ForumNational Peer Review Process Sharpens Our Science
...It's the best research plan you've ever written as an ARS scientist. Your write-up went to a panel of your peers. If things went as usual, the panelists spent about an hour scrutinizing your plan in a lively discussion. Now you're reading their review. They approve of your research approach and procedures. They've given you thumbs up for the merit and significance of your research plan and its alignment with the ARS National Program in your subject area. And the panelists have determined that you have a high probability of accomplishing your project's objectives. You're especially pleased that the panel recommends establishing a new research collaboration with a university colleague on the other coast. It's something you've had in the back of your mind for a while. But this green light is the boost you need to make the contact. You begin drafting your response to the panel...
Scenarios like this one are now occurring at ARS laboratories across
the nation. That's in response to our improved process for peer review,
instituted to ensure research quality. The experts who make up these
panels are providing candid, comprehensive, and constructive assessments
of ARS scientists' multiyear research plans. In fact, the procedure
may be one of the best-ever opportunities for our scientists to obtain
extremely well-thought-out feedback for improving the scientific basis
of their plans.
Now in its third year, the procedure is generating fresh ideas, encouraging
creativity, and identifying alternative approaches and analytical techniques.
In brief, here's how it works:
Panels of four to seven members review, over several days, project
plans that are within the same National Program of research. ARS has
22 of these National Programs. Each panel provides in-depth suggestions
and recommendations for up to 25 research plans. Scientists respond
in writing, modifying their research plans accordingly.
Panelists include well-recognized authoritiessuch as university
professors, federal scientists, and industry consultantsfrom relevant
disciplines. For objectivity and credibility, their names and affiliations
remain unknown to the scientists whose work they evaluate.
To date, peer panels have reviewed more than 250 agency research projects,
including those in such National Programs as Food Safety (#108), Animal
Health (#103), Air Quality (#203), Water Quality and Management (#201),
and Plant Biological and Molecular Processes (#302). The completed reviews
describe the work of some 900 scientists, technicians, and other staff
whose specialties range from agronomy to veterinary medicine.
A common panel recommendation advises scientists to consider research
plansor protocolsnot included in the original project. In
other instances, reviewers may have praise for the protocol already
in place. As one panel noted, for example, "This entire objective
is very high risk, but the payoff is potentially high. The plan articulates
a clear, stepwise protocol."
Another frequent recommendation counsels scientists to set up collaborations
with groups of scientists at different institutions. For instance, one
panel observed, "Collaborations between ARS scientists appear to
be well established and functional. However, the group could benefit
from international collaboration with scientists working in similar
scales and settings." Comments such as these result from the spirited
exchanges of panel members meeting around the table to challenge each
other's ideas. We think this group approach is proving superior to having
individual reviewers work alone.
There are other benefits of our revamped review process. By law, the
majority of panel members must come from outside ARS. This promotes
impartiality and minimizes conflicts of interest. What's more, this
exposure of ARS research to outsiders gives these pre-eminent professionals
an insider's look at the scope and quality of our work. Their favorable
reviews reinforce agency scientists' morale and enhance recruitment
of top new talent.
Requiring a written response to panel reviews ensures that each recommendation
gets scientists' close attention and consideration. A surprise benefit:
scientists report that the thinking and writing they put into the process
has greatly improved their grant-writing skills.
ARS developed the national peer review in response to the mandates
of the Agricultural Research, Extension, and Education Reform Act of
1998 (Public Law 105-185). Our approach is part of an ongoing effort
to increase the federal government's accountability to every client,
most notably the American taxpayer. Our new system complies with all
the Act's requirements, including that all ARS research be reviewed
every 5 years.
We submit results of the reviews to the National Agricultural Research,
Extension, Education, and Economics Advisory Board. Several members
of that board have commented favorably on the rigor and integrity of
the new peer review process.
More information appears at a web site run by the ARS Office of Scientific Quality Review, the staff that manages the process: http://www.ars.usda.gov/osqr. The National Programs under which the reviews are clustered are described on the World Wide Web at http://www.nps.ars.usda.gov.
Edward B. Knipling
"Forum" was published in the May 2002 issue of Agricultural Research magazine.