Forum—National Peer Review Process Sharpens Our Science

...It's the best research plan you've ever written as an ARS scientist. Your write-up went to a panel of your peers. If things went as usual, the panelists spent about an hour scrutinizing your plan in a lively discussion. Now you're reading their review. They approve of your research approach and procedures. They've given you thumbs up for the merit and significance of your research plan and its alignment with the ARS National Program in your subject area. And the panelists have determined that you have a high probability of accomplishing your project's objectives. You're especially pleased that the panel recommends establishing a new research collaboration with a university colleague on the other coast. It's something you've had in the back of your mind for a while. But this green light is the boost you need to make the contact. You begin drafting your response to the panel...

Scenarios like this one are now occurring at ARS laboratories across the nation. That's in response to our improved process for peer review, instituted to ensure research quality. The experts who make up these panels are providing candid, comprehensive, and constructive assessments of ARS scientists' multiyear research plans. In fact, the procedure may be one of the best-ever opportunities for our scientists to obtain extremely well-thought-out feedback for improving the scientific basis of their plans.

Now in its third year, the procedure is generating fresh ideas, encouraging creativity, and identifying alternative approaches and analytical techniques. In brief, here's how it works:

Panels of four to seven members review, over several days, project plans that are within the same National Program of research. ARS has 22 of these National Programs. Each panel provides in-depth suggestions and recommendations for up to 25 research plans. Scientists respond in writing, modifying their research plans accordingly.

Panelists include well-recognized authorities—such as university professors, federal scientists, and industry consultants—from relevant disciplines. For objectivity and credibility, their names and affiliations remain unknown to the scientists whose work they evaluate.

To date, peer panels have reviewed more than 250 agency research projects, including those in such National Programs as Food Safety (#108), Animal Health (#103), Air Quality (#203), Water Quality and Management (#201), and Plant Biological and Molecular Processes (#302). The completed reviews describe the work of some 900 scientists, technicians, and other staff whose specialties range from agronomy to veterinary medicine.

A common panel recommendation advises scientists to consider research plans—or protocols—not included in the original project. In other instances, reviewers may have praise for the protocol already in place. As one panel noted, for example, "This entire objective is very high risk, but the payoff is potentially high. The plan articulates a clear, stepwise protocol."

Another frequent recommendation counsels scientists to set up collaborations with groups of scientists at different institutions. For instance, one panel observed, "Collaborations between ARS scientists appear to be well established and functional. However, the group could benefit from international collaboration with scientists working in similar scales and settings." Comments such as these result from the spirited exchanges of panel members meeting around the table to challenge each other's ideas. We think this group approach is proving superior to having individual reviewers work alone.

There are other benefits of our revamped review process. By law, the majority of panel members must come from outside ARS. This promotes impartiality and minimizes conflicts of interest. What's more, this exposure of ARS research to outsiders gives these pre-eminent professionals an insider's look at the scope and quality of our work. Their favorable reviews reinforce agency scientists' morale and enhance recruitment of top new talent.

Requiring a written response to panel reviews ensures that each recommendation gets scientists' close attention and consideration. A surprise benefit: scientists report that the thinking and writing they put into the process has greatly improved their grant-writing skills.

ARS developed the national peer review in response to the mandates of the Agricultural Research, Extension, and Education Reform Act of 1998 (Public Law 105-185). Our approach is part of an ongoing effort to increase the federal government's accountability to every client, most notably the American taxpayer. Our new system complies with all the Act's requirements, including that all ARS research be reviewed every 5 years.

We submit results of the reviews to the National Agricultural Research, Extension, Education, and Economics Advisory Board. Several members of that board have commented favorably on the rigor and integrity of the new peer review process.

More information appears at a web site run by the ARS Office of Scientific Quality Review, the staff that manages the process: http://www.ars.usda.gov/osqr. The National Programs under which the reviews are clustered are described on the World Wide Web at http://www.nps.ars.usda.gov.

Edward B. Knipling
Acting Administrator
Agricultural Research Service
Washington, D.C.

"Forum" was published in the May 2002 issue of Agricultural Research magazine.