Location: Nutrient Data
Title: Validation study of the USDA’s data quality evaluation system Authors
Submitted to: Journal of Food Composition and Analysis
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: March 25, 2008
Publication Date: June 9, 2008
Citation: Bhagwat, S.A., Patterson, K.K., Holden, J.M. 2009. Validation study of the USDA’s data quality evaluation system. Journal of Food Composition and Analysis. http://dx.doi.org/10.1016/j.jfca.2008.06.009. Interpretive Summary: The Nutrient Data laboratory (NDL) of USDA is responsible for the National Nutrient Database for Standard Reference (SR). The NDL also prepares databases for emerging bioactive compounds. Data used in these databases is collected from various sources, including analytical data from NDL’s NFNAP (National Food and Nutrient Analysis Program) program, from scientific journals, from manufacturers etc. To assure quality of these data NDL has developed a data quality evaluation system. The system consists of 5 evaluation categories: sampling plan, sample handling, number of samples, analytical method, and analytical quality control. A series of algorithms to address critical issues within each category and to generate ratings for each category have been developed. We conducted a study to validate the objectivity of the system modules, robustness of the rating scales, and variability of responses by different individuals while evaluating the same published articles. 39 individuals from 13 different countries involved in food composition work and 4 individuals from NDL participated in the study and evaluated 3 published articles, one on vitamin K in leaf lettuce, one on catechin in grapes, and one on riboflavin in mushrooms. Preliminary observations revealed reasonable consistency in the ratings of analytical method and sampling plan ratings for both the nutrients. But there was considerable variability in the ratings of sample handling, analytical quality control, and number of samples. These categories need clearer sets of questions and better elaboration of definitions. Final confidence codes did not differ considerably in spite of variability in the ratings of some categories. Therefore equal emphasis on all categories needs reconsideration.
Technical Abstract: The Nutrient Data Laboratory (NDL) of USDA conducted a validation study of the USDA Data Quality Evaluation System (DQES). The system evaluates the quality of analytical data by rating important documentation concerning the analytical method, analytical quality control, number of samples, sampling plan, and sample handling and creates “Quality Index” and “Confidence Code” for each nutrient and food. Objectives of this validation study were to: 1)measure the variability of ratings assigned by different evaluators, 2) assess the objectivity of the “critical” questions of the DQES categories, and 3)test the robustness of the rating scale. Methods: Out of 39 individuals who participated in the International Postgraduate Course for Food Composition and four nutritionists from NDL, 37 completed an exercise using the DQES evaluating a research article containing analytical data on vitamin K in lettuce, 25 evaluated an article containing data on catechin in black grapes, and 16 evaluated an article containing data on riboflavin in portabella mushrooms. The various rating scores assigned by the participants were analyzed to assess the success of above objectives. The maximum score for each category was 20. Results: There was a greater variation among individuals in the ratings for “number of samples” and “analytical quality control” categories of data source evaluation than the other three categories. Overall, seventy five percent of the participants assigned a confidence code of “C” to the vitamin K in lettuce, while 70% and 100% assigned confidence codes of “B” and “C” to catechin in black grapes and riboflavin in mushrooms, respectively. Conclusions: Consistent results can be obtained by different evaluators. Clear documentation by the authors and training for evaluators to understand basic concepts related to data quality evaluation will be helpful. Future work will assess the assignment of equal rating points for all the categories. Modifications in “sampling plan” category to accommodate country and population size are necessary. The USDA Data Quality Evaluation System represents one of the first efforts to standardize and harmonize the evaluation of analytical data quality across the international food composition network.