Page Banner

United States Department of Agriculture

Agricultural Research Service


Location: Sustainable Biofuels and Co-Products

Title: On the proper use of tools

item Haas, Michael

Submitted to: European Journal of Lipid Science and Technology
Publication Type: Other
Publication Acceptance Date: November 29, 2010
Publication Date: December 9, 2010
Citation: Haas, M.J. 2010. On the proper use of tools. European Journal of Lipid Science and Technology. 112(12):1287-1289.

Technical Abstract: In the ‘Odds and Ends’ drawer of my desk is an old multi-bladed pocket knife, made of high quality steel. Although its two smallest blades are intact and sharp, its larger main blade is snapped off, a stub compared to its original length. This stub is a testament to the fact that even very good knives make very poor prybars. Similarly, in my gun cupboard sits a shotgun, many of whose screws have badly chipped and rounded slots for the corresponding screwdriver. These are a permanent testimony to the day that, as a much younger man, I learned that most screwdrivers and their companion screws have tapered blades and correspondingly tapered slots, but that firearms screws and screwdrivers have parallel faces. Regular screwdrivers therefore do not properly fit the screws in firearms, and when used in this improper application can slip out and destroy the screw head. These two examples illustrate the fact that a good tool used inappropriately is a recipe for disaster. In a Letter to the Editor in this issue, Dr. Albert Dijkstra essentially observes that the same goes for the tools we use in our research. He makes this observation in regard to response surface methodology (RSM), which, to paraphrase Wikipedia, is a statistics-based experimental method that explores the relationships between multiple experimental variables and one or more response variables. Response Surface Methodology involves the use of a sequence of designed experiments to identify the conditions that will give an optimal response. The technique has become popular in some types of research because it can produce a maximum amount of data from a relatively small number of reactions, and can yield powerful predictive results. In that it allows identification of interactions between multiple variables in an experimental design, RSM offers a power that is lacking in the more conventional approach of changing one variable at a time. An experienced scientist, Dr. Dijkstra has spent a long and productive career conducting research and managing research teams, projects and entire research centers. He is very good at what he does, and he is widely regarded as a wise and accomplished man. In his letter he presents a multi-faceted commentary on the use, or rather the misuses, of statistically designed experimental planning and the consequent application of response surface methodology to the results. I have a high regard for the author of the Letter, a skilled and wise researcher with a multitude of accomplishments in science and technology. Also, I agree with several of the cautions voiced in his Letter. But, as in looking at a glass with water in it and seeing it as either half empty or half full, I see some of the points raised in the Letter in a somewhat different light than does the author. After reading the Letter, please consider the following comments, whose numbers correspond to the similarly numbered items there: 1. Dubious significance: Here the author of the Letter observes that by addition of extra terms to a data fitting equation one can reduce the apparent standard error of the fit to the data. It is correctly noted that to do so without increasing the number of experimental determinations as well is ill advised. Perhaps RSM has become too easy to apply, and thus too easy to apply incorrectly, such as in without appreciating this fact. Some contemporary RSM programs will help recognize such a situation as this, reporting an ‘adjusted’ R2 value to alert the researcher to the presence of an excessive number of terms in the equation of fit. The Letter’s admonishment to conduct multiple repeat determinations in order to determine the standard error of the measurements is wise advice, best heeded by all who implement RSM. Also, the observation is well made that just because your computer will print results out to seven digits does not mean that the last four or five have any meaning or should be reported. 2. Reproducibility: This comment alleges that it would be wise to repeat entire RSM experiments, in order to determine if the original set of data is accurate. It also speculates that researchers don’t usually do so for fear that their encouraging first set of results may be thereby proven incorrect. I suspect that this is no greater a reason for a lack of duplication than with other experimental design methods (a poor researcher is a poor researcher largely irrespective of his research tools). In fact a well designed RSM protocol has sufficient internal controls that a single run of the experiment gives a good feeling for the reliability of the data. 3. Repeatability: An argument for the validation of RSM results by the use of multiple labs to repeat the work. Such validation would add reliability to all research data, not just that from RSM. I do not see a reason to demand more validation from RSM than we routinely demand from other types of experimental design. 4. The use of absolute values: An argument for conducting experiments in a varying order, to minimize the effects of long term fluctuations. Again, this seems to demand of RSM that which we do not routinely demand of other experimental methods. In fact, the Letter goes on to acknowledge that the better practitioners of RSM adopt a random order in conducting their experiments, to minimize the effect of such fluctuations. 5. The choice of process variables: The author of the Letter observes that that in applying an RSM approach one may ignore the investigation of some major parameters affecting the system. I agree, but again cannot see this as a problem unique to RSM. Failure to correctly perceive and investigate the parameters most likely to impact our systems is a danger we all face, whether we use RSM or not. Should I choose to explore whether my tomatoes grow well if I read to them from Shakespeare, Goethe, or Aldo Leopold, while failing to provide sunlight and water, neither RSM nor any other experimental design approach will prevent the world from judging me a fool. 6. The value of process variables: And yes, if I should choose to study the effects of water, but investigate the effects of only two levels, 0 and 25 liters per day per two liter pot, I should not be surprised if the plants die under both treatment regimes and I fail to identify water as a requirement in tomato agriculture. The Letter’s caution here that it is very important to have a feel for the process before selecting the variables and their settings is excellent. RSM provides the world no more insulation against shabby researchers than do other experimental approaches. 7. RSM is superficial: This section faults RSM methods for being applicable in cases where one has no hypothetical mechanism from which to identify variables, interactions, and relationships. Exercise more of the familiarization with your system that is suggested in item no. 6 and you may come up with a hypothesis to guide a better RSM examination. Even so, I am not convinced that the ability to function in the absence of hypothesis is necessarily fatal. If one designs and conducts an RSM experiment for which major effecting variables are not examined, the fit of the results to the data will be poor, and the serious investigator will immediately see the need for more study to flesh out the real impacting variables in the system. 8. Non-elimination of outliers/ rogue values: Here the author of the letter relates his past improvement of the quality of a data set by deleting a data point that his experience led him to recognize as an outlier. He faults RSM for lacking this capability. In fact, some RSM programs can conduct outlier searches and alert the experimenter of their presence. More importantly, RSM does not remove the researcher from the dance, does not allow him or her to retire their powers of observation from the field of battle. In fact, I am unaware of any experimental approach in which this would be ad

Last Modified: 11/26/2015