Location: Food Animal Environmental Systems Research Unit
Title: Weighting formulas for the leastsquares analysis of binding phenomena data
AuthorsTellinghuisen, Joel  VANDERBILT UNIVERSITY  
Bolster, Carl 
Submitted to: Journal of Physical Chemistry
Publication Type: Peer Reviewed Journal Publication Acceptance Date: February 20, 2009 Publication Date: April 4, 2009 Repository URL: 235472  http://hdl.handle.net/10113/43804 Citation: Tellinghuisen, J., Bolster, C.H. 2009. Weighting formulas for the leastsquares analysis of binding phenomena data. Journal of Physical Chemistry. 113:61516157. Interpretive Summary: The Langmuir equation is of the form of a rectangular hyperbola, y = a b x /(1 + b x ), and is widely used for fitting phosphorus sorption data to obtain model soil sorption constants. The most common approach for fitting this model to data is through the use of leastsquares analysis. However, it is commonly the case in most sorption studies that the "independent variable" is actually a directly measured quantity, and the dependent variable is simply a computed function of the independent variable. These circumstances violate one of the fundamental tenets of most leastsquares methods — that the independent variable be errorfree. We treat this problem by deriving weighting formulas for the leastsquares analysis of such data for the rectangular hyperbola and all of its common linearized versions — the double reciprocal, yreciprocal, and xreciprocal forms. We verify the correctness of these expressions by computing the nonlinear least squares parameter standard errors for exactly fitting data, and we confirm their utility through Monte Carlo simulations. Our findings will improve the statistical treatment of phosphorus sorption data. Technical Abstract: The rectangular hyperbola, y = a b x /(1 + b x ), is widely used as a fit model in the analysis of data obtained in studies of complexation, sorption, fluorescence quenching, and enzyme kinetics. Frequently the "independent variable" x is actually a directly measured quantity, and y may be a simply computed function of x, like y = x0 – x. These circumstances violate one of the fundamental tenets of most least squares methods — that the independent variable be errorfree — and they lead to fully correlated error in x and y. Using an effective variance approach, we treat this problem to derive weighting formulas for the leastsquares analysis of such data by the given equation and by all of its common linearized versions — the double reciprocal, yreciprocal, and xreciprocal forms. We verify the correctness of these expressions by computing the nonlinear least squares parameter standard errors for exactly fitting data, and we confirm their utility through Monte Carlo simulations. The latter reveal a problem with the inversion methods when the inverted data are moderately uncertain (~30%), leading to the recommendation that the reciprocal methods not be used for such data. For benchmark tests, results are presented for specific data sets having error in x alone and in both x and x0. The actual estimates of a and b and their standard errors vary somewhat with the choice of fit model, with one important exception: The Deming/Lybanon algorithm treats multiple uncertain variables equivalently and returns a single set of parameters and standard errors independent of the manner in which the fit model is expressed.
