Animal Waste Management Research Site Logo
ARS Home About Us Helptop nav spacerContact Us En Espanoltop nav spacer
Printable VersionPrintable Version     E-mail this pageE-mail this page
Agricultural Research Service United States Department of Agriculture
Search
  Advanced Search
 
Programs and Projects
Subjects of Investigation
Research Information for Stakeholders
 

Research Project: EFFICIENT MANAGEMENT AND USE OF ANIMAL MANURE TO PROTECT HUMAN HEALTH AND ENVIRONMENTAL QUALITY

Location: Animal Waste Management Research

Title: Weighting formulas for the least-squares analysis of binding phenomena data

Authors
item Tellinghuisen, Joel - VANDERBILT UNIVERSITY
item Bolster, Carl

Submitted to: Journal of Physical Chemistry
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: February 20, 2009
Publication Date: April 4, 2009
Repository URL: 235472 - http://hdl.handle.net/10113/43804
Citation: Tellinghuisen, J., Bolster, C.H. 2009. Weighting formulas for the least-squares analysis of binding phenomena data. Journal of Physical Chemistry. 113:6151-6157.

Interpretive Summary: The Langmuir equation is of the form of a rectangular hyperbola, y = a b x /(1 + b x ), and is widely used for fitting phosphorus sorption data to obtain model soil sorption constants. The most common approach for fitting this model to data is through the use of least-squares analysis. However, it is commonly the case in most sorption studies that the "independent variable" is actually a directly measured quantity, and the dependent variable is simply a computed function of the independent variable. These circumstances violate one of the fundamental tenets of most least-squares methods — that the independent variable be error-free. We treat this problem by deriving weighting formulas for the least-squares analysis of such data for the rectangular hyperbola and all of its common linearized versions — the double reciprocal, y-reciprocal, and x-reciprocal forms. We verify the correctness of these expressions by computing the nonlinear least squares parameter standard errors for exactly fitting data, and we confirm their utility through Monte Carlo simulations. Our findings will improve the statistical treatment of phosphorus sorption data.

Technical Abstract: The rectangular hyperbola, y = a b x /(1 + b x ), is widely used as a fit model in the analysis of data obtained in studies of complexation, sorption, fluorescence quenching, and enzyme kinetics. Frequently the "independent variable" x is actually a directly measured quantity, and y may be a simply computed function of x, like y = x0 – x. These circumstances violate one of the fundamental tenets of most least squares methods — that the independent variable be error-free — and they lead to fully correlated error in x and y. Using an effective variance approach, we treat this problem to derive weighting formulas for the least-squares analysis of such data by the given equation and by all of its common linearized versions — the double reciprocal, y-reciprocal, and x-reciprocal forms. We verify the correctness of these expressions by computing the nonlinear least squares parameter standard errors for exactly fitting data, and we confirm their utility through Monte Carlo simulations. The latter reveal a problem with the inversion methods when the inverted data are moderately uncertain (~30%), leading to the recommendation that the reciprocal methods not be used for such data. For benchmark tests, results are presented for specific data sets having error in x alone and in both x and x0. The actual estimates of a and b and their standard errors vary somewhat with the choice of fit model, with one important exception: The Deming/Lybanon algorithm treats multiple uncertain variables equivalently and returns a single set of parameters and standard errors independent of the manner in which the fit model is expressed.

   

 
Project Team
Sistani, Karamat
Loughrin, John
Bolster, Carl
Cook, Kimberly - Kim
Lovanh, Nanh
 
Publications
   Publications
 
Related National Programs
  Food Safety, (animal and plant products) (108)
  Air Quality (203)
  Manure and Byproduct Utilization (206)
 
 
Last Modified: 05/25/2013
ARS Home | USDA.gov | Site Map | Policies and Links 
FOIA | Accessibility Statement | Privacy Policy | Nondiscrimination Statement | Information Quality | USA.gov | White House