Technical Abstract: The Conservation Effects Assessment Project (CEAP) was established to quantify water quality benefits of voluntary agricultural conservation practices supported through U.S. Department of Agriculture (USDA) financial incentives. In 2004, a set of watershed assessment studies were begun in large, rain-fed agricultural watersheds of the U.S. with varying landscapes, climates, and cropping systems, and a range of water quality concerns. This paper reviews results from the USDA Agricultural Research Service (ARS) ‘benchmark’ watershed assessment studies and examines the challenge of identifying water quality impacts in large watersheds. Watershed study goals included watershed-scale modeling, field research to assess practices and ecosystem responses, and evaluation of practice placement in watersheds. These goals could not all be met within five years, but important lessons were learned to inform policy and further research. Field and modeling studies showed practices can improve water quality, but water quality problems persist in larger agricultural watersheds. This apparent dissociation between practice-focused assessment and watershed monitoring occurred because: 1) Conservation practices were not targeted according to critical sources and pathways of contaminants; 2) Sediment in streams often originated more from channel and bank erosion than from erosion of soil in fields; 3) Timing lags, historical legacies, and shifting climate combined to mask effects of practice implementation; and 4) Water quality management strategies address single contaminants and may neglect inherent trade-offs among contaminants. These lessons can be leveraged to improve strategies for implementing conservation programs and to set water quality goals with realistic timelines. Continued research on agricultural water quality could be improved by better integrating modeling and monitoring capabilities, and by addressing provisioning of ecosystem services in agricultural watersheds.