Foundation model-based apple ripeness and size estimation for selective harvesting

Authors: ZHU, KEYI - Michigan State University; LI, JIAJIA - Michigan State University; ZHANG, KAIXIANG - Michigan State University; ARUNACHALAM, CHAARAN - Michigan State University; BHATTACHARYA, SIDDHARTHA - Michigan State University; Lu, Renfu; LI, ZHAOJIAN - Michigan State University

Submitted to: Computers and Electronics in Agriculture
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 4/11/2025
Publication Date: 4/26/2025
Citation: Zhu, K., Li, J., Zhang, K., Arunachalam, C., Bhattacharya, S., Lu, R., Li, Z. 2025. Foundation model-based apple ripeness and size estimation for selective harvesting. Computers and Electronics in Agriculture. 236. Article 110407. https://doi.org/10.1016/j.compag.2025.110407.
DOI: https://doi.org/10.1016/j.compag.2025.110407

Interpretive Summary: Harvesting is a single largest cost in apple production, requiring a significant amount of manual labor. Automated harvesting technology is thus urgently needed to reduce the fruit industry’s reliance on labor and address rising labor and production costs. Currently, many robotic harvesting systems are being developed, but they often indiscriminately harvest fruits regardless of their ripeness and/or quality grade such as color and size. This could result in increased postharvest handling cost and lower product quality and thus profitability for growers. This study introduces a novel foundation model-based framework for effective estimation of apple ripeness and size on trees. Two publicly available ‘Fuji’ apple image datasets collected at two different geographic locations were curated, integrated, and annotated to generate one single comprehensive dataset, which contains more than 4,000 images and 16,000 fruits labeled with fruit size and ripeness based on image color and capture date. Grounding-DINO, a language model-based object detector, was used for robust fruit detection and ripeness classification (i.e., ‘Ripe’ versus ‘Unripe’), which outperformed other state-of-the-art AI models. In addition, we also evaluated six fruit size estimation algorithms and identified an optimal model with low estimation errors. The newly annotated and integrated apple image dataset and fruit ripeness and size estimation algorithms are made publicly available, which provide valuable benchmarks for future studies in automated and selective harvesting of apples.

Technical Abstract: Harvesting is a critical task in the tree fruit industry, requiring extensive manual labor and substantial costs and exposing workers to potential hazards. Recent advances in automated harvesting offer a promising solution by enabling efficient, cost-effective, and ergonomic fruit picking within tight harvesting windows. However, existing harvesting technologies often indiscriminately harvest fruits, including those that are unripe or undersized, which increase postharvest handling costs and lower product quality and thus profitability for growers. This study introduces a novel foundation model-based framework for effective apple ripeness and size estimation. Specifically, we curated two publicly available RGBD-based Fuji apple image datasets, integrating expanded annotations for ripeness ("Ripe" vs. "Unripe") based on fruit color and image capture dates. The resulting comprehensive dataset, Fuji-Ripeness-Size Dataset, includes 4,027 images and 16,257 apples with ripeness and size labels. Using Grounding-DINO, a language-model-based object detector, we achieved robust apple detection and ripeness classification, outperforming other state-of-the-art models. In addition, we developed and evaluated six fruit size estimation algorithms, selecting the one with the lowest mean average error and standard deviation for optimal performance. The Fuji-Ripeness-Size Dataset and the apple detection and size estimation algorithms are made publicly available, which provides valuable benchmarks for future studies in automated and selective apple harvesting.