Detect, segment, cluster: Apple localization for robotic harvesting in complex orchards

Authors: BHATTACHARYA, SIDDHARTHA - Michigan State University; ARUNACHALAM, CHAARAN - Michigan State University; ZHANG, KAIXIANG - Michigan State University; LI, JIAJIA - Michigan State University; Lu, Renfu; LI, ZHAOJIAN - Michigan State University

Submitted to: Computers and Electronics in Agriculture
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 11/16/2025
Publication Date: 11/19/2025
Citation: Bhattacharya, S., Arunachalam, C., Zhang, K., Li, J., Lu, R., Li, Z. 2025. Detect, segment, cluster: Apple localization for robotic harvesting in complex orchards. Computers and Electronics in Agriculture. 12. Article 101642. https://doi.org/10.1016/j.atech.2025.101642.
DOI: https://doi.org/10.1016/j.atech.2025.101642

Interpretive Summary: Automated harvesting of apples and other tree fruits is urgently needed to control the rising production cost and mitigate labor shortage for U.S. fruit growers. Accurate detection and three-dimensional (3D) localization of apples in complex orchard environments, where fruit occlusions by leaves and branches are commonplace, is critical to successful robotic harvesting. Different fruit localization methods have been developed by leveraging color and 3D depth sensors and deep learning algorithms. However, they often fail to provide sufficient localization accuracies due to fruit clustering, leave/branch occlusions and noisy depth data. To overcome these limitations, we proposed a new, unified 3D localization framework that seamlessly integrates a foundation vision-language model for apple detection, a high-resolution instance segmentation model for apple segmentation, and a spatial clustering technique for refining the 3D position of target apples on trees. To evaluate the localization performance of the new method, we collected 2D color and 3D depth images from 219 real and artificial apples in different indoor and outdoor environments. The new model was evaluated against three state-of-the-art models using the collected image data. Results showed that our model has achieved a mean absolute localization error of 7.69 mm, which represents a 74.5% improvement over the baseline models. The new 3D localization method has the potential to significantly improve the performance of the robotic apple harvester developed by our research team. This would help us achieve the goal of efficient, robust commercial harvesting of apples.

Technical Abstract: Efficient robotic apple harvesting hinges on robust 3D perception systems capable of accurately identifying and localizing fruits in complex orchard environments. Different localization algorithms leveraging RGB-D sensors and deep learning methods have been developed through extrapolating 2D detections into the 3D space. But they often suffer from occlusions and noisy depth data, resulting in limited real-world accuracy. To overcome these limitations, we proposed a unified 3D localization framework that seamlessly integrates vision-language model (VLM)-based apple detection, instance segmentation, and unsupervised point-cloud clustering. Our method first detects and segments apples in 2D images using a foundation VLM (i.e., Grounding-DINO) and a high-resolution instance segmentation model. The identified apple pixels are then projected into the 3D space, where Density-Based Spatial Clustering of Applications with Noise (DBSCAN) are used to refine localization by filtering out noise and unreliable depth measurements. We benchmark our approach against conventional 2D-to-3D extrapolation and 3D localization techniques. Evaluated on a dataset of RGB-D images containing 219 apples collected from indoor and outdoor environments, including both real and artificial apples with varying degrees of leave/branch occlusions, our method achieved a mean absolute localization error of 7.69 mm, which represents a 74.5% improvement over the best-performing baseline methods. The proposed 3D localization method has potential to significantly improve the performance of robotic apple harvesters.