Algorithm design and integration for a robotic apple harvesting system

Authors: ZHANG, KAIXIANG - Michigan State University; LAMMERS, KYLE - Michigan State University; CHU, PENGYU - Michigan State University; DICKINSON, NATHAN - Michigan State University; LI, ZHAOJIAN - Michigan State University; Lu, Renfu

Submitted to: IEEE RSJ International Conference on Intelligent Robots and Systems
Publication Type: Proceedings
Publication Acceptance Date: 6/30/2022
Publication Date: 10/23/2022
Citation: Zhang, K., Lammers, K., Chu, P., Dickinson, N., Li, Z., Lu, R. 2022. Algorithm design and integration for a robotic apple harvesting system. In: Proceedings of 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 23-27, 2022, Kyto, Japan. p. 9217-9224. https://ieeexplore.ieee.org/document/9981417.

Interpretive Summary: Apple production is still labor intensive and harvesting alone accounts for about 15% of total production cost in the U.S. With increased labor shortage and higher labor cost, the U.S. apple industry urgently needs automated systems to harvest apples. In this paper, we report on a new, improved algorithm design and systems integration for our recently developed robotic apple harvester prototype. A robust calibration method, an improved multi-viewing perception system with a deep learning-based algorithm for fruit detection and localization, and an efficient fruit dropping module were designed and implemented. A unified planning algorithm for fruit picking and releasing, coupled with an efficient nonlinear control scheme, was proposed to enhance fruit picking rate by the harvesting robot. Experiments were conducted for the robot prototype integrated with the new algorithm and system improvements under both indoor simulated and real orchard environments during the 2021 harvest season. Results showed that the multi-viewing perception system achieved 93.9% fruit detection rate compared to 90.5% detection rate with a single vision system. The harvesting robot achieved an average picking rate of 3.6 seconds per fruit, a significant improvement over the previous version of the robot, and it also compares favorably to other reported harvesting robots with a picking rate in the range of 7-10 seconds per fruit. Field experiments also identified several areas for improvement. This research represents an important step towards the goal of developing a commercially viable robotic harvesting technology for the U.S. apple industry.

Technical Abstract: Due to labor shortage and rising labor cost for the U.S. apple industry, there is an urgent need for the development of commercially viable robotic systems to harvest apples. In this paper, we present a system overview and algorithm design of our recently developed robotic apple harvester prototype. It covers the main methods and advancements in robust extrinsic parameter calibration, deep learning-based multi-view fruit detection and localization, unified picking and dropping planning, and dexterous manipulation control. Our robotic system integrates several core modules, including calibration, visual perception, planning, and control. Indoor and field experiments were conducted to evaluate the performance of the developed system, which achieved an average picking rate of 3.6 seconds per apple. This is a significant improvement over other reported apple harvesting robots with a picking rate in the range of 7-10 seconds per apple. The current prototype shows promising performance towards further development of efficient and automated apple harvesting technology. Finally, limitations of the current system and future work are discussed.