Location: Sugarbeet and Bean Research
2024 Annual Report
Objectives
1. Enable a new, efficient, and cost-effective robotic technology, coupled with automated infield sorting and quality tracking technologies, for commercial harvesting of apples.
2. Develop a new imaging technology, based on structured illumination integrated with artificial intelligence and advanced data analytics, with substantially improved capabilities for commercial quality inspection of fruits and vegetables.
Approach
Development of enabling technologies for automated fruit harvesting and nondestructive quality inspection during postharvest handling can provide an effective solution to the labor availability and cost issues, and enhance production efficiency, product quality, and thus profitability and sustainability for the specialty crop industries. In recent years, much research has been focused on fruit robotic harvesting, but the progress has been slow and unsatisfactory in meeting industry needs, mainly due to the several key technical hurdles encountered in robotic perception (identifying and localizing fruit), manipulation (reaching out for and picking fruit), and systems integration and coordination. While machine vision technology is widely used for postharvest quality inspection of horticultural products, it still is short of meeting industry expectations in detection of quality-degrading defects and symptoms. This research is therefore aimed at developing a new, cost effective robotic technology for automated harvesting of apples and a new generation imaging technology with substantially enhanced capabilities for quality inspection of fruits and vegetables (e.g., pickling cucumber and tomato) during postharvest handling. Innovative concepts and designs, coupled with artificial intelligence, will be used in the development of the new robotic harvesting system for fruit imaging, detection, localization, and picking. The new robotic system will be integrated with the recently developed apple harvest and infield sorting machine, to enable automated harvesting, sorting, grading and tracking of apples in the orchard. Moreover, a new imaging system, using our newly developed technique on improved reflectance for imaging structures, will be assembled to enable rapid, real-time inspection of harvested horticultural products for quality-degrading defects caused by bruising, physiological disorders, and disease infection. The new knowledge and technologies generated from the research will enable growers and packers/processors to achieve significant labor and cost savings in harvesting, enhance product marketability, and reduce postharvest product loss.
Progress Report
OBJECTIVE 1: Robotic harvesting needs to be cost effective and efficient in fruit harvesting to meet grower expectations. Modular designs are advantageous in reducing overall system costs and enabling better, or easier, adaption to, or integration with, different harvesting platforms. Towards meeting this goal and built on the promising performance of the single-arm harvesting robot developed earlier, we have designed and built a new dual-arm robotic harvesting system for enhancing fruit harvest efficiency. The dual-arm harvesting robot shares one common perception system and one central vacuum system, thus making the system more compact, more energy efficient and cost effective, compared to the single-arm robotic system. A new version of the perception module with two line lasers, based on our innovative Active LAser-Camera Scanning (ALACS) technology, was designed and assembled to provide fruit localization information for the two robot arms to pick target fruits. New planning/coordination algorithms were developed for control and coordination of the two arms for fruit harvesting.
Furthermore, a fruit handling system has been assembled and integrated with the harvesting robot for receiving and transporting picked fruits to the bin. Field tests were conducted of the dual-arm harvesting robot in orchards with complex tree canopies during 2023 harvesting season. The harvesting robot achieved 60% successful picking rate and improved harvesting efficiency by 9% to 34%, compared to the single-arm configuration. A non-provisional patent application has been filed for the new dual-arm robotic harvesting system.
Based on the promising performance of the dual-arm robot in the field harvesting test, we are redesigning the control of the vacuum system to further improve the harvesting efficiency of the dual-arm robot. New controls and control logic are being incorporated with the vacuum system for more efficient coordination of the two robot arms for faster apple picking. In addition, efforts have been made to develop new, improved algorithms for more accurate detection and localization of fruits. Different image segmentation techniques were developed and compared, and improved fruit segmentation results were obtained, compared to the current segmentation algorithm. Furthermore, new algorithms for detecting and localizing branches, based on image segmentation and deep learning techniques, are being developed, which are important to the development of an effective obstacle avoidance algorithm for robotic picking of fruits that are occluded or blocked by branches.
OBJECTIVE 2: Structured-light imaging (SLI), as a promising emerging technique, enables more effective detection of many types of subsurface and surface defects of horticultural and food products, which may be difficult to detect using conventional imaging techniques. The conventional SLI technique, however, can only be implemented when objects or products are in stationary status. We have made progress on implementing the SLI technique in online settings, where samples are moving. An online SLI prototype has been constructed, which enables collection of pattern images from moving samples at different conveyor speeds. Theoretical and experimental analyses were conducted to select appropriate parameters related to the acquisition of pattern images from moving samples. The parameters studied include conveyor speed, spatial frequency of the structured light, orientation of structured light patterns (parallel and perpendicular to the moving direction), frequency of the structured-light pattern shift, and imaging frame rate by the camera and its synchronization with the light projector. Experiments were performed with reference and apple samples to determine the optimum combinations of the parameters for implementing the SLI technique for online applications. Results showed that the new SLI prototype was able to obtain good-quality pattern images that are needed for image demodulation at a conveyor speed of up to 400 mm/s when samples were illuminated at lower spatial frequencies (e.g., 0.05 cycles/mm). This represents a significant step in the implementation of the SLI technique for online detection of surface and/or subsurface defects from moving samples. However, at higher spatial frequencies, some abnormal stripes were observed around the edge of samples for both direct and alternate component images. A further investigation of an effective method to correct these abnormal stripes is being conducted. A manuscript on the design, calibration and optimization of the online SLI system prototype is being prepared for journal publication. Experiments are being conducted on the effectiveness of the SLI for detecting defects of apples and other fruits in moving status.
Accomplishments
1. A new, more efficient dual-arm apple harvesting robot with great promise to save labor in the apple industry. Harvesting labor is the single largest cost in production of apples and other tree fruits. Harvest automation is thus urgently needed to address the rising costs and growing shortage of labor for fruit production. Based on the previous single-arm robotic harvesting technology, ARS researchers in East Lansing, Michigan, in collaboration with Michigan State University, developed a new dual-arm harvesting robot to enhance fruit harvest efficiency and cost effectiveness. The new robot demonstrated up to 34% improvements in harvesting efficiency, compared to the single-arm robot, with great potential for further performance enhancement. This new robot design provides a commercially viable solution to automated harvesting of apples, which is critical to the long-term sustainability and global competitiveness of the U.S. apple industry.
Review Publications
Chu, P., Li, Z., Zhang, K., Lammers, K., Lu, R. 2024. High-precision fruit localization using active laser-camera scanning: Robust laser line extraction for 2D-3D transformation. Smart Agricultural Technology. 2024(7). Article 100391. https://doi.org/10.1016/j.atech.2023.100391.
Zhang, K., Chu, P., Lammers, K., Li, Z., Lu, R. 2023. Active laser-camera scanning for high-precision fruit localization in robotic harvesting: system design and calibration. Horticulturae. 10(1):40. https://doi.org/10.3390/horticulturae10010040.
Zhang, K., Lammers, K., Chu, P., Li, Z., Lu, R. 2023. An automated apple harvesting robot—from system design to field evaluation. Journal of Field Robotics. 1-17. https://doi.org/10.1002/rob.22268.
Li, J., Lu, Y., Lu, R. 2023. Identification of early decayed oranges using structured-illumination reflectance imaging coupled with fast demodulation and improved image processing algorithms. Postharvest Biology and Technology. 207. Article 112627. https://doi.org/10.1016/j.postharvbio.2023.112627.
Chu, P., Li, Z., Zhang, K., Chen, D., Lammers, K., Lu, R. 2023. O2RNet: Occluder-occludee relational network for robust apple detection of clustered orchard environments. Smart Agricultural Technology. 5. Article 100284. https://doi.org/10.1016/j.atech.2023.100284.
Burks, T., Watson, A., Frederick, Q., Migliaccio, K., Lu, R. 2023. Frontier: Creating parallel SmartAg systems certificate programs for engineering and applied science graduate students. Journal of the ASABE. 66(5):1187-1203. https://doi.org/10.13031/ja.15358.