Zheng Zhou1†
Umair Zahid2†
Yaqoob Majeed2*Nisha2Sadaf Mustafa2Muhammad Muzzammil Sajjad2Hafiz Danish Butt2
Longsheng Fu3- 1College of Engineering, Heilongjiang Bayi Agricultural University, Daqing, China
- 2Department of Food Engineering, Faculty of Agricultural Engineering and Technology, University of Agriculture Faisalabad, Faisalabad, Pakistan
- 3College of Mechanical and Electronic Engineering, Northwest A&F University, Yangling, China
On-farm sorting and transportation of postharvest fruit include sorting out defective products, grading them into categories based on quality, distributing them into bins, and carrying bins to field collecting stations. Advances in artificial intelligence (AI) can speed up on-farm sorting and transportation with high accuracy and robustness and significantly reduce postharvest losses. The primary objective of this literature review is to provide an overview to present a critical analysis and identify the challenges and opportunities of AI applications for on-farm sorting and transportation, with a focus on fruit. The challenges of on-farm sorting and transportation were discussed to specify the role of AI. Sensors and techniques for data acquisition were investigated to illustrate the tasks that AI models have addressed for on-farm sorting and transportation. AI models proposed in previous studies were compared to investigate the adequate approaches for on-farm sorting and transportation. Finally, the advantages and limitations of utilizing AI have been discussed, and in-depth analysis has been provided to identify future research directions. We anticipate that this survey will pave the way for further studies on the implementation of automated systems for on-farm fruit sorting and transportation.
Introduction
On-farm sorting involves the removal of damaged, diseased, and rotten produce, and then categorizing the produce into bins or trays based on size, color, maturity, and ripening stage (Majeed and Waseem, 2022). The bins and trays are carried from the picking point to the collection center, which is usually at the end of a row or field. On-farm transportation is the operation of transporting bins and trays to the collection center. For large fruit-producing companies, the operations of sorting and transporting fruit for packing are conducted in the sorting line of the factory. However, in small farms that lack an integrated production line, these tasks may be conducted in the field or warehouse. On-farm sorting and transportation are important for small commercial farms and orchards because the produce is sorted and separated into bins on-site and directly delivered to storage before being sold in the fresh market. The operations of on-farm sorting and transportation are shown in Figure 1.
Figure 1 Flow diagram of operations involved in on-farm sorting and transportation.
Producers are often reluctant to utilize manual on-farm sorting approaches because of higher labor costs and lower efficiency. High profits would be hard to achieve without on-farm sorting operations to remove products such as overripe, rotten, and defective fruit that are unsuitable for the fresh market. Manual on-farm transportation is a repetitive task that requires pickers to walk in the field carrying full and empty trays and bins between the picking location and the collection station. With the development of artificial intelligence (AI) technology, automated on-farm sorting and transportation have been used as an alternate and viable method to replace manual sorting and transportation of fruits. In recent years, AI-based systems have progressively been employed for on-farm sorting and transportation tasks, greatly improving the production efficiency of agriculture. The deployment of AI can speed up post-harvest handling processes and significantly reduce post-harvest losses and the chance of mechanical injury (Idama and Uguru, 2021). Additionally, it enhances the facility and safety level of human tasks for handling fresh horticultural produce (Bader and Rahimifard, 2020).
A number of AI solutions have been provided for maintaining the quality of fresh fruit products at the on-farm and post-harvest stages. In the last few years, significant reviews attempts have been devoted to the development of automated agricultural systems that are capable of efficiently performing tedious field tasks such as fruit yield estimation (Maheswari et al., 2021), shoot thinning (Majeed et al., 2020; Majeed et al., 2021), non-destructive defect detection (Nturambirwe and Opara, 2020), and mechanical harvesting (Zhang et al., 2020) by using AI applications. However, available reviews lack information on the advancement of AI techniques such as computer vision, machine learning, and deep learning for on-farm fruit sorting and transportation. Thus, a literature review is needed to determine the applications of AI for on-farm sorting and transportation of fruits, identify the challenges faced by these technologies and discuss their prospects and opportunities. On-farm sorting and transportation for small commercial farms tend to be low-cost, reliable, and miniaturized to work in conditions of machine vibration and uneven terrain, which is different from factory sorting lines and transportation. Hence, we excluded the studies on AI models for fruit quality detection for factory sorting lines and narrowed the scope to the techniques that are suitable for on-farm sorting lines and transportation, making the paper more relevant to the topic.
This paper reviews the studies on advanced technologies that have been applied to on-farm fruit sorting and transportation. Key findings related to data acquisition sensors, the use of AI models, and their benefits for on-farm sorting and transportation are presented. In addition, it addresses knowledge gaps regarding the use of AI in on-farm sorting and transportation and suggests future research directions to overcome these gaps.
Applications of AI for on-farm sorting and transportation
Applications of AI for on-farm sorting
Conventional fruit sorting methods are based on visual assessment. Various factors such as ripeness, quality, decay, disease, and injury are considered for sorting (Wendel et al., 2018; Gabriëls et al., 2020; Kang and Gwak, 2021). On-farm sorting requires a considerable amount of labor and low productivity is prone to human fatigue, and is biased by inspector experience, resulting in product variability and failure to meet quality standards (Rysz and Mehta, 2021). Given these challenges, recent studies have focused on the application of AI models to on-farm sorting to increase harvest automation and reduce labor costs. Table 1 shows examples of on-farm sorting tasks solved by AI models.
Table 1 On-farm sorting tasks solved by AI model.
On-farm sorting tasks that can be addressed by AI models mostly include maturity evaluation, quality identification, injury detection, and decay and disease detection for fruits such as apples, mangoes, coconuts, blueberries, etc. (Kuzy et al., 2018; Caladcad et al., 2020; Gabriëls et al., 2020; Kang and Gwak, 2021). On-farm sorting aims to remove fruit that is defective and not suitable for the fresh market and to grade fruit immediately after harvest so that it can be stored differently to achieve cost savings. AI systems have been deployed to replace human workers, overcoming visual variability, inaccuracy, and fatigue in quality grading. Machine vision is a potentially viable solution for improving harvest efficiency by providing an alternative way to automatically and non-destructively grade harvested fruits in the field. The sorting machine is highly dependent on an integrated mechanism, which is mainly composed of an image processing unit, a fruit conveyor, a paddle, and a bin filler, as shown in Figure 2. Implementing a computer-controlled hydraulic system for on-farm sorting machines has been greatly beneficial to achieve significant cost savings and meet commercial needs.
Figure 2 Schematic of a sorting machine: (A) image processing unit, fruit conveyor, and paddle; (B) bin filler (Lu et al., 2022).
Applications of AI for on-farm transportation
Pickers spend a considerable amount of time walking to carry full containers, i.e., trays and bags, to the collection center at the end of the row and walking back with the empty container to resume picking. Workers who use the conventional manual picking method are at risk of musculoskeletal problems due to carrying heavy loads and repetitive hand and upper body activities. Faced with the challenges of labor costs, low harvesting efficiency, and occupational injuries, harvesting aids such as on-farm transportation co-robots (Figure 3) have been introduced to assist in harvesting fruits such as apples, strawberries, and grapes (Ye et al., 2018; Faheem et al., 2021; Peng et al., 2021; Lu et al., 2022).
Figure 3 Co-robot for on-farm transportation of strawberries: (A, B) control boxes, (C) GPS antennas, (D, G) DC motors, (E) return button, (F) and (H) steer-driving system, (I) emergency button.
Previous research on on-farm transportation has focused on co-robot scheduling, maneuvering strategies, and minimizing damage to products during transportation. With the goal of developing a co-robot to carry trays and transport fruit in the harvest field, human-robot collaboration models that include picking and walking have been investigated for co-robot scheduling and dispatching policies (Seyyedhasani et al., 2020; Peng et al., 2021; Rysz and Mehta, 2021). Faced with the challenges of bin handling in commercial orchards, navigation, and maneuvering strategies for bin carriers have been proposed to achieve autonomous navigation between tree rows to perform bin handling tasks (Ye et al., 2018; Blok et al., 2019). A fleet of robotic N-trailer vehicles has also been introduced to assist with crop transportation as an alternative to the ultimate mechanization of manual harvesting activities to improve field efficiency (Guevara et al., 2021). To minimize the mechanical damage induced by vibration during post-harvest handling, the vibration mechanism of hanging grape clusters was analyzed to achieve low vibration of the fruit during on-farm transportation (Faheem et al., 2021).
Data acquisition sensors and techniques for on-farm sorting and transportation
Data acquisition sensors and techniques for on-farm sorting
Multiple sensors have been used for image acquisition in on-farm sorting and transportation, including a red-green-blue (RGB) camera, a charge-coupled device (CCD) camera, a hyperspectral camera, a near-infrared (NIR) sensor, visible and near-infrared spectroscopy, and a thermal camera. According to previous studies, the RGB camera is currently the most widely employed for on-farm sorting, especially for surface damage detection, color grading, mass and volume estimation of apples, and ripeness of avocados (Jaramillo-Acevedo et al., 2020; Lu et al., 2022; Mansuri et al., 2022). A CCD camera was utilized for the size and color grading of apples and mass grading of mangoes (Momin et al., 2017; Zhang et al., 2021). A thermographic imaging system combining a thermal camera (7.5–13 µm) with heat lamps was built for internal bruise detection in blueberries (Kuzy et al., 2018). A hyperspectral camera mounted on a ground-based mobile platform was used to detect ripeness in mango and apples (Wendel et al., 2018; Çetin et al., 2022; Wang et al., 2022). A NIR spectrometer (850–1,888 nm) was used for spectral measurements to determine the total soluble solid content in “Medjool” dates to predict fruit quality (Ben-Zvi et al., 2017). VNIR spectroscopy (200–1,000 nm and 673–1,100 nm) was employed for internal browning in mangoes (Gabriëls et al., 2020; Raghavendra et al., 2021). In addition to imaging and spectral techniques, a tapping system using an omnidirectional microphone to record the tapping sound was developed for ripeness detection in coconuts (Caladcad et al., 2020). Table 2 represents the key instruments required for data acquisition for on-farm fruit sorting.
Table 2 Key instruments required for on-farm sorting data acquisition.
The RGB camera is the most employed sensor for automated on-farm sorting and grading. It should be noted that using a machine vision-based AI algorithm to sort and grade fruits and vegetables need high-quality and decipherable images. Hence, an RGB camera is the most appropriate sensor for the job. RGB cameras are low-cost and can be used to measure texture, color, geometric shape, and defects in fruits and vegetables. The main disadvantage of an RGB camera is its sensitivity to changes in illumination under different lighting conditions. Besides RGB cameras, CCD cameras are also widely used for on-farm sorting because of their low cost. Hyperspectral, NIR, and thermal cameras are more expensive sensors and provide different spectral ranges from visible light, which are universally utilized for internal defect and ingredient detection due to their sensitive spectral range for internal damage and composition. Hyperspectral imaging includes a large amount of spectral data, providing more information for defect and contaminant detection in on-farm sorting. However, the cost is the major drawback of hyperspectral cameras.
Data acquisition sensors and techniques for on-farm transportation
With regards to cameras and sensors employed for on-farm transportation, LiDAR sensors, and positioning systems are more commonly used than vision-based sensors due to their robustness for robot localization and navigation in outdoor environments. A 2D LiDAR scanner was placed on the front of the orchard transport robot to observe tree growth on each side (Blok et al., 2019). Positioning systems such as GNSS, GPS, RTK-GNSS, and RTK-Global Positioning System (RTK-GPS) were used to obtain ground truth position information and schedule the path for the bin-handling machines (Ye et al., 2018; Blok et al., 2019; Peng and Vougioukas, 2020; Peng et al., 2021). An inertial measurement unit (IMU) was employed for the measurement of the location and heading angle of on-farm harvesting-aid robots (Ye et al., 2018; Blok et al., 2019). A wheel encoder was used to measure wheel odometry (Blok et al., 2019). A load cell was used to sense the load on the tray to calculate the amount of fruit collected for transportation path scheduling (Anjom and Vougioukas, 2019; Peng and Vougioukas, 2020; Peng et al., 2021). A uniaxial force sensor was inserted between the grape clusters to calculate the peak hanging force and realize the vibrations generated in the grape clusters during postharvest handling (Faheem et al., 2021). Table 3 shows the key instruments required for data acquisition during on-farm transportation.
Table 3 Key instruments required for on-farm transportation data acquisition.
In previous studies, LiDAR sensors have been most commonly used for on-farm transportation. They have been widely used for yield estimation and geometric measurements of canopy value at the harvest stage due to their ability to be unaffected by illumination (Westling et al., 2020; Chen et al., 2022; Yu et al., 2022). For on-farm transportation, the main ability of LiDAR is that it can directly deliver 3D localization data to co-robots navigating in the field and orchard. Positioning systems such as GNSS, GPS, RTK-GNSS, and RTK-GPS have been employed to obtain the locations of co-robots for bin and tray scheduling. Other sensors, such as the IMU, wheel encoder, load cell, and force sensor, have been adopted to control on-farm vehicles and quantify harvesting efficiency for co-robot route planning. The accuracy of the sensors is crucial to the localization and navigation of on-farm transport vehicles. With the development of sensor technology, the accuracy of LIDAR scanners and positioning systems has been limited to the centimeter level, which is within the tolerance of precision agriculture (Ye et al., 2018; Blok et al., 2019). The accuracy of the IMU used to measure the heading angle of the on-farm vehicle was 0.1–5°. The accuracy and details of other sensors, such as wheel encoders, load cells, and force sensors, employed to evaluate harvest efficiency were not mentioned in the relevant literature, but they can be considered suitable for on-farm vehicles based on the efficiency of the experimental results.
AI models for on-farm sorting and transportation
AI models for on-farm sorting
AI models have been used for on-farm sorting along with the significant development of machine learning, deep learning, and machine vision techniques. On-farm grading is mostly aimed at identifying the size, mass, ripeness, surface and internal defects, and diseases in fruit for further transportation, storage, and packaging. Surface and internal defect detection for fruit classification is useful to improve product quality and boost profits. Moreover, on-farm defect detection could analyze product degradation and damage and take preventative actions to halt the deterioration process. Fruit firmness analysis can assist in improving the packaging process by maintaining consistency. AI-based techniques provide a solution for automatically unloading graded fruit into bins to improve harvesting efficiency.
Various image processing methods and deep learning models have been introduced to achieve automated on-farm sorting. Artificial neural network (ANN) and support vector machine (SVM) models have been developed to estimate the mass and volume of apples and mangoes (Utai et al., 2019; Mansuri et al., 2022). Classification of bruises in blueberries was performed using linear discriminant analysis, SVM, random forest (RF), K-nearest-neighbors, and logistic regression classifiers (Kuzy et al., 2018). ANN, RF, SVM, convolutional neural network (CNN), multiple linear regression (MLR), and partial least squares (PLS) were used to classify the ripeness and firmness levels of apple, coconut, kiwi, and mango (Torkashvand et al., 2017; Torkashvand et al., 2017; Wendel et al., 2018; Caladcad et al., 2020; Wang et al., 2022). ANN and PLS were used for identifying browning and total soluble solids in “Medjool” dates and mangoes for quality measurement (Ben-Zvi et al., 2017; Gabriëls et al., 2020). The performance of the AI models is presented in Table 4.
Table 4 Accuracy of the AI models for on-farm handling.
Spectroscopy techniques such as hyperspectral spectroscopy and visible and near-infrared spectroscopy, which include large amounts of data, have been used for the chemical detection of various fruits due to their sensitivity to nutritional and constituent content that can be detected by external inspection (Ben-Zvi et al., 2017; Jaramillo-Acevedo et al., 2020; Çetin et al., 2022). ANN models, which are robust for pattern recognition in a large amount of data, are used for hyperspectral imaging analysis in chemical detection, such as total soluble solids content, mineral nutrient content, and dry matter content for mangoes, bananas, blueberries, etc. Visible image size and external defects. ANN models have shown great potential in grading, which are superior to other models such as PLS, SVM, and RF (Wendel et al., 2018; Gabriëls et al., 2020). Visible imaging has been employed to determine physical features such as size, mass, color, and external defects and bruises in fruits. CNN models have been introduced to replace manual inspection in fruit sorting by extracting physical features from images and using them to detect the size, mass, color, and external defects and bruises (Wendel et al., 2018; Kang and Gwak, 2021). ANN and CNN models provide solutions for making quality decisions and improving postharvest efficiency through on-farm sorting, which will be soon widely adopted for automated on-farm sorting systems, especially in small commercial farms and orchards.
AI models for on-farm transportation
Workers’ picking speeds vary based on fruit distribution and the individual’s work pace. To address this issue, crop transport harvesting-aid robots have been developed to manage full and empty bins/trays as harvesting aids to increase the productivity, safety, and aesthetics of manual harvesting (Fei and Vougioukas, 2021; Guevara et al., 2021). Since harvesting-aid robots are subject to the constraints of space between crop rows and the cooperation with pickers, maneuvering strategies are required to determine the transport path and operation to deliver full and empty bins in the field. Modeling frameworks for the combined tasks of manual harvesting and robot-assisted crop transport have been proposed to support specialty crops that require labor to distribute and pick fruit. Models describing the interactions between robots and humans during fruit harvesting, such as deterministic predictive dynamic scheduling (Peng and Vougioukas, 2020), a stochastic model (Seyyedhasani et al., 2020), and a risk-averse optimization approach (Rysz and Mehta, 2021), have been utilized to identify optimal harvester servicing policies. Evaluating the efficiency of pickers is vital for scheduling co-robots to transport empty and full bins in the field. Mechanistic gray box models were addressed to predict the time when a picker will fill up the tray being currently harvested to calculate tray transport request time, with the goal of increasing harvesting efficiency (Anjom and Vougioukas, 2019). Field navigation is another core feature for on-farm transport harvesting-aid robots, which includes co-robot localization, a vehicle steering scheme, and path scheduling. Probabilistic localization algorithms, i.e., the Kalman filter and the particle filter using a 2D light detection and ranging (LiDAR) scanner, were developed for in-row robot navigation in orchards (Blok et al., 2019). Maneuvering strategies were created for four steering vehicles to complete different bin-handling tasks in commercial orchards (Ye et al., 2018). To support cooperation between pickers and harvesting-aid robots, cooperative navigation strategies were used to generate a feasible path to arrange and allocate vehicles and trailers for transporting empty and full trays between field rows (Guevara et al., 2021; Peng et al., 2021).
Small crews of harvesting-aid robots that assist big picking crews by transferring full and empty trays can boost harvesting efficiency by minimizing pickers’ unproductive walking times. The study of robotic postharvest handling can benefit specialty fruits such as blueberries, lychees, grapes, and cherries. Harvesting-aid robots have been used in conjunction with pickers, following the departure times, harvesting sequence, computed vehicle routes, and the number of trailers as guidelines. Harvesting-aid robots can handle the transport of empty and full bins and trays between the collection stations and the pickers, thus improving harvesting efficiency by decreasing the unproductive walking times of the pickers in the field. A good maneuvering strategy for a four-wheel independently steered robotic bin-handling machine can effectively accomplish the navigation of bin transport on the desired trajectories without damaging fruit trees or hitting bins. Hanging force analysis for grape clusters during on-farm transportation is helpful to realize the low vibration of fruit clusters, which provides the theoretical basis for reducing berry drop during on-farm transportation (Faheem et al., 2021).
Challenges and future directions for AI in on-farm sorting and transportation
Challenges and future directions for AI models in on-farm sorting
I. The most common sensors used for on-farm sorting are RGB and CCD cameras. However, RGB and CCD cameras can only detect surface and superficial appearance parameters such as shape, color, and size. There is a need for sensors that meet the requirements for high throughput function and low cost before this technology is more widely adopted for small farms and orchards. Automated sorting and grading systems using sensors such as hyperspectral cameras, lasers, and NIR for defect detection have been developed for factory sorting lines. To improve the performance of internal disease and defect detection in the factory line, more advanced sensors, and techniques could be adopted and modified to identify surface and internal diseases and defects of fruits and vegetables for the on-farm sorting system.
II. AI models have been more universally developed to determine maturity level by fruit shape, size, mass, and nutritional substance for harvesting than for in-field sorting and grading. Accordingly, AI models for harvesting have achieved better performance than the ones for on-farm sorting. To improve the performance of on-farm sorting to address this problem, research such as that on ripeness and quality detection (Overbeck et al., 2017; Halstead et al., 2018) for fruit and vegetable harvesting could be introduced and modified. Furthermore, AI models and systems for on-farm sorting need to be further optimized based on the factory sorting line to meet the requirements of working in field conditions, which requires systems that are reliable and prone to being functional to work with machine vibrations and uneven terrain with a high sorting throughput.
III. Another limitation is the dataset used for the AI model of on-farm sorting. In previous studies, datasets were mostly built for individual research using limited varieties and cultivars. Fruit samples were randomly selected to ensure the diversity of the dataset, and the fruit quality used as a reference for the AI models was manually checked and classified by experts (Caladcad et al., 2020; Mansuri et al., 2022; Wang et al., 2022). These data collections to build datasets were conducted for individual research, which could lead to tendencies of model overfitting and reduce the accuracy of the AI model. Additionally, AI models may be biased as data collection was conducted either indoors or outside, with varying environmental conditions affecting illumination. As a result, the diversity of datasets for on-farm sorting has not been fully developed. Hence, it is suggested that we use additional data samples, including more diverse production circumstances and different cultivars, to improve the generalization ability of AI models. Open-source image datasets could be another solution for expanding the samples to avoid overfitting in AI model training. Furthermore, a findable and accessible dataset that includes more fruit types and varieties conducted in different environmental conditions would be beneficial to expand the dataset for generalization and development of AI algorithms and to prompt the on-farm sorting industry for small farms and orchards.
IV. On-farm post-harvest processing of fruit comprises a series of operations, including sorting and grading, bin filling, cleaning, and waxing. Most of the studies have focused on the on-farm processing of sorting, grading, and bin filling. There are few studies related to the on-farm cleaning and waxing of fruit. Small farms and orchards usually skip the process of cleaning and waxing and deliver the produce directly to the market due to the lack of low-cost cleaning and waxing machinery. Without cleaning and waxing, the shelf life of the fruit would be greatly compromised during storage, which would ultimately affect the quality and profit of the fruit. Low-cost on-farm cleaning and waxing machinery could be a solution for small farms and orchards. The development of on-farm cleaning and waxing machinery integrated with sorting and grading systems can be an opportunity to improve the robustness and efficiency of on-farm handling on small farms and orchards.
V. Studies of on-farm handling of vegetables are rarely found in the published literature. This is because most vegetables are less profitable than fruits, so farms would rather invest less in AI and robotics for the on-farm sorting of vegetables to balance inputs and outputs. However, organic vegetables, which are more dependent on skilled management, have become popular in the fresh market, providing an opportunity for the development of AI in the on-farm sorting of specialty vegetables such as tomatoes, peppers, and cucumbers. We anticipate that there will be more demand for AI applications in the on-farm sorting of vegetables in the near future.
VI. Initially, researchers only focused on the AI models for the sorting lines of big commercial producers, and the AI models tended to be developed for sorting the produce that was directly transported from farms and orchards to factories, rather than for the on-farm stage that occurs before transportation (Nasiri et al., 2019; Yang et al., 2022). Small farms and orchards had limited access to automated fruit sorting lines due to their high cost. The development of sensor technology and AI models has opened up the potential for automated fruit sorting in small farms and orchards. In recent years, customer demand for high-quality produce has promoted the on-farm management of small farms and orchards. Consequently, small producers have gradually realized the importance of replacing costly labor with automated sorting systems to save costs and improve efficiency, whereupon AI models for on-farm sorting systems have been introduced. However, AI models for on-farm sorting are still less focused on researchers compared to those for big factories. The limitation of factory sorting used for on-farm sorting is that the AI models and sensors may not be efficient for the on-farm environment due to operational demands such as lighting, working conditions, and throughput requirements. As a matter of fact, the purpose of factory and on-farm sorting is the same: to sort and grade produce and remove unwanted items. Therefore, an on-farm fruit sorting system could adopt the AI models developed for factory sorting lines due to their prevalence and effectiveness, but it should be modified and fine-tuned to meet the requirements and environmental conditions of an on-farm operation. Meanwhile, the development of flexible and low-cost sorting lines for on-farm sorting could provide solutions for reducing the costs for big factories. Hence, it is anticipated that the development of AI models for fruit sorting would be beneficial for both factory and on-farm sorting.
Challenges and future directions for AI models in on-farm transportation
I. Harvesting-aid robots have been introduced to transport empty and full trays between picking crews and the collection station at the end of the field. Collaboration models between pickers and robots have been studied for harvesting-aid robot scheduling to increase harvest efficiency. Reactive scheduling has been replaced by predictive scheduling because it can reduce or eliminate the wait time for the robot to move to the pickers’ location. Using co-robot scheduling models, harvesting-aid robots for fragile fruits such as strawberries and table grapes have been introduced as prototypes and simulated using manual harvest data. However, most of the on-farm transporting harvesting-aid robots have not been practically applied in commercial farms or orchards. Mechanizing harvesting-aid robots, which can be practically used for transporting empty and full trays in commercial fields, constitutes one of the biggest challenges for on-farm transportation. Harvesting-aid robots integrate technologies such as automatic navigation, localization, wireless data transmission, and maneuvering strategies. Further work will involve combining these technologies to improve the practicality of harvesting-aid robots. The core part of on-farm harvesting-aid robots is route planning. Advanced computing technologies such as search algorithms have been developed to meet the requirements of the logistics industry. The search algorithm can find the best solutions for the desired goal by searching for all the possible alternatives, which is achieved in two phases, i.e., problem definition and search space exploration. The search algorithm has shown its efficiency and accuracy in multi-robot navigation and cooperation (Xu et al., 2020; Sahu et al., 2023). AI models based on search algorithms can potentially be used for on-farm harvesting-aid robots to minimize the energy cost and time taken in complex on-farm tasks.
II. Safety and collision issues are another concern for on-farm transport as most transport robots adopt route planning strategies for navigation. Obstacle and headland detection are crucial for orchard robots to avoid endangering humans and obstacles on the navigation path. In addition, they need to turn or stop when encountering extreme terrain and at the edge of the road. However, this problem has not been previously addressed and needs to be further investigated for on-farm transport robots. Although positioning systems such as RTK-GNSS have been utilized to identify environmental scenarios in farming vehicles, the tree canopies close to the vehicles can obstruct the satellite signals and degrade the accuracy. This can be addressed by better-utilizing sensors such as 3D LiDAR and RGB-depth cameras and integrating them with AI models to classify objects, areas, and paved and unpaved roads in orchards. The implementation of automatic driving and obstacle avoidance for on-farm transport could improve human safety and prevent collisions between robots and vehicles. Promisingly, on-farm transport robots integrated with positioning systems and machine vision systems for inter-vehicle coordination and safe navigation will be the focus of research in the near future.
III. Mechanical damage induced by the vibration of produce during bin filling and on-farm transportation can reduce the quality and shelf life of fruits and vegetables. Bruising and abrasions can be exacerbated by overweight bins and bumps in the uneven terrain of fields and orchards. Only a few studies have focused on bin management to overcome mechanical damage during bin filling and on-farm transport. On-farm bin fillers, which are usually incorporated with on-farm sorting and grading platforms, play an important role in evenly distributing fruits and vegetables to reduce bruising and abrasion. Sensors such as an IR sensor and a Hall-effect sensor have been used for bin fillers to measure the fall distance between the fruit and the filler for bruise prevention during catching and distributing the fruit into bins (Peng et al., 2022). However, it can only detect the distance between the filler and the area directly below the sensor, resulting in inaccuracy in the distance and distribution evaluation of fruits and vegetables. This problem can be addressed by introducing AI techniques using sensors such as a depth camera or 3D LiDAR, which provide a wider field of view of the fruit load distribution and thus better control the position of the bin filler, preventing mechanical damage during bin filling.
IV. Previous studies have focused on on-farm bin handling from the perspective of maneuvering strategies and vehicle positioning. The decision-making process for storage and transportation at the post-harvest stage plays a crucial role in the long-term stability of produce. To achieve the goal of planning post-harvest storage and transportation, farmers, and distributors need to monitor the on-farm harvesting process. Bin monitoring is one of the main ways to monitor the harvesting process, but it has rarely been studied in the literature. Load cells have been used to measure the number of trays to track the picking speed of strawberries for transport co-robot arrangements, which could alternatively be applied to measure the quantity of bin weight to determine the harvesting speed. Furthermore, smart bins equipped with sensors that can detect the condition of fruits and vegetables inside the bins, such as temperature, humidity, ethylene, and carbon dioxide production, can be a powerful tool to make appropriate decisions for storage and transportation at the post-harvest stage.
Conclusion
Advanced technologies in AI are increasingly being used to automate a variety of on-farm sorting and transportation operations. In recent years, AI approaches have proven to be useful and effective for automated fruit sorting and transportation at the on-farm post-harvest stage. This paper focused on determining the applications of AI for on-farm fruit sorting and transportation, identifying the challenges faced by these applications, and discussing the prospects and opportunities. Key findings related to data acquisition sensors, the use of AI models, and their merits are presented. Limitations and future research directions have been elaborated to provide an overview and in-depth analysis of potential applications of AI for on-farm sorting and transportation. We believe that these investigations will help direct future research toward building autonomous systems for on-farm sorting and transportation. In the context of increasing labor costs and the demand for better-quality produce in the fresh market, there will be a great need for AI-based systems for on-farm sorting and transportation. Given the rapid advancement of AI technologies, it is predictable that the use of AI for on-farm sorting and transportation will have a widespread application in precision agriculture in the near future.
Author contributions
ZZ and UZ contributed in editing of draft of manuscript. YM contributed in conceptualization, project administration, and editing of final draft of manuscript. N, SM, MMS, and HDB contributed in survey of literature, graph preparation. LF contributed in evaluation of manuscript. All authors contributed to the article and approved the submitted version.
Funding
This research was supported by the National Natural Science Foundation of China (NSFC) (32201655), Heilongjiang Human Resources and Social Security Bureau, the National Research Program for Universities, the Higher Education Commission (HEC), Pakistan (20-15545/NRPU/R&D/HEC/2021), and the University of Agriculture Faisalabad, Pakistan.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Anjom, F., Vougioukas, S. G. (2019). Online prediction of tray-transport request time using mechanistic grey box models for improved scheduling of robotic strawberry harvest-aids. Biosyst. Eng. 188, 265–287. doi: 10.1016/j.biosystemseng.2019.10.025
Bader, F., Rahimifard, S. (2020). A methodology for the selection of industrial robots in food handling. Innovative Food Sci. Emerging Technol. 64, 102379. doi: 10.1016/j.ifset.2020.102379
Ben-Zvi, R., Ignat, T., Alchanatis, V., Hoffman, A., Borochov-Neori, H., Immerman, A., et al. (2017). New postharvest aproach for high quality fresh ‘Medjhool’ date. Postharvest biology and technology 124, 35–44. doi: 10.1016/j.postharvbio.2016.09.004
Blok, P. M., van Boheemen, K., van Evert, F. K., IJsselmuiden, J., Kim, G. H. (2019). Robot navigation in orchards with localization based on particle filter and kalman filter. Comput. Electron. Agric. 157, 261–269. doi: 10.1016/j.compag.2018.12.046
Caladcad, J. A., Cabahug, S., Catamco, M. R., Villaceran, P. E., Cosgafa, L., Cabizares, K. N., et al. (2020). Determining Philippine coconut maturity level using machine learning algorithms based on acoustic signal. Comput. Electron. Agric. 172, 105327. doi: 10.1016/j.compag.2020.105327
Çetin, N., Karaman, K., Kavuncuoğlu, E., Yıldırım, B., Jahanbakhshi, A. (2022). Using hyperspectral imaging technology and machine learning algorithms for assessing internal quality parameters of apple fruits. Chemometrics and Intelligent Laboratory Systems 230, 104650.
Chen, R., Zhang, C., Xu, B., Zhu, Y., Zhao, F., Han, S., et al. (2022). Predicting individual apple tree yield using UAV multi-source remote sensing data and ensemble learning. Comput. Electron. Agric. 201, 107275. doi: 10.1016/j.compag.2022.107275
Faheem, M., Liu, J., Chang, G., Ahmad, I., Peng, Y. (2021). Hanging force analysis for realizing low vibration of grape clusters during speedy robotic postharvest handling. Int. J. Agric. Biol. Eng. 14 (4), 62–71. doi: 10.25165/j.ijabe.20211404.6406
Fei, Z., Vougioukas, S. G. (2021). Co-Robotic harvest-aid platforms: Real-time control of picker lift heights to maximize harvesting efficiency. Comput. Electron. Agric. 180, 105894. doi: 10.1016/j.compag.2020.105894
Gabriëls, S. H. E. J., Mishra, P., Mensink, M. G. J., Spoelstra, P., Woltering, E. J. (2020). Non-destructive measurement of internal browning in mangoes using visible and near-infrared spectroscopy supported by artificial neural network analysis. Postharvest Biol. Technol. 166, 111206. doi: 10.1016/j.postharvbio.2020.111206
Gené-Mola, J., Gregorio, E., Guevara, J., Auat, F., Sanz-Cortiella, R., Escolà, A., et al. (2019). Fruit detection in an apple orchard using a mobile terrestrial laser scanner. Biosyst. Eng. 187, 171–184. doi: 10.1016/j.biosystemseng.2019.08.017
Guevara, L., Rocha, R. P., Cheein, F. A. (2021). Improving the manual harvesting operation efficiency by coordinating a fleet of n-trailer vehicles. Comput. Electron. Agric. 185, 106103. doi: 10.1016/j.compag.2021.106103
Halstead, M., McCool, C., Denman, S., Perez, T., Fookes, C. (2018). Fruit quantity and ripeness estimation using a robotic vision system. IEEE Robotics Automation Lett. 3 (4), 2995–3002. doi: 10.1109/LRA.2018.2849514
Idama, O., Uguru, H. (2021). Robotization of tomato fruits production to enhance food security. J. Eng. Res. Reports JERR 1), 67–75. doi: 10.9734/jerr/2021/v20i117248
Jaramillo-Acevedo, C. A., Choque-Valderrama, W. E., Guerrero-Álvarez, G. E., Meneses-Escobar, C. A. (2020). Hass avocado ripeness classification by mobile devices using digital image processing and ANN methods. Int. J. Food Eng. 16 (12), 20190161. doi: 10.1515/ijfe-2019-0161
Kang, J., Gwak, J. (2021). Ensemble of multi-task deep convolutional neural networks using transfer learning for fruit freshness classification. Multimedia Tools Appl. 81: 22355–22377. doi: 10.1007/s11042-021-11282-4
Kuzy, J., Jiang, Y., Li, C. (2018). Blueberry bruise detection by pulsed thermographic imaging. Postharvest Biol. Technol. 136, 166–177. doi: 10.1016/j.postharvbio.2017.10.011
Lu, J., Ehsani, R., Shi, Y., Abdulridha, J., de Castro, A. I., Xu, Y. (2017). Field detection of anthracnose crown rot in strawberry using spectroscopy technology. Comput. Electron. Agric. 135, 289–299. doi: 10.1016/j.compag.2017.01.017
Lu, Y., Lu, R., Zhang, Z. (2022). Development and preliminary evaluation of a new apple harvest assist and in-field sorting machine. Appl. Eng. Agric. 38 (1), 23–35. doi: 10.13031/AEA.14522
Maheswari, P., Raja, P., Apolo-Apolo, O. E., Pérez-Ruiz, M. (2021). Intelligent fruit yield estimation for orchards using deep learning based semantic segmentation techniques–a review. Front. Plant Sci. 12. doi: 10.3389/FPLS.2021.684328/BIBTEX
Majeed, Y., Karkee, M., Zhang, Q. (2020). Estimating the trajectories of vine cordons in full foliage canopies for automated green shoot thinning in vineyards. Comput. Electron. Agric. 176, 105671. doi: 10.1016/j.compag.2020.105671
Majeed, Y., Karkee, M., Zhang, Q., Fu, L., Whiting, M. D. (2021). Development and performance evaluation of a machine vision system and an integrated prototype for automated green shoot thinning in vineyards. J. Field Robotics 38 (6), 898–916. doi: 10.1002/rob.22013
Majeed, Y., Waseem, M. (2022). “Postharvest handling systems,” in Encyclopedia of smart agriculture technologies. Ed. Zhang, Q. (Cham: Springer). doi: 10.1007/978-3-030-89123-7_125-1
Mansuri, S., Gautam, P., Jain, D., Nickhil, C. (2022). Computer vision model for estimating the mass and volume of freshly harvested Thai apple ber (Ziziphus mauritiana l.) and its variation with storage days. Scientia Hortic. 305, 111436. doi: 10.1016/j.scienta.2022.111436
Momin, M. A., Rahman, M. T., Sultana, M. S., Igathinathane, C., Ziauddin, A. T. M., Grift, T. E. (2017). Geometry-based mass grading of mango fruits using image processing. Inf. Process. Agric. 4 (2), 150–160. doi: 10.1016/j.inpa.2017.03.003
Nasiri, A., Taheri-Garavand, A., Zhang, Y. (2019). Image-based deep learning automated sorting of dte fruit. Postharvest Biol. Technol. 153, 133–141. doi: 10.1016/j.postharvbio.2019.04.003
Nturambirwe, J. F. I., Opara, U. L. (2020). Machine learning applications to non-destructive defect detection in horticultural products. Biosyst. Eng. 189, 60–83. doi: 10.1016/J.BIOSYSTEMSENG.2019.11.011
Overbeck, V., Schmitz, M., Blanke, M. (2017). Non-destructive sensor-based prediction of maturity and optimum harvest date of sweet cherry fruit. Sensors (Switzerland) 17 (2), 277. doi: 10.3390/s17020277
Peng, C., Vougioukas, S. G. (2020). Deterministic predictive dynamic scheduling for crop-transport co-robots acting as harvesting aids. Comput. Electron. Agric. 178, 105702. doi: 10.1016/j.compag.2020.105702
Peng, C., Vougioukas, S., Slaughter, D., Fei, Z., Arikapudi, R. (2021). A strawberry harvest-aiding system with crop-transport harvesting-aid robots: Design, development, and field evaluation. ArXiv Preprint. doi: 10.48550/arxiv.2107.13063
Raghavendra, A., Guru, D. S., Rao, M. K. (2021). Mango internal defect detection based on optimal wavelength selection method using NIR spectroscopy. Artificial Intelligence in Agriculture 5, 43–51.
Rysz, M. W., Mehta, S. S. (2021). A risk-averse optimization approach to human-robot collaboration in robotic fruit harvesting. Comput. Electron. Agric. 182, 106018. doi: 10.1016/j.compag.2021.106018
Sahu, B., Das, P. K., Kumar, R. (20232023). A modified cuckoo search algorithm implemented with SCA and PSO for multi-robot cooperation and path planning. Cogn. Syst. Res. 179, 24–42. doi: 10.1016/j.cogsys.2023.01.005
Seyyedhasani, H., Peng, C., Jang, W., Vougioukas, S. G. (2020). Collaboration of human pickers and crop-transporting robots during harvesting – part I: Model and simulator development. Comput. Electron. Agric. 172, 105324. doi: 10.1016/j.compag.2020.105324
Torkashvand, A. M., Ahmadi, A., Nikravesh, N. L. (2017). Prediction of kiwifruit firmness using fruit mineral nutrient concentration by artificial neural network (ANN) and multiple linear regressions (MLR). J. Integr. Agric. 16 (7), 1634–1644. doi: 10.1016/S2095-3119(16)61546-0
Utai, K., Nagle, M., Hämmerle, S., Spreer, W., Mahayothee, B. (2019). Mass estimation of mango fruits (Mangifera indica l., cv., ‘Nam dokmai’) by linking image processing and artificial neural network. Eng. Agriculture Environ. Food 12 (1), 103–110. doi: 10.1016/j.eaef.2018.10.003
Wang, F., Zhao, C., Yang, H., Jiang, H., Li, L., Yang, G. (2022). Non-destructive and in-site estimation of apple quality and maturity by hyperspectral imaging. Comput. Electron. Agric. 195, 106843. doi: 10.1016/j.compag.2022.106843
Wendel, A., Underwood, J., Walsh, K. (2018). Maturity estimation of mangoes using hyperspectral imaging from a ground based mobile platform. Comput. Electron. Agric. 155, 298–313. doi: 10.1016/j.compag.2018.10.021
Westling, F., Mahumud, K., Underwood, J., Bally, I. (2020). Replacing traditional light measurement with LiDAR based methods in orchards. Comput. Electron. Agric. 179, 105798. doi: 10.1016/j.compag.2020.105798
Xu, Y., Guan, G., Song, Q., Jiang, C., Wang, L. (2020). Heuristic and random search algorithm in optimization of route planning for robot’s geomagnetic navigation. Comput. Commun. 154, 12–17. doi: 10.1016/j.comcom.2020.02.043
Yang, Y., Peng, J., Fan, P. (2022). A non-destructive dropped fruit impact signal imaging-based deep learning approach for smart sorting of kiwifruit. Comput. Electron. Agric. 202, 107380. doi: 10.1016/j.compag.2022.107380
Ye, Y., He, L., Wang, Z., Jones, D., Hollinger, G. A., Taylor, M. E., et al. (2018). Orchard maneuvering strategy for a robotic bin-handling machine. Biosyst. Eng. 169, 85–103. doi: 10.1016/j.biosystemseng.2017.12.005
Yu, T., Hu, C., Xie, Y., Liu, J., Li, P. (2022). Mature pomegranate fruit detection and location combining improved f-PointNet with 3D point cloud clustering in orchard. Comput. Electron. Agric. 200, 107233. doi: 10.1016/j.compag.2022.107233
Zhang, Z., Igathinathane, C., Li, J., Cen, H., Lu, Y., Flores, P. (2020). Technology progress in mechanical harvest of fresh market apples. Comput. Electron. Agric. 175, 105606. doi: 10.1016/j.compag.2020.105606
Zhang, J., Lai, S., Yu, H., Wang, E., Wang, X., Zhu, Z. (2021). Fruit classification utilizing a robotic gripper with integrated sensors and adaptive grasping. Math. Problems Eng. 2021, 7157763. doi: 10.1155/2021/7157763
Zhang, Z., Lu, Y., Lu, R. (2021). Development and evaluation of an apple infield grading and sorting system. Postharvest Biol. Technol. 180, 111588. doi: 10.1016/j.postharvbio.2021.111588
Keywords: deep learning, precision farming, machine vision, postharvest handling, infield transportation
Citation: Zhou Z, Zahid U, Majeed Y, Nisha, Mustafa S, Sajjad MM, Butt HD and Fu L (2023) Advancement in artificial intelligence for on-farm fruit sorting and transportation. Front. Plant Sci. 14:1082860. doi: 10.3389/fpls.2023.1082860
Received: 28 October 2022; Accepted: 14 March 2023;
Published: 06 April 2023.
Edited by:
Dmitri Voronine, University of South Florida, United StatesReviewed by:
Uzair Aslam Bhatti, Hainan University, ChinaPeter Oudemans, Rutgers, The State University of New Jersey, United States
Copyright © 2023 Zhou, Zahid, Majeed, Nisha, Mustafa, Sajjad, Butt and Fu. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Yaqoob Majeed, eWFxb29iLm1hamVlZEB1YWYuZWR1LnBr
†These authors have contributed equally to this work