Design and experimentation of multi-fruit envelope-cutting kiwifruit picking robot

Currently kiwifruit picking process mainly leverages manual labor, which has low productivity and high labor intensity, meanwhile, the existing kiwifruit picking machinery also has low picking efficiency and easily damages fruits. In this regard, a kiwifruit picking robot suitable for orchard operations was developed in this paper for kiwifruit grown in orchard trellis style. First, based on the analysis of kiwifruit growth pattern and cultivation parameters, the expected design requirements and objectives of a kiwifruit picking robot were proposed, and the expected workflow of the robot in the kiwifruit orchard environment was given, which in turn led to a multi-fruit envelope-cutting kiwifruit picking robot was designed. Then, the D-H method was used to establish the kinematic Equations of the kiwifruit-picking robot, the forward and inverse kinematic calculations were carried out, and the Monte Carlo method was used to analyze the workspace of the robot. By planning the trajectory of the robotic arm and calculating critical nodes in the picking path, the scheme of trajectory planning of the robot was given, and MATLAB software was applied to simulate the motion trajectory as well as to verify the feasibility of the trajectory planning scheme and the picking strategy. Finally, a kiwifruit picking test bed was set up to conduct picking tests in the form of fruit clusters. The results show that the average time to pick each cluster of fruit was 9.7s, the picking success rate was 88.0%, and the picking damage rate was 7.3%. All the indicators met the requirements of the expected design of the kiwifruit-picking robot.

1 Introduction

Kiwifruit is highly nutritious, known as the “king of fruit”, and has significant market demand. China is the world’s largest agricultural country in terms of kiwifruit planting area and production, with its annual yield exceeding 3 million tons (Fazayeli et al., 2019). In order to timely, efficiently, and carefully harvest kiwifruit, annual orchards require a large seasonal manual labor for harvesting, with labor costs accounting for more than 25% of annual production costs (García-Quiroga et al., 2015). With the scale-up of the kiwifruit planting industry, the traditional manual picking method is difficult to meet the current market demand. Faced with the increasing labor shortage, the development of efficient and adaptable kiwifruit harvesting robots is of great practical value.

Fruit and vegetable-picking robots are generally composed of robotic arm, end-effector, mobile mechanism, and control system. In recent years, scholars have conducted a series of studies on fruit and vegetable picking robots such as kiwifruit (Fang et al., 2023; Gao et al., 2023), apple (Miao and Zheng, 2019; Kuznetsova et al., 2020), citrus (Mehta and Burks, 2014; Sun et al., 2023), tomato (Ling et al., 2019; Wang T. et al., 2023) and so on. For example, Williams et al. (2020) designed a four-armed parallel kiwifruit-picking robot, which separated the fruit from the stalk by rotation after the end-effector gripped the kiwifruit with two fingers, the fruit was automatically collected into a fruit box after each picking action the average time required for harvesting each fruit was 5.5 seconds, with a fruit recognition rate of 76.3% and a successful harvesting rate of 51%. However, the robot had a problem that utilized two fingers to hold the fruit process, which was easy to affect the neighboring fruits. Arad et al. (2020) designed a six-degrees-of-freedom robotic arm bell pepper picking robot, after obtaining the position of the bell pepper with an RGB-D camera, the end-effector grasped the bell pepper and cut off the fruit stalk by using a vibrating cutter. Subsequently, the picked fruit was sent by the robotic arm to the designated collection box, and the average time required for harvesting each fruit was 24 seconds, with a harvesting success rate of up to 61%. Since the robot didn’t have an automatic collection device, it resulted in it taking up half of the overall harvest time in collecting the fruit. Rong et al. (2022) designed a watermelon harvesting robot that employed the improved YOLOv5s-CBAM algorithm for watermelon identification and localization. After the detection system recognized the watermelon, the control system drove four flexible fingers to clamp the watermelon, whose harvesting success rate was 93.3%. However, the robot’s detection system was susceptible to the effects of strong illumination and occlusion, and the harvesting success rate would be significantly reduced when the watermelon was partially shaded by leaves.

Improving fruit-picking efficiency and ensuring non-destructive fruit picking are two key criteria of fruit-picking robots, in which non-destructive picking is mainly ensured by the end-effector (Fu et al., 2015). For example, Mu et al. (2020) designed a wrapped kiwifruit-picking end-effector that used two bionic fingers to wrap the fruit before repeatedly bending the fingers to separate the fruit from the stalk. Silwal et al. (2017) developed a clamped apple-picking end-effector consisting of three actuators and three fingers, where after the fruit was clamped by the three fingers, the fruit was rotated by the base until the fruit was separated from the stalk. However, the end-effector was not equipped with a pressure sensor, which made it easy to damage the fruit when the clamping force was too large. And Hohimer et al. (2019) designed a 3-finger pneumatic gripping soft end-effector that could control picking pressure by adjusting the input air pressure, preventing fruit damage, though with lower picking efficiency. Wang et al. (2018) designed a bite-type citrus picking end-effector by simulating the biting action of a snake’s mouth, which could cut off the fruit stalk while gripping the fruit, However, there was a risk of fruit damage if the clamping was not precise. Xu L. et al. (2018) designed a suction-type navel orange picking end-effector, which used a suction cup to hold the fruit and then non-destructively separated the fruit from the stalk using a cutter. However, during fruit capture, it was prone to sucking debris, such as branches and leaves around the fruit, into the end-effector.

The above studies provide valuable insights for the development of harvesting techniques for kiwifruit and other fruits and vegetables. However, the current kiwifruit-picking end-effector is mostly of a single-fruit clamping or sucking structure, which is susceptible to two problems during the fruit-picking process: A clamping or sucking method of kiwifruit is unstable when gripping the fruit, leading to potential fruit loss. Approximately 87% of kiwifruit are distributed in clusters within the canopies (Fu et al., 2019), and the use of single-fruit picking has a high likelihood of impacting neighboring fruits, thereby leading to fruit drop. Given the above analysis, a kiwifruit picking robot was designed for kiwifruit grown in orchard trellis type, which could realize the continuous picking of multiple fruits by adopting the operation mode of enveloping the fruits - cutting off the fruit stalks - and collecting the fruits in pipelines; Then, MATLAB software was used to perform forward and reverse kinematics analysis of the kiwifruit picking robot, and the workspace range of the robot was solved. By planning the trajectory of the robotic arm and calculating the critical nodes in the picking path, a scheme of the robot picking strategy was given. Finally, a kiwifruit-picking test bed was set up to verify the performance of the kiwifruit-picking end-effector.

2 Design requirements

2.1 Kiwifruit cultivation parameters

In this study, the kiwifruit planted in trellis style at the kiwifruit industry base in Mei Xian County, Baoji City, Shanxi Province was used as the research object, and the kiwifruit orchard planting scenarios were characterized as follows. ① The trellis for kiwifruit cultivation is mainly made of steel tubes and steel wires, with supporting steel tubes between the trunks, and steel wires at the top of the tubes to fix the extended branches, so that the fruits grow naturally drooping. ② Orchard trellises have an average lateral spacing of 4m, a longitudinal spacing of 2m, and a height of 2m. ③ The fruits are generally distributed in clusters in a spatial range of 1.5~1.8m in height from the ground, and there is generally 3~7 fruits in each cluster, as shown in Figure 1.

Figure 1

Figure 1 Kiwifruit orchard environment: (A) parameters of the trellis, (B) distribution of kiwifruit on the trellis. (A) Basic parameters of the kiwifruit.

In this paper, 50 ripened brown kiwifruits produced in Mei Xian County were randomly selected, and their physical parameters were measured. Electronic vernier calipers (with a precision of 0.01 mm) were used to measure the length, width, and thickness of the fruits, and the length of the fruit stalk. The weight of the fruits was measured using electronic scales (with a precision of 0.1 g). The data are presented in Table 1.

Table 1

Table 1 Basic parameters of the kiwifruit.

When the kiwifruit is mature, the fruit stalk and the fruit form a separation layer with vascular bundles as the main tissue structure. Vascular bundles are tubular hollow fibrous tissue that can withstand strong tensile stresses but not easily withstand shear forces. The main cause of fruit abscission is the shear force on the separating layer of the fruiting stalk, which causes the vascular bundles to be fractured. In this regard, if a mechanical device is used to grab the kiwifruits and then a cutter is used to cut off the fruit stalks, the fruits can be picked without damage.

From the study in reference (Fu et al., 2015), it could be observed that when the fruit stalk formed a 180° angle with the inertia axis of the fruit, the maximum force required to separate the fruit stalk was 12.8 N. When the angle was 90°, the force required for separation was 1.7 N, and at a 60° angle, the minimum force needed to separate the fruit stalk was 1.3 N. The reasons were analyzed as follows: when the clamp angle was 180°, the vascular bundles were mainly subjected to tensile stress, and the separating force of the fruit stalk was larger, and when the clamp angle was 90° and 60°, the vascular bundles were mainly subjected to shear force, and the separating force of the fruit stalk was smaller. Thus, when the cutter is used to separate the fruit from the fruit stalk, the optimal angle between the cutter and the fruit stalk ranges from 60° to 90°, which corresponds to a range of 1.3 to 1.7 N of separation cutting force.

2.2 Picking requirements and design goals

The design requirements of the kiwifruit picking robot consist of two parts: functional requirements and performance requirements. The functional requirements mainly include: automatically identifying fruit trees, recognizing fruits, gathering fruits, separating fruits from fruit stalks, and collecting fruits. The performance requirements include: automatically adjusting the speed of the mobile device according to the orchard environment, adjusting the angle of the end-effector to recognize fruits, fully aggregating fruits, quickly collecting kiwifruit, and fully adapting to the orchard picking environment.

Currently, the orchard mostly adopts trellis cultivation, which has a larger space underneath and is not impacted by branches, leaves, etc., which is favorable for the implementation of the picking action. According to the physical properties and growth environment of kiwifruit, the working space range of the picking robot is designed as follows: X-direction travel is 1.2~1.5 m, Y-direction travel is 1.5~1.8 m, and Z-direction travel is 1.5~2.0 m. The expected picking indicators are as follows: the fruit picking speed is 7~10 s/cluster, with a fruit picking success rate of more than 90% and a fruit damage rate of less than 10%.

2.3 Expected workflow

The operation flow of the kiwifruit-picking robot is shown in Figure 2. Firstly, the robot enters the kiwifruit orchard via the autonomous mobile platform to locate the target fruit trees. And the end-effector is raised to the suitable kiwifruit-picking area by the lifting device. Subsequently, the end-effector recognizes the fruit by the identification system, approaches the fruit by enveloping or grasping or sucking, and picks the fruit by cutting or pulling or twisting. Finally, the end-effector puts the fruit into a fruit collection box by putting it back or just throwing it down or piping it.

Figure 2

Figure 2 Expected workflow of kiwifruit: (A) environmental map of kiwifruit orchards, (B) autonomous mobile platforms, (C) three methods for end-effector capture of kiwifruits, (D) locating the target kiwifruits, (E) three methods of separating the fruits from the fruiting stalks, (F) three methods of collecting fruits after picking them.

In the operation phase of Figures 2C–F, the end-effector fruit-picking process can be divided into three stages: capturing fruits, separating fruit stalks, and collecting fruits. Among them, the methods to capture the fruits mainly include grabbing, sucking, and enveloping; the methods to separate the fruit stalks mainly are cutting, pulling, and twisting; and the key methods to collect the fruits mainly consist of putting them back, throwing them down and collecting them in Pipelines. The specific analysis of the implementation of each stage is shown in Table 2.

Table 2

Table 2 End-effector harvesting action method.

In contrast to crowned trees such as apples and citrus, kiwifruit is a vine and is characterized by clustering (see Figure 1B). Generally, managers in kiwifruit orchards tend to use a trellis planting pattern, where the fruits hang down on vines and are evenly distributed in the form of clusters.

1. Analyzed for kiwifruit capture methods: If the kiwifruit is captured by grabbing, generally only one fruit can be picked at a time, while the kiwifruit grows in clusters, in order to pick them individually, it is necessary to separate the adjacent fruits. However, the grabbing type end-effector used in this process is prone to damaging the adjacent fruits. If the kiwifruit is captured by suction, two problems may arise. Firstly, if the suction force is too strong, it can easily damage the fruit. On the other hand, if the suction force is too weak, the fruit may easily drop. Secondly, during the suction process, it is likely to suck in the fruit leaves and vines surrounding the fruit into the end-effector. However, when the kiwifruit is captured by means of enveloping, it is possible to address the characteristics of kiwifruit clusters by enveloping the fruit inside the end-effector in the form of fruit clusters, which reduces the rate of damage to the fruit during the fruit capture process.

2. Analyzed for the methods of separating fruit stalk in kiwifruit: Non-destructive and rapid separation of fruit from fruit stalks is the optimal principle for fruit stalk separation. For kiwifruit, this vine plant, the use of pulling for fruit stalk separation can cause damage if the strength is not controlled properly, as it may result in the vine being pulled directly from the trellis and affecting the surrounding fruit. Additionally, since kiwifruit is typically harvested when it is still slightly unripe, the connection between the fruit stalk and the fruit is relatively strong. If the twisted fruit stalk separation method is used, it can easily cause fruit damage. On the other hand, using the cut-off fruit stalk separation method allows for non-destructive separation of the fruit stalk and avoids the occurrence of the aforementioned situations.

3. Analyzed for kiwifruit collection methods:Non-destructive and rapid are also optimal principles for fruit collection. If the robotic arm with the end-effector puts the fruits back into the collection device every time they are picked, it reduces the rate of fruit damage, but it is time-consuming. On the other hand, if the direct throw-down method is used, although it is a rapid collection method, it is easy to damage the fruits. However, with the pipeline collection method, the picked fruits fall directly into the collection device along the conveyor pipeline. This method can compensate for the shortcomings of the above two collection methods and achieve rapid and non-destructive fruit collection.

In summary, after comparative analysis, it was decided in this paper to use enveloping to capture kiwifruits, cutting to separate the fruit stalks, and piping to collect the kiwifruits.

3 Structural design

3.1 Overall structure

The overall structure of the kiwifruit picking robot, as shown in Figure 3A, is primarily composed of an end-effector, robotic arm, retractable hose, fruits collection box, and crawler chassis. A tracked chassis is used as the mobile device for the kiwifruit-picking robot, allowing it to adapt to the kiwifruit orchard environment and enhancing the robot’s flexibility. A navigation camera is installed at the bottom of the tracked chassis to enable the precise positioning of kiwifruit trees in the orchard. A 6-degree-of-freedom polar coordinate robotic arm has been chosen as a support device for the kiwifruit-picking robot to assist in the end-effector-picking operation. Furthermore, to prevent the kiwifruit’s height during picking from exceeding the robotic arm’s reachable range, a retractable support frame is added beneath the base of the robotic arm to increase the height and optimize space utilization. The collection device utilizes a retractable hose and a fruit collection box. One end of the hose is securely fixed beneath the end-effector, connecting to the envelope-picking bin, while the other end was linked to the fruit collection box.

Figure 3

Figure 3 Schematic structure of kiwifruit picking robot: (A) overall structure, (B) picking end-effector. (1. end-effector 2. retractable hose 3. cushion 4. fruit collection box 5. navigation camera 6. tracked chassis 7. support frame 8. robotic arm 9. cutter 10. photoelectric detection sensor 11. hall sensor 12. baffle plate 13. lateral plate 14. fruit collection end 15. connector 16. curved guiding rod 17. rotary stepper motor 18. picking bin 19. fixing rod 20. infrared recognition sensor).

The end-effector structure, as shown in Figure 3B, comprises an external picking bin, an internal rotary cutter mechanism, and a fruit collection end. Two infrared recognition sensors are installed on both sides of the lateral plate to recognize the fruits, and a photoelectric detection sensor is positioned on one side of the baffle plate to determine whether the fruits have entered the picking bin. The rotary cutter mechanism is mainly composed of a rotating shaft, curved guide rods, fixed rods, cutters, and a rotary stepper motor. During operation, the rotary stepper motor drives the cutter mechanism to rotate, causing the cutters to cut off the fruit stalks.

The kiwifruit picking robot parameters is shown in Table 3.

Table 3

Table 3 Parameters of the kiwifruit picking robot.

3.2 Working principle

The working principle of the kiwifruit-picking robot is as follows: The robot initiates system initialization. The robot uses fused SLAM technology based on multiple sensors (including a 16-line 3D LIDAR, IMU and differential GPS/INS navigation module supporting RTK technology) to navigate and localize the fruit trees in the orchard, and uses the Dijkstra’s algorithm and dynamic windowing approach to globally plan the path and avoid dynamic obstacles detected by the LIDAR, and when it reaches the location of the fruit trees, the robot stops moving; The robotic arm delivers the end-effector in the suitable kiwifruit-picking area; Recognition sensors begin scanning the surroundings for kiwifruits, and once they are detected, the robotic arm raises the end-effector from below the kiwifruits to envelope them into the picking bin. When the photoelectric detection sensor confirms that all kiwifruits are inside the picking bin, the robotic arm halts its movement; The rotary cutter turns clockwise to cut off the stalks of the kiwifruits in the picking bin, and kiwifruits fall on curved guide rods; The rotary cutter continues to rotate until it resets, during which time the kiwifruits slowly descend the curved guide rods and drop into the fruit collection end; Finally, the picked kiwifruits are conveyed into the fruit collection box through the retractable hose, completing the single-picking process.

3.3 End-effector design

3.3.1 Structural design of picking bin

The existing kiwifruit picking end-effector mostly leverage a clamping structure to harvest the fruit. However, the clamping structure is susceptible to two issues during the picking process. Firstly, since kiwifruits naturally grow in clusters, the clamping mechanism can only pick one fruit at a time. To achieve single picking, it becomes necessary to separate adjacent fruits first, a process in which the clamping structure may inadvertently damage neighboring fruits. Secondly, when clamping fruits of different sizes, excessive clamping force can lead to fruit damage, while inadequate force can cause the fruits to drop. Aiming at the above problems, and considering that kiwifruit in clusters hangs down on the fruit tree vines in orchards, an envelope-type picking end-effector with clusters as the form of harvesting was proposed in this paper, as shown in Figure 4A. During the robot’s picking operation, the end-effector sufficiently envelopes the target fruit clusters from below, and prevents direct contact between the mechanical structure and the fruits. It is expected that by using multi-fruit envelope picking, not only the fruit picking efficiency can be improved, but also the fruit damage rate can be greatly reduced.

Figure 4

Figure 4 Design of the structural and parameters of the picking bin: (A) schematic diagram of picking process, (B) structure of the envelope type picking bin, (C) the process of enveloping fruits, (D) two forms of expected distribution of fruits, (E) the fruit collection end transmits the fruits.

The structure of the picking bin is shown in Figure 4B, and it consists of two side plates and four baffle plates connected. To prevent damage to fruit outside the picking bin when adjacent fruits are separated, the picking port’s shape has been designed as a “hexagonal shape”, with dimensions $l_1$=90mm, $l_2$=135mm, $h_1$=390mm, $h_2$=270mm, as shown in Figure 4C. During the enveloping process, the fruits outside the picking bin are separated from the picked fruits by the baffles. The side plates and baffle plates don’t need to carry heavy loads but require a certain level of toughness to prevent end-effector failure during the picking process. Therefore, 8mm thick carbon fiber plates were chosen for production.

According to the kiwifruit cultivation parameters in section 2.1, kiwifruit has 3~7 fruits per cluster, with the average value of fruit length being 67.2mm, the maximum value being 72.7mm, and the minimum value being 61.4mm; the average value of fruit width being 53mm, the maximum value being 56.7mm, and the minimum value being 49.5mm; the average value of fruit thickness being 47.9mm, the maximum value being 48.3mm, and the minimum value being 43.4mm. In this regard, in order to meet the requirements of multi-fruits picking by the end-effector, as well as to meet the requirements that the fruits in the picking bin can be fully detected by the photoelectric detection sensor (refer to section 3.3.3, the width of detection $w_{max}$=70mm), the port side of the picking bin was designed to be $l_3\times l_4$, i.e. side plate edge length $l_3$=200mm and baffle plate edge length $l_4$=150mm. It is expected that the picking bin port can hold up to about 6 to 8 fruits, as shown in Figure 4D. In addition, a fruit collection end was installed below the end-effector to be able to assist in delivering the picked kiwifruits to the fruit collection box. According to the growth parameters of the fruits, the diameter of the fruit collection end was designed to be D= $\varnothing$120mm, and it is expected that 1~2 fruits can be delivered to the fruit collection box at the same time, as shown in Figure 4E.

3.3.2 Parameter design of curved guide rods

The curved guide rod primarily serves as a buffer and takes over the picked fruits during the rotation process, effectively reducing the risk of fruit damage, as shown in Figure 5B. Additionally, it plays a guiding role during rotation, assisting the sequentially falling of the picked kiwifruits into the fruit collection end. The curved structure can enhance the smoothness of guidance, and the surface of the curved guide rod is covered with a layer of polyethylene foam cotton, a material known for its softness, excellent toughness, flexibility, and cushioning properties (Zhang et al., 2023). This added layer provides better protection for the fruits, minimizing the risk of damage.

Figure 5

Figure 5 Structure and workflow of curved guide rods: (A) limit distribution diagram for side-by-side arrangement of fruits in the picking bin, (B) structural diagram of curved guide rod, (C) schematic diagram of cutting off all fruit stalks, (D) schematic diagram of the curved guide rod delivering fruit 1, (E) schematic diagram of the fruits picking state.

The radius of curvature, width, and other parameters of the curved guide rod have a significant impact on the picking process, given its direct contact with the kiwifruits. If the radius of curvature is too small, there’s a risk of pushing kiwifruits out of the picking bin by the curved guide rod during the enveloping process, potentially causing fruit damage by the cutter. Conversely, if the radius of curvature is too large, it may lead to picked kiwifruits piling up and not passing through the fruit collection end sequentially. The limiting distribution of the side-by-side arrangement of fruits in the picking bin is shown in Figures 5A, B, with fruit 1 being the first to touch the curved guide rod when four kiwifruits are at the same height. Therefore, during the enveloping process, as long as it is avoided that fruit 1 is pushed out of the picking bin by the curved guide rod, i.e., the height of the fruit from the port of the picking bin $h_4$ is greater than the length of the fruit itself $l_5$, the rest of the fruits will enter the picking bin completely. At this point $h_4$ is approximately equal to the curved guide rod inner radius $r_2$. By combining the design parameters of the picking bin and the fruit parameters (refer to section 3.3.1), it was determined that the outer radius of the curved guide rod should be $r_1$=100mm, with an inner radius of $r_2$=72.5mm. The workflow of the curved guide rod is shown in Figures 5C–E, where the kiwifruits rotate with the curved guide rod and fall smoothly into the fruit collection end.

3.3.3 Control system design of end-effector

The hardware composition of the end-effector control system, as shown in Figure 6, includes a controller with an STM32 microcontroller at its core, two infrared recognition sensors, a photoelectric detection sensor, a Hall sensor, a rotary stepper motor, and a motor driver. The control system is powered by two power modules, where the sensor, microcontroller, and motor driver control signal voltage are 5 V, and the stepper motor driver operates at 24 V. The control system was programmed in C language. The microcontroller gathers sensor data, then controls the direction signal, number of pulses, as well as pulse frequency of the stepper motor driver after processing to achieve orderly control of the end-effector actions.

Figure 6

Figure 6 Hardware composition.

Two infrared identification sensors (Guangdong Xin de Electronics, model GP-2D12) were symmetrically installed on the outer edges of the side plates of the end-effector, as shown in Figure 3B. This model of sensor is capable of recognizing fruits within a range of 20 to 80 cm.

The photoelectric detection sensor (Zhongshan Covey Company, model CX-3LIKNP) was installed in the middle of the end-effector single-side baffle, as shown in Figure 3B. The signal from this sensor is a bar-shaped colorless light spot, with a detection width range of 10~70mm and a length range of 0~100cm, and only detects preset colors. Typically, matured kiwifruits in the orchard exhibit a brown color, while the fruit stalks are dark green. Therefore, this sensor is set to detect brown. Based on the parameters of the picking bin and fruit parameters (refer to section 3.3.1), the photoelectric detection sensor is set to detect a range of 0 to 20cm. When the kiwifruits enter the picking bin, the sensor triggers an alarm, and the alarm ceases once all the kiwifruits are inside the picking bin. When the alarm stops, the microcontroller commands the stepper motor driver to drive the rotary stepper motor, thereby driving the rotary cutter mechanism to sever the fruit stalks of the fruits in the picking bin.

To prevent the rotary stepping motor from over-traveling, the Hall position sensor (Tianjin Yue er xing Electronic Technology Co., Ltd., YS44E model) was installed on the fixed rod, as shown in Figure 3B.

The rotary stepper motor (Guangdong Lv wei Technology Motor Company, model 57BYG250B, motor driver model TB-6600) was installed on the connector and is responsible for driving the rotary cutter mechanism to cut the stalks of the fruits in the picking bin, as shown in Figure 3B. This model of stepper motor can provide a thrust of more than 60N and a torque of up to 6N.m at a set speed of 12r/min.

4 Kinematic analysis

Robot kinematics is primarily concerned with analyzing the relationship between the position of the end-effector and the joint variables of the robotic arm (Wang K. et al., 2020). This chapter lays the foundation for the trajectory planning and picking strategy of the robot in chapter 5 by calculating the forward and inverse solutions of the robot kinematics and analyzing the workspace, as well as providing theoretical guidance for the motion of the robotic arm during the experiment.

4.1 Forward kinematic analysis

The kinematic equations of the kiwifruit-picking robot were established using the D-H parameter method, and the connecting rod coordinates are shown in Figure 7. The D-H parameters for each joint were determined based on the connecting rod parameters and the relationships within the established coordinate system, as shown in Table 4.

Figure 7

Figure 7 The connecting rod coordinate system.

Table 4

Table 4 D-H parameters of the robotic arm.

The transformation matrix Equation 1 and Equation 2 between neighboring connecting rod coordinate systems are:

$\begin{array}{ll}{}_i^{i-1}T=\text{Rot}(z_{i-1},{\theta }_i)\times \text{Trans}(z_{i-1},d_i)\times \text{Trans}(x_i,a_i)\times \text{Rot}(x_i,{\alpha }_i)& (1)\end{array}$

$\begin{array}{ll}{}_i^{i-1}T=\left[\begin{array}{cccc}cos{\theta }_i& -sin{\theta }_{i}cos{\alpha }_i& sin{\theta }_{i}sin{\alpha }_i& a_{i}cos{\theta }_i\\ sin{\theta }_i& cos{\theta }_{i}cos{\alpha }_i& -cos{\theta }_{i}sin{\alpha }_i& a_{i}sin{\theta }_i\\ 0& sin{\alpha }_i& cos{\alpha }_i& d_i\\ \mathit{0}& \mathit{0}& \mathit{0}& \mathit{1}\end{array}\right]& (2)\end{array}$

Where ${}_i^{i-1}T$ represents the Homogeneous transformation matrix of the connecting rod relative to the connecting rod $i-1$;$\text{Trans}()$presents a function of the translation of the coordinate system about the axis; $\text{Rot}()$represents a function of the rotation of the coordinate system about the axis. Each connecting rod Substituting the parameters of each connecting rod in Table 4 into Equation 1 can obtain the transformation matrix Equation 3 of each connecting rod:

$\begin{array}{ll}\begin{array}{c}{}_1^{0}T=\left[\begin{array}{cccc}{c}_{\mathit{1}}& \mathit{0}& s_{\mathit{1}}& \mathit{0}\\ s_{\mathit{1}}& \mathit{0}& -c_{\mathit{1}}& \mathit{0}\\ \mathit{0}& \mathit{1}& \mathit{0}& \mathit{90}\\ \mathit{0}& \mathit{0}& \mathit{0}& \mathit{1}\end{array}\right] 2_1^{}T=\left[\begin{array}{cccc}{c}_{\mathit{2}}& -s_{\mathit{2}}& \mathit{0}& \mathit{425}{c}_{\mathit{2}}\\ s_{\mathit{2}}& c_{\mathit{2}}& \mathit{0}& \mathit{425}{s}_{\mathit{2}}\\ \mathit{0}& \mathit{0}& \mathit{1}& \mathit{90}\\ \mathit{0}& \mathit{0}& 0& \mathit{1}\end{array}\right] 3_2^{}T=\left[\begin{array}{cccc}{c}_{\mathit{3}}& -s_{\mathit{3}}& \mathit{0}& \mathit{390}{c}_{\mathit{3}}\\ s_{\mathit{3}}& c_{\mathit{3}}& \mathit{0}& \mathit{390}{s}_{\mathit{3}}\\ \mathit{0}& \mathit{0}& \mathit{1}& \mathit{0}\\ \mathit{0}& \mathit{0}& \mathit{0}& \mathit{1}\end{array}\right]\\ 4_3^{}T=\left[\begin{array}{cccc}{c}_{\mathit{4}}& \mathit{0}& s_{\mathit{4}}& \mathit{0}\\ s_{\mathit{4}}& \mathit{0}& -c_{\mathit{4}}& \mathit{0}\\ \mathit{0}& \mathit{1}& \mathit{0}& \mathit{110}\\ \mathit{0}& \mathit{0}& \mathit{0}& \mathit{1}\end{array}\right] 5_4^{}T=\left[\begin{array}{cccc}{c}_{\mathit{5}}& \mathit{0}& -s_{\mathit{5}}& \mathit{0}\\ s_{\mathit{5}}& \mathit{0}& c_{\mathit{5}}& \mathit{0}\\ \mathit{0}& -\mathit{1}& \mathit{0}& \mathit{90}\\ \mathit{0}& \mathit{0}& \mathit{0}& \mathit{1}\end{array}\right] {}_6^{5}T=\left[\begin{array}{cccc}{c}_{\mathit{6}}& -s_{\mathit{6}}& \mathit{0}& \mathit{0}\\ s_{\mathit{6}}& c_{\mathit{6}}& \mathit{0}& \mathit{0}\\ \mathit{0}& \mathit{0}& \mathit{1}& \mathit{80}\\ \mathit{0}& \mathit{0}& \mathit{0}& \mathit{1}\end{array}\right]\end{array}& (3)\end{array}$

Where the kiwifruit picking robot equation of motion $T_6^0$ is expressed as the position of the end-effector with respect to the base coordinate system:

$\begin{array}{ll}{}_6^{0}T=T_1^{0}T_2^{1}T_3^{2}T_4^{3}T_5^{4}T_6^5=\left[\begin{array}{cccc}{n}_x& o_x& a_x& p_x\\ n_y& o_y& a_y& p_y\\ n_z& o_z& a_z& p_z\\ 0& 0& 0& 1\end{array}\right]& (4)\end{array}$

Where $c_i$is an abbreviation of $cos{\theta }_i$, $s_i$is an abbreviation of $sin{\theta }_i$, T represents the corresponding change matrix, $\text{T}_1^0$, $\text{T}_2^1$, $\text{T}_3^2$, $\text{T}_4^3$, $\text{T}_5^4$, represent the change matrices for each joint, respectively. $\text{T}_5^0$is represented as the position of the end-effector relative to the base coordinate system.

To simplify writing, letting: $cos({\theta }_{I }+{\theta }_j)$ = $c_{ij}$, $sin({\theta }_{I }+{\theta }_j)$ = $s_{ij}$, $cos({\theta }_{I }+{\theta }_j+{\theta }_k)$ = $c_{ijk}$, $cos({\theta }_{I }+{\theta }_j+{\theta }_k)$ = $c_{ijk}$

And by substituting them into Equation 4, Equation 5 can be obtained:

Among them:

$\begin{array}{ll}\{\begin{array}{l}{n}_x=c_6(s_1{s}_5+c_5{c}_1{c}_{234})-s_6{c}_1{s}_{234}\\ n_y=c_6(c_5{s}_1{c}_{234}-c_1{s}_5)-s_6{s}_5{s}_{234}\\ n_z=c_5{c}_6{s}_{234}+s_6{c}_{234}\\ o_x=-s_6(c_5{c}_1{c}_{234}+s_1{s}_5)-c_6{c}_1{s}_{234}\\ o_y=-s_6(c_5{s}_1{c}_{234}-c_1{s}_5)-c_6{s}_1{s}_{234}\\ o_z=c_6{c}_{234}-c_5{s}_6{s}_{234}\\ a_x=-s_5{c}_1{c}_{234}+c_5{s}_1\\ a_y=-s_5{s}_1{c}_{234}-c_1{c}_5\\ a_z=-s_5{s}_{234}\\ p_x=90c_1{s}_{234}+110s_1+80(c_5{s}_1-s_5{c}_1{c}_{234})+425c_1{c}_2+390c_1{c}_{23}\\ p_y=90s_1{s}_{234}-110c_1-80(s_5{s}_1{c}_{234}+c_1{c}_5)+425c_2{s}_1+390s_1{c}_{23}\\ p_z=90-90c_{234}+425s_2+390s_{23}-80s_5{s}_{234}\end{array}& (5)\end{array}$

4.2 Inverse kinematic analysis

The kinematic inverse solution is that the angle of each joint is calculated by knowing the position and attitude of the robot’s end-effector to achieve precise positioning and control of the robot. In this section, the matrix inverse multiplication analytic method was used to solve the kinematic inverse solution of the kiwifruit picking robot.

(1) Solving for ${\theta }_1$、${\theta }_5$ and ${\theta }_6$:

Multiply both sides of Equation 4 to the left by $T_1^0{}^{-1}T_6^5{}^{-1}$:

$\begin{array}{ll}T_1^0{}^{-1}T_6^{0}T_6^5{}^{-1}=T_2^{1}T_3^{2}T_4^{3}T_5^4& (6)\end{array}$

The left side of the equation:

$$\left[\begin{array}{cccc}{c}_6(n_x{c}_1+n_y{s}_1)-s_6(o_x{c}_1+o_y{s}_1)& s_6(n_x{c}_1+n_y{s}_1)+c_6(o_x{c}_1+o_y{s}_1)& a_x{c}_1+a_y{s}_1& p_x{c}_1-80(a_x{c}_1+a_y{s}_1)+p_y{s}_1\\ n_x{c}_6-o_z{s}_6& o_z{c}_6+n_z{s}_6& a_z& p_z-90-80a_z\\ s_6(o_y{c}_1-o_x{s}_1)-s_6(n_y{c}_1-n_x{s}_1)& -s_6(n_y{c}_1-n_x{s}_1)-c_6(o_y{c}_1-o_x{s}_1)& a_x{s}_1-a_y{c}_1& -p_y{c}_1+80(a_y{c}_1-a_x{s}_1)+p_x{s}_1\\ 0& 0& 0& 1\end{array}\right]$$

The right side of the equation:

$$\left[\begin{array}{cccc}{c}_{\mathit{234}}{c}_{\mathit{5}}& -s_{\mathit{234}}& -c_{\mathit{234}}{s}_{\mathit{5}}& a_3{c}_{\mathit{23}}+a_{\mathit{2}}{c}_{\mathit{2}}+\mathit{90}{s}_{\mathit{234}}\\ s_{\mathit{234}}{c}_{\mathit{5}}& c_{\mathit{234}}& -s_{\mathit{234}}{s}_{\mathit{5}}& a_{\mathit{3}}{s}_{\mathit{23}}+a_{\mathit{2}}{s}_{\mathit{2}}-\mathit{90}{c}_{\mathit{234}}\\ s_{\mathit{5}}& \mathit{0}& c_{\mathit{5}}& \mathit{110}\\ \mathit{0}& \mathit{0}& \mathit{0}& \mathit{1}\end{array}\right]$$

According to the equality of the corresponding row 3, column 4 on the left and right sides of Equation 6, it can be seen Equation 7:

$\begin{array}{ll}-p_y{c}_1+80(a_y{c}_1-a_x{s}_1)+p_x{s}_1=110& (7)\end{array}$

Organize and obtain Equation 8:

$$(80a_y-p_y)c_1-(80a_x-p_x)s_1=110,\text{ and letting }80a_y-p_y=m,80a_x-p_x=n$$

$\begin{array}{ll}{\theta }_1=A\tan 2(m,n)-A\tan 2(110,\pm \sqrt{m^2+n^2-{110}^2})& (8)\end{array}$

From Equation 6, Equation 9 can be obtained $\{\begin{array}{l}{a}_x{s}_1-a_y{c}_1=c_5\\ s_6(o_y{c}_1-o_x{s}_1)-c6(n_y{c}_1-n_x{s}_1)=s_5\end{array}$:

$\begin{array}{ll}\{\begin{array}{l}{\theta }_5=\pm \arccos (a_x{s}_1-a_y{c}_1)\\ {\theta }_6=A\tan 2(\frac{n_x{s}_1-n_y{o}_x}{s_5},\frac{o_x{s}_1-o_y{c}_1}{s_5})\end{array}& (9)\end{array}$

(2) Solving for ${\theta }_2$、 ${\theta }_3$ and ${\theta }_4$:

Multiply both sides of Equation 4 to the right by $T_5^4{}^{-1}$:

$\begin{array}{ll}T_1^0{}^{-1}T_6^{0}T_6^5{}^{-1}T_5^4{}^{-1}={ }_2^{1}T{3}_2^{}T{4}_3^{}T& (10)\end{array}$

Using Equation 10 the left and right sides of row 1, column 4 correspond to be equal, and row 2, column 4 correspond to be equal can be shown:

$\begin{array}{ll}\{\begin{array}{l}90\left(s_6\left(n_x{c}_1+n_y{s}_1\right)+c_6\left(o_x{c}_1+o_y{s}_1\right)\right)-90\left(a_x{c}_1+a_y{s}_1\right)+p_x{c}_1+p_y{s}_1=a_3{c}_{23}+a_2{c}_2\\ p_z-90-80a_z+90\left(o_z{c}_6+n_z{s}_6\right)=a_3{s}_{23}+a_2{s}_2\end{array}& (11)\end{array}$

Combined with the above method of finding the value, according to Equation 11 can be obtained Equation 12:

$\begin{array}{ll}\{\begin{array}{l}{\theta }_3=\pm \arccos \left(\frac{m_2+n_2-a_2{}^2-a_3{}^2}{2a_2{a}_3}\right)\\ {\theta }_2=A\tan 2\left(\frac{(a_3{c}_3+a_2)n-a_3{s}_{3}m}{a_2{}^2+a_3{}^2+2a_2{a}_3{c}_3},\frac{m+a_3{s}_3{s}_2}{a_3{c}_3+a_2}\right)\\ m=90\left(s_6\left(n_x{c}_1+n_y{s}_1\right)+c_6\left(o_x{c}_1+o_y{s}_1\right)\right)-90\left(a_x{c}_1+a_y{s}_1\right)+p_x{c}_1+p_y{s}_1\\ n=p_z-90-80a_z+90\left(o_z{c}_6+n_z{s}_6\right)\end{array}& (12)\end{array}$

According to the equality of the corresponding 2nd row, 2nd column and 1st row, 2nd column on the left and right sides of Equation 6, it can be seen Equation 13:

$\begin{array}{ll}{\theta }_4=A\tan 2\left(-s_6\left(n_x{c}_1+n_y{s}_1\right)-c_6\left(o_x{c}_1+o_y{s}_1\right),o_z{c}_6+n_z{s}_6\right)-{\theta }_2-{\theta }_3& (13)\end{array}$

4.3 Workspace analysis

The workspace is the collection of spatial points that can be reached by the end-effector during the picking process of the robot, and its shape and size are the key factors that determine the robot’s work performance. Based on the Robotics Toolbox in MATLAB software, the Monte Carlo method was used to solve the approximate solution in motion space (Xu Z. et al., 2018). Through the forward kinematics Equation 4, the position vector of the end-effector relative to the base was solved, and then multiple end position coordinate points were randomly generated by the Rand function to obtain the workspace as shown in Figure 8.

Figure 8

Figure 8 Workspace of the kiwifruit picking robot: (A) XOZ planar projection, (B) YOZ planar projection, (C) XOY planar projection, (D) Three-dimensional space.

From Figure 8, the workspace of the kiwifruit-picking robot is a heart-like columnar space, X-direction travel is about 1.4 m; Y-direction travel is about 1.8 m; and Z-direction travel is about 2.0 m. And the dexterous workspace is shown in Figure 8C. The space avoids collisions between the robotic arm and the internal branches of the kiwifruit tree, as well as interference between the arm and the mechanics of the robot. During the actual picking task, the position of the picked kiwifruit should be within the dexterity workspace. The simulation results provide the foundation for the trajectory planning of the robotic arm and the design of the picking strategy in the subsequent section.

5 Picking trajectory planning

When the robot has finished picking kiwifruit $i-1$ and will pick kiwifruit $i$, the picking motion can be divided into three phases:

1. Leaving the kiwi $i-1$ place, avoiding branches and leaves.

2. Moving within a safe area where there will be no collision with the branching vine, approaching the target kiwi $i$.

3. Arriving at the target kiwi, the end-effector performs the picking.

During the transfer of the picking target, it was necessary to ensure that the branches of the fruit tree don’t interfere with the robot. During trajectory planning, a safety point H was added to split the motion of the second stage into two parts to ensure that the positional transitions of the robotic arm could take place in a safe area. As a result, each picking task involved the planning of four trajectory segments and five points, as illustrated in Figure 9.

Figure 9

Figure 9 Trajectory planning node between two picks.

When $i\ne 0$, the robot has just finished picking kiwifruit $i-1$and will pick kiwifruit $i$. The path for trajectory planning is:

$$P_{i-1}\to P{r}_{i-1}\to H_i\to P{r}_i\to P_i$$

When $i=0$, the robot has just started to work, the robot arm is in the zero position, and the coordinates position of the end-effector is the point $\text{H}$at this time. The path for trajectory planning is:

$$H_0\to P{r}_0\to P_0$$

5.1 Trajectory planning of the robotic arm

The purpose of robotic arm trajectory planning is to ensure that the position, velocity, and acceleration function curves of each joint variable are continuous and smooth during the completion of operational tasks (Yang and Cheng, 2018). When the kiwifruit-picking robot carries out fruit-picking work, the movement of the end-effector should follow to a specific trajectory. The continuous trajectory function minimizes the impact vibrations on each joint of the robotic arm during operation, resulting in smoother robot operation. To meet actual requirements, joint space trajectory planning was implemented for real-time control of the robotic arm by adjusting each joint angle.

Knowing the joint position, velocity, and acceleration at the initial and termination moments, a fifth-degree polynomial can be used to ensure smooth continuity of the joint angle, angular velocity, and angular acceleration, that is:

$\begin{array}{ll}\theta (t)=a_0+a_{1}t+a_2{t}^2+a_3{t}^3+a_4{t}^4+a_5{t}^5& (14)\end{array}$

If the six constraints on the angle, angular velocity, and angular acceleration of the initial and terminating joints are:

$\begin{array}{ll}\theta (t_0)={\theta }_0,\theta (t_f)={\theta }_f,\dot{\theta }(t_0)={\dot{\theta }}_0,\dot{\theta }(t_f)={\dot{\theta }}_f,\ddot{\theta }(t_0)={\ddot{\theta }}_0,\ddot{\theta }(t_f)={\ddot{\theta }}_f& (15)\end{array}$

By solving for the first and second-order derivatives of Equation 14, the angular velocity function can be obtained as:

$\begin{array}{ll}\{\begin{array}{l}{\theta }_f=a_0+a_{1}t+a_2{t}^2+a_3{t}^3+a_4{t}^4+a_5{t}^5\\ {\dot{\theta }}_f=a_1+2a_{2}t+3a_3{t}^2+4a_4{t}^3+5a_5{t}^4\\ {\ddot{\theta }}_f=2a_2+6a_{3}t+12a_4{t}^2+20a_5{t}^3\end{array}& (16)\end{array}$

Combining Equations 14–16, the coefficients of the fifth-degree polynomial can be obtained as Equation 17:

$\begin{array}{ll}\{\begin{array}{l}{a}_0={\theta }_0\\ a_1={\dot{\theta }}_0\\ a_2=\frac{{\ddot{\theta }}_0}{2}\\ a_3=\frac{20{\theta }_f-20{\theta }_0-(8{\dot{\theta }}_f+12{\dot{\theta }}_0)t_e-(3{\ddot{\theta }}_0-{\ddot{\theta }}_f)t_f^2}{2t_f^3}\\ a_4=\frac{30{\theta }_0-30{\theta }_f+(14{\dot{\theta }}_f+16{\dot{\theta }}_0)t_e-(3{\ddot{\theta }}_0-2{\ddot{\theta }}_f)t_f^2}{2t_f^4}\\ a_5=\frac{12{\theta }_f-12{\theta }_0-(6{\dot{\theta }}_f+6{\dot{\theta }}_0)t_e-({\ddot{\theta }}_0-{\ddot{\theta }}_f)t_f^2}{2t_f^5}\end{array}& (17)\end{array}$

A time-varying function is utilized to represent each joint variable during joint space trajectory planning, by which the process of trajectory change of the robot over a certain period of time is illustrated. The start point coordinate ${\text{q}}_0$and end point coordinate ${\text{q}}_1$of the robot are given.Utilizing thefunction, the call format is Equation 18:

$\begin{array}{ll}[\begin{array}{ccc}\dot{o}& \dot{p}& \dot{q}\end{array} ]=jtraj({\text{q}}_0,{\text{q}}_1,\text{t})& (18)\end{array}$

Where: $\dot{o}$ stands for displacement; $\dot{p}$ stands for velocity; $\dot{q}$ stands for acceleration; $t$stands for simulation time. The function $\text{jtraj}$is a fifth-degree polynomial interpolation, with default initial and final velocities set to zero. The simulation time was 2s, and the $\text{jtraj}$function was called to observe the motion change process of each joint. The plot function was then used to generate the displacement, angular velocity, and angular acceleration curves of the robot, as shown in Figure 10.

Figure 10

Figure 10 Dynamics of the joints over time: (A) displacement curve, (B) velocity curve, (C) acceleration curve.

Figure 10A reflects the movement process of each joint over time. Figure 10B reflects the velocity variation of each joint over time, and they satisfy the requirement that the starting and ending velocities of the fifth-degree polynomial interpolation are all zero. The change in acceleration for each joint is depicted in Figure 10C and the curved motion is smooth without abrupt changes. Velocity and acceleration curves are smooth and excessive, without inflection points, interruptions, jumps, and other phenomena, indicating that the robot ran smoothly.

5.2 Calculation of critical nodes of the path

There are several critical nodes in the path of picking: picking point $P_{i-1}$, picking point $P_i$, approach point $P{r}_{i-1}$, approach point $P{r}_i$, safety point $H_i$. The critical nodes constrain the position in the picking path to avoid the interference of branches and leaves to the robotic arm. Therefore, the position of critical nodes directly affects the motion trajectory of the robot.

The critical nodes can be designed by the RRT (Rapidly Expanding Randomized Tree) algorithm. RRT algorithm is a random sampling algorithm that uses the starting point as the root node, increases the number of nodes by random same, piling, and connects the nodes to generate a random tree (Li et al., 2020; Wang J. et al., 2020; Wang L. et al., 2023). The nodes that do not satisfy the constraint requirements are discarded during the generation of the next node. When the random tree is able to contain the goal point or enter within the goal area, then there exists a route that connects the start point to the endpoint.

The critical nodes in the picking path were computed using the RRT algorithm.Settling the current picking position as $p_i$, random sampling in the free space to get the node as $p_{rand1}$, at this time there are only two nodes $p_i$ and $p_{rand1}$, so $p_i$that is the closest node to $p_{rand1}$. Connecting the two nodes and getting the set m (k is a coefficient) of the connecting line transits, which is Equation 19:

$\begin{array}{ll}m=\left\{m|m=p_i+k\left(p_{rand}-p_i\right)\right\}& (19)\end{array}$

If the path point m crosses the obstacle space $C_{\mathit{0}}$, which is Equation 20:

$\begin{array}{ll}m\in C_0& (20)\end{array}$

Then the node $p_{rand1}$is discarded and a new node $p_{rand2}$ is generated from scratch. If (as shown in Equation 21)

$\begin{array}{ll}m\notin C_0& (21)\end{array}$

Then grow one step from $p_i$ towards $p_{rand}$ to get the new node $p_{i+1}$. Then continuing to randomly generate new nodes, at this time it is necessary to determine the nearest node with which to connect, so the distance d is calculated as Equation 22:

$\begin{array}{ll}d=\Vert p_{rand3}-p_k\Vert & (22)\end{array}$

After computational comparison, $p_{rand3}$ selected the closest node for connection and followed the same method as before to determine the obstacle space of the path and finally generate a new node. By analogy, all path nodes before reaching the next picking point can be obtained.

The environmental parameters during the kiwifruit picking cycle were set and the critical nodes were calculated using the RRT algorithm program. The simulation results are shown in Figure 11, which shows that the upper black area simulates the branch obstacle space, and the lower black area simulates the body of the kiwifruit-picking robot, with the red dots in the upper left corner representing the last kiwifruit picked, and the green dots in the upper right corner representing the next kiwifruit to be picked.

Figure 11

Figure 11 Generation of critical nodes in the picking path: (A) step 100, (B) step 80, (C) step 60, (D) step 40.

5.3 Dynamic simulation of picking trajectory

Based on the kinematic forward and inverse solutions and the workspace analysis of the picking robot in Chapter 4, the MATLAB software was applied to simulate the trajectory of the kiwifruit picking robot. Two clusters of fruits located at different heights on a kiwifruit tree were used as an example to simulate the growth of kiwifruit grown on a trellis in an orchard, and the simulation results are shown in Figure 12 (Brown areas represent kiwifruit trees, red areas represent kiwifruit clusters, and green areas are modeled as the spatial distribution of fruit).

Figure 12

Figure 12 Dynamic simulation of the picking trajectory: (A) t=0 s, (B) t=4 s, (C) t=8 s, (D) t=16 s.

Starting from recognizing the first cluster of kiwifruits and ending with recognizing the second cluster of kiwifruits, the robot took a total time of 16 seconds. At t = 0 s, the robot is in the initial state. At 1~4 s, the robot follows the preset picking route, and the robotic arm smoothly avoids the branches and leaves and delivers the end-effector to the fruit picking area. At 5-8 s, the end-effector recognizes the first cluster of fruits and completes the harvesting of the first fruit cluster by enveloping the fruits, cutting off the fruit stalks, collecting the fruits, and other actions. At 9-16 s, the robotic arm first returns to the safe area and then follows the preset picking route to recognize the second cluster of kiwifruits and complete the picking without disturbing other fruits. The simulation results show that the time for the robot to pick a cluster of fruits was 8s, and the picking trajectory met the requirements of the kiwifruit picking operation space.

6 Test and results

6.1 Test bed set Up

In the later experimental phase of this study, due to the influence of kiwifruit fruit harvesting season, a test bed was set up in the laboratory environment to simulate a kiwifruit orchard for the picking experiment, as shown in Figure 13. The test bed mainly includes: the AUBO-E5 robotic arm test bed, trellis model, the kiwifruit picking end-effector and its control system, Leno R7000P portable computer, collection device, etc.

Figure 13

Figure 13 Kiwifruit harvesting test bed: (A) trellis model, (B) end-effector, (C) AUBO-E5 test bed, (D) collection device, (E) test scenarios.

6.2 Test method

The picking experiment was carried out on August 3, 2023. AUBO-E5 robotic arm was used to assist the end-effector operation, and the picking operation flow is shown in Figure 7 (refer to section 3.2). In this paper, 125 kiwifruits of different shapes were selected for testing to verify the harvesting performance of the end-effector. Based on the characteristics of kiwifruit clusters, 125 fruits were divided into 5 groups according to the number of fruits per cluster: 3, 4, 5, 6, and 7, and each group had 5 clusters with the same number of fruits. In this section, the picking success rate, average picking time per picking, and picking damage rate were used as the evaluation indexes for fast and non-destructive picking. The definitions are shown in Equations 23–25 below:

$\begin{array}{ll}T=\frac{t_0-t_1}{5}& (23)\end{array}$

$\begin{array}{ll}S=\frac{n}{h}\times 100\%& (24)\end{array}$

$\begin{array}{ll}D=\frac{m}{n}\times 100\%& (25)\end{array}$

Where T is the average picking time (includes recognizing fruit clusters, enveloping fruit clusters, and cutting the fruit stalks of the fruit in the picking bin); t₀ is the time of the start of picking; t₁ is the time of the end of picking; S is the success rate; n is the number of fruits picked successfully; h is the total number of fruits; D is the damage rate of fruits picked successfully; and m is the number of damaged fruits.

6.3 Analysis of results

Picking experiments were carried out on 5 groups of 25 clusters of fruits respectively, as shown in Figure 14. The robotic arm first sent the end-effector to the area suitable for picking fruits, and after the fruits were detected by the recognition sensor, the robotic arm carried the end-effector upwardly to envelop and pick the fruits, and the picked fruits fell into a fruit collection box through the hose.

Figure 14

Figure 14 Kiwifruit picking process: (A) recognizing the fruits (front view and side view), (B) the process from the beginning of the envelopment of the fruit to the complete envelopment of the fruit, (C) the process of rotating the cutter mechanism to cut off all fruit stalks in the picking bin, (D) during the resetting process, the picked kiwifruits fall through the fruit collection end along the hose into the fruit collection box. (represents rotary cutter real-time status, represents fruit real-time status).

During the above fruit-picking process, the robotic arm carried the end-effector picking operation at a speed of 1.5 m/s, and the stepper motor carried the rotary cutter at a speed of 6 r/min to separate the fruits from the fruit stalks, and the results of the picking experiments as shown in Table 5.

Table 5

Table 5 Results of picking experiments.

According to Table 5, the average time for picking each cluster of fruits was 9.7 seconds, with a success rate of 88.0% and a damage rate of 7.3%. When the number of fruits picked in each cluster was 3~4, the picking success rate was 100%, and the fruit damage rate was 0. When the number of fruits picked in each cluster was 5~7, the more fruits picked in each cluster, the lower the picking success rate, and the higher the fruit damage rate.

The causes of fruit damage were analyzed as follows: (Figure 15A represents the unpicked fruit from fruit trees; Figure 15B represents the undamaged picked fruits) ① Fruit leaves interfered with the color detection sensor, leading to damage to one or more fruits by the rotary cutter before they all entered the picking bin, as shown in Figure 15C. ② During picking or collection, fruit-to-fruit collisions could result in damage to the skin of overripe fruits (Note: Typically, when kiwifruit is picked in an orchard, the fruit is not fully ripe, and it is firmer, reducing the likelihood of skin damage), as shown in Figure 15D.

Figure 15

Figure 15 Status of picked fruits: (A) unpicked fruit from fruit trees, (B) undamaged picked fruits, (C) fruits damaged by the rotary cutter before all of the fruits entered the picking bin, (D) fruits with mutual collisions resulting in damage to the epidermis.

The factors contributing to fruit-picking failures were analyzed as follows. While kiwifruit grown on trellises typically fall within the same height range, the fruits become more irregular when a fruit cluster contains a higher number of fruits. Because the picking bin has a fixed size, some fruits may be blocked outside the picking bin during the process of enveloping fruits. However, the robot doesn’t adjust the enveloping angle correspondingly, leading to the failure to pick those fruits. Subsequently, there might be a second failure to pick the initially unpicked fruits within the cluster due to interference from branches or leaves.

7 Conclusion

1. For kiwifruit grown in orchard trellises, a multi-fruit picking kiwifruit robot was designed by analyzing kiwifruit cultivation parameters and design requirements. A continuous picking method of automatically recognizing fruits - fully enveloping fruits - non-destructively separating fruit stalks - and automatically collecting fruits was proposed, which can pick kiwifruits in clusters. The designed picking end-effector, after the fruit are recognized by sensors, uses envelopment to gather the fruits into the picking bin and then employs a rotary cutter mechanism to separate the fruit stalks without causing damage. Additionally, curved guide rods were also set to buffer and guide the picked fruit to reduce the rate of fruit damage.

2. The kinematic equations of a kiwifruit picking robot were established by the D-H method, and forward and inverse kinematic calculations were performed. Firstly, the robot reachable workspace and dexterous workspace ranges were solved using Robotics Toolbox in MATLAB software and the Monte Carlo method. Secondly, the trajectory planning of the robotic arm was performed by using the fifth-degree polynomial interpolation method, and the critical nodes in the picking path were calculated by using the RRT algorithm, furthermore, the scheme of the robot’s picking strategy was given. Finally, MATLAB was applied to simulate the motion trajectory of the kiwifruit picking robot to verify the feasibility of the trajectory planning scheme and the picking strategy.

3. A kiwifruit-picking test bed was established in a laboratory. A total of 125 fruits kiwifruits were selected and divided into five groups in the form of fruit clusters. The picking test was conducted with the mechanical arm set at a speed of 1.5m/s and the stepper motor running at 6r/min. The results showed that the average time required to pick each fruit cluster was 9.7 seconds, with a picking success rate of 88.0% and a picking damage rate of 7.3%. Moreover, the causes of fruit damage and fruit picking failures during the picking process were analyzed and elaborated in detail.

4. The research work of this study has a certain significance for the simultaneous harvesting of multiple fruits in kiwifruit, but it still needs to be further investigated in the future. For example, ① When the kiwifruit ripening season of the next year is approaching, picking experiments are conducted in real environments of kiwifruit orchards to test various index parameters of kiwifruit. ② Research on more advanced detection systems, on the one hand, can allow the robot to accurately locate the target fruit tree, on the other hand, can allow the end-effector to avoid the interference of external conditions such as fruit leaves, light, etc., to accurately locate the position of the target fruit. ③ The end-effector is further designed from the lightweight point of view on the premise of being able to achieve the target picking index.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material. Further inquiries can be directed to the corresponding author.

Author contributions

MF: Conceptualization, Methodology, Validation, Writing – original draft. SG: Conceptualization, Formal analysis, Methodology, Software, Validation, Writing – original draft, Writing – review & editing. AC: Formal analysis, Supervision, Writing – review & editing. RC: Software, Writing – review & editing. XC: Supervision, Writing – review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This research was funded by the National Natural Science Foundation of China (Grant No. 51975114).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Arad, B., Balendonck, J., Barth, R., Ben-Shahar, O., Edan, Y., Hellström, T., et al. (2020). Development of a sweet pepper harvesting robot. J. Field Robot. 37 (6), 1027–1039. doi: 10.1002/rob.21937

CrossRef Full Text | Google Scholar

Fang, W., Wu, Z., Li, W., Sun, X., Mao, W., Li, R., et al. (2023). Fruit detachment force of multiple varieties kiwifruit with different fruit-stem angles for designing universal robotic picking end-effector. Comput. Electron Agric. 213, 108225. doi: 10.1016/j.compag.2023.108225

CrossRef Full Text | Google Scholar

Fazayeli, A., Kamgar, S., Nassiri, S. M., Fazayeli, H., de la Guardia, M. (2019). Dielectric spectroscopy as a potential technique for prediction of kiwifruit quality indices during storage. Inf. Process. 6 (4), 479–486. doi: 10.1016/j.inpa.2019.02.002

CrossRef Full Text | Google Scholar

Fu, L., Tola, E., Al-Mallahi, A., Li, R., Cui, Y. (2019). A novel image processing algorithm to separate linearly clustered kiwifruits. Biosyst. Eng. 183, 184–195. doi: 10.1016/j.biosystemseng.2019.04.024

CrossRef Full Text | Google Scholar

Fu, L., Zhang, F., Ge, Y., Li, Z., Wang, B., Cui, Y. (2015). Development and experiment of end-effector for kiwifruit harvesting robot. Trans. Chin. Soc Agric. Mach. 46 (3), 1–8. doi: 10.6041/j.issn.1000-1298.2015.03.001

CrossRef Full Text | Google Scholar

Gao, C., He, L., Fang, W., Wu, Z., Jiang, H., Li, R., et al. (2023). A novel pollination robot for kiwifruit flower based on preferential flowers selection and precisely target. Comput. Electron Agric. 207, 107762. doi: 10.1016/j.compag.2023.107762

CrossRef Full Text | Google Scholar

García-Quiroga, M., Nunes-Damaceno, M., Gómez-López, M., Arbones-Maciñeira, E., Muñoz-Ferreiro, N., Vázquez-Odériz, M. L., et al. (2015). Kiwifruit in syrup: Consumer acceptance, purchase intention and influence of processing and storage time on physicochemical and sensory characteristics. Food Bioproc Tech. 8, 2268–2278. doi: 10.1007/s11947-015-1571-3

CrossRef Full Text | Google Scholar

Hohimer, C. J., Wang, H., Bhusal, S., Miller, J., Mo, C., Karkee, M. (2019). Design and field evaluation of a robotic apple harvesting system with a 3D-printed soft-robotic end-effector. T ASABE. 62 (2), 405–414. doi: 10.13031/trans.12986

CrossRef Full Text | Google Scholar

Kuznetsova, A., Maleva, T., Soloviev, V. (2020). Using YOLOv3 algorithm with pre-and post-processing for apple detection in fruit-harvesting robot. Agronomy 10 (7), 1016. doi: 10.3390/agronomy10071016

CrossRef Full Text | Google Scholar

Ling, X., Zhao, Y., Gong, L., Liu, C., Wang, T. (2019). Dual-arm cooperation and implementing for robotic harvesting tomato using binocular vision. Robot Auton Syst. 114, 134–143. doi: 10.1016/j.robot.2019.01.019

CrossRef Full Text | Google Scholar

Li, Y., Wei, W., Gao, Y., Wang, D., Fan, Z. (2020). PQ-RRT*: an improved path planning algorithm for mobile robots. Expert Syst. Appl. 152, 113425. doi: 10.1016/j.eswa.2020.113425

CrossRef Full Text | Google Scholar

Mehta, S. S., Burks, T. F. (2014). Vision-based control of robotic manipulator for citrus harvesting. Comput. Electron Agric. 102, 146–158. doi: 10.1016/j.compag.2014.01.003

CrossRef Full Text | Google Scholar

Miao, Y. B., Zheng, J. F. (2019). Development of compliant constant-force mechanism for end effector of apple picking robot. Trans. Chin. Soc Agric. Eng. 35 (10), 19–25. doi: 10.11975/j.issn.1002-6819.2019.10.003

CrossRef Full Text | Google Scholar

Mu, L., Cui, G., Liu, Y., Cui, Y., Fu, L., Gejima, Y. (2020). Design and simulation of an integrated end-effector for picking kiwifruit by robot. Inf. Process. 7 (1), 58–71. doi: 10.1016/j.inpa.2019.05.004

CrossRef Full Text | Google Scholar

Rong, J., Fu, J., Zhang, Z., Yin, J., Tan, Y., Yuan, T., et al. (2022). Development and evaluation of a watermelon-harvesting robot prototype: vision system and end-effector. Agronomy 12 (11), 2836. doi: 10.3390/agronomy12112836

CrossRef Full Text | Google Scholar

Silwal, A., Davidson, J. R., Karkee, M., Mo, C., Zhang, Q., Lewis, K. (2017). Design, integration, and field evaluation of a robotic apple harvester. J. Field Robot. 34 (6), 1140–1159. doi: 10.1002/rob.21715

CrossRef Full Text | Google Scholar

Sun, Q., Zhong, M., Chai, X., Zeng, Z., Yin, H., Zhou, G., et al. (2023). Citrus pose estimation from an RGB image for automated harvesting. Comput. Electron Agric. 211, 108022. doi: 10.1016/j.compag.2023.108022

CrossRef Full Text | Google Scholar

Wang, J., Chi, W., Li, C., Wang, C., Meng, M. Q. H. (2020). Neural RRT*: Learning-based optimal path planning. IEEE Trans. Autom. Sci. Eng. 17 (4), 1748–1758. doi: 10.1109/TASE.2020.2976560

CrossRef Full Text | Google Scholar

Wang, T., Du, W., Zeng, L., Su, L., Zhao, Y., Gu, F., et al. (2023). Design and testing of an end-effector for tomato picking. Agronomy 13 (3), 947. doi: 10.3390/agronomy13030947

CrossRef Full Text | Google Scholar

Wang, Y., Xu, H., Zhang, M., Ma, J., Liu, B., He, Y. (2018). Design and experiment of bite-model end-effector for citrus harvesting by simulating with mouth of snake. Trans. Chin. Soc Agric. Mach. 49 (10). doi: 10.6041/j.issn.1000-1298.2018.10.007

CrossRef Full Text | Google Scholar

Wang, L., Yang, X., Chen, Z., Wang, B. (2023). Application of the improved rapidly exploring random tree algorithm to an insect-like mobile robot in a narrow environment. Biomimetics 8 (4), 374. doi: 10.3390/biomimetics8040374

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, K., Zhang, W., Luo, Z., Zhank, Y. (2020). Design and experiment of hitting pine cone picking robot. Trans. Chin. Soc Agric. Mach. 51 (8), 26–33. doi: 10.6041/j.issn.1000-1298.2020.08.003

CrossRef Full Text | Google Scholar

Williams, H., Ting, C., Nejati, M., Jones, M. H., Penhall, N., Lim, J., et al. (2020). Improvements to and large-scale evaluation of a robotic kiwifruit harvester. J. Field Robot. 37 (2), 187–201. doi: 10.1002/rob.21890

CrossRef Full Text | Google Scholar

Xu, L., Liu, X., Zhang, K., Xing, J., Yuan, Q., Chen, J., et al. (2018). Design and test of end-effector for navel orange picking robot. Trans. Chin. Soc Agric. Eng. 34 (12), 53–61. doi: 10.11975/j.issn.1002-6819.2018.12.007

CrossRef Full Text | Google Scholar

Xu, Z., Zhao., Z., He, S., He, J., Wu, Q. (2018). Improvement of Monte Carlo method for robot workspace solution and volume calculation. Optics precis Eng. 26 (11), 2703–2713. doi: 10.3788/OPE.20182611.2703

CrossRef Full Text | Google Scholar

Zhang, F., Zhang, X., Yu, K., Li, K., Hou, J., Yang, Y., et al. (2023). Preparation and microcellular foaming of crosslinked polyethylene-octene elastomer by ionic modification. J. Supercrit Fluids. 202, 106035. doi: 10.1016/j.supflu.2023.106035

CrossRef Full Text | Google Scholar