1. Introduction
Citrus is one of the agricultural products grown on a large scale and a fruit very close to the public. The current citrus picking operation mainly relies on human labor, which is labor-intensive and has high production cost, and the human picking cost reaches more than 50% of the total cost [1]. With the development of science and technology, the hope of reducing human picking costs is placed on the development of robots that can replace workers in picking work, and the development of citrus picking robots can help improve picking efficiency and reduce citrus production costs [2].
Since the mid-1980s, robots and technologies related to citrus picking have been appearing, such as the fruit identification machine taught by Professor Koichi Ozaki of the Faculty of Engineering at Utsunomiya University in Japan, which consists of three cameras and a polarizing filter that identifies the location of the fruit [3].
Citrus picking robots have developed to the present, mostly semi-automatic, still need human assistance, cannot really be free from human. Most of the robots are four-wheeled or two-wheeled, and can only pick on flatter terrain. Many citruses are produced from small plantations where the land is not flat and continuous. The use of the machine requires the user to learn the operation manual, and the robot is highly integrated, so once some parts are damaged during the work, they all need to be replaced, and the maintenance cost is high, which cannot effectively reduce the labor cost.
In this paper, we propose to design a fully automated crawler-type citrus picking robot, which can realize the functions of automatic walking, automatic citrus identification, automatic picking, automatic unloading, automatic reset and automatic fault checking, etc. We hope to solve the problems that previous citrus picking robots cannot effectively reduce labor cost, high maintenance cost and do not adapt to uneven land. The advantage of this design is the reduction of human intervention. The machine is easy to use, low threshold, and can be operated by users without professional knowledge. The modular design and automatic fault detection function reduce the difficulty of maintenance and achieve a more effective reduction of human cost.
2. Systematic Architecture Design
In order to realize the basic functions of the robot, five modules corresponding to the functions are designed, and each module is designed as follows.
1) Automatic obstacle avoidance: In order to realize the automatic obstacle avoidance function, the HC-SR04 ultrasonic module is used to detect obstacles. The ultrasonic module continuously emits ultrasonic waves outward and detects the echoes, and transmits the detected data to the STM32 main controller after detecting the echoes, and the STM32 main controller calculates after the formula
where s is the measured distance, t is the time from the emission of ultrasonic waves to the reception of the echo.
It can then be derived whether there is an obstacle ahead and the distance of the obstacle.
2) Automatic identification: OpenMV4 camera is used to realize the automatic identification function. After the robot is powered on, the camera starts to rotate according to the predetermined program to find the citrus, and after finding the citrus, the data is transmitted to STM32, which makes the corresponding action according to the data. After the citrus is picked and dropped into the car box, the camera loses the target and starts to search again until the robot detects the need to unload.
3) Automatic picking function: The three-degree-of-freedom robotic arm is used to realize the picking function, and the camera is used to realize the automatic picking function. The robotic arm only receives signals from the STM32 to perform the action and does not transmit data itself. After getting the incoming data from the camera, the STM32 converts the data into a pulse signal to the robotic arm servo, and the robotic arm receives the corresponding pulse signal to make the picking and reset action [4].
4) Automatic unloading function: In order to realize the automatic unloading function, the cart box has built-in load cell and infrared sensor. The infrared sensor is installed at the top of the box, when picking citrus to a certain number, it will be detected by the infrared sensor at the top of the box, then the infrared sensor transmits data to STM32, the load cell is installed with the bottom of the box, when picking citrus to a certain number more than the set weight, the load cell transmits data to STM32, when both sensors transmit data, the representative needs to go back to unload [5]. The trunk is controlled by two servos, one at the bottom of the trunk and one controlling the door, when the trunk receives the unloading signal from STM32, the servo at the bottom of the trunk rotates to make the trunk tilt, while the other servo controls the door to open, and after the trunk tilts to the preset angle, the trunk is reset and the door is closed at the same time.
5) Automatic walking function: using two 12V motors, the motor both receive STM32 signal and transmits data, the motor receives signal from STM32 through DRV8833 driver chip, STM32 sends signal to the motor according to the data returned from other modules, the motor receives signal and rotates. When the motor starts to rotate, the motor comes with a Hall sensor, which will transmit its own speed to the STM32 through a pulse signal, and the STM32 derives the speed according to this signal, and then uses the PWM signal to control the motor speed after the PID algorithm is rectified in the program [6].
This design of citrus picking robot mainly consists of five modules and STM32 main controller, as shown in Figure 1. Five modules are connected to each other through STM32F103ZET6, the camera and automatic obstacle avoidance module pass data to STM32, and STM32 controls the motor and robot arm rotation according to the data, the car box passes information to STM32, and also receives STM32 information for unloading and open and close the door action.
3. System Mechanical Structure Design
The mechanical structure is assembled by five modules, which are as modular as possible and made into disassembled parts to facilitate robot transportation and user maintenance. The three-dimensional modeling of the citrus picking robot is shown in Figure 2.
In terms of the robot’s mechanical structure, the structure is divided into the following parts.
![]()
Figure 1. System control diagram of the citrus picking robot.
1) Automatic obstacle avoidance module: In order to adapt to the vibration of citrus picking robot movement, the ultrasonic module of automatic obstacle avoidance, STM32 main controller and corresponding accessories, power supply, etc. need to be protected by a box, which leaves a hole for the ultrasonic module to send out ultrasonic waves and receive echoes, and the rest is wrapped in the box to isolate the fragile circuit part from the outside world.
2) Automatic identification module: including OpenMV camera and a two-degree of freedom gimbal, this part is independent relative to the STM32 main controller. OpenMV has its own processor, which processes the data of the camera and controls the gimbal rotation. OpenMV’s processor processes the data while sending pulse signals to control the gimbal rotation according to the predefined program, so that the camera rotates to find the citrus [7] and realize the function of automatic identification of citrus. The actual camera is as follows Figure 3.
3) Automatic picking module: automatic picking needs three degrees of freedom, so the mechanical arm with four servos is chosen, which can rotate, reach forward and reach upward. The end has a gripper to hold the citrus, and a hose is installed under the gripper, so the citrus rolls directly into the box after clamping. The actual robot arm is as follows Figure 4.
![]()
Figure 4. Mechanical arm physical picture.
4) Automatic unloading module: use acrylic transparent board to make the cart box, convenient to make while you can clearly see the situation inside the cart box. The location where the citrus is placed after picking, the cart box also adds cushioning material to protect the citrus from damage due to collision or bumps. It is connected to the robot arm through a hose.
5) Auto travel module: including tracks, motors and Hall sensors for motors. The citrus planting land is mostly mud, and the citrus picking robot will work on uneven land most of the time. Using tracks instead of ordinary tires can greatly improve the robot’s adaptation to the citrus planting land, whether it is a mud road or a rugged mountain road, it has far superiority over three or four wheels.
The tracks replace the tires, and the chassis is filled with iron material. The ultrasonic module is put into the protection box, which is placed at the front of the robot, and the camera head and robot arm are put on the protection box. The box is placed at the rear of the robot and connected to the robotic arm by a hose. Each part is fixed with screws.
4. System Design
4.1. System Hardware Design
1) The choice of main controller: STM32 is a 32-bit microprocessor integrated circuit developed by STMicroelectronics, with Arm’s Cortex architecture as the core. It is a good fit for our citrus picking robot. The 103STM32FZET6 is chosen as the main controller, which has 144 I/O ports and is powerful and low cost [8].
2) Selection of recognition module: OpenMV is a low-cost, open source, integrated machine vision, image processing module. It supports Micro Python language. OpenMV’s main functions on the citrus picking robot are: controlling the rotation of the head, acquiring the environment image, finding the target citrus, and returning data to the STM32 main controller [9].
3) Multi-core design of identification module and main controller: The camera requires more program code and arithmetic power than other sensors, so using only one STM32 to take up all the work will slow down the program efficiency and the time [10] of the whole workflow. It is necessary to use openMV’s own processor to share the work of STM32 main controller. OpenMV’s own processor is an STM32H7 chip, which is used to process the raw data of the camera and control the camera head rotation, the Tripod head is a two-degree of freedom acrylic Tripod head, which is made according to the size of OpenMV camera. The multi-core processing makes the camera module independent relative to the STM32 main controller, and the camera module only passes the simplified data to the STM32 main controller after its own processing, thus improving the efficiency of the STM32 main controller.
4) Power supply circuit design: In order to facilitate the production of robots and user maintenance, an additional voltage regulator module is selected to adjust the voltage for each module and the main controller. In order to protect the chip, a TPS63020 voltage regulator module is used in parallel with the power supply to supply the camera and STM32 main controller separately, so that the voltage of the camera and STM32 can be protected from other modules [11]. Use LM2596 step-down module to connect out the power supply to power the automatic obstacle avoidance module and the car box sensor. Two MT3608 DC-DC boost modules are used to power the other parts, one of them supplies 8 V to the robotic arm servo, Tripod head servo, and car box servo, and the other supplies 12 V or more to the motor driver module. The circuit is connected as Figure 5. When designing the circuit of the citrus picking robot pay attention to as modular as possible, not only to facilitate the production of citrus harvesting robot, but also to reduce the difficulty of maintenance, where bad to replace where, rather than over-integration, resulting in a small point of bad scrap a large piece, the user maintenance becomes simple, can be more permanent use, and further efforts to share the cost of labor.
5) Control circuit design: the data output of infrared sensor and load cell is connected to the PA4 and PB4 pins of STM32, STM32 receives data from two sensors to determine whether it is necessary to unload, and when unloading, it sends pulse signals through PA5 and PA6 pins to control two tiller rotations of the car box to unload [12]. The camera transmits data to the STM32 through the serial port, and the STM32 processes the data and then controls the four servos of the robot arm through the four pins from PB5 to PB8 to complete the picking action. The PA8 pin of the STM32 is connected to the Echo pin of the ultrasonic module, which is set high when the ultrasonic module starts working, and the PA7 is connected to the Trig pin, which is set low when the ultrasonic module receives the echo. The STM32 determines whether the PA7 pin is low or not,
and then controls the two motors through the DVR8833 chip. The Hall sensors on the motors are connected to pins PB1 to PB4 of the STM32, which feed the speed data to the STM32. The system control circuit is as follows Figure 6.
4.2. System Software Design
Use keil5 software to create a new project, configure parameters according to the STM32F103ZET6 main controller model we choose, and create at least one.c suffix file for each module. c suffix file for each module, so that the automatic obstacle avoidance, robot arm, motor, car box, four-module functions into the corresponding c suffix file, so that the main function code segment as small as possible. Set a section of code for automatic fault detection in the main function to facilitate user maintenance. The workflow of automatic detection, automatic obstacle avoidance and automatic citrus picking after power on is as follows Figure 7.
1) Automatic obstacle avoidance software design: STM32 main controller to the ultrasonic module transmitter set high, while the STM32 timer 3 start timing, receive the echo ultrasonic module receiver set high, timer 3 stop timing,
extract the time in the timer, according to the formula to calculate whether there are obstacles, if there are obstacles, avoid obstacles, otherwise continue to move forward.
2) Camera looking for citrus: judge the ripeness of this citrus according to its color, use the color recognition and tracking function of OpenMV to determine the two-dimensional coordinates of the citrus, the recognition lock setting of OpenMV is as follows Figure 8. After that, the distance of the citrus relative to the camera is obtained according to the pre-defined reference, and the 3D coordinates are simplified by the processor that comes with OpenMV, and then passed to the STM32 main controller through the serial port. The camera processor part of the code is shown in Figure 9.
3) Workflow of the robot arm: The STM32 main controller receives the 3D coordinates through the serial port and uses the STM32 built-in program algorithm to change the coordinates into a pulse signal to control the servo of the robot arm, so that the servo of the robot arm can receive [13]. The robot arm moves by receiving the pulse signal from STM32. After the robot arm reaches the designated position, the STM32 main controller then sends a pulse signal to control the control servo of the end gripper to cut off the citrus stalk.
4) Cart unloading design: the load cell and infrared sensor start to work from the robot power on, the combination of the two makes the detection more accurate, avoiding errors such as leaves covering the infrared sensor and mistakenly considered full or the average weight of citrus is too different from the preset average weight leading to misjudgment, both sensors are judged to be full before executing the next procedure [14]. When the citrus is not full to cover the infrared
![]()
Figure 8. The recognition lock setting of OpenMv.
![]()
Figure 9. Camera processor part of the code.
sensor and the weight does not reach the preset value of the load cell, the two sensors give the STM32 main controller to keep high, otherwise keep low, the STM32 main controller detects that both are low to stop the program of picking citrus and start to execute the program of unloading back to the unloading point, the unloading process is: at the unloading point the STM32 controls the box and the door of the two The unloading process is as follows: at the unloading point, STM32 controls the two servos of the car box and the car door, tilts the car box to open the car door, tilts to the limit angle and then tilts back while closing the car door. The unloading process of the car box is as follows Figure 10.
5. Simulation
The Proteus 8 Professional simulation software was chosen to perform the simulation. This software is an EDA tool software published by Lab Center Electronics, UK. It not only has the simulation functions of other EDA tool software, but also can simulate microcontrollers and peripheral devices, which is very suitable for the simulation of citrus picking robots, and its component library comes with all the relevant chips needed for citrus picking robots [15].
The STM32F103ZET6 was chosen as the core for the simulation, and the Hall sensor module was replaced by a CLOCK component, which can generate a pulse signal to be sent to the STM32 main controller and can replace the Hall sensor in the simulation. The IR sensor is replaced by an IR diode that comes with the Proteus 8 Professional software. The simulation design of this design is as follows Figure 11.
1) Simulation process: the STM32 main controller controls the robotic arm servo to make movements according to the incoming data from the camera, and the 3D coordinate data is built into the program in the simulation and changed during each simulation. Six different sets of 3D coordinates are set to represent the positions of six citrus, and the simulation is performed one by one [16]. Then three different routes are set to simulate the successful obstacle algorithm, and the ultrasonic module is changed several times during the simulation to replace the obstacle encountered. The successful action rate of the robot arm indicates the successful citrus picking rate. The rate of successful obstacle avoidance indicates whether the motor has a corresponding speed change after the ultrasonic module transmits data to the STM32 main controller.
2) Simulation results: After six rounds of simulation, the percentage of successful picking action of the robot arm reaches 91.6% and the rate of successful obstacle avoidance reaches 93.3%, which shows that the designed control circuit can fully meet the expected requirements, i.e., it can control the robot arm to perform correct picking action according to the incoming data from the camera and can avoid obstacles according to the data from the ultrasonic module. However, the stability of the robotic arm module and the automatic obstacle avoidance module needs to be strengthened. Overall, the automatic walking function, automatic obstacle avoidance function and automatic picking function are achieved. The simulation result statistics are as follows Table 1.
6. Conclusion
Image recognition and recognition data processing require greater arithmetic power compared to the mechanical structure control, and two controllers have greater advantages over one controller, and a multi-core design is necessary. The robot’s track design can better adapt to the uneven land, and has better load capacity and stability compared to ordinary wheels. To further reduce labor costs, the robot’s automaticity is also important. In addition to automatic citrus searching, automatic identification, automatic picking and automatic loading
![]()
Table 1. Simulation results statistics.
and unloading, more additional automatic functions can be considered for the robot, such as one-key self-testing, which is convenient for users to operate and maintain; Internet functions, and the development of a corresponding APP, which makes it easier for users to manage the robot on their cell phones. It is believed that with the continuous improvement of technology, all these functions are expected to be realized in the future, thus reducing the burden of farmers, improving productivity and further contributing to agricultural development.
Fundamental Project
Guangxi University Student Innovation and Entrepreneurship Training Program Project (S202110596203).