UNIVERSITY PARK, Pa. — Penn State agricultural engineers have developed, for the first time, a prototype “end-effector” capable of deftly removing unwanted apples from trees — the first step toward robotic, green-fruit thinning.

The development is important, according to Long He, assistant professor of agricultural and biological engineering, because manual thinning is a labor-intensive task, and the shrinking labor force in apple production makes manual thinning economically infeasible. His research group in the College of Agricultural Sciences conducted a new study that led to the end-effector.

The apple crop is a high-value agricultural commodity in the U.S., with an annual total production of nearly 10 billion pounds and valued at nearly $3 billion, according to He, who is a leader in agricultural robotics research, previously developing automated components for mushroom picking and apple tree pruning. Green-fruit thinning — the process of discarding excess fruitlets in early summer, mainly to increase the remaining fruit size and quality — is one of the most important aspects of apple production.

“Eventually — in the next decade, we hope — this end effector will be combined with a machine vision component and a locomotion system, creating a mechanism that eventually will be able to accomplish robotic green-fruit thinning in apple orchards,” He said. “I realize this sounds wildly futuristic, but I think this is where we are headed, sooner than later, and this end-effector was the needed first step.”

Prior to designing the end-effector, lead researcher Magni Hussain, a doctoral degree student in the Department of Agricultural and Biological Engineering, performed a series of fruit-removal-dynamics tests to determine the forces required for robotic thinning, using pulling or stem-cutting methods on three different apple cultivars. The work was done at Penn State’s Fruit Research and Extension Center in Biglerville, in south-central Pennsylvania.

After finding that fruit removal using the pulling method resulted in a high occurrence of spur-end stem detachment — which can leave a wound that affects neighboring fruits in a cluster — Hussain built a stem-cutting end-effector prototype and conducted extensive field tests. He tested two end-effector prototype configurations: one placing the end-effector on a handheld bar; the other integrating the end-effector with a robotic manipulator.

The success rates of green-fruit removal for all end-effector-prototype experiments were over 90%, the researchers recently reported in the Journal of the American Society of Agricultural and Biological Engineers. Linking the end-effector with the robotic manipulator demonstrated the potential of a robotic green-fruit system to remove fruit at different locations and orientations. Robotic green-fruit thinning can potentially be as selective as manual thinning, the researchers suggested.

Hussain and He are already looking to the next stage of the research and the work is complex. Hussain is now working on the machine-vision component of the overall robotic system, “trying to design it from the ground up, so to speak," he said. The challenge is significant, he conceded, because for such a vision system to work, there are several components to consider.

“First and foremost, it has to detect green fruit, and I've implemented a deep-learning algorithm into the system to enable it to not only detect the green fruit in a tree environment, but also generate their pixel masks,” Hussain said. “And then, on top of detecting the fruit, I'm also inserting an algorithm that will allow it to detect the orientation of the fruit. Because when you're trying to remove green fruit, knowing their orientations is important for properly orienting an end-effector for fruit removal.”

James Schupp, professor of pomology, and Paul Heinemann, professor of agricultural and biological engineering, also contributed to this research.

This research was supported in part by U.S. Department of Agriculture’s National Institute of Food and Agriculture.