Vision-based Agricultural Implement Awareness

Autonomous tractors have recently become a reality through the advent of robotics technologies that enable autonomous driving and control. This serves as a potential game changer for addressing labor and sustainability issues in agriculture towards feeding a growing global population. However, unlike autonomous cars, tractors are not used for the sole purpose of transportation, but rather for performing an agricultural task (e.g. planting, harvesting, baling, tillage). Tractors serve as a universal power platform to pull different tools (called implements) to perform these tasks. Each of these tasks can required completing different operation from the tractor, making autonomous operation challenging as the function of the tractor needs to change based on what is connected to it. In order for the tractor to behave in the way that it should for a task, if first needs to recognize what job it needs to perform, the position of the tool for the task, and the operating state of the tool - each of these problems relate to awareness of the implement by the tractor.

The requirements of this project is to:
(1) determine the vision-based sensor suite (e.g. Lidar, 2d camera, 3D camera) that can be mounted to a tractor for implement detection,
(2) determine an embedded computing platform appropriate for the implement awareness problem that can be powered by a 12V vehicle system and operate in mobile machine conditions
(3) develop software systems to:
(i) recognize the type of implement connected to a tractor
(ii) determine the position of the implement relative to the tractor in 3D space to an accuracy less than 5 cm
(iii) determine key operating characteristics of the implement (different based on the type of implement)
(4) demonstrate the operation of the system with an example agricultural implement (or similar representation)

It is expected this project will require the use of multiple sensor types (Lidar, cameras, etc.), as well as multiple vision processing approaches (deterministic computer vision algorithms, deep learning recognition. etc.).