Object detection and semantic segmentation are two of the most widely adopted deep learning algorithms in agricultural applications. One of the major sources of variability in image quality acquired outdoors for such tasks is changing lighting conditions that can alter the appearance of the objects or the contents of the entire image. While transfer learning and data augmentation reduce the need for large amount of data to train deep neural networks to some extent, the large variety of cultivars and the lack of shared datasets in agriculture makes wide-scale field deployments difficult. In this paper, we present an active lighting-based camera system that generates robust and uniform images in any lighting conditions. We provide extensive validation experiments to evaluate the consistency in the quality of the images. Metrics for assessing image uniformity like Structural Similarity (SSIM) index and the Peak Signal to Noise Ratio (PSNR) ranged from 83.78 to 93.79 and from 25.30 to 31.78 respectively, showing stability over a day with changing sunlight. The validation stage also showed that the generated images effectively reduce the amount of training samples for object detection using deep neural networks. The camera system was then deployed in real field experiments for counting buds in dormant vines and shoots in early season grape vines as well as for counting apples in orchards. The mean absolute errors obtained when compared to ground truth were 5%, 2.55% and 8.57%, respectively.

Reinforcement learning is known to be sample inefficient, preventing its application to many real-world problems, especially with high dimensional observations like images. Transferring knowledge from other auxiliary tasks is a powerful tool for improving the learning efficiency. However, the usage of auxiliary tasks has been limited so far due to the difficulty in selecting and combining different auxiliary tasks. In this work, we propose a principled online learning algorithm that dynamically combines different auxiliary tasks to speed up training for reinforcement learning. Our method is based on the idea that auxiliary tasks should provide gradient directions that, in the long term, help to decrease the loss of the main task. We show in various environments that our algorithm can effectively combine a variety of different auxiliary tasks and achieves significant speedup compared to previous heuristic approches of adapting auxiliary task weights.

Extensive genetic and phenotypic research is necessary for any effective plant breeding program. Such studies, however, require an immense amount of time and resources. In order to expedite the breeding process, we provide a novel method for rapid genotype prediction using in-situ images of plants. In this method, significant single nucleotide polymorphisms (SNPs) are first identified using a novel autoencoder framework with the goal of being more robust to false positive associations than standard genome wide association studies (GWAS). On-field images of various plant varieties are then used to train Convolutional Neural Networks (CNNs) to predict candidate alleles and validate phenotypic relationships. This image-based system allows for easy use on new plant varieties to gain real-time genetic information for better harvest prediction. The feasibility of our method for rapid genotype prediction was demonstrated on 345 Sorghum bicolor varieties with corresponding uncontrolled images 60 days after seed planting. Our autoencoder identified 4 significant SNPs that had an average allele classification accuracy of 70.58% on 68 previously unseen plant varieties.

The need for fast and precise measurements of plant attributes makes robotic solutions an ideal replacement for labor-intensive phenotyping processes. In this work we present a deep learning-based high throughput, online pipeline for in-situ sorghum stalk detection and grasping. We use a variation of Generative Adversarial Network (GAN) for stalk segmentation trained on a relatively small number of images followed by a grasp point generation pipeline. The presented pipeline is robust to field challenges such as occlusions, high stalk density and lighting variation, and was deployed on a custom-built ground robot. We tested our end-to-end system in a field of Sorghum bicolor in South Carolina, USA, achieving an average grasping accuracy of 74.13% and a stalk detection F1 score of 0.90. Grasp point detection for plant manipulation takes an average of 0.98 seconds, and pixel-wise stalk detection takes 0.2 seconds per image.