Extensive genetic and phenotypic research is necessary for any effective plant breeding program. Such studies, however, require an immense amount of time and resources. In order to expedite the breeding process, we provide a novel method for rapid genotype prediction using in-situ images of plants. In this method, significant single nucleotide polymorphisms (SNPs) are first identified using a novel autoencoder framework with the goal of being more robust to false positive associations than standard genome wide association studies (GWAS). On-field images of various plant varieties are then used to train Convolutional Neural Networks (CNNs) to predict candidate alleles and validate phenotypic relationships. This image-based system allows for easy use on new plant varieties to gain real-time genetic information for better harvest prediction. The feasibility of our method for rapid genotype prediction was demonstrated on 345 Sorghum bicolor varieties with corresponding uncontrolled images 60 days after seed planting. Our autoencoder identified 4 significant SNPs that had an average allele classification accuracy of 70.58% on 68 previously unseen plant varieties.

This paper describes a keypoint detection algorithm to accurately detect round fruits in high resolution imagery. The significant challenge associated with round fruits such as grapes and apples is that the surface is smooth and lacks definition and contrasting features, the contours of the fruit may be partially occluded, and the color of the fruit often blends with background foliage. We propose a fruit detection algorithm that utilizes the gradual variation of intensity and gradient orientation on the surface of the fruit. Candidate fruit locations, or “seed points” are tested for both monotonically decreasing intensity and gradient orientation profiles. Candidate fruit locations that pass the initial filter are classified using modified histogram of oriented gradients combined with a pairwise intensity comparison texture descriptor and random forest classifier. We analyse the performance of the fruit detection algorithm on image datasets of grapes and apples using human labeled images as ground truth. Our method to detect candidate fruit locations is scale invariant, robust to partial occlusions and more accurate than existing methods. We achieve overall F1 accuracy score of 0.82 for grapes and 0.80 for apples. We demonstrate our method is more accurate than existing methods.

We present a long-range visual signal detection system that is suitable for an unmanned aerial vehicle to find an optical signal released at a desired landing site for the purposes of cargo delivery or rescue situations where radio signals or other communication systems are not available or the wind conditions at the landing site need to be signaled. The challenge here is to have a signal and detection system that works from long range (> 1000m) amongst ground clutter during various seasonal conditions on passive imagery. We use a smoke-grenade as a ground signal, which has the advantageous properties of being easy to carry by ground crews because of its light weight and small size, but when released has a long visual signaling range. We employ a camera system on the UAV with a visual texture feature extraction approach in a machine learning framework to classify image patches as `signal' or `background'. We study conventional approaches and develop a visual feature descriptor that can better differentiate the appearance of the visual signal under varying conditions and, when used to train a random-forest classifier, outperforms commonly used feature descriptors. The system was rigorously and quantitatively evaluated on data collected from a camera mounted on a helicopter and flown towards a plume of signal smoke over a variety of seasons, ground conditions, weather conditions, and environments. Our system was capable of detecting the smoke cloud with both precision and recall rates greater than 0.95 from ranges between 1000m and 1500m. Further, we develop a method to estimate wind orientation and approximate wind strength by assessing the shape of the smoke signal. We present a preliminary evaluation of the wind estimation in conditions with different wind intensities and orientations relative to the approach direction.

Separating the direct and global components of radiance can aid shape recovery algorithms and can provide useful information about materials in a scene. Practical methods for finding the direct and global components use multiple images captured under varying illumination patterns and require the scene, light source and camera to remain stationary during the image acquisition process. In this paper, we develop a motion compensation method that relaxes this condition and allows direct-global separation to be performed on video sequences of dynamic scenes captured by moving projector-camera systems. Key to our method is being able to register frames in a video sequence to each other in the presence of time varying, high frequency active illumination patterns. We compare our motion compensated method to alternatives such as single shot separation and frame interleaving as well as ground truth. We present results on challenging video sequences that include various types of motions and deformations in scenes that contain complex materials like fabric, skin, leaves and wax.

Shipdeck landing is one of the most challenging tasks for a rotorcraft. Current autonomous rotorcraft use shipdeck mounted transponders to measure the relative pose of the vehicle to the landing pad. This tracking system is not only expensive but renders an unequipped ship unlandable. We address the challenge of tracking a shipdeck without additional infrastructure on the deck. We present two methods based on video and lidar that are able to track the shipdeck starting at a considerable distance from the ship. This redundant sensor design enables us to have two independent tracking systems. We show the results of the tracking algorithms in three different environments - field testing results on actual helicopter flights, in simulation with a moving shipdeck for lidar based tracking and in laboratory using an occluded, and, moving scaled model of a landing deck for camera based tracking. The complimentary modalities allow shipdeck tracking under varying conditions.

Here we present an approach to estimate the global pose of a vehicle in the face of two distinct problems; first, when using stereo visual odometry for relative motion estimation, a lack of features at close range causes a bias in the motion estimate. The other challenge is localizing in the global coordinate frame using very infrequent GPS measurements. Solving these problems we demonstrate a method to estimate and correct for the bias in visual odometry and a sensor fusion algorithm capable of exploiting sparse global measurements. Our graph-based state estimation framework is capable of inferring global orientation using a unified representation of local and global measurements and recovers from inaccurate initial estimates of the state, as intermittently available GPS information may delay the observability of the entire state. We also demonstrate a reduction of the complexity of the problem to achieve real-time throughput. In our experiments, we show in an outdoor dataset with distant features where our bias corrected visual odometry solution makes a fivefold improvement in the accuracy of the estimated translation compared to a standard approach. For a traverse of 2km we demonstrate the capabilities of our graph-based state estimation approach to successfully infer global orientation with as few as 6 GPS measurements and with two-fold improvement in mean position error using the corrected visual odometry.

Here we consider the problem of automatically segmenting images taken from a boat or low-flying aircraft. Such a capability is important for autonomous river following and mapping. The need for accurate segmentation in a wide variety of riverine environments challenges the state of the art vision-based methods that have been used in more structured environments such as roads and highways. Apart from the lack of structure, the principal difficulty is the large spatial and temporal variations in the appearance of water in the presence of nearby vegetation and with reflections from the sky. We propose a self-supervised method to segment images into `sky', `river' and `shore' (vegetation + structures) regions. Our approach uses assumptions about river scene structure to learn appearance models based on features like color, texture and image location which are used to segment the image. We validated our algorithm by testing on four datasets captured under varying conditions on different rivers. Our self-supervised algorithm had higher accuracy rates than a supervised alternative, often significantly more accurate, and does not need to be retrained to work under different conditions.

We consider the perceptual challenges inherent in the robotic manipulation of previously unseen socks, with the end goal of manipulation by a household robot for laundry. The task poses challenging problems in modeling the appearance, shape and configuration of these textile items that tend to exhibit high variability in texture, design, and style while being highly articulated objects.

Geolocation of a ground object or target of interest from live video is a common task required of small and micro unmanned aerial vehicles (SUAVs and MAVs) in surveillance and rescue applications. However, such vehicles commonly carry low-cost and light-weight sensors providing poor bandwidth and accuracy whose contribution to observations is nonlinear, resulting in poor geolocation performance by standard techniques. This paper proposes the application of an efficient over-parameterized state representation to the problem of geolocation that is able to handle large, time-varying, and non-Gaussian sensor error to produce better geolocation estimates than typical approaches and which provides computing and communication benefits in applications such as predictive control and distributed collaboration. We evaluate our filter on real flight data, demonstrating its ability to efficiently produce a solution with tight confidence bounds given highly uncertain data.

The harvest yield in vineyards can vary significantly from year to year and also spatially within plots due to variations in climate, soil conditions and pests. Fine grained knowledge of crop yields can allow viticulturists to better manage their vineyards. The current industry practice for yield prediction is destructive, expensive and spatially sparse - during the growing season sparse samples are taken and extrapolated to determine overall yield. We present an automated method that uses computer vision to detect and count grape berries. The method could potentially be deployed across large vineyards taking measurements at every vine in a non-destructive manner. Our berry detection uses both shape and visual texture and we can demonstrate detection of green berries against a green leaf background. Berry detections are counted and the eventual harvest yield is predicted. Results are presented for 224 vines (over 450 meters) of two different grape varieties and compared against the actual harvest yield as groundtruth. We calibrate our berry count to yield and find that we can predict yield of individual vineyard rows to within 9.8% of actual crop weight.

Reliable real-time localization is a key component of autonomous industrial vehicle systems. We consider the problem of using on-board vision to determine a vehicle's pose in a known, but non-static, environment. While feasible technologies exist for vehicle localization, many are not suited for industrial settings where the vehicle must operate dependably both indoors and outdoors and in a range of lighting conditions. We extend the capabilities of an existing vision-based localization system, in a continued effort to improve the robustness, reliability and utility of an automated industrial vehicle system. The vehicle pose is estimated by comparing an edge-filtered version of a video stream to an available 3D edge map of the site. We enhance the previous system by additionally filtering the camera input for straight lines using a Hough transform, observing that the 3D environment map contains only linear features. In addition, we present an automated approach for generating 3D edge maps from laser point clouds, removing the need for manual map surveying and also reducing the time for map generation down from days to minutes. We present extensive localization results in multiple lighting conditions comparing the system with and without the proposed enhancements.

This paper presents a vision-based method of vehicle localisation that has been developed and tested on a large forklift type robotic vehicle which operates in a mainly outdoor industrial setting. The localiser uses a sparse 3D-edge- map of the environment and a particle filter to estimate the pose of the vehicle. The vehicle operates in dynamic and non-uniform outdoor lighting conditions, an issue that is addressed by using knowledge of the scene to intelligently adjust the camera exposure and hence improve the quality of the information in the image. Results from the industrial vehicle are shown and compared to another laser-based localiser which acts as a ground truth. An improved likelihood metric, using per- edge calculation, is presented and has shown to be 40% more accurate in estimating rotation. Visual localization results from the vehicle driving an arbitrary 1.5 km path during a bright sunny period show an average position error of 0.44 m and rotation error of 0.62deg.

Conventional cameras have limited dynamic range, and as a result vision-based robots cannot effectively view an environment made up of both sunny outdoor areas and darker indoor areas. This paper presents an approach to extend the effective dynamic range of a camera, achieved by changing the exposure level of the camera in real-time to form a sequence of images which collectively cover a wide range of radiance. Individual control algorithms for each image have been developed to maximize the viewable area across the sequence. Spatial discrepancies between images, caused by the moving robot, are improved by a real-time image registration process. The sequence is then combined by merging color and contour information. By integrating these techniques it becomes possible to operate a vision-based robot in wide radiance range scenes