We present an algorithm for infinitesimal motion estimation and segmentation from multiple central panoramic views. We first show that the central panoramic optical flows corresponding to independent motions lie in orthogonal ten-dimensional subspaces of a higher-dimensional linear space. We then propose a factorization-based technique that estimates the number of independent motions, the segmentation of the image measurements and the motion of each object relative to the camera from a set of image points and their optical flows in multiple frames. Finally, we present the experimental results on motion estimation and segmentation for a real image sequence with two independently moving mobile robots, and evaluate the performance of our algorithm by comparing the vision estimates with GPS measurements gathered by the mobile robots.

We consider the problem of having a team of nonholonomic mobile robots follow a desired leader-follower formation using omnidirectional vision. By specifying the desired formation in the image plane, we translate the control problem into a separate visual servoing task for each follower. We use a rank constraint on the omnidirectional optical flows across multiple frames to estimate the position and velocities of the leaders in the image plane of each follower. We show that the direct feedback-linearization of the leader-follower dynamics suffers from degenerate configurations due to the nonholonomic constraints of the robots and the nonlinearity of the omnidirectional projection model. We therefore design a nonlinear tracking controller that avoids such degenerate configurations, while preserving the formation input-to-state stability. Our control law naturally incorporates collision avoidance by exploiting the geometry of omnidirectional cameras. We present simulations and experiments evaluating our omnidirectional vision-based formation control scheme.

We consider the problem of having a group of nonholonomic mobile robots equipped with omnidirectional cameras maintain a desired leader-follower formation. Our approach is to translate the formation control problem from the configuration space into a separate visual servoing task for each follower. We derive the questions of motion of the leader in the image plane of the follower and propose two control schemes for the follower. The first one is based on feedback linearization and is either string stable or leader-to-formation stable, depending on the sensing capabilities of the followers. The second one assumes a kinematic model for the evolution of the leader velocities and combines a Luenberger observer with a linear control law that is locally stable. We present simulation results evaluating our vision-based follow-the-leader control strategies.

We present a multiple view algorithm for vision based landing of an unmanned aerial vehicle. Our algorithm is based on our results in multiple view geometry which exploit the rank deficiency of the so called multiple view matrix. We show how the use of multiple views significantly improves motion and structure estimation. We compare our algorithm to our previous linear and non-linear two-view algorithms using an actual flight test. Our results show that the vision-based state estimates are accurate to within 7cm in each axis of translation and 4 degrees in each axis of rotation.

We consider the problem of having a team of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) pursue a second team of evaders while concurrently building a map in an unknown environment. We cast the problem in a probabilistic game theoretical framework, and consider two computationally feasible greedy pursuit policies: local-mar and global-max. To implement this scenario on real UAVs and UGVs, we propose a distributed hierarchical hybrid system architecture which emphasizes the autonomy of each agent, yet allows for coordinated team efforts. We describe the implementation of the architecture on a fleet of UAVs and UGVs, detailing components such as high-level pursuit policy computation, map building and interagent communication, and low-level navigation, sensing, and control. We present both simulation and experimental results of real pursuit-evasion games involving our fleet of UAVs and UGVs, and evaluate the pursuit policies relating expected capture times to the speed and intelligence of the evaders and the sensing capabilities of the pursuers.

We present the design and implementation of a real-time computer vision system for a rotor-craft unmanned aerial vehicle to land onto a known landing target. This vision system consists of customized software and off-the-shelf hardware which perform image processing, segmentation, feature point extraction, camera pan/tilt control, and motion estimation. We introduce the design of a landing target which significantly simplifies the computer vision tasks such as corner detection and correspondence matching. Customized algorithms are developed to allow for realtime computation at a frame rate of 30Hz. Such algorithms include certain linear and nonlinear optimization schemes for model-based camera pose estimation. We present results from an actual flight test which show the vision-based state estimates are accurate to within 5cm in each axis of translation and 5 degrees in each axis of rotation, making vision a viable sensor to be placed in the control loop of a hierarchical flight management system.

Presents the implementation of a hierarchical architecture for the coordination and control of a heterogeneous team of autonomous agents. We consider the problem of having a team of agents pursue a second team of evaders while building a map of the environment. The control architecture emphasizes the autonomy of each agent yet allows for coordinated efforts among them. We address the technical challenges and implementation issues of multi-agent operation. Finally we present experimental results of a pursuit-evasion game scenario between unmanned ground and aerial vehicles.