Traditionally, unmanned aerial vehicles (UAVs) rely on CMOS-based cameras to collect images about the world below. One of the most successful applications of UAVs is to generate orthomosaics or orthomaps, in which a series of images are integrated to develop a larger map. However, using CMOS-based cameras with global or rolling shutters means that orthomaps are vulnerable to challenging light conditions, motion blur, and high-speed motion of independently moving objects (IMOs) under the camera. Event cameras are less sensitive to these issues, as their pixels trigger asynchronously on brightness changes. This work introduces the first orthomosaic approach using event cameras. We focus on addressing high-dynamic range and low-light problems in orthomosaics. In contrast to existing methods relying only on CMOS cameras, our approach enables map generation even in challenging light conditions, including direct sunlight and after sunset. The source code for EvMAPPER, the high-altitude hardware, and the dataset collected in this paper are available open source11https://evmapper.fcladera.com.

Bowen Jiang, Yangxinyu Xie, Xiaomeng Wang, Yuan Yuan, Zhuoqun Hao, Xinyi Bai, Weijie J Su, Camillo Jose Taylor, Tanwi Mallick. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

This work introduces an enhanced approach to generating scene graphs by incorporating both a relationship hierarchy and commonsense knowledge. Specifically, we begin by proposing a hierarchical relation head that exploits an informative hierarchical structure. It jointly predicts the relation super-category between object pairs in an image, along with detailed relations under each super-category. Following this, we implement a robust commonsense validation pipeline that harnesses foundation models to critique the results from the scene graph prediction system, removing nonsensical predicates even with a small language-only model. Extensive experiments on Visual Genome and Open-Image V6 datasets demonstrate that the proposed modules can be seamlessly integrated as plug-and-play enhancements to existing scene graph generation algorithms. The results show significant improvements with an extensive set of reasonable predictions beyond dataset annotations. Codes are available at https://github.com/bowen-upenn/scene_graph_commonsense11This work has been accepted at the 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo Jose Taylor, Dan Roth. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Multi-robot collaboration in large-scale environments with limited-sized teams and without external infrastructure is challenging, since the software framework required to support complex tasks must be robust to unreliable and intermittent communication links. In this work, we present MOCHA (Multi-robot Opportunistic Communication for Heterogeneous Collaboration), a framework for resilient multi-robot collaboration that enables large-scale exploration in the absence of continuous communications. MOCHA is based on a gossip communication protocol that allows robots to interact opportunistically whenever communication links are available, propagating information on a peer-to-peer basis. We demonstrate the performance of MOCHA through real-world experiments with commercial-off-the-shelf (COTS) communication hardware. We further explore the system’s scalability in simulation, evaluating the performance of our approach as the number of robots increases and communication ranges vary. Finally, we demonstrate how MOCHA can be tightly integrated with the planning stack of autonomous robots. We show a communication-aware planning algorithm for a high-altitude aerial robot executing a collaborative task while maximizing the amount of information shared with ground robots.The source code for MOCHA and the high-altitude UAV planning system is available open source1.

Traditional approaches for active mapping focus on building geometric maps. For most real-world applications, however, actionable information is related to semantically meaningful objects in the environment. We propose an approach to the active metric-semantic mapping problem that enables multiple heterogeneous robots to collaboratively build a map of the environment. The robots actively explore to minimize the uncertainties in both semantic (object classification) and geometric (object modeling) information. We represent the environment using informative but sparse object models, each consisting of a basic shape and a semantic class label, and characterize uncertainties empirically using a large amount of real-world data. Given a prior map, we use this model to select actions for each robot to minimize uncertainties. The performance of our algorithm is demonstrated through multi-robot experiments in diverse real-world environments. The proposed framework is applicable to a wide range of real-world problems, such as precision agriculture, infrastructure inspection, and asset mapping in factories.

In this paper, we present a low-latency odometry system designed for spinning lidars. Many existing lidar odometry methods wait for an entire sweep from the lidar before processing the data. This introduces a large delay between the first laser firing and its pose estimate. To reduce this latency, we treat the spinning lidar as a streaming sensor and process packets as they arrive. This effectively distributes expensive operations across time, resulting in a very fast and lightweight system with a much higher throughput and lower latency. Our open source implementation is available at https://github.com/versatran01/llol.

One of the fundamental challenges in realizing the potential of legged robots is generating plans to traverse challenging terrains. Control actions must be carefully selected so the robot will not crash or slip. The high dimensionality of the joint space makes directly planning low-level actions from onboard perception difficult, and control stacks that do not consider the low-level mechanisms of the robot in planning are ill-suited to handle fine-grained obstacles. One method for dealing with this is selecting footstep locations based on terrain characteristics. However, incorporating robot dynamics into footstep planning requires significant computation, much more than in the quasi-static case. In this work, we present an LSTM-based planning framework that learns probability distributions over likely footstep locations using both terrain lookahead and the robot's dynamics, and leverages the LSTM's sequential nature to find footsteps in linear time. Our framework can also be used as a module to speed up sampling-based planners. We validate our approach on a simulated one-legged hopper over a variety of uneven terrains.

In this paper, we describe Direct Sparse Odometry Lite (DSOL), an improved version of Direct Sparse Odometry (DSO) [1]. We propose several algorithmic and implementation enhancements which speed up computation by a significant factor (on average 5x) even on resource-constrained platforms. The increase in speed allows us to process images at higher frame rates, which in turn provides better results on rapid motions. Our open-source implementation is available at https://github.com/versatran01/dso1.

The papers in this special section were presented at the Computer Vision and Pattern Recognition conference.

In this work we investigate the problem of uncertainty estimation for image-guided depth completion. We extend Deep Basis Fitting (DBF) [54] for depth completion within a Bayesian evidence framework to provide calibrated perpixel variance. The DBF approach frames the depth completion problem in terms of a network that produces a set of low-dimensional depth bases and a differentiable least squares fitting module that computes the basis weights using the sparse depths. By adopting a Bayesian treatment, our Bayesian Deep Basis Fitting (BDBF) approach is able to 1) predict high-quality uncertainty estimates and 2) enable depth completion with few or no sparse measurements. We conduct controlled experiments to compare BDBF against commonly used techniques for uncertainty estimation under various scenarios. Results show that our method produces better uncertainty estimates with accurate depth prediction.

We present an empirical investigation of a new mapping system based on a graph of panoramic depth images. Panoramic images efficiently capture range measurements taken by a spinning lidar sensor, recording fine detail on the order of a few centimeters within maps of expansive scope on the order of tens of millions of cubic meters. The flexibility of the system is demonstrated by running the same mapping software against data collected by hand-carrying a paired lidar and IMU around a laboratory space at walking pace, moving them outdoors through a campus environment at running pace, driving the sensors on a small wheeled vehicle on- and off-road, flying the sensors through a forest, carrying them on the back of a legged robot navigating an underground coal mine, and mounting them on the roof of a car driven on public roads. The full 3D maps are built online with a median update time of less than ten milliseconds on an embedded NVIDIA Jetson AGX Xavier system.

Autonomous flight through unknown environments in the presence of obstacles is a challenging problem for micro aerial vehicles (MAVs). A majority of the current state-of-art research assumes obstacles as opaque objects that can be easily sensed by optical sensors such as cameras or LiDARs. However in indoor environments with glass walls and windows, or scenarios with smoke and dust, robots (even birds) have a difficult time navigating through the unknown space.In this paper, we present the design of a new class of micro aerial vehicles that achieves autonomous navigation and are robust to collisions. In particular, we present the Tiercel MAV: a small, agile, light weight and collision-resilient robot powered by a cellphone grade CPU. Our design exploits contact to infer the presence of transparent or reflective obstacles like glass walls, integrating touch with visual perception for SLAM. The Tiercel is able to localize using visual-inertial odometry (VIO) running on board the robot with a single downward facing fisheye camera and an IMU. We show how our collision detector design and experimental set up enable us to characterize the impact of collisions on VIO. We further develop a planning strategy to enable the Tiercel to fly autonomously in an unknown space, sustaining collisions and creating a 2D map of the environment. Finally we demonstrate a swarm of three autonomous Tiercel robots safely navigating and colliding through an obstacle field to reach their objectives.

We address the localization of robots in a multi-MAV system where external infrastructure like GPS or motion capture systems may not be available. Our approach lends itself to implementation on platforms with several constraints on size, weight, and power (SWaP). Particularly, our framework fuses the onboard VIO with the anonymous, visual-based robot-to-robot detection to estimate all robot poses in one common frame, addressing three main challenges: 1) the initial configuration of the robot team is unknown, 2) the data association between each vision-based detection and robot targets is unknown, and 3) the vision-based detection yields false negatives, false positives, inaccurate, and provides noisy bearing, distance measurements of other robots. Our approach extends the Coupled Probabilistic Data Association Filter [1] to cope with nonlinear measurements. We demonstrate the superior performance of our approach over a simple VIO-based method in a simulation with the measurement models statistically modeled using the real experimental data. We also show how onboard sensing, estimation, and control can be used for formation flight.

In this work we propose long wave infrared (LWIR) imagery as a viable supporting modality for semantic segmentation using learning-based techniques. We first address the problem of RGB-thermal camera calibration by proposing a passive calibration target and procedure that is both portable and easy to use. Second, we present PST900, a dataset of 894 synchronized and calibrated RGB and Thermal image pairs with per pixel human annotations across four distinct classes from the DARPA Subterranean Challenge. Lastly, we propose a CNN architecture for fast semantic segmentation that combines both RGB and Thermal imagery in a way that leverages RGB imagery independently. We compare our method against the state-of-the-art and show that our method outperforms them in our dataset.

In this paper we consider the task of image-guided depth completion where our system must infer the depth at every pixel of an input image based on the image content and a sparse set of depth measurements. We propose a novel approach that builds upon the strengths of modern deep learning techniques and classical optimization algorithms and significantly improves performance. The proposed method replaces the final 1 × 1 convolutional layer employed in most depth completion networks with a least squares fitting module which computes weights by fitting the implicit depth bases to the given sparse depth measurements. In addition, we show how our proposed method can be naturally extended to a multi-scale formulation for improved self-supervised training. We demonstrate through extensive experiments on various datasets that our approach achieves consistent improvements over state-of-the-art baseline methods with small computational overhead.

In this work, we present an information based exploration strategy tailored for the generation of high resolution 3D maps. We employ RGBD panoramas because they have been shown to provide memory efficient high quality representations of space. Robots explore the environment by selecting locations with maximal Cauchy-Schwarz Quadratic Mutual Information (CSQMI) computed on an angle enhanced occupancy grid to collect these RGBD panoramas. By employing the angle enhanced occupancy grid, the resulting exploration strategy emphasizes perspective in addition to binary coverage. Furthermore, the goal selection strategy is improved by using image morphology to reduce the search space over which CSQMI is computed. We present experimental results demonstrating the improved performance in perception related tasks by capturing panoramas using this approach, near frontier exploration, and a control of logging images at regular intervals while teleoperating the robot through the workspace. Collect imagery was passed through an object detection library with our perspective aware approach yielding a greater number of successful detections compared to near frontier exploration.

In this paper we describe the Open Vision Computer (OVC) which was designed to support high speed, vision guided autonomous drone flight. In particular our aim was to develop a system that would be suitable for relatively small-scale flying platforms where size, weight, power consumption and computational performance were all important considerations. This manuscript describes the primary features of our OVC system and explains how they are used to support fully autonomous indoor and outdoor exploration and navigation operations on our Falcon 250 quadrotor platform.

We present an approach to depth estimation that fuses information from a stereo pair with sparse range measurements derived from a LIDAR sensor or a range camera. The goal of this work is to exploit the complementary strengths of the two sensor modalities, the accurate but sparse range measurements and the ambiguous but dense stereo information. These two sources are effectively and efficiently fused by combining ideas from anisotropic diffusion and semi-global matching.We evaluate our approach on the KITTI 2015 and Middlebury 2014 datasets, using randomly sampled ground truth range measurements as our sparse depth input. We achieve significant performance improvements with a small fraction of range measurements on both datasets. We also provide qualitative results from our platform using the PMDTec Monstar sensor. Our entire pipeline runs on an NVIDIA TX-2 platform at 5Hz on 1280×1024 stereo images with 128 disparity levels.

High speed navigation through unknown environments is a challenging problem in robotics. It requires fast computation and tight integration of all the subsystems on the robot such that the latency in the perception-action loop is as small as possible. Aerial robots add a limitation of payload capacity, which restricts the amount of computation that can be carried onboard. This requires efficient algorithms for each component in the navigation system. In this paper, we describe our quadrotor system which is able to smoothly navigate through mixed indoor and outdoor environments and is able to fly at speeds of more than 18 m/s. We provide an overview of our system and details about the specific component technologies that enable the high speed navigation capability of our platform. We demonstrate the robustness of our system through high speed autonomous flights and navigation through a variety of obstacle rich environments.

We present a novel fruit counting pipeline that combines deep segmentation, frame to frame tracking, and 3D localization to accurately count visible fruits across a sequence of images. Our pipeline works on image streams from a monocular camera, both in natural light, as well as with controlled illumination at night. We first train a Fully Convolutional Network (FCN) and segment video frame images into fruit and non-fruit pixels. We then track fruits across frames using the Hungarian Algorithm where the objective cost is determined from a Kalman Filter corrected Kanade-Lucas-Tomasi (KLT) Tracker. In order to correct the estimated count from tracking process, we combine tracking results with a Structure from Motion (SfM) algorithm to calculate relative 3D locations and size estimates to reject outliers and double counted fruit tracks. We evaluate our algorithm by comparing with ground-truth human-annotated visual counts. Our results demonstrate that our pipeline is able to accurately and reliably count fruits across image sequences, and the correction step can significantly improve the counting accuracy and robustness. Although discussed in the context of fruit counting, our work can extend to detection, tracking, and counting of a variety of other stationary features of interest such as leaf-spots, wilt, and blossom.

In the last decade, multi-rotor Micro Aerial Vehicles (MAVs) have attracted great attention from robotics researchers. Offering affordable agility and maneuverability, multi-rotor aircrafts have become the most commonly used platforms for robotics applications. Amongst the most promising applications are inspection of power-lines, cell-towers, large and constrained infrastructures and precision agriculture. While GPS offers an easy solution for outdoor autonomy, using on-board sensors is the only solution for autonomy in constrained indoor environments. In this paper, we present our results on autonomous inspection of completely dark, featureless, symmetric dam penstocks using cameras and range sensors. We use a hex-rotor platform equipped with an IMU, four cameras and two lidars. One of the cameras tracks features on the walls using the on-board illumination to estimate the position along the tunnel axis unobservable to range sensors while all of the cameras are used for panoramic image construction. The two lidars estimate the remaining degrees of freedom (DOF). Outputs of the two estimators are fused using an Unscented Kalman Filter (UKF). A moderately trained operator defines waypoints using the Remote Control (RC). We demonstrate our results from Carters Dam, GA and Glen Canyon Dam, AZ which include panoramic images for cracks and rusty spot detection and 6-DOF estimation results with ground truth comparisons. To our knowledge ours is the only study that can autonomously inspect environments with no geometric cues and poor to no external illumination using MAVs.

In this paper we propose a new approach to recovering the relative position and orientation of a pair of platforms moving on the plane by fusing the kinds of estimates provided by visual odometry systems with distance measurements obtained from ranging radio systems. By combining these two complementary sources of information we are able to provide estimates that could not be obtained using either source separately. We show that the localization problem can be phrased as an optimization problem where we are interested in maximizing a convex function subject to a set of convex constraints and we propose an efficient and effective solution scheme that leverages the structure of the problem. Experimental results are provided to demonstrate that the proposed method performs creditably on actual robotic platforms.

This paper presents a novel approach to solving optical flow problems using a discrete, tree-structured MRF derived from a hierarchical segmentation of the image. Our method can be used to find globally-optimal matching solutions even for problems involving very large motions. Experiments demonstrate that our approach is competitive on the MPI-Sintel dataset and that it can significantly outperform existing methods on problems involving large motions.

This work demonstrates the use of depth panoramas in the construction of detailed 3D models of extended environments. The paper describes an approach to the acquisition of such panoramic images using a robotic platform that collects sequences of depth images with a commodity depth sensor. These sequences are stitched into panoramic images that efficiently capture scene information while reducing noise in the captured imagery. Scene structure is extracted from the panoramas and used to register a collection of panoramas to obtain coverage of extended areas. The presented approach maintains fine geometry detail in 3D reconstructions, uses a modest amount of memory, and enjoys excellent scaling characteristics across large environments.

We present an approach for 2D sensor network localization when only bearing measurements are available and no global coordinate frame is known. Our work builds off of the linear constraint given in Kennedy et al. (2012) for sets of nodes that form triangles. We extend that constraint to general networks and present methods for locally optimizing the resulting cost function. We also show how these methods can be used for 3D network localization when the vertical axis is known. The algorithms are evaluated on both synthetic and real datasets, and we also show how they can be applied to the “structure from motion” problem in the field of computer vision.

We present a family of online algorithms for real-time factorization-based structure from motion, leveraging a relationship between the incremental singular value decomposition and recent work in online matrix completion. Our methods are orders of magnitude faster than previous state of the art, can handle missing data and a variable number of feature points, and are robust to noise and sparse outliers. Experiments show that they perform well in both online and batch settings. We also provide an implementation which is able to produce 3D models in real time using a laptop with a webcam.

In this paper we explore approaches to accelerating segmentation and edge detection algorithms based on the \emph{gPb} framework. The paper characterizes the performance of a simple but effective edge detection scheme which can be computed rapidly and offers performance that is competitive with the pB detector. The paper also describes an approach for computing a reduced order normalized cut that captures the essential features of the original problem but can be computed in less than half a second on a standard computing platform.

Mobile manipulators have brought a new level of flexibility to traditional automation tasks such as tabletop manipulation, but are not yet capable of the same speed and reliability as industrial automation. We present approaches to 3D perception and manipulator motion planning that enable a general purpose robotic platform to recognize and manipulate a variety of objects at a rate of one pick-and-place operation every 6.7 s, and work with a conveyor belt carrying objects at a speed of 33cm/s.

We present an approach for sensor network localization when provided with a set of angular constraints. This problem arises in camera networks when angles between nearby points can be measured but depth measurements are not readily available. We provide contributions for two different variations on this problem. First, when each node is aware of a global coordinate frame, we present a novel method for identifying the components of the problem that are rigidly constrained. Second, in the more difficult case where only relative angles are known, we propose a novel spectral solution that achieves a globally-optimal embedding under transitively-triangular constraints, which we show encompass a wide range of real-world conditions. We demonstrate the utility of our algorithm on both synthetic data and data from quadrotor robot formations.

This paper presents a novel variant of structured lighting which exploits the inherent blur in the projector system to overcome the discrepancy in resolution between typical Digital SLR cameras and typical projector systems. More specifically, the scheme estimates the coordinates of the projection of each illuminated scene point in the projector frame with subpixel precision and this additional level of accuracy helps to improve the quality of the resulting 3D reconstructions.

This article describes a novel approach to localizing networks of embedded cameras and sensors. In this scheme, the cameras and the sensors are equipped with controllable light sources (either visible or infrared), which are used for signaling. Each camera node can then determine automatically the bearing to all of the nodes that are visible from its vantage point. By fusing these measurements with the measurements obtained from onboard accelerometers, the camera nodes are able to determine the relative positions and orientations of other nodes in the network. The method uses angular measurements derived from images, rather than range measurements derived from time-of-flight or signal attenuation. The scheme can be implemented relatively easily with commonly available components, and it scales well since the localization calculations exploit the sparse structure of the system of measurements. Additionally, the method provides estimates of camera orientation which cannot be determined solely from range measurements. The localization technology could serve as a basic capability on which higher-level applications could be built. The method could also be used to automatically survey the locations of sensors of interest, to implement distributed surveillance systems, or to analyze the structure of a scene, based on images obtained from multiple registered vantage points. It also provides a mechanism for integrating the imagery obtained from the cameras with the measurements obtained from distributed sensors.

Multi-robot map building has advanced to the point where high quality occupancy grid data may be collected by multiple robots collaborating with only intermittent connectivity. However, the tasking of these agents to most efficiently build the map is a problem that has seen less attention. Unfamiliar, highly cluttered environments can con found exploration strategies that rely solely on occupancy grid frontier identification or even semantic classification methods keyed on geometric features. To reason about partial maps of novel, highly cluttered locations, hypotheses about significant structure in the disposition of free space may be used to guide exploration task assignment. A parsing of map data into places with semantic significance to the exploration task provides a foundation from which one may infer an efficient exploration strategy.

This paper presents a scheme which takes as input a 3D point cloud and an associated color image and parses the scene into a collection of salient planar surfaces. The scheme makes use of a fast color segmentation scheme to divide the color image into coherent regions and the groupings suggested by this procedure are used to inform and accelerate a RANSAC based interpretation process. Results on real data sets are presented.

The decomposition of robotics software into a collection of loosely coupled processes has become a core design principle of virtually every large robotics software engineering effort over the past decade. Recently, the ROS software platform from Willow Garage has gained significant traction due to its adoption of sound design principles and significant software library contributions from Willow Garage itself. This paper describes a binding from the Haskell programming language to basic ROS interfaces. The novelty of these bindings is that they allow for, and encourage, a higher level of abstraction in writing programs for robots that treat streams of values as first-class citizens. This approach makes the fusing, transforming, and filtering of streams fully generic and compositional while maintaining full compatibility with the existing ROS ecosystem.

The management of finite resources is central to many robot behaviors. Some robotic systems must maintain invariants regarding the disposition of feet for balancing, others have grippers for manipulating their environments, while yet others must respect strict rules governing the usage of objects in the environment. Yet the specifics of such resource management responsibilities are almost universally locked behind opaque controllers whose lack of type information greatly impedes rigorous static analysis. We present an application of dependent type theory and linear logic for the static analysis of robot behavior programs that manage both robot and environment state, with a worked assembly task example. This approach offers static, formal guarantees with respect to safety requirements attached to primitive actions, as well as introspection of expected state at each step of a scripted sequence of actions allowing for the automatic generation of dynamic, sensor-based, runtime verification of successful execution.

This paper describes a feature based approach to segmenting images into coherent regions. The method draws inspiration from earlier work on randomized projection schemes for approximate nearest neighbor computation. The method proceeds by first computing a descriptor vector for each of the pixels in the image. These vectors are then randomly hashed to yield binary vectors. Salient clusters in the hash space are automatically identified by considering the populations associated with various hash codes. Since the method avoids the explicit vector distance computations associated with other methods, it is very amenable to fast implementation. Experimental results are presented on standard data sets. 1 Introduction and Related Work Segmentation, the problem of breaking an image into coherent regions is, of course, a fundamental problem in Computer Vision. This paper proposes a new approach to the segmentation problem that leverages ideas developed in the Theoretical Computer Science literature to derive a new feature space based clustering algorithm that is amenable to real time implementation.

Collision detection and estimation from a monocular visual sensor is an important enabling technology for safe navigation of small or micro air vehicles in near earth flight. In this paper, we introduce a new approach called expansion segmentation, which simultaneously detects “collision danger regions” of significant positive divergence in inertial aided video, and estimates maximum likelihood time to collision (TTC) in a correspondenceless framework within the danger regions. This approach was motivated from a literature review which showed that existing approaches make strong assumptions about scene structure or camera motion, or pose collision detection without determining obstacle boundaries, both of which limit the operational envelope of a deployable system. Expansion segmentation is based on a new formulation of 6-DOF inertial aided TTC estimation, and a new derivation of a first order TTC uncertainty model due to subpixel quantization error and epipolar geometry uncertainty. Proof of concept results are shown in a custom designed urban flight simulator and on operational flight data from a small air vehicle.

Graph cuts have become an increasingly important tool for solving a number of energy minimization problems in computer vision and other fields. In this paper, the graph cut problem is reformulated as an unconstrained l1 norm minimization that can be solved effectively using interior point methods. This reformulation exposes connections between the graph cuts and other related continuous optimization problems. Eventually the problem is reduced to solving a sequence of sparse linear systems involving the Laplacian of the underlying graph. The proposed procedure exploits the structure of these linear systems in a manner that is easily amenable to parallel implementations. Experimental results obtained by applying the procedure to graphs derived from image processing problems are provided.

A functional approach to programming robot swarms brings with it well-defined properties that allow for automated concurrency and distributed execution. Further, the particular expressiveness of a pure functional language with first-class closures captures so cleanly certain biologically-inspired behaviors that program specification often becomes compact enough to allow a programmer to visually inspect the program code for the entire swarm at once. This benefit comes in contrast to more piece-meal construction methods used to build-up robot software from discrete components. While such programming models capture the engineered structure of a robotic system, the dynamic, decentralized qualities sought after in robot swarms are well-matched by the idioms of functional concurrent programming.

This paper describes an approach to cooperative localization which finds its roots in robust estimation, employing an unknown-but-bounded error model for sensor measurements. In this framework, range and bearing measurements obtained by the robots are viewed as constraints which implicitly define a set of feasible solutions in the joint configuration space of the robot team. The scheme produces bounded uncertainty estimates for the relative configuration of the team by using convex optimization techniques to approximate the projection of this feasible set onto various subspaces of the configuration space. The scheme can also be used to localize distributed sensor nodes. An important advantage of the proposed approach is that it is able to produce bounded uncertainty estimates for the relative configuration of the robots even in the case where the relative orientations of the robots are completely unknown. This is an important practical advance since errors in relative orientation are often a major contributor to positioning uncertainty in multi-robot localization schemes.

This video shows an uncut sequence of a 15 module robot in a bi-pedal configuration subject to a variety of disturbances yet staying on task. As the robot walks, a large disturbance is introduced. A human kicks the robot in its midsection causing the robot to fall into three separate clusters of five modules. Each of these pieces diagnose their state sensing gravity and their connectedness. They each perform a self-righting maneuver, then begin to search for each other. This is a first demonstration towards the development of solving larger self-reassembly problems that include greater levels of randomization and entropy (more pieces more widely distributed.) The 2006 Robotics Science and Systems workshop on self-reconfigurable robots developed as one of the grand challenges for this area, the ability to survive and self-repair after an explosion. This work will push the technologies required for integrated sensing, localization, distributed control, under highly unstructured conditions.

This paper introduces a new challenge problem: designing robotic systems to recover after disassembly from high-energy events and a first implemented solution of a simplified problem. It uses vision-based localization for self- reassembly. The control architecture for the various states of the robot, from fully-assembled to the modes for sequential docking, are explained and inter-module communication details for the robotic system are described.

Inter-component communication has received considerable attention by robotic software architects as various frameworks and toolkits have matured. While the resulting software platforms have proven useful for abstracting hardware interfaces and the complex networking issues that are often present in robot deployment scenarios, they typically present users with arduous paths for integrating new hardware and software, while making no allowance for humans as peers on the network. We explore how mature standards and protocols born of the Internet community can be leveraged to empower software designed for robots and sensor networks. Here we present software approaches that facilitate the integration of new hardware devices, software platforms, and humans in ways that are natural and intuitive for each

Communication is essential for coordination in most cooperative control and sensing paradigms. In this paper, we present an experimental study of strategies for maintaining end-to-end communication links for tasks such as surveillance and search and rescue where team connectivity is essential for providing situational awareness to a base station. We consider the differences between monitoring point-to-point signal strength versus data throughput and present experimental results with our multi-robot testbed in outdoor environments

In this work, we present a framework for the development of a planar mobile robot controller based on image plane feedback. We show that the design of such a motion controller can be accomplished in the image plane by making use of a subset of the parameters that relate the image plane to the ground plane, while still leveraging the simplifications offered by modeling the system as a differentially flat system. Our method relies on a waypoint-based trajectory generator, with all the waypoints specified in the image, as seen by an overhead observer. We present some results from simulation as well as from experiments that validate the ideas presented in this work and discuss some ideas for future work.

This paper addresses the problem of coordinating aerial and ground vehicles in tasks that involve exploration, identification of targets and maintaining a connected communication network. We focus on the problem of localizing vehicles in urban environments where GPS signals are often unreliable or unavailable. We first describe our multi-robot testbed and the control software used to coordinate ground and aerial vehicles. We present the results of experiments in air-ground localization analyzing three complementary approaches to determining the positions of vehicles on the ground. We show that the coordination of aerial vehicles with ground vehicles is necessary to get accurate estimates of the state of the system.

We describe our implementation of a distributed sensor database that was designed to support the activities of teams of mobile robots as they explore an environment. Importantly, this approach effectively separates the process of acquiring sensor data from that of exploiting it. This allows us to develop applications where robots and human users can automatically discover and utilize sensor measurements acquired by other robots in the team. We also explain our approach to implementing distributed queries, an important capability that allows us to perform queries in a way that makes best use of the limited available communication bandwidth. Finally, we briefly describe how we have used this system to support situational awareness tasks.

Communication is essential for coordination in most cooperative control and sensing paradigms. In this paper, we investigate the construction of a map of radio signal strength that can be used to plan multirobot tasks and also serve as useful perceptual information. We show how nominal models of an urban environment, such as those obtained by aerial surveillance, can be used to generate strategies for exploration and present preliminary experimental results with our multi-robot testbed.

As the price of cameras and computing elements continue to fall it becomes increasingly feasible to consider the deployment of smart camera networks. Such networks would be composed of small, networked computers equipped with inexpensive image sensors. Consider, for example, the proliferation of camera equipped cell phones. Such camera networks could be used to support a wide variety of applications including environmental modeling, 3D model construction and surveillance. A number of research efforts at a variety of institutions are currently directed towards realizing aspects of this vision. One critical problem that must be addressed in such systems is the issue of localization. That is, in order to take full advantage of the images gathered from multiple vantage points it is helpful to know where the cameras are located with respect to each other. In our system each of the smart cameras is equipped with a co-located controllable light source which it can use to signal other smart cameras in the vicinity. By analyzing the images that it acquires over time, each smart camera is able to locate and identify other smart cameras in the scene. This arrangement makes it possible to directly determine the epipolar geometry of the camera system from image measurements and, hence, recover the relative positions and orientations of the smart camera nodes. We will demonstrate a small scale version of an auto configuring camera network consisting of 3 to 5 smart cameras, we will show how they can accurately localize each other in real time and how they adapt to changes in the configuration.

We describe an approach to recovering surface models of complex scenes from the quasisparse data returned by a feature based stereo system. The method can be used to merge stereo results obtained from different viewpoints into a single coherent surface mesh. The technique proceeds by exploiting the freespace theorem which provides a principled mechanism for reasoning about the structure of the scene based on quasisparse correspondences in multiple image. Effective methods for overcoming the difficulties posed by missing features and outliers are discussed. Results obtained by applying this approach to actual images are presented.

In this paper, a vision-based stabilization and output tracking control method for a four-rotor helicopter has been proposed. A novel 2 camera method has been described for estimating the full 6 DOF pose of the helicopter. This two camera system is consisting of a pan-tilt ground camera and an onboard camera. The pose estimation algorithm is compared in simulation to other methods (such as four point method, and a stereo method) and is shown to be less sensitive to feature detection errors on the image plane. The proposed pose estimation algorithm and non-linear control techniques have been implemented on a remote controlled quadrotor helicopter.

This paper presents ROCI, a framework for developing applications for multi-robot teams. In ROCI, each robot is considered a node, which contains several modules and may export different types of services and capabilities to other nodes. Each node runs a kernel that mediates the interactions of the robots in a team. This kernel keeps an updated database of all nodes and the functionalities that they export. Multi-robot applications can be built dynamically by connecting modules that may be running on different nodes over the network. As an example, we present an obstacle avoidance task implemented using our framework and also discuss the use of ROCI in a multi-robot scenario.

In this paper, we investigate data fusion techniques for target tracking using distributed sensors. Specifically, we are interested in how pairs of bearing or range sensors can be best assigned to targets in order to minimize the expected error in the estimates. We refer to this as the focus of attention (FOA) problem. In its general form, FOA is NP-hard and not well approximable. However, for specific geometries we obtain significant approximation results: a 2-approximation algorithm for stereo cameras on a line, a PTAS for when the cameras are equidistant, and a 1.42 approximation for equally spaced range sensors on a circle. By reposing as a maximization problem - where the goal is to maximize the number of tracks with bounded error - we are able to leverage results from maximum set-packing to render the problem approximable. We demonstrate the results in simulation for a target tracking task, and for localizing a team of mobile agents in a sensor network. These results provide insights into sensor/target assignment strategies, as well as sensor placement in a distributed network.

We build on previous work, on the development of a computer controlled wheelchair equipped with a suite of sensors and a novel interface for human-robot interaction. In this paper, we present experimental results and usability studies for the wheelchair. The architecture for human-robot interaction is hierarchical, with the lowest level corresponding to trajectory control, the intermediate level being behavioral and the highest level involving the composition of behaviors and navigation. Our experimental results illustrate the benefits of a shared-control paradigm where the human operator selects the appropriate behavior(s) or goals while the software is responsible for executing behaviors and generating safe trajectories. Experiments with human users highlight advantages of augmentation in wheelchairs.

In this paper the problem of controlling the motion of a nonholonomic vehicle along a desired trajectory using observations from an overhead camera is considered. The control problem is formulated in the image plane. We show that the system in the image plane is differentially flat and use this property to generate effective control strategies using only visual feedback. Simulation results illustrate the methodology and show robustness to errors in the camera calibration parameters.

We offer a new approach to the multi-robot localization problem. Using an unknown-but-bounded model for sensor error, we are able to define convex polytopes in the configuration space of the robot team that represent the set of configurations consistent with all sensor measurements. Estimates for the uncertainty in various parameters of the team's configuration such as the absolute position of a single robot, or the relative positions of two or more nodes can be obtained by projecting this polytope onto appropriately chosen subspaces of the configuration space. We propose a novel approach to approximating these projections using linear programming techniques. The approach can handle both bearing and range measurements with a computational complexity scaling polynomially in the number of roots. Finally, the workload is readily distributed - requiring only the communication of sensor measurements between robots. We provide simulation results for this approach implemented on a multi-robot team.

This paper presents a sensor based algorithm for guiding a nonholonomic platform, such as a wheelchair, through a doorway. The controller uses information from a camera system and a laser range finder to perform image-based navigation. Simulations of the resultant switching controller are presented along with experimental results. A simple obstacle avoidance algorithm is also implemented on the experimental platform. Finally, we have considered the input of limited field-of-view constraints on this controller. All of these components together lead to a modal, image-based approach that will safely and robustly navigate a nonholonomic robot with sensor constraints through a doorway.

Addresses the problem of human robot interaction with application to the design of assistive devices. We describe the design and development of a prototype of a smart wheelchair that can be commanded by a rider. Specifically, we focus on (a) the vision-based human interaction interface; (b) the suite of sensors on the chair, and (c) the software architecture and the control algorithms used to control the chair.

This paper presents an approach to the problem of controlling the configuration of a team of mobile agents equipped with cameras so as to optimize the quality of the estimates derived from their measurements. The issue of optimizing the robots' configuration is particularly important in the context of teams equipped with vision sensors since most estimation schemes of interest will involve some form of triangulation. We provide a theoretical framework for tackling the sensor planning problem and a practical computational strategy, inspired by work on particle filtering, for implementing the approach. We extend our previous work by showing how modeled system dynamics and configuration space obstacles can be handled. These ideas have been demonstrated both in simulation and on actual robotic platforms. The results indicate that the framework is able to solve fairly difficult sensor planning problems online without requiring excessive amounts of computational resources.

We describe a framework for cooperative control of a group of nonholonomic mobile robots that allows us to build complex systems from simple controllers and estimators. The resultant modular approach is attractive because of the potential for reusability. Our approach to composition also guarantees stability and convergence in a wide range of tasks. There are two key features in our approach: 1) a paradigm for switching between simple decentralized controllers that allows for changes in formation; 2) the use of information from a single type of sensor, an omnidirectional camera, for all our controllers. We describe estimators that abstract the sensory information at different levels, enabling both decentralized and centralized cooperative control. Our results include numerical simulations and experiments using a testbed consisting of three nonholonomic robots.

This paper presents a simple approach to capturing the appearance and structure of immersive scenes based on the imagery acquired with an omnidirectional video camera. The scheme proceeds by combining techniques from structure-from-motion with ideas from image-based rendering. An interactive photogrammetric modeling scheme is used to recover the locations of a set of salient features in the scene (points and lines) from image measurements in a small set of keyframe images. The estimates obtained from this process are then used as a basis for estimating the position and orientation of the camera at every frame in the video clip. By augmenting the video sequence with pose information, we provide the end-user with the ability to index the video sequence spatially as opposed to temporally. This allows the user to explore the immersive scene by interactively selecting the desired viewpoint and viewing direction.

This paper describes an approach to estimating the trajectory of a moving camera based on the measurements acquired with an inertial sensor and estimates obtained by applying a structure from motion algorithm to a small set of keyframes in the video sequence. The problem is formulated as an offline trajectory fitting task rather than an online integration problem. This approach avoids many of the issues usually associated with inertial estimation schemes. One of the main advantages of the proposed technique is that it can be applied in situations where approaches based on feature tracking would have significant difficulties. Results obtained by applying the procedure to extended sequences acquired with both conventional and omnidirectional cameras are presented.

The paper presents an approach to the problem of controlling the configuration of a team of mobile agents equipped with cameras so as to optimize the quality of the estimates derived from their measurements. The issue of optimizing the robots' configuration is particularly important in the context of teams equipped with vision sensors since most estimation schemes of interest will involve some form of triangulation. We provide a theoretical framework for tackling the sensor planning problem and a practical computational strategy, inspired by work on particle filtering, for implementing the approach. The ideas have been demonstrated both in simulation and on actual robotic platforms. The results indicate that the framework is able to solve fairly difficult sensor planning problems online without requiring excessive amounts of computational resources.

This paper considers the problem of vision-based control of a nonholonomic mobile robot. We describe the design and implementation of real-time estimation and control algorithms on a car-like robot platform using a single omni-directional camera as a sensor without explicit use of odometry. We provide experimental results for each of these vision-based control objects. The algorithms are packaged as control modes and can be combined hierarchically to perform higher level tasks involving multiple robots.

We describe a framework for coordinating multiple robots in cooperative manipulation tasks in which vision is used for establishing relative position and orientation and maintaining formation. The two key contributions are a cooperative scheme for localizing the robots based on visual imagery that is more robust than decentralized localization, and a set of control algorithms that allow the robots to maintain a prescribed formation (shape and size). The ability to maintain a prescribed formation allows the robots to "trap" objects in their midst, and to "flow" the formation to a desired position. We derive the cooperative localization and control algorithms and present experimental results that illustrate the implementation and the performance of these algorithms.

This paper deals with the problem of recovering the dimensions of an object and its pose from a single image acquired with a camera of unknown focal length. It is assumed that the object in question can be modeled as a polyhedron where the coordinates of the vertices can be expressed as a linear function of a dimension vector. The reconstruction program takes as input, a set of correspondences between features in the model and features in the image. From this information, the program determines an appropriate projection model for the camera, the dimensions of the object, its pose relative to the camera and, in the case of perspective projection, the focal length of the camera. This paper describes how the reconstruction problem can be framed as an optimization over a compact set with low dimension (no more than four). This optimization problem can be solved efficiently by coupling standard nonlinear optimization techniques with a multistart method. The result is an efficient, reliable solution system that does not require initial estimates for any of the parameters being estimated.

This paper investigates the problem of recovering information about the configuration of an articulated object, such as a human figure, from point correspondences in a single-image. Unlike previous approaches, the proposed reconstruction method does not assume that the imagery was acquired with a calibrated camera. An analysis is presented which demonstrates that there are a family of solutions to this reconstruction problem parameterized by a single variable. A simple and effective algorithm is proposed for recovering the entire set of solutions by considering the foreshortening of the segments of the model in the image. Results obtained by applying this algorithm to real images are presented.

The problem of controlling the spatial position and orientation of a robotic platform based on the image data obtained from a video camera mounted on that platform is considered. More specifically, we propose control laws that will cause the robot to achieve and maintain a fixed position and orientation with respect to a set of feature points in the scene. We demonstrate analytically that the proposed control scheme is globally convergent even in the presence of large calibration errors in both the intrinsic parameters of the camera and in the extrinsic parameters which relate the frame of reference of the camera to the body frame of the robot platform which is being controlled. Furthermore, no a priori knowledge about the structure of the scene is assumed.

We present a software architecture used to demonstrate the ability of a tank-like robot to climb multiple flights of stairs. The algorithm we use incorporates intelligent sensor fusion and a hierarchical modular structure. This structure simplifies the overall design, enables reuse, and promotes extensibility. The modular nature of the design also allows for a variety of user interface modes with different levels of user autonomy. The robustness of our algorithm allows us to ascend and descend stairs autonomously in many environments with different textures and configurations, all with minimal adjustment to the algorithm. Our approach develops an extensible sensor and task model that reduces the effect of sensor inaccuracies and geometric constraints, uses minimal information about the environment, and utilizes characteristic features in the environment for navigation.

This paper investigates the problem of recovering information about the configuration of an articulated object, such as a human figure, from point correspondences in a single image. Unlike previous approaches, the proposed reconstruction method does not assume that the imagery was acquired with a calibrated camera. An analysis is presented which demonstrates that there is a family of solutions to this reconstruction problem parameterized by a single variable. A simple and effective algorithm is proposed for recovering the entire set of solutions by considering the foreshortening of the segments of the model in the image. Results obtained by applying this algorithm to real images are presented.

This paper deals with the problem of recovering the dimensions of an object and its pose from a single image acquired with a camera of unknown focal length. It is assumed that the object in question can be modeled as a polyhedron where the coordinates of the vertices can be expressed as a linear function of a dimension vector, /spl lambda/. The reconstruction program takes as input a set of correspondences between features in the model and features in the image. From this information the program determines an appropriate projection model for the camera (scaled orthographic or perspective), the dimensions of the object, its pose relative to the camera and, in the case of perspective projection, the focal length of the camera. We demonstrate that this reconstruction task can be framed as an unconstrained optimization problem involving a small number of variables, no more than four, regardless of the number of parameters in the dimension vector.

In this paper the problem of controlling the spatial position and orientation of a robotic platform based on the image data obtained from a video camera mounted on that platform is considered. More specifically, we propose control laws that generate translational and angular velocities that will cause the robot to achieve and maintain a fixed position and orientation with respect to a set of feature points in the scene. The proposed control schemes make use of well established techniques for computing estimates for the relative orientation of two camera positions from a set of feature correspondences. An important advantage of these control schemes is that it is possible to demonstrate analytically that they are globally convergent even in the presence of large calibration errors in both the intrinsic parameters of the camera and in the extrinsic parameters which relate the frame of reference of the camera to the body frame of the robot platform which is being controlled. Furthermore no a priori knowledge about the structure of the scene is assumed.

This paper will present the results of a comparative study of a set of vision-based control strategies that have been applied to the problem of steering an autonomous vehicle along a highway. The aim of this work has been to further our understanding of the characteristics of various control laws that could be applied to this problem with a view to making informed design decisions. The control strategies that we explored include a lead lag control law, a full-state linear controller and input-output linearizing control law. Each of these control strategies was implemented and tested on our experimental vehicle, a Honda Accord LX, both with and without a curvature feedforward component.

This paper considers the problem of systematically exploring an unfamiliar environment in search of one or more recognizable targets. The proposed exploration algorithm is based on a novel representation of environments containing visual landmarks, called the boundary place graph. This representation records the set of recognizable objects (landmarks) that are visible from the boundary of each configuration space obstacle. The exploration algorithm constructs the boundary place graph incrementally from sensor data. Once the robot has completely explored an environment, it can use the constructed representation to carry out further navigation tasks. We provide a necessary and sufficient condition under which the algorithm is guaranteed to discover all landmarks. This algorithm has been implemented on our mobile robot platform RJ, and results from these experiments are presented.

We present a new approach for modeling and rendering existing architectural scenes from a sparse set of still photographs.Our modeling approach, which combines both geometry-based and imagebased techniques, has two components.The first component is a photogrammetric modeling method which facilitates the recovery of the basic geometry of the photographed scene.Our photogrammetric modeling approach is effective, convenient, and robust because it exploits the constraints that are characteristic of architectural scenes.The second component is a model-based stereo algorithm, which recovers how the real scene deviates from the basic model.By making use of the model, our stereo technique robustly recovers accurate depth from widely-spaced image pairs.Consequently, our approach can model large architectural environments with far fewer photographs than current image-based modeling approaches.For producing renderings, we present view-dependent texture mapping, a method of compositing multiple views of a scene that better simulates geometric detail on basic models.Our approach can be used to recover models for use in either geometry-based or image-based rendering systems.We present results that demonstrate our approach's ability to create realistic renderings of architectural scenes from viewpoints far from the original photographs.

This paper presents a new method for recovering the three dimensional structure of a scene composed of straight line segments using the image data obtained from a moving camera. The recovery algorithm is formulated in terms of an objective function which measures the total squared distance in the image plane between the observed edge segments and the projections (perspective) of the reconstructed lines. This objective function is minimized with respect to the line parameters and the camera positions to obtain an estimate for the structure of the scene. The effectiveness of this approach is demonstrated quantitatively through extensive simulations and qualitatively with actual image sequences. The implementation is being made publicly available.>

An approach for recovering the structure of a rigid scene composed of straight line segments and the pose of moving camera from multiple images under perspective projection is presented. Recovery is formulated as minimizing the total squared image distance between measured segments and the projection of the reconstructed infinite lines with respect to structural and motion parameters. An efficient algorithm for minimizing this nonlinear objective function is presented. The approach can be directly used for accurately locating correspondences in multieyed stereo, as well as for pose estimation from landmarks. Its effectiveness is demonstrated quantitatively through simulation and qualitatively with a real image sequence.>