Instance and semantic segmentation form the backbone of robotic perception and are crucial to many tasks. While most research in the area focuses on improving segmentation quality metrics, there are plenty of applications where approximate methods are adequate as long as they are fast, especially in applications with large amounts of data like precision agriculture. In order to apply the recent successes of machine learning and computer vision on a large scale using robotics, efficient and general algorithms must be designed to intelligently split point clouds into small, yet actionable, portions that can then be processed by more complex algorithms. In this paper, we capitalize on a similarity between the current state-of-the-art for roughly segmenting corn plants and a commonly used density-based clustering algorithm, Quickshift. Exploiting this similarity we propose a novel algorithm, Ground-Density Quickshift++, with the goal of producing a general and scalable field segmentation algorithm that segments individual plants and their stems. This algorithm produces quantitatively better results than the current state-of-the-art on both plant separation and stem segmentation while being less sensitive to input parameters and maintaining the same algorithmic time complexity. When incorporated into field-scale phenotyping systems, the proposed algorithm should work as a drop-in replacement that can greatly improve the accuracy of results while ensuring that performance and scalability remain undiminished.

The determination of a crop's growth-stage is critical information for precision agriculture. Estimates of the growth-stage are used to guide irrigation and the application of agrochemicals. Of particular importance is the use of fertilizers, however, growth-stage estimates may also suggest further investigation of potential crop infections and infestations. Traditionally, the growth-stage is based upon a manual random sample of a very small number of plants that are then analyzed to produce an estimate for the entire crop (up to thousands of acres). In order to increase the sample size (and thus accuracy) and to enable precision agriculture to address non-uniform crop development across a field, we present an analysis methodology that facilitates the automated growth-stage analysis of dense point clouds that are derived from drone imagery. Our method utilizes a standard camera drone and does not use specialized sensors or geo-spatial tagging. We propose a multi-stage unsupervised method, which provides information about the individual plant locations in a field plot with a high probability. The method also produces a measure of individual plant heights, which along with their location are critical for later growth-stage estimation and necessary for robotic precision application. We confirm our method's efficacy with experimental results on corn fields in Minnesota.

In view planning, the position and orientation of the cameras have been a major contributing factor to the quality of the resulting 3D model. In applications such as precision agriculture, a dense and accurate reconstruction must be obtained quickly while the data is still actionable. Instead of using an arbitrarily large number of images taken from every possible position and orientation in order to cover the desired area of study, a more optimal approach is required. We present an efficient and realistic pipeline, which aims to optimize the positioning of cameras and hence the quality of the 3D reconstruction of a field of row crops. This is achieved with four steps; an initial flight to obtain a sparse point cloud, the fitting of a simple mesh model, the planning of images via a discrete optimization process, and a second flight to obtain the final reconstruction. We demonstrate the effectiveness of our method by comparing it with baseline methods commonly used for agricultural data collection and processing.

Representations in the form of Symmetric Positive Definite (SPD) matrices have been popularized in a variety of visual learning applications due to their demonstrated ability to capture rich second-order statistics of visual data. There exist several similarity measures for comparing SPD matrices with documented benefits. However, selecting an appropriate measure for a given problem remains a challenge and in most cases, is the result of a trial-and-error process. In this paper, we propose to learn similarity measures in a data-driven manner. To this end, we capitalize on the αβ-log-det divergence, which is a meta-divergence parametrized by scalars α and β, subsuming a wide family of popular information divergences on SPD matrices for distinct and discrete values of these parameters. Our key idea is to cast these parameters in a continuum and learn them from data. We systematically extend this idea to learn vector-valued parameters, thereby increasing the expressiveness of the underlying non-linear measure. We conjoin the divergence learning problem with several standard tasks in machine learning, including supervised discriminative dictionary learning and unsupervised SPD matrix clustering. We present Riemannian gradient descent schemes for optimizing our formulations efficiently, and show the usefulness of our method on eight standard computer vision tasks.

Reinforcement learning has been widely applied in exploration, navigation, manipulation, and other fields. Most of the relevant techniques generate kinematic commands (e.g., move, stop, turn) for agents based on the current state information. However, recent dense action representations based research, such as spatial action maps, pointing way-points to the agent in the same domain as its observation of the state shows great promise in mobile manipulation tasks. Inspired by that, we make the first step towards using a spatial action maps based method to effectively explore novel environmental spaces. To reduce the chance of redundant exploration, the visit frequency map (VFM) and its corresponding reward function are introduced to direct the agent to actively search previously unexplored areas. In the experimental section, our work was compared to the same method without VFM and the method based on traditional steering commands with the same input data in various environments. The results show conclusively that our method is more efficient than other methods. The project page is: https://github.com/zxwang96/sam-exploration

A major component of an efficient farming strategy is the precise detection and characterization of plant deficiencies followed by the proper deployment of fertilizers. Through the thoughtful utilization of modern computer vision techniques, it is possible to achieve positive financial and environmental results for these tasks. This work introduces an automation framework that attempts to address the three main drawbacks of existing approaches: 1) lack of generality (methods are tuned for specific data sets); 2) difficulty to apply in variable field conditions; and 3) lack of tool sophistication that limits their applicability. The cultivation of corn lies in the core of the American and global economy with 81.7 million acres harvested only in the USA for the year 2018. The ubiquity of its cultivation makes it an ideal candidate to highlight the large economic benefits from even a small improvement in nutrient deficiency detection. The proposed methodology utilizes drone collected images to detect nitrogen (N) deficiencies in maize fields and assess their severity using low-cost RGB sensors. The proposed methodology is twofold. A low complexity recommendation scheme identifies candidate plants exhibiting N deficiency and, with minimal interaction, assists the annotator in the creation of a training data set that is then used to train an object detection deep neural network. Results on data from experimental fields support the merits of the proposed methodology with mean average precision for the detection of N-deficient leaves reaching 82.3%.Note to Practitioners—The motivation behind this article is the problem of inefficient fertilizer application in corn fields throughout the cultivation season. Current widely spread techniques to counter plant malnutrition suggest the application of excessive amounts of nitrogen fertilizer prior to seeding or the uniform application during the plant growth. These practices result in financial losses and have severe environmental consequences, e.g., the dead zone in the Gulf of Mexico. We propose an automation framework that automatically detects corn nitrogen deficiencies in the field during the plants’ growth, and to achieve our goal, we employ low-cost robotic platforms and RGB sensors. The framework that we developed is able to detect the characteristic pattern of nitrogen deficiency on corn leaves and provide an estimation of the in-field spatial variability of the deficiency.

As Unmanned Aerial Vehicles (UAVs) become more commonly used in industry, their performance will continue to be challenged. A performance bottleneck that is crucial to overcome is the design of electric propulsion systems for UAVs that operate in disparate flight modes (e.g., hovering and forward-moving flight). While flight mode dissimilarity presents a fundamental design challenge for fixed-geometry propulsion systems, variable-geometry systems such as the Variable Pitch Propeller (VPP) ones are able to provide superior propulsion performance across a wide range of flight modes. This work builds on previous work by the authors and presents a VPP system control and estimation framework for safe, near-minimum-electrical-effort propulsion system behavior across the whole operation state space of any UAV. Multiple simulated validations are presented to support the feasibility of the approach.

Continuous representations of objects have always been used in robotics in the form of geometric primitives and surface models. Recently, learning techniques have emerged which allow more complex continuous representations to be learned from data, but these learning techniques require training data in the form of watertight meshes which restricts their application as meshes of this form are difficult to obtain from real data. This paper proposes a modification to existing methods that allows real world point cloud data to be used for training these surface representations allowing the techniques to be used in broader applications. The modification is evaluated on ModelNet10 to quantify the difference between the existing and the proposed methods as well as on a novel precision agriculture dataset that has been released publicly to show the modification's applicability to new areas. The proposed method enables obtaining training data from real world sensors that produce point clouds rather than requiring an expensive meshing step which may not be possible for some applications. This opens the possibility of using techniques like this for complex shapes in areas like grasping and agricultural data collection.

Tumbling locomotion allows for small robots to traverse comparatively rough terrain, however, their motion is complex and difficult to control. Existing tumbling robot control methods involve manual control or the assumption of at terrain. Reinforcement learning allows for the exploration and exploitation of diverse environments. By utilizing reinforcement learning with domain randomization, a robust control policy can be learned in simulation then transferred to the real world. In this paper, we demonstrate autonomous setpoint navigation with a tumbling robot prototype on at and non- at terrain. The flexibility of this system improves the viability of nontraditional robots for navigational tasks.

The Crossing or Not-Crossing (C/NC) problem is important to autonomous vehicles (AVs) for safe vehicle/pedestrian interactions. However, this problem setup often ignores pedestrians walking along the direction of the vehicles' movement (LONG). To enhance the AVs' awareness of pedestrian behavior, we make the first step towards extending the C/NC to the C/NC/LONG problem and recognize them based on single body pose. In contrast, previous C/NC state classifiers depend on multiple poses or contextual information. Our proposed shallow neural network classifier aims to recognize these three states swiftly. We tested it on the JAAD dataset and reported an average 81.23% accuracy.

Presented in this paper are the design and experiments for a transformable Vertical Take Off and Landing (VTOL) UAV. This work demonstrates shape-shifting transformation, building upon the conceptual designs put forth in [1] and [2], along with hardware prototyping and component testing from [3]. A deterministic model is presented to characterize the flight of the MIST-UAV in simulation. Experimental results from the platform demonstrate for the first time successful in-air transformation from multi-rotor, tail-sitter, and fixed-wing operation. Experiments also validated transformation repeatability, successfully testing multiple sequential transformations.

In response to an abundance of applications, Unmanned Aerial Vehicles are being called upon to perform missions of high difficulty for increasingly long periods of time. Traditional paradigms of propeller design and actuation are reaching a design ceiling, motivating creative approaches to the design of propeller-based propulsion mechanisms. Within the last decade, one particular kind of mechanism, the variable-pitch propeller, has been studied by researchers for its applications to the class of small UAVs. This paper pushes for new results in this area by exploring the use of Variable Pitch Propulsion (VPP) to minimize power consumption for small, versatile UAVs. A control algorithm is presented to minimize the consumed electrical power during a quasi-steady propulsive state. In particular, the algorithm is not confined to operation in limited regions of the state space, but it seeks to minimize power at whatever point in the state space a steady state is reached. Several experimental results are presented to validate the approach.

We present the Signature of Topologically Persistent Points (STPP), a global descriptor that encodes topological invariants of 3D point cloud data. These topological invariants include the zeroth and first homology groups and are computed using persistent homology, a method for finding the features of a topological space at different spatial resolutions. STPP is a competitive 3D point cloud descriptor when compared to the state of art and is resilient to noisy sensor data. We demonstrate experimentally on a publicly available RGB-D dataset that STPP can be used as a distinctive signature, thus allowing for 3D point cloud processing tasks such as object detection and classification.

Financial and social elements of modern societies are closely connected to the cultivation of corn. Due to its massive production, deficiencies during the cultivation process directly translate to major financial losses. Since proper surveillance in a large scale is still very challenging, the companies that specialize in optimizing crop yield are trying to address the problem at its root by developing hybrid plants able to resist the harsh conditions of the field. The selection of the best hybrid is not easy and every year hundreds of test plants with different phenotypic characteristics are planted while their performance is quantified by inconsistent and rough measurements gathered by humans. We propose a pipeline that takes advantage of the structure from motion technology to create a detailed 3D point cloud of a few plants and segment it into the basic elements of the scene; the ground, the plants, the plant stems, and the plant leaves. The focus is on the segmentation process through which several phenotypic characteristics of individual plants can be extracted. As an example, we show the results for the plant counting and plant height estimation processes where we achieve an accuracy of 88.1% and 89.2%.

Deep neural networks typically require large amounts of annotated data to be trained effectively. However, in several scientific disciplines, including medical image analysis, generating such large annotated datasets requires specialized domain knowledge, and hence is usually very expensive. In this work, we present a novel application of active learning to data sample selection for training Convolutional Neural Networks (CNN) for Cancerous Tissue Recognition (CTR). Our main idea is to steer annotation efforts towards selecting the most informative samples for training the CNN. To quantify informativeness, we explore three choices based on discrete entropy, best-vs-second-best, and k-nearest neighbor agreement. Our results on three different types of cancer datasets consistently demonstrate that under limited annotated samples, our proposed training scheme converges faster than classical randomized stochastic gradient descent, while achieving the same (or sometimes superior) classification accuracy.

Aerial robotic platforms are an increasingly sought-after solution for a variety of sensing, monitoring, and transportation challenges. However, as invaluable as unmanned aerial vehicles (UAVs) have been for these applications, fixed-wing and multi-rotor systems each have individual limitations. Fixed-wing UAVs are generally capable of high-altitude surveillance and long flight times, while quad-rotors are most effective when used for their maneuverability and close-quarters surveying. This paper improves upon the prototypes discussed in [1] by creating a series of three next-generation prototypes to isolate the aspects of solar powered fixed-wing flight, quad-rotor flight, and transformation modes of the SUAV:Q platform. Improvements to the transformation mechanism, airframe design, variable pitch propulsion system, and custom-designed power electronics are presented along with validation of the designs through empirical testing.

The recent activity in the area of autonomous vehicle navigation has initiated a series of reactions that stirred the automobile industry, pushing for the fast commercialization of this technology which, until recently, seemed futuristic. The LiDAR sensor is able to provide a detailed understanding of the environment surrounding the vehicle making it useful in a plethora of autonomous driving scenarios. Segmenting the 3D point cloud that is provided by modern LiDAR sensors, is the first important step towards the situational assessment pipeline that aims for the safety of the passengers. This step needs to provide accurate segmentation of the ground surface and the obstacles in the vehicle's path, and to process each point cloud in real time. The proposed pipeline aims to solve the problem of 3D point cloud segmentation for data received from a LiDAR in a fast and low complexity manner that targets real world applications. The two-step algorithm first extracts the ground surface in an iterative fashion using deterministically assigned seed points, and then clusters the remaining non-ground points taking advantage of the structure of the LiDAR point cloud. Our proposed algorithms outperform similar approaches in running time, while producing similar results and support the validity of this pipeline as a segmentation tool for real world applications.

Given the wide variety of flight conditions typically encountered by fixed-wing aerial vehicles, the flight performance of a solar-powered unmanned aerial vehicle (SUAV) depends on many factors. Predicting the performance for a given application requires characterization of both system and environmental components. Furthermore, a SUAV with a transformable airframe increases the number of characterizable states, where each state features a unique set of capabilities. This results in significant differences with respect to power consumption, solar panel orientation, and increases the design limitations on system components. This paper characterizes the energy collection, propulsion, and power electronics subsystems of a transformable SUAV developed at the University of Minnesota. Clear-sky and solar panel models are used to predict the power available for a given location and time. Propulsion system design is validated with flight data, and a proposed model correlates propulsion limits with available energy. Power electronics are modeled and simulated to determine hardware and tracking algorithm efficiencies. Finally, a pulsed battery charging methodology is implemented in hardware and evaluated against conventional charging techniques.

Small robots benefit from the ability to go places where humans cannot and are attractive for numerous practical reasons such as portability and manufacturing simplicity. However, with smaller scale comes more difficulty traversing rough terrain, especially for robots which use wheel-based locomotion. Previous approaches to overcome this drawback have included auxiliary mechanisms such as jumping, transformations of the robot or its appendages, and alternative forms of locomotion such as aerial flight capability. This paper presents a small scale robot that is capable of both ground travel and aerial flight. In combination, these methods of locomotion allow for efficient ground-based movement as well as the ability to overcome obstacles and explore otherwise unreachable locations through air travel. The novel aspect of the robot design is a transformation between ground and air configurations. This feature offers advantages over previous approaches such as a highly compact ground configuration and protection of delicate flight hardware when not in use. In this paper, the robot concept is compared to other approaches to address ground robot mobility drawbacks. This is followed by a detailed design description with a focus on the transformation between the ground and air modes. Lastly, a fully functional prototype is presented which is capable of ground and air locomotion and the transformation between these configurations.

Financial and social elements of modern societies are closely connected to the cultivation of corn. Due to the massive production of corn, deficiencies during the cultivation process directly translate to major financial losses. The early detection and treatment of crops deficiencies is thus a task of great significance. Towards an automated health condition assessment, this study introduces a scheme for the computation of plant health indices. Based on the 3D reconstruction of small batches of corn plants, an alternative to existing cumbersome Leaf Area Index (LAI) estimation methodologies is presented. The use of 3D models provides an elevated information content, when compared to planar methods, mainly due to the reduced loss attributed to leaf occlusions. High resolution images of corn stalks are collected and used to obtain 3D models of plants of interest. Based on the extracted 3D point clouds, an accurate calculation of the Leaf Area Index (LAI) of the plants is performed. An experimental validation (using artificially made corn plants used as ground truth of the LAI estimation), emulating real world scenarios, supports the efficacy of the proposed methodology. The conclusions of this work, suggest a fully automated scheme for information gathering in modern farms capable of replacing current labor intensive procedures, thus greatly impacting the timely detection of crop deficiencies.

In this article, we propose a transit service Feeder to tackle the last-mile problem, that is, passengers’ destinations lay beyond a walking distance from a public transit station. Feeder utilizes ridesharing-based vehicles (e.g., minibus) to deliver passengers from existing transit stations to selected stops closer to their destinations. We infer real-time passenger demand (e.g., exiting stations and times) for Feeder design by utilizing extreme-scale urban infrastructures, which consist of 10 million cellphones, 27 thousand vehicles, and 17 thousand smartcard readers for 16 million smartcards in a Chinese city, Shenzhen. Regarding these numerous devices as pervasive sensors, we mine both online and offline data for a two-end Feeder service: a back-end Feeder server to calculate service schedules and front-end customized Feeder devices in vehicles for real-time schedule downloading. We implement Feeder using a fleet of vehicles with customized hardware in a subway station of Shenzhen by collecting data for 30 days. The evaluation results show that compared to the ground truth, Feeder reduces last-mile distances by 68% and travel time by 56%, on average.

Computer-Aided Diagnosis (CAD) has witnessed a rapid growth over the past decade, providing a variety of automated tools for the analysis of medical images. In surgical pathology, such tools enhance the diagnosing capabilities of pathologists by allowing them to review and diagnose a larger number of cases daily. Geared towards developing such tools, the main goal of this paper is to identify useful computer vision based feature descriptors for recognizing cancerous tissues in histopathologic images. To this end, we use images of Hematoxylin & Eosin-stained microscopic sections of breast and prostate carcinomas, and myometrial leiomyosarcomas, and provide an exhaustive evaluation of several state of the art feature representations for this task. Among the various image descriptors that we chose to compare, including representations based on convolutional neural networks, Fisher vectors, and sparse codes, we found that working with covariance based descriptors shows superior performance on all three types of cancer considered. While covariance descriptors are known to be effective for texture recognition, it is the first time that they are demonstrated to be useful for the proposed task and evaluated against deep learning models. Capitalizing on Region Covariance Descriptors (RCDs), we derive a powerful image descriptor for cancerous tissue recognition termed, Covariance Kernel Descriptor (CKD), which consistently outperformed all the considered image representations. Our experiments show that using CKD lead to 92.83%, 91.51%, and 98.10% classification accuracy for the recognition of breast carcinomas, prostate carcinomas, and myometrial leiomyosarcomas, respectively.

In this paper, we present an approach to segment 3D point cloud data using ideas from persistent homology theory. The proposed algorithms first generate a simplicial complex representation of the point cloud dataset. Next, we compute the zeroth homology group of the complex which corresponds to the number of connected components. Finally, we extract the clusters of each connected component in the dataset. We show that this technique has several advantages over state of the art methods such as the ability to provide a stable segmentation of point cloud data under noisy or poor sampling conditions and its independence of a fixed distance metric.

Selecting an aerial platform for an application typically requires compromise. A choice must be made between the flight time and long-range capabilities of a fixed-wing aircraft or the maneuverability and stationary characteristics of a multi-rotor platform. Recent developments of small-scale solar-powered UAVs have leveraged the advances in solar cell, energy storage, and propulsion system technology to reach extended flight times capable of all-day and multi-day flight. This paper presents the concept of a small-scale hybrid unmanned aerial vehicle capable of augmenting the maneuverability of a quad-rotor with the energy collection and supply of a solar-powered fixed-wing aircraft. An investigation into the aircraft design, transforming mechanism, and energy management of the multi-state system is presented. A proof-of-concept prototype has been constructed to demonstrate the airframe operating in a quad-rotor configuration. Power electronics capable of simultaneous battery charging and power loading from a solar array have been validated. Additional work in optimization of the propulsion system and airframe needs to be completed to maximize the performance of the hybrid system.

Analysis of behavior using video is a promising approach for identifying risk markers for psychopathology that can be applied in a wide range of populations. Computer vision techniques are needed to automatically code behavior in order to reduce time and effort in these analyses. This paper discusses algorithms developed for the automatic analysis of video data from a study regarding the impact of environmental factors on youths with obsessive-compulsive disorder. Overhead videos of subjects washing hands were automatically annotated for activities such as rinsing, applying soap, and turning on/off the water faucet. These automated annotations were created by using a color-based background subtraction method to create a foreground probability score which is then used to determine if various labeled regions of interest (ROIs) are activated. These activation signals are then characterized and used to determine when different substeps of the handwashing procedure are performed. Automated annotations were validated by comparisons with hand-labeled ground truth.

A `region' is an important concept in interpreting 3D point cloud data since regions may correspond to objects in a scene. To correctly interpret 3D point cloud data, we need to partition the dataset into regions that correspond to objects or parts of an object. In this paper, we present a region growing approach that combines global (topological) and local (color, surface normal) information to segment 3D point cloud data. Using ideas from persistent homology theory, our algorithm grows a simplicial complex representation of the point cloud dataset. At each step in the growth process we compute the zeroth homology group of the complex, which corresponds to the number of connected components, and use color and surface normal statistics to build regions. Lastly, we extract out the segmented regions of the dataset. We show that this method provides a stable segmentation of point cloud data in the presence of noise and poorly sampled data, thus providing advantages over contemporary region-based segmentation techniques.

Throughout the wide range of aerial robot related applications, selecting a particular airframe is often a trade-off. Fixed-wing small-scale unmanned aerial vehicles (UAVs) typically have difficulty surveying at low altitudes while quadrotor UAVs, having more maneuverability, suffer from limited flight time. Recent prior work [1] proposes a solar-powered small-scale aerial vehicle designed to transform between fixed-wing and quad-rotor configurations. Surplus energy collected and stored while in a fixed-wing configuration is utilized while in a quad-rotor configuration. This paper presents an improvement to the robot's design in [1] by pursuing a modular airframe, an optimization of the hybrid propulsion system, and solar power electronics. Two prototypes of the robot have been fabricated for independent testing of the airframe in fixed-wing and quad-rotor states. Validation of the solar power electronics and hybrid propulsion system designs were demonstrated through a combination of simulation and empirical data from prototype hardware.

This work focuses on the design and predicted performance of a two meter wingspan solar powered unmanned aerial vehicle (UAV). Such a platform would be ideal for distributed robotics applications because it combines the portability and deployment simplicity of a small airframe with the long flight time of a solar UAV. Methods to design and predict properties of a two meter solar UAV are described including airframe type selection, mass estimation, and propulsion requirements. A simplified approach to predict flight time is presented as well as an improved metric for quantifying multiday flight robustness. Maximum flight time for the two meter airframe considered is estimated to be greater than ten hours which is an order of magnitude improvement over reported commercially available options. In terms of multi-day flight capability, total mass is predicted to be within the bounds of a realizable aircraft based on extrapolation from larger experimentally tested multi-day solar UAVs.

Active Constraint Learning (ACL) is continuously gaining popularity in the area of constrained clustering due to its ability to achieve performance gains via incorporating minimal feedback from a human annotator for selected instances. For constrained clustering algorithms, such instances are integrated in the form of Must-Link (ML) and Cannot-Link (CL) constraints. Existing iterative uncertainty reduction schemes, introduce high computational burden particularly when they process larger datasets that are usually present in computer vision and visual learning applications. For scenarios that multiple agents (i.e., robots) require user feedback for performing recognition tasks, minimizing the interaction between the user and the agents, without compromising performance, is an essential task. In this study, a non-iterative ACL scheme with proven performance benefits is presented. We select to demonstrate the effectiveness of our methodology by building on the well known K-Means algorithm for clustering; one can easily extend it to alternative clustering schemes. The proposed methodology introduces the use of the Silhouette values, conventionally used for measuring clustering performance, in order to rank the degree of information content of the various samples. In addition, an efficient greedy selection scheme was devised for selecting the most informative samples for human annotation. To the best of our knowledge, this is the first active constrained clustering methodology with the ability to process computer vision datasets that this study targets. Performance results are shown on various computer vision benchmarks and support the merits of adopting the proposed scheme.

Symmetric Positive Definite (SPD) matrices emerge as data descriptors in several applications of computer vision such as object tracking, texture recognition, and diffusion tensor imaging. Clustering these data matrices forms an integral part of these applications, for which soft-clustering algorithms (K-Means, expectation maximization, etc.) are generally used. As is well-known, these algorithms need the number of clusters to be specified, which is difficult when the dataset scales. To address this issue, we resort to the classical nonparametric Bayesian framework by modeling the data as a mixture model using the Dirichlet process (DP) prior. Since these matrices do not conform to the Euclidean geometry, rather belongs to a curved Riemannian manifold,existing DP models cannot be directly applied. Thus, in this paper, we propose a novel DP mixture model framework for SPD matrices. Using the log-determinant divergence as the underlying dissimilarity measure to compare these matrices, and further using the connection between this measure and the Wishart distribution, we derive a novel DPM model based on the Wishart-Inverse-Wishart conjugate pair. We apply this model to several applications in computer vision. Our experiments demonstrate that our model is scalable to the dataset size and at the same time achieves superior accuracy compared to several state-of-the-art parametric and nonparametric clustering algorithms.

In this paper, we introduce a dictionary learning framework using RGB-D covariance descriptors on point cloud data for performing object classification. Dictionary learning in combination with RGB-D covariance descriptors provides a compact and flexible description of point cloud data. Furthermore, the proposed framework is ideal for updating and sharing dictionaries among robots in a decentralized or cloud network. This work demonstrates the increased performance of 3D object classification utilizing covariance descriptors and dictionary learning over previous results with experiments performed on a publicly available RGB-D database.

An object recognition engine needs to extract discriminative features from data representing an object and accurately classify the object to be of practical use in robotics. Furthermore, the classification of the object must be rapidly performed in the presence of a voluminous stream of data. These conditions call for a distributed and scalable architecture that can utilize a cloud computing infrastructure for performing object recognition. This paper introduces a Cloud-based Object Recognition Engine (CORE) to address these needs. CORE is able to train on large-scale datasets, perform classification of 3D point cloud data, and efficiently transfer data in a robotic network.

Determining and detecting risk markers for mental illness remains a labor intensive process, requiring vast amounts of observations by clinical professionals. Motor stereotypies, which are defined as involuntary repetitive motor behaviors, invariant in form, that, to an observer, appear to serve no purpose, are a class of risk markers which are very amenable to video analysis. These behaviors are associated with mental illnesses such as Autism, Rett Syndrome, and other developmental disabilities. This paper investigates the application of innovative automated methods to recognize these subtle motor indicators. To train and test our methods, a dataset of actions resembling motor stereotypies was created by engaging the normally developing children at the University of Minnesota's Shirley G. Moore Laboratory School. Comparison to a publicly available dataset depicting a subset of behaviors is performed as well. This work demonstrates the applicability of various techniques in the behavioral science domain. The results show that these techniques can perform well on a difficult and challenging real-world scenario.

Unmanned solar powered aircraft offer a unique set of advanced capabilities and have set general aviation records for longest continuous flight and greatest sustained altitude. However, the application of solar powered flight to small scale solar powered unmanned aerial vehicles (UAVs) has seen sparse research activity and is only partially explored. The use of solar power as an energy resource allows small scale UAVs to carry heavier, more powerful sensor payloads, and can extend flight times to over 24 hours, thereby achieving multi-day flight. This work focuses on recent developments by the Center for Distributed Robotics on a four meter wingspan solar UAV designed for low altitude aerial sensing applications. Highlighted in this paper are aspects of airframe, propulsion, and electronics hardware design as well as experiments that quantify the solar power system and airframe performance.

The continuously growing need for increasing the production of food and reducing the degradation of water supplies, has led to the development of several precision agriculture systems over the past decade so as to meet the needs of modern societies. The present study describes a methodology for the detection and characterization of Nitrogen (N) deficiencies in corn fields. Current methods of field surveillance are either completed manually or with the assistance of satellite imaging, which offer infrequent and costly information to the farmers about the state of their fields. The proposed methodology promotes the use of small-scale Unmanned Aerial Vehicles (UAVs) and Computer Vision algorithms that operate with information in the visual (RGB) spectrum. Through this implementation, a lower cost solution for identifying N deficiencies is promoted. We provide extensive results on the use of commercial RGB sensors for delivering the essential information to farmers regarding the condition of their field, targeting the reduction of N fertilizers and the increase of the crop performance. Data is first collected by a UAV that hovers over a stressed area and collects high resolution RGB images at a low altitude. A recommendation algorithm identifies potential segments of the images that are candidates exhibiting N deficiency. Based on the feedback from experts in the area a training set is constructed utilizing the initial suggestions of the recommendation algorithm. Supervised learning methods are then used to characterize crop leaves that exhibit signs of N deficiency. The performance of 84.2% strongly supports the potential of this scheme to identify N-deficient leaves even in the case of images where the unhealthy leaves are heavily occluded by other healthy or stressed leaves.

Sparse models have proven to be extremely successful in image processing and computer vision. However, a majority of the effort has been focused on sparse representation of vectors and low-rank models for general matrices. The success of sparse modeling, along with popularity of region covariances, has inspired the development of sparse coding approaches for these positive definite descriptors. While in earlier work, the dictionary was formed from all, or a random subset of, the training signals, it is clearly advantageous to learn a concise dictionary from the entire training set. In this paper, we propose a novel approach for dictionary learning over positive definite matrices. The dictionary is learned by alternating minimization between sparse coding and dictionary update stages, and different atom update methods are described. A discriminative version of the dictionary learning approach is also proposed, which simultaneously learns dictionaries for different classes in classification or clustering. Experimental results demonstrate the advantage of learning dictionaries from data both from reconstruction and classification viewpoints. Finally, a software library is presented comprising C++ binaries for all the positive definite sparse coding and dictionary learning approaches presented here.

Tractor-trailer freight hauling has increased markedly within the United States over the past several years, resulting in higher truck volumes. commercial heavy vehicle drivers are required under federal Hours Of Services rules to rest and take breaks to mitigate driving while fatigued. Although there are many rest area facilities available to truck drivers, there is a lack of persistent timely information on truck parking availability. An automated real-time sensing system to directly detect and disseminate parking space occupancy from truck parking facilities is described in detail. The methodology and system architecture are presented in which robust, persistent, parking occupancy detection is achieved by extending Structure from Motion (SfM) techniques using a multiplicity of commercial, off-the-shelf cameras. The system architecture allows the approach to be scaled to a region-wide comprehensive truck parking information system for commercial heavy vehicle drivers and operators. Per parking space detection accuracy of 99% is achieved over continuous operation. Classification accuracy under diverse scene and parking behavior scenarios is discussed.

Object segmentation and classification is an important and difficult task in robotic vision. The task is complicated even further when the different objects are partially or completely occluded. Allowing a robot to take measurements from varying points of view can help in alleviating or completely removing occlusions. A robot equipped with an RGB-D sensor has the capability of searching for new and better points of view to facilitate object recognition. In this work, a motion control algorithm is designed and implemented on a mobile robot to facilitate object classification in RGB-D data of clustered objects.

In this paper, we introduce a new covariance based feature descriptor to be used on “colored” point clouds gathered by a mobile robot equipped with an RGB-D camera. Although many recent descriptors provide adequate results, there is not yet a clear consensus on how to best tackle “colored” point clouds. We present the notion of a covariance on RGB-D data. Covariances have not only been proven to be successful in image processing, but in other domains as well. Their main advantage is that they provide a compact and flexible description of point clouds. Our work is a first step towards demonstrating the usability of the concept of covariances in conjunction with RGB-D data. Experiments performed on an RGB-D database and compared to previous results show the increased performance of our method.

In this paper, we present two real-time methods for controlling data transmission in a robotic network that utilizes a remote computing infrastructure. The proposed algorithms use information and communication theory concepts to perform a highly efficient transfer of RGB-D data from a client (robot) to a server (cloud). We show that this approach makes it possible to conserve bandwidth and reduce network latency while allowing a mobile robot to perform vision tasks.

In recent years, there has been extensive research on sparse representation of vector-valued signals. In the matrix case, the data points are merely vectorized and treated as vectors thereafter (for example, image patches). However, this approach cannot be used for all matrices, as it may destroy the inherent structure of the data. Symmetric positive definite (SPD) matrices constitute one such class of signals, where their implicit structure of positive eigenvalues is lost upon vectorization. This paper proposes a novel sparse coding technique for positive definite matrices, which respects the structure of the Riemannian manifold and preserves the positivity of their eigenvalues, without resorting to vectorization. Synthetic and real-world computer vision experiments with region covariance descriptors demonstrate the need for and the applicability of the new sparse coding model. This work serves to bridge the gap between the sparse modeling paradigm and the space of positive definite matrices.

This paper presents a new nearest neighbor (NN) retrieval framework: robust sparse hashing (RSH). Our approach is inspired by the success of dictionary learning for sparse coding. Our key idea is to sparse code the data using a learned dictionary, and then to generate hash codes out of these sparse codes for accurate and fast NN retrieval. But, direct application of sparse coding to NN retrieval poses a technical difficulty: when data are noisy or uncertain (which is the case with most real-world data sets), for a query point, an exact match of the hash code generated from the sparse code seldom happens, thereby breaking the NN retrieval. Borrowing ideas from robust optimization theory, we circumvent this difficulty via our novel robust dictionary learning and sparse coding framework called RSH, by learning dictionaries on the robustified counterparts of the perturbed data points. The algorithm is applied to NN retrieval on both simulated and real-world data. Our results demonstrate that RSH holds significant promise for efficient NN retrieval against the state of the art.

An aircraft that is capable of continuous flight offers a new level of autonomous capacity for unmanned aerial vehicles. We present an overview of the components and concepts of a small scale unmanned aircraft that is capable of sustaining powered flight without a theoretical time limit. We then propose metrics that quantify the robustness of continuous flight achieved and optimization criteria to maximize these metrics. Finally, the criteria are applied to a fabricated and flight tested small scale high efficiency aircraft prototype to determine the optimal battery and photovoltaic array mass for robust continuous flight.

Computer vision as an entire field has a wide and diverse range of applications. The specific application for this project was in the realm of dance, notably ballet and choreography. This project was proof-of-concept for a choreography assistance tool used to recognize and record dance movements demonstrated by a choreographer. Keeping the commercial arena in mind, the Kinect from Microsoft was chosen as the imaging hardware, and a pilot set chosen to verify recognition feasibility. Before implementing a classifier, all training and test data was transformed to a more applicable representation scheme to only pass the important aspects to the classifier to distinguish moves for the pilot set. In addition, several classification algorithms using the Nearest Neighbor (NN) and Support Vector Machine (SVM) methods were tested and compared from a single dictionary as well as on several different subjects. The results were promising given the framework of the project, and several new expansions of this work are proposed.

Presented are results demonstrating that, in developing a system with its first objective being the sustained detection of adults and young children as they move and interact in a normal preschool setting, the direct application of the straightforward RGB-D innovations presented here significantly outperforms even far more algorithmically advanced methods relying solely on images. The use of multiple RGB-D sensors by this project for depth-aware object localization economically resolves numerous issues regularly frustrating earlier vision-only detection and human surveillance methods, issues such as occlusions, illumination changes, unexpected postures, atypical morphologies, erratic or unanticipated motions, reflections, and misleading textures and colorations. This multiple RGB-D installation forms the front-end for a multi-step pipeline, the first portion of which seeks to isolate, in situ, 3D renderings of classroom occupants sufficient for a later analysis of their behaviors and interactions. Towards this end, a voxel-based approach to foreground/background separation and an effective adaptation of supervoxel clustering for 3D were developed, and 3D and image-only methods were tested and compared. The project's setting is highly challenging, but then so are its longer term goals: the automated detection of early childhood precursors, ofttimes very subtle, to a number of increasingly common developmental disorders.

Covariance matrices have found success in several computer vision applications, including activity recognition, visual surveillance, and diffusion tensor imaging. This is because they provide an easy platform for fusing multiple features compactly. An important task in all of these applications is to compare two covariance matrices using a (dis)similarity function, for which the common choice is the Riemannian metric on the manifold inhabited by these matrices. As this Riemannian manifold is not flat, the dissimilarities should take into account the curvature of the manifold. As a result, such distance computations tend to slow down, especially when the matrix dimensions are large or gradients are required. Further, suitability of the metric to enable efficient nearest neighbor retrieval is an important requirement in the contemporary times of big data analytics. To alleviate these difficulties, this paper proposes a novel dissimilarity measure for covariances, the Jensen-Bregman LogDet Divergence (JBLD). This divergence enjoys several desirable theoretical properties and at the same time is computationally less demanding (compared to standard measures). Utilizing the fact that the square root of JBLD is a metric, we address the problem of efficient nearest neighbor retrieval on large covariance datasets via a metric tree data structure. To this end, we propose a K-Means clustering algorithm on JBLD. We demonstrate the superior performance of JBLD on covariance datasets from several computer vision applications.

Object recognition algorithms often focus on determining the class of a detected object in a scene. Two significant phases are usually involved in object recognition. The first phase is the object representation phase, in which the most suitable features that provide the best discriminative power under constraints such as lighting, resolution, scale, and view variations are chosen to describe the objects. The second phase is to use this representation space to develop models for each object class using discriminative classifiers. In this paper, we focus on composite objects, i.e., objects with two or more simpler classes that are interconnected in a complicated manner. One classic example of such a scenario is a bicyclist. A bicyclist consists of a bicycle and a human who rides the bicycle. When we are faced with the task of classifying bicyclists and pedestrians, it is counterintuitive and often hard to come up with a discriminative classifier to distinguish the two classes. We explore global image analysis based on bag of visual words to compare the results with local image analysis, in which we attempt to distinguish the individual parts of the composite object. We also propose a unified naive Bayes framework and a combined histogram feature method for combining the individual classifiers for enhanced performance.

We study Nearest Neighbors (NN) retrieval by introducing a new approach: Robust Sparse Hashing (RSH). Our approach is inspired by the success of dictionary learning for sparse coding; the key innovation is to use learned sparse codes as hashcodes for speeding up NN. But sparse coding suffers from a major drawback: when data are noisy or uncertain, for a query point, an exact match of the hashcode seldom happens, breaking the NN retrieval. We tackle this difficulty via our novel dictionary learning and sparse coding framework called RSH by learning dictionaries on the robustified counterparts of uncertain data points. The algorithm is applied to NN retrieval for Scale Invariant Feature Transform (SIFT) descriptors. The results demonstrate that RSH is noise tolerant, and at the same time shows promising NN performance over the state-of-the-art.

Object recognition entails extracting information about which object class(es) are present in an image. In order to enhance the performance of object recognition, reducing the redundancy in the data is absolutely essential. Prior literature [1, 2] introduced local and global self-similarity features to highlight the areas in an image which are useful for object classification and detection. We introduce an efficient self-similarity measure based on sparse representations and propose two different descriptors. Our measure of self-similarity is determined across multiple scales and is more efficient than prior work. We test our self similarity descriptor using support vector machine based classification on the PASCAL VOC 2007 database consisting of 20 object classes. Comparative results indicate performance competitive with the prior approaches of computing self-similarity descriptors.

One of the most important tasks for mobile robots is to sense their environment. Further tasks might include the recognition of objects in the surrounding environment. Three dimensional range finders have become the sensors of choice for mapping the environment of a robot. Recognizing objects in point clouds provided by such sensors is a difficult task. The main contribution of this paper is the introduction of a new covariance based point cloud descriptor for such object recognition. Covariance based descriptors have been very successful in image processing. One of the main advantages of these descriptors is their relatively small size. The comparisons between different covariance matrices can also be made very efficient. Experiments with real world and synthetic data will show the superior performance of the covariance descriptors on point clouds compared to state-of-the-art methods.

Tumbling robots provide the potential to produce increased mobility on smaller scales with respect to their size and/or complexity. In this paper we explore the frictional interactions between a tumbling robot and the terrain while climbing a single vertical step to illustrate the advantages of tumbling. We present a set of parametric configuration equations that express the relationships between the robot's configuration parameters (morphology, geometry, mass, etc.), the environmental/task parameters (step geometry, available coefficients of friction, etc.), and the performance parameters (step height). The required body coefficient of friction is examined in detail for idealized tumbling and wheel-tail robots.

Fast image recognition and classification is extremely important in various robotics applications such as exploration, rescue, localization, etc. k-nearest neighbor (kNN) classifiers are popular tools used in classification since they involve no explicit training phase, and are simple to implement. However, they often require large amounts of training data to work well in practice. In this paper, we propose a batch-mode active learning algorithm for efficient training of kNN classifiers, that substantially reduces the amount of training required. As opposed to much previous work on iterative single-sample active selection, the proposed system selects samples in batches. We propose a coverage formulation that enforces selected samples to be distributed such that all data points have labeled samples at a bounded maximum distance, given the training budget, so that there are labeled neighbors in a small neighborhood of each point. Using submodular function optimization, the proposed algorithm presents a near-optimal selection strategy for an otherwise intractable problem. Further we employ uncertainty sampling along with coverage to incorporate model information and improve classification. Finally, we use locality sensitive hashing for fast retrieval of nearest neighbors during active selection as well as classification, which provides 1-2 orders of magnitude speedups thus allowing real-time classification with large datasets.

Clinical studies confirm that mental illnesses such as autism, Obsessive Compulsive Disorder (OCD), etc. show behavioral abnormalities even at very young ages; the early diagnosis of which can help steer effective treatments. Most often, the behavior of such at-risk children deviate in very subtle ways from that of a normal child; correct diagnosis of which requires prolonged and continuous monitoring of their activities by a clinician, which is a difficult and time intensive task. As a result, the development of automation tools for assisting in such monitoring activities will be an important step towards effective utilization of the diagnostic resources. In this paper, we approach the problem from a computer vision standpoint, and propose a novel system for the automatic monitoring of the behavior of children in their natural environment through the deployment of multiple non-invasive sensors (cameras and depth sensors). We provide details of our system, together with algorithms for the robust tracking of the activities of the children. Our experiments, conducted in the Shirley G. Moore Laboratory School, demonstrate the effectiveness of our methodology.

Action classification is an important component of human-computer interaction. Trajectory classification is an effective way of performing action recognition with significant success reported in the literature. We compare two different representation schemes, raw multivariate time-series data and the covariance descriptors of the trajectories, and apply sparse representation techniques for classifying the various actions. The features are sparse coded using the Orthogonal Matching Pursuit algorithm, and the gestures and actions are classified based on the reconstruction residuals. We demonstrate the performance of our approach on standardized datasets such as the Australian Sign Language (AusLan) and UCF Motion Capture datasets, collected using high-quality motion capture systems, as well as motion capture data obtained from a Microsoft Kinect sensor.

Mobile robots are often proposed as a favorable substitute to human correspondence in emergency response, disaster relief, and environmental monitoring scenarios. In this work, the next iteration of the Aquapod is proposed as a method to facilitate collection of subsurface liquid samples in order to assess toxicity levels in a body of water. This amphibious small form-factor robot is equipped with a buoyancy control unit, detachable fluidic sampling unit, and a wide range of sensing and processing capabilities. The robot was designed to move and collect water samples to a maximum depth of ten meters. Its unique form of tumbling locomotion results in a versatile platform that can be used in both terrestrial and aquatic environments leveraging its high mobility-to-size ratio.

Early intervention in mental disorders can dramatically increase an individual's quality of life. Additionally, when symptoms of mental illness appear in childhood or adolescence, they represent the later stages of a process that began years earlier. One goal of psychiatric research is to identify risk-markers: genetic, neural, behavioral and/or social deviations that indicate elevated risk of a particular mental disorder. Ideally, screening of risk-markers should occur in a community setting, and not a clinical setting which may be time-consuming and resource-intensive. Given this situation, a system for automatically detecting risk-markers in children would be highly valuable. In this paper, we describe such a system that has been installed at the Shirley G. Moore Lab School, a research pre-school at the University of Minnesota. This system consists of multiple RGB+D sensors and is able to detect children and adults in the classroom, tracking them as they move around the room. We use the tracking results to extract high-level information about the behavior and social interaction of children, that can then be used to screen for early signs of mental disorders.

In developmental disorders such as autism and schizophrenia, observing behavioral precursors in very early childhood can allow for early intervention and can improve patient outcomes. While such precursors open the possibility of broad and large-scale screening, until now they have been identified only through experts' painstaking examinations and their manual annotations of limited, unprocessed video footage. Here we introduce a system to automate and assist in such procedures. Employing multiple inexpensive real-time rgb+depth (rgb+d) sensors recording from multiple viewpoints, our non-invasive system - now installed at the Shirley G. Moore Lab School, a research preschool - is being developed to monitor and reconstruct the play and interactions of preschoolers. The system's role is to help in assessing the growing volumes of its on-site recordings and to provide the data needed to uncover additional neuromotor behavioral markers via techniques such as data mining.

Machine learning techniques for computer vision applications like object recognition, scene classification, etc., require a large number of training samples for satisfactory performance. Especially when classification is to be performed over many categories, providing enough training samples for each category is infeasible. This paper describes new ideas in multiclass active learning to deal with the training bottleneck, making it easier to train large multiclass image classification systems. First, we propose a new interaction modality for training which requires only yes-no type binary feedback instead of a precise category label. The modality is especially powerful in the presence of hundreds of categories. For the proposed modality, we develop a Value-of-Information (VOI) algorithm that chooses informative queries while also considering user annotation cost. Second, we propose an active selection measure that works with many categories and is extremely fast to compute. This measure is employed to perform a fast seed search before computing VOI, resulting in an algorithm that scales linearly with dataset size. Third, we use locality sensitive hashing to provide a very fast approximation to active learning, which gives sublinear time scaling, allowing application to very large datasets. The approximation provides up to two orders of magnitude speedups with little loss in accuracy. Thorough empirical evaluation of classification accuracy, noise sensitivity, imbalanced data, and computational performance on a diverse set of image datasets demonstrates the strengths of the proposed algorithms.

Covariance matrices of multivariate data capture feature correlations compactly, and being very robust to noise, they have been used extensively as feature descriptors in many areas in computer vision, like, people appearance tracking, DTI imaging, face recognition, etc. Since these matrices do not adhere to the Euclidean geometry, clustering algorithms using the traditional distance measures cannot be directly extended to them. Prior work in this area has been restricted to using K-means type clustering over the Rieman-nian space using the Riemannian metric. As the applications scale, it is not practical to assume the number of components in a clustering model, failing any soft-clustering algorithm. In this paper, a novel application of the Dirich-let Process Mixture Model framework is proposed towards unsupervised clustering of symmetric positive definite matrices. We approach the problem by extending the existing K-means type clustering algorithms based on the logdet divergence measure and derive the counterpart of it in a Bayesian framework, which leads to the Wishart-Inverse Wishart conjugate pair. Alternative possibilities based on the matrix Frobenius norm and log-Euclidean measures are also proposed. The models are extensively compared using two real-world datasets against the state-of-the-art algorithms and demonstrate superior performance.

The idea of learning overcomplete dictionaries based on the paradigm of compressive sensing has found numerous applications, among which image denoising is considered one of the most successful. But many state-of-the-art denoising techniques inherently assume that the signal noise is Gaussian. We instead propose to learn overcomplete dictionaries where the signal is allowed to have both Gaussian and (sparse) Laplacian noise. Dictionary learning in this setting leads to a difficult non-convex optimization problem, which is further exacerbated by large input datasets. We tackle these difficulties by developing an efficient online algorithm that scales to data size. To assess the efficacy of our model, we apply it to dictionary learning for data that naturally satisfy our noise model, namely, Scale Invariant Feature Transform (SIFT) descriptors. For these data, we measure performance of the learned dictionary on the task of nearest-neighbor retrieval: compared to methods that do not explicitly model sparse noise our method exhibits superior performance.

Covariance matrices provide compact, informative feature descriptors for use in several computer vision applications, such as people-appearance tracking, diffusion-tensor imaging, activity recognition, among others. A key task in many of these applications is to compare different covariance matrices using a (dis)similarity function. A natural choice here is the Riemannian metric corresponding to the manifold inhabited by covariance matrices. But computations involving this metric are expensive, especially for large matrices and even more so, in gradient-based algorithms. To alleviate these difficulties, we advocate a novel dissimilarity measure for covariance matrices: the Jensen-Bregman LogDet Divergence. This divergence enjoys several useful theoretical properties, but its greatest benefits are: (i) lower computational costs (compared to standard approaches); and (ii) amenability for use in nearest-neighbor retrieval. We show numerous experiments to substantiate these claims.

Sparse models have proven to be extremely successful in image processing and computer vision, and most efforts have been focused on sparse representation of vectors. The success of sparse modeling and the popularity of region covariances have inspired the development of sparse coding approaches for positive definite matrices. While in earlier work [1], the dictionary was pre-determined, it is clearly advantageous to learn a concise dictionary adaptively from the data at hand. In this paper, we propose a novel approach for dictionary learning over positive definite matrices. The dictionary is learned by alternating minimization between the sparse coding and dictionary update stages, and two different atom update methods are described. The online versions of the dictionary update techniques are also outlined. Experimental results demonstrate that the proposed learning methods yield better dictionaries for positive definite sparse coding. The learned dictionaries are applied to texture and face data, leading to improved classification accuracy and strong detection performance, respectively.

Abstract-A robot scrub nurse (RSN) is an example of a robotic assistant for surgical environments. Ideally, by taking over management of instruments, it would lower costs of an operation and cut down on mistakes. Of vital importance for such robots is how they interface with the environment. A scrub nurse robot requires the ability to sense the human operators before it can assist. Computer vision offers here a number of advantages over other sensing modalities. In this paper we examine a visual tracking system for a robot scrub nurse. The system works by estimating the hand position and orientation of the main surgeon. This information is needed to guide the robot in delivering instruments directly to the surgeon. Our work outlines the entire visual tracking process and evaluates robustness and accuracy. The end result is a re-implementable and working application, suitable for surgical environments, that also offers a degree of operation robustness.

The literature on distributed robotic teams working in adversarial settings focuses primarily on external entities attempting to thwart the team. The question is raised, however, about what can be done when the adversary is within the team itself? That is, if one team member turns traitor? This paper presents an initial investigation into this question. A method is developed which can be used to classify the behavior of other team members, and provide a measure of how similar their behavior is to the expected behavior. A simulation and a real-world experiment are presented, and results show that expected and traitorous behavior are distinguishable in an example real world setting. I.

As mobile robots decrease in size so does their ability to traverse rough terrain. New forms of locomotion beyond the basic wheel are being explored to overcome this fault. This paper expands on the mechanical design of a previous robot with a high mobility-to-size ratio. To accomplish high mobility the robot uses tumbling as its form of locomotion. By actively involving the body of the robot in the locomotion it can scale larger obstacles and will not get stuck in compliant terrain like similar sized wheeled robots. To accommodate real-world environments the new design has been waterproofed and moreover can be completely submerged in water to operate on a lake or stream floor. Additionally, this robot is equipped with a buoyancy control unit which will allow the robot to either sink or float in water, offering many unique applications in environmental monitoring and surveillance. This paper describes a first generation, radio controlled prototype of the design.

In this paper we introduce tumbling, a relatively unexplored method of locomotion in which the robot utilizes net body rotations while ambulating. Tumbling for mobile robots is attractive in that it can enable increased mobility on smaller scales, often while reducing hardware requirements. As motivation for this interesting form of locomotion we provide a geometric analysis of a vertical step climbing task, one that tumbling robots perform well with respect to their size and complexity. In addition to our analysis we present results of a hardware experiment with a tumbling robot performing the task for varying combinations of frictional coefficients at the step and ground.

This paper describes the use of a probabilistic quality metric for planning camera placement for 3D reconstructions. A probabilistic quality metric estimates the probability of a reconstruction achieving a desired goal. This probabilistic model leads to the natural integration of many different factors influencing the quality of a reconstruction without relying on arbitrary weights for those factors. The specific factors addressed here are occlusions, feature matching, and feature localization. It is demonstrated how these factors impact the quality of a reconstruction and how they can be accounted for in a probabilistic manner. The developed quality metric is then used to optimize a camera network for patient tracking during tomotherapy.

The utility of miniature ground robots has long been limited by reduced locomotion capabilities compared to larger robots. Many avenues of research have been pursued to improve ground locomotion to alleviate this issue. In this paper, another option is explored in which a small ground robot is equipped with the ability to fly, allowing it to move to previously unreachable areas if necessary. The robot design is presented with an emphasis on mechanical aspects. The design utilizes a minimalistic two-wheeled ground mode to minimize weight, and a rotary-wing flight mode, enabling transformations at will. The transition between modes requires a transformation wherein the robot tips itself on-end and unfolds rotors or vise versa. The design presented herein is an improvement upon previous designs using the same concept; it is more robust in both its transformation process and locomotion capabilities [1], [2]. In addition to a new robot design, two new principles for the design of such robot are proposed: protection of flight mode components and isolation of the two modes' drivetrains.

This work presents a novel multi-robot coverage scheme for an unknown number of robots; it focuses on optimizing the number of robots and each path cost. Coverage problems traditionally deal with how a given number of robots covers the entire environment. This work, however, presents solutions of not only (i) how to cover the area (locations of interest) within the minimum time, but simultaneously (ii) how to find the optimal number of robots for a given time. Also, we consider the worst but realistic case of all robots starting at the same location instead of assuming randomly initialized positions. The minimum coverage time depends upon the number of robots used. Our research specifies (iii) how to find the minimum coverage time without knowing the number of robots. Finally, we present a deterministic coverage algorithm based on finding the shortest paths in order to optimize the number of robots and corresponding paths.

This paper presents a method to perform Euclidean calibration on camera and projector-based structured light systems, without assuming specific scene structure. The vast majority of the methods in the literature rely on prior knowledge of the 3D scene geometry in order to perform calibration, i.e., the nature of the occluding bodies in the scene needs to be known beforehand in order to calibrate structured light systems. Examples of prior knowledge used include using known stationary occluding bodies, precisely maneuvering known occluding bodies, knowing the exact world location of projected points or lines, or ensuring the entire scene obeys some other specific setup. By using multiple cameras, the method presented in this paper is able to calibrate camera and projector systems without requiring any of these constraints on occluding bodies in the scene. The method presented optimizes the calibration of the scene in terms of image-based reprojection error. Simulations are shown which characterize the effect noise has on the system, and experimental verification is performed on complex and cluttered scenes. The main contribution of this paper is the elimination of the requirement of using known occluding bodies in the scene for camera and projector-based structured light system calibration, which has not been extensively studied.

Multi-class classification schemes typically require human input in the form of precise category names or numbers for each example to be annotated - providing this can be impractical for the user when a large (and possibly unknown) number of categories are present. In this paper, we propose a multi-class active learning model that requires only binary (yes/no type) feedback from the user. For instance, given two images the user only has to say whether they belong to the same class or not. We first show the interactive benefits of such a scheme with user experiments. We then propose a Value of Information (VOI)-based active selection algorithm in the binary feedback model. The algorithm iteratively selects image pairs for annotation so as to maximize accuracy, while also minimizing user annotation effort. To our knowledge, this is the first multi-class active learning approach that requires only yes/no inputs. Experiments show that the proposed method can substantially minimize user supervision compared to the traditional training model, on problems with as many as 100 classes. We also demonstrate that the system is robust to real-world issues such as class population imbalance and labeling noise.

Many surveillance and reconnaissance tasks make use of multi-cameras in order to ensure that a particular mission is accomplished. These networks of cameras are useful as they can reduce the cost of human observers, are continuously observant (unlike humans who may fall asleep on the job), and can be implemented for a fairly low cost. However, in many scenarios, it does not make sense or may not be possible to have a fixed camera installation because the surveillance may only be needed for a short duration or it may take too long to do a proper install and observation is needed now. In terms of short terms and immediacy, mobile robots acting as a camera network provide an interesting middle ground. They can be deployed quickly to cover immediate needs, and they can be packed up and moved to another area if needs change. However, as the duration of the mission in which they are used increases, the robotic team will run out of power. This paper addresses some of the issues with keeping a surveillance team active while their batteries drain. Multiple task-reallocation methods are used in conjunction with an analysis of the effects of fixed vs mobile docking stations. Simulations were run requiring the team to provide camera coverage of a group of mobile “pedestrians” moving dynamically through a scene and the results are presented.

Accurate image classification is crucial in many robotics and surveillance applications - for example, a vision system on a robot needs to accurately recognize the objects seen by its camera. Object recognition systems typically need a large amount of training data for satisfactory performance. The problem is particularly acute when many object categories are present. In this paper we present a batch-mode active learning framework for multi-class image classification systems. In active learning, images are to be chosen for interactive labeling, instead of passively accepting training data. Our framework addresses two important issues: i) it handles redundancy between different images which is crucial when batch-mode selection is performed; and ii) we pose batch-selection as a submodular function optimization problem that makes an inherently intractable problem efficient to solve, while having approximation guarantees. We show results on image classification data in which our approach substantially reduces the amount of training required over the baseline.

Small ground robots remain limited in their locomotion capabilities, often prevented from accessing areas restricted by tall obstacles or rough terrain. This paper presents the improved design of a hybrid-locomotion robot made to address this issue. It uses wheels for ground travel and rotary-wing flight for scaling obstacles and flying over rough terrain. The robot's initial design suffered from a number of issues that prevented it from functioning fully, such as overheating motors, inadequate control electronics, and insufficient landing gear. Several improvements have been made to the robot's design to correct these problems. These obstacles, and the solutions implemented in the improved design, have enabled several design principles to be formulated for miniature hybrid-locomotion robots. It is found that hybrid-locomotion vehicles utilizing rotary-wing flight are most useful when the design is optimized for ground mode performance. Collapsibility is necessary in such vehicles to reduce the impact of the helicopter rotor on the size of the ground mode. Finally, since a large number of actions are necessary to propel and transform the robot, integrating multiple functions into each mechanism can reduce the mass of the robot.

Physical constraints that underly the formation of periodic motions can be effectively used to accurately reconstruct the periodic motion from even single camera views. As shown in our earlier work, this reduces to a problem of geometric inference. In this paper, we focus on periodic motions exhibited by humans, which are generally not perfectly periodic, and explore the suitability of the reconstruction techniques in these scenarios. We examine the degree of periodicity of human gait empirically, including the applicability of our motion model. Importantly, we illustrate the usefulness of these techniques by applying them to the task of clinical gait analysis. A computational tool to analyze periodic human motion can prove to be invaluable in medical applications either in terms of assessing deviations from normal patterns or evaluating changes resulting from therapy or other clinical procedures.

We present a method that is suitable for clustering of vehicle trajectories obtained by an automated vision system. We combine ideas from two spectral clustering methods and propose a trajectory-similarity measure based on the Hausdorff distance, with modifications to improve its robustness and account for the fact that trajectories are ordered collections of points. We compare the proposed method with two well-known trajectory-clustering methods on a few real-world data sets.

Cameras are becoming a common tool for automated vision purposes due to their low cost. In an era of growing security concerns, camera surveillance systems have become not only important but also necessary. Algorithms for several tasks such as detecting abandoned objects and tracking people have already been successfully developed. While tracking people is relatively easy, counting people in groups is much more challenging. The mutual occlusions between people in a group make it difficult to provide an exact count. The aim of this work is to present a method of estimating the number of people in group scenarios. Several considerations for counting people are illustrated in this paper, and experimental results of the method are described and discussed.

One of the principal bottlenecks in applying learning techniques to classification problems is the large amount of labeled training data required. Especially for images and video, providing training data is very expensive in terms of human time and effort. In this paper we propose an active learning approach to tackle the problem. Instead of passively accepting random training examples, the active learning algorithm iteratively selects unlabeled examples for the user to label, so that human effort is focused on labeling the most “useful” examples. Our method relies on the idea of uncertainty sampling, in which the algorithm selects unlabeled examples that it finds hardest to classify. Specifically, we propose an uncertainty measure that generalizes margin-based uncertainty to the multi-class case and is easy to compute, so that active learning can handle a large number of classes and large data sizes efficiently. We demonstrate results for letter and digit recognition on datasets from the UCI repository, object recognition results on the Caltech-101 dataset, and scene categorization results on a dataset of 13 natural scene categories. The proposed method gives large reductions in the number of training examples required over random selection to achieve similar classification accuracy, with little computational overhead.

Accurate localization of landmarks in the vicinity of a robot is a first step towards solving the SLAM problem. In this work, we propose algorithms to accurately estimate the 3D location of the landmarks from the robot only from a single image taken from its on board camera. Our approach differs from previous efforts in this domain in that it first reconstructs accurately the 3D environment from a single image, then it defines a coordinate system over the environment, and later it performs the desired localization with respect to this coordinate system using the environment's features. The ground plane from the given image is accurately estimated and this precedes segmentation of the image into ground and vertical regions. A Markov Random Field (MRF) based 3D reconstruction is performed to build an approximate depth map of the given image. This map is robust against texture variations due to shadows, terrain differences, etc. A texture segmentation algorithm is also applied to determine the ground plane accurately. Once the ground plane is estimated, we use the respective camera's intrinsic and extrinsic calibration information to calculate accurate 3D information about the features in the scene.

The desire for a high mobility-to-size ratio in mobile robots has led to the exploration of many new methods of locomotion, one of which is tumbling. We believe that tumbling has a great potential to produce high mobility-to-size ratios in miniature mobile robots. In this paper we discuss tumbling locomotion and introduce the Adelopod, a small two-armed tumbling robot recently developed at the University of Minnesota Center for Distributed Robotics. The Adelopod achieves high mobility with low hardware complexity, making it a desirable addition to any heterogeneous team of robots.

In order to increase the effectiveness of robotic systems, the robots must be able to negotiate a variety of terrains. In urban environments this task becomes challenging as environments built for humans often have impediments such as stairs and thresholds which may be difficult or impossible for wheeled or tracked vehicles to go over. This is further complicated in search and rescue scenarios where debris and rubble from collapsed structures may further complicate the environment. In order to address these limitations a novel robotic platform, the Loper, has been developed. The Loper utilizes a Tri-lobe wheel and a compliant chassis in order to traverse difficult terrain. In addition, multiple gait configurations of the Tri-lobe wheel enable the Loper to overcome a variety of obstacles, both indoors and out.

Effective robotic autonomy requires an accurate estimation of the remaining power that the robot is carrying. There are a number of methods to estimate the remaining capacity of the robot's batteries. The accuracy of these approaches varies depending upon the chemical composition of the batteries as well as the means by which the monitoring is conducted. In this paper, an overview of various battery capacity estimation methodologies will be presented as well as a discussion of a specific implementation utilized on robotic platforms developed at the Center for Distributed Robotics at the University of Minnesota. This approach involves the novel use of fuel-gauging electronics for mobile robotic platforms. Experiments were conducted to show the accuracy of this method for both charge and discharge of the polymer Lithium-ion batteries in use.

This paper presents a new vision-based leader-follower formation algorithm where the leader's trajectory is unknown to the robots which are following. Formation schemes in straight lines and diagonal formations are introduced which are both stable and observable in the presence of limited views. The algorithms are novel since they only use local image measurements through a pinhole camera to estimate the leader's position. This approach does not require specialized markings nor extensive robot communications. The algorithms are also decentralized. We apply an input-output feedback linearization for system stability and utilize an Extended Kalman Filter (EKF) for estimation. Simulations illustrate how the proposed formation controls work. Real experiments utilizing multiple miniature robots are also presented and illustrate the challenges associated with noisy images in real-world applications.

The majority of work related to grasp planning has centered on the understanding of what constitutes a good grasp. However, to reach a good grasp we must first find the relative position of the gripper, its approach vector, and finger configuration. This search problem is the focus of our paper. We propose an on-line method that uses pairwise shape descriptors to quickly find good alignments between the gripper contact surface and the target. Having found a good fit, we then evaluate how alignment quality relates to grasp quality and what can be done to speed up the exploration of the DOF space.

This paper presents a mathematical basis for judging the quality of camera and projector placement in 3D for structured light systems. Two important quality metrics are considered: visibility, which measures how much of the target object is visible; and scale, which measures the error in detecting the visible portions. A novel method for computing each of these metrics is presented. An example is discussed which demonstrates use of these two metrics. The proposed techniques have direct applicability to the task of monitoring patient safety for radiation therapy applications.

Autonomous estimation of the altitude of an Unmanned Aerial Vehicle (UAV) is extremely important when dealing with flight maneuvers like landing, steady flight, etc. Vision based techniques for solving this problem have been underutilized. In this paper, we propose a new algorithm to estimate the altitude of a UAV from top-down aerial images taken from a single on-board camera. We use a semi-supervised machine learning approach to solve the problem. The basic idea of our technique is to learn the mapping between the texture information contained in an image to a possible altitude value. We learn an over complete sparse basis set from a corpus of unlabeled images capturing the texture variations. This is followed by regression of this basis set against a training set of altitudes. Finally, a spatio-temporal Markov Random Field is modeled over the altitudes in test images, which is maximized over the posterior distribution using the MAP estimate by solving a quadratic optimization problem with L1 regularity constraints. The method is evaluated in a laboratory setting with a real helicopter and is found to provide promising results with sufficiently fast turnaround time.

Robotic teams are often proposed for solving a number of problems, ranging from exploring unknown environments to monitoring areas for security or environmental contamination. These teams are composed of individual robots which may lack the capabilities to complete a task on their own. One critical capability required by teams regardless of the mission is the ability to have sufficient battery life to remain active for the duration of the mission. We present an approach for maintaining battery life by developing a hierarchical team composed of deployable robots and docking stations. Unlike other approaches, the approach presented here focuses on docking stations supporting multiple deployed robots simultaneously. In order to do so the docking stations must continually optimize their locations with respect to the robots in need of service. Discussion of the optimization is presented, along with simulation in multiple environments to illustrate the scalability of the approach to large robotic teams. The on-going transition of this algorithm to actual hardware is also discussed.

Cameras are becoming a common tool for automated vision purposes due to their low cost. Many surveillance and inspection systems include cameras as their sensor of choice. How useful these camera systems are is very dependent upon the positioning of the cameras. This is especially true if the cameras are to be used in automated systems as a beneficial camera placement will simplify image processing operations. Therefore, a reliable positioning algorithm can lower the processing requirements of the system. In this paper several considerations for improving camera placement are investigated with the goal of developing a general algorithm that can be applied to a variety of systems. This paper presents this algorithm for placement problem in the context of computer vision and robotics. Simulated results of our method are then shown and discussed, along with an outline of future work.

Tumbling is an exciting new area of robotic locomotion that takes advantage of ground-body interactions to achieve rich motions with minimal hardware complexity. The increased mobility of tumbling robots, however, comes at the price of increased control complexity. In this paper, we propose a novel method to handle the issues of tumbling locomotion which takes the problem and separates into locally independent subproblems. Our approach provides an intuitive geometric solution to the tumbling control problem without sacrificing performance. We provide a running example throughout the paper to help solidify the ideas presented.

Mobility in small ground robots is often improved by novel mechanisms or wheel designs. This can be successful when navigating rough terrain or climbing small obstacles. However, few such robots are capable of scaling obstacles of arbitrary height or traversing all types of terrain. This paper presents a concept for a miniature robot that combines wheeled ground locomotion with rotary-wing flight capabilities, which has the potential to offer the best features of both helicopters and ground vehicles while addressing the aforementioned challenges to mobility.

The forward moving target segmentation from a moving platform with a pinhole camera is an important and relatively unexplored problem in visual tracking. This paper proposes a novel segmentation algorithm for extracting a moving target when using a moving platform (that follows a similar trajectory and has a camera that is not calibrated for the particular scene). When the target has unpredictable motions, we are unable to model it and the pertinent backgrounds are very different. We introduce a new entropy-based clustering algorithm in order to find a bounding box representing the target. A target model based on graph representation is used for matching the moving target. To demonstrate the robust target segmentation scheme, we apply the method to a team of miniature robots (the Explorers developed at the University of Minnesota) in real tracking missions.

Periodicity has been recognized as an important cue for tasks like activity recognition and gait analysis. However, most existing techniques analyze periodic motions only in image coordinates, making them very dependent on the viewing angle. In this paper we propose a new technique for reconstructing periodic point trajectories in 3D given only their apparent trajectories in image coordinates from a single stationary camera. We show that this reconstruction is possible without performing a costly gradient descent-type optimization, and is based only on a single SVD. This new algorithm is shown to accurately reconstruct natural human motions, allowing them to be compared in 3D world coordinates, independent of the angle from which they were originally viewed.

In this paper, we consider the problem of localizing a projectile in 3D based on its apparent motion in a stationary monocular view. A thorough theoretical analysis is developed, from which we establish the minimum conditions for the existence of a unique solution. The theoretical results obtained have important implications for applications involving projectile motion. A robust, nonlinear optimization-based formulation is proposed, and the use of a local optimization method is justified by detailed examination of the local convexity structure of the cost function. The potential of this approach is validated by experimental results.

A key research issue in activity recognition in real-world applications, such as in intelligent transportation systems (ITS), is to automatically learn robust models of activities that require minimal human training. In this paper, we contribute a novel approach for learning sequenced spatiotemporal activities in outdoor traffic intersections. Concretely, by representing the activities as sequences of actions, we contribute a semisupervised learning algorithm that learns activities as complete stochastic context-free grammars (SCFGs), namely, the grammar structure and the parameters. Our approach has been implemented and tested on real-world scenes, and we present experimental results of the grammar learning and activity recognition applied to data collection and traffic monitoring applications using video data.

We present a vehicle tracker for automatic data collection at intersections and freeways. Key features of our approach are the ability to measure the real-world state of tracked vehicles, to hand off targets between cameras, and to track simultaneously from multiple views for improved handling of occlusions and scene clutter. Further, the proposed measurement model supports both full and partial measurements from any number of views, which are seamlessly fused by the estimation procedure. State constraints are incorporated in the tracking method, and issues caused by stop-and-go traffic and turning vehicles are addressed. The use of constrained estimation, the support of partial measurements, and the multi- view capability represents a significant improvement of our past efforts in automating the tracking of vehicles in challenging situations.

The use of cameras is becoming more prevalent by the day owing to the variety of applications they have. However, each application requires a specific placement of the cameras for best performance. Therefore, determining this placement has been a problem of much work in the field of computer vision. However, most of the current approaches deal with a scene that does not change once the cameras have been placed. In this paper a system is developed and successfully tested that can not only distribute cameras around a scene such that the observability is improved, but can reorganize the cameras if the pattern of activity within a scene changes with time. The cameras can be distributed in either two or three dimensional space, and this was tested with the use of mobile robotic cameras and stationary cameras.

The purpose of this paper is to describe the Loper, a multi-purpose robotic platform under development at the University of Minnesota's Center for Distributed Robotics. Leper's unique Tri-lobe wheel design and highly compliant chassis make the platform especially suited for overcoming many of the challenges associated with search operations in urban settings. The mechanically simple design and use of commercially available components make Loper easily maintainable. The platform also features long operational time, onboard sensor processing, dedicated motion control, and four reconfigurable sensor bays.

We propose a novel online framework for detecting moving shadows in video sequences using statistical learning techniques. In this framework, support vector machines are applied to obtain a classifier that can differentiate between moving shadows and other foreground objects. The co-training algorithm of Blum and Mitchell is then used in an online setting to improve accuracy with the help of unlabeled data. We evaluate the concept of co-training and show its viability even when explicit assumptions made by the algorithm are not satisfied. Thus, given a small random set of labeled examples (in our application domain, shadow and foreground), the system gives encouraging generalization performance using a semi-supervised approach. In dynamic environments such as those induced by robot motion, the view changes significantly and traditional algorithms do not work well. Our method can handle such changing conditions by adapting online using a semi-supervised approach.

The desire for a high mobility-to-size ratio in mobile robots has led to the exploration of many new methods of locomotion, one of which is tumbling. To the authors' knowledge, there are very few tumbling robots in existence and no formalized methods for their control. In this paper we begin addressing these issues by presenting an approach for deriving motion primitives for tumbling robots. We apply our method to the specific case of a two-armed tumbling robot and include the final derived motion primitives. Additionally we discuss in general the motion of tumbling robots and introduce 3 useful gaits derived from the resulting primitives of our approach.

We propose a novel adaptive technique for detecting moving shadows and distinguishing them from moving objects in video sequences. Most methods for detecting shadows work in a static setting with significant human input. To remove these limitations, we propose a more general semi-supervised learning technique to tackle the problem. First, we exploit characteristic differences in color and edges in the video frames to come up with a set of features useful for classification. Second, we use a learning technique that employs Support Vector Machines and the Co-training algorithm, that relies on a small set of human-labeled data. We observe a surprising phenomenon that Co-training can counter the effects of changing underlying probability distributions in the feature space. From the standpoint of detecting shadows, once deployed, the proposed method can dynamically adapt to varying conditions without any manual intervention, and performs better classification than previous methods on static and dynamic environments alike. The strengths of the proposed technique are the small quantity of human labeled data required, and the ability to adapt automatically to changing scene conditions.

Automatic detection of dynamic events in video sequences has a variety of applications including visual surveillance and monitoring, video highlight extraction, intelligent transportation systems, video summarization, and many more. Learning an accurate description of the various events in real-world scenes is challenging owing to the limited user-labeled data as well as the large variations in the pattern of the events. Pattern differences arise either due to the nature of the events themselves such as the spatio-temporal events or due to missing or ambiguous data interpretation using computer vision methods. In this work, we introduce a novel method for representing and classifying events in video sequences using reversible context-free grammars. The grammars are learned using a semi-supervised learning method. More concretely, by using the classification entropy as a heuristic cost function, the grammars are iteratively learned using a search method. Experimental results demonstrating the efficacy of the learning algorithm and the event detection method applied to traffic video sequences are presented.

In this paper, we propose a multi-level shadow identification scheme which is generally applicable without restrictions on the number of light sources, illumination conditions, surface orientations, and object sizes. In the first level, we use a background segmentation technique to identify foreground regions which include moving shadows. In the second step, pixel-based decisions are made by comparing the current frame with the background model to distinguish between shadows and actual foreground. In the third step, this result is improved using blob-level reasoning which works on geometric constraints of identified shadow and foreground blobs. Results on various indoor and outdoor sequences under different illumination conditions show the success of the proposed approach.

The eROSI (educational, research-oriented, sensing, inexpensive) robot is a robotic platform designed by the Center for Distributed Robotics at the University of Minnesota. It was designed for two purposes; an educational platform and a distributed robotics platform. These two purposes require very different and oftentimes contradictory abilities. As an educational platform, it must be easy to use and program for. It must also be non-intimidating to students who may not have much experience with robotics. As a research platform, it must be powerful enough to run useful behaviors, and expandable with new sensors. These two different purposes have driven the design of the eROSI robot through three generations of development. The first two generations have been tested and evaluated in terms of its two intended purposes. The feedback generated was then used to modify the design of the robot to better accommodate its intended purposes. The eROSI robot is still an ongoing project and aims to create a better platform for both research and education.

Many researchers study the search of unknown areas with teams of cooperating robots. Robot cooperation often relies upon shared representations built from shared information. When robots share information, they inadvertently share the error associated with that information. In this research, we realistically assume that all position-based data has some amount of localization error. We model the propagation of this error in cooperative search. We conjecture that propagation of error differs based on whether search targets are shared or locally discovered. Search errors due to inaccurate position estimates are quantified and compared in both approaches via simulations.

Multirobot coordination techniques range from sharing state and intent information to explicit coordination strategies. The choice of cooperation strategy can affect task performance and resource utilization. This paper investigates the use of local search queues, rather than globally shared search queues. This work shows this change improves search in terms of interference and communications loads, two key factors in multirobot search, without significantly impacting time-to-search and coverage metrics.

We present a novel technique for the detection of thrown objects and other free-flying bodies in video sequences. Our method runs in real-time and was designed to be used as a component in a deployed surveillance system. We detect regions of interesting motion that fit certain size, compactness and speed criteria, and use the expectation maximization algorithm to detect objects on parabolic trajectories over a short time window. The system was shown to successfully detect thrown objects of various sizes in a large test set of indoor and outdoor videos.

We address the problem of determining and ob- taining the visibility of a moving target from multiple angles using a mobile robot. The pursuer uses a modified form of pursuit curve in order to reach the aim point (or multiple aim points) which is at a pre-determined distance and angle from the target. Unlike existing works that track a target to localize it, or follow a target by keeping it in the current frame of view generally from behind the target, the focus of our work is on an efficient pursuit of the target such that a certain set of images of it from various orientations can be acquired. We present a method to approach a target in motion from various angles and an analysis of whether the task completion is possible in the cases of a target performing a random walk and an adversarial target.

The University of Minnesota Center for Distributed Robotics and the Digital Technology Center hosted the second annual Technology Day Camp, a week long camp targeting underrepresented students such as girls, African Americans, and Hispanics from the Twin Cities metro area. Students were surveyed prior to the camp revealing a strong proclivity towards math and science, but a resistance to the perceived lifestyle of computer scientists. With an emphasis on the college experience, building hardware, creating software, and interacting with robots, the camp implemented proposed changes from the previous year as well as new activities with a special emphasis on robotics. The successes and failures are discussed in an effort to provide insight for organizations hosting similar programs, new research questions are presented, and a materials list is provided.

This paper presents a novel technique for camera tampering detection. It is implemented in real-time and was developed for use in surveillance and security applications. This method identifies camera tampering by detecting large differences between older frames of video and more recent frames. A buffer of incoming video frames is kept and three different measures of image dissimilarity are used to compare the frames. After normalization, a set of conditions is tested to decide if camera tampering has occurred. The effects of adjusting the internal parameters of the algorithm are examined. The performance of this method is shown to be extremely favorable in real-world settings.

This work presents a method for detecting abandoned objects in real-world conditions. The method presented here addresses the online and real time aspects of such systems, utilizes logic to differentiate between abandoned objects and stationary people, and is robust to temporary occlusion of potential abandoned objects. The capacity to not detect still people as abandoned objects is a major aspect that differentiates this work from others in the literature. Results are presented on 3 hours 36 minutes of footage over four videos representing both sparsely and densely populated real-world situations, also differentiating this work from others in the literature

Within the fields of law enforcement and urban search and rescue, there is always a need to obtain information from areas that may be hard to reach or unsafe to enter. One method of obtaining this reconnaissance information is to deploy a robot as a projectile. This may be accomplished with mechanical aids or simply by throwing the robot manually. This rapid deployment method has the ability to attain locations inaccessible to other technologies. The miniature nature of the presented design has the ability to operate discretely and avoid detection, making it desirable for law enforcement. In the case of urban search and rescue, the diminutive form minimizes the impact on potentially unsound structures. During the course of deployment, unexpected impacts and drops are inevitable, generating a need for an impact invariant design. Design decisions to create this system are presented, and experimental validation of design aspects is discussed

Women and minorities are underrepresented in the IT field at the high school, university, and industry levels. Efforts to address this imbalance are often too late to solve underlying problems such as perceived ineptitude and actual inexperience. By designing and hosting a program for these underrepresented students in the middle grades, the Center for Distributed Robotics at the University of Minnesota hopes to establish a successful annual robotics day camp which would inspire both women and minorities to pursue careers in technology. Detailed accounts of the goals and methodology are provided. Initial survey results reveal a very positive response from the campers as well as strengths and weaknesses which would be useful in designing or refining similar camps

Distributed robotic teams have long been touted as potential means to avoid sending humans into harmful situations. The ability of robotic teams to operate for extended periods without the fatigue human teams can experience, coupled with the ability to transport a variety of sensing and manipulation equipment and reducing costs of operation make them an attractive solution. Limited sensing capabilities, power, and mobility of individual robotic platforms can be overcome by forming teams of heterogeneous robots. This work addresses the power limitations associated with individual robots, which have finite amounts of power, and thus limited operational lifetimes. This work presents a method by which mobile docking stations can optimize their locations in order to maximize the power available to the deployed robots. Simulated results are presented in which teams of docking stations continuously recover and recharge a much larger team of deployed robots. It is assumed that the deployed robots are able to maintain a communication link between themselves and the docking station. This communication link is used to provide position information and available power to the docking stations. The communication link may be direct from the robot to docking station or it may require the use of ad hoc routing through other deployed robots

Robust motion recovery in tracking multiple targets using image features is affected by difficulties in obtaining good correspondences over long sequences. Difficulties are introduced by occlusions, scale changes, as well as disappearance of features with the rotation of targets. In this work, we describe an adaptive geometric template-based method for robust motion recovery from features. A geometric template consists of nodes containing salient features (e.g., corner features). The spatial configuration of the features is modeled using a spanning tree. This paper makes the following two contributions: (i) an adaptive geometric template to model the varying number of features on a target, and (U) an iterative data association method for the features based on the uncertainties in the estimated template structure in conjunction with its individual features. We present experimental results for tracking multiple targets over long outdoor image sequences with multiple persistent occlusions. A comparison of the results of the data association method with a standard Mahalanobis distance gating applied to individual features is also presented

We address the problem of automatically learning the layout of a traffic intersection from trajectories of vehicles obtained by a vision tracking system. We present a similarity measure which is suitable for use with spectral clustering in problems that emphasize spatial distinctions between vehicle trajectories. The robustness of the method to small perturbations and its sensitivity to the choice of parameters are evaluated using real-world data

Large scale robotic teams are capable of working independently or cooperatively to carry out a variety of missions. However, for large teams of robots to function for extended periods of time, the individual members of a team must be able to generate or find energy to re-supply themselves. One approach to providing power for a robotic team is to couple larger systems with significant energy reserves so that the smaller systems can be recharged directly from the larger. This paper presents an implementation of such an approach. Here, a modular docking station is given locomotion through the cooperation of two larger robots. The docking station is capable of transporting, deploying, retrieving, and recharging many smaller robots. The kinematic model which will govern the cooperation of the maneuvering robots and will be used to develop control is presented and discussed. The design of the individual bays of the docking station and how they facilitate the deployment, recovery, and recharge of the smaller robots is also presented. The development of this system makes possible a number of applications, including autonomous long-term environmental monitoring and reconnaissance in various locations

/sup T/he University of Minnesota's Scout is a small cylindrical robot capable of rolling and jumping. Models describing the robot's motion are developed. These models can be employed for motion prediction and simulation. The results suggest that the determining factor of the Scout's behavior is the length of the winch cable.

The performance of computer vision systems for measurement, surveillance, reconstruction, gait recognition, and many other applications, depends heavily on the placement of cameras observing the scene. This work addresses the question of the optimal placement of cameras to maximize the performance of real-world vision systems in a variety of applications. Specifically, our goal is to optimize the aggregate observability of the tasks being performed by the subjects in an area. We develop a general analytical formulation of the observation problem, in terms of the statistics of the motion in the scene and the total resolution of the observed actions that is applicable to many observation tasks and multi-camera systems. An optimization approach is used to find the internal and external (mounting position and orientation) camera parameters that optimize the observation criteria. We demonstrate the method for multi-camera systems in real-world monitoring applications, both indoor and outdoor.

Monitoring traffic intersections in real-time and predicting possible collisions is an important first step towards building an early collision warning system. We present the general vision methods used in a system addressing this problem and describe the practical adaptations necessary to achieve real-time performance. A novel method for three-dimensional vehicle size estimation is presented. We also describe a method for target localization in real-world coordinates which allows for sequential incorporation of measurements from multiple cameras into a single target's state vector. Additionally, a fast implementation of a false-positive reduction method for the foreground pixel masks is developed. Finally, a low-overhead collision prediction algorithm using the time-as-axis paradigm is presented. The proposed system was able to perform in real-time on videos of quarter-VGA (320/spl times/240) resolution. The errors in target position and dimension estimates in a test video sequence are quantified.

The performance of systems for human activity recognition depends heavily on the placement of cameras observing the scene. This work addresses the question of the optimal placement of cameras to maximize the performance of these types of recognition tasks. Specifically, our goal is to optimize the quality of the joint observability of the tasks being performed by the subjects in an area. We develop a general analytical formulation of the observation problem, in terms of the statistics of the motion in the scene and the total resolution of the observed actions that is applicable to many observation tasks and multi-camera systems. A nonlinear optimization approach is used to find the internal and external (mounting position and orientation) camera parameters that optimize the recognition criteria. In these experiments, a single camera is repositioned using a mobile robot. Initial results for the problem of human activity recognition are presented.

The Research Oriented Sensing Inexpensive (ROSI) robot is a new, highly capable mobile sensor node designed to enable collaborative applications. Not only capable of two-way communication, ROSI robots also possess a full suite of traditional small scale sensors which can be augmented with visual information from a digital camera via a semi-dedicated onboard processor. This is accomplished in the same form factor of the original Scouts using commodity components for a low cost. This work details an approach to localizing robots with respect to each other using a collaborative sensing mechanism. Determination of relative angles, overlapping field of view and direction are explored.

Monitoring traffic intersections in real time and predicting possible collisions is an important first step towards building an early collision-warning system. We present a vision-based system addressing this problem and describe the practical adaptations necessary to achieve real-time performance. Innovative low-overhead collision-prediction algorithms (such as the one using the time-as-axis paradigm) are presented. The proposed system was able to perform successfully in real time on videos of quarter-video graphics array (VGA) (320 /spl times/ 240) resolution under various weather conditions. The errors in target position and dimension estimates in a test video sequence are quantified and several experimental results are presented.

This paper presents a vision-based method to automatically detect individuals loitering about inner-city bus stops. Using a stationary camera view of a bus stop, pedestrians are segmented and tracked throughout the scene. The system takes snapshots of individuals when a clean, nonobstructed view of a pedestrian is found. The snapshots are then used to classify the individual images into a database, using an appearance-based method. The features used to correlate individual images are based on short-term biometrics, which are changeable but stay valid for short periods of time; this system uses clothing color. A linear discriminant method is applied to the color information to enhance the differences and minimize similarities between the different individuals in the feature space. To determine if a given individual is loitering, time stamps collected with the snapshots in their corresponding database class can be used to judge how long an individual has been present. An experiment was performed using a 30-min video of a busy bus stop with six individuals loitering about it. Results show that the system successfully classifies images of all six individuals as loitering.

The problem of vision-guided control of miniature mobile robots is investigated. Untethered mobile robots with small physical dimensions of around 10 cm or less do not permit powerful onboard computers because of size and power constraints. These challenges have, in the past, reduced the functionality of such devices to that of a complex remote control vehicle with fancy sensors. With the help of a computationally more powerful entity such as a larger companion robot, the control loop can be closed. Using the miniature robot's video transmission or that of an observer to localize it in the world, control commands can be computed and relayed to the inept robot. The result is a system that exhibits autonomous capabilities. The framework presented here solves the problem of climbing stairs with the miniature Scout robot. The robot's unique locomotion mode, the jump, is employed to hop one step at a time. Methods for externally tracking the Scout are developed. A large number of real-world experiments are conducted and the results discussed.

We propose a motion recognition strategy that represents a videoclip as a set of filtered images, each of which encodes a short period of motion history. Given a set of videoclips whose motion types are known, a filtered image classifier is built using support vector machines. In offline classification, the label of a test videoclip is obtained by applying majority voting over its filtered images. In online classification, the most probable type of action at an instance is determined by applying the majority voting over the most recent filtered images, which are within a sliding window. The effectiveness of this strategy was demonstrated on real datasets where the videoclips were recorded using a fixed camera whose optical axis is perpendicular to the person's trajectory. In offline recognition, the proposed strategy outperforms a principal component analysis based recognition algorithm. In online recognition, the proposed strategy cannot only classify motions correctly and identify the transition between different types of motions, but also identify the existence of an unknown motion type. This latter capability and the efficiency of the proposed strategy make it possible to create a real-time motion recognition system that can not only make classifications in real-time, but also learn new types of actions and recognize them in the future.

The Scout is a miniature robot capable of performing surveillance and reconnaissance applications. However, the small size of the Scout limits the on-board processing capability and thus requires that the primary sensing modality (vision) be processed remotely. This creates a dependence upon a video channel which can become a severe bottleneck when attempting to use large numbers of robots. This paper discusses the newest generation of the Scout, the COTS 3.0, which attempts to reduce the effect of the communication bottleneck by using more local sensing capabilities to achieve a degree of autonomous functionality. Several experiments showing the durability of the new COTS Scout and the autonomous capability are presented.

We introduce a vision-based system to monitor for suspicious human activities at a bus stop. The system currently examines for drug dealing activity. To accomplish this goal, the system must measure how long individuals loiter around the bus stop. To facilitate this, the system must track individuals from the video feed, identify them, and keep a record of how long they spend at the bus stop. The system is broken into three distinct portions: background subtraction, object tracking, and human recognition. The background subtraction and object tracking modules use off-the-shelf algorithms and are shown to work well following people as they walk around a bus stop. The human recognition module segments the image of an individual into three portions corresponding to the head, torso, and legs. Using the median color of each of these regions, two people can be quickly compared to see if they are the same person.

This paper compares the performances of several algorithms that address the problem of Simultaneous Localization and Mapping (SLAM) for the case of very small, resource-limited robots. These robots have poor odometry and can typically only carry a single monocular camera. These algorithms do not make the typical SLAM assumption that metric distance/bearing information to landmarks is available. Instead, the robot registers a distinctive sensor "signature", based on its current location, which is used to match robot positions. The performances of a physics-inspired maximum likelihood (ML) estimator, the iterated form of the Extended Kalman Filter (IEKF), and a batch-processed linearized ML estimator are compared under various odometric noise models.

Classical dynamic contours (snakes) suffer from their inability to properly address the crossover problem. In the closed contour case, a part of the snake inverts and expands without bounds. This paper proposes a novel approach to handle inversions by splitting the contour at the points of self-intersection. Separated parts that are inverted are destroyed while the remaining parts are used to seed new snakes. Collisions of snakes cause them to merge. The probability of crossovers is further reduced by an efficient reparameterization procedure. The method is successfully applied to the challenging task of tracking the globule in a motion lamp.

This paper presents a camera-based system for tracking vehicles at outdoor scenes such as traffic intersections. Two different computer vision modalities, namely, the connected regions obtained through region segmentation and color analysis, obtained through a mean-shift tracking procedure are combined sequentially using an extended Kalman filter to provide the position of each target. Data association ambiguities arising in blob tracking are handled by using oriented bounding boxes and a joint probabilistic data association filter. We show that the above tracking formulation can provide reasonable tracking despite the stop-and-go motion of vehicles and clutter in traffic intersections.

This paper describes a dual-camera system intended to accomplish the task of vision-based activity recognition at multiple resolutions. The system is comprised of a wide-angle, fixed field of view camera coupled with a computer-controlled pan/tilt/zoom-lens camera to make detailed measurements of people for activity recognition applications. We demonstrate the use of the system in both indoor and outdoor environments.

We present a simple and efficient algorithm for determining the position of static occluding bodies within a scene viewed by one or more static cameras. All information about the occluding bodies is derived from the perimeter of a figure moving through the scene. Once the positions of the occlusions are learned, successful reasoning about the observability of future figures in the scene is demonstrated. The method is extended to derive an estimate of the 3D position of the occluding bodies from multiple views. Several experimental results are described.

In this paper, we address the problem of recovering the intrinsic and extrinsic parameters of a camera or a group of cameras in a setting overlooking a traffic scene. Unlike many other settings, conventional camera calibration techniques are not applicable in this case. We present a method that uses certain geometric primitives commonly found in traffic scenes in order to recover calibration parameters. These primitives provide needed redundancy and are weighted depending on the significance of there corresponding image features. We show experimentally that these primitives are capable of achieving accurate results suitable for most traffic monitoring applications.

The paper deals with the problem of classification of human activities from video as one way of performing activity monitoring. Our approach uses motion features that are computed very efficiently and subsequently projected into a lower dimension space where matching is performed. Each action is represented as a manifold in this lower dimension space and matching is done by comparing these manifolds. To demonstrate the effectiveness of this approach, it was used on a large data set of similar actions, each performed by many different actors. Classification results are accurate and show that this approach can handle many challenges such as variations in performers' physical attributes, color of clothing, and style of motion. An important result is that the recovery of three-dimensional properties of a moving person, or even two-dimensional tracking of the person's limbs, is not a necessary step that must precede action recognition.

When designing a mobile robotic team, an engineer is faced with many design choices. This paper discusses the design of a team consisting of two different models of robots with significantly different sensing and control capabilities intended to accomplish a similar task. Two new robotic platforms, the COTS Scout and the MegaScout are described along with their respective design considerations.

To safely and efficiently guide search and rescue operations in disaster areas, gathering of relevant information such as the locations of victims, must occur swiftly. Using the concept of repellent virtual pheromones inspired by insect colony coordination behaviors, miniature robots can be quickly dispersed to survey a disaster site. Assisted by visual servoing, dispersion of the miniature robots can quickly cover an area. An external observer such as another robot or an overhead camera is brought into the control loop to provide each miniature robot estimations of the positions of all the other nearby robots in the robotic team. Each robot can then move away from the other nearby robots, resulting in the robot collective swiftly dispersing through the local area. The technique has been implemented using the miniature scout robots, developed by the Center for Distributed Robotics at the University of Minnesota, which are well-suited to surveillance and reconnaissance missions.

This paper addresses the problem of localization and map construction by a mobile robot in an indoor environment. Instead of trying to build high-fidelity geometric maps, we focus on constructing topological maps, as they are lees sensitive to poor odometry estimates and position errors. We propose a method for incrementally building topological maps for a robot, which uses a panoramic camera to obtain images at various locations along its path and uses the features it tracks in the images to update the topological map. The method is very general and does not require the environment to have uniquely distinctive features.

The University of Minnesota's Scout robot is a small cylindrical robot capable of rolling and jumping. In this paper, models describing the robot's motion under its mode of actuation are developed. These models can be employed for Scout motion prediction and simulation. The models suggest that the determining factor of the Scout's behavior is the length of the winch cable. The validity of the models is verified with empirical data.

In this paper, we introduce a novel method for employing image-based rendering to extend the range of use of human motion recognition systems. We demonstrate the use of image-based rendering to generate additional training sets for view-dependent human motion recognition systems. Input views orthogonal to the direction of motion are created automatically to construct the proper view from a combination of non-orthogonal views taken from several cameras. To extend motion recognition systems, image-based rendering can be utilized in two ways: (i) to generate additional training sets for these systems containing a large number of non-orthogonal views, and (ii) to generate orthogonal views (the views those systems are trained to recognize) from a combination of non-orthogonal views taken from several cameras. In this case, image-based rendering is used to generate views orthogonal to the mean direction of motion. We tested the method using an existing view-dependent human motion recognition system on two different sequences of motion, and promising initial results were obtained.

The Scouts, developed at the University of Minnesota, are miniature robots designed mainly for reconnaissance and surveillance tasks. The Scout's small size and multiple mobility and sensing modes allow it to efficiently navigate and carry out certain missions in indoor environments and on relatively smooth surfaces. However, like many other small-sized robots, its miniature size and limited battery life become a bottleneck when it comes to cover long distances, particularly on rough, outdoor surfaces. In order to address this issue, we present a method to carry the Scouts to and from their main mission locations by adding a larger robot, a Pioneer 2-AT, to the team. The Pioneer carries a team of Scouts in a color marked box that it holds with its grippers. The Scouts jump out of the box, carry out their mission, and autonomously find the box and jump back in.

This paper addresses the problem of simultaneous localization and mapping (SLAM) for the case of very small, resource-limited robots which have poor odometry and can typically only carry a single monocular camera. We propose a modification to the standard SLAM algorithm in which the assumption that the robots can obtain metric distance/bearing information to landmarks is relaxed. Instead, the robot registers a distinctive sensor "signature", based on its current location, which is used to match robot positions. In our formulation of this non-linear estimation problem, we infer implicit position measurements from an image recognition algorithm. The iterated form of the extended Kalman filter (IEKF) is employed to process all measurements.

The goal of this project is to monitor activities at traffic intersections for detecting/predicting situations that may lead to accidents. Some of the key elements for robust intersection monitoring are camera calibration, motion tracking, incident detection, etc. In this paper, we consider the motion-tracking problem. A multilevel tracking approach using Kalman filter is presented for tracking vehicles and pedestrians at intersections. The approach combines low-level image-based blob tracking with high-level Kalman filtering for position and shape estimation. An intermediate occlusion-reasoning module serves the purpose of detecting occlusions and filtering relevant measurements. Motion segmentation is performed by using a mixture of Gaussian models which helps us achieve fairly reliable tracking in a variety of complex outdoor scenes. A visualization module is also presented. This module is very useful for visualizing the results of the tracker and serves as a platform for the incident detection module.

When a distributed robotic system is assigned to perform reconnaissance or surveillance, restrictions inherent to the design of an individual robot limit the system's performance in certain environments. Finding an ideal portable robotic platform capable of deploying and returning information in spatially restrictive areas is not a simple task. The Scout robot, developed at the University of Minnesota, is a viable robotic platform for these types of missions. The small form factor of the Scout allows for deployment, placement, and concealment of a team of robots equipped with a variety of sensory packages. However, the design of the Scout requires a compromise in power, sensor types, locomotion, and size; together these factors prevent an individual Scout from operating ideally in some environments. Several attempts to address these deficiencies have been implemented and are discussed. Among the prototype solutions are actuating wheels, allowing the Scout to increase ground clearance in varying terrains, a grappling hook enabling the Scout to obtain a position of elevated observation, and infrared emitters to facilitate low light operation.

Presents a method for localizing miniature mobile robots (Scouts) using dynamic contours. An observer robot with a camera follows the miniature robot as it moves and jumps in the workspace. Dynamic contours are very effective in tracking the fast accelerations and decelerations of the Scout robot. We show initial experimental results with particular emphasis on the task of monitoring a Scout during jumps.

Small scale distributed robotic systems are ideal for tasks that larger, more expensive robots may not be able to undertake such as surveillance or inspection in enclosed areas. Small scale robots also have the advantage of being easily portable to an area of interest. However, the advantages related to the small size and portability cause an increase in the complexity of development. In addition, the more pronounced effects of interacting with the environment in terms of mobility and communications lead to robotic systems of limited functionality. In order to address such deficiencies, several novel developments and improvements have been made to the Scout robot, which will be discussed. These improvements include the development of a communication relay system facilitating longer distance operation, an improved actuating wheel system increasing the Scout's mobility and a grappling hook system to elevate the Scout. By enhancing the Scout robotic team in this way, the functional use of the team is expanded, allowing more practical use.

Search and rescue operations in large disaster sites require quick gathering of relevant information. Both the knowledge of the location of victims and the environmental/structural conditions must be available to safely and efficiently guide rescue personnel. A major hurdle for robots in such scenarios is stairs. A system for autonomous surmounting of stairs is proposed in which a Scout robot jumps from step to step. The robot's height is only about a quarter step in size. Control of the Scout is accomplished using visual servoing. An external observer such as another robot is brought into the control loop to provide the Scout with an estimation of its pose with respect to the stairs. This cooperation is necessary as the Scout must refrain from ill-fated motions that may lead it back down to where it started its ascend. Initial experimental results are presented along with a discussion of the issues involved.

This paper presents algorithms for vision-based detection and classification of vehicles in monocular image sequences of traffic scenes recorded by a stationary camera. Processing is done at three levels: raw images, region level, and vehicle level. Vehicles are modeled as rectangular patches with certain dynamic behavior. The proposed method is based on the establishment of correspondences between regions and vehicles, as the vehicles move through the image sequence. Experimental results from highway scenes are provided which demonstrate the effectiveness of the method. We also briefly describe an interactive camera calibration tool that we have developed for recovering the camera parameters using features in the image selected by the user.

We have designed and built a set of miniature robots called Scouts and have developed a distributed software system to control them. This paper addresses the fundamental choices we made in the design of the control software, describes experimental results in a surveillance task, and analyzes the factors that affect robot performance. Space and power limitations on the Scouts severely restrict the computational power of their on-board computers, requiring a proxy-processing scheme in which the robots depend on remote computers for their computing needs. While this allows the robots to be autonomous, the fact that robots' behaviors are executed remotely introduces an additional complication-sensor data and motion commands have to be exchanged using wireless communications channels. Communications channels cannot always be shared, thus requiring the robots to obtain exclusive access to them. We present experimental results on a surveillance task in which multiple robots patrol an area and watch for motion. We discuss how the limited communications bandwidth affects robot performance in accomplishing the task, and analyze how performance depends on the number of robots that share the bandwidth.

Using robots for surveillance and reconnaissance applications requires a versatile connection between the human operator and robotic hardware. Some application domains require a fully teleoperated system while others may benefit by giving robots more autonomy. This paper describes a robotic control architecture which merges both paradigms. The whole scheme is implemented using the miniature Scout robot and involves a suite of user interfaces that can be tailored to specific surveillance and reconnaissance missions. Hardware capabilities are presented and a visual servoing strategy, important for semi-autonomous Scout operation, is discussed.

We describe a distributed software system for controlling a group of miniature robots using a very low capacity communication system. Space and power limitations on the miniature robots drastically restrict the capacity of the communication system and require sharing bandwidth and other resources among the robots. We have developed a process management/scheduling system that dynamically assigns resources to each robot in an attempt to maximize the utilization of the available resources while still maintaining a priori behavior priorities. We describe a surveillance task in which the robots patrol an area and watch for motion, and present experimental results.

This paper presents algorithms for vision-based monitoring of weaving sections. These algorithms have been developed for the Minnesota Department of Transportation in order to acquire data for several weaving sections in the Twin Cities area. Unlike commercially available systems, the proposed algorithms can track and count vehicles as they change lanes. Furthermore, they provide the velocity and the direction of each vehicle in the weaving section. Experimental results from various weaving sections under various weather conditions are presented. The proposed methods are based on the establishment of correspondences among blobs and vehicles as the vehicles move through the weaving section. The blob tracking problem is formulated as a bipartite graph optimization problem.

This paper introduces a broadly applicable technique for visibly improving the digitized, grey-level outputs produced by a host of iterative geometric diffusion methods. By replacing standard, central-difference estimates of discrete spatial gradients with alternating image derivative estimates that are offset by known, complementary biases, errors accumulated during iteration are reduced and the quality of geometric diffusions is improved. This unexpected synergy occurs at no added computational cost over central difference methods. Very simple to implement, the innovation is introduced at the level of spatial derivatives; hence, for a given process, any derived higher level mathematical properties-for example, group invariance or scale-space properties-can be preserved.

Presents a miniature robotic system ("scout") useful for reconnaissance and surveillance missions. A large number of scout robots are deployed and controlled by humans and/or larger "ranger" robots. The specially designed and constructed scouts are extremely small (roughly 116cc volume) yet are readily deployable (by tossing or launching), have multiple mobility modes, have multiple sensing capabilities, can transmit and receive data and instructions, and have a limited capability for autonomous action. The rangers are significantly larger vehicles, based on a commercial-off-the-shelf platform, augmented with scout launchers, radios, and additional sensors. Together, the scouts and rangers form a hierarchical team capable of carrying out complex missions in a wide variety of environments.

Grasping is one of the most fundamental and challenging tasks in robotics. Applications range from space missions (e.g., collection of rock samples) to industrial automation. In this work, we use a camera mounted on the end-effector of a manipulator to grasp an unknown object in the workspace. A novel deformable contour model is used to determine plausible grasp axes of the target object. Potential grasp point pairs are generated, ranked based upon measurements taken from the contour, and a vision-guided grasp of the object using the highest ranked grasp point pair is executed. Several experimental results are presented.

In this paper we present an active video module that consists of a miniature video sensor, a wireless video transmitter and a pan-tilt mechanism driven by micromotors. The video module is part of a miniature mobile robot that is projected to areas of the environment to be surveyed. A single-chip CMOS video sensor and miniature brushless DC gearmotors are used to comply with restrictions imposed by the robotic system in terms of payload weight volume and power consumption. Different types of actuation are analyzed for compatibility with a mesoscale robotic system. Applications of an active video module are discussed.

Distributed robotics uses multiple robots through coordination of task to solve problems. Developing a distributed robotic system introduces a number of challenges, one of which is the intra-robot communication and networking of data and commands. Due to the characteristics of the targeted robotic missions and environments, the wireless networking requirements impose stringent constraints in terms of available memory, processing, and power consumption. Designing under these constraints and still providing support for such functions as synchronous commands and proxy processing is the true challenge. This paper discusses research in providing a lightweight wireless routing/networking solution that can dynamically tune the intra-robotic networking to adapt to the robotic team's mission needs and the environment situation encountered.

We undertook a study to determine if the automatic detection and counting of vehicle occupants is feasible. In the present paper, we report our findings regarding the appropriate sensor phenomenology and arrangement for the task. We propose a novel system based on fusion of near-infrared imaging signals and demonstrate its adequacy with theoretical and experimental arguments. We also propose a fuzzy neural network classifier to operate upon the fused near-infrared imagery and perform the occupant detection and counting function. We demonstrate experimentally that the combination of fused near-infrared phenomenology and fuzzy neural classification produces a robust solution to the problem of automatic vehicle occupant counting. We substantiate our argument by providing comparative experimental results for vehicle occupant counters based on visible, single near-infrared, and fused near-infrared bands. Our proposed solution can find a more general applicability as the basis for a reliable face detector both indoors and outdoors.

This paper presents a method for computing the skeleton of planar shapes and objects which exhibit sparseness (lack of connectivity), within their image regions. Such sparseness in images may occur due to poor lighting conditions, incorrect thresholding or image subsampling. Furthermore, in document image analysis, sparse shapes are characteristic of texts faded due to aging and/or poor ink quality. Due to the lack of pixel level connectivity, conventional skeletonization techniques perform poorly on such (sparse) shapes. Given the pixel distribution for a shape, the proposed method involves an iterative evolution of a piecewise-linear approximation of the shape skeleton by using a minimum spanning tree-based self-organizing map (SOM). By constraining the SOM to lie on the edges of the Delaunay triangulation of the shape distribution, the adjacency relationships between regions in the shape are detected and used in the evolution of the skeleton. The SOM, on convergence, gives the final skeletal shape. The skeletonization is invariant to Euclidean transformations. The potential of the method is demonstrated on a variety of sparse shapes from different application domains.

Robotic manipulators in contemporary work-cells are often incapable of solving even simple pick-and-place operations. More specifically, most systems require each object to be supplied in the same and pre-defined way. Introducing regrasping to the production cycle relaxes the constraints imposed on supply mechanisms. Regrasping describes the operation that must be performed whenever an object's pick-up grasp is incompatible with its put-down grasp. The paper presents a novel approach to solve the regrasp problem for robots equipped with parallel-jaw end-effecters. The proposed method first evaluates the object's possible grasps and placements. These are subsequently combined to form grasp-placement-grasp triples. A regrasp sequence leading from the pick-up to the put-down grasp is generated by searching through the resulting space of compatible grasp-placement-grasp triples. The algorithm takes kinematic and geometric constraints of both the manipulator and the objects into account. Computational concerns are addressed by subdividing the calculation into an off-line and a fast online phase.

Presents a method for computing the shape skeleton of planar objects in presence of noise occurring inside the image regions. Such noise may be due to poor control of lighting conditions, incorrect thresholding or image subsampling. Binary images of objects with such noise exhibit sparseness (lack of connectivity), within their image regions. Such non-contiguity may also be observed in thresholded images of objects which consist of regions having varying albedo. The problem of obtaining the skeletal description of sparse shapes is ill posed in the sense of conventional skeletonization techniques. We propose a skeletonization method which is based on obtaining the shape skeleton by evolving an approximation of the principal curve of the shape distribution. Our method is implemented as a batch mode Kohonen self-organizing map algorithm and involves iterating the following two steps: (1) Voronoi tessellation of the data, (2) kernel smoothing on the Voronoi centroids. Adjacency relationships between the Voronoi regions are obtained by computing a Delaunay triangulation of the centroids. The Voronoi centroids are connected by a minimum spanning tree after each iteration. The final shape skeleton is obtained by joining centroids which are disjoint in the spanning tree, but have adjacent Voronoi regions. The skeletal descriptions obtained with the method are invariant to translation, rotation, and scale changes of the shape. The potential of the method is demonstrated on industrial objects having varying shape complexity under different imaging conditions.

Positioning an eye-in-hand robotic system with respect to a static target is a challenging research problem since it involves recognition of the object and the desired pose at which alignment is to occur, planning a trajectory for the robot to attain this pose, and careful calibration of the system and the environment. In this paper we introduce a unified framework based on image morphing, to address the above problems and apply it to the task of translational and rotational alignment of an eye-in-hand system to planar objects or planar projections of 3D objects. In our method the desired manipulator pose for each object is defined and stored as a view of the object taken from this pose. The identity of an unknown object in the workspace is established by morphing its image to the views in the database, and using a quantification of the morph as a dissimilarity measure. The synthetic images generated during the morph are used guide an eye-in-hand system to the desired pose. The framework can accommodate partially occluded or deformable targets and smooth trajectories can be generated since an arbitrary number of intermediate images can be used.

We present a method for determining the skeletal shape description for letters in texts faded due to ageing and/or poor ink quality. The proposed algorithm is interesting in that it neither involves assumptions about demarcation of object regions from the background, nor does it require pixel connectivity in the text regions. Consequently, it may be applied for obtaining the shape descriptions of "sparse" regions, which are characteristic of letters in faded documents. Given the pixel distribution for a letter or a word from a faded document, the method involves an iterative evolution of a piecewise-linear approximation of the principal curve of this pixel distribution. By constraining the principal curve to lie on the edges of the Delaunay triangulation of the shape distribution, the adjacency relationships between regions in the shape can be detected and used in evolving the skeleton. The approximation of the principal curve, on convergence, gives the final skeletal shape. The skeletonization is invariant to Euclidean transformations and is adaptive in terms of the topology of the underlying shape distribution as well as in the number of units needed for the piece-wise approximation of the principal curve.

We propose a novel user-dependent method for the recognition of on-line handwritten notes. The method employs as a dissimilarity measure the "degree of morphing" between an input curve and a template curve. A physics-based approach substantiates the "degree of morphing" as a deformation energy and casts the problem as an energy minimization problem. The method operates upon key segmentation points that are provided by an appropriate segmentation algorithm. The segmentation objective is not to locate letters, but instead to locate corners and some key low curvature points (an easier task). This is part of the method's strategy to see the word as a generic on-line curve. Due to this strategy, the proposed method can handle collectively both cursive words and hand-drawn line figures, the two key ingredients of handwritten notes. Most importantly, the proposed system achieves high recognition rates without ever resorting to statistical models.

This paper describes a novel approach to real-time vehicle following. A scheme, called "symmetry axis detection and filtering based on symmetry constraints", is proposed and has been implemented. This scheme incorporates many elements of our research efforts in sensor-based control of mobile robots and manipulators. The proposed algorithm detects and tracks the rear portion of the exterior of a leading vehicle via a camera mounted on a vehicle that belongs to a platoon. Our scheme uses the symmetry property of the shape of most vehicles. In particular, our technique takes advantage of the stable contour symmetry instead of the intensity symmetry. An algorithm that employs a voting technique is proposed in order to detect the vertical symmetry axis of the leading vehicle. A filtering method based on symmetry constraints is performed to eliminate the pixels which do not contribute to the leading vehicle's contour. As a result of the filtering, a distinct symmetric contour is obtained. Robustness and real-time performance of the symmetry-based scheme are greatly enhanced by using an adaptive processing window, the local symmetry properties, and a filtering procedure.

A novel method for 2D shape recognition is proposed. The method employs as a dissimilarity measure the degree of morphing between a test shape and a reference shape. A physics-based approach substantiates the degree of morphing as a deformation energy and casts the problem as an energy minimization problem. The method operates upon key segmentation points that are provided by an appropriate segmentation algorithm. The recognition paradigm is invariant to translation, rotation, and scaling. It can handle both convex and non-convex shapes. The proposed system exhibits robust recognition behavior and real-time performance in a series of experiments. The experiments also highlight the ability of the method to recognize deformable shapes.

Flexible operation of a robotic agent in an uncalibrated environment requires the ability to recover unknown or partially known parameters of the workspace through sensing. Of the sensors available to a robotic agent, visual sensors provide information that is richer and more complete than other sensors. In this paper we present robust techniques for the derivation of depth from feature points on a target's surface and for the accurate and high-speed tracking of moving targets. We use these techniques in a system that operates with little or no a priori knowledge of object- and camera-related parameters to robustly determine such object-related parameters as velocity and depth. Such determination of extrinsic environmental parameters is essential for performing higher level tasks such as inspection, exploration, tracking, grasping, and collision-free motion planning. For both applications, we use the Minnesota robotic visual tracker (MRVT) (a single visual sensor mounted on the end-effector of a robotic manipulator combined with a real-time vision system) to automatically select feature points on surfaces, to derive an estimate of the environmental parameter in question, and to supply a control vector based upon these estimates to guide the manipulator.

A novel method that recognizes on-line handwritten patterns (typical in pen-based computing applications) is proposed. The method combines the advantages of both global and local recognition methods, works in real-time, and avoids the use of statistical models that require extensive user data. It is also one of the first methods that handles collectively cursive words, and hand-drawn line figures. The proposed system achieves pattern recognition through the use of shape metamorphosis. It is based on the premise that if two shapes are similar they don't have to undergo a substantial metamorphosis process in order for one to assume the shape of the other. In other words, the "degree of morphing" becomes the primary matching criterion. The notion of the "degree of morphing" is quantified through an energy minimization approach. The potential of the method is highlighted by a set of experiments.

A novel curve segmentation algorithm for determining control points for deformable-model-based target tracking is proposed. The algorithm is parameterless enabling a fully-fledged automated tracking regardless of the shape of the object being tracked. Compared with other curve segmentation algorithms, it selects a minimal number of control points that yet deliver a superior shape description. The algorithm is comparatively tested with other curve segmentation algorithms in a variety of characteristic target outlines.

Many researchers have turned to sensing, and in particular computer vision, to create more flexible robotic systems. Computer vision is often required to provide data for the grasping of a target. Using a vision system for grasping presents several issues with respect to sensing, control, and system configuration. This paper presents some of these issues in concert with the options available to the researcher and the trade-offs to be expected when integrating a vision system with a robotic system for the purpose of grasping objects. The paper includes experimental results from a particular configuration that characterize the type and frequency of errors encountered while performing various vision-guided grasping tasks. These error classes and their frequency of occurrence lend insight into the problems encountered during visual grasping and into the possible solution of these problems.

Presents a model-based approach for visual tracking and eye-in-hand robotic visual servoing. Our approach uses active deformable models to track a rigid or a semi-rigid object in the manipulator's workspace. These deformable models approximate the contour of the object boundary, defined by a set of control points. During tracking, the control points are updated at frame rates by minimizing an energy function involving the relative position of model points, the image data, and the characteristics of figure pixels. When visual servoing is combined with the use of active deformable models, movement of the manipulator can compensate for translations and deformations of the object's image. Experimental results are presented.

This paper addresses the issues of time and compression efficiency in image transformation. We propose a novel basis selection scheme which improves transformation efficiency by exploiting the energy compacting characteristic in the frequency domain. In a typical complete transformation, a large number of basis functions have low coding efficiency, and thus are not necessary in the encoding process. By removing these functions from the basis set, we improve the time and the compression efficiency of the encoding process while maintaining a high reproduction quality. We have chosen to use the Gabor transform to demonstrate our proposed method. Experimental results with the Gabor transform are presented to demonstrate the effectiveness of our method. Finally, issues related to the application of this approach in image sequence encoding and the adaptation of our approach to other transformation schemes are discussed.

This paper addresses the issues of compression and computation efficiency of transform-based image compression methods. Most transform-based compression systems operate based on the assumption that the input images are mostly conventional types of images. We argue that these systems will not perform efficiently in some image domains in which the aforementioned assumption does not hold. We believe an image encoding system must take into consideration the directional and frequency attributes in the input images in order to achieve high encoding quality. We present a novel adaptive approach for image transformation. Experimental results show that for an "unconventional" image, the proposed system produces better image quality than the JPEG at the same bit-rates.

Traditionally, the robotic visual servoing/tracking problem has received attention from researchers for its interesting control and computer vision issues. However many visual servoing tasks also require the ability to automatically detect moving objects. Until recently, very few efforts have been reported in the area of automatic detection of servoing targets. This paper presents a robust detection scheme for use in robotic visual servoing experiments. It detects and tracks moving objects through the use of a "figure/ground" approach. Experimentation has shown the feasibility of this approach under general conditions. This paper provides a description of the authors system implementation in an experimental robotic system, along with a collection of results. The paper also contains a discussion of problems and issues for future work.

Recent work combining robotics with vision has emphasized an active vision paradigm where the system changes the pose of the camera to improve environmental knowledge or to establish and presence a desired relationship between the robot and objects in the environment. Much of this work has concentrated upon the active observation of objects by the robotic agent. In this paper we present extensions to the controlled active vision framework that focus upon the autonomous grasping of a moving or static object in the manipulator's workspace. Our work extends the capabilities of an eye-in-hand system beyond those as a "pointer" or a "camera orienter" to provide the flexibility required to robustly interact with the environment in the presence of uncertainty. The proposed work is experimentally verified using the Minnesota robotic visual tracker to automatically select object features, to derive estimates of unknown environmental parameters, and to supply a control vector based upon these estimates to guide the manipulator in the grasping of a moving or static object.

Algorithms for full 3D robotic visual tracking of moving targets whose motion is 3D and consists of translational and rotational component are presented. The objective of the system is to track selected features on moving objects and to place their projections on the image plane at desired positions by appropriate camera motion. The most important characteristics of the proposed algorithms are the use of a single camera mounted on the end-effector of a robotic manipulator (eye-in-hand configuration), and the fact that these algorithms do not require accurate knowledge of the relative distance of the target object from the camera frame. The detection of motion is based on a cross-correlation technique known as sum-of squares differences (SSD) algorithm. The camera model used introduces a number of parameters that are estimated online, further reducing the algorithms' reliance on precise calibration of the system. An adaptive control algorithm compensates for modeling errors, tracking errors, and unavoidable computations delays which result from time-consuming image processing.>

Flexible operation of a robotic agent in an uncalibrated environment requires the ability to recover unknown or partially known parameters of the workspace through sensing. Of the sensors available to a robotic agent, visual sensors provide information that is richer and more complete than other sensors. In this paper we present robust techniques for the derivation of depth from feature points on a target's surface and for the accurate and high-speed tracking of moving targets. We use these techniques in a system that operates with little or no a priori knowledge of the object- and camera-related parameters to robustly determine such object-related parameters as velocity and depth. Such determination of extrinsic environmental parameters is essential for performing higher level tasks such as inspection, exploration, tracking, grasping, and collision-free motion planning. For both applications, we use the Minnesota Robotic Visual Tracker (a single visual sensor mounted on the end-effector of a robotic manipulator combined with a real-time vision system) to automatically select feature points on surfaces, to derive an estimate of the environmental parameter in question, and to supply a control vector based upon these estimates to guide the manipulator.>

In this paper, we propose a novel Gabor transformation scheme that is based on the QR decomposition. This Gabor-QR scheme computes the exact Gabor coefficients by solving a system of linear equations Ax=b. The computation is more efficient than iterative schemes and less complicated than other matrix-based methods. In addition, the proposed Gabor-QR scheme allows the use of incomplete basis sets, which is uncommon in most matrix-based Gabor transformations. Experimental results from image encoding are presented to highlight the major contributions of the proposed method. Due to its computational efficiency, the Gabor-QR decomposition scheme has a strong potential as an efficient method for image sequence encoding.>

Algorithms for 3D robotic visual tracking of moving targets whose motion is 3D and consists of translational and rotational components are presented. The objective of the system is to track selected features on moving objects and to place their projections on the image plane at desired positions by appropriate camera motion. The most important characteristics of the proposed algorithms are the use of a single camera mounted on the end-effector of a robotic manipulator (eye-in-hand configuration), and the fact that these algorithms do not require accurate knowledge of the relative distance of the target object from the camera frame. This fact makes these algorithms particularly useful in environments that are difficult to calibrate. The camera model used introduces a number of parameters that are estimated on-line, further reducing the algorithms' reliance on precise calibration of the system. An adaptive control algorithm compensates for modeling errors, tracking errors, and unavoidable computational delays which result from time-consuming image processing. Experimental results are presented to verify the efficacy of the proposed algorithms. These experiments were performed using a multi-robotic system consisting of Puma 560 manipulators.>

Accurate knowledge of depth continues to be of critical importance in robotic systems. Without accurate depth knowledge, tasks such as inspection, tracking, grasping, and collision-free motion planning prove to be difficult and often unattainable. Traditional visual depth recovery has relied upon techniques that require the solution of the correspondence problem or require known lighting conditions and Lambertian surfaces. In this paper, we present a technique for the derivation of depth from feature points on a target's surface using the controlled active vision framework. We use a single visual sensor mounted on the end-effector of a robotic manipulator to automatically select feature points and to derive depth estimates for those features using adaptive control techniques. Movements of the manipulator produce displacements that are measured using a sum-of-squared difference (SSD) optical flow. The measured displacements are fed into the controller to alter the path of the manipulator and to refine the depth estimate.>

Flexible operation of a robotic agent in an uncalibrated environment requires the ability to recover unknown or partially known parameters of the workspace through sensing. Of the sensors available to a robotic agent, visual sensors provide information that is richer and more complete than other sensors. In this paper we present robust techniques for the derivation of depth from feature points on a target's surface and for the accurate and high-speed tracking of moving targets. We use these techniques in a system that operates with little or no a priori knowledge of the object-related parameters present in the environment. The system is designed under the controlled active vision framework and robustly determines parameters such as velocity for tracking moving objects and depth maps of objects with unknown depths and surface structure. Such determination of intrinsic environmental parameters is essential for performing higher level tasks such as inspection, exploration, tracking grasping, and collision-free motion planning. For both applications, we use the Minnesota Robotic Visual Tracker (a single visual sensor mounted on the end-effector of a robotic manipulator combined with a real-time vision system) to automatically select feature points on surfaces, to derive an estimate of the environmental parameter in question, and to apply a control vector based upon these estimates to guide the manipulator. The paper concludes with applications of these techniques to transportation problems such as vehicle tracking.>

A new approach for robotic visual tracking and servoing is discussed. Deformable active models are introduced as a powerful means for tracking a moving rigid object (an eye-in-hand robot arm is used). Deformable models imitate, in real-time, the dynamic behavior of elastic structures. These computer-generated models are designed to capture the silhouette of rigid objects with well defined boundaries in terms of image gradient. By means of an eye-in-hand robot arm configuration, the desired motion of the end-effector is computed with the objective of keeping the target's position and shape invariant with respect to the camera frame. Experimental results are presented for the tracking of a rigid object moving in the three-dimensional space.>

The authors present algorithms for robotic (eye-in-hand configuration) real-time visual tracking of arbitrary 3D objects traveling at unknown velocities in a 2D space (depth is given as known). Visual tracking is formulated as a problem of combining control with computer vision. A mathematical formulation of the control problem that includes information from a novel feedback vision sensor and represents everything with respect to the camera frame is presented. The sum-of-squared differences (SSD) optical flow is used to compute the vector of discrete displacements each instant of time. These displacements can be fed either directly to a PI (proportional-integral) controller or to a pole assignment controller or discrete steady-state Kalman filter. In the latter case, the Kalman filter calculates the estimated values of the system's states and the exogenous disturbances, and a discrete LQG (linear-quadratic Gaussian) controller computes the desired motion of the robotic system. The outputs of the controllers are sent to the Cartesian robotic controller. Performance results are presented.>

The authors address the problem of integrating the human operator with autonomous robotic visual tracking and servoing modules. A CCD (charge coupled device) camera is mounted on the end-effector of a robot and the task is to servo around a static or moving rigid target. In manual control mode, the human operator, with the help of joystick and a monitor, commands robot motions in order to compensate for tracking errors. In shared control mode, the human operator and the autonomous visual tracking modules command motion along orthogonal sets of degrees of freedom. In autonomous control mode, the autonomous visual tracking modules are in full control of the servoing functions. Finally, in traded control mode, the control can be transferred from the autonomous visual modules to the human operator and vice versa. The authors present an experimental setup where all these different schemes have been tested. Experimental results of all modes of operation are presented and the related issues are discussed. In certain degrees of freedom the autonomous modules perform better than the human operator. On the other hand, the human operator can compensate fast for failures in tracking while the autonomous modules fail.>

This paper presents a new approach for visual tracking and servoing in robotics. We introduce deformable active models as a powerful means for tracking a rigid object in movement within the manipulator's workspace. Deformable models imitate, in real-time, the dynamic behavior of elastic structures. These computer-generated models are designed to capture the silhouette of rigid objects with well-defined boundaries, in terms of image gradient. By means of an eye-in-hand robot arm configuration, the desired motion of the end-effector is computed with the objective of keeping the target's position and shape invariant with respect to the camera frame. Optimal estimation and control techniques (ZQG regulator) have been successfully implemented in order to deal with noisy measurements provided by our vision sensor. Experimental results are presented for the tracking of a rigid object moving in a plane parallel to the image plane (three degree-of-freedom visual servoing). The experiments performed in a real-time environment show the effectiveness and robustness of the proposed method for servoing tasks based on visnal feedback control.

Using active monocular vision for 3-D visual control tasks is difficult since the translational and the rotational degrees of freedom are strongly coupled. The paper addresses several issues in 3-D visual control and presents adaptive control schemes for the problem of robotic visual servoing (eye-in-hand configuration) around a static rigid target. The objective is to move the image projections of several feature points of the static rigid target to some desired image positions. The inverse perspective transformation is assumed partially unknown. The adaptive controllers compensate for the servoing errors, the partially unknown camera parameters, and the computational delays which are introduced by the time-consuming vision algorithms. The authors present a stability analysis along with a study of the conditions that the feature points must satisfy in order for the problem to be solvable. Finally, several experimental results are presented to verify the validity and the efficacy of the proposed algorithms.>

Algorithms for robotic real-time visual tracking of arbitrary 3-D objects traveling at unknown velocities in a 2-D space are presented. The problem of visual tracking is formulated as a problem of combining control with computer vision. A mathematical formulation that is general enough to be extended to the problem of tracking 3-D objects in 3-D space is presented. The authors propose the use of sum-of-squared differences optical flow for the computation of the vector of discrete displacements each instant of time. These displacements can be fed either directly to a PI controller, a pole assignment controller, or a discrete steady-state Kalman filter. In the latter case, the Kalman filter calculates the estimated values of the system's states and exogenous disturbances, and a discrete LQG controller computes the desired motion of the robotic system. The outputs of the controllers are sent to a Cartesian robotic controller that drives the robot.>