This paper addresses the detection of 3D hand pointing direction from RGB-D data by a mobile robot. Considering ubiquitous forms of pointing gestures, the 3D pointing direction is assumed to be inferable from hand data only. First, a novel sequential network-based learning model is developed for the simultaneous detection of hands and humans in RGB data. Differing from previous work, its performance is shown to be both accurate and fast. Following, a new geometric method for estimating the 3D pointing direction from depth data of the detected hand is presented along with a mathematical analysis of sensor noise sensitivity. Two new data sets for pointing gesture classification and continuous 3D pointing direction with varying proximity, arm pose and background are presented. As there are no such data sets to the best of our knowledge, both will be publicly available. Differing from previous work, the robot is able to estimate the 3D hand direction both accurately and fast regardless of hand proximity, background variability or the detectability of specific human parts - as demonstrated by end-to-end experimental results.

This paper is focused on safe mapless navigation of mobile robots in unknown and possibly complex environments containing both internal and dynamic obstacles. We present a novel modular approach that combines the strengths of artificial potential functions (APF) with deep reinforcement learning. Differing from related work, the robot learns how to adjust the two input parameters of the APF controller as necessary through soft actor-critic algorithm. Environmental complexity measures are introduced in order to ensure that the robot's training covers a range of learning scenarios that vary in regard to maneuvering difficulty. Our experimental results show that differing from the classical navigation methods and end-to-end models, the robot can navigate successfully on its own even in complex scenarios with moving entities without requiring any maps.

In this paper, we consider the generation of generic object candidates by a mobile robot that is endowed with a pan-tilt monocular camera. This is an important problem because these candidates serve as basis for the robot to categorize and/or recognize the objects in its surroundings. The previously proposed methods either do not have a means of enabling the robot to look around through moving its camera or do not take advantage of the temporal coherence of the video data. We present a novel approach that enables the robot to achieve both of these capabilities simultaneously. In this approach, the robot's camera movements are governed by a family of controllers whose constructions depend on the set of object candidates that have been hitherto generated, but not directly looked at. In parallel, the robot discovers the object candidates through tracking segments and determining spatio-temporally coherent ones. The advantage of the proposed approach is that while the robot can explore its surroundings by simply looking around prior to more sophisticated exploration behavior involving possibly bodily locomotion the generated object candidates turn out to be consolidated across the visual stream in comparison to single-shot methods. This is demonstrated in extensive experimental results with a robot operating indoors varying in clutter as well as outdoors.

In this paper, we are concerned with the automation of single robot exogeneous assembly in simple planar settings. Here, a set of of unactuated disk-shaped parts needs to be serviced by a single robot that resides outside the workspace and is capable of moving the parts one at a time through sliding or lifting slightly. The task is to have the parts end up in their respective goal positions without any collisions whilst moving. We present an automated reactive approach through the composition of one-part movements. The movement of each part is achieved by a controller obtained through the projection of a carefully constructed vector field on the respective configuration subspace. Such a scheme is known to accommodate positional variations naturally. Once the movement terminates, the robot chooses the next part to move in a cyclic manner. The contribution of this paper is to show for the first time that with certain restrictions on the allowed goal configurations, the assembly task is either successfully completed or terminated (rather than useless cycling of a part or from part to part) while the generated sequence of motions never causes collisions among the parts.

This paper presents an exploration approach for discovering canonical appearances in unknown environments using an autonomous ground robot equipped with a depth sensor. This approach is based on the previously proposed two-stage algorithm that alternates between local and global decision making for efficient topological mapping based on bubble space representation. Differing from it, the approach aims to identify vantage viewpoints with characterizing views for subsequent appearance-based learning as well as achieving complete coverage. This is demonstrated by a series of experiments using an outdoor benchmark data set including a comparative study with evaluation metrics including the exploration path length and number of canonical appearances discovered.

In many multirobot applications, the overall goal topology of the robot formation is more important than achieving a specific adjacency. In such cases, it is convenient to revise the given goal adjacency matrix so that it is more apt to the robots' configuration. In this paper, we present a solution to this problem. For this, we first formulate the problem as finding an optimal isomorphic adjacency matrix and then propose a two-stage method for solving it. In the first stage, an optimal orthogonal matrix that solves the problem is determined while in the second stage, the search for the optimal permutation matrix is conducted - based on the approximation of the resulting orthogonal matrix by a non-commutative convex combination of permutation matrices. Extensive simulation results indicate that an adjacency matrix that is more apt to the robots' locations can be determined with the optimality of the resulting solution tuning the accuracy parameter.

This paper considers the merging of appearance-based spatial knowledge among robots having compatible visual sensing. Each robot is assumed to retain its knowledge in its individual long-term spatial memory where i) the place knowledge and their spatial relations are retained in an organized manner in place and map memories respectively; and ii) a ‘place’ refers to a spatial region as designated by a collection of associated appearances. In the proposed approach, each robot communicates with another robot, receives its memory and then merges the received knowledge with its own. The novelty of the merging process is that it is done in two stages: merging of place knowledge followed by the merging of map knowledge. As each robot's place memory is processed as a whole or in portions, the merging process scales easily with respect to the amount and overlap of the appearance data. Furthermore, the merging can be done in decentralized manner. Our experimental results with a team of three robots demonstrate that the resulting merged knowledge enables the robots to reason about learned places.

This paper is on simultaneous exploration and topological map building in completely unknown environments. We propose an approach that consists of a continuous loop of local decision-making (where potential search directions are added to the current node) and global decision-making (where the robot moves along one of these directions and adds possibly a new node to the map ) using the previously proposed bubble space representation. As the robot switches between the two in a continual manner, it creates a coarse graph representation of the environment. There is a node for every explored location with a set of associated edges that either indicate potential search directions or connect to the other explored locations. The so-far constructed topological map contains all the required information in regards to deciding where to move with comparatively low memory requirements. Experimental results with an extensive real data set demonstrate that large terrains can be explored and mapped efficiently using this approach.

In this work, we consider long-term topological place learning and present an approach that enables the robot to learn in an unsupervised, organized and incremental manner. The knowledge associated with the previously visited places is internally stored in the form of bubble descriptor semantic tree (BDST) using the previously proposed bubble space representation. The BDST is generated and maintained without any external supervision. It organizes the learned knowledge where the terminal nodes are viewed as corresponding to distinct places while its structure encodes their semantic hierarchy. In case the robot is not able to recognize a place with its current BDST, it learns it via updating the BDST incrementally based on the hierarchical single link clustering algorithm SLINK. The proposed approach is evaluated experimentally using combined benchmark datasets from indoor and outdoor settings with recognition rates comparable to those of state-of-the-art approaches while the robot is able to retain efficiently and use the knowledge associated with the learned places.

This paper introduces a novel approach to topo-logical place detection. The approach is based on previously proposed bubble space representation - where all sensory features and their relative S2- geometry are encoded in a manner that is implicitly dependent on robot pose. Its novelty is that ensuring sensory data reliability is integrated with place detection. This is achieved via checking for informativeness, coherence and plenitude using only the bubble space representation of the incoming sensory data. The stringency of these checks is controllable via a set of associated parameters. Experimental results with benchmark datasets indicate correct detection rates comparable to state-of-the-art approaches in place detection. Furthermore, the detected places can then be immediately used to generate the nodes in topological maps.

This paper addresses the coordinated navigation of multiple independently actuated disk-shaped robots-all placed within the same disk-shaped workspace. Assuming perfect sensing, shared-centralized communications and computation, as well as perfect actuation, we encode complete information about the goal, obstacles, and workspace boundary using an artificial potential function over the configuration space of the robots' simultaneous nonoverlapping positions. The closed-loop dynamics governing the motion of each (velocity-controlled) robot take the form of the appropriate projection of the gradient of this function. We impose (conservative) restrictions on the allowable goal positions that yield sufficient conditions for convergence: We prove that this construction is an essential navigation function that guarantees collision-free motion of each robot to its destination from almost all initial free placements. The results of an extensive simulation study investigate practical issues such as average resulting trajectory length and robustness against simulated sensor noise.

This paper studies the problem of communication network topology in a multirobot system while the agents are engaged in a given task. The contribution of this paper is to propose a centralized approach to network evolution. In this approach, a communication coordinator is responsible for determining the network topology. The robots periodically send their state information to the communication coordinator. In turn, the communication coordinator considers the individual communication payoff functions of all the robots, their current states and the current network simultaneously and finds a network topology acceptable to all the robots. It models the network topology formation as a pairwise game where it forms or severs pairwise links based on the improvement the resulting network offers the robot pairs relative to the current network. We show that with the assumed form of communication payoff functions, each pairwise game is ensured of convergence to a pairwise stable network. Furthermore, simulation results provide statistical results on the resulting network topology, the number of game moves and the processing time as well as comparative results with the proximity based approach.

Place representation is a key element in topological maps. This paper presents bubble space - a novel representation for “places” (nodes) in topological maps. The novelties of this model are two-fold: First, a mathematical formalism that defines bubble space is presented. This formalism extends previously proposed bubble memory to accommodate two new variables - varying robot pose and multiple features. Each bubble surface preserves the local S2-metric relations of the incoming sensory data from the robot's viewpoint. Secondly, for learning and recognition, bubble surfaces can be transformed into bubble descriptors that are compact and rotationally invariant, while being computable in an incremental manner. The proposed model is evaluated with support vector machine based decision making in two different settings: first with a mobile robot placed in a variety of locations and secondly using benchmark visual data.

This paper presents a novel approach to the navigation of dynamically communicating robots via a bidirectional interaction model between the robot network and the continuous states. First, the robot dynamics is formulated as being dependent on the communication network where two robots - if in communication - can access each other's position and goal information. Next, three alternative strategies for establishing the communication network depending on the robots' states are presented: deterministic, game-theoretic and mixed approaches. In the first approach, the network is defined deterministically based on the robots' states and the communication range. The game-theoretic network formation is based on utilizing the conflict between the communication gain and cost. The mixed approach integrates features from deterministic and game-theoretic approaches. An extensive statistical study investigates comparative performance characteristics for exploration, zone and procession type goals.

This paper considers the problem of planar multi-robot realizations of connectivity graphs. A realization is a set of planar positions for a team of robots with a connectivity graph that is identical to an a priori given connectivity graph with the additional constraint that it must be feasible. Feasibility means that that the robots must not be overlapping with each other. As the associated mathematical problem is known to be NP-hard, a stochastic approach based on genetic algorithms is proposed. First, a population set based on randomly generated planar and feasible multi-robot positions is generated. Next, a fitness function that measures the similarity of the graph of each member is constructed. Finally, new reproduction operators that enable the evolution of generations are introduced. An extensive statistical study with different number of robots demonstrates that the proposed algorithm can be used to obtain fairly complicated network topologies.

The paper addresses the warehouseman's problem with geometrical simplifications under the more realistic case of imperfect sensory information. In this scenario, a 2D workspace contains an actuated robot and a set of unactuated parts. The discrepancy between the robot's and/or the parts' real and measured positions may lead to jerky movements or even collisions in the parts' moving problem we are concerned with. Thus, we need to approximate the state information - taking the highly nonlinear nature of the resulting system into account. This is accomplished using particle filters - which implement recursive Bayesian filter in nonlinear and/or nongaussian environments. For the model of parts which turns out to be linear, the approach reduces to Kalman filtering. First the robot's dynamic model and the measurement model are modified to incorporate the inaccuracies in the sensory data; and then the particle filter is utilized to get improved positional estimate. Enhancements in the robot's movements and reduction in the number of collisions have been verified through extensive computer simulations

This paper presents a mathematical model for saccadic motion and fixations. We relate this issue to the problem of motion planning and show that a family of artificial potential functions can be used for creating saccadic motion. The advantage of this approach is that finding the next fixation point does not require an explicit visual search - which is computationally costly and may be problematic in real-time applications. Rather, the system naturally 'slides' from the current fixation into the next. Thus real-time performance on cheap hardware can easily be achieved. Experimental results serve to provide insight into the performance of a robot APES implementing this approach

A collection of unactuated disk-shaped "parts" must be brought by an actuated manipulator robot into a specified configuration from arbitrary initial conditions. The task is cast as a noncooperative game played among the parts-which in turn yields a feedback-based event-driven approach to plan generation and execution. The correctness of this approach, an open question, has been demonstrated in simpler settings and is further suggested by the extensive experiments reported here using an actual working implementation with EDAR-a mobile robot operating in a purely feedback-based event-driven manner. These results verify the reliability of this approach against uncertainties in sensory information and unanticipated changes in workspace configuration.

Biological vision systems explore their environment via allocating their visual resources to only the interesting parts of a scene. This is achieved by a selective attention mechanism that controls eye movements. The data thus generated is a sequence of subimages of different locations and thus a sequence of features extracted from those images - referred to as attentional sequence. In higher level visual processing leading to scene cognition, it is hypothesized that the information contained in attentional sequences are combined and utilized by special mechanisms - although still poorly understood. However, developing models of such mechanisms prove out to be crucial - if we are to understand and mimic this behavior in robotic systems. In this paper, we consider the recognition problem and present two approaches to using attentional sequences for recognition: Markovian and evidential reasoning. Experimental results with our mobile robot APES reveal that simple shapes can be modeled and recognized by these methods - using as few as ten fixations and very simple features. For more complex scenes, longer attentional sequences or more sophisticated features may be required for cognition.

Robot vision systems inspired by human vision are required to employ mechanisms similar to those which are known to play key roles in human visual performance. Some of the more crucial properties are: 1) ability to actively select and filter visual input by changing its focus of attention both mechanically and mentally; 2) a retina model which will reduce computational complexity like in the human eye; 3) levels of representation; 4) serial processing of incoming stream of visual data; and 5) memory mechanisms that will keep track of eye movements and store functional models of the environment. APES is a mobile robot developed in our laboratory to meet these requirements simultaneously and to act as a test-bed for biologically motivated robot vision research. In this paper we describe APES and the original approaches taken to realize the above properties of biological vision.

This paper deals with the parts moving problem based on an event-driven planning and control. We are interested in developing feedback based approaches to the automatic generation of actuator commands that cause the robot to move a set of parts from an arbitrary initial disassembled configuration to a specified final configuration. In the Phase 1 of this project, a composite algorithm that reactively switches between different feedback controllers is shown to induce a noncooperative game being played among the parts being manipulated. This paper describes experimental results with EDAR (event-driven assembler robot) developed for moving parts based on feedback techniques.

Selectively perceiving systems explore their environment by fixating on and analyzing potentially interesting parts of a scene in a continual manner. During this process, a sequence of snapshots, each containing visual data with distinct spatio-temporal properties, is collected. We propose a visual memory model that enables the integration of this spatio-temporal data. This memory model, referred to as the bubble model, uses a spherical surface attached to the viewpoint in order to store and manipulate information collected during selective perception process. The bubble surface can be deformed by a number of control points to represent visual information obtained from different fixation points. Bubbles are indexed by the viewpoint vectors so that they can be recalled or updated as necessary. Different bubbles attached to the same viewpoint can store multiple simple and complex features. Using time stamps and parametric surface representations the model can also be used to extract changes and similarities in the environment.

The ability to combine vision and behavior has proven to be vital for a system to achieve real-time performance on a variety of tasks. Using biological evidence, it has been shown that such systems must have oculomotion, attention and spatio-temporal reasoning capabilities. The paper describes BUVIS an automated visual inspection system-which is designed to have these capabilities-and selective perceptual processing thus achieved can result in real-time behavior in visual inspection of manufactured parts.

In this study, ultrasound image sequences of fetus head are examined for the bi-parietal diameter measurement. There are five parts of the fetus head that have to be seen clearly before such a measurement can be made. Our approach is based on model-based multi-objective analysis. For each part, an objective function encoding both data and anatomic constraints is formulated. The resulting multi-objective problem is then transformed into a problem of coupled differential equations and is then solved using numerical integration. Throughout the process, continuity principle is used between the frames so that the results of analysis on one frame can be used on the next frame-whenever possible.

The ability to combine vision and behaviour is vital for a robot to achieve real time performance on a variety of tasks in a complex and dynamic environment. The design and development of such systems have been motivated by physiological and psychological studies on biological systems. These studies indicate that perception systems must have focusing, oculomotion and multi-level processing capabilities. This paper describes a mobile robot having a biologically motivated vision system. The robot-APES-is designed to have both physical and mental attention capabilities, end selective perceptual processing thus achieved can result in real-time behavior in simple tasks involving recognition.

This paper offers a new approach to integration of modules in an intelligent sensor system. Such a system requires that a set of modules-each doing a smaller portion of the overall task-be integrated within a unifying framework. From the perspective of computational systems, this problem holds a considerable interest because it is characterized by a set of coexisting mathematical objectives that need to be optimized simultaneously. In this sense, the design considerations necessitate the introduction of problem solving with multiple objectives. This paper explores these issues in the instance when each module is associated with a mathematical objective that is a function of the outputs of other modules. The integration problem is formulated and what is required of a good solution is presented. This examination interprets the decentralized mediation of conflicting subgoals as promoting a N-player game amongst the modules to be integrated and proposes a game-theoretic integration framework. We model the interaction among the modules as a noncooperative game and argue that this strategy leads to a framework in which the solutions correspond to a compromise decision. The application of this framework in image analysis motivates the hope that a framework such as game-theoretic integration will facilitate the development of general design principles for "modular" systems.>

A variety of architectures have been proposed for integrating vision modules into a complete system. An examination of these systems indicates that they either lack analytic/computational tractability or are based on restrictive assumptions such as global additivity or sequential, decoupled solution of objectives. The authors introduce a framework of integration which overcomes both of these shortcomings. This proposed model is based on game-theoretic setting in which a correspondence between a vision system and a N-player game is established. The authors demonstrate the power of such a model within a comparative study in the context of a system aimed at delineating 2-D deformable contours from noisy images.>

Describes a novel integration architecture for decentralized decision making in modular intelligent sensor systems. In contrast to previous approaches, this framework preserves the decentralized and coexisting natures of the objectives and is analytically and computationally tractable. The starting point for this approach is noncooperative N-player game theory.>

In relaxation labeling, the domain model is incorporated into the system through the constraint functions ( compatibility coefficients). The selection of these functions is shown to be critical. In particular, for certain choices of the contraint functions, some processes will have a single nontrivial convergence point regardless of the initial labeling. Criteria on the constraint functions which result in a single convergence point are presented for those relaxation labeling processes whose formulations are within the framework of an optimization problem. We then present the necessary and sufficient conditions on the constraint €unctions under which the first of these sufficiency criteria is satisfied. The othpr conditions can be tested using standard techniques. These criteria yield a formal guideline for detecting some of the inadmi-ible constraint functions.