As robots enter the human environment, there are increasing needs for novice users to be able to program robots with ease. A successful robot programming system should be intuitive, interactive, and intention aware. Intuitiveness refers to the use of intuitive user interfaces such as speech and hand gestures. Interactivity refers to the system's ability to let the user interact preemptively with the robot to take its control at any given time. Intention awareness refers to the system's ability to recognize and adapt to user intent. This paper focuses on the intention awareness problem for interactive multi-modal robot programming system. In our framework, user intent takes on the form of a robot program, which in our context is a sequential set of commands with parameters. To solve the intention recognition and adaptation problem, the system converts robot programs into a set of Markov chains. The system can then deduce the most likely program the user intends to execute based on a given observation sequence. It then adapts this program based on additional interaction. The system is implemented on a mobile vacuum cleaning robot with a user who is wearing sensor gloves, inductive position sensors, and a microphone.

This paper introduces a novel approach to program a robot interactively through a multi-modal interface. The key characteristic of this approach is that the user can provide feedback interactively at any time - during both the programming and the execution phase. The framework takes a three-step approach to the problem: multi-modal recognition, intention interpretation, and prioritized task execution. The multi-modal recognition module translates hand gestures and spontaneous speech into a structured symbolic data stream without abstracting away the user's intent. The intention interpretation module selects the appropriate primitives to generate a task based on the user's input, the system's current state, and robot sensor data. Finally, the prioritized task execution module selects and executes skill primitives based on the system's current state, sensor inputs, and prior tasks. The framework is demonstrated by interactively controlling and programming a vacuum-cleaning robot.

To combine sensor information from distributed robot teams, it is critical to know the locations of all the robots relative to each other. This paper presents a novel fault tolerant localization algorithm developed for centimeter-scale robots, called Millibots. To determine their locations, the Millibots measure the distances between themselves with an ultrasonic distance sensor. They then combine these distance measurements with dead reckoning in a maximum likelihood estimator. The focus of this paper is on detecting and isolating measurement faults that commonly occur in this localization system. Such failures include dead reckoning errors when the robots collide with undetected obstacles, and distance measurement errors due to destructive interference between direct and multi-path ultrasound wavefronts. Simulations show that the fault tolerance algorithm accurately detects erroneous measurements and significantly improves the reliability and accuracy of the localization system.

Presents a distributed planning and control architecture for autonomous multi-manipulator systems (MMS). The control architecture is implemented using an agent-based approach. A team of distributed and autonomous agents is deployed to model the flexible assembly system in such a way that the agents negotiate, collaborate, and cooperate to achieve the goals of assembly tasks. The main focus of the paper is on assembly task allocation and assembly task execution. We describe the agent models and communication mechanism, and explain how they handle complex interactions among agents. A distributed trajectory planning approach based on artificial potential fields is also presented. Experimental results show that our multi-agent planning and control framework is suitable for flexible robotic assembly tasks. Our approach addresses the issues of flexibility, scalability, reconfigurability, and fault-tolerance. We anticipate that the same approach can be applied to other flexible manufacturing environments.

To focus on the research issues surrounding collaborative behavior in multiple mobile-robotic systems, a great amount of low-level infrastructure is required. To facilitate our on-going research into multi-robot systems, we have developed RAVE, a software framework that provides a real and virtual environment for running and managing multiple heterogeneous mobile-robot systems. This framework simplifies the implementation and development of collaborative robotic systems by providing the following capabilities: the ability to run systems off-line in simulation, user-interfaces for observing and commanding simulated and real robots, transparent transference of simulated robot programs to real robots, the ability to have simulated robots interact with real robots, and the ability to place virtual sensors on real robots to augment or experiment with their performance.

Gestures provide a rich and intuitive form of interaction for controlling robots. This paper presents an approach for controlling a mobile robot with hand gestures. The system uses hidden Markov models (HMMs) to spot and recognize gestures captured with a data glove. To spot gestures from a sequence of hand positions that may include nongestures, we have introduced a "wait state" in the HMM. The system is currently capable of spotting six gestures reliably. These gestures are mapped to robot commands under two different modes of operation: local and global control. In the local control mode, the gestures are interpreted in the robot's local frame of reference, allowing the user to accelerate, decelerate, and turn. In the global control mode, the gestures are interpreted in the world frame, allowing the robot to move to the location at which the user is pointing.

The Gyrover is a single wheel gyroscopically stabilized mobile robot developed at Carnegie Mellon University. An internal pendulum serves as a counter weight for a drive motor that causes fore/aft motion, while a tilt-mechanism on a large gyroscope provides a mechanism for lateral actuation. In this paper we develop a detailed dynamic model for the Gyrover and use this model in an extended Kalman filter to estimate the complete state. A linearized version of the model is used to develop a state feedback controller. The design methodology is based on a semi-definite programming procedure which optimize the stability region subject to a set of linear matrix inequalities that capture stability and pole placement constraints. Finally, the controller design combined with the extended Kalman filter are verified on the prototype.

Significant advances have been made in the area of macro-planning for assembly operations (i.e.,, dividing a product into sub-assemblies, determining the sequence of assembly operations). On the contrary, the state of the art in micro-planning (i.e., tool selection, path planning for tool and part movements) is rather primitive. To generate a realizable assembly plan, both macro-planning and micro-planning problems need to be solved. Tooling considerations are an important component of the micro-planning problem. Therefore, we present a methodology for modeling assembly tools, selecting tools for assembly operations, and generating detailed tool movements. Techniques described in this paper can be combined with macro-planning techniques to result in a complete assembly planner.

A rapidly deployable fault tolerant manipulator system consists of modular hardware and support software that allow the user to quickly configure and deploy a fault tolerant manipulator that is custom-tailored for a given task. The main focus of this paper is on the task based design component of such a system, that is, the determination of the optimal manipulator configuration, its base position and the corresponding joint space trajectory for a given task. We introduce a novel agent-based solution approach to task based design and illustrate it with a fault tolerant manipulator design for a satellite docking operation aboard the space shuttle.

A rapidly deployable manipulator system combines the flexibility of reconfigurable modular hardware with modular programming tools, allowing the user to rapidly create a manipulator which is custom-tailored for a given task. This article describes two main aspects of such a system, namely, the reconfigurable modular manipulator system (RMMS) hardware and the corresponding control software.

Whether a task can be completed after a failure of one of the degrees-of-freedom of a redundant manipulator depends on the joint angle at which the failure takes place. It is possible to achieve fault tolerance by globally planning a trajectory that avoids unfavorable joint positions before a failure occurs. In this paper we present a trajectory planning algorithm that guarantees fault tolerance while simultaneously satisfying joint limit and obstacle avoidance requirements.

The application of robots in critical missions in hazardous environments requires the development of reliable or fault tolerant manipulators. In this paper we define fault tolerance as the ability to continue the performance of a task after immobilization of a joint due to failure. Initially, no joint limits are considered, in which case we prove the existence of fault tolerant manipulators and develop an analysis tool to determine the fault tolerant work space. We also derive design templates for spatial fault tolerant manipulators. When joint limits are introduced, analytic solutions become infeasible but instead a numerical design procedure can be used, as is illustrated through an example.>