Context-aware intelligent environments are computing systems embedded within physical spaces. They are equipped with input and output computing devices for users and sensors to provide contextual information to the system. These environments provide new challenges to interface designers due to a number of differences from typical desktop computing environments, including the lack of a single focal point for the user, a dynamic set of interaction devices, the sensor-rich nature of the environment, the potential of multiple simultaneous users, and the opportunity for diverse interaction modalities. This essay describes these challenges and focuses on issues involving multiple interaction modalities and automatic system behaviors.

We describe a real-time system for detecting pointing gestures and estimating the direction of pointing using stereo cameras. Previously, similar systems were implemented using color-based blob trackers, which relied on effective skin color detection; this approach is sensitive to lighting changes and the clothing worn by the user. In contrast, we used a stereo system that produces dense disparity maps in real-time. Disparity maps are considerably less sensitive to lighting changes. Our system subtracts the background, analyzes the foreground pixels to break the body into parts using a robust mixture model, and estimates the direction of pointing. We have tested the system on both coarse and fine pointing by selecting the targets in a room and controlling the cursor on a wall screen, respectively.

Background maintenance is a frequent element of video surveillance systems. We develop Wallflower, a three-component system for background maintenance: the pixel-level component performs Wiener filtering to make probabilistic predictions of the expected background; the region-level component fills in homogeneous regions of foreground objects; and the frame-level component detects sudden, global changes in the image and swaps in better approximations of the background. We compare our system with 8 other background subtraction algorithms. Wallflower is shown to outperform previous algorithms by handling a greater set of the difficult situations that can occur. Finally, we analyze the experimental results and propose normative principles for background maintenance.

In this paper we report on experiments with a system for autonomously driving two vehicles based on complex mission specifications. We show that the system is able to plan local paths in obstacle fields based on sensor data, to plan and update global paths to goals based on frequent obstacle map updates, and to modify mission execution, e.g., the ordering of the goals, based on the updated paths to the goals. Two recently developed sensors are used for obstacle detection: a high-speed laser rangefinder and a video-rate stereo system. An updated version of a dynamic path planner D* is used for online computation of routes. A new mission planning and execution monitoring tool, GRAMMPS, is used for managing the allocation and ordering of goals between vehicles. We report on experiments conducted in an outdoor test site with two HMMWVs. Implementation details and performance analysis, including failure modes, are described based on a series of twelve experiments, each over 1/2 km distance with up to nine goals. This system is the first multivehicle and multigoal system to be demonstrated in real, natural environments with this degree of generality. The work reported here includes a number of results not previously published, including the use of a real-time stereo machine, a high performance laser rangefinder and the GRAMMPS planning system.

For a system of cooperative mobile robots to be effective in real-world applications it must be able to efficiently execute a wide class of complex tasks in potentially unknown and unstructured environments. Previous research in multi-robot systems has either been limited to relatively structured domains or to small classes of feasible missions. This paper describes a field-capable system called GRAMMPS which addresses this problem by coupling a general-purpose interpreted grammar for task definition with dynamic planning techniques. GRAMMPS supports a general class of local navigation systems and heterogeneous groups of robots, providing optimal execution of missions given current world knowledge. Simulations illustrating the capabilities of this system are provided. Results showing successful runs of this system on two autonomous off-road vehicles are also given.

Mobile robots operating in vast outdoor unstructured environments often only have incomplete maps and must deal with new objects found during traversal. Path planning in such sparsely occupied regions must be incremental to accommodate new information, and, must use efficient representations. In previous work we have developed an optimal method D* to plan paths when the environment is not known ahead of time, but, rather is discovered as the robot moves around. To date, D* has been applied to a uniform grid representation for obstacles and free space. In this paper we propose the use of D* with framed quadtrees to improve the efficiency of planning paths in sparse environments. The new system has been tested in simulation as well on an autonomous jeep, equipped with local obstacle avoidance capabilities. We show how the use of framed quadtrees improves performance in terms of path length, computation speed, and memory requirements.

Planning for multiple mobile robots in dynamic environments involves determining the optimal path each robot should follow to accomplish the goals of the mission, given the current knowledge available about the world. As knowledge increases or improves, the planning system should dynamically reassign robots to goals in order to continually minimize the time to complete the mission. In this paper, an example problem in this domain is explored and performance results of such a dynamic planning system are presented. The system was able to dynamically optimize the motion of 3 robots toward 6 goals in real time, improving the average overall mission performance compared to a static planner by 25%. A preliminary design for a practical solution to a wider class of problems is also discussed.