Prior research has investigated the effect of interactive social agents presented on computer screens or embodied in robots. Much of this research has been pursued in labs and brief field studies. Comparatively little is known about social agents embedded in the workplace, where employees have repeated interactions with the agent, alone and with others. We designed a social robot snack delivery service for a workplace, and evaluated the service over four months allowing each employee to use it for two months. We report on how employees responded to the robot and the service over repeated encounters. Employees attached different social roles to the robot beyond a delivery person as they incorporated the robot's visit into their workplace routines. Beyond one-on-one interaction, the robot created a ripple effect in the workplace, triggering new behaviors among employees, including politeness, protection of the robot, mimicry, social comparison, and even jealousy. We discuss the implications of these ripple effects for designing services incorporating social agents.

Creating and sustaining rapport between robots and people is critical for successful robotic services. As a first step towards this goal, we explored a personalization strategy with a snack delivery robot. We designed a social robotic snack delivery service, and, for half of the participants, personalized the service based on participants' service usage and interactions with the robot. The service ran for each participant for two months. We evaluated this strategy during a 4-month field experiment. The results show that, as compared with the social service alone, adding personalized service improved rapport, cooperation, and engagement with the robot during service encounters.

Current manufacturing practices require complete physical separation between people and active industrial robots. These precautions ensure safety, but are inefficient in terms of time and resources, and place limits on the types of tasks that can be performed. In this paper, we present a real-time, sensor-based approach for ensuring the safety of people in close proximity to robots in an industrial workcell. Our approach fuses data from multiple 3D imaging sensors of different modalities into a volumetric evidence grid and segments the volume into regions corresponding to background, robots, and people. Surrounding each robot is a danger zone that dynamically updates according to the robot's position and trajectory. Similarly, surrounding each person is a dynamically updated safety zone. A collision between danger and safety zones indicates an impending actual collision, and the affected robot is stopped until the problem is resolved. We demonstrate and experimentally evaluate the concept in a prototype industrial workcell augmented with stereo and range cameras.

We describe a real-time pedestrian detection system intended for use in automotive applications. Our system demonstrates superior detection performance when compared to many state-of-the-art detectors and is able to run at a speed of 14 fps on an Intel Core i7 computer when applied to 640×480 images. Our approach uses an analysis of geometric constraints to efficiently search feature pyramids and increases detection accuracy by using a multiresolution representation of a pedestrian model to detect small pixel-sized pedestrians normally missed by a single representation approach. We have evaluated our system on the Caltech Pedestrian benchmark which is currently the largest publicly available pedestrian dataset at the time of this publication. Our system shows a detection rate of 61% with 1 false positive per image (FPPI) whereas recent other state-of-the-art detectors show a detection rate of 50% ~ 61% under the `reasonable' test scenario (explained later). Furthermore, we also demonstrate the practicality of our system by conducting a series of use case experiments on selected videos of Caltech dataset.

This paper presents a monocular vision based 3D bicycle tracking framework for intelligent vehicles based on a detection method exploiting a deformable part model and a tracking method using an Interacting Multiple Model (IMM) algorithm. Bicycle tracking is important because bicycles share the road with vehicles and can move at comparable speeds in urban environments. From a computer vision standpoint, bicycle detection is challenging as bicycle's appearance can change dramatically between viewpoints and a person riding on the bicycle is a non-rigid object. To this end, we present a tracking-by-detection method to detect and track bicycles that takes into account these difficult issues. First, a mixture model of multiple viewpoints is defined and trained via a Latent Support Vector Machine (LSVM) to detect bicycles under a variety of circumstances. Each model uses a part based representation. This robust bicycle detector provides a series of measurements (i.e., bounding boxes) in the context of the Kalman filter. Second, to exploit the unique characteristics of bicycle tracking, two motion models based on bicycle's kinematics are fused using an IMM algorithm. For each motion model, an extended Kalman filter (EKF) is used to estimate the position and velocity of a bicycle in the vehicle coordinates. Finally, a single bicycle tracking method using an IMM algorithm is extended to that of multiple bicycle tracking by incorporating a Rao-Blackwellized Particle Filter which runs a particle filter for a data association and an IMM filter for each bicycle tracking. We demonstrate the effectiveness of this approach through a series of experiments run on a new bicycle dataset captured from a vehicle-mounted camera.

Robots that operate in the real world will make mistakes. Thus, those who design and build systems will need to understand how best to provide ways for robots to mitigate those mistakes. Building on diverse research literatures, we consider how to mitigate breakdowns in services provided by robots. Expectancy-setting strategies forewarn people of a robot's limitations so people will expect mistakes. Recovery strategies, including apologies, compensation, and options for the user, aim to reduce the negative consequence of breakdowns. We tested these strategies in an online scenario study with 317 participants. A breakdown in robotic service had severe impact on evaluations of the service and the robot, but forewarning and recovery strategies reduced the negative impact of the breakdown. People's orientation toward services influenced which recovery strategy worked best. Those with a relational orientation responded best to an apology; those with a utilitarian orientation responded best to compensation. We discuss robotic service design to mitigate service problems.

This paper presents a vision-based framework for intelligent vehicles to detect and track people riding bicycles in urban traffic environments. To deal with dramatic appearance changes of a bicycle according to different viewpoints as well as nonrigid nature of human appearance, a method is proposed which employs complementary detection and tracking algorithms. In the detection phase, we use multiple view-based detectors: frontal, rear, and right/left side view. For each view detector, a linear Support Vector Machine (SVM) is used for object classification in combination with Histograms of Oriented Gradients (HOG) which is one of the most discriminative features. Furthermore, a real-time enhancement for the detection process is implemented using the Integral Histogram method and a coarse-to-fine cascade approach. Tracking phase is performed by a multiple patch-based Lucas-Kanade tracker. We first run the Harris corner detector over the bounding box which is the result of our detector. Each of the corner points can be a good feature to track and, in consequence, becomes a template of each instance of multiple Lucas-Kanade trackers. To manage the set of patches efficiently, a novel method based on spectral clustering algorithm is proposed. Quantitative experiments have been conducted to show the effectiveness of each component of the proposed framework.

For an autonomous vehicle, detecting and tracking other vehicles is a critical task. Determining the orientation of a detected vehicle is necessary for assessing whether the vehicle is a potential hazard. If a detected vehicle is moving, the orientation can be inferred from its trajectory, but if the vehicle is stationary, the orientation must be determined directly. In this paper, we focus on vision-based algorithms for determining vehicle orientation of vehicles in images. We train a set of Histogram of Oriented Gradients (HOG) classifiers to recognize different orientations of vehicles detected in imagery. We find that these orientation-specific classifiers perform well, achieving a 88% classification accuracy on a test database of 284 images. We also investigate how combinations of orientation-specific classifiers can be employed to distinguish subsets of orientations, such as driver's side versus passenger's side views. Finally, we compare a vehicle detector formed from orientation-specific classifiers to an orientation-independent classifier and find that, counter-intuitively, the orientation-independent classifier outperforms the set of orientation-specific classifiers.

In this paper we present an algorithm that allows a human to naturally and easily teach a mobile robot how to recognize objects in its environment. The human selects the object by pointing at it using a laser pointer. The robot recognizes the laser reflections with its cameras and uses this data to generate an initial 2D segmentation of the object. The 3D position of SURF feature points are extracted from the designated area using stereo vision. As the robot moves around the object, new views of the object are obtained from which feature points are extracted. These features are filtered using active vision. The complete object representation consists of feature points registered with 3D pose data. We describe the method and show that it works well by performing experiments on real world data collected with our robot. We use an extensive dataset of 21 objects, differing in size, shape and texture.

We present the design of the Snackbot, a robot that will deliver snacks in our university buildings. The robot is intended to provide a useful, continuing service and to serve as a research platform for long-term Human-Robot Interaction. Our design process, which occurred over 24 months, is documented as a contribution for others in HRI who may be developing social robots that offer services. We describe the phases of the design project, and the design decisions and tradeoffs that led to the current version of the robot.

This paper describes the obstacle detection and tracking algorithms developed for Boss, which is Carnegie Mellon University 's winning entry in the 2007 DARPA Urban Challenge. We describe the tracking subsystem and show how it functions in the context of the larger perception system. The tracking subsystem gives the robot the ability to understand complex scenarios of urban driving to safely operate in the proximity of other vehicles. The tracking system fuses sensor data from more than a dozen sensors with additional information about the environment to generate a coherent situational model. A novel multiple-model approach is used to track the objects based on the quality of the sensor data. Finally, the architecture of the tracking subsystem explicitly abstracts each of the levels of processing. The subsystem can easily be extended by adding new sensors and validation algorithms.

This paper describes an algorithm for an autonomous car to identify the shape of a roadway by detecting geometric features via LIDAR. The data from multiple LIDAR are fused together to detect both obstacles as well as geometric features such as curbs, berms, and shoulders. These features identify the boundaries of the roadway and are used by a stochastic state estimator to identify the most likely road shape. This algorithm has been used successfully to allow an autonomous car to drive on paved roadways as well as on off-road trails without requiring different sets of parameters for the different domains.

Effective human/robot interfaces which mimic how humans interact with one another could ultimately lead to robots being accepted in a wider domain of applications. We present a framework for interactive task training of a mobile robot where the robot learns how to do various tasks while observing a human. In addition to observation, the robot listens to the human's speech and interprets the speech as behaviors that are required to be executed. This is especially important where individual steps of a given task may have contingencies that have to be dealt with depending on the situation. Finally, the context of the location where the task takes place and the people present factor heavily into the robot's interpretation of how to execute the task. In this paper, we describe the task training framework, describe how environmental context and communicative dialog with the human help the robot learn the task, and illustrate the utility of this approach with several experimental case studies.

In order for robots to act as valuable assistants for non-expert users, they need to be able to learn new abilities and do so through natural methods of communication. Furthermore, it is often desirable that tasks be learned quickly without having to provide multiple demonstrations. Training should also be conducted in such a way that the user has a clear understanding of the manner in which environmental features affect the behavior of the learned activity, so that execution behavior is predictable. We present an interactive framework for teaching a robot the flow of an activity composed of elements from a set of primitive behaviors and previously trained activities. Conditional branching and looping, order-independent activity execution, and contingency (or interrupt) actions can all be captured by our activity structures. Additional convenience functionality to aid in the training process is also provided. By providing a natural method of communicating production rules analogous to rigid programming structures, well-defined tasks can be trained easily. We demonstrate our task training procedure on a mobile robot.

The essence of the signal-to-symbol problem consists of associating a symbolic description of an object (e.g., a chair) to a signal (e.g., an image) that captures the real object. Robots that interact with humans in natural environments must be able to solve this problem correctly and robustly. However, the problem of providing complete object models a priori to a robot so that it can understand its environment from any viewpoint is extremely difficult to solve. Additionally, many objects have different uses which in turn can cause ambiguities when a robot attempts to reason about the activities of a human and their interactions with those objects. In this paper, we build upon the fact that robots that co-exist with humans should have the ability of observing humans using the different objects and learn the corresponding object definitions. We contribute an object recognition algorithm, FOCUS, that is robust to the variations of signals, combines structure and function of an object, and generalizes to multiple similar objects. FOCUS, which stands for Finding Object Classification through Use and Structure, combines an activity recognizer capable of capturing how an object is used with a traditional visual structure processor. FOCUS learns structural properties (visual features) of objects by knowing first the object's affordance properties and observing humans interacting with that object with known activities. The strength of the method relies on the fact that we can define multiple aspects of an object model, i.e., structure and use, that are individually robust but insufficient to define the object, but can do when combined.

Meetings are a very important part of everyday life for professionals working in universities, companies or governmental institutions. We have designed a physical awareness system called CAMEO (Camera Assisted Meeting Event Observer), a hardware/software system to record and monitor people's activities in meetings. CAMEO captures a high resolution omnidirectional view of the meeting by stitching images coming from almost concentric cameras. Besides recording capability, CAMEO automatically detects people and learns a person-specific facial appearance model (PS-FAM) for each of the participants. The PSFAMs allow more robust/reliable tracking and identification. In this paper, we describe the video-capturing device, photometric/geometric autocalibration process, and the multiple people tracking system. The effectiveness and robustness of the proposed system is demonstrated over several real-time experiments and a large data set of videos.

We present a meeting recorder infrastructure used to record and annotate events that occur in meetings. Multiple data streams are recorded and analyzed in order to infer a higher-level state of the group’s activities. We describe the hardware and software systems used to capture people’s activities as well as the methods used to characterize them.

In this paper, we make two contributions. First, we present a new domain, called Segway Soccer, for investigating the coordination of dynamically formed, mixed human-robot teams within the realm of a team task that requires real-time decision making and response. Segway Soccer is a game of soccer between two teams consisting of Segway riding humans and Segway RMP-based robots. We believe Segway Soccer is the first game involving both humans and robots in cooperative roles and with similar capabilities. In conjunction with this new domain, we present our work towards developing a soccer playing robot using the Segway RMP platform and vision as its primary sensing modality. As Segway Soccer is set in the outdoors, we have developed novel vision algorithms to adapt to changes in lighting conditions. We present the domain of Segway Soccer, its inherent challenges, and our work towards this goal.

This paper compares the performances of several algorithms that address the problem of Simultaneous Localization and Mapping (SLAM) for the case of very small, resource-limited robots. These robots have poor odometry and can typically only carry a single monocular camera. These algorithms do not make the typical SLAM assumption that metric distance/bearing information to landmarks is available. Instead, the robot registers a distinctive sensor "signature", based on its current location, which is used to match robot positions. The performances of a physics-inspired maximum likelihood (ML) estimator, the iterated form of the Extended Kalman Filter (IEKF), and a batch-processed linearized ML estimator are compared under various odometric noise models.

Static cameras are pervasive in a variety of environments. However it remains a challenging problem to extract and reason about high-level features from real-time and continuous observation of an environment. In this paper, we present CAMEO, the Camera Assisted Meeting Event Observer, which is a physical awareness system designed for use by an agent-based electronic assistant. CAMEO is an inexpensive high-resolution omnidirectional vision system designed to be used in meeting environments. The multiple camera design achieves the desired high image resolution and lower cost that can be achieved when compared to traditional omnicameras that make use of a single camera and mirror solution.

We have designed a physical awareness system called CAMEO, the camera assisted meeting event observer, which consists of a multi-camera omnidirectional vision system designed to be used in meeting environments. CAMEO is designed to monitor the activities of people in meetings so that it can generate a semantically-indexed summary of what occurred in the meeting. In this paper, we describe CAMEO's fast people detection and tracking module. This module makes use of a combination of frame differencing, face detection, and adaptive color blob tracking based on mean shift analysis to detect and track people in the panoramic image. We describe this algorithm and present experimental results from captured meeting logs.

To safely and efficiently guide search and rescue operations in disaster areas, gathering of relevant information such as the locations of victims, must occur swiftly. Using the concept of repellent virtual pheromones inspired by insect colony coordination behaviors, miniature robots can be quickly dispersed to survey a disaster site. Assisted by visual servoing, dispersion of the miniature robots can quickly cover an area. An external observer such as another robot or an overhead camera is brought into the control loop to provide each miniature robot estimations of the positions of all the other nearby robots in the robotic team. Each robot can then move away from the other nearby robots, resulting in the robot collective swiftly dispersing through the local area. The technique has been implemented using the miniature scout robots, developed by the Center for Distributed Robotics at the University of Minnesota, which are well-suited to surveillance and reconnaissance missions.

This paper addresses the problem of localization and map construction by a mobile robot in an indoor environment. Instead of trying to build high-fidelity geometric maps, we focus on constructing topological maps, as they are lees sensitive to poor odometry estimates and position errors. We propose a method for incrementally building topological maps for a robot, which uses a panoramic camera to obtain images at various locations along its path and uses the features it tracks in the images to update the topological map. The method is very general and does not require the environment to have uniquely distinctive features.

This paper addresses the problem of simultaneous localization and mapping (SLAM) for the case of very small, resource-limited robots which have poor odometry and can typically only carry a single monocular camera. We propose a modification to the standard SLAM algorithm in which the assumption that the robots can obtain metric distance/bearing information to landmarks is relaxed. Instead, the robot registers a distinctive sensor "signature", based on its current location, which is used to match robot positions. In our formulation of this non-linear estimation problem, we infer implicit position measurements from an image recognition algorithm. The iterated form of the extended Kalman filter (IEKF) is employed to process all measurements.

When a distributed robotic system is assigned to perform reconnaissance or surveillance, restrictions inherent to the design of an individual robot limit the system's performance in certain environments. Finding an ideal portable robotic platform capable of deploying and returning information in spatially restrictive areas is not a simple task. The Scout robot, developed at the University of Minnesota, is a viable robotic platform for these types of missions. The small form factor of the Scout allows for deployment, placement, and concealment of a team of robots equipped with a variety of sensory packages. However, the design of the Scout requires a compromise in power, sensor types, locomotion, and size; together these factors prevent an individual Scout from operating ideally in some environments. Several attempts to address these deficiencies have been implemented and are discussed. Among the prototype solutions are actuating wheels, allowing the Scout to increase ground clearance in varying terrains, a grappling hook enabling the Scout to obtain a position of elevated observation, and infrared emitters to facilitate low light operation.

Search and rescue operations in large disaster sites require quick gathering of relevant information. Both the knowledge of the location of victims and the environmental/structural conditions must be available to safely and efficiently guide rescue personnel. A major hurdle for robots in such scenarios is stairs. A system for autonomous surmounting of stairs is proposed in which a Scout robot jumps from step to step. The robot's height is only about a quarter step in size. Control of the Scout is accomplished using visual servoing. An external observer such as another robot is brought into the control loop to provide the Scout with an estimation of its pose with respect to the stairs. This cooperation is necessary as the Scout must refrain from ill-fated motions that may lead it back down to where it started its ascend. Initial experimental results are presented along with a discussion of the issues involved.

We have designed and built a set of miniature robots called Scouts and have developed a distributed software system to control them. This paper addresses the fundamental choices we made in the design of the control software, describes experimental results in a surveillance task, and analyzes the factors that affect robot performance. Space and power limitations on the Scouts severely restrict the computational power of their on-board computers, requiring a proxy-processing scheme in which the robots depend on remote computers for their computing needs. While this allows the robots to be autonomous, the fact that robots' behaviors are executed remotely introduces an additional complication-sensor data and motion commands have to be exchanged using wireless communications channels. Communications channels cannot always be shared, thus requiring the robots to obtain exclusive access to them. We present experimental results on a surveillance task in which multiple robots patrol an area and watch for motion. We discuss how the limited communications bandwidth affects robot performance in accomplishing the task, and analyze how performance depends on the number of robots that share the bandwidth.

Using robots for surveillance and reconnaissance applications requires a versatile connection between the human operator and robotic hardware. Some application domains require a fully teleoperated system while others may benefit by giving robots more autonomy. This paper describes a robotic control architecture which merges both paradigms. The whole scheme is implemented using the miniature Scout robot and involves a suite of user interfaces that can be tailored to specific surveillance and reconnaissance missions. Hardware capabilities are presented and a visual servoing strategy, important for semi-autonomous Scout operation, is discussed.

We describe a distributed software system for controlling a group of miniature robots using a very low capacity communication system. Space and power limitations on the miniature robots drastically restrict the capacity of the communication system and require sharing bandwidth and other resources among the robots. We have developed a process management/scheduling system that dynamically assigns resources to each robot in an attempt to maximize the utilization of the available resources while still maintaining a priori behavior priorities. We describe a surveillance task in which the robots patrol an area and watch for motion, and present experimental results.

Presents a miniature robotic system ("scout") useful for reconnaissance and surveillance missions. A large number of scout robots are deployed and controlled by humans and/or larger "ranger" robots. The specially designed and constructed scouts are extremely small (roughly 116cc volume) yet are readily deployable (by tossing or launching), have multiple mobility modes, have multiple sensing capabilities, can transmit and receive data and instructions, and have a limited capability for autonomous action. The rangers are significantly larger vehicles, based on a commercial-off-the-shelf platform, augmented with scout launchers, radios, and additional sensors. Together, the scouts and rangers form a hierarchical team capable of carrying out complex missions in a wide variety of environments.

This paper describes a demonstration-based programming system in which a mobile robot observes the actions of a human performing a multi-step task. From these observations, the robot determines which of its pre-learned capabilities are required to replicate the task and in what sequence they must be ordered. The focus of this paper is on the hidden Markov model method used to learn and classify the actions as "gestures". A preliminary system demonstration is also described in which the robot observes the human performing a block distribution task. During the demonstration, the robot actively follows the demonstrator to maintain its vantage point and to infer spatial relationships.