Augmented reality (AR) is an interface that can potentially provide more intuitive and immersive experiences, particularly for automotive applications. In this paper we present the Personal Navi, a vehicular AR navigational aid designed for use with a see-thru 3D volumetric Head Up Display (HUD). In one study, we found that the Personal Navi visuals helped participants recognize turn locations earlier than when provided with conventional navigation aids alone. The interface also helped keep users' eyes up and fixated on the driving environment more. In a second study, we tested participants' depth perception of AR visuals provided with our prototype 3D-HUD. We displayed the same visuals with a fixed focal depth display (2D-HUD) and found that for applications such as Personal Navi, where the spatial location of AR visuals are an important component of the interface, a dynamically adjustable focal plane (3D-HUD) is necessary for proper registration. We argue that AR-based HUD navigational aids should be implemented only with 3D-HUDs because of inferior depth perception with 2D-HUDs.

With the capability of fast, wireless communication, combined with cloud and location-based services, modern drivers can potentially access a wide variety of information about their automobile's environment. This paper presents a system for information query by the driver by using a simple pointing mechanism, combined with visual feedback in the form of a 3-D Head-up Display (3D-HUD). Because of its 3-D properties, the HUD can also be used for Augmented Reality (AR) as it allows physical elements in the driver's field of view to be annotated with computer graphics. The combination of simple natural user input tailored for the constraints of the driver with a see-thru 3D-HUD allows drivers to query information while minimizing visual and manual distraction.

Making left turns across oncoming traffic without a protected left-turn signal is a significant safety concern at intersections. In a left turn situation, the driver typically does not have the right of way and must determine when to initiate the turn maneuver safely. It has been reported that a driver's inability to correctly judge the velocity and time gap of the oncoming vehicles is a major cause of left turn crashes. Although the position and velocity of surrounding vehicles is available using camera and laser based vehicle detection and tracking, methods on how to effectively communicate such information to help the driver have been relatively under-explored. In this paper, we describe a left turn aid that displays a 3 second projected path of the oncoming vehicle in the driver's environment with a 3D Head-Up Display (3D-HUD). Utilizing the abilities of our 3D-HUD to show the projected path in Augmented Reality (AR) could help increase driver intuition and alleviate visual distraction as compared to other possible non-AR solutions. Through an iterative process utilizing early user feedback, the design of the left turn aid was refined to interfere less with the driver view and be more effective. A pilot study has been designed for a driving simulation environment and can be used to evaluate the potential of the proposed AR left turn aid in helping the driver be more cautious or efficient when turning left.

This proxemics study examines whether the physical distance between robots and humans differ based on the following factors: 1) age: children vs. adults, 2) who initiates the approach: humans approaching the robot vs. robot approaching humans, 3) prompting: verbal invitation vs. non-verbal gesture (e.g., beckoning), and 4) informing: announcement vs. permission vs. nothing. Results showed that both verbal and non-verbal prompting had significant influence on physical distance. Physiological data is also used to detect the appropriate timing of approach for a more natural and comfortable interaction.

In-vehicle contextual augmented reality (I-CAR) has the potential to provide novel visual feedback to drivers for an enhanced driving experience. To enable I-CAR, we present a parametrized road trench model (RTM) for dynamically extracting display surfaces from a driver's point of view that is adaptable to constantly changing road curvature and intersections. We use computer vision algorithms to analyze and extract road features that are used to estimate the parameters of the RTM. GPS coordinates are used to quickly compute lighting parameters for shading and shadows. Novel driver-based applications that use the RTM are presented.

The role of context in recognizing a person's affect is being increasingly studied. In particular, context arising from the presence of multi-modal information such as faces, speech and head pose has been used in recent studies to recognize facial expressions. In most approaches, the modalities are independently considered and the effect of one modality on the other, which we call inter-modal influence (e.g. speech or head pose modifying the facial appearance) is not modeled. In this paper, we describe a system that utilizes context from the presence of such inter-modal influences to recognize facial expressions. To do so, we use 2-D contextual masks which are activated within the facial expression recognition pipeline depending on the prevailing context. We also describe a framework called the Context Engine. The Context Engine offers a scalable mechanism for extending the current system to address additional modes of context that may arise during human-machine interactions. Results on standard data sets demonstrate the utility of modeling inter-modal contextual effects in recognizing facial expressions.

Two studies examined the different features of humanoid robots and the influence on children's affective behavior. The first study looked at interaction styles and general features of robots. The second study looked at how the robot's attention influences children's behavior and engagement. Through activities familiar to young children (e.g., table setting, story telling), the first study found that cooperative interaction style elicited more oculesic behavior and social engagement. The second study found that quality of attention, type of attention, and length of interaction influences affective behavior and engagement. In the quality of attention, Wizard-of-Oz (woz) elicited the most affective behavior, but automatic attention worked as well as woz when the interaction was short. The type of attention going from nonverbal to verbal attention increased children's oculesic behavior, utterance, and physiological response. Affective interactions did not seem to depend on a single mechanism, but a well-chosen confluence of technical features.

This paper considers a shortcutting heuristic to smooth jerky trajectories for many-DOF robot manipulators subject to collision constraints, velocity bounds, and acceleration bounds. The heuristic repeatedly picks two points on the trajectory and attempts to replace the intermediate trajectory with a shorter, collision-free segment. Here, we construct segments that interpolate between endpoints with specified velocity in a time-optimal fashion, while respecting velocity and acceleration bounds. These trajectory segments consist of parabolic and straight-line curves, and can be computed in closed form. Experiments on reaching tasks in cluttered human environments demonstrate that the technique can generate smooth, collision-free, and natural-looking motion in seconds for a PUMA manipulator and the Honda ASIMO robot.

We present a model that is capable of synchronizing expressive gestures with speech. The model, implemented on a Honda humanoid robot, can generate a full range of gesture types, such as emblems, iconic and metaphoric gestures, deictic pointing and beat gestures. Arbitrary input text is analyzed with a part-of-speech tagger and a text-to-speech engine for timing information of spoken words. In addition, style tags can be optionally added to specify the level of excitement or topic changes. The text, combined with any tags, is then processed by several grammars, one for each gesture type to produce several candidate gestures for each word of the text. The model then selects probabilistically amongst the gesture types based on the desired degree of expressivity. Once a gesture type is selected, it coincides with a particular gesture template, consisting of trajectory curves that define the gesture. Speech timing patterns and style parameters are used to modulate the shape of the curve before it sent to the whole body control system on the robot. Evaluation of the model's parameters were performed, demonstrating the ability of observers to differentiate varying levels of expressiveness, excitement and speech synchronization. Modification of gesture speed for trajectory tracking found that positive associations like happiness and excitement accompanied faster speeds, with negative associations like sadness or tiredness occurred at slower speeds.

Extended human-robot interactions possess unique aspects which are not exhibited in short-term interactions spanning a few minutes or extremely long-term spanning days. In order to comprehensively monitor such interactions, we need special recording mechanisms which ensure the interaction is captured at multiple spatio-temporal scales, viewpoints and modalities(audio, video, physio). To minimize cognitive burden, we need tools which can automate the process of annotating and analyzing the resulting data. In addition, we also require these tools to be able to provide a unified, multi-scale view of the data and help discover patterns in the interaction process. In this paper, we describe recording and analysis tools which are helping us analyze extended human-robot interactions with children as subjects. We also provide some experimental results which highlight the utility of such tools.

Humanoid robots consist of biologically inspired features, human-like appearance, and intelligent behavior that naturally elicit social responses. Complex interactions are now possible, where children interact and learn from robots. A pilot study attempted to determine which features in robots led to changes in learning and behavior. Three common learning styles, lecture, cooperative, and self-directed, were implemented into ASIMO to see if children can learn from robots. General features such as monotone robot-like voice and human-like voice were compared. Thirty-seven children between the ages 4-to 10- years participated in the study. Each child engaged in a table-setting task with ASIMO that exhibited different learning styles and general features. Children answered questions in relation to a table-setting task with a learning measure. Promissory evidence shows that learning styles and general features matter especially for younger children.

We present a human-robot interactive scenario consisting of a memory card game between Hondapsilas humanoid robot ASIMO and a human player. The game features perception exclusively through ASIMOpsilas on-board cameras and both reactive and proactive behaviors specific to different situational contexts in the memory game. ASIMO is able to build a dynamic environmental map of relevant objects in the game such as the table and card layout as well as understand activities from the player such as pointing at cards, flipping cards and removing them from the table. Our system architecture, called the Cognitive Map, treats the memory game as a multi-agent system, with modules acting independently and communicating with each other via messages through a shared blackboard system. The game behavior module can model game state and contextual information to make decisions based on different pattern recognition modules. Behavior is then sent through high-level command interfaces to be resolved into actual physical actions by the robot via a multi-modal communication module. The experience gained in modeling this interactive scenario will allow us to reuse the architecture to create new scenarios and explore new research directions in learning how to respond to new interactive situations.

The successful acquisition and organization of a large number of skills for humanoid robots can be facilitated with a collection of performable tasks organized in a task matrix. Tasks in the matrix can utilize particular preconditions and in conditions to enable execution, motion trajectories to specify desired movement, and references to other tasks to perform subtasks. Interaction between the matrix and external modules such as goal planners is achieved via a high-level interface that categorizes a task using its semantics and execution parameters, allowing queries on the matrix to be performed using different selection criteria. Performable tasks are stored in an XML-based file format that can be readily edited and processed by other applications. In its current implementation, the matrix is populated with sets of primitive tasks (e.g. reaching, grasping, arm-waving) and macro tasks that reference multiple primitive tasks (pick-and-place and facing-and-waving)

Reaching is a critical task for humanoid robots, requiring the application of state-of-the-art algorithms for motion planning and inverse kinematics. Practical algorithms for solving these subproblems are currently not complete, offering resolution completeness or probabilistic completeness instead. While this lesser provision of completeness is acceptable in many cases, naively combining state-of-the-art approaches in motion planning and inverse kinematics can lead to a method that provides no measure of completeness. We present a probabilistically complete solution to the reaching problem for humanoid robots in static environments, and evaluate it against two other methods using a kinematically simulated humanoid in a virtual environment

Simulation of the musculoskeletal system has important applications in biomechanics, biomedical engineering, surgery simulation, and computer graphics. The accuracy of the muscle, bone, and tendon geometry as well as the accuracy of muscle and tendon dynamic deformation are of paramount importance in all these applications. We present a framework for extracting and simulating high resolution musculoskeletal geometry from the segmented visible human data set. We simulate 30 contact/collision coupled muscles in the upper limb and describe a computationally tractable implementation using an embedded mesh framework. Muscle geometry is embedded in a nonmanifold, connectivity preserving simulation mesh molded out of a lower resolution BCC lattice containing identical, well-shaped elements, leading to a relaxed time step restriction for stability and, thus, reduced computational cost. The muscles are endowed with a transversely isotropic, quasi-incompressible constitutive model that incorporates muscle fiber fields as well as passive and active components. The simulation takes advantage of a new robust finite element technique that handles both degenerate and inverted tetrahedra.

We present a general joint component framework model that is capable of exhibiting complex behavior of joints in articulated figures. The joints are capable of handling non-orthogonal, non-intersecting axes of rotation and changing joint centers that are often found in the kinematics of real anatomical joints. The adjustment of joint articulation is done with a relatively small set of intuitive parameters compared to the number of articulations in the motions they parameterize. This is done by making various linear and nonlinear joint dependencies implicit within our framework. An animator is restricted from putting the skeleton in an infeasible pose. We have used our joint framework model to successfully model highly-articulated complex joints such as the human spine and shoulder.