This paper explores the feasibility of employing EEG-based intention detection for real-time robot assistive control. We focus on predicting and distinguishing motor intentions of left/right arm movements by presenting: i) an offline data collection and training pipeline, used to train a classifier for left/right motion intention prediction, and ii) an online real-time prediction pipeline leveraging the trained classifier and integrated with an assistive robot. Central to our approach is a rich feature representation composed of the tangent space projection of time-windowed sample covariance matrices from EEG filtered signals and derivatives; allowing for a simple SVM classifier to achieve unprecedented accuracy and real-time performance. In pre-recorded real-time settings (160 Hz), a peak accuracy of 86.88% is achieved, surpassing prior works. In robot-in-the-loop settings, our system successfully detects intended motion solely from EEG data with 70% accuracy, triggering a robot to execute an assistive task. We provide a comprehensive evaluation of the proposed classifier.

Robotic grasping of 3D deformable objects is critical for real-world applications such as food handling and robotic surgery. Unlike rigid and articulated objects, 3D deformable objects have infinite degrees of freedom. Fully defining their state requires 3D deformation and stress fields, which are exceptionally difficult to analytically compute or experimentally measure. Thus, evaluating grasp candidates for grasp planning typically requires accurate, but slow 3D finite element method (FEM) simulation. Sampling-based grasp planning is often impractical, as it requires evaluation of a large number of grasp candidates. Gradient-based grasp planning can be more efficient, but requires a differentiable model to synthesize optimal grasps from initial candidates. Differentiable FEM simulators may fill this role, but are typically no faster than standard FEM. In this work, we propose learning a predictive graph neural network (GNN), DefGraspNets, to act as our differentiable model. We train DefGraspNets to predict 3D stress and deformation fields based on FEM-based grasp simulations. DefGraspNets not only runs up to 1500x faster than the FEM simulator, but also enables fast gradient-based grasp optimization over 3D stress and deformation metrics. We design DefGraspNets to align with real-world grasp planning practices and demonstrate generalization across multiple test sets, including real-world experiments.

The automated cleaning of surfaces such as furniture, bathroom sinks, and even human bodies is challenging due to the three-dimensional nature of their geometries. Yet, enabling robots to effectively and safely perform these tasks would not only reduce user efforts spent on household cleaning chores, but would also alleviate the strenuous workload of caretakers as the elderly population continues to grow at an unprecedented rate. In this work, we unify the applications of wiping objects and bathing humans as a general contour-following problem. To this end, we utilize a depth camera-based soft tactile sensor to extract the contact geometries and force-correlated measures during interaction between the robot and the target object or body part, and design a general contour-following controller that not only maintains contact with the target throughout the cleaning process, but also regulates the amount of force applied. Our system enables successful cleaning of pipes, shelving, and even human limbs and torsos without the need for data-driven methods such as deep learning, upon which the majority of existing works have relied.

Soft optical tactile sensors enable robots to manipulate deformable objects by capturing important features such as high-resolution contact geometry and estimations of object compliance. This work presents a variable stiffness soft tactile end-effector called StRETcH, a Soft to Resistive Elastic Tactile Hand, that is easily manufactured and integrated with a robotic arm. An elastic membrane is suspended between two robotic fingers, and a depth sensor capturing the deformations of the elastic membrane enables sub-millimeter accurate estimates of contact geometries. The parallel-jaw gripper varies the stiffness of the membrane by uniaxially stretching it, which controllably modulates StRETcH’s effective modulus from approximately 4kPa to 9kPa. This work uses StRETcH to reconstruct the contact geometry of rigid and deformable objects, estimate the stiffness of four balloons filled with different substances, and manipulate dough into a desired shape.

Deformable solid objects such as clay or dough are prevalent in industrial and home environments. However, robotic manipulation of such objects has largely remained unexplored in literature due to the high complexity involved in representing and modeling their deformation. This work addresses the problem of shaping elasto-plastic dough by proposing to use a novel elastic end-effector to roll dough in a reinforcement learning framework. The transition model for the end-effector-to-dough interactions is learned from one hour of robot exploration, and doughs of different hydration levels are rolled out into varying lengths. Experimental results are encouraging, with the proposed framework accomplishing the task of rolling out dough into a specified length with 60% fewer actions than a heuristic method. Furthermore, we show that estimating stiffness using the soft end-effector can be used to effectively initialize models, improving robot performance by approximately 40% over incorrect model initialization.

The safety and comfort of drivers have been improved over the decades as a result of our broadened understanding of driver modeling and behavior prediction. Despite these remarkable advances in autonomous and interactive systems, there is a significant lack of approaches that consider the passengers and the vehicle as components of a dynamical vibro-acoustical system. Sound in vehicles is not only informative of the state of the vehicle and the environment, but can also critically affect the driver's performance, attention, and comfort. This paper aims to investigate the interplay between the perceived sounds of a vehicle and psychoacoustic annoyance (PA) metrics. Our goal is to create an intelligent agent that would act to improve driving pleasantness through acoustic-driven learning. To tackle the problem of choosing the correct actions to reduce the acoustic annoyance, the paper presents a method based on reinforcement learning that learns from the environment, i.e., the vehicle interior. The method actively changes the state inside the vehicle (e.g., closing or opening the window and choosing the cruise speed) in order to minimize acoustic annoyance experienced by the driver. The results of this work, performed using the GTA V simulator, showed that the trained agent successfully learned to take the correct actions to reduce PA metrics. The paper also present to the community a new multi-modal dataset composed of several rides on a real vehicle and an in-depth analysis of the influence of vehicle's signal on the acoustic annoyance.

If robots and humans are to coexist and cooperate in society, it would be useful for robots to be able to engage in tactile interactions. Touch is an intuitive communication tool as well as a fundamental method by which we assist each other physically. Tactile abilities are challenging to engineer in robots, since both mechanical safety and sensory intelligence are imperative. Existing work reveals a trade-off between these principles- tactile interfaces that are high in resolution are not easily adapted to human-sized geometries, nor are they generally compliant enough to guarantee safety. On the other hand, soft tactile interfaces deliver intrinsically safe mechanical properties, but their non-linear characteristics render them difficult for use in timely sensing and control. We propose a robotic system that is equipped with a completely soft and therefore safe tactile interface that is large enough to interact with human upper limbs, while producing high resolution tactile sensory readings via depth camera imaging of the soft interface. We present and validate a data-driven model that maps point cloud data to contact forces, and verify its efficacy by demonstrating two real-world applications. In particular, the robot is able to react to a human finger's pokes and change its pose based on the tactile input. In addition, we also demonstrate that the robot can act as an assistive device that dynamically supports and follows a human forearm from underneath.

Granular media (e.g., cereal grains, plastic resin pellets, and pills) are ubiquitous in robotics-integrated industries, such as agriculture, manufacturing, and pharmaceutical development. This prevalence mandates the accurate and efficient simulation of these materials. This work presents a software and hardware framework that automatically calibrates a fast physics simulator to accurately simulate granular materials by inferring material properties from real-world depth images of granular formations (i.e., piles and rings). Specifically, coefficients of sliding friction, rolling friction, and restitution of grains are estimated from summary statistics of grain formations using likelihood-free Bayesian inference. The calibrated simulator accurately predicts unseen granular formations in both simulation and experiment; furthermore, simulator predictions are shown to generalize to more complex tasks, including using a robot to pour grains into a bowl, as well as to create a desired pattern of piles and rings.

Deploying robot learning frameworks in unconstrained environments requires robustness and tractability. We must not only equip the robot with a sufficient range of sensing capabilities, but also provide training data in a sample-efficient manner. To this end, we identify and address a need specifically in robot learning from demonstration (LfD) literature to account for not only end-effector pose and wrench signals, but also tactile signals for contact. While traditional pose and wrench signals have proven to be sufficient for robots to learn basic position and force-control behaviors, they are inherently too constraining for the learning of general manipulation tasks. In particular, useful manipulation tasks often rely on the geometry of the contact interaction. To explore the value of geometry-based tactile signals, we utilize a LfD framework built upon hidden Markov models and Gaussian mixture regression, adapt it to our robotic system equipped with a soft tactile sensor, and validate its performance with an edge-following task and a manipulation task involving different object geometries.

There are a lack of quantitative measures for clinically assessing upper limb function. Conventional biomechanical performance measures are restricted to specialist labs due to hardware cost and complexity, while the resulting measurements require specialists for analysis. Depth cameras are low cost and portable systems that can track surrogate joint positions. However, these motions may not be biologically consistent, which can result in noisy, inaccurate movements. This paper introduces a rigid body modelling method to enforce biological feasibility of the recovered motions. This method is evaluated on an existing depth camera assessment: the reachable workspace (RW) measure for assessing gross shoulder function. As a rigid body model is used, position estimates of new proximal targets can be added, resulting in a proximal function (PF) measure for assessing a subject's ability to touch specific body landmarks. The accuracy, and repeatability of these measures is assessed on ten asymptomatic subjects, with and without rigid body constraints. This analysis is performed both on a low-cost depth camera system and a gold-standard active motion capture system. The addition of rigid body constraints was found to improve accuracy and concordance of the depth camera system, particularly in lateral reaching movements. Both RW and PF measures were found to be feasible candidates for clinical assessment, with future analysis needed to determine their ability to detect changes within specific patient populations.

At the core of most three-dimensional alignment and tracking tasks resides the critical problem of point correspondence. In this context, the design of descriptors that efficiently and uniquely identifies keypoints, to be matched, is of central importance. Numerous descriptors have been developed for dealing with affine/perspective warps, but few can also handle non-rigid deformations. In this paper, we introduce a novel binary RGB-D descriptor invariant to isometric deformations. Our method uses geodesic isocurves on smooth textured manifolds. It combines appearance and geometric information from RGB-D images to tackle non-rigid transformations. We used our descriptor to track multiple textured depth maps and demonstrate that it produces reliable feature descriptors even in the presence of strong non-rigid deformations and depth noise. The experiments show that our descriptor outperforms different state-of-the-art descriptors in both precision-recall and recognition rate metrics. We also provide to the community a new dataset composed of annotated RGB-D images of different objects (shirts, cloths, paintings, bags), subjected to strong non-rigid deformations, to evaluate point correspondence algorithms.

As the demand for robotic applications in unconstrained and dynamic environments rises, so does the benefit of advancing the state of the art in soft robotic technologies. However, the complex capabilities of soft robots elicited by their high-dimensional, non-linear characteristics simultaneously yield difficult challenges in control and sensing. Moreover, embedding tactile sensing capabilities in soft materials is often expensive and difficult to fabricate. In recent years, however, the invention of small-scale depth-sensing cameras introduced a promising channel for soft tactile sensor design. In this work, we propose a novel soft device inspired by the human fingertip that not only utilizes a small depth camera as the perception mechanism, but also possesses compliance-modulating capabilities. We demonstrate its ability to accurately estimate contact regions upon interaction with an external obstacle, and show that the estimation sensitivity can be modulated via internal fluid states. In addition, we determine an empirical model of the device's force-deformation characteristics under simplifying assumptions, and validate its performance with real-time force matching control experiments.

Despite the impressive progress being made in autonomous vehicles, human drivers will remain ubiquitous in the imminent years. Therefore, intelligent hybrid vehicular systems must be aware of the interactions between humans and the environment (e.g., sound, vibration, speed, etc.). In this paper, we evaluate the effect of acoustic annoyance on drivers in a real-world driving study. We found significant differences in driving styles elicited by annoying acoustics and present an online classifier that uses onboard inertial measurement unit measurements to distinguish whether a driver is annoyed with 77% accuracy. Moreover, we directly measured the forces applied on the passenger with a pressure mat lined on the car seat, and empirically confirm that our proposed passenger dynamics model is reasonable. However, due to our acoustically induced driving styles not being polarizing enough, we were unable to show that passengers' self-reported ride comfort changed with acoustic annoyance.

Service robots will require several important manipulation skills, including the ability to accurately measure and pour liquids. Prior work on robotic liquid pouring has primarily focused on visual techniques for sensing liquids, but these techniques fall short when liquids are obscured by opaque or closed containers. This paper proposes a complementary method for liquid perception via haptic sensing. The robot moves a container through a series of tilting motions and observes the wrenches induced at the manipulator's wrist by the liquid's shifting center of mass. That data is then analyzed with a physics-based model to estimate the liquid's mass and volume. In experiments, this method achieves error margins of less than lg and 2mL for an unknown liquid in a 600mL cylindrical container. The model can also predict the viscosity of fluids, which can be used for classifying water, oil, and honey with an accuracy of 98%. The estimated volume is used to precisely pour 100mL of water with less than 4% average error.

Kinetic and dynamic motion analysis provides quantitative, functional assessments of human ability that are unobtainable through static imaging methods or subjective surveys. While biomechanics facilities are equipped to perform this measurement and analysis, the clinical translation of these methods is limited by the specialized skills and equipment needed. This paper presents and validates a method for estimating dynamic effects such as joint torques and body momenta using a single depth camera. An allometrically scaled, sagittal plane dynamic model is used to estimate the joint torques at the ankles, knees, hips, and low back, as well as the torso momenta, and shear and normal loads at the L5-S1 disk. These dynamic metrics are applied to the sit-to-stand motion and validated against a gold-standard biomechanical system consisting of full-body active motion-capture and force-sensing systems. The metrics obtained from the proposed method were found to have excellent concordance with peak metrics that are consistent with prior biomechanical studies. This suggests the feasibility of using this system for rapid clinical assessment, with applications in diagnostics, longitudinal tracking, and quantifying patient recovery.

The study of joint kinematics and dynamics has broad clinical applications, including the identification of pathological motions or compensation strategies and the analysis of dynamic stability. High-end motion capture systems, however, are expensive and require dedicated camera spaces with lengthy setup and data processing commitments. Depth cameras, such as the Microsoft Kinect, provide an inexpensive, marker-free alternative at the sacrifice of joint-position accuracy. In this work, we present a fast framework for adding biomechanical constraints to the joint estimates provided by a depth camera system. We also present a new model for the lower lumbar joint angle. We validate key joint position, angle, and velocity measurements against a gold standard active motion-capture system on ten healthy subjects performing sit to stand (STS). Our method showed significant improvement in mean absolute error and intraclass correlation coefficients for the recovered joint angles and position-based metrics. These improvements suggest that depth cameras can provide an accurate and clinically viable method of rapidly assessing the kinematics and kinetics of the STS action, providing data for further analysis using biomechanical or machine learning methods.

Our goal is for people to be physically comfortable when taking objects from robots. This puts a burden on the robot to hand over the object in such a way that a person can easily reach it, without needing to strain or twist their arm - a way that is conducive to ergonomic human grasping configurations. To achieve this, the robot needs to understand what makes a configuration more or less ergonomic to the person, i.e. their ergonomic cost function. In this work, we formulate learning a person's ergonomic cost as an online estimation problem. The robot can implicitly make queries to the person by handing them objects in different configurations, and gets observations in response about the way they choose to take the object. We compare the performance of both passive and active approaches for solving this problem in simulation, as well as in an in-person user study.

Surface electromyography is currently the sensing modality of choice for control of biosignal-driven prostheses and exoskeletons; however, the sensor's noisy and aggregate nature inhibits collection of distinguishable signal streams to robustly manipulate multiple device degrees of freedom (DoF). We here explore 2D B-mode ultrasound as an alternative source of muscle activation data (namely, muscle deformation) that can be more precisely localized, allowing for the theoretical collection of multiple naturally-varying signals that could be used to control high-DoF assistive devices. We here present a proof-of-concept study showing a) the observability of muscle deformation via ultrasound, and b) novel descriptions of the spatially-varying nature of the signal. These analyses are accomplished through the study of nine volumetric scans of the biceps brachii under varied elbow angle and loading conditions, collected and spatially localized using an ultrasound scanner and motion capture. We here establish the feasibility of measuring several force-associated deformation signals (including muscle cross-sectional area and thickness) via real-time ultrasound scanning and quantify the spatial variation of these signals. Additionally, we propose future applications for both our signal characterizations and the generated muscle volume data set, including better design of assistive device sensor locations and validation of existing muscle deformation models.

Soft material robots are attractive for safe interaction with humans and unstructured environments due to their compliance and low intrinsic stiffness and mass. These properties enable new capabilities such as the ability to conform to environmental geometry for tactile sensing and to undergo large shape changes for actuation. Due to the complex coupling between sensing and actuation in high-dimensional nonlinear soft systems, prior work in soft robotics has primarily focused on either sensing or actuation. This paper presents SOFTcell, a novel controllable stiffness tactile device that incorporates both optical sensing and pneumatic actuation. We report details on the device's design and implementation and analyze results from characterization experiments on sensitivity and performance, which show that SOFTcell can controllably increase its effective modulus from 4.4kPa to 46.1kPa. Additionally, we demonstrate the utility of SOFTcell for grasping in a reactive control task in which tactile data is used to detect fingertip shear as a grasped object slips, and cell pressurization is used to prevent the slip without the need to adjust fingertip position.

Given the current capabilities of autonomous vehicles, one can easily imagine autonomous vehicles being released on the road in the near future. However, it can be assumed that this transition will not be instantaneous, suggesting that autonomous vehicles will have to be capable of driving in a mixed environment, with both humans and autonomous vehicles. To guarantee smooth integration and maintain the nuanced social interactions on the road, a shared mental model must be developed. This means that the behaviors of human-driven vehicles and their typical interactions in collaborative maneuvers must be modeled and understood in an accurate and precise manner. Then, by integrating such models into autonomous planning, we can develop control frameworks that mimic this shared understanding. We present a driver modeling framework that estimates an empirical reachable set to capture typical lane changing behaviors. This method can predict driver behaviors with up to 90% accuracy and cumulative errors less than 1 m. Leveraging this driver model in an optimization-based trajectory planning framework, we can generate trajectories that are similar to those performed by humans. By using this modeling and planning framework, we can improve understanding and integration of nuanced interactions to improve collaboration between humans and autonomy.

Given the current capabilities of autonomous vehicles, one can easily imagine autonomy released on the road in the near future. However, it can be assumed that the transition will not be instantaneous, meaning they will have to be capable of driving well in a mixed environment, with both humans and other autonomous vehicles on the road. This leaves a number of concerns for autonomous vehicles in terms of dealing with human uncertainty and understanding of cooperation on the road. This work demonstrates the need for focusing on communication and collaboration between autonomy and human drivers. After analyzing how drivers perform cooperative maneuvers (e.g. lane changing), key cues were identified for conveying intent through nonverbal communication. It was found that human observers can predict lane changes with over two seconds in prior to the lane departure, without use of a turning signal. Building on this concept, an autonomous control scheme is proposed that aims to capture these subtle motions before executing a lane change. To compare the proposed human-inspired methods, three possible control schemes for autonomous vehicles are implemented for a validation study on human subjects to provide feedback on their experience. By properly conveying intent through nuanced trajectory planning, we show that drivers can predict the autonomous vehicle's actions with 40% increase in prediction time when compared to traditional control methods, both as a passenger and while observing the autonomous vehicle.

The performance of human-robot collaboration tasks can be improved by incorporating predictions of the human collaborator's movement intentions. These predictions allow a collaborative robot to both provide appropriate assistance and plan its own motion so it does not interfere with the human. In the specific case of human reach intent prediction, prior work has divided the task into two pieces: recognition of human activities and prediction of reach intent. In this work, we propose a joint model for simultaneous recognition of human activities and prediction of reach intent based on skeletal pose. Since future reach intent is tightly linked to the action a person is performing at present, we hypothesize that this joint model will produce better performance on the recognition and prediction tasks than past approaches. In addition, our approach incorporates a simple human kinematic model which allows us to generate features that compactly capture the reachability of objects in the environment and the motion cost to reach those objects, which we anticipate will improve performance. Experiments using the CAD-120 benchmark dataset show that both the joint modeling approach and the human kinematic features give improved F1 scores versus the previous state of the art.

In light of growing attention of intelligent vehicle systems, we have present an assessment of methods for driver models that predict driver behaviors. This work looks at varying datasets to see the affects on intent detection algorithms. The motivation is to understand and assess how data is mapped from datasets to discrete states or modes of intent. Using a model of a human driver's decision making process to estimate intent, we build techniques for analyzing and learning human behaviors to improve understanding. We derive models based off of human perception and interaction with the environment (e.g. other vehicles on the road), that is generalizable and flexible enough to detect intent across different drivers. The resulting detection scheme is able to determine driver intent with high accuracy across multiple drivers, relying on a large dataset consisting of lane changes under varying environmental constraints. By comparing different labeling methods, we assess the effectiveness of learned models under different class variations. This allows us to derive accurate and general models for detecting intent that rely on the subtle variations and behaviors that humans exhibit while driving.

We present a smartphone-based system for real-time tele-monitoring of physical activity in patients with chronic heart-failure (CHF). We recently completed a pilot study with 15 subjects to evaluate the feasibility of the proposed monitoring in the real world and examine its requirements, privacy implications, usability, and other challenges encountered by the participants and healthcare providers. Our tele-monitoring system was designed to assess patient activity via minute-by-minute energy expenditure (EE) estimated from accelerometry. In addition, we tracked relative user location via global positioning system (GPS) to track outdoors activity and measure walking distance. The system also administered daily surveys to inquire about vital signs and general cardiovascular symptoms. The collected data were securely transmitted to a central server where they were analyzed in real time and were accessible to the study medical staff to monitor patient health status and provide medical intervention if needed. Although the system was designed for tele-monitoring individuals with CHF, the challenges, privacy considerations, and lessons learned from this pilot study apply to other chronic health conditions, such as diabetes and hypertension, that would benefit from continuous monitoring through mobile-health (mHealth) technologies.

Although the positive effects of exercise on the well-being and quality of independent living for older adults are well accepted, many elderly individuals lack access to exercise facilities, or the skills and motivation to perform exercise at home. To provide a more engaging environment that promotes physical activity, various fitness applications have been proposed. Many of the available products, however, are geared toward a younger population and are not appropriate or engaging for an older population. To address these issues, we developed an automated interactive exercise coaching system using the Microsoft Kinect. The coaching system guides users through a series of video exercises, tracks and measures their movements, provides real-time feedback, and records their performance over time. Our system consists of exercises to improve balance, flexibility, strength, and endurance, with the aim of reducing fall risk and improving performance of daily activities. In this paper, we report on the development of the exercise system, discuss the results of our recent field pilot study with six independently living elderly individuals, and highlight the lessons learned relating to the in-home system setup, user tracking, feedback, and exercise performance evaluation.

In this paper, we propose a method for temporal segmentation of human repetitive actions based on frequency analysis of kinematic parameters, zero-velocity crossing detection, and adaptive k-means clustering. Since the human motion data may be captured with different modalities which have different temporal sampling rate and accuracy (e.g., Optical motion capture systems vs. Microsoft Kinect), we first apply a generic full-body kinematic model with an unscented Kalman filter to convert the motion data into a unified representation that is robust to noise. Furthermore, we extract the most representative kinematic parameters via the primary frequency analysis. The sequences are segmented based on zero-velocity crossing of the selected parameters followed by an adaptive k-means clustering to identify the repetition segments. Experimental results demonstrate that for the motion data captured by both the motion capture system and the Microsoft Kinect, our proposed algorithm obtains robust segmentation of repetitive action sequences.

In order to develop provably safe human-in-the-loop systems, accurate and precise models of human behavior must be developed. Driving is a good example of such a system because the driver has full control of the vehicle, and her likely actions are highly dependent on her mental state and the context of the current situation. This paper presents a testbed for collecting driver data that allows us to collect realistic data, while maintaining safety and control of the environmental surroundings. We extend previous work that focuses on set predictions consisting of trajectories observed from the nonlinear dynamics and behaviors of the human driven car, accounting for the driver mental state, the context or situation that the vehicle is in, and the surrounding environment in both highway and intersection scenarios. This allows us to predict driving behavior over long time horizons with extremely high accuracy. By using this realistic data and flexible algorithm, a precise and accurate driver model can be developed that is tailored to an individual and usable in semi-autonomous frameworks.

We present a framework for parameter and state estimation of personalized human kinematic models from motion capture data. These models can be used to optimize a variety of human-robot collaboration scenarios for the comfort or ergonomics of an individual human collaborator. Our approach offers two main advantages over prior approaches from the literature and commercial software: the kinematic models are estimated for a specific individual without a priori assumptions on limb dimensions or range of motion, and our kinematic formalism explicitly encodes the natural kinematic constraints of the human body. The personalized models are tested in a human-robot collaborative manipulation experiment. We find that human subjects with a restricted range of motion rotate their torso significantly less during bimanual object handoffs if the robot uses a personalized kinematic model to plan the handoff configuration, as compared to previous approaches using generic human kinematic models.

Existing commercial driver assistance systems, including automatic braking systems and lane-keeping systems, may monitor the state of the vehicle or the environment to determine whether the systems should intervene. However, the state of the human driver is not typically included in the decision making process. In this paper, we propose to use hidden mode stochastic hybrid systems to model the interaction between the human driver and the vehicle. We show that by monitoring the human behavior as well as the vehicle state, we can infer the human state and enhance the quality of decision making in a driver assistance system. The resulting control policy is obtained by solving an optimal planning problem of the proposed hidden mode hybrid system. The policy can automatically balance the decision making about when to give warning to the driver and when to actually intervene in the control of the vehicle.

Assistive devices such as exoskeletons are capable of providing rehabilitative improvement and independence for individuals suffering from musculoskeletal conditions. Typical devices use either active assistance methods such as DC motors or passive methods such as springs. Active methods require a continuous power input, while passive methods are limited by user capability. This work introduces an Active/Passive EXoskeleton (APEX) framework. This device can passively provide continuous assistance, only requiring energy to change the dynamic properties of the passive state. The first prototype (APEX-α) is introduced and tested on six healthy subjects who performed hammer curls. It was found that changes in the passive state of the APEX-α affect the number of curls performed by an individual. By changing the passive state of the exoskeleton, increases in curl count of 65 - 92% were observed. This indicates the potential for such devices to provide assistance to an individual through the use of lightweight, energy efficient active/passive actuators.

Threat assessment during semiautonomous driving is used to determine when correcting a driver's input is required. Since current semiautonomous systems perform threat assessment by predicting a vehicle's future state while treating the driver's input as a disturbance, autonomous controller intervention is limited to a restricted regime. Improving vehicle safety demands threat assessment that occurs over longer prediction horizons wherein a driver cannot be treated as a malicious agent. In this paper, we describe a real-time semiautonomous system that utilizes empirical observations of a driver's pose to inform an autonomous controller that corrects a driver's input when possible in a safe manner. We measure the performance of our system using several metrics that evaluate the informativeness of the prediction and the utility of the intervention procedure. A multisubject driving experiment illustrates the usefulness, with respect to these metrics, of incorporating the driver's pose while designing a semiautonomous system.

In this paper we address calibration of camera arrays for life-size 3D video acquisition and display where mosaicking and multiviewpoint stereo are combined to provide an immersive experience which, by its size, resolution, and three-dimensionality, is meant to rival being there. This coupling of multiview and mosaicking requires integration of numerous synchronized video streams in a single presentation, aligned for both panoramic blending and epipolar rectification. The calibration framework we have developed extends the classical checkerboard approach through a modular multi-stage pipeline performing global optimization across intrinsics, extrinsics, panoramas, multiview epipolar alignments, and color correction. We demonstrate the methodology on several multiview camera arrays with various configurations aimed at mosaicking and epipolar light-field analysis. The results of this analysis have driven real-time life-sized panoramic 3D displays of captured events such as a concert, a fashion show, and sports activity.

For collision avoidance systems to be accepted by human drivers, it is important to keep the rate of unnecessary interventions very low. This is challenging since the decision to intervene or not is based on incomplete and uncertain information. The contribution of this paper is a decision making strategy for collision avoidance systems which allows the system to occasionally postpone a decision in order to collect more information. The problem is formulated in the framework of statistical decision theory, and the core of the algorithm is to run a preposterior analysis to estimate the benefit of deciding with the additional information. A final decision is made by comparing this benefit with the cost of delaying the intervention. The proposed approach is evaluated in simulation at a two-way stop road intersection for stop sign violation scenarios. The results show that the ability to postpone decisions leads to a significant reduction of false alarms and does not impair the ability of the collision avoidance system to prevent accidents.

Over the years, a large number of methods have been proposed to analyze human pose and motion information from images, videos, and recently from depth data. Most methods, however, have been evaluated on datasets that were too specific to each application, limited to a particular modality, and more importantly, captured under unknown conditions. To address these issues, we introduce the Berkeley Multimodal Human Action Database (MHAD) consisting of temporally synchronized and geometrically calibrated data from an optical motion capture system, multi-baseline stereo cameras from multiple views, depth sensors, accelerometers and microphones. This controlled multimodal dataset provides researchers an inclusive testbed to develop and benchmark new algorithms across multiple modalities under known capture conditions in various research domains. To demonstrate possible use of MHAD for action recognition, we compare results using the popular Bag-of-Words algorithm adapted to each modality independently with the results of various combinations of modalities using the Multiple Kernel Learning. Our comparative results show that multimodal analysis of human motion yields better action recognition rates than unimodal analysis.

The paper focuses on 3D structure and motion factorization from uncalibrated image sequences. A rank-4 affine factorization algorithm and a robust structure and motion factorization scheme are proposed to handle outlying and missing data. The novelty and main contribution of the paper are as follows: (i) The rank-4 factorization algorithm is a new addition to previous affine factorization family using rank-3 constraint; (ii) the outliers and image uncertainty are estimated directly from the image reprojection residuals; and (iii) the robust factorization scheme is proved empirically to be more efficient and accurate than other robust algorithms. Extensive experiments on synthetic data and real images validate the proposed approach.

We offer the explanation that computation can be seen as a transformation or function of information.

This article presents a psychophysical study that measures the perceptual thresholds of a new factor called Color-plus-Depth Level-of-Details (CZLoD) peculiar to polygon-based 3D tele-immersive video. The results demonstrate the existence of Just Noticeable Degradation and Just Unacceptable Degradation thresholds on the factor. In light of the results, we design and implement a real-time perception-based quality adaptor for 3D tele-immersive video. Our experimental results show that the adaptation scheme can reduce resource usage (e.g., CPU cycles) while considerably enhancing the overall perceived visual quality. Our analysis confirms the potential temporal and spatial performance benefits achievable with CZLoD adaptation.

Within biomedical engineering, the use of wearable wireless accelerometers for gait analysis can provide useful information for multiple health-related applications. The minimization of hardware for an accurate and simple estimation of the patient's velocity and stride length represents a difficult task. In this paper we propose a new methodology to determine the velocity and stride length of the patient through the application of the wavelet transform to the waist acceleration signal minimizing the hardware requirements. We introduce 4 novel formulations to take into account the trade-off between accuracy and computational costs, showing that an adaptive optimum approach can deliver results with average errors below 5%.

This paper proposes a cost function constructed from human data, the human-based cost, which is used to gauge the "human-like" nature of robotic walking. This cost function is constructed by utilizing motion capture data from a 9 subject straight line walking experiment. Employing a novel technique to process the data, we determine the times when the number of contact points change during the course of a step which automatically determines the ordering of discrete events or the domain breakdown along with the amount of time spent in each domain. The result is a weighted graph or walking cycle, associated with each of the subjects walking gaits. Finding a weighted cycle that minimizes the cut distance between this collection of graphs produces an optimal or universal domain graph for walking together with an optimal walking cycle. In essence, we find a single domain graph and the time spent in each domain that yields the most "natural" and "human-like" bipedal walking. The human-based cost is then defined as the cut distance from this optimal gait. The main findings of this paper are two-fold: (1) when the human-based cost is computed for subjects in the experiment it detects medical conditions that result in aberrations in their walking, and (2) when the human-based cost is computed for existing robotic models the more human-like walking gaits are correctly identified.

Local photometric descriptors are a crucial low level component of numerous computer vision algorithms. In practice, these descriptors are constructed to be invariant to a class of transformations. However, the development of a descriptor that is simultaneously robust to noise and invariant under general deformation has proven difficult. In this paper, we introduce the Topological-Attributed Relational Graph (T-ARG), a new local photometric descriptor constructed from homology that is provably invariant to locally bounded deformation. This new robust topological descriptor is backed by a formal mathematical framework. We apply T-ARG to a set of benchmark images to evaluate its performance. Results indicate that T-ARG significantly outperforms traditional descriptors for noisy, deforming images.

This paper presents a psychophysical study that measures the perceptual thresholds of a new factor called Color-plus-Depth Level-of-Detail peculiar to polygon-based 3D tele-immersive video. The results demonstrate the existence of Just Noticeable Degradation and Just Unacceptable Degradation thresholds on the factor. In light of the results, we describe the design and implementation of a real-time perception-based quality adaptor for 3D tele-immersive video. Our experimental results show that the adaptation scheme can reduce resource usage while considerably enhancing the overall perceived visual quality.

This paper presents a psychophysical approach to control a new factor called Color-plus-Depth Level-of-Detail in polygon-based 3D tele-immersive video. Based on our psychophysical study that demonstrates the existence of perceptual thresholds on the factor, we present a real-time perception-based quality adaptor for 3D tele-immersive video. Our experimental results show that the adaptation scheme can reduce resource usage while considerably enhancing the overall perceived visual quality.

The growing popularity of 3-D movies has led to the rapid development of numerous affordable consumer 3-D displays. In contrast, the development of technology to generate 3-D content has lagged behind considerably. In spite of significant improvements to the quality of imaging devices, the accuracy of the algorithms that generate 3-D data, and the hardware available to render such data, the algorithms available to calibrate, reconstruct, and then visualize such data remain difficult to use, extremely noise sensitive, and unreasonably slow. In this paper, we present a multi-camera system that creates a highly accurate (on the order of a centimeter), 3-D reconstruction of an environment in real-time (under 30 ms) that allows for remote interaction between users. This paper focuses on addressing the aforementioned deficiencies by describing algorithms to calibrate, reconstruct, and render objects in the system. We demonstrate the accuracy and speed of our results on a variety of benchmarks and data collected from our own system.

One of the oldest problems in the study of dynamical systems is the calculation of an optimal control. Though the determination of a numerical solution for the general non-convex optimal control problem for hybrid systems has been pursued relentlessly to date, it has proven difficult, since it demands nominal mode scheduling. In this paper, we calculate a numerical solution to the optimal control problem for a constrained switched nonlinear dynamical system with a running and final cost. The control parameter has a discrete component, the sequence of modes, and two continuous components, the duration of each mode and the continuous input while in each mode. To overcome the complexity posed by the discrete optimization problem, we propose a bi-level hierarchical optimization algorithm: at the higher level, the algorithm updates the mode sequence by using a single-mode variation technique, and at the lower level, the algorithm considers a fixed mode sequence and minimizes the cost functional over the continuous components. Numerical examples detail the potential of our proposed methodology.

Though the variety of desktop real time stereo vision systems has grown considerably in the past several years, few make any verifiable claims about the accuracy of the algorithms used to construct 3D data or describe how the data generated by such systems, which is large in size, can be effectively distributed. In this paper, we describe a system that creates an accurate (on the order of a centimeter), 3D reconstruction of an environment in real time (under 30 ms) that also allows for remote interaction between users. This paper addresses how to reconstruct, compress, and visualize the 3D environment. In contrast to most commercial desktop real time stereo vision systems our algorithm produces 3D meshes instead of dense point clouds, which we show allows for better quality visualizations. The chosen representation of the data also allows for high compression ratios for transfer to remote sites. We demonstrate the accuracy and speed of our results on a variety of benchmarks.

We present an indoor localization application leveraging the sensing capabilities of current state of the art smart phones. To the best of our knowledge, our application is the first one to be implemented in smart phones and integrating both offline and online phases of fingerprinting, delivering an accuracy of up to 1.5 meters. In particular, we have studied the possibilities offered by WiFi radio, cellular communications radio, accelerometer and magnetometer, already embedded in smart phones, with the intention to build a multimodal solution for localization. We have also implemented a new approach for the statistical processing of radio signal strengths, showing that it can outperform existing deterministic techniques.

Though the quality of imaging devices, the accuracy of algorithms that construct 3D data, and the hardware available to render such data have all improved, the algorithms available to calibrate, reconstruct, and then visualize such data are difficult to use, extremely noise sensitive, and unreasonably slow. In this paper, we describe a multi-camera system that creates a highly accurate (on the order of a centimeter), 3D reconstruction of an environment in real time (under 30 ms) that allows for remote interaction between users. The paper addresses the aforementioned deficiencies by featuring an overview of the technology and algorithms used to calibrate, reconstruct, and render objects in the system. The algorithm produces partial 3D meshes, instead of dense point clouds, which are combined on the renderer to create a unified model of the environment. The chosen representation of the data allows for high compression ratios for transfer to remote sites. We demonstrate the accuracy and speed of our results on a variety of benchmarks and data collected from our own system.

In this paper, sensor network scenarios are considered where the underlying signals of interest exhibit a degree of sparsity, which means that in an appropriate basis, they can be expressed in terms of a small number of nonzero coefficients. Following the emerging theory of compressive sensing (CS), an overall architecture is considered where the sensors acquire potentially noisy projections of the data, and the underlying sparsity is exploited to recover useful information about the signals of interest, which will be referred to as distributed sensor perception. First, we discuss the question of which projections of the data should be acquired, and how many of them. Then, we discuss how to take advantage of possible joint sparsity of the signals acquired by multiple sensors, and show how this can further improve the inference of the events from the sensor network. Two practical sensor applications are demonstrated, namely, distributed wearable action recognition using low-power motion sensors and distributed object recognition using high-power camera sensors. Experimental data support the utility of the CS framework in distributed sensor perception.

Sensor networks have been widely used for surveillance, monitoring, and tracking. Camera networks, in particular, provide a large amount of information that has traditionally been processed in a centralized manner employing a priori knowledge of camera location and of the physical layout of the environment. Unfortunately, these conventional requirements are far too demanding for ad-hoc distributed networks. In this article, we present a simplicial representation of a camera network called the camera network complex ( CN-complex), that accurately captures topological information about the visual coverage of the network. This representation provides a coordinate-free calibration of the sensor network and demands no localization of the cameras or objects in the environment. A distributed, robust algorithm, validated via two experimental setups, is presented for the construction of the representation using only binary detection information. We demonstrate the utility of this representation in capturing holes in the coverage, performing tracking of agents, and identifying homotopic paths.

Gait analysis is important for the diagnosis of many neurological diseases such as Parkinson's. The discovery and interpretation of minor gait abnormalities can aid in early diagnosis. We have used an inertial measuring system mounted on the subject's foot to provide numerical measures of a subject's gait (3-D displacements and rotations), thereby creating an automated tool intended to facilitate diagnosis and enable quantitative prognostication of various neurological disorders in which gait is disturbed. This paper describes the process used for ensuring that these inertial measurement units yield accurate and reliable displacement and rotation data, and for validating the preciseness and robustness of the gait-deconstruction algorithms. It also presents initial results from control subjects, focusing on understanding the data recorded by the shoe-mounted sensor to quantify relevant gait-related motions.

Three-dimensional tele-immersive (3DTI) environments have great potential to promote collaborative work among geographically distributed users. However, most existing 3DTI systems only work with two sites due to the huge demand of resources and the lack of a simple yet powerful networking model to handle connectivity, scalability, and quality-of-service (QoS) guarantees. In this article, we explore the design space from the angle of multi-stream management to enable multi-party 3DTI communication. Multiple correlated 3D video streams are employed to provide a comprehensive representation of the physical scene in each 3DTI environment, and are rendered together to establish a common cyberspace among all participating 3DTI environments. The existence of multi-stream correlation provides the unique opportunity for new approaches in QoS provisioning. Previous work mostly concentrated on compression and adaptation techniques on the per-stream basis while ignoring the application layer semantics and the coordination required among streams. We propose an innovative and generalized ViewCast model to coordinate the multi-stream content dissemination over an overlay network. ViewCast leverages view semantics in 3D free-viewpoint video systems to fill the gap between high-level user interest and low-level stream management. In ViewCast, only the view information is specified by the user/application, while the underlying control dynamically performs stream differentiation, selection, coordination, and dissemination. We present the details of ViewCast and evaluate it through both simulation and 3DTI sessions among tele-immersive environments residing in different institutes across the Internet2. Our experimental results demonstrate the implementation feasibility and performance enhancement of ViewCast in supporting multi-party 3DTI collaboration.

Three-dimensional tele-immersive (3DTI) environments have great potential to promote collaborative work among geographically distributed users. However, most existing 3DTI systems only work with two sites due to the huge demand of resources and the lack of a simple yet powerful networking model to handle connectivity, scalability, and quality-of-service (QoS) guarantees.

The use of wireless sensor networks as a means for providing remote healthcare provides a unique opportunity to reduce the healthcare cost through more efficient use of clinical resources and earlier detection of medical conditions. Despite the initial promising results, there remain significant obstacles to apply this technology to the practical medical care context. In this paper, we investigate one of the critical issues-how to assure the timely and reliable delivery of life-critical medical data in the pervasive patient monitoring networks. In this paper, our study is based on the remote health monitoring system CareNet which is built on top of a two-tier wireless sensor networks. In particular, we present an integrated admission control and routing solution to ensure the quality of admitted sensor delivery services. Using an optimization-based approach, our solution can maximally utilize the scarce wireless resource and satisfy the monitoring service requirements from the most number of patients. The novelty of our approach come from its patient activity awareness, which is used to provide more accurate estimation of the network capacity and worst-case assurance of the service quality under all patient movement scenarios. Extensive experiment and simulation results are presented to validate our proposed solution.

Recent advances in wireless sensor technology facilitate the development of remote healthcare systems, which can significantly reduce the healthcare cost. Despite the initial promising results, there remain many obstacles to apply this technology to the practical medical care context. One of the critical issues is to assure the timely and robust delivery of the life-critical medical data in the resource-constrained wireless sensor networking environment. This paper addresses this issue and presents a quality of service (QoS) support mechanism for wireless remote healthcare system, which integrates XML-based QoS specification, patient admission policy, and differentiated scheduling and queue management. The proposed QoS support mechanism is implemented in CareNet, our wireless sensor system for remote healthcare. Extensive performance study is presented to validate and evaluate our solution.

Human movement models often divide movements into parts. In walking, the stride can be segmented into four different parts, and in golf and other sports, the swing is divided into sections based on the primary direction of motion. These parts are often divided based on key events, also called temporal parameters. When analyzing a movement, it is important to correctly locate these key events, and so automated techniques are needed. There exist many methods for dividing specific actions using data from specific sensors, but for new sensors or sensing positions, new techniques must be developed. We introduce a generic method for temporal parameter extraction called the hidden Markov event model based on hidden Markov models. Our method constrains the state structure to facilitate precise location of key events. This method can be quickly adapted to new movements and new sensors/sensor placements. Furthermore, it generalizes well to subjects not used for training. A multiobjective optimization technique using genetic algorithms is applied to decrease error and increase cross-subject generalizability. Further, collaborative techniques are explored. We validate this method on a walking dataset by using inertial sensors placed on various locations on a human body. Our technique is designed to be computationally complex for training, but computationally simple at runtime to allow deployment on resource-constrained sensor nodes.

In this paper, we present a framework for immersive 3D video conferencing and geographically distributed collaboration. Our multi-camera system performs a full-body 3D reconstruction of users in real time and renders their image in a virtual space allowing remote interaction between users and the virtual environment. The paper features an overview of the technology and algorithms used for calibration, capturing, and reconstruction. We introduce stereo mapping using adaptive triangulation which allows for fast (under 25 ms) and robust real-time 3D reconstruction. The chosen representation of the data provides high compression ratios for transfer to a remote site. The algorithm produces partial 3D meshes, instead of dense point clouds, which are combined on the renderer to create a unified model of the user. We have successfully demonstrated the use of our system in various applications such as remote dancing and immersive Tai Chi learning.

In this paper we present a framework for immersive virtual environment intended for remote collaboration and training of physical activities. Our multi-camera system performs full-body 3D reconstruction of human user(s) in real time and renders their image in the virtual space allowing remote users to interact. The paper features a short overview of the technology used for the capturing and reconstruction. Some of the applications where we have successfully demonstrated use of the system in combination with the tele-immersive virtual environment are described. Finally, we address current drawbacks with regard to data capturing and networking and provide some ideas for future work.

We present a novel approach to the problem of representation and recognition of human actions, that uses an optimal control based model to connect the high-level goals of a human subject to the low-level movement trajectories captured by a computer vision system. These models quantify the high-level goals as a performance criterion or cost function which the human sensorimotor system optimizes by picking the control strategy that achieves the best possible performance. We show that the human body can be modeled as a hybrid linear system that can operate in one of several possible modes, where each mode corresponds to a particular high-level goal or cost function. The problem of action recognition, then is to infer the current mode of the system from observations of the movement trajectory. We demonstrate our approach on 3D visual data of human arm motion.

Advances in technology have led to development of various sensing, computing and communication devices that can be woven into the physical environment of our daily lives. Such systems enable on-body and mobile health-care monitoring, can integrate information from different sources, and can initiate actions or trigger alarms when needed. In this talk, we describe a collaborative signal processing scheme for physical movement monitoring with motion sensors. The signal processing consists of preprocessing, feature extraction and classification. We define a measure on feature "significance" as well as features' correlations. We characterize a graph model for collaborative signal processing based on the aforementioned measures, and illustrate how this model can be utilized to efficiently synthesize computation and communication for highly resource constrained wearable and mobile systems. We are examining the optimal positioning of sensors on the body for given physical activities, and focus on the segmentation and classification problem of the analysis of the continuous measurements of the observations obtained from the sensors. We have experimental data from different age subjects and show the individual differences amongst subjects.

The study of 3D Tele-immersion impact on remote collaborative work represents a very interesting and challenging research topic. In this paper, we introduce the latest accomplishments of TEEVE research which merges computer science with dance choreography. This collaborative research model is ideal for creative, interdisciplinary problem solving. TEEVE offers an entirely new interface for dance choreography as a creative tool and alternative performance venue.

Tele-immersion is emerging as a new medium that creates 3D photorealistic, immersive, and interactive experience between geographically dispersed users. However, most existing tele-immersive systems can only support two-way collaboration. In this paper we propose a multi-layer framework and a new data dissemination protocol to support multi-site collaboration. The problem context is unique as multiple remote sites participate in an interactive tele-immersive session, where each site has multiple correlated 3D video streams to send (later referred as multi-stream/multi-site environments). The key challenge is to disseminate such large number of 3D live video streams among these sites subject to the bandwidth and latency constraints while satisfying QoS guarantees in visual quality. Among our findings is that the simple randomized algorithm outperforms many other static algorithms in the unique context. Moreover, the streams generated from one site have high semantic correlation, because often cameras at one site are shooting the same scene, only from different angles. We exploit the stream correlation in the multicast protocol to minimize the level of loss.

Real-time distributed multi-party/multi-stream systems are becoming more popular in many areas such as 3D tele-immersion, multi-camera conferencing and security surveillance. However, the construction of such systems in large scale is impeded by the huge demand of computing and networking resources and the lack of a simple yet powerful networking model to handle interconnection, scalability and quality of service (QoS) guarantees. We make two main contributions in the paper: (1) we propose a novel generalized ViewCast model for multi-party/multi-stream video-mediated systems that fills the gap between high-level user interest and low level per-stream management, and (2) we demonstrate the ViewCast model by applying it to the multi-party 3D Tele-Immersive (3DTI) collaboration among geographically dispersed users. More specifically, we show how the ViewCast model is used in supporting stream data dissemination, coordination and QoS management among multiple 3D tele-immersive environments. We present our experimental results in both real implementation and simulation to show that our ViewCast-based solution achieves high efficiency, scalability, and quality in supporting multi-party 3DTI collaboration.

This paper describes a framework for constructing a three-dimensional immersive environment that can be used for training physical activities. In the proposed system, an immersive environment is constructed through realistic reconstruction of a scene by applying a stereo algorithm on a set of images that is captured from multiple viewpoints. Specifically, twelve camera clusters that consist of four camera quadruples are used to capture full body human motions, where each camera cluster is processed by a pc independently and synchronously so that they can compute 3d information of the scene in real-time fashion. This paper discusses in detail system architectures that enable synchronous operations across multiple computers while achieving parallel computations within each multi-processor system. A set of experiments are performed to learn taichi lessons in this environment where students are instructed to follow pre-recorded teacher's movements while observing both their own motions and the teacher in real-time from arbitrary viewpoints.

We first present the tele-immersive environments developed jointly by University of Illinois at Urbana-Champaign and University of California at Berkeley. The environment features 3D full and real body capturing, wide field of view, multi-display 3D rendering, and attachment free participant. We then describe a study of collaborative dancing between remotely located dancers in the shared virtual space. Two professional dancers are invited to the tele-immersive site of each university. As a preliminary experiment, we let the dancers perform elementary body movements and coordinate their dancing. The coordination requires one dancer to take the lead while the other follows her by appropriate movements. During the experiment, the dancers are dancing at various motion rates to evaluate how well the collaborative dancing is supported with the current technical boundary. Our important findings indicate that 1) tele-immersive environments have strong potential impact on the concept of choreography and communication of live dance performance, 2) the presence of multi-display system, real body 3D rendering, audio channel, and less intrusive-ness greatly enhances the immersive and dancing experience, and 3) the level of synchronization achieved by the dancers is higher than that expected from the video rate

We present a study of collaborative dancing between remote dancers in a tele-immersive environment which features 3D full and real body capturing, wide field of view, multi-display 3D rendering, and attachment free participant. We invite two professional dancers to perform collaborative dancing in the environment. The coordination requires one dancer to take the lead while the other follows by appropriate movement. Throughout the experiment, the dancers are dancing at various motion rates to evaluate how well the collaborative dancing is supported with the current technical boundary. Our important findings indicate that 1) tele-immersive environments have strong potential impact on the concept of choreography and communication of live dance performance, 2) the presence of multi-view display, real body 3D rendering, audio channel, and less intrusiveness greatly enhances the immersive and dancing experience, and 3) the level of synchronization achieved by the dancers is higher than that expected from the video rate.

Tele-immersive environments will improve the state of collaboration among distributed participants. However, along with the promise a new set of challenges have emerged including the real-time acquisition, streaming and rendering of 3D scenes to convey a realistic sense of immersive spaces. Unlike 2D video conferencing, a 3D tele-immersive environment employs multiple 3D cameras to cover a much wider field of view, thus generating a very large volume of data that need to be carefully coordinated, organized, and synchronized for Internet transmission, rendering and display. This is a challenging task and a dynamic bandwidth management must be in place. To achieve this goal, we propose a multi-stream adaptation framework for bandwidth management in 3D tele-immersion. The adaptation framework relies on the hierarchy of mechanisms and services that exploits the semantic link of multiple 3D video streams in the tele-immersive environment. We implement a prototype of the framework that integrates semantic stream selection, content adaptation, and 3D data compression services with user preference. The experimental results have demonstrated that the framework shows a good quality of the resulting composite 3D rendered video in case of sufficient bandwidth, while it adapts individual 3D video streams in a coordinated and user-friendly fashion, and yields graceful quality degradation in case of low bandwidth availability.

We present here a comprehensive imaging theory about cone mirrors in a single-view-point (SVP) configuration and show that an SVP cone mirror catadioptric system is not only practical but also has unique advantages for certain applications. We show its merits and weaknesses and how to build a workable system.

We address the problem of selecting sensors so as to minimize the error in estimating the position of a target. We consider a generic sensor model where the measurements can be interpreted as polygonal, convex subsets of the plane. In our model, the measurements are merged by intersecting corresponding subsets, and the measurement uncertainty corresponds to the area of the intersection. This model applies to a large class of sensors, including cameras. We present an approximation algorithm which guarantees that the resulting error in estimation is within factor 2 of the least possible error. In establishing this result, we formally prove that a constant number of sensors suffice for a good estimate-an observation made by many researchers. We demonstrate the utility of this result in an experiment where 19 cameras are used to estimate the position of a target on a known plane. In the second part of this paper, we study relaxations of the problem formulation. We consider 1) a scenario where we are given a set of possible locations of the target (instead of a single estimate) and 2) relaxations of the sensing model. Note to Practitioners-This paper addresses a problem which arises in applications where many sensors are used to estimate the position of a target. For most sensing models, the estimates get better as the number of sensors increases. On the other hand, energy and communication constraints may render it impossible to use the measurements from all sensors. In this case, we face the sensor selection problem: how to select a "good" subset of sensors so as to obtain "good" estimates. We show that under a fairly restricted sensing model, a constant number of sensors are always competitive with respect to all sensors and present an algorithm for selecting such sensors. In obtaining this result, we assume that the sensor locations are known. In future research, we will investigate methods that are robust with respect to errors in sensor localization/calibration

We address the problem of selecting sensors so as to minimize the error in estimating the position of a target. We consider a generic sensor model where the measurements can be interpreted as polygonal, convex subsets of the plane. This model applies to a large class of sensors including cameras. We present an approximation algorithm which guarantees that the resulting error in estimation is within a factor 2 of the least possible error. In establishing this result, we formally prove that a constant number of sensors suffice for a good estimate-an observation made by many researchers. In the second part of the paper, we study the scenario where the target's position is given by an uncertainty region and present algorithms for both probabilistic and online versions of this problem.

Tele-immersive 3D multi-camera room environments are starting to emerge and with them new challenging research questions. One important question is how to organize the large amount of visual data, being captured, processed, transmitted and displayed, and their corresponding resources, over current COTS computing and networking infrastructures so that "everybody" would be able to install and use tele-immersive environments for conferencing and other activities. In this paper, we propose a novel cross-layer control and streaming framework over general purpose delivery infrastructure, called TEEVE (tele-immersive environments for everybody). TEEVE aims for effective and adaptive coordination, synchronization, and soft QoS-enabled delivery of tele-immersive visual streams to remote room(s). The TEEVE experiments between two tele-immersive rooms residing in different institutions more than 2000 miles apart show that we can sustain communication of up to 12 3D video streams with 4/spl sim/5 3D frames per second for each stream, yielding 4/spl sim/5 tele-immersive video rate.

In this paper we propose a distributed and scalable algorithm that eliminates congestion within a sensor network, and that ensures the fair delivery of packets to a central node, or base station. We say that fairness is achieved when equal number of packets are received from each node. Since in general we have many sensors transmitting data to the base station, we consider the scenario where we have manyto-one multihop routing, noting that it can easily be extended to unicast or many-to-many routing. Such routing structures often result in the sensors closer to the base station experiencing congestion, which inevitably cause packets originating from sensors further away from the base station to have a higher probability of being dropped. Our algorithm exists in the transport layer of the traditional network stack model, and is designed to work with any MAC protocol in the data-link layer with minor modifications. Our solution is scalable, each sensor mote requires state proportional to the number of its neighbors. Finally, we demonstrate the effectiveness of our solution with both simulations and actual implementation in UC Berkeley’s sensor motes.

Autonomous robots need to acquire both omnidirectional view and stereo in real-time without sacrificing too much image resolution. The recent catadioptric omni-directional vision sensors provide simple affordable real-time omni-directional images but only at the cost of rather low image resolutions. This work describes a novel omni-directional stereo image sensor providing the highest resolution while still retaining the simplicity and real-time advantages that are unique to the family of catadioptric sensors. Only one simple reflective surface, a beam splitter and two regular (perspective) cameras are needed.

Provides an overview of the technical articles and features presented in this issue.

Summary form only given, as follows. The complete presentation was not made available for publication as part of the conference proceedings. The Center for Information Technology Research in the Interest of Society (CITRIS) is a partnership between four UC campuses – Berkeley (lead campus) Davis, Merced, Santa Cruz, the state of California and private industry. It is one of four UC Centers for Science and Innovation formed to help solve societal scale problems, which affect the quality of life of every Californian. The presenter will describe some of the projects going on at UC campuses to help solve these daunting challenges: Education – distance learning ; Energy Efficiency – smart buildings ; Environment - water, air, environment, forest fires; Health Care - monitoring devices; Homeland Security and Defense; Seismic Safety; and Transportation.

Pinhole camera model is a simplified subset of geometric optics. In special cases like the image formation of the cone (a degenerate conic section) mirror in an omnidirectional view catadioptric system, there are more complex optical phenomena involved that the simple pinhole model can not explain. We show that using the full geometric optics model a true single viewpoint cone mirror omni-directional system can be built. We show how such a system is built first, and then show in detail how each optical phenomenon works together to make the system true single viewpoint. The new system requires only simple off-the-shelf components and still outperforms other single viewpoint omni-systems for many applications.

No abstract available.

We present two families of reflective surfaces that are capable of providing a wide field of view, and yet still approximate a perspective projection to a high degree. These surfaces are derived by considering a plane perpendicular to the axis of a surface of revolution and finding the equations governing the distortion of the image of the plane in this surface. We then view this relation as a differential equation and prescribe the distortion term to be linear. By choosing appropriate initial conditions for the differential equation and solving it numerically, we derive the surface shape and obtain a precise estimate as to what degree the resulting sensor can approximate a perspective projection. Thus these surfaces act as computational sensors, allowing for a wide-angle perspective view of a scene without processing the image in software. The applications of such a sensor should be numerous, including surveillance, robotics and tradition photography.

In this paper, we discuss matching of diffusion tensor (DT) MRIs of the human brain. Issues concerned with matching and transforming these complex images are discussed. A number of similarity measures are proposed, based on indices derived from the DT, the DT itself and the DT deviatoric. Each measure is used to drive an elastic matching algorithm applied to the task of registration of 3D images of the human brain. The performance of the various similarity measures is compared empirically by use of several quality of match measures computed over a pair of matched images. Results indicate that the best matches are obtained from a Euclidean difference measure using the full DT. 1 Introduction Diffusion tensor (DT) imaging is a recent innovation in magnetic resonance imaging (MRI), [1]. In DT imaging, the measurement acquired at each voxel in an image volume is a symmetric second order tensor, which describes the local water diffusion properties of the material being imaged. Th...

In this paper we discuss work on the use of diffusion tensor MRIs for inter-subject brain matching. A multiresolution elastic matching algorithm for spatial normalisation of 3D image data, has been adapted for use with diffusion tensor data. The hope is that by exploitation of the added information contained in the diffusion tensor image, improved anatomical matches can be found, particularly in white matter regions of the brain. Results show that by matching on the diffusion tensor alone, anisotropic regions of the brain (white matter) are aligned better than if the match is computed on standard structural data. However, there is a cost of some accuracy in the alignment of prominent features in more conventional, structural MRI data, such as PD-, TI- and T2-weighted imagery. If both types of data to drive the matching process, prominent features in both images can be aligned simultaneously. The motivation for this work lies in the characterisation of the distribution of brain images taken from population groups.

The light reflected from a surface depends on the scene geometry, the incident illumination and the surface material. A novel methodology is presented which extracts reflectivity information of the various materials in the scene independent of incident light and scene geometry. A scene is captured under different narrow-band color filters and the spectral derivatives of the scene are computed. The resulting spectral derivatives form a spectral gradient at each pixel. This spectral gradient is a material descriptor which is invariant to scene geometry and incident illumination for smooth diffuse surfaces. Spectral gradients can discriminate among smooth dielectrics with different reflectance properties independent of viewing conditions.

We address the problem of automatically reconstructing m-manifolds of unknown topology from unorganized points in metric p-spaces obtained from a noisy measurement process . The point set is first approximated by a collection of oriented primitive fuzzy sets over a range of resolutions. Hierarchical multiresolution representation is then computed based on the relation of relative containment defined on the collection. Finally, manifold structure is recovered by establishing connectivity between these primitives based on proximity, compatibility of position and orientation and local topological constraints. The method has been successfully applied to the problem of surface reconstruction from polynocular-stereo data with many outliers.

In this paper we address an application of computer vision which can in the future change completely our way of communicating over the network. We present our version of a testbed for telecollaboration. It is based on a highly accurate and precise stereo algorithm. The results demonstrate the live (on-line) recovery of 3D models of a dynamically changing environment and the simultaneous display and manipulation of the models.

We study occluding contour artifacts in area-based stereo matching: they are false responses of the matching operator to the occlusion boundary and cause the objects extend beyond their true boundaries in disparity maps. Most of the matching methods suffer from these artifacts; the effect is so strong that it cannot be ignored. We show what gives rise to the artifacts and design a matching criterion that accommodates the presence of occlusions as opposed to methods that identify and remove the artifacts. This approach leads to the problem of measurement contamination studied in statistics. We show that such a problem is hard given finite computational resources, unless more independent measurements directly related to occluding contours is available. What can be achieved is a substantial reduction of he artifacts, especially for large matching templates. Reduced artifacts allow for easier hierarchical matching and for easy fusion of reconstructions from different viewpoints into a coherent whole.

We address scaling of the "dynamic systems" approach for robot planning to multi-agent cooperation. To accommodate this extension it is necessary to carefully consider how individual behaviors contribute to the vector field. To avoid spurious minima and related problems a competition dynamics is introduced and its stability is analyzed. A system of two cooperating agents is designed, and examples are presented to illustrate the utility of this approach.

We present a new integrated approach to the two-dimensional part segmentation, shape, and motion estimation of moving multipart objects. Our technique exploits the relationship between the geometry and the observed deformations of the apparent contour of a moving multipart object and its structure. The novelty of the technique is that no prior model of the object or of its parts is employed. We develop aPart Segmentation Algorithm(PSA) that recursively recovers all the moving parts of an object by monitoring and reasoning over the changes of its deforming apparent contour. To parameterize and segment over time a deforming apparent contour, we fit initially a single deformable model whose global and local deformations over time allow us to hypothesize an underlying part structure. This hypothesis is verified by further monitoring the relative motion among the model's parts and the satisfaction of certain criteria. Upon verifying the part hypothesis, the initial deformable model is split into two or more models that better fit the apparent contour. This recursive operation allows the refinement over time of the number and shape of the extracted parts. When multiple deformable models are used to model the apparent contour of a multipart object, there is an uncertainty concerning the deformable model to which the data points should apply forces to. To address this problem, we present a new algorithm for force assignment that assigns forces from the data to multiple models. This algorithm allows partial overlap between the parts' models and the determination of their joint location. Finally, the effectiveness of the approach is demonstrated through a series of experiments involving a variety of objects.

A method for integrating image segmentation information into geometric models is presented. The resulting object representation has advantages of both model-based and view-based representations, in that model geometry plus learned appearance information is used to improve the prediction of object appearance over purely geometric methods. The combined models are constructed over a training set of imagery using prior geometric models. Segmentation features are matched to the geometric models, and an evidential framework is used to characterize the segmentations of model features. To test the validity of the models, a pose adjustment system was modified to incorporate the prior segmentation information. Results indicate that the inclusion of the segmentation information significantly improves pose adjustment accuracy over using purely geometric information for model appearance.

Shadows are a frequent occurrence, but they cannot be infallibly recognized until a scene's geometry and lighting are known. We present a number of cues which together strongly suggest the identification of a shadow and which can be examined with low cost. The techniques are: a color image segmentation method that recovers single material surfaces as single image regions irregardless of the surface partially in shadow, a method to recover the penumbra and umbra of shadow; a method for determining whether some object could be obstructing a light source. The last cue requires the examination of well understood shadows in the scene. Our observer is equipped with an extendable probe for casting its own shadows. Actively obtained shadows allow the observer to experimentally determine the location of the light sources in the scene. The system has been tested both indoors and out.>

As it occurs in humans, robotic systems should be able to respond to unexpected tactile events by orienting their visual attention toward the location of the stimuli. This implies two basic problems: 1) it is necessary to develop a general method for integrating attentive processes which belong to different sensory modalities according to the attended task; 2) for the specific case of touch-driven shift of gaze, a sensory motor transformation needs to be identified, which links the stimulation of tactile receptors to the spatial position of the camera, via the current posture of the system. In this paper we describe a general framework for integrating multimodal attentive mechanisms, and we show how the visuo-tactile coordination can be autonomously learnt on the basis of sensory consistency and feedback. After the general presentation of the method, we consider the case of a robotic system composed of a 2-DOF arm and a 2-DOF head. Experiments with this system show that it discovers its own functional model without any external intervention and adapts it continuously during normal operation. The approach gives good results while presenting the advantages of autonomy and adaptability.

Presents a collaborative effort to design and implement a cooperative material handling system by a small team of human and robotic agents in an unstructured indoor environment. The authors' approach makes fundamental use of the human agents' expertise for aspects of task planning, task monitoring, and error recovery. The authors' system is neither fully autonomous nor fully teleoperated. It is designed to make effective use of the human's abilities within the present state of the art of autonomous systems. The authors' robotic agents refer to systems which are each equipped with at least one sensing modality and which possess some capability for self-orientation and/or mobility. The authors' robotic agents are not required to be homogeneous with respect to either capabilities or function. The authors' research stresses both paradigms and testbed experimentation. Theory issues include the requisite coordination principles and techniques which are fundamental to a cooperative multiagent system's basic functioning. The authors have constructed an experimental distributed multiagent-architecture testbed facility. The required modular components of this testbed are currently operational and have been tested individually. The authors' current research focuses on the agents' integration in a scenario for cooperative material handling.

Functionality of an object defines its applicability to a task, In this paper, we introduce a representation for functionality and present a methodology for its recovery. In addition, we introduce force-shape maps as means to integrate and classify the recovered functionality, Since functionality describes an interaction, the representation for an object must include not only its intrinsic (material) but also its functional (how it is used) properties. In order to recover the functionality of an object, we have developed a formalism based on discrete event system theory and on the paradigm of active perception. This formalism allows us to express an investigating task in the form of a finite automaton for controlling and observing the interactions in our task. Our current investigation focuses on manipulatory interactions, and thus far, the functionality of piercing. We envision an expandable system having at its disposal a score of investigative procedures and through them to be able to classify and recognize an object based on its functional attributes.

We present a novel, robust, integrated approach to segmentation shape and motion estimation of articulated objects. Initially, we assume the object consists of a single part, and we fit a deformable model to the given data using our physics-based framework. As the object attains new postures, we decide based on certain criteria if and when to replace the initial model with two new models. These criteria are based on the model's state and the given data. We then fit the models to the data using a novel algorithm for assigning forces from the data to the two models, which allows partial overlap between them and determination of joint location. This approach is applied iteratively until all the object's moving parts are identified. Furthermore, we define new global deformations and we demonstrate our technique in a series of experiments, where Kalman filtering is employed to account for noise and occlusion.>

This paper discusses some representation issues and challenges involved in object recognition. It is intended as a step toward assessing current object representation schemes and proposing design and evaluation criteria for future ones.>

The application of tools by artificial systems depends on the ability to recognize an object as having a particular function as well as on the ability to express the functional interaction. This paper focuses on the operation of piercing as means of investigating the issue of functional representation. The operation is expressed using a formalism, based on discrete event system (DES) theory and on the paradigm of active perception. The visual and tactile data acquired during piercing motor actions performed with a robotic manipulator is discussed.

The authors present modeling, analysis, and synthesis of visual behaviors of agents engaged in navigational tasks. They consider situations in which two agents can navigate independently or in cooperation. For the purpose of modeling the behaviors, a formalism is adopted from the discrete events systems (DES) theory that is suitable for investigating control-theoretic issues of a system. The focus is on the identification of elementary behaviors and their composition, leading to more complex behaviors. Two kinds of elementary behaviors are identified: one where observations are directly connected with actions, and one where observations and actions are either received or transmitted. The use of the DES formalism allows synthesis of complex behaviors in a systematic fashion and guarantees their controllability.>

A strategy for acquiring 3-D data of an unknown scene, using range images obtained by a light stripe range finder is addressed. The foci of attention are occluded regions, i.e., only the scene at the borders of the occlusions is modeled to compute the next move. Since the system has knowledge of the sensor geometry, it can resolve the appearance of occlusions by analyzing them. The problem of 3-D data acquisition is divided into two subproblems due to two types of occlusions. An occlusion arises either when the reflected laser light does not reach the camera or when the directed laser light does not reach the scene surface. After taking the range image of a scene, the regions of no data due to the first kind of occlusion are extracted. The missing data are acquired by rotating the sensor system in the scanning plane, which is defined by the first scan. After a complete image of the surface illuminated from the first scanning plane has been built, the regions of missing data due to the second kind of occlusions are located. Then, the directions of the next scanning planes for further 3-D data acquisition are computed.>

The findings of a workshop, the goals of which were to identify applications, research problems, and designs of high performance computing and communications (HPCC) systems for supporting applications are discussed. In computer vision, the main scientific issues are machine learning, surface reconstruction, inverse optics and integration, model acquisition, and perception and action. In speech and natural language processing (SNLP), issues were identified statistical analysis in corpus-based speech and language understanding, search strategies for language analysis, auditory and vocal-tract modeling, integration of multiple levels of speech and language analyses, and connectionist systems. In AI, important issues that need immediate attention include the development of efficient machine learning and heuristic search methods that can adapt to different architectural configurations, and the design and construction of scalable and verifiable knowledge bases, active memories, and artificial neural networks.>

Presents an integrated framework for segmenting dense range data of complex 3-D scenes into surface (bi-quadrics) and volumetric (superquadrics) primitives, without a priori domain knowledge or stored models. Surface segmentation is performed by a novel local-to-global iterative regression approach of searching for the best piecewise description of the data in terms of biquadric models. Region adjacency information, surface discontinuities, and global shape properties are extracted and used to guide the volumetric segmentation. Superquadric models are recovered by a global-to-local residual-driven procedure, which recursively segments the scene to derive the part-structure. A set of acceptance criteria provide the objective evaluation of intermediate descriptions, and decide whether to terminate the procedure, or selectively refine the segmentation. The control module generates hypotheses about superquadric models at clusters of underestimated data and performs controlled extrapolation of part-models by shrinking the global model. The authors present results on real range images of scenes of varying complexity, including objects with occluding parts, and scenes where surface segmentation is not sufficient to guide the volumetric segmentation.>

The problem of development and implementation of a discrete event dynamic system observer for a moving agent is considered. The authors present a modeling approach for the visual system and its observer, where the 'events' are defined as ranges on parameter subsets. In particular, the proposed system is used for observing a manipulation process, where a robot hand manipulates an object. the hand/object interaction is recognised over time and a stabilizing observer is constructed. The resulting robot arm behavior is constructed as a hybrid intelligent mechanism. The work examines closely the possibilities for errors, mistakes and uncertainties in the manipulation system, observer construction process and event identification mechanisms. Some results from a sequence of a peg-in-hole operation are documented.>

The strategy of acquiring 3-D data of unknown scenes is discussed. The analysis is limited to range images obtained by a light stripe range finder. Prior knowledge given to the system is the knowledge of the sensor geometry. The emphasis on the occluded regions, i.e. their shape and the height of the scene at their borders. Since the system has knowledge of the sensor geometry it can resolve the appearance of occlusions by analyzing them.>

The authors propose that some aspects of task-based learning in robotics can be approached using nativist and constructionist views on human sensorimotor development as a metaphor. They use findings in developmental psychology and neurophysiology, as well as machine perception, to guide the overall design of robotic system that attempts to learn sensorimotor binding rules for simple actions. Visually driven grasping was chosen as the experimental task. The learning was empirical in nature, and was done by having the robot observe repeated interactions with the task environment. The technique of nonparametric projection pursuit regression was used to accomplish reinforcement data sets that capture task invariants. The learning process generally implied failures along the way. Therefore, the mechanics of the untrained robotic system must be able to tolerate mistakes during learning and not be damaged. This problem was addressed by the use of an instrumented compliant robot wrist that controlled impact forces.>

It is noted that material properties like penetrability, compliance, and surface roughness are important in the characterization of the environment. The authors have designed and implemented exploratory procedures on a robotic system to actively explore a surface to extract its material properties. The exploratory procedures for exploration are integrated into an active perceptual scheme for legged locomotion. The perceptual scheme is designed around creating the ability for the robot to sense variations in terrain properties while it is walking, so that it may be able to avoid sinking, slipping, and falling due to unexpected changes in the terrain properties, and make suitable changes in its foot forces to continue locomotion. The active perceptual scheme is implemented by simulating a leg-ankle-foot system with a PUMA arm-compliant wrist-foot system and an accelerometer mounted on the foot to detect slip.>

The authors address the problem of observing a moving agent. They advocate a modeling approach for the visual system and its observer, where a discrete event dynamic system framework is developed and events are defined as ranges on parameter subsets. A system for observing a manipulation process in which a robot hand manipulates an object is proposed. The hand/object interaction is recognized over time, and a stabilizing observer is constructed. Low-level modules are developed for recognizing the events that cause state transitions within the dynamic manipulation system. The work examines closely the possibilities for errors, mistakes, and uncertainties in the manipulation system, observer construction process, and even identification mechanisms. The system utilized different tracking techniques to observe and recognize the task in an active and goal-directed manner.>

The question of what are the necessary elements to integrate a robotics system that would enable it to carry out a task, i.e. pick-up and transport objects in an unknown environment, is addressed. One of the major concerns is to insure adequate data throughput and fast communication between modules within the system, so that haptic tasks can be adequately carried out. The communication issues involved in the development of such a system are discussed.>

For robots to traverse rugged terrain successfully using legged locomotion, they need not only constantly to maintain structural stability but also, and perhaps more importantly, to detect and adapt to changes in the terrain properties. The authors address the issue of exploration to extract material properties from a given surface for the specific purpose of aiding in and improving the quality of legged locomotion. While it is important to evaluate terrain properties prior to the start of locomotion, it is even more important to evaluate these properties actively during locomotion so that the robot does not sink, slip or fall. It is proposed that the legs of a robot be used not only for stepping and walking but also as probes to examine those properties of the surface that would contribute to the efficiency of locomotion, one way or another. The proposed framework for active perception for legged locomotion suggests that for stable stepping and walking in an unknown environment, it is necessary actively to recover the material properties of penetrability, compliance and surface traction from the supporting surface. These attributes must be recovered by exploratory procedures that are built into the mobile robotic system. This paper focusses on the implementation of the perceptual scheme so that feedback from the measurement of material properties is used to control robot foot forces during legged locomotion.>

The authors address the problem of observing a moving agent. In particular, they propose a system for observing a manipulation process, where a robot hand manipulates an object. A discrete event dynamic system framework is developed for the hand/object interaction over time and a stabilizing observer is constructed. Low-level modules are developed for recognizing the 'events' that cause state transitions within the dynamic manipulation system. The work examines closely the possibilities for errors, mistakes and uncertainties in the manipulation system, observer construction process and event identification mechanisms. The system utilizes different tracking techniques in order to observe and recognize the task in an active, adaptive and goal-directed manner.>

A paradigm of iterative, interactive scene segmentation and simplification of random heaps of unknown objects via vision and manipulation is introduced. The scene simplification is based on the graph operations of vertex and edge removal. These operations are defined isomorphic to the pick and push manipulation actions. Sensors are used as graph generators and the manipulator is used as the decomposing mechanism of the graphs. The model is a nondeterministic finite-state Turing machine. A vision system, a manipulator, and force/torque and other sensory input are integrated into a robot work cell. Experiments conducted to test convergence and error recovery of four different strategies are discussed. It is found that under certain conditions the strategies can tolerate errors in the sensory data, recover from pathological states, and converge.>

A paradigm is presented for the segmentation of images into piecewise continuous patches. Data aggregation is performed via model recovery in terms of variable-order bivariate polynomials using iterative regression. All the recovered models are candidates for the final description of the data. Selection of the models is achieved through a maximization of the quadratic Boolean problem. The procedure can be adapted to prefer certain kinds of descriptions (one which describes more data points, or has smaller error, or has a lower order model). A fast optimization procedure for model selection is discussed. The approach combines model extraction and model selection in a dynamic way. Partial recovery of the models is followed by the optimization (selection) procedure where only the best models are allowed to develop further. The results are comparable with the results obtained when using the selection module only after all the models are fully recovered, while the computational complexity is significantly reduced. The procedure was tested on real range and intensity images. >

A computational model for color image segmentation is proposed based on the physical properties of sensors, illumination lights, and surface reflectances. For image segmentation to depend only on the change of material surface, variations of surface reflection due to illumination, shading, shadows, and highlights should be discounted from a measured image. The authors use the dichromatic model for dielectric materials and develop a metric space of intensity, hue, and saturation in order to interpret various light-surface interactions better. Using the established model for surface reflection, the authors perform color image segmentation based on the material change with detection of highlights and small interreflections between adjacent objects. A reference plate is used to whiten global illumination. The detected interreflections represent local variation of illumination.>

A method for recovery of compact volumetric models for shape representation of single-part objects in computer vision is introduced. The models are superquadrics with parametric deformations (bending, tapering, and cavity deformation). The input for the model recovery is three-dimensional range points. Model recovery is formulated as a least-squares minimization of a cost function for all range points belonging to a single part. During an iterative gradient descent minimization process, all model parameters are adjusted simultaneously, recovery position, orientation, size, and shape of the model, such that most of the given range points lie close to the model's surface. A specific solution among several acceptable solutions, where are all minima in the parameter space, can be reached by constraining the search to a part of the parameter space. The many shallow local minima in the parameter space are avoided as a solution by using a stochastic technique during minimization. Results using real range data show that the recovered models are stable and that the recovery procedure is fast.>

The design of an end effector under development at Pennsylvania University is reported. The rationale supporting this mechanism is explored, its geometry is described and experimental results from the first prototype are presented. The gripper has three fingers, one fixed with respect to the palm and the other two able to rotate synchronously about different points under the palm. Some ideas for future work are presented.>

Active perception (active vision specifically) is defined as a study of modeling and control strategies for perception. Local methods are distinguished from global models by their extent of application in space and time. The local models represent procedures and parameters such as optical distortions of the lens, focal lens, spatial resolution, bandpass filter, etc, The global models, on the other hand, characterize the overall performance and make predictions on how the individual modules interact. The control strategies are formulated as a search of such sequences of steps that would minimize a loss function while still seeking the most information. Examples are shown as the existence proof of the proposed theory on obtaining range from focus and stereo/vergence on 2-D segmentation of an image and 3-D shape parameterization.>

This paper specifies constraints based on the geometry of the grasped object, on geometry of the hand and the kinematics of the constrained object which determine how to grasp an object.

The purpose of this correspondence is to show the design considerations in the choice of mechanisms when a flexible querysystem of visual scenes is being constructed. More concretely, the issues are: ¿ flexibility in adding new information to the knowledge base; ¿ power of inferencing; ¿ avoiding unnecessary generation of hypotheses where a great deal of image processing has to be perfected in order to test it; ¿ having the power of automatic generation of recognition strategies.

In this note we present an algorithm for packing spheres in an arbitrary shaped volume. This algorithm is similar to Blum's transform in that it fits spheres into a volume, but it is different in that it fits only tangential spheres, and thereby the data reduction is larger than by Blum's transform. The spheres are of variable radii, which enables us to achieve a hierarchy of intrinsic volume properties, i.e., from gross to more detailed. The result of this algorithm is a graph where the nodes are the centers of spheres and the arcs are the connections between two tangent spheres. Analysis of computational complexity and the time and error considerations are provided.

Representing complex three-dimensional objects in a computer involves more than just evaluating its display capabilities. Other factors are the uses and costs of the representation, what operations can be performed on it and, ultimately, how useful it is for computer recognition or description or three-dimensional objects. Many of the questions which are posed arise from the joint consideration of computer graphics and computer vision, and a specific representation hierarchy is proposed for complex objects which makes them amenable to display, manipulation, measurement, and analysis.

A program which recognizes real roads, their intersections, and objects which are road-like is presented. Although the work has been strongly motivated by the real roads as they are seen from the ERTS satellite, the visual model of a road can be used for recognition of any road-like objects. This model is structured in such a way that it can handle real roads as well as rivers and streams as they are seen on satellite pictures, tracks in bubble chambers, and veins under the appropriate magnification.