Labeling articulated objects in unconstrained settings has a wide variety of applications including entertainment, neuroscience, psychology, ethology, and many fields of medicine. Large offline labeled datasets do not exist for all but the most common articulated object categories (e.g., humans). Hand labeling these landmarks within a video sequence is a laborious task. Learned landmark detectors can help, but can be error-prone when trained from only a few examples. Multi-camera systems that train fine-grained detectors have shown significant promise in detecting such errors, allowing for self-supervised solutions that only need a small percentage of the video sequence to be hand-labeled. The approach, however, is based on calibrated cameras and rigid geometry, making it expensive, difficult to manage, and impractical in real-world scenarios. In this paper, we address these bottlenecks by combining a non-rigid 3D neural prior with deep flow to obtain high-fidelity landmark estimates from videos with only two or three uncalibrated, handheld cameras. With just a few annotations (representing $1-2\%$ of the frames), we are able to produce 2D results comparable to state-of-the-art fully supervised methods, along with 3D reconstructions that are impossible with other existing approaches. Our Multi-view Bootstrapping in the Wild (MBW) approach demonstrates impressive results on standard human datasets, as well as tigers, cheetahs, fish, colobus monkeys, chimpanzees, and flamingos from videos captured casually in a zoo. We release the codebase for MBW as well as this challenging zoo dataset consisting of image frames of tail-end distribution categories with their corresponding 2D and 3D labels generated from minimal human intervention.

Multi-view triangulation is the gold standard for 3D reconstruction from 2D correspondences given known calibration and sufficient views. However in practice, expensive multi-view setups – involving tens sometimes hundreds of cameras – are required in order to obtain the high fidelity 3D reconstructions necessary for many modern applications. In this paper we present a novel approach that leverages recent advances in 2D-3D lifting using neural shape priors while also enforcing multi-view equivariance. We show how our method can achieve comparable fidelity to expensive calibrated multi-view rigs using a limited (2-3) number of uncalibrated camera views.

Training deep belief networks (DBNs) is normally done with large data sets. Our goal is to predict traces of the surface of the tongue in ultrasound images of human speech. Hand-tracing is labor-intensive; the dataset is highly imbalanced since many images are extremely similar. We propose a bootstrapping method which handles this imbalance by iteratively selecting a small subset of images to be hand-traced (thereby reducing human labor time), then (re)training the DBN, making use of an entropy-based diversity measure for the initial selection, thereby achieving over a two-fold reduction in human time required for tracing with human-level accuracy.

We present the Computer Expression Recognition Toolbox (CERT), a software tool for fully automatic real-time facial expression recognition, and officially release it for free academic use. CERT can automatically code the intensity of 19 different facial actions from the Facial Action Unit Coding System (FACS) and 6 different prototypical facial expressions. It also estimates the locations of 10 facial features as well as the 3-D orientation (yaw, pitch, roll) of the head. On a database of posed facial expressions, Extended Cohn-Kanade (CK+[1]), CERT achieves an average recognition performance (probability of correctness on a two-alternative forced choice (2AFC) task between one positive and one negative example) of 90.1% when analyzing facial actions. On a spontaneous facial expression dataset, CERT achieves an accuracy of nearly 80%. In a standard dual core laptop, CERT can process 320 × 240 video images in real time at approximately 10 frames per second.

We describe a system for automatically extracting dynamics of tongue gestures from ultrasound images of the tongue using translational deep belief networks (tDBNs). In tDBNs, a joint model of the input and output vectors are learned during a generative pretraining stage, and then a translation step is used to transform input-only vectors into this joint representation. A final fine-tuning stage is then used to reconstruct the desired outputs given input vectors. We show that this technique dramatically improves performance on segmenting ultrasound image sequences of continuous speech into individual consonant gestures compared with the original DBN method of as well as alternative methods using PCA and support vector machines.

We investigate the effect of reduction induced by repetition during articulation. Specifically, we report how tongue movement differs between the first mention of Mandarin words and that of later repetitions using ultrasound imaging. Two analyses were carried out in this paper: tongue deformation and timing. We used Dynamic Time Warping to measure the tongue deformation from a neutral position. We used Functional Data Analysis to measure the timing difference between the first and later repetitions. We found that the tongue deviates less from the neutral and moves faster in time for the later repetitions, namely the more reduced ones. Our study shows promise for more thorough investigation of speech reduction from the articulatory perspective, and provides insights for constructing applications for speech synthesis/recognition towards more natural speech. Index Terms: Speech reduction, ultrasound imaging, Mandarin Chinese, dynamic time warping, functional data analysis

Currently, most systems for human-robot teaching allow only one mode of teacher-student interaction (e.g., teaching by demonstration or feedback), and teaching episodes have to be carefully set-up by an expert. To understand how we might integrate multiple, interleaved forms of human instruction into a robot learner, we performed a behavioral study in which 44 untrained humans were allowed to freely mix interaction modes to teach a simulated robot (secretly controlled by a human) a complex task. Analysis of transcripts showed that human teachers often give instructions that are nontrivial to interpret and not easily translated into a form useable by machine learning algorithms. In particular, humans often use implicit instructions, fail to clearly indicate the boundaries of procedures, and tightly interleave testing, feedback, and new instruction. In this paper, we detail these teaching patterns and discuss the challenges they pose to automatic teaching interpretation as well as the machine-learning algorithms that must ultimately process these instructions. We highlight the challenges by demonstrating the difficulties of an initial automatic teacher interpretation system.

Ultrasound has become a useful tool for speech scientists studying mechanisms of language sound production. State-of-the-art methods for extracting tongue contours from ultrasound images of the mouth, typically based on active contour snakes, require considerable manual interaction by an expert linguist. In this paper we describe a novel method for fully automatic extraction of tongue contours based on a hierarchy of restricted Boltzmann machines (RBMs), i.e. deep belief networks (DBNs). Usually, DBNs are first trained generatively on sensor data, then discriminatively to predict human-provided labels of the data. In this paper we introduce the translational RBM (tRBM), which allows the DBN to make use of both human labels and raw sensor data at all stages of learning. This method yields performance in contour extraction comparable to human labelers, without any temporal smoothing or human intervention, and runs in real-time.

Social interactions unfold over time, at multiple time scales, and can be observed through multiple sensory modalities. In this paper, we propose a machine learning framework for selecting and combining low-level sensory features from different modalities to produce high-level characterizations of human-robot social interactions in real-time.

Localizing facial features is a critical component in many computer vision applications such as expression recognition, face recognition, face tracking, animation, and red-eye correction. Practical applications require detectors that operate reliably under a wide range of conditions, including variations in illumination, pose, ethnicity, gender and age. One challenge for the development of such detectors is the inherent trade-off between robustness and precision. Robust detectors tend to provide poor localization and detectors sensitive to small changes in local structure, which are needed for precise localization, generate a large number of false alarms. Here we present an approach to this trade-off based on context dependent inference. First, robust detectors are used to detect contexts in which target features occur, then precise detectors are trained to localize the features given the detected context. This paper describes the approach and presents a thorough empirical examination of the parameters needed to achieve practical levels of performance, including the size of the training database, size of the detector's receptive fields and methods for information integration. The approach operates in real time and achieves, to our knowledge, the most accurate localization performance to date.

Machine learning approaches have produced some of the highest reported performances for facial expression recognition. However, to date, nearly all automatic facial expression recognition research has focused on optimizing performance on a few databases that were collected under controlled lighting conditions on a relatively small number of subjects. This paper explores whether current machine learning methods can be used to develop an expression recognition system that operates reliably in more realistic conditions. We explore the necessary characteristics of the training data set, image registration, feature representation, and machine learning algorithms. A new database, GENKI, is presented which contains pictures, photographed by the subjects themselves, from thousands of different people in many different real-world imaging conditions. Results suggest that human-level expression recognition accuracy in real-life illumination conditions is achievable with machine learning technology. However, the data sets currently used in the automatic expression recognition literature to evaluate progress may be overly constrained and could potentially lead research into locally optimal algorithmic solutions.

Social robots face the fundamental challenge of detecting and adapting their behavior to the current social mood. For example, robots that assist teachers in early education must choose different behaviors depending on whether the children are crying, laughing, sleeping, or singing songs. Interactive robotic applications require perceptual algorithms that both run in real time and are adaptable to the challenging conditions of daily life. This paper explores a novel approach to auditory mood detection which was born out of our experience immersing social robots in classroom environments. We propose a new set of low-level spectral contrast features that extends a class of features which have proven very successful for object recognition in the modern computer vision literature. Features are selected and combined using machine learning approaches so as to make decisions about the ongoing auditory mood. We demonstrate excellent performance on two standard emotional speech databases (the Berlin Emotional Speech [W. Burkhardt et al., 2005], and the ORATOR dataset [H. Quast, 2001]). In addition we establish strong baseline performance for mood detection on a database collected from a social robot immersed in a classroom of 18-24 months old children [J. Movellan er al., 2007]. This approach operates in real time at little computational cost. It has the potential to greatly enhance the effectiveness of social robots in daily life environments.

Many popular optimization algorithms, like the Levenberg-Marquardt algorithm (LMA), use heuristic-based controllers'' that modulate the behavior of the optimizer during the optimization process. For example, in the LMA a damping parameter is dynamically modified based on a set rules that were developed using various heuristic arguments. Reinforcement learning (RL) is a machine learning approach to learn optimal controllers by examples and thus is an obvious candidate to improve the heuristic-based controllers implicit in the most popular and heavily used optimization algorithms. Improving the performance of off-the-shelf optimizers is particularly important for time-constrained optimization problems. For example the LMA algorithm has become popular for many real-time computer vision problems, including object tracking from video, where only a small amount of time can be allocated to the optimizer on each incoming video frame. Here we show that a popular modern reinforcement learning technique using a very simply state space can dramatically improve the performance of general purpose optimizers, like the LMA. Most surprisingly the controllers learned for a particular domain appear to work very well also on very different optimization domains. For example we used RL methods to train a new controller for the damping parameter of the LMA. This controller was trained on a collection of classic, relatively small, non-linear regression problems. The modified LMA performed better than the standard LMA on these problems. Most surprisingly, it also dramatically outperformed the standard LMA on a difficult large scale computer vision problem for which it had not been trained before. Thus the controller appeared to have extracted control rules that were not just domain specific but generalized across a wide range of optimization domains."

This paper presents the results of a panel discussion titled "The Future of HRI," held during an NSF workshop for graduate students on human-robot interaction in August 2006. The panel divided the workshop into groups tasked with inventing models of the field, and then asked these groups their opinions on the future of the field. In general, the workshop participants shared the belief that HRI can and should be seen as a single scientific discipline, despite the fact that it encompasses a variety of beliefs, methods, and philosophies drawn from several "core" disciplines in traditional areas of study. HRI researchers share many interrelated goals, participants felt, and enhancing the lines of communication between different areas would help speed up progress in the field. Common concerns included the unavailability of common robust platforms, the emphasis on human perception over robot perception, and the paucity of longitudinal real-world studies. The authors point to the current lack of consensus on research paradigms and platforms to argue that the field is not yet in the phase that philosopher Thomas Kuhn would call "normal science," but believe the field shows signs of approaching that phase.

The goal of the RUBI project is to accelerate progress in the development of social robots by addressing the problem at multiple levels, including the development of a scientific agenda, research methods, formal approaches, software, and hardware. The project is based on the idea that progress will go hand-in-hand with the emergence of a new scientific discipline that focuses on understanding the organization of adaptive behavior in real-time within the environments in which organisms operate. As such, the RUBI project emphasizes the process of design by immersion, i.e., embedding scientists, engineers and robots in everyday life environments so as to have these environments shape the hardware, software, and scientific questions as early as possible in the development process. The focus of the project so far has been on social robots that interact with 18 to 24 month old toddlers as part of their daily activities at the Early Childhood Education Center at the University of California, San Diego. In this document we present an overall assessment of the lessons and progress through year two of the project.

We present a systematic comparison of machine learning methods applied to the problem of fully automatic recognition of facial expressions. We report results on a series of experiments comparing recognition engines, including AdaBoost, support vector machines, linear discriminant analysis. We also explored feature selection techniques, including the use of AdaBoost for feature selection prior to classification by SVM or LDA. Best results were obtained by selecting a subset of Gabor filters using AdaBoost followed by classification with support vector machines. The system operates in real-time, and obtained 93% correct generalization to novel subjects for a 7-way forced choice on the Cohn-Kanade expression dataset. The outputs of the classifiers change smoothly as a function of time and thus can be used to measure facial expression dynamics. We applied the system to to fully automated recognition of facial actions (FACS). The present system classifies 17 action units, whether they occur singly or in combination with other actions, with a mean accuracy of 94.8%. We present preliminary results for applying this system to spontaneous facial expressions.

This paper reports progress in the development of a humanoid robot designed for real-time face-to-face interaction with humans. An essential component for face-to-face interaction with humans is being able to find faces, tracking them, and smoothly moving back and forth between gazing to a face and other objects of interest. In this paper we propose a system that integrates peripheral vision and foveal vision in a principled manner using particle filters. The developed system generates hypotheses about face position by using peripheral vision and verifies them by integrating peripheral vision and foveal vision. Even though a face may not be present in foveal vision, while the robot is gazing at another object, it keeps plausible hypotheses about the location of the human face with peripheral vision and restarts the face-following by verifying the hypothesis later. Moreover, this fundamental function provides more rich sensory information for human-robot interaction, such as a human's facial expression and lip motions during utterances.

This paper describes an automated tool for classifying tone quality (a quality related to timbre). This tool provides real-time visual feedback to players of clarinet to help improve tone production technique. A neural network architecture is employed to build a graphical biofeedback device that allows the user to immediately "see" what changes in technique lead to better tone-quality. The tone is also classified and probability estimates are shown in a bar graph, giving quantitative feedback to the user.