The safety and comfort of drivers have been improved over the decades as a result of our broadened understanding of driver modeling and behavior prediction. Despite these remarkable advances in autonomous and interactive systems, there is a significant lack of approaches that consider the passengers and the vehicle as components of a dynamical vibro-acoustical system. Sound in vehicles is not only informative of the state of the vehicle and the environment, but can also critically affect the driver's performance, attention, and comfort. This paper aims to investigate the interplay between the perceived sounds of a vehicle and psychoacoustic annoyance (PA) metrics. Our goal is to create an intelligent agent that would act to improve driving pleasantness through acoustic-driven learning. To tackle the problem of choosing the correct actions to reduce the acoustic annoyance, the paper presents a method based on reinforcement learning that learns from the environment, i.e., the vehicle interior. The method actively changes the state inside the vehicle (e.g., closing or opening the window and choosing the cruise speed) in order to minimize acoustic annoyance experienced by the driver. The results of this work, performed using the GTA V simulator, showed that the trained agent successfully learned to take the correct actions to reduce PA metrics. The paper also present to the community a new multi-modal dataset composed of several rides on a real vehicle and an in-depth analysis of the influence of vehicle's signal on the acoustic annoyance.

There are a lack of quantitative measures for clinically assessing upper limb function. Conventional biomechanical performance measures are restricted to specialist labs due to hardware cost and complexity, while the resulting measurements require specialists for analysis. Depth cameras are low cost and portable systems that can track surrogate joint positions. However, these motions may not be biologically consistent, which can result in noisy, inaccurate movements. This paper introduces a rigid body modelling method to enforce biological feasibility of the recovered motions. This method is evaluated on an existing depth camera assessment: the reachable workspace (RW) measure for assessing gross shoulder function. As a rigid body model is used, position estimates of new proximal targets can be added, resulting in a proximal function (PF) measure for assessing a subject's ability to touch specific body landmarks. The accuracy, and repeatability of these measures is assessed on ten asymptomatic subjects, with and without rigid body constraints. This analysis is performed both on a low-cost depth camera system and a gold-standard active motion capture system. The addition of rigid body constraints was found to improve accuracy and concordance of the depth camera system, particularly in lateral reaching movements. Both RW and PF measures were found to be feasible candidates for clinical assessment, with future analysis needed to determine their ability to detect changes within specific patient populations.

We present a smartphone-based system for real-time tele-monitoring of physical activity in patients with chronic heart-failure (CHF). We recently completed a pilot study with 15 subjects to evaluate the feasibility of the proposed monitoring in the real world and examine its requirements, privacy implications, usability, and other challenges encountered by the participants and healthcare providers. Our tele-monitoring system was designed to assess patient activity via minute-by-minute energy expenditure (EE) estimated from accelerometry. In addition, we tracked relative user location via global positioning system (GPS) to track outdoors activity and measure walking distance. The system also administered daily surveys to inquire about vital signs and general cardiovascular symptoms. The collected data were securely transmitted to a central server where they were analyzed in real time and were accessible to the study medical staff to monitor patient health status and provide medical intervention if needed. Although the system was designed for tele-monitoring individuals with CHF, the challenges, privacy considerations, and lessons learned from this pilot study apply to other chronic health conditions, such as diabetes and hypertension, that would benefit from continuous monitoring through mobile-health (mHealth) technologies.

Although the positive effects of exercise on the well-being and quality of independent living for older adults are well accepted, many elderly individuals lack access to exercise facilities, or the skills and motivation to perform exercise at home. To provide a more engaging environment that promotes physical activity, various fitness applications have been proposed. Many of the available products, however, are geared toward a younger population and are not appropriate or engaging for an older population. To address these issues, we developed an automated interactive exercise coaching system using the Microsoft Kinect. The coaching system guides users through a series of video exercises, tracks and measures their movements, provides real-time feedback, and records their performance over time. Our system consists of exercises to improve balance, flexibility, strength, and endurance, with the aim of reducing fall risk and improving performance of daily activities. In this paper, we report on the development of the exercise system, discuss the results of our recent field pilot study with six independently living elderly individuals, and highlight the lessons learned relating to the in-home system setup, user tracking, feedback, and exercise performance evaluation.

In this paper, we propose a method for temporal segmentation of human repetitive actions based on frequency analysis of kinematic parameters, zero-velocity crossing detection, and adaptive k-means clustering. Since the human motion data may be captured with different modalities which have different temporal sampling rate and accuracy (e.g., Optical motion capture systems vs. Microsoft Kinect), we first apply a generic full-body kinematic model with an unscented Kalman filter to convert the motion data into a unified representation that is robust to noise. Furthermore, we extract the most representative kinematic parameters via the primary frequency analysis. The sequences are segmented based on zero-velocity crossing of the selected parameters followed by an adaptive k-means clustering to identify the repetition segments. Experimental results demonstrate that for the motion data captured by both the motion capture system and the Microsoft Kinect, our proposed algorithm obtains robust segmentation of repetitive action sequences.

In this paper we address calibration of camera arrays for life-size 3D video acquisition and display where mosaicking and multiviewpoint stereo are combined to provide an immersive experience which, by its size, resolution, and three-dimensionality, is meant to rival being there. This coupling of multiview and mosaicking requires integration of numerous synchronized video streams in a single presentation, aligned for both panoramic blending and epipolar rectification. The calibration framework we have developed extends the classical checkerboard approach through a modular multi-stage pipeline performing global optimization across intrinsics, extrinsics, panoramas, multiview epipolar alignments, and color correction. We demonstrate the methodology on several multiview camera arrays with various configurations aimed at mosaicking and epipolar light-field analysis. The results of this analysis have driven real-time life-sized panoramic 3D displays of captured events such as a concert, a fashion show, and sports activity.

Over the years, a large number of methods have been proposed to analyze human pose and motion information from images, videos, and recently from depth data. Most methods, however, have been evaluated on datasets that were too specific to each application, limited to a particular modality, and more importantly, captured under unknown conditions. To address these issues, we introduce the Berkeley Multimodal Human Action Database (MHAD) consisting of temporally synchronized and geometrically calibrated data from an optical motion capture system, multi-baseline stereo cameras from multiple views, depth sensors, accelerometers and microphones. This controlled multimodal dataset provides researchers an inclusive testbed to develop and benchmark new algorithms across multiple modalities under known capture conditions in various research domains. To demonstrate possible use of MHAD for action recognition, we compare results using the popular Bag-of-Words algorithm adapted to each modality independently with the results of various combinations of modalities using the Multiple Kernel Learning. Our comparative results show that multimodal analysis of human motion yields better action recognition rates than unimodal analysis.

This article presents a psychophysical study that measures the perceptual thresholds of a new factor called Color-plus-Depth Level-of-Details (CZLoD) peculiar to polygon-based 3D tele-immersive video. The results demonstrate the existence of Just Noticeable Degradation and Just Unacceptable Degradation thresholds on the factor. In light of the results, we design and implement a real-time perception-based quality adaptor for 3D tele-immersive video. Our experimental results show that the adaptation scheme can reduce resource usage (e.g., CPU cycles) while considerably enhancing the overall perceived visual quality. Our analysis confirms the potential temporal and spatial performance benefits achievable with CZLoD adaptation.

This paper presents a psychophysical study that measures the perceptual thresholds of a new factor called Color-plus-Depth Level-of-Detail peculiar to polygon-based 3D tele-immersive video. The results demonstrate the existence of Just Noticeable Degradation and Just Unacceptable Degradation thresholds on the factor. In light of the results, we describe the design and implementation of a real-time perception-based quality adaptor for 3D tele-immersive video. Our experimental results show that the adaptation scheme can reduce resource usage while considerably enhancing the overall perceived visual quality.

This paper presents a psychophysical approach to control a new factor called Color-plus-Depth Level-of-Detail in polygon-based 3D tele-immersive video. Based on our psychophysical study that demonstrates the existence of perceptual thresholds on the factor, we present a real-time perception-based quality adaptor for 3D tele-immersive video. Our experimental results show that the adaptation scheme can reduce resource usage while considerably enhancing the overall perceived visual quality.

The growing popularity of 3-D movies has led to the rapid development of numerous affordable consumer 3-D displays. In contrast, the development of technology to generate 3-D content has lagged behind considerably. In spite of significant improvements to the quality of imaging devices, the accuracy of the algorithms that generate 3-D data, and the hardware available to render such data, the algorithms available to calibrate, reconstruct, and then visualize such data remain difficult to use, extremely noise sensitive, and unreasonably slow. In this paper, we present a multi-camera system that creates a highly accurate (on the order of a centimeter), 3-D reconstruction of an environment in real-time (under 30 ms) that allows for remote interaction between users. This paper focuses on addressing the aforementioned deficiencies by describing algorithms to calibrate, reconstruct, and render objects in the system. We demonstrate the accuracy and speed of our results on a variety of benchmarks and data collected from our own system.

Though the variety of desktop real time stereo vision systems has grown considerably in the past several years, few make any verifiable claims about the accuracy of the algorithms used to construct 3D data or describe how the data generated by such systems, which is large in size, can be effectively distributed. In this paper, we describe a system that creates an accurate (on the order of a centimeter), 3D reconstruction of an environment in real time (under 30 ms) that also allows for remote interaction between users. This paper addresses how to reconstruct, compress, and visualize the 3D environment. In contrast to most commercial desktop real time stereo vision systems our algorithm produces 3D meshes instead of dense point clouds, which we show allows for better quality visualizations. The chosen representation of the data also allows for high compression ratios for transfer to remote sites. We demonstrate the accuracy and speed of our results on a variety of benchmarks.

Though the quality of imaging devices, the accuracy of algorithms that construct 3D data, and the hardware available to render such data have all improved, the algorithms available to calibrate, reconstruct, and then visualize such data are difficult to use, extremely noise sensitive, and unreasonably slow. In this paper, we describe a multi-camera system that creates a highly accurate (on the order of a centimeter), 3D reconstruction of an environment in real time (under 30 ms) that allows for remote interaction between users. The paper addresses the aforementioned deficiencies by featuring an overview of the technology and algorithms used to calibrate, reconstruct, and render objects in the system. The algorithm produces partial 3D meshes, instead of dense point clouds, which are combined on the renderer to create a unified model of the environment. The chosen representation of the data allows for high compression ratios for transfer to remote sites. We demonstrate the accuracy and speed of our results on a variety of benchmarks and data collected from our own system.

Three-dimensional tele-immersive (3DTI) environments have great potential to promote collaborative work among geographically distributed users. However, most existing 3DTI systems only work with two sites due to the huge demand of resources and the lack of a simple yet powerful networking model to handle connectivity, scalability, and quality-of-service (QoS) guarantees. In this article, we explore the design space from the angle of multi-stream management to enable multi-party 3DTI communication. Multiple correlated 3D video streams are employed to provide a comprehensive representation of the physical scene in each 3DTI environment, and are rendered together to establish a common cyberspace among all participating 3DTI environments. The existence of multi-stream correlation provides the unique opportunity for new approaches in QoS provisioning. Previous work mostly concentrated on compression and adaptation techniques on the per-stream basis while ignoring the application layer semantics and the coordination required among streams. We propose an innovative and generalized ViewCast model to coordinate the multi-stream content dissemination over an overlay network. ViewCast leverages view semantics in 3D free-viewpoint video systems to fill the gap between high-level user interest and low-level stream management. In ViewCast, only the view information is specified by the user/application, while the underlying control dynamically performs stream differentiation, selection, coordination, and dissemination. We present the details of ViewCast and evaluate it through both simulation and 3DTI sessions among tele-immersive environments residing in different institutes across the Internet2. Our experimental results demonstrate the implementation feasibility and performance enhancement of ViewCast in supporting multi-party 3DTI collaboration.

Three-dimensional tele-immersive (3DTI) environments have great potential to promote collaborative work among geographically distributed users. However, most existing 3DTI systems only work with two sites due to the huge demand of resources and the lack of a simple yet powerful networking model to handle connectivity, scalability, and quality-of-service (QoS) guarantees.

In this paper, we present a framework for immersive 3D video conferencing and geographically distributed collaboration. Our multi-camera system performs a full-body 3D reconstruction of users in real time and renders their image in a virtual space allowing remote interaction between users and the virtual environment. The paper features an overview of the technology and algorithms used for calibration, capturing, and reconstruction. We introduce stereo mapping using adaptive triangulation which allows for fast (under 25 ms) and robust real-time 3D reconstruction. The chosen representation of the data provides high compression ratios for transfer to a remote site. The algorithm produces partial 3D meshes, instead of dense point clouds, which are combined on the renderer to create a unified model of the user. We have successfully demonstrated the use of our system in various applications such as remote dancing and immersive Tai Chi learning.

In this paper we present a framework for immersive virtual environment intended for remote collaboration and training of physical activities. Our multi-camera system performs full-body 3D reconstruction of human user(s) in real time and renders their image in the virtual space allowing remote users to interact. The paper features a short overview of the technology used for the capturing and reconstruction. Some of the applications where we have successfully demonstrated use of the system in combination with the tele-immersive virtual environment are described. Finally, we address current drawbacks with regard to data capturing and networking and provide some ideas for future work.

The study of 3D Tele-immersion impact on remote collaborative work represents a very interesting and challenging research topic. In this paper, we introduce the latest accomplishments of TEEVE research which merges computer science with dance choreography. This collaborative research model is ideal for creative, interdisciplinary problem solving. TEEVE offers an entirely new interface for dance choreography as a creative tool and alternative performance venue.

Tele-immersion is emerging as a new medium that creates 3D photorealistic, immersive, and interactive experience between geographically dispersed users. However, most existing tele-immersive systems can only support two-way collaboration. In this paper we propose a multi-layer framework and a new data dissemination protocol to support multi-site collaboration. The problem context is unique as multiple remote sites participate in an interactive tele-immersive session, where each site has multiple correlated 3D video streams to send (later referred as multi-stream/multi-site environments). The key challenge is to disseminate such large number of 3D live video streams among these sites subject to the bandwidth and latency constraints while satisfying QoS guarantees in visual quality. Among our findings is that the simple randomized algorithm outperforms many other static algorithms in the unique context. Moreover, the streams generated from one site have high semantic correlation, because often cameras at one site are shooting the same scene, only from different angles. We exploit the stream correlation in the multicast protocol to minimize the level of loss.

Real-time distributed multi-party/multi-stream systems are becoming more popular in many areas such as 3D tele-immersion, multi-camera conferencing and security surveillance. However, the construction of such systems in large scale is impeded by the huge demand of computing and networking resources and the lack of a simple yet powerful networking model to handle interconnection, scalability and quality of service (QoS) guarantees. We make two main contributions in the paper: (1) we propose a novel generalized ViewCast model for multi-party/multi-stream video-mediated systems that fills the gap between high-level user interest and low level per-stream management, and (2) we demonstrate the ViewCast model by applying it to the multi-party 3D Tele-Immersive (3DTI) collaboration among geographically dispersed users. More specifically, we show how the ViewCast model is used in supporting stream data dissemination, coordination and QoS management among multiple 3D tele-immersive environments. We present our experimental results in both real implementation and simulation to show that our ViewCast-based solution achieves high efficiency, scalability, and quality in supporting multi-party 3DTI collaboration.