We consider the topic of data imputation, a foundational task in machine learning that addresses issues with missing data. To that end, we propose MCFlow, a deep framework for imputation that leverages normalizing flow generative models and Monte Carlo sampling. We address the causality dilemma that arises when training models with incomplete data by introducing an iterative learning scheme which alternately updates the density estimate and the values of the missing entries in the training data. We provide extensive empirical validation of the effectiveness of the proposed method on standard multivariate and image datasets, and benchmark its performance against state-of-the-art alternatives. We demonstrate that MCFlow is superior to competing methods in terms of the quality of the imputed data, as well as with regards to its ability to preserve the semantic structure of the data.

Human-robot collaboration towards achieving a common goal is most effective when the robot has the capability to estimate the intentions and needs of its human partner, and to plan complementary actions accordingly. To this end, synergistic coupling between inference engines and task-planning algorithms is essential: the earlier the robot can anticipate the actions performed by its partner, the safer and more seamless the interaction between the two parties will be.In this work, we propose a perception-based analytics framework that incorporates discriminative and generative models, which together estimate the current and future class of an action being performed by a human. The analytics leverage a sequence of human skeletal joint locations extracted from a depth map video stream of the human partner. The generative model ingests current and previous joint positions and outputs a sequence of predicted future positions. The discriminative model produces a vector of probabilities indicating the likelihood that the future action belongs to each class within a set of action classes being considered. The information on current and future actions is fed to a task planning module which selects the robot collaborative action that better suits the estimated present and future human states.

Process monitoring and verification have a wide range of uses in the medical and healthcare fields. Currently, such tasks are often carried out by a trained specialist, which makes them expensive, inefficient, and time-consuming. Recent advances in automated video- and multimodal-data-based action and activity recognition have made it possible to reduce the extent of manual intervention required to effectively carry out process supervision tasks. In this paper, we propose algorithms for automated egocentric human action and activity recognition from multimodal data, with a target application of monitoring and assisting a user perform a multistep medical procedure. We propose a supervised deep multimodal fusion framework that relies on concurrent processing of motion data acquired with wearable sensors and video data acquired with an egocentric or body-mounted camera. We demonstrate the effectiveness of the algorithm on a public multimodal dataset and conclude that automated process monitoring via the use of multiple heterogeneous sensors is a viable alternative to its manual counterpart. Furthermore, we demonstrate that the application of previously proposed adaptive sampling schemes to the video processing branch of the multimodal framework results in significant performance improvements.

In recent years, Deep Learning has been successfully applied to multimodal learning problems, with the aim of learning useful joint representations in data fusion applications. When the available modalities consist of time series data such as video, audio and sensor signals, it becomes imperative to consider their temporal structure during the fusion process. In this paper, we propose the Correlational Recurrent Neural Network (CorrRNN), a novel temporal fusion model for fusing multiple input modalities that are inherently temporal in nature. Key features of our proposed model include: (i) simultaneous learning of the joint representation and temporal dependencies between modalities, (ii) use of multiple loss terms in the objective function, including a maximum correlation loss term to enhance learning of cross-modal information, and (iii) the use of an attention model to dynamically adjust the contribution of different input modalities to the joint representation. We validate our model via experimentation on two different tasks: video-and sensor-based activity classification, and audio-visual speech recognition. We empirically analyze the contributions of different components of the proposed CorrRNN model, and demonstrate its robustness, effectiveness and state-of-the-art performance on multiple datasets.

The tasks of color and hyperspectral image reconstruction have been addressed in the context of Compressive Sensing (CS) frameworks in the past. Traditional CS methodologies exploit the underlying assumption that images are intrinsically sparse in some domain in order to reconstruct the image with few linear measurements, relative to its original dimensionality. Since the different color or spectral planes of such imagery are usually correlated, exploiting joint sparsity principles has been shown to be beneficial. Specifically, images reconstructed from a given number of measurements with so-called Multiple Measurement Vector (MMV) approaches have better fidelity relative to those yielded by Single Measurement Vector (SMV) frameworks which don't exploit joint sparsity constraints. Standard MMV approaches, however, operate by vectorizing the data, and effectively fail to preserve the intrinsic high-dimensional structure of the imagery. In this paper, we introduce a tensorial MMV approach that exploits joint sparsity constraints across both spatial dimensions of the images as opposed to only the rows or columns, as in traditional vectorial approaches, while still leveraging joint sparsity assumptions across the color or spectral planes. We demonstrate empirically that our method provides better reconstruction fidelity given a fixed number of measurements, and that it is also more computationally efficient.

This paper investigates the discriminative capabilities of facial action units (AUs) exhibited by an individual while performing a task on a tablet computer in a semi-unconstrained environment. To that end, AUs are measured on a frame-by-frame basis from videos of 96 different subjects participating in a game-show-like quiz game that included a prize incentive. We propose a method that leverages the activation characteristics, as well as the temporal dynamics of facial behavior. In order to demonstrate the discriminative capabilities of the proposed approach, we perform identity matching across all subject pairs. Overall, the rank-1 matching performance of our algorithm ranges from 55% and up to 85%, on scenarios where the emotional disparity between the reference and query samples is largest and smallest, respectively. We believe these results represent a significant improvement relative to existing work relying on the use of AUs for human identification, in particular because the experimental settings guarantee that the facial expressions involved are spontaneous.

Hyperspectral imaging has a wide range of applications; however, due to the high dimensionality of the data involved, the complexity and cost of hyperspectral imagers can be prohibitive. Exploiting redundancies along the spatial and spectral dimensions of a hyperspectral image of a scene has created new paradigms that do away with the limitations of traditional imaging systems. While Compressive Sensing (CS) approaches have been proposed and simulated with success on already acquired hyperspectral imagery, most of the existing work relies on the capability to simultaneously measure the spatial and spectral dimensions of the hyperspectral cube. Most real-life devices, however, are limited to sampling one or two dimensions at a time, which renders a significant portion of the existing work unfeasible. In this paper we propose a novel CS framework that is a hybrid between traditional vectorized approaches and recently proposed tensorial approaches, and that is compatible with real-life devices both in terms of the acquisition and reconstruction requirements.