Imitation learning (IL) aims to extract knowledge from human experts' demonstrations or artificially created agents to replicate their behaviors. It promotes interdisciplinary communication and real-world automation applications. However, the process of replicating behaviors still exhibits various problems, such as the performance is highly dependent on the demonstration quality, and most trained agents are limited to perform well in task-specific environments. In this survey, we provide an insightful review on IL. We first introduce the background knowledge from development history and preliminaries, followed by presenting different taxonomies within IL and key milestones of the field. We then detail challenges in learning strategies and present research opportunities with learning policy from suboptimal demonstration, voice instructions, and other associated optimization schemes.

Video and audio streams are essential and mutually complementary in multimedia immersive application scenarios. Recent studies have explored the field of deep neural net-work application on multimedia production, e.g., visually guided generation of binaural audio, where Difference Mask (DM) is the predominant strategy in the state-of-the-art (SOTA) work. However, this strategy is not interpretable and requires adding the ground truth output as the input, limiting applicability. Besides, the generated audio has a relatively low spatial sensation. This paper aims to develop an interpretable and robust approach to visually guided binaural audio generation. Specifically, we generalize a concept and new strategy from Difference Mask, named Binaural Ratio, to interpret its binaural property relevant to the Inter-aural Time Difference (ITD) and Inter-aural Level Difference (ILD). In the new strategy, the model input can be natural and arbitrary mono audio instead of the direct sum of left and right audio, i.e., ground truth output. Moreover, we identify that one reason for the low spatial sensation is the bias toward mono. Thus, we tackle it by designing new network variants to learn the Binaural Ratio robustly. Experiments show that our proposed approach significantly outperforms the SOTA methods in both objective and subjective evaluation metrics.

The principal component analysis network (PCANet) is an unsupervised deep network, utilizing principal components as convolution filters in its layers. Albeit powerful, the PCANet suffers from two fundamental problems responsible for its performance degradation. First, the principal components transform the data as column vectors (which we call the amalgamated view) and incur a loss of spatial information present in the data. Second, the generalized pooling in the PCANet is unable to incorporate spatial statistics of the natural images, and it also induces redundancy among the features. In this research, we first propose a tensor-factorization-based deep network called the tensor factorization network (TFNet). The TFNet extracts features by preserving the spatial view of the data (which we call the minutiae view). We then proposed HybridNet, which simultaneously extracts information with the two views of the data since their integration can improve the performance of classification systems. Finally, to alleviate the feature redundancy among hybrid features, we propose Attn-HybridNet to perform attention-based feature selection and fusion to improve their discriminability. Classification results on multiple real-world datasets using features extracted by our proposed Attn-HybridNet achieves significantly better performance over other popular baseline methods, demonstrating the effectiveness of the proposed techniques.

Record linkage algorithms match and link records from different databases that refer to the same real-world entity based on direct and/or quasi-identifiers, such as name, address, age, and gender, available in the records. Since these identifiers generally contain personal identifiable information (PII) about the entities, record linkage algorithms need to be developed with privacy constraints. Known as privacy-preserving record linkage (PPRL), many research studies have been conducted to perform the linkage on encoded and/or encrypted identifiers. Differential privacy (DP) combined with computationally efficient encoding methods, e.g. Bloom filter encoding, has been used to develop PPRL with provable privacy guarantees. The standard DP notion does not however address other constraints, among which the most important ones are fairness-bias and cost of linkage in terms of number of record pairs to be compared. In this work, we propose new notions of fairness-constrained DP and fairness and cost-constrained DP for PPRL and develop a framework for PPRL with these new notions of DP combined with Bloom filter encoding. We provide theoretical proofs for the new DP notions for fairness and cost-constrained PPRL and experimentally evaluate them on two datasets containing person-specific data. Our experimental results show that with these new notions of DP, PPRL with better performance (compared to the standard DP notion for PPRL) can be achieved with regard to privacy, cost and fairness constraints.

Preventing water pipe leaks and breaks has high priority for water utilities. It is a critical task for the utility to reduce water loss through leaks and breaks detection in water mains. The failure prediction and data analytics research have been conducted for an Australian water utility over the last few years to enhance the prediction of leaks and breaks detection in water mains. Intelligent sensing at sensitive locations with current research aids in prioritising investigation and prevention of potential breaks and leaks in water mains. The purpose of this work is to integrate the predictive analytics and intelligent sensing applications to identify high risk mains prior to failures. Predictive analytics and minimum night flow (MNF) analysis have been utilised to prioritise risky zones over the whole water network, and then risky pipes are identified to optimise sensors deployment. The sensing data is being collected for analysis and validation, and a machine learning model is being built based on the analysis results. This work is currently under progress and the planned outcomes will help the utility reduce water loss, improve leak detection, and enhance customer satisfaction by automating the process of leak detection using a data driven approach with smart sensors.

Multimodal sentiment analysis utilizes multiple heterogeneous modalities for sentiment classification. The recent multimodal fusion schemes customize LSTMs to discover intra-modal dynamics and design sophisticated attention mechanisms to discover the inter-modal dynamics from multimodal sequences. Although powerful, these schemes completely rely on attention mechanisms which is problematic due to two major drawbacks 1) deceptive attention masks, and 2) training dynamics. Nevertheless, strenuous efforts are required to optimize hyperparameters of these consolidate architectures, in particular their custom-designed LSTMs constrained by attention schemes. In this research, we first propose a common network to discover both intra-modal and inter-modal dynamics by utilizing basic LSTMs and tensor based convolution networks. We then propose unique networks to encapsulate temporal-granularity among the modalities which is essential while extracting information within asynchronous sequences. We then integrate these two kinds of information via a fusion layer and call our novel multimodal fusion scheme as Deep-HOSeq (Deep network with higher order Common and Unique Sequence information). The proposed Deep-HOSeq efficiently discovers all-important information from multimodal sequences and the effectiveness of utilizing both types of information is empirically demonstrated on CMU-MOSEI and CMU-MOSI benchmark datasets. The source code of proposed Deep-HOSeq is available at https://github.com/sverma88/Deep-HOSeq-ICDM-2020.

Multimodal sentiment analysis combines information available from visual, textual, and acoustic representations for sentiment prediction. The recent multimodal fusion schemes combine multiple modalities as a tensor and obtain either; the common information by utilizing neural networks, or the unique information by modeling low-rank representation of the tensor. However, both of these information are essential as they render inter-modal and intra-modal relationships of the data. In this research, we first propose a novel deep architecture to extract the common information from the multi-mode representations. Furthermore, we propose unique networks to obtain the modality-specific information that enhances the generalization performance of our multimodal system. Finally, we integrate these two aspects of information via a fusion layer and propose a novel multimodal data fusion architecture, which we call DeepCU (Deep network with both Common and Unique latent information). The proposed DeepCU consolidates the two networks for joint utilization and discovery of all-important latent information. Comprehensive experiments are conducted to demonstrate the effectiveness of utilizing both common and unique information discovered by DeepCU on multiple real-world datasets. The source code of proposed DeepCU is available at https://github.com/sverma88/DeepCU-IJCAI19.

Growing awareness towards ethical use of machine learning (ML) models has created a surge for the development of fair models. Existing work in this regard assumes the presence of sensitive attributes in the data and hence can build classifiers whose decisions remain agnostic to such attributes. However, in the real world settings, the end-user of the ML model is unaware of the training data; besides, building custom models is not always feasible. Moreover, utilizing a pre-trained model with high accuracy on certain dataset can not be assumed to be fair. Unknown biases in the training data are the true culprit for unfair models (i.e., disparate performance for groups in the dataset). In this preliminary research, we propose a different lens for building fair models by enabling the user with tools to discover blind spots and biases in a pre-trained model and augment them with corrective measures.

Real world data is usually generated over multiple time periods associated with multiple labels, which can be represented as multiple labeled tensor sequences. These sequences are linked together, sharing some common features while exhibiting their own unique features. Conventional tensor factorization techniques are limited to extract either common or unique features, but not both simultaneously. However, both types of these features are important in many machine learning systems as they inherently affect the systems' performance. In this paper, we propose a novel supervised tensor factorization technique which simultaneously extracts ordered common and unique features. Classification results using features extracted by our method on CIFAR-10 database achieves significantly better performance over other factorization methods, illustrating the effectiveness of the proposed technique.