Modeling the moving behaviors and predicting the future paths of pedestrians, especially for those in complex scenes, remain a challenging problem in machine learning. We recognize that human motion trajectories, governed by social norms and constrained by physical structures of the surrounding environment, are both forward predictable and backward predictable. Motivated by this observation, we develop a new approach, calledreciprocal twin networks, for human trajectory learning and prediction. We design two networks, a forward prediction network to predict future trajectory from past observations and a backward prediction that performs the trajectory prediction backward in time. The backward prediction network serves as the inverse operation of the forward prediction network, forming a reciprocal constraint. During the training stage, this reciprocal constraint allows them to be jointly learned for accurate and robust human trajectory prediction. During the inference stage, we borrow the concept of adversarial attack of deep neural networks, which iteratively modifies the input of the network to match the given or forced network output, and develop a new method, calledreciprocal attack for matched prediction, to achieve accurate human trajectory prediction. Our experimental results on benchmark datasets demonstrate that our new method outperforms the state-of-the-art methods for human trajectory prediction.

Deep neural networks recognize objects by analyzing local image details and summarizing their information along the inference layers to derive the final decision. Because of this, they are prone to adversarial attacks. On the other hand, human eyes recognize objects based on their global structures and semantic cues, instead of local image textures. In this work, we propose to develop a structure-oriented progressive low-rank image completion method to remove unneeded texture details from the input images and shift the bias of deep neural networks towards global object structures and semantic cues. We formulate the problem into a low-rank matrix completion problem with progressively smoothed rank functions to avoid local minimums. Our experimental results demonstrate the proposed method is able to successfully remove the insignificant local image details while preserving important global object structures.

Media content has become a primary source of information, entertainment, and even education. The ability to provide video content querying as well as interactive experiences is a new challenge. To this end, question answering (QA) systems such as Alexa and Google Assistant have become quite established in consumer markets but are limited to general information and lack context awareness. In this paper, we propose Context-QA, a light-weight context-aware QA framework, to provide QA experiences on multimedia content. The context awareness is achieved through our innovative Staged QA Controller algorithm that keeps the search for answers in the context most relevant to the question. Our evaluation results show that Context-QA improves the quality of the answers by up to 49% and uses up to 56% less time compared to the conventional QA model. Subjective tests show Context-QA improved results over conventional QA models, with 90% reporting enjoying this new media form.

Deep neural networks are fragile under adversarial attacks. In this work, we propose to develop a new defense method based on image restoration to remove adversarial attack noise. Using the gradient information back-propagated over the network to the input image, we identify high-sensitivity keypoints which have significant contributions to the image classification performance. We then partition the image pixels into the two groups: high-sensitivity and low-sensitivity points. For low-sensitivity pixels, we use a total variation (TV) norm-based image smoothing method to remove adversarial attack noise. For those high-sensitivity keypoints, we develop a structure-preserving low-rank image completion method. Based on matrix analysis and optimization, we derive an iterative solution for this optimization problem. Our extensive experimental results on the CIFAR-10, SVHN, and Tiny-ImageNet datasets have demonstrated that our method significantly outperforms other defense methods which are based on image de-noising or restoration, especially under powerful adversarial attacks.

In this work, we develop a joint sample discovery and iterative model evolution method for semi-supervised learning on very small labeled training sets. We propose a master-teacher-student model framework to provide multi-layer guidance during the model evolution process with multiple iterations and generations. The teacher model is constructed by performing an exponential moving average of the student models obtained from past training steps. The master network combines the knowledge of the student and teacher models with additional access to newly discovered samples. The master and teacher models are then used to guide the training of the student network by enforcing the consistency between their predictions of unlabeled samples and evolve all models when more and more samples are discovered. Our extensive experiments demonstrate that the process of discovering confident samples from the unlabeled dataset, once coupled with the master-teacher-student network evolution, can significantly improve the overall semi-supervised learning performance. For example, on the CIFAR-10 dataset, with a small set of 250 labeled samples, our method achieves an error rate of 11.58%, more than 38% lower than Mean-Teacher (49.91%). When coupled with the MixMatch augmentation and loss function, the improvements are also significant.

We observe that the human trajectory is not only forward predictable, but also backward predictable. Both forward and backward trajectories follow the same social norms and obey the same physical constraints with the only difference in their time directions. Based on this unique property, we develop a new approach, called reciprocal learning, for human trajectory prediction. Two networks, forward and backward prediction networks, are tightly coupled, satisfying the reciprocal constraint, which allows them to be jointly learned. Based on this constraint, we borrow the concept of adversarial attacks of deep neural networks, which iteratively modifies the input of the network to match the given or forced network output, and develop a new method for network prediction, called reciprocal attack for matched prediction. It further improves the prediction accuracy. Our experimental results on benchmark datasets demonstrate that our new method outperforms the state-of-the-art methods for human trajectory prediction.