Multi-horizon time series forecasting plays an important role in many industrial and business decision processes. To grasp complex and various patterns across different time series is the crucial step in achieving promising performance. However, most deep learning-based forecasting approaches simply take series-specific static (i.e. time-invariant) covariates as input features, which can fail to capture the complex pattern variation for each possible time series. In this paper, we propose a novel multiplicative attention-based architecture to tackle such forecasting problem. Our modification to multi-head attention layers leverages the series-specific covariates to build flexible attention functions for each possible time series. This improvement contributes to greater representation capacity to grasp different patterns across related time series. Experiment results demonstrate that our approach achieves state-of-the-art performance on a variety of real-world datasets.

Generating videos with semantic meaning, such as gestures in sign language, is a challenging problem. The model should not only learn to generate videos with realistic appearance, but also take notice of crucial details in frames to convey precise information. In this paper, we focus on the problem of generating long-term gesture videos containing precise and complete semantic meanings. We develop a novel architecture to learn the temporal and spatial transforms in regions of interest, i.e., gesticulating hands or face in our case. We adopt a hierarchical approach for generating gesture videos, by first making predictions on future pose configurations, and then using the encoder-decoder architecture to synthesize future frames based on the predicted pose structures. We develop the scheme of action progress in our architecture to represent how far the action has been performed during its expected execution, and to instruct our model to synthesize actions with various paces. Our approach is evaluated on two challenging datasets for the task of gesture video generation. Experimental results show that our method can produce gesture videos with more realistic appearance and precise meaning than the state-of-the-art video generation approaches.

This work develops a continuous sign language (SL) recognition framework with deep neural networks, which directly transcribes videos of SL sentences to sequences of ordered gloss labels. Previous methods dealing with continuous SL recognition usually employ hidden Markov models with limited capacity to capture the temporal information. In contrast, our proposed architecture adopts deep convolutional neural networks with stacked temporal fusion layers as the feature extraction module, and bidirectional recurrent neural networks as the sequence learning module. We propose an iterative optimization process for our architecture to fully exploit the representation capability of deep neural networks with limited data. We first train the end-to-end recognition model for alignment proposal, and then use the alignment proposal as strong supervisory information to directly tune the feature extraction module. This training process can run iteratively to achieve improvements on the recognition performance. We further contribute by exploring the multimodal fusion of RGB images and optical flow in sign language. Our method is evaluated on two challenging SL recognition benchmarks, and outperforms the state of the art by a relative improvement of more than 15% on both databases.

Models applied on real time response tasks, like click-through rate (CTR) prediction model, require high accuracy and rigorous response time. Therefore, top-performing deep models of high depth and complexity are not well suited for these applications with the limitations on the inference time. In order to get neural networks of better performance given the time limitations, we propose a universal framework that exploits a booster net to help train the lightweight net for prediction. We dub the whole process rocket launching, where the booster net is used to guide the learning of our light net throughout the whole training process. We analyze different loss functions aiming at pushing the light net to behave similarly to the booster net. Besides, we use one technique called gradient block to improve the performance of light net and booster net further. Experiments on benchmark datasets and real-life industrial advertisement data show the effectiveness of our proposed method.

This work presents a weakly supervised framework with deep neural networks for vision-based continuous sign language recognition, where the ordered gloss labels but no exact temporal locations are available with the video of sign sentence, and the amount of labeled sentences for training is limited. Our approach addresses the mapping of video segments to glosses by introducing recurrent convolutional neural network for spatio-temporal feature extraction and sequence learning. We design a three-stage optimization process for our architecture. First, we develop an end-to-end sequence learning scheme and employ connectionist temporal classification (CTC) as the objective function for alignment proposal. Second, we take the alignment proposal as stronger supervision to tune our feature extractor. Finally, we optimize the sequence learning model with the improved feature representations, and design a weakly supervised detection network for regularization. We apply the proposed approach to a real-world continuous sign language recognition benchmark, and our method, with no extra supervision, achieves results comparable to the state-of-the-art.

Alignment is an important preprocessing step for image information retrieval. With template information, images should be aligned precisely to retrieve information inside. These images may contain repeated characters or radicals, and their textures may be not appropriate for keypoint extractions. In this paper, a novel robust image alignment algorithm is proposed for images with text template using clustering and robust subspace learning. Experiment shows it can perform well even in extreme cases.

Recent progress on automatic generation of image captions has shown that it is possible to describe the most salient information conveyed by images with accurate and meaningful sentences. In this paper, we propose an image captioning system that exploits the parallel structures between images and sentences. In our model, the process of generating the next word, given the previously generated ones, is aligned with the visual perception experience where the attention shifts among the visual regions-such transitions impose a thread of ordering in visual perception. This alignment characterizes the flow of latent meaning, which encodes what is semantically shared by both the visual scene and the text description. Our system also makes another novel modeling contribution by introducing scene-specific contexts that capture higher-level semantic information encoded in an image. The contexts adapt language models for word generation to specific scene types. We benchmark our system and contrast to published results on several popular datasets, using both automatic evaluation metrics and human evaluation. We show that either region-based attention or scene-specific contexts improves systems without those components. Furthermore, combining these two modeling ingredients attains the state-of-the-art performance.

In most model-free tracking algorithms, context of the target is usually taken as source of negative examples for training the appearance model and thus not fully utilized. In fact, the target is embedded in its context, and there are spatial constraints between them with the potential motion correlations and relative locations. This paper presents a part-based model to describe the spatial constraints between the target object and its context. Auxiliary objects are selected as parts of the part based model to represent the context information. These auxiliary objects should have some properties: (i) co-occurrence with the target, (ii) having distinct appearance, and (iii) having homogeneous motion as an invisible visual region. A template-based model concerning about the object appearance is implemented to help estimate the reliability of tracking and make the part-based model updated properly. We test the proposed method on a recent benchmark with 50 video sequences evaluated, and a superior performance is achieved compared with state-of-the-arts.

In this paper, we propose a real-time system of hand pose estimation that infers the hand pose and shows the finger positions in each frame. The system is designed through data driven methodology. For the system to perform in real-time, we employ a retrieval method based on an inverted-file index with edge-based descriptors. To strengthen the discriminability, we combine a robust orientation assignment method with the descriptors. A novel alignment method is designed to transfer pose information from the retrieval results to the recognized image. To refine the retrieval results in the measurement of pose similarity, a mixed criterion that considers both outline deviation and descriptor matching is also proposed. The proposed system is quite scalable with respect to poses, since unobserved poses can be added freely. To evaluate the retrieval performance, we apply criteria that assess about the deviation of the corresponding positions of the query pose from the retrieval results. The evaluation is carried out on a retrieval dataset with manually marked images. Our proposed method effectively estimates pose at 20 frames per second, as demonstrated in our experiments on both real and publicly available datasets.