Event camera has unique advantages in high temporal resolution and dynamic range and has shown potentials in several computer vision tasks. However, due to the novelty of this hardware, there’s a lack of large benchmark DVS event-stream datasets, including datasets for object recognition. In this work, we proposed an encoder-decoder method to augment event stream dataset from image and optical flow with arbitrary temporal resolution for object recognition task. We believe this proposed method can be generalized well in augmenting event stream vision data for object recognition and will help advance the development of event vision paradigm.

Recent years have shown rapid development of digital fabrication techniques, making manufacturing individual models reachable for ordinary users. Thus, tools for designing customized objects in a user-friendly way are in high demand. In this paper, we tackle the problem of generating a collage of patterns along a given boundary, aimed at digital fabrication. We represent the packing space by a pipe-like closed shape along the boundary and use ellipses as packing elements for computing an initial layout of the patterns. Then we search for the best matching pattern for each ellipse and construct the initial pattern collage in an automatic manner. To facilitate editing the collage, we provide interactive operations which allow the user to adjust the layout at the coarse level. The patterns are fine-tuned based on a spring–mass system after each interaction step. After this interactive process, the collage result is further optimized to enforce connectivity. Finally, we perform structural analysis on the collage and enhance its stability, so that the result can be fabricated. To demonstrate the effectiveness of our method, we show results fabricated by 3D printing and laser cutting.

Image captioning is an important problem in artificial intelligence, related to both computer vision and natural language processing. There are two main problems in existing methods: in the training phase, it is difficult to find which parts of the captions are more essential to the image; in the caption generation phase, the objects or the scenes are sometimes misrecognized. In this paper, we consider the training images as the references and propose a Reference based Long Short Term Memory (R-LSTM) model, aiming to solve these two problems in one goal. When training the model, we assign different weights to different words, which enables the network to better learn the key information of the captions. When generating a caption, the consensus score is utilized to exploit the reference information of neighbor images, which might fix the misrecognition and make the descriptions more natural-sounding. The proposed R-LSTM model outperforms the state-of-the-art approaches on the benchmark dataset MS COCO and obtains top 2 position on 11 of the 14 metrics on the online test server.

Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language.In this paper, we pose the problem of multimodal sentiment analysis as modeling intra-modality and inter-modality dynamics.We introduce a novel model, termed Tensor Fusion Network, which learns both such dynamics end-to-end.The proposed approach is tailored for the volatile nature of spoken language in online videos as well as accompanying gestures and voice.In the experiments, our model outperforms state-ofthe-art approaches for both multimodal and unimodal sentiment analysis.

With the increasing popularity of video sharing websites such as YouTube and Facebook, multimodal sentiment analysis has received increasing attention from the scientific community. Contrary to previous works in multimodal sentiment analysis which focus on holistic information in speech segments such as bag of words representations and average facial expression intensity, we propose a novel deep architecture for multimodal sentiment analysis that is able to perform modality fusion at the word level. In this paper, we propose the Gated Multimodal Embedding LSTM with Temporal Attention (GME-LSTM(A)) model that is composed of 2 modules. The Gated Multimodal Embedding allows us to alleviate the difficulties of fusion when there are noisy modalities. The LSTM with Temporal Attention can perform word level fusion at a finer fusion resolution between the input modalities and attends to the most important time steps. As a result, the GME-LSTM(A) is able to better model the multimodal structure of speech through time and perform better sentiment comprehension. We demonstrate the effectiveness of this approach on the publicly-available Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis (CMU-MOSI) dataset by achieving state-of-the-art sentiment classification and regression results. Qualitative analysis on our model emphasizes the importance of the Temporal Attention Layer in sentiment prediction because the additional acoustic and visual modalities are noisy. We also demonstrate the effectiveness of the Gated Multimodal Embedding in selectively filtering these noisy modalities out. These results and analysis open new areas in the study of sentiment analysis in human communication and provide new models for multimodal fusion.