Understanding narrated instructional videos is important for both research and real-world web applications.Motivated by video dense captioning, we propose a model to generate procedure captions from narrated instructional videos which are a sequence of stepwise clips with description.Previous works on video dense captioning learn video segments and generate captions without considering transcripts.We argue that transcripts in narrated instructional videos can enhance video representation by providing fine-grained complimentary and semantic textual information.In this paper, we introduce a framework to ( 1) extract procedures by a cross-modality module, which fuses video content with the entire transcript; and (2) generate captions by encoding video frames as well as a snippet of transcripts within each extracted procedure.Experiments show that our model can achieve state-of-the-art performance in procedure extraction and captioning, and the ablation studies demonstrate that both the video frames and the transcripts are important for the task.

We study how to learn a semantic parser of state-of-the-art accuracy with less supervised training data.We conduct our study on WikiSQL, the largest hand-annotated semantic parsing dataset to date.First, we demonstrate that question generation is an effective method that empowers us to learn a state-ofthe-art neural network based semantic parser with thirty percent of the supervised training data.Second, we show that applying question generation to the full supervised training data further improves the state-of-the-art model.In addition, we observe that there is a logarithmic relationship between the accuracy of a semantic parser and the amount of training data.

We present a general solution towards building task-oriented dialogue systems for online shopping, aiming to assist online customers in completing various purchase-related tasks, such as searching products and answering questions, in a natural language conversation manner. As a pioneering work, we show what & how existing NLP techniques, data resources, and crowdsourcing can be leveraged to build such task-oriented dialogue systems for E-commerce usage. To demonstrate its effectiveness, we integrate our system into a mobile online shopping app. To the best of our knowledge, this is the first time that an AI bot in Chinese is practically used in online shopping scenario with millions of real consumers. Interesting and insightful observations are shown in the experimental part, based on the analysis of human-bot conversation log. Several current challenges are also pointed out as our future directions.

This paper presents how to generate questions from given passages using neural networks, where large scale QA pairs are automatically crawled and processed from Community-QA website, and used as training data.The contribution of the paper is 2-fold: First, two types of question generation approaches are proposed, one is a retrieval-based method using convolution neural network (CNN), the other is a generation-based method using recurrent neural network (RNN); Second, we show how to leverage the generated questions to improve existing question answering systems.We evaluate our question generation method for the answer sentence selection task on three benchmark datasets, including SQuAD, MS MARCO, and WikiQA.Experimental results show that, by using generated questions as an extra signal, significant QA improvement can be achieved.

Most current chatbot engines are designed to reply to user utterances based on existing utterance-response (or Q-R) 1 pairs.In this paper, we present DocChat, a novel information retrieval approach for chatbot engines that can leverage unstructured documents, instead of Q-R pairs, to respond to utterances.A learning to rank model with features designed at different levels of granularity is proposed to measure the relevance between utterances and responses directly.We evaluate our proposed approach in both English and Chinese: (i) For English, we evaluate Doc-Chat on WikiQA and QASent, two answer sentence selection tasks, and compare it with state-of-the-art methods.Reasonable improvements and good adaptability are observed.(ii) For Chinese, we compare DocChat with XiaoIce 2 , a famous chitchat engine in China, and side-by-side evaluation shows that DocChat is a perfect complement for chatbot engines using Q-R pairs as main source of responses.