In the field of autonomous driving and intelligent transportation systems, 3D object detection plays a critical role in ensuring safe and efficient driving. Achieving reliable object detection relies on robust sensing technologies, where LiDAR provides accurate spatial perception, and radar offers extended detection range and speed information because of its longer wave-lengths. To capitalize on the strengths of both sensors, a novel RCS-based LiDAR-Radar fusion network, named RLRFusion, is proposed for 3D object detection in this paper. The network takes LiDAR and radar point clouds as inputs and processes them through dual bird's-eye view (BEV) feature extraction streams, followed by the BEV fusion and detection module to produce the detection results. In input-level fusion, a cross-modal pillar encoder is introduced to address the sparse radar data and its lack of height information. In feature-level fusion, an RCS-aware fusion encoder leverages the Radar Cross Section (RCS) distribution by mapping pillar features to their surroundings, enhancing object size estimation and mitigating the challenges faced by LiDAR in adverse weather conditions. Experimental results show that RLRFusion achieves competitive performance on the nuScenes dataset, with strong detection results even in rainy conditions. The source code of our method is available at: https://github.com/djZzgroupIRLRFusion.

This paper addresses the efficiency challenge of Neural Architecture Search (NAS) by formulating the task as a ranking problem.Previous methods require numerous training examples to estimate the accurate performance of architectures, although the actual goal is to find the distinction between "good" and "bad" candidates.Here we do not resort to performance predictors.Instead, we propose a performance ranking method (RankNAS) via pairwise ranking.It enables efficient architecture search using much fewer training examples.Moreover, we develop an architecture selection method to prune the search space and concentrate on more promising candidates.Extensive experiments on machine translation and language modeling tasks show that RankNAS can design high-performance architectures while being orders of magnitude faster than state-ofthe-art NAS systems.

Multimodal summarization with multimodal output (MSMO) is to generate a multimodal summary for a multimodal news report, which has been proven to effectively improve users' satisfaction. The existing MSMO methods are trained by the target of text modality, leading to the modality-bias problem that ignores the quality of model-selected image during training. To alleviate this problem, we propose a multimodal objective function with the guidance of multimodal reference to use the loss from the summary generation and the image selection. Due to the lack of multimodal reference data, we present two strategies, i.e., ROUGE-ranking and Order-ranking, to construct the multimodal reference by extending the text reference. Meanwhile, to better evaluate multimodal outputs, we propose a novel evaluation metric based on joint multimodal representation, projecting the model output and multimodal reference into a joint semantic space during evaluation. Experimental results have shown that our proposed model achieves the new state-of-the-art on both automatic and manual evaluation metrics. Besides, our proposed evaluation method can effectively improve the correlation with human judgments.

Neural architecture search (NAS) has advanced significantly in recent years but most NAS systems restrict search to learning architectures of a recurrent or convolutional cell.In this paper, we extend the search space of NAS.In particular, we present a general approach to learn both intra-cell and inter-cell architectures (call it ESS).For a better search result, we design a joint learning method to perform intra-cell and inter-cell NAS simultaneously.We implement our model in a differentiable architecture search system.For recurrent neural language modeling, it outperforms a strong baseline significantly on the PTB and Wiki-Text data, with a new state-of-the-art on PTB.Moreover, the learned architectures show good transferability to other systems.E.g., they improve state-of-the-art systems on the CoNLL and WNUT named entity recognition (NER) tasks and CoNLL chunking task, indicating a promising line of research on large-scale prelearned architectures.

In encoder-decoder neural models, multiple encoders are in general used to represent the contextual information in addition to the individual sentence.In this paper, we investigate multi-encoder approaches in document-level neural machine translation (NMT).Surprisingly, we find that the context encoder does not only encode the surrounding sentences but also behaves as a noise generator.This makes us rethink the real benefits of multi-encoder in context-aware translation -some of the improvements come from robust training.We compare several methods that introduce noise and/or well-tuned dropout setup into the training of these encoders.Experimental results show that noisy training plays an important role in multi-encoder-based NMT, especially when the training data is small.Also, we establish a new state-of-the-art on IWSLT Fr-En task by careful use of noise generation and dropout methods.

Traditional neural machine translation is limited to the topmost encoder layer's context representation and cannot directly perceive the lower encoder layers.Existing solutions usually rely on the adjustment of network architecture, making the calculation more complicated or introducing additional structural restrictions.In this work, we propose layer-wise multi-view learning to solve this problem, circumventing the necessity to change the model structure.We regard each encoder layer's off-the-shelf output, a by-product in layer-by-layer encoding, as the redundant view for the input sentence.In this way, in addition to the topmost encoder layer (referred to as the primary view), we also incorporate an intermediate encoder layer as the auxiliary view.We feed the two views to a partially shared decoder to maintain independent prediction.Consistency regularization based on KL divergence is used to encourage the two views to learn from each other.Extensive experimental results on five translation tasks show that our approach yields stable improvements over multiple strong baselines.As another bonus, our method is agnostic to network architectures and can maintain the same inference speed as the original model.

With the booming of content “re-creation” in social media platforms, character-orientedvideo summary has become a crucial form of user-generated video content. However, artificial extraction could be time-consuming with high missing rate, while traditional techniques on person search may incur heavy burden of computing resources. At the same time, in social media platforms, videos are usually accompanied with rich textual information, e.g., subtitles or bullet-screen comments which provide the multi-view description of videos. Thus, there exists a potential to leverage textual information to enhance the character-oriented video summarization. To that end, in this paper, we propose a novel framework for jointly modeling visual and textual information. Specifically, we first locate characters indiscriminately through detection methods, and then identify these characters via re-identification to extract potential key-frames, in which appropriate source of textual information will be automatically selected and integrated based on the features of specific frame. Finally, key-frames will be aggregated as the character-oriented summarization. Experiments on real-world data sets validate that our solution outperforms several state-of-the-art baselines on both person search and summarization tasks, which prove the effectiveness of our solution on the character-oriented video summarization problem.

The field of conversation generation using neural networks has attracted increasing attention from researchers for several years. However, traditional neural language models tend to generate a generic reply with poor semantic logic and no emotion. This article proposes an emotional conversation generation model based on a Bayesian deep neural network that can generate replies with rich emotions, clear themes, and diverse sentences. The topic and emotional keywords of the replies are pregenerated by introducing commonsense knowledge in the model. The reply is divided into multiple clauses, and then a multidimensional generator based on the transformer mechanism proposed in this article is used to iteratively generate clauses from two dimensions: sentence granularity and sentence structure. Subjective and objective experiments prove that compared with existing models, the proposed model effectively improves the semantic logic and emotional accuracy of replies. This model also significantly enhances the diversity of replies, largely overcoming the shortcomings of traditional models that generate safe replies.

Transformer is the state-of-the-art model in recent machine translation evaluations. Two strands of research are promising to improve models of this kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de facto standard for development of the Transformer system, and the other uses deeper language representation but faces the difficulty arising from learning deep networks. Here, we continue the line of research on the latter. We claim that a truly deep Transformer model can surpass the Transformer-Big counterpart by 1) proper use of layer normalization and 2) a novel way of passing the combination of previous layers to the next. On WMT’16 English-German and NIST OpenMT’12 Chinese-English tasks, our deep system (30/25-layer encoder) outperforms the shallow Transformer-Big/Base baseline (6-layer encoder) by 0.4-2.4 BLEU points. As another bonus, the deep model is 1.6X smaller in size and 3X faster in training than Transformer-Big.

How people divide their attention across their friends can help to understand key issues in the realm of geopolitics. Such attention exploration allows us to compare people who focus a large portion of their attention on a small set of close friends with those disperse their attention more widely. Using 2.5 million twitter data written by 130 thousands users, we find the balance of attention is a relatively stable property of people across different modalities of interaction. It displays subtle variation across people with different characteristics and different modalities of interaction. Specifically, people's attention is more focused in mention interactions, while those active in socialization tend to allocate higher portion of total attention to their close friends. Besides external interactions, people's inner interests also affect their attention allocation. People spreading multiple memes tend to be focused, and those with more even distribution of memes are focused on their intimate friends. Finally, people's relationships also plays an important role in their attention allocation. People are more likely to focus their attention on those most like them, and this similarity sequentially enhances the intimate relationship between them.

Sentiment analysis, which is a fundamental research in the field of natural language processing and artificial intelligence field, has received much attention these years because of its practical applicability and the challenges. However, existing methods only focus on local text information and ignore the background knowledge (such as the director of a movie, the producer of a product). In this paper, we propose a novel LSTM with Background Knowledge Attention Model (LSTM-BKAM) for sentiment analysis. Our model incorporates background knowledge based attentions over different semantic parts of a sentence. The experiment results show that our model achieves state-of-the-art, and substantially better than other approaches.

Topic classification plays an important role in facilitating security-related applications, which can help people reduce data scope and acquire key information quickly. Conversation is one of the important ways of communication between people. The utterances in a conversation may contain vital clues, such as people's opinions, emotions and political slants. To explore more effective approaches for Chinese conversational topic classification, in this paper, we propose a neural network architecture with pre-trained contextualized representation. We firstly apply pretrained BERT model to fine-tune and generate the conversational embeddings, which are the inputs of our neural network models. Then we design several models based on neural networks to extract task-oriented advanced features for topic classification. Experimental results indicate that the models based on our neural network architecture all outperform the baseline only fine-tuned with the pre-trained BERT model. It demonstrates that the pretrained representations are effective to Chinese conversational topic classification, and the proposed architecture can further capture the salient features from the representations. And we release the code and dataset of this paper that can be obtained from https://github.com/njoe9/pretrained_representation.

Spoken Language Understanding (SLU), which typically involves intent determination and slot filling, is a core component of spoken dialogue systems.Joint learning has shown to be effective in SLU given that slot tags and intents are supposed to share knowledge with each other.However, most existing joint learning methods only consider joint learning by sharing parameters on surface level rather than semantic level.In this work, we propose a novel self-attentive model with gate mechanism to fully utilize the semantic correlation between slot and intent.Our model first obtains intent-augmented embeddings based on neural network with self-attention mechanism.And then the intent semantic representation is utilized as the gate for labelling slot tags.The objectives of both tasks are optimized simultaneously via joint learning in an end-to-end way.We conduct experiment on popular benchmark ATIS.The results show that our model achieves state-of-the-art and outperforms other popular methods by a large margin in terms of both intent detection error rate and slot filling F1-score.This paper gives a new perspective for research on SLU.

Spoken language understanding (SLU), which mainly involves intent prediction and slot filling, is a core component of a spoken dialogue system. Usually, intent determination and slot filling are carried out independently. Recently, joint learning of intent determination and slot filling has been proved effective in SLU. In this paper, we propose a novel joint multi-task learning framework for SLU, which predicts user intent and slot label via shared LSTM architecture, as well as next word's part of speech (POS) via neural language model. The proposed model exploits the correlation among different tasks and makes full advantage of all supervised signals. We conduct experiments on popular benchmark ATIS dataset, which consists of rich dialogues collected from real world. The experiment results show that our model achieves state-of-the-art in terms of several popular metrics.

Generative Adversarial Networks (GAN), which are capable of generating realistic synthetic real-valued data, have achieved great progress in machine learning field. However, generator in GAN framework requires being differentiable, which means that the generator cannot produce discrete data, and it poses great challenge for GAN applied in Natural Language Processing (NLP) research. To unlock the potential of GAN in NLP, we develop a novel Text-to-Text Generative Adversarial Networks (TT-GAN), through which we can get generated text based on semantic information translated from source text. We demonstrate that our model can generate not only realistic texts, but also the source text's paraphrase or its semantic summarization. As our best knowledge, it is the first framework capable of generating natural language on semantic level in real sense, and gives a new perspective to apply GAN on NLP research.

A fixed-length feature vector is required for many machine learning algorithms in NLP field. Word embeddings have been very successful at learning lexical information. However, they can't capture the compositional meaning of sentences, which prevents them from a deeper understanding of language. In this paper, we introduce a novel hierarchical tree long short-term memory (HTLSTM) model that learns vector representations for sentences of arbitrary syntactic type and length. We propose to split one sentence into three hierarchies: short phrase, long phrase and full sentence level. The HTLSTM model gives our algorithm the potential to fully consider the hierarchical information and longterm dependencies of language. We design the experiments on both English and Chinese corpus to evaluate our model on sentiment analysis task. And the results show that our model outperforms several existing state of the art approaches significantly.

Paraphrases are sentences or phrases that convey the same meaning using different words. Paraphrase recognition is of interest for many current Natural Language Processing (NLP) tasks. As understood in linguistics, the phenomenon of paraphrases is difficult to characterize. In this article, we present a novel approach to the task of paraphrase identification. The proposed approach measures similarity between two sentences based on both the lexical and semantic levels, via combining neural networks and keywords jointly. In particular, we employ a vector offset, which implies the relation of given inputs in vector space, as the representation of a neural classifier. We conduct experiments on the Microsoft Research Paraphrase Corpus (MSRP)1and SICK dataset, which are both standard datasets for evaluating approaches to paraphrase identification. The experiments showed that our proposed approach makes much progress and achieves state-of-the-art results.

Word segmentation is the first step in Chinese natural language processing, and the error caused by word segmentation can be transmitted to the whole system. In order to reduce the impact of word segmentation and improve the overall performance of Chinese short text classification system, we propose a hybrid model of character-level and word-level features based on recurrent neural network (RNN) with long short-term memory (LSTM). By integrating character-level feature into word-level feature, the missing semantic information by the error of word segmentation will be constructed, meanwhile the wrong semantic relevance will be reduced. The final feature representation is that it suppressed the error of word segmentation in the case of maintaining most of the semantic features of the sentence. The whole model is finally trained end-to-end with supervised Chinese short text classification task. Results demonstrate that the proposed model in this paper is able to represent Chinese short text effectively, and the performances of 32-class and 5-class categorization outperform some remarkable methods.

Emotion entrainment accounts for the rhythmic convergence of human emotions through social interactions. This phenomenon abounds in various disciplines, i.e. effervescency in soccer games, anger proliferation in violence incidents, or anxiety diffusion in disasters. Although emotion entrainment is highly relevant to the quality of human daily life, the principles underpinning this phenomenon is still unclear. Previous dynamic models try to explain entrainment phenomenon by assuming symmetrical coupling among identical individuals. Yet this assumption clearly does not hold in real-world human interactions. As such, we propose an alternative model that captures asymmetric relationships. In depicting the coupling mechanism, the effect of social influence is also encoded. Experimental results on two emergent social events suggest that the proposed model characterizes emotion trends with high accuracy. Also, we explain the emotion dynamics by analyzing the reconstructed entrainment matrix. Our work may present practical implications for those who want to guide or regulate the emotion evolution in emergency events discussed online.

The sociological theory of entrainment accounts for the synchronization of human rhythmic modalities through social interactions: they coordinate in a variety of dimensions including linguistic styles, facial expressions, music pace, applause, and so on. Though highly relevant, emotion entrainment has received little attention to date. In addition, most previous studies on entrainment are done through small scale or controlled laboratory studies. In this paper, we investigate emotion entrainment in the context of online social media. To the best of our knowledge, this is the first time that emotion entrainment has been examined on a large scale, real world setting. For this purpose, we propose a framework that can model entrainment phenomenon and measure its effect. Our framework differentiates from previous research by its model-free essential and discerning in entrainment directions. These traits enable us to model entrainment dynamics under few assumptions, and distinguish emotion flow of entrainment. In our studies, we investigate entrainment patterns under different emotion states, i.e. positive, neutral and negative. We discover that entrainments under different emotions all follow a power law distribution. Besides, people are willing to entrain to others under positive emotion, and users with positive emotion are more likely to be entrained. By inspecting the interactions between entrainment and emotion, we reveal that entrainment has an effect of negotiating different emotion types toward an even distribution.