Visual question answering (VQA) is a difficult multimodal task that requires answering questions about images. It requires a fine-grained level of understanding of both the visual content of the image and the textual content of the question. However, most of the existing models perform weakly in filtering noisy information and are unable to fuse features from multiple modalities effectively. To resolve the above restriction, we propose a novel multimodal gate fusion network (MGFN), which consists of an attention-on-attention interaction module (AoAIM) and a multimodal gate fusion module (MGFM). The role of AoAIM is to capture intra-modal and inter-modal dependencies and to filter out some irrelevant attention. The proposed MGFM can effectively fuse textual and visual features based on the relative importance of textual and visual modalities. We have performed many ablation experiments on the VQA-v2 dataset to validate the effectiveness of AoAIM and MGFM. The ablation experiments demonstrate that both AoAIM and MGFM play a key role in improving the performance of the model. By embedding these two modules, MGFN performs better than the previous state-of-the-art (SOTA) model on the VQA-v2 dataset. Particularly, the MGFN achieves an overall accuracy of 71.68% on the test-dev set and 72.12% on the test-std set.

Because of the limited context information, it is a challenging task to classify short texts. Most existing methods only focus on extracting high-quality local features or global features from text to construct text representations, which is not comprehensive enough. This article proposes a global–local enhancement network (GLEN), which can construct a high-quality text representation by integrating the global and local features of the text. To improve the quality of global and local features, a group-wise enhancement mechanism is introduced, which can effectively enhance the important features while weakening the unimportant features. For the information-loss problem of traditional pooling operation, we designed a global–local pooling mechanism, which can filter out local features that are more relevant to the whole information of the text. Seven benchmark datasets of text classification were selected to test the performance of the model, and GLEN obtained the best results on most datasets. The ablation experiment on GLEN shows that both the group-wise enhancement mechanism and the global–local pooling mechanism can effectively improve the performance of the model.

Recently, the existing visual question answering (VQA) models based on the attention mechanism have achieved state-of-art results. However, attention-based networks only rely on question guidance to capture relevant image features, ignoring the high-level semantic information of the image. Therefore, the key challenge of the VQA task lies in obtaining effective semantic embedding and fine-grained visual understanding during the reasoning process. In this research, we propose a novel visual-semantic dual channel network to answer related questions from both visual and semantic perspectives. Specifically, the visual channel uses the relational reasoning method with an attention mechanism to capture visual objects and their relations, while the semantic channel can capture high-level semantic information from the global and local image by the semantic attention module. We confirmed the effectiveness of the proposed model and each module through extensive experiments on two versions of VQA datasets. Interpretability shows that the visual-semantic dual channel network can dynamically model to infer the most relevant answer to the question.

Recently, various attention-based networks have achieved state-of-art results on image captioning tasks. However, this simple mechanism is insufficient to modelling and reasoning the relationships between the visual regions required for scene understanding. In this research, we propose a visual relational reasoning module to implicit learning semantic and spatial relationships between pairs of relevant visual objects and infers the feature output that is most relevant to the currently generated word. Furthermore, a context gate is introduced to dynamically control the contribution of visual region attention modules and visual relational reasoning module which allows predicting different words according to different type of features (visual or visual relationship). We evaluate our model on the MSCOCO dataset and achieved state-of-the-art results. Qualitative analysis shows that our visual relational reasoning model can dynamically model and reason the most relevant features of different types of generated words and improve the quality of the caption.

Text classification requires a comprehensive consideration of global and local information for the text. However, most methods only treat the global and local features of the text as two separate parts and ignore the relationship between them. In this paper, we propose a Dynamic Global-Local Attention Network based on Capsules (DGLA) that can use global features to dynamically adjust the importance of local features (e.g., sentence-level features or phrase-level features). The global features of the text are extracted by the capsule network, which can capture the mutual positional relationship of the input features to mine more hidden information. Furthermore, we have designed two global-local attention mechanisms within DGLA to measure the importance of two different local features and effectively leverage the advantages of these two attention mechanisms through the residual network. The performance of the model was evaluated on seven benchmark text classification datasets, and DGLA achieved the highest accuracy on all datasets. Ablation experiments show that the global-local attention mechanism can significantly improve the performance of the model.

Sina Weibo sentiment analysis technology provides the methods to survey public emotion about the related events or products in China. Most of the current works in sentiment analysis are to apply neural networks, such as convolution neural network (CNN), long short-term memory (LSTM), or C-LSTM. In this article, a novel structure of a hybrid neural network model is proposed to deal with the polysemy phenomena of words and topic confusion with Sina Weibo. First, the embeddings from language models (ELMo) and some statistical methods based on the corpus and sentiment lexicon are employed to extract the features. This method uses latent semantic relationships in different linguistic contexts and cooccurrence statistical features between words in Weibo. Second, for the classification model, unlike traditional C-LSTM which feeds CNN's output into LSTM, we employ several filters with variable window sizes to extract a sequence of high-level word representation in different granularity distributions of text data in multichannel CNN. At the same time, obtain the sentence representation in Bi-LSTM. Then, concatenate the outputs of multichannel CNN and Bi-LSTM. In conclusion, the results indicate that the proposed model performs better on the precision, recall, and F1-score for Weibo sentiment analysis.