Inspired by structured state space models and graph neural network modeling, we proposes a novel graph-aware reasoning (GAR) model to effectively solve the problem between memory utilization efficiency and reasoning navigation. First, we introduce graph networks into the navigation framework to enhance the model modeling ability for long sequence dependencies. It is integration graph neural networks into the state space to ensures that the agent can accurately find topological paths based on instructions and environmental interactions. Then, we design a bidirectional selective state space model to enhance the agent-aware of spatial information. When without relying on the attention network, we capture the context information in the image through position embedding. Finally, we fuison the features in the bidirectional state space and effectively compresses and improves the fine-grained image features by a residual connection methods. Our experimental results on R2R and REVERIE datasets show that GAR reduces GPU memory usage by 4.45% compared with the DUET baseline model.

Visual language navigation is an exciting and challenging multi-modal task. Most existing research focuses on the fusion of visual features and semantic space, which ignoring the importance of local highlight features and semantic knowledge alignment in images for agent navigation. Therefore, this paper proposes a novel visual language model combining Knowledge-augmented Reasoning and Soft-Prompt (KRSP) learning. First, we perform fine-grained processing of local regions in the image and to map context image features and text knowledge to the same common sub-space. We focus on regional knowledge to increase the model reasoning ability. Next, soft-prompt learning aligns keywords and sub-visual information in instruction features to solve the path mismatch problem in coarse-grained instructions. We use a large-scale pre-training model CoCoOp to collect highly matched soft action prompts into a unified instruction set. Finally, we propose a general cross-modal feature alignment loss function. The potential semantic correlation between sub-visual information and instruction space is closer through the penalty mechanism of the alignment function. This paper verifies the method effectiveness on the R2R and REVERIE datasets, and the experimental results show that KRSP achieves state-of-the-art performance. Among them, the KRSP of SPL evaluation metric increased by 4.5% in unseen scenarios.

Large-scale multi-agent path finding (MAPF ) has the problem of shared features, which leads to a large amount of bandwidth and additional overhead. Therefore, this paper proposes a multi-agent path finding method that combines heuristic action-aware network and dual Q network(HADQ). First, we use a convolutional neural network to encoder observation input at field of view scope. On this basis, we combine self-attention network and graph neural network to query the priority of agent communication to reduce collisions. we improves the timeliness of agents timely communication by reducing the size of shared features. Then, we embedding the agent shortest path into the training process as a heuristic guide. Finally, this paper uses the deep Q network to map the observation values into the actions of the agent, thereby constructing a dynamic communication topology network. Compared with the state-of-the-art MAPF method, the method proposed in this paper has certain improvements in solution quality and success rate. Experimental results show that even in unseen data sets. HADQ can also show good generalization ability and robustness.

Image captioning is a cross-modal problem combining computer vision and natural language processing. A typical image captioning model uses a convolutional neural network to extract the features of an image and then uses an Long Short-Term Memory network to transform the representations of the features. However, this method has problems such as not including high-level semantics in the visual network and exposure bias in the language network. To overcome these problems, this paper proposes a novel image captioning model that combines relationship-aware and reinforcement learning. First, we design a relational awareness network as the visual network to mine the latent relationships between objects in an image. Then, a context semantic relational network is proposed to improve the accuracy of image captioning. The context semantic network can generate feature representations for arbitrary pixel positions in an image without association with any specific visual concepts. Subsequently, the high-level context semantics are used as external knowledge to guide the language network in generating sentences. Finally, a policy gradient training algorithm is designed to simplify the state value function in reinforcement learning. We have verified the effectiveness of the model on the MS-COCO and Flickr 30K datasets. The experimental results show that the model proposed in this paper achieves state-of-the-art results.

Image captioning is an interesting and challenging task. The previously established image captioning approach is based mainly on the encoder-decoder architecture, but it suffers from problems such as inaccurate captioning information, and the generated captioning sentences are not sufficiently rich. This paper proposes a novel image captioning model that is based on a self-attention network and a scene graph relationship network. First, an improved self-attention network is added to the extraction of visual features to evaluate the effectiveness of image global information for image generation. Then, we design a visual intensity parameter to coordinate the strategies of visual features and language model for word generation. Finally, a graph convolutional network is designed to extract the relationships from the scene information to render the generated caption more exciting and to increase the accuracy of the fine-grained captioning. We demonstrated the satisfactory performance of the model on the MS-COCO and Flickr 30K datasets. The experimental results demonstrate that the proposed model realizes state-of-the-art performance.

Image captioning is a multi-modal complex task in machine learning. Traditional methods focus only on entities in visual strategy networks, and can't reason about the relationship between entities and attributes. There are problems of exposure bias and error accumulation in language strategy networks. To this end, this paper proposes a multi-level visual fusion network model based on reinforcement learning. In the visual strategy network, multi-level neural network modules are used to transform visual features into feature sets of visual knowledge. The fusion network generates function words that make the description more fluent, and is used for the interaction between the visual strategy network and the language strategy network. The self-criticism strategy gradient algorithm based on reinforcement learning in language strategy networks is used to achieve end-to-end optimization of visual fusion networks. We evaluated our model on the Flickr 30K and MS-COCO datasets, and verified the accuracy of the model and the diversity of model learning subtitles through experiments. Our model achieves better performance over state-of-the-art methods.