With the development of steel materials, metallographic analysis has become increasingly important. Unfortunately, grain size analysis is a manual process that requires experts to evaluate metallographic photographs, which is unreliable and time-consuming. To resolve this problem, we propose a novel classification method based on deep learning, namely, GSNets, a family of hybrid models, which can effectively introduce guided self-attention for classifying grain size. Concretely, we build our models from three insights: 1) Introducing our novel guided self-attention module can assist the model in finding the generalized necessarily distinct vectors capable of retaining intricate relational connections and rich local feature information; 2) By improving the pixelwise linear independence of the feature map, the highly condensed semantic representation will be captured by the model; 3) Our novel triple-stream merging module can significantly improve the generalization capability and efficiency of the model. Experiments show that our GSNet yields a classification accuracy of 90.1%, surpassing the state-of-the-art Swin Transformer V2 by 1.9% on the steel grain size dataset, which comprises 3599 images with 14 grain size levels. Furthermore, we intuitively believe our approach is applicable to broader applications such as object detection and semantic segmentation.

In the Vision-and-Language Navigation (VLN) task, an agent must comprehend natural language instructions and execute precise navigation in complex environments. While significant progress has been made in the VLN field, the limited availability of navigation data hinders existing methods from fully learning the commonsense relationships between rooms and landmarks, which are crucial for environmental understanding and successful navigation. To address this issue, this work proposes a Visual and Textual Commonsense-Enhanced Layout Learning Model (ViTeC). We leverage the open-world knowledge embedded in large models by utilizing ChatGPT and BLIP-2 to provide commonsense information about environments. Specifically, BLIP-2 analyzes the room type corresponding to each panoramic image, while ChatGPT infers and provides knowledge about the most common landmarks within each room type. Moreover, to compensate for the agent’s lack of commonsense at the visual level, we employ Stable Diffusion to generate commonsense-based visual images, enhancing the agent’s visual perception. To ensure the agent effectively learns commonsense about the environment, we designed a Text Commonsense Layout Learning Module and a Visual Commonsense Layout Learning Module. These modules help the agent acquire environmental commonsense from both linguistic and visual perspectives, enabling it to utilize commonsense information effectively during navigation, thereby improving its environmental understanding and reasoning capabilities. Experimental results demonstrate that ViTeC achieves strong performance on REVERIE, R2R, and SOON datasets, exhibiting good generalization ability in complex environments. This validates the effectiveness of ViTeC in enhancing the agent’s environmental understanding and navigation capabilities.

Multi-object tracking achieves the acquisition of target location information and identity information through two subtasks, detection and re-identification (ReID). The existing commonly used one-shot framework has speed advantages, but the two subtasks have different feature requirements, which leads to competitive learning in the training and thus weakens the feature quality. We propose a feature decoupling based multi-object tracking framework FDTrack for contradictory feature requirements. Through the mutual inhibition of the two subtasks, the features of the backbone network are decoupled. Then the decoupled features are self-constrained to enhance effective features. Considering the instability of the target state and the different confidence of the detections, a more reasonable association strategy is employed to maximize the matchings between detections, thus recovering low-confidence targets. FDTrack is extensively tested on the MOT17 and MOT20 benchmarks. The experimental results show that FDTrack surpasses the previous state-of-the-art (SOTA) methods and has good anti-interference and real-time performance. Moreover, our proposed modules have good portability and can be applied in other one-shot trackers to achieve performance improvement.

Abstract Point cloud data in the real world is often affected by occlusion and light reflection, leading to incompleteness of the data. Large‐region missing point clouds will cause great deviations in downstream tasks. A dual feature fusion network (DFF‐Net) is proposed to improve the accuracy of the completion of a large missing region of the point cloud. First, a dual feature encoder is designed to extract and fuse the global and local features of the input point cloud. Subsequently, a decoder is used to directly generate a point cloud of missing region that retains local details. In order to make the generated point cloud more detailed, a loss function with multiple terms is employed to emphasise the distribution density and visual quality of the generated point cloud. A large number of experiments show that the authors’ DFF‐Net is better than the previous state‐of‐the‐art methods in the aspect of point cloud completion.