LiDAR, as an excellent sensor, can provide positions, motion states, and other objective attribute information of objects in the 3D world. Inevitably, the inherent sparsity of point cloud and the problem of occlusion tend to cause incomplete semantic and geometry information of long-range small objects, posing challenges to 3D object detection. The multi-view models take advantage of the complementary information among bird’s eye view (BEV), range view (RV), and other views to alleviate the above issues. However, most of the existing methods coarsely learn the views’ features and neglect the learning of semantic information, which further leads to unsatisfactory detection performance. To this end, this paper proposes a Local-to-Global Semantic Learning Network (LGSLNet) for multi-view 3D object detection from point cloud. The proposed LGSLNet can effectively learn semantic information to explore the local semantics contained in various channels of RV features and to fuse them with BEV features. It has two branches with different backbones. In the BEV branch, the voxels quantized from the point cloud are extracted by sparse convolutional networks and compressed to BEV features. In the RV branch, a multi-scale backbone with semantic-aware convolution (SAC) is designed to learn the local semantic information of the RV. It allows for adaptation to the 3D location using the auxiliary network. In the fusion module, the bidirectional cross-view channel attention (Bi-CCA) is designed to compensate for the semantic information between multiple views and aggregate new RV and BEV features. Extensive experiments on the KITTI, ONCE, and nuScenes 3D object detection datasets demonstrate the superiority of our proposed method.

Motion models are vital for solving multiple object tracking (MOT), which makes instance-level position predictions of targets to handle occlusions and noisy detections. Recent methods have proposed the use of Single Object Tracking (SOT) techniques to build motion models and unify the SOT tracker with the object detector into a single network for high-efficiency MOT. However, three feature incompatibility issues in the required features of this paradigm are ignored, leading to inferior performance. First, the object detector requires class-specific features to localize objects of pre-defined classes. Contrarily, target-specific features are required in SOT to track the target of interest with an unknown category. Second, MOT relies on intra-class differences to associate targets of the same identity (ID). On the other hand, the SOT trackers focus on inter-class differences to distinguish the tracking target from the background. Third, classification confidence is used to determine the existence of targets, which is obtained with category-related features and cannot accurately reveal the existence of targets in tracking scenes. To address these issues, we propose a novel Task-specific Feature Encoding Network (TFEN) to extract task-driven features for different sub-networks. Besides, we propose a novel Quadruplet State Sampling (QSS) strategy to form the training samples of the motion model and guide the SOT trackers to capture identity-discriminative features in position predictions. Finally, we propose an Existence Aware Tracking (EAT) algorithm by estimating the existence confidence of targets and re-considering low-scored predictions to recover missed targets. Experimental results indicate that the proposed Discriminative Motion Model-based tracker (DMMTracker) can effectively address these issues when employing SOT trackers as motion models, leading to highly competitive results on MOT benchmarks.

Tracking-by-detection algorithms have considerably enhanced tracking performance with the introduction of recent convolutional neural networks (CNNs). However, most trackers directly exploit standard scalar-output CNN features, which may not capture enough feature encoding information, instead of aggregated CNN features of vector-output form. In this paper, we propose an end-to-end feature aggregation capsule framework. First, based on the existing CNN network, we aggregate a certain number of similar position-aware CNN features into a capsule to model the feature similarity. The acquired vector-level feature capsules (rather than previous scalar-level pointwise features) are utilized for differentiation learning. We then propose a group attention module to better model the entity representation between different capsule groups thus optimizes total discriminative capability. Third, to reduce the prediction interference resulted by the side effect of dimension rising within capsules, we propose a penalty attention module. Such strategy could dynamically adjust values of neurons by estimating whether they are beneficial or harmful to tracking. Experimental results on five representative benchmarks (UAVDT, DTB70, UAV123, VOT2016 and VOT2018) demonstrate the excellent tracking performance of our dual attention based capsule tracker (DACapT). Specially, it exceeds the previous top tracker by 4.6%/1.9% in precision/success evaluations on UAVDT.

In this work, the authors propose a novel self‐supervised learning method based on mutual information to learn representations from the videos without manual annotation. Different video clips sampled from the same video usually have coherence in the temporal domain. To guide the network to learn such temporal coherence, they maximise the mutual information between global features extracted from different clips sampled from the same video (Global‐MI). However, maximising the Global‐MI leads the network to seek shared content from different video clips and may make the network degenerate to focus on the background of the video. Considering the structure of the video, they further maximise the average mutual information between the global feature and local patches of multiple regions of the video clip (multi‐region Local‐MI). Their approach, which is called Max‐GL, learns the temporal coherence by jointly maximising the Global‐MI and multi‐region Local‐MI. Experiments are conducted to validate the effectiveness of the proposed Max‐GL. Experimental results show that the Max‐GL can serve as an effective pre‐training method for the task of action recognition in videos. Additional experiments for the task of action similarity labelling and dynamic scene recognition also validate the generalisation of the learned representations of the Max‐GL.

Human action recognition in videos is an important research topic in computer vision due to its wide applications. Actions naturally contain both spatial and temporal information. The key to action recognition is to model the spatial and temporal structures of actions. In this study, the authors propose an attention‐based spatial–temporal hierarchical convolutional long short‐term memory (ST‐HConvLSTM) network to model the structures of actions in the spatial and temporal domains. The ST‐HConvLSTM consists of two parts: a spatial–temporal attention module and a novel LSTM‐like architecture named hierarchical ConvLSTM (HConvLSTM). The HConvLSTM can model the spatial and temporal structures of actions. The spatial–temporal attention module can figure out which part of the video is more discriminative for action recognition and makes the HConvLSTM focus on it. In addition, a weighted fusion strategy is proposed to fuse the appearance information and motion information of the video. The proposed ST‐HConvLSTM is evaluated on UCF101, HMDB51 and Kinetics datasets. Experimental results show that the authors’ proposed ST‐HConvLSTM achieves state‐of‐the‐art performance compared with other recent LSTM‐like architectures and attention‐based methods.

The level set method based on bias correction can segment images with gentle intensity inhomogeneity effectively. However, most level set methods fail to segment severe inhomogeneous images due to the use of fixed scale clustering criterion. To deal with this problem, an adaptive multilayer level set method is proposed to segment images with severe intensity inhomogeneity. First, an improved global adaptive scale operator and a local adaptive scale operator are designed to adaptively adjust the scale of clustering kernel function according to the degree of intensity inhomogeneity. Then, an adaptive multilayer level set structure is constructed with the two designed scale operators. The number of layers and the scale of each layer in the multilayer structure are adaptively determined based on the degree of intensity inhomogeneity, which not only provides appropriate candidate scales in each pixel but also allows the model to detect global contrast information. With the dual minimisation method, image segmentation and bias correction can be achieved simultaneously. In addition, a hybrid bias field initialisation procedure is proposed to enhance the robustness of the proposed method. Experimental results demonstrate the effectiveness and robustness of the proposed method in segmenting images with intensity inhomogeneity.