Optical flow is the process of estimating motion in scenes. Each object in the scene has a homogeneous motion, i.e., moves in the same direction with the same velocity. Therefore, connecting the parts of an image globally provides an essential cue for learning accurate motion. Convolution-based methods estimate the motion features from the local regions, which miss this important cue. Recently, some methods used Transformer to model global dependencies to improve optical flow. However, Transformer suffers from excessive attention computations and still brings irrelevant parts into the region of interest. Therefore, we propose a deformable cross-attention for optical flow estimation, which provides two important advantages: connecting the parts of the image globally while deforming the attention to the objects’ shapes in the image and reducing the memory consumption. Our proposed method achieved competitive performance on Sintel and KITTI 2015 datasets in terms of accuracy and efficiency.

Person search is a challenging retrieval problem which aims at matching pedestrians with the same identity over non-overlapping camera views. In this paper, we adopt Swin Transformer as the backbone network to extract discriminative features. We propose a symmetric online instance matching loss which transfers the symmetric idea from KL divergence to the online instance matching loss. The purpose is to strengthen the robustness of the person search model under the condition of limited training identities. We compared with the state-of-the-arts on two mainstream benchmarks: CUHK-SYSU and PRW datasets. Experimental results demonstrate the effectiveness of our method. Especially, we achieve better performance on the PRW dataset with an improvement of 6.5% and 3.5% at the mAP and top-1 accuracy, respectively.

In this work, we propose a self-supervised scene flow framework for joint learning of optical flow, stereo depth, camera pose, and rigidity map and handle the occlusion during training. Specifically, we propose a feature masking method to alleviate the occlusion impact on the correlation result and reduce the outliers in both optical flow and depth map. We use the improved optical flow and depth to estimate the camera motion directly using the Perspective-n-Point method, which improves it accordingly. Furthermore, we recursively update the optical flow in both occluded and non-occluded regions with self-supervised cues learned from the rigid and optical flows. Reducing the error in the occluded regions enhances the rigidity map and improves the final optical flow accordingly. Our model achieved the state-of-the-art performance on KITTI 2015 benchmark for optical flow and produced competitive results for the depth, pose, and segmentation tasks.

Change detection (CD) is an important means to monitor environmental changes on Earth. Recently, many methods based on deep learning have been proposed for CD tasks. However, existing CD methods still have difficulties in the detection of changing target edges and small changing targets. In this letter, we propose a novel CD method named FCDNet based on full-scale skip connections (FSC) and coordinate attention (CA). FCDNet makes full use of shallow information and high-level semantics through FSC between different levels of encoder features and decoder features, which can effectively alleviate the missed detection of small targets in the CD task. In addition, we design a multi-receptive field position enhancement module (MRPEM) based on CA. MRPEM enhances the local relationship of features through convolution operations of different kernel sizes, and establishes long-distance dependency of features with the application of CA, thus facilitate accurate detection of the edges of changing targets. We also introduce Depthwise Over-parameterized Convolutional Layer (DOConv) in our network architecture, which can improve model performance without increasing computational complexity during inference. The experimental results show that our method is comparable to state-of-the-art (SOTA) methods on the Season-Varying Change Detection (SVCD) dataset.

Person search, which requires both pedestrian detection and person re-identification, is a challenging computer vision task applied to real-world scenarios. The challenges faced by detection and re-identification, such as occlusion, poor illumination, confusing background, are still urgent for person search. In addition, one-step methods for person search need to deal with the divergence between two tasks. In this work, we propose an end-to-end model containing the feature extractor, the region proposal network, and the multi-task learning module. In order to process divergence between detection and re-identification, we introduce switchable normalization and gradient centralization to improve the stability of the model. To solve the imbalance problem of hard examples, we introduce focal loss as a classification loss in the multi-task learning module. The experimental results on two bench-marks, i.e., CUHK-SYSU and PRW, well demonstrate that our method outperforms the state-of-the-art one-step methods.

Many deep learning applications benefit from multi-task learning with several related objectives. In autonomous driving scenarios, being able to accurately infer motion and spatial information is essential for scene understanding. In this paper, we combine an adaptive feature refinement module and a unified framework for joint learning of optical flow, depth and camera pose estimation in an unsupervised manner. The feature refinement module is embedded into motion estimation and depth prediction sub-networks, which can exploit more channel-wise relationships and contextual information for feature learning. Given a monocular video, our network firstly estimates depth and camera motion, and calculates rigid optical flow. Then, we design an auxiliary flow network for inferring non-rigid flow fields. In addition, a forward-backward consistency check is adopted for occlusion reasoning. Extensive experiments on KITTI dataset demonstrate that the proposed method achieves potential results comparing to recent deep learning networks.

Recently, optical flow estimation benefits greatly from deep learning based techniques. Most approaches use encoder-decoder architecture (U-Net) or spatial pyramid network (SPN) to learn optical flow. Both U-Net and SPN can extract multi-scale features and can predict optical flow directly. However, existing networks ignore to exploit the global information among channel features and inter-spatial relationship of features. In this paper, we propose a dual self-attention pyramid network, which adaptively integrates local features with their global dependencies and focuses on important features and suppresses unimportant features. Specifically, we introduce two types of attention modules into SPN, which emphasizes meaningful features along channel and spatial axes. The channel attention can adaptively re-weight channel-wise features by considering interdependencies among channels. Moreover, the spatial attention can utilize global contextual information to emphasize or suppress features in different spatial locations. In addition, two attention modules are embedded into each pyramidal level, which can refine features at different scale. We evaluate our method on MPI-Sintel and KITTI. The experimental results show that using the dual self-attention module can improve the representation power of network and further increase the accuracy of optical flow estimation.

Supervised depth prediction and optical flow estimation have achieved promising performance due to the advanced deep network architectures. Since the ground truths are difficult to be collected, many recent works try to learn the depth and flow in an unsupervised manner. However, existing methods only use features from convolutional layers or a simple aggregation of multi-level features to predict the depth and flow maps, which is insufficient to exploit context information. In this paper, we attempt to exploit object contextual information and investigate the effect of the object context for joint learning of depth and optical flow. Specifically, we present a novel combination of object context and the framework of joint learning depth and optical flow. Our proposed network can exploit and integrate the object context for both tasks by aggregating the context according to pair-wise similarities. Furthermore, we adopt the existing spatial pyramid network (SPN) to estimate the depth and flow in a coarse-to-fine strategy effectively. Given temporally adjacent stereo pairs, our network can be trained end-to-end in an unsupervised manner and can predict the depth and flow maps simultaneously. We conduct experiments on two publicly available datasets, KITTI2012 and KITTI2015. Our proposed approach yields comparable performance on both depth and flow tasks, compared to the recent deep learning-based approaches. Experimental results demonstrate that exploiting object contextual information is useful and beneficial for depth and optical flow estimation.

Variational models for optical flow estimation usually define an energy function that contains prior assumptions to explore rudimentary statistics of images. However, such methods cannot learn motion knowledge from the pre-prepared data and have many parameters that need to be set manually. Nowadays, convolutional neural networks (CNNs) have been used in optical flow estimation successfully, which can learn weights from the training dataset and can predict optical flow end-to-end. In this paper, we propose an attention guided network for learning optical flow, named AD-Net, which contains several attention units for modelling the relativities between the channels. Further, we introduce dilated convolution into supervised network for reducing the loss of motion details. In addition, some prior auxiliary constraints are embedded in the supervised network as auxiliary loss terms. Our proposed approach is tested on MPI-Sintel and KITTI2012 datasets and can preserve motion edges and details effectively.

Recently, learning to estimate optical flow via deep convolutional networks is attracting significant attention. In this paper, we introduce a spatial-channel attention module into optical flow estimation, which infers attention maps along two separated dimensions, channel and spatial, and then integrates these separated attention maps into a fusion attention map for feature refinement. We embed this module into spatial pyramid network, which can adaptively learn the channel and spatial attention maps at each level for modifying the different scaled features and can further improve the accuracy of optical flow estimation. Our network is trained on FlyingChairs and FlyingThings3D datasets with a supervised manner, and is further tested on MPI-Sintel benchmark. The experimental results show that using the spatial-channel attention unit is beneficial for dense flow estimation and our approach is comparable with the state-of-the-art methods.