Jointly estimating the optical flow and depth tasks in real-world scenes presents considerable hurdles due to some phenomena, such as occlusion, ambiguous textures, and illumination variation. The lack of guidance from the labeled data makes these challenges harder to overcome. This paper presents a novel approach to learning the regions with high uncertainties in a self-supervised manner. Our method allows the network to learn these regions by leveraging its predictions’ confidence. The uncertainties are then utilized in the estimation and refinement processes for optical flow and depth. We also introduce a novel uncertainty-guided smoothness regularization technique that leverages the uncertainty map to increase the robustness through concentrating the smoothness on regions with low confidence scores. Our results in the KITTI real-world dataset demonstrate the effectiveness of our approach for enhancing the predictions, particularly in the challenging scenes, showcasing the potential of our approach for real-world applications.

Missing data is ubiquitous phenomenon in the time series community, significantly challenging forecasting due to incomplete ground truth and sparse data. Most previous Multi-variate Time Series Forecasting with Missing Values (MTSFMV) approaches usually assume static missing patterns, neglecting the dynamic changes over time and space, leading to suboptimal forecasting results. To tackle these challenges, we propose novel STMPANets, which are capable of perceiving time-varying spatiotemporal missing patterns to refine the forecasting sequences. Specifically, we decompose the series into seasonal trend components, allowing STMPANets to highlight inherent sequence properties and adapt to missing patterns. We then propose a Multi-granularity Conditional Partial TCN (MGCPT) to regulate the imputation rate of missing values over time, modeling temporal correlation. Additionally, we design an Adaptive Dynamic GCN (ADGCN) to capture spatial dependencies by perceiving dynamic missing patterns. Extensive experiments demonstrate that STMPANets outperform state-of-the-art models.

Recurrently refining the optical flow based on a single high-resolution feature demonstrates high performance. We exploit the strength of this strategy to build a novel architecture for the joint learning of optical flow and depth. Our pro-posed architecture is improved to work in the case of training on unlabeled data, which is extremely challenging. The loss is computed for the iterations carried out over a single high-resolution feature, where the reconstruction loss fails to optimize the accuracy particularity in occluded regions. Therefore, we propose to hierarchically refine the optical flow across multiple scales while feeding the rigid flow calculated from depth and camera pose to provide more refinement. We further propose a self-supervised patch-based similarity loss to be optimized with the reconstruction loss to improve accuracy in the occluded regions. Our proposed method demonstrates efficient performance on the KITTI 2015 dataset, with more improvement in the occluded regions.

Scene flow describes the 3D motion in a scene. It can be modeled as a single task or as a composite of the auxiliary tasks of depth, camera motion, and optical flow estimation. Deep learning's emergence in recent years has broadened the horizons for new methodologies in estimating these tasks, either as separate tasks or as joint tasks to reconstruct the scene flow. The sequence of images that are either synthesized or captured by a camera is used as input for these methods, which face the challenge of dealing with various situations in images to provide the most accurate motion, such as image quality. Nowadays, images have been superseded by point clouds, which provide 3D information, thereby expediting and enhancing the estimated motion. In this paper, we dig deeply into scene flow estimation in the deep learning era. We provide a comprehensive overview of the important topics regarding both image-based and point-cloud-based methods. In addition, we cover the methodologies for each category, highlighting the network architecture. Furthermore, we provide a comparison between these methods in terms of performance and efficiency. Finally, we conclude this survey with insights and discussions on the open issues and future research directions.

Optical flow is the process of estimating motion in scenes. Each object in the scene has a homogeneous motion, i.e., moves in the same direction with the same velocity. Therefore, connecting the parts of an image globally provides an essential cue for learning accurate motion. Convolution-based methods estimate the motion features from the local regions, which miss this important cue. Recently, some methods used Transformer to model global dependencies to improve optical flow. However, Transformer suffers from excessive attention computations and still brings irrelevant parts into the region of interest. Therefore, we propose a deformable cross-attention for optical flow estimation, which provides two important advantages: connecting the parts of the image globally while deforming the attention to the objects’ shapes in the image and reducing the memory consumption. Our proposed method achieved competitive performance on Sintel and KITTI 2015 datasets in terms of accuracy and efficiency.

Scene flow estimation is the task of obtaining 3D motion from a dynamic scene. Due to the sparseness of point clouds, extracting features for a local group of points separately may result in different features that may all belong to the same object. This difference makes global correlation prone to producing an unacceptable flow. Local correlation restricts the algorithm to capturing limited movements and fails when fast movement or large deformation of an object occurs. Therefore, we propose a transformer-based scene flow method that can perform global feature modeling through a self-attention layer. Moreover, we propose a cross-attention-based flow embedding layer for global feature matching. We further propose a learnable attention-based up-sampling layer to up-sample the estimated flow to higher resolution based on a single feature scale, eliminating the need to model global dependencies at all scales. Experimental results show that our model produces competitive results on Flyingthings3D and KITTI datasets with efficient performance.

In this work, we propose a self-supervised scene flow framework for joint learning of optical flow, stereo depth, camera pose, and rigidity map and handle the occlusion during training. Specifically, we propose a feature masking method to alleviate the occlusion impact on the correlation result and reduce the outliers in both optical flow and depth map. We use the improved optical flow and depth to estimate the camera motion directly using the Perspective-n-Point method, which improves it accordingly. Furthermore, we recursively update the optical flow in both occluded and non-occluded regions with self-supervised cues learned from the rigid and optical flows. Reducing the error in the occluded regions enhances the rigidity map and improves the final optical flow accordingly. Our model achieved the state-of-the-art performance on KITTI 2015 benchmark for optical flow and produced competitive results for the depth, pose, and segmentation tasks.

Person search, which requires both pedestrian detection and person re-identification, is a challenging computer vision task applied to real-world scenarios. The challenges faced by detection and re-identification, such as occlusion, poor illumination, confusing background, are still urgent for person search. In addition, one-step methods for person search need to deal with the divergence between two tasks. In this work, we propose an end-to-end model containing the feature extractor, the region proposal network, and the multi-task learning module. In order to process divergence between detection and re-identification, we introduce switchable normalization and gradient centralization to improve the stability of the model. To solve the imbalance problem of hard examples, we introduce focal loss as a classification loss in the multi-task learning module. The experimental results on two bench-marks, i.e., CUHK-SYSU and PRW, well demonstrate that our method outperforms the state-of-the-art one-step methods.