Graph Convolutional Networks (GCNs) have achieved notable success in skeleton-based action recognition. However, research scholars face issues such as low discriminability between actions and inadequate model posture adaptability in the field of action recognition. Furthermore, existing methods tend to ignore the temporal dependencies between joint points, employing a static adjacency matrix across all frames, which restricts the spatiotemporal modeling capacity of GCNs. To address these limitations, we propose a Dual Adjacency Graph Convolutional Network (DA-GCN) training framework combined with reinforcement learning. Our framework dynamically adjusts the spatial positions of joint points in skeleton data using a reinforcement learning policy network. This approach directly improves the differentiation between different actions while minimizing the computational demands associated with feature extraction. Moreover, DA-GCN employs a dual adjacency matrix design, constructing channel-dependent and temporal-dependent matrices to capture the spatiotemporal adjacency dependencies of joint points effectively. To our knowledge, this is the first study to leverage reinforcement learning to directly increase distinctions within the original skeleton data. Experimental results indicate that our method achieves excellent performance in the Northwestern-UCLA, NTU-RGB+D, and NTU-RGB+D 120 datasets, validating its effectiveness in action recognition tasks.

Utilizing infrastructure and vehicle-side information to track and forecast the behaviors of surrounding traffic participants can significantly improve decision-making and safety in autonomous driving. However, the lack of real-world sequential datasets limits research in this area. To address this issue, we introduce V2X-Seq, the first large-scale sequential V2X dataset, which includes data frames, trajectories, vector maps, and traffic lights captured from natural scenery. V2X-Seq comprises two parts: the sequential perception dataset, which includes more than 15,000 frames captured from 95 scenarios, and the trajectory forecasting dataset, which contains about 80,000 infrastructure-view scenarios, 80,000 vehicle-view scenarios, and 50,000 cooperative-view scenarios captured from 28 intersections' areas, covering 672 hours of data. Based on V2X-Seq, we introduce three new tasks for vehicle-infrastructure cooperative (VIC) autonomous driving: VIC3D Tracking, Online-VIC Forecasting, and Offline-VIC Forecasting. We also provide benchmarks for the introduced tasks. Find data, code, and more up-to-date information at https://github.com/AIR-THU/DAIR-V2X-Seq.

Facial action unit (AU) detection is a challenging task due to the scarcity of manual annotations. Recent works on AU detection with self-supervised learning have emerged to address this problem, aiming to learn meaningful AU representations from numerous unlabeled data. However, most existing AU detection works with self-supervised learning utilize global facial features only, while AU-related properties such as locality and relevance are not fully explored. In this paper, we propose a novel self-supervised framework for AU detection with the region and relation learning. In particular, AU related attention map is utilized to guide the model to focus more on AU-specific regions to enhance the integrity of AU local features. Meanwhile, an improved Optimal Transport (OT) algorithm is introduced to exploit the correlation characteristics among AUs. In addition, Swin Transformer is exploited to model the long-distance dependencies within each AU region during feature learning. The evaluation results on BP4D and DISFA demonstrate that our proposed method is comparable or even superior to the state-of-the-art self-supervised learning methods and supervised AU detection methods.

Human action recognition is one of the challenging and active research fields due to its wide range of applications. Recently, graph convolutions for skeleton-based action recognition have attracted much attention. Generally, the adjacency matrices of the graph are fixed to the hand-crafted physical connectivity of the human joints, or learned adaptively via deep learning. The hand-crafted or learned adjacency matrices are fixed when processing each frame of an action sequence. However, the interactions of different subsets of joints may play a core role at different phases of an action. Therefore, it is reasonable to evolve the graph topology with time. In this paper, a recurrent graph convolution is proposed, in which the graph topology is evolved via a long short-term memory (LSTM) network. The proposed recurrent graph convolutional network (R-GCN) can recurrently learn the data-dependent graph topologies for different layers, different time steps and different kinds of actions. Experimental results on the NTU RGB+D and Kinetics-Skeleton datasets demonstrate the advantages of the proposed R-GCN.

Continuous gesture recognition aims at recognizing the ongoing gestures from continuous gesture sequences and is more meaningful for the scenarios, where the start and end frames of each gesture instance are generally unknown in practical applications. This paper presents an effective deep architecture for continuous gesture recognition. First, continuous gesture sequences are segmented into isolated gesture instances using the proposed temporal dilated Res3D network. A balanced squared hinge loss function is proposed to deal with the imbalance between boundaries and nonboundaries. Temporal dilation can preserve the temporal information for the dense detection of the boundaries at fine granularity, and the large temporal receptive field makes the segmentation results more reasonable and effective. Then, the recognition network is constructed based on the 3-D convolutional neural network (3DCNN), the convolutional long-short-term-memory network (ConvLSTM), and the 2-D convolutional neural network (2DCNN) for isolated gesture recognition. The “3DCNN-ConvLSTM-2DCNN” architecture is more effective to learn long-term and deep spatiotemporal features. The proposed segmentation and recognition networks obtain the Jaccard index of 0.7163 on the Chalearn LAP ConGD dataset, which is 0.106 higher than the winner of 2017 ChaLearn LAP Large-Scale Continuous Gesture Recognition Challenge.

PixelNet has achieved great success in dense prediction problems with a pure pixel-level architecture, but there is still much room for improvement. In this paper, we start from PixelNet and discuss the pixel-level architecture called hypercol-umn and its limitations in building feature representation with rich semantic information. To achieve this goal, we propose a concept in the context of neural networks called reflective field, representing the area reflected by the origin input. Furthermore, the proposed reflective field is used to solve the limitations of the hypercolumn architecture. Specifically, we give the method of calculating the size of the reflective field and analyze the effective reflective field in the calculated area. Then, we use the reflective field to build a new hypercolumn architecture, which has a more rational construction. The results on PASCAL VOC segmentation dataset with our new architecture are improved.

Colour vision deficiency (CVD) is a genetic condition that has troubled people for a long time. This study proposes an improved colour‐to‐grey method for CVD using image segmentation and a colour difference model. In this method, the colour image is first segmented using a region growing method so that each region corresponds to one colour. Next, the colour difference is computed between arbitrary segmented region pairs. Finally, the greyscale image is obtained by minimising a target function. Experimental results show that compared with state‐of‐the‐art colour‐to‐grey methods, the proposed algorithm can improve the E ‐score by about 10.99%.

Point cloud fusion is a significant process in many computer vision applications such as multi-robot maps building or multi-SLAM. When robots capture maps in different places, the maps suffer from large angle difference between respective viewpoints of the robots and spatial misalignment. These problems bring great challenges to point cloud fusion. In order to overcome such difficulty, in this study, we propose a 3D viewpoint calibration method. The method relies on the novel combination of 3D-SIFT (scale-invariant feature transform) keypoints and FPFH (fast point feature histogram) features, which are used for feature matching. Based on the feature correspondences, we compute the transformation to resolve the difference of the viewpoints. The experiments demonstrate that (1) the keypoints and features used in our method are distinctive and robust to camera viewpoint change; and (2) using viewpoint calibration can reduce the iterations of registration algorithms, and transforming different viewpoints to a close one could save computation time and improve accuracy.

Heavy noises and large amounts of holes exist in depth images captured by sensors, such as Kinect, which would severely hinder the application of depth information. In this paper, a novel depth enhancement algorithm with improved exemplar-based inpainting and joint trilateral guided filtering is proposed. The improved examplar-based inpainting method is applied to fill the holes in the depth images, in which the level set distance component is introduced in the priority evaluation function. Then a joint trilateral guided filter is adopted to denoise and smooth the inpainted results. Experimental results reveal that the proposed algorithm can achieve better enhancement results compared with the existing methods in terms of subjective and objective quality measurements.

Human gesture recognition is one of the central research fields of computer vision, and effective gesture recognition is still challenging up to now. In this paper, we present a pyramidal 3D convolutional network framework for large-scale isolated human gesture recognition. 3D convolutional networks are utilized to learn the spatiotemporal features from gesture video files. Pyramid input is proposed to preserve the multi-scale contextual information of gestures, and each pyramid segment is uniformly sampled with temporal jitter. Pyramid fusion layers are inserted into the 3D convolutional networks to fuse the features of pyramid input. This strategy makes the networks recognize human gestures from the entire video files, not just from segmented clips independently. We present the experiment results on the 2016 ChaLearn LAP Large-scale Isolated Gesture Recognition Challenge, in which we placed third.

Human activity recognition plays an important role in personal assistive robot, being able to recognize human activity and perform corresponding assistive action is a great challenges for personal assistive robot. Human body is an articulated system of rigid segments that can be divided into five parts, but many existing methods always identify actions based on the motion trajectories of whole body. In this paper, taking into account the fact that most actions can be performed by a few limbs and the other limbs should not impact on the action recognition, we proposed an activity recognition method based on limb weights. The weight of each limb is composed of consistency weight and uniqueness weight, which are learned according to the similarity degree among different sequences for each specific action. The covariance descriptor, which is the concatenation of eigenvalues extracted from covariance matrices, is adopted to represent the motion trajectory of each limb. In order to distinguish action instances from each other in the feature sequences, a simple annotation method is used. Experimental results on the Cornell activity dataset and the Lab dataset show that the proposed method not only can outperform the state-of-the-art algorithms, but also is appropriate to recognize the actions whose non-core limbs' trajectories are different from each other.

Images obtained under low‐light conditions tend to have the characteristics of low‐grey levels, high‐noise levels, and indistinguishable details. Image degradation not only affects the recognition of images, but also influences the performance of the computer vision system. The low‐light image enhancement algorithm based on the dark channel prior de‐hazing technique can enhance the contrast of images effectively and can highlight the details of images. However, the dark channel prior de‐hazing technique ignores the effects of noise, which leads to significant noise amplification after the enhancement process. In this study, a de‐hazing‐based simultaneous enhancement and noise reduction algorithm of are proposed by analysing the essence of the dark channel prior de‐hazing technique and bilateral filter. First, the authors estimate the values of the initial parameters of the hazy image model by de‐hazing technique. Then, they correct the parameters of the hazy image model alternately with the iterative joint bilateral filter. Experimental results indicate that the proposed algorithm can simultaneously enhance the low‐light images and reduce noise effectively. The proposed algorithm could also perform quite well compared with the current common image enhancement and noise reduction algorithms in terms of the subjective visual effects and objective quality assessments.

Correlation Noise Modeling (CNM) is a key factor to influence the performance of Distributed Video Coding (DVC). In most current CNM solutions, the distribution parameter is estimated based on the motion compensated residual frames and kept constant during the decoding process. A progressive correlation noise refinement method is proposed in this paper for transform domain Wyner-Ziv video coding to model the correlation noise more accurately, in which the estimated correlation noise is refined by using previously decoded bitplanes and quantization errors as bitplane decoding proceeds. The experimental results show that our proposed correlation noise refinement method could provide considerable bitrate savings and PSNR gains for transform domain Wyner-Ziv video coding system.