Under-display Camera is an emerging technology for full-screen display with a camera under the display. However, the current implementation of UDC causes serious image degradation. Incident light required for camera imaging undergoes attenuation and diffraction when passing through the display. Current UDC image restoration methods predominantly utilize convolutional networks, whereas transformer-based methods with superior performance are lacking. This paper proposes a Segmentation-Guided Sparse Transformer method (SGSFormer) for restoring images from UDC degraded images. Specifically, we utilize sparse self-attention to filter out redundant information and noise, directing the model’s attention to focus on the features more relevant to the degraded regions in need of reconstruction. Moreover, we integrate an instance segmentation map as prior information to guide sparse self-attention in filtering and focusing on the correct regions. Extensive experiments exhibit the superior performance of our model over the state-of-the-art methods.

Controllable character image animation has a wide range of applications. Although existing studies have consistently improved performance, challenges persist in the field of character image animation, particularly concerning stability in complex backgrounds and tasks involving multiple characters. To address these challenges, we propose a novel multi-condition guided framework for character image animation, employing several well-designed input modules to enhance the implicit decoupling capability of the model. First, the optical flow guider calculates the background optical flow map as guidance information, which enables the model to implicitly learn to decouple the background motion into background constants and background momentum during training, and generate a stable background by setting zero background momentum during inference. Second, the depth order guider calculates the order map of the characters, which transforms the depth information into the positional information of multiple characters. This facilitates the implicit learning of decoupling different characters, especially in accurately separating the occluded body parts of multiple characters. Third, the reference pose map is input to enhance the ability to decouple character texture and pose information in the reference image. Furthermore, to fill the gap of fair evaluation of multi-character image animation, we propose a new benchmark comprising about 4,000 frames. Extensive qualitative and quantitative evaluations demonstrate that our method excels in generating high-quality character animations, especially in scenarios of complex backgrounds and multiple characters.

Abstract In occlusion and interaction scenarios, human action recognition (HAR) accuracy is low. To address this issue, this paper proposes a novel multi‐modal fusion framework for HAR. In this framework, a module called improved attention long short‐term memory (IAL) is proposed, which combines the improved SE‐ResNet50 (ISE‐ResNet50) with long short‐term memory (LSTM). IAL can extract the video sequence features and the skeleton sequence features of human behaviour. To improve the performance of HAR at a high semantic level, the obtained multi‐modal sequence features are fed into a couple hidden Markov model (CHMM), and a multi‐modal IAL+CHMM method called IALC is developed based on a probability graph model. To test the performance of the proposed method, experiments are conducted on the HMDB51, UCF101, Kinetics 400k, and ActivityNet datasets, and the obtained recognition accuracy are 86.40%, 97.78%, 81.12%, and 69.36% on the four datasets, respectively. The experimental results show that when the environment is complex, the proposed multi‐modal fusion method for HAR based on the IALC can achieve more accurate target recognition results.

Semantic segmentation of airborne laser scanning (ALS) point clouds using deep learning is a hot research in remote sensing and photogrammetry. A current trend is to aggregate contextual features from different scales for boosting network generalization and diversity discrimination capabilities. One main challenge is how to achieve effective fusion with multiscale information. In this letter, we propose a muti-level context feature fusion network (MCFN) for semantic segmentation of ALS point cloud based on an encoder-decoder structure. More specifically, we design the squeeze-expansion shared MLP module (SE-MLP) following kernel point convolution (KPConv) in the encoding stage, which can extend the receptive field of KPConv. To aggregate low-level features and high-level representations, we establish channel self-attention between skip connections. In the decoding stage, we develop a cross-layer attention fusion module (CAF) to generate additional discriminative channel features by fusing multi-scale features at different upsampling layers. Experiments on the ISPRS and LASDU datasets demonstrate the superiority of the proposed method. Code: https://github.com/SC-shendazt/MCFN.

Kernel point convolution (KPConv) can effectively represent the point features of point cloud data. However, KPConv-based methods just consider the local information of each point, which is very difficult to characterize the intrinsic properties of ALS point clouds for complex laser scanning conditions. Therefore, we rethink KPConv and propose a recurrent residual dual attention network (RRDAN) based on the encoder-decoder structure for the semantic segmentation of ALS point cloud data. In the encoder stage, we design an attention kernel point convolution (AKPConv) block by using a scaling factor of batch normalization to highlight the significant channel information. Then, we use the AKPConv block to develop a recurrent residual kernel attention (RRKA) module to iteratively aggregate the local neighborhood features. In the decoder stage, we design a global and local channel attention (GLCA) module with global connection and local 1D convolution to interact the global and local information after fusing the upsampled high-level representations and the skip-connected low-level features. In addition, to reduce the influence of the long-tail distribution of reflection intensity, we apply gamma transformation to correct the data as normal distribution. The proposed RRDAN can achieve diversified feature aggregation to implement the refined semantic segmentation of ALS point clouds. We evaluate our method on two ALS datasets (i.e., ISPRS and DCF2019) to demonstrate its performance compared to a few advanced methods. Code: https://github.com/SC-shendazt/RRDAN.