The proposed method in the study, called DEMMNet-KD, aims to address the challenges of scene parsing in RGB-D indoor scenes. The study recognizes the importance of lightweight models for pixel-intensive tasks like scene parsing, and proposes a depth enhancement mask mapping network (DEMMNet) with multi-teacher knowledge distillation approach to achieve both reduced model size and high accuracy. DEMMNet introduces a new segmentation paradigm called class-position separate segmentation to reduce the difficulty of delineating and classifying the segmentation region. It also includes a depth spatial enhancement module based on the attention mechanism to effectively utilize depth information in a single-stream network design. To improve performance without increasing complexity, DEMMNet-KD utilizes Mix Transformer encoder (MiT) as a backbone and employs multi-teacher knowledge distillation. This allows semantic knowledge to be transferred from large teacher networks with different performances to the student network. The experiments conducted on NYUDv2 and SUN RGB-D datasets demonstrate that DEMMNet-KD achieves state-of-the-art performance with fewer additional learnable parameters and minimal computational effort. Utilizing MiT-B2 as the backbone network, the DEMMNet-KD method attained mean Intersection over Union (mIoU) scores of 54.71% and 51.03% on the two datasets, respectively, thereby achieving an optimal balance between accuracy and efficiency. Furthermore, the scalability of this method has been demonstrated across various domains and modalities. Overall, the proposed DEMMNet-KD approach provides an efficient solution for scene parsing in RGB-D indoor scenes, enhancing the capabilities of bionic binocular robots in perceiving their environments and serving human society effectively. The code is available at https://github.com/SHARKALAKALA/DEMMNet.

Scene parsing of high-resolution remote sensing images with complex backgrounds has received extensive attention in recent years. As unimodal networks are significantly affected by weather conditions, reflecting complex ground conditions fully and accurately is difficult; therefore, multimodal scene analysis is particularly important. Current multimodal scene-parsing networks often employ a dual-coding architecture to achieve high-performance segmentation. Because prompt learning allows models to understand and capture contextual information more effectively, the proposed prompt injection module (PIM) extracts relevant information from frozen normalized digital surface model (nDSM) features and integrates it into the infrared, red, and green (IRRG) branches through a modal embedding block. To extract the contextual semantic relationships between the local and global features in the image efficiently, we also design a dynamic filter block for feature enhancement. This design facilitates the mutual complementarity and guidance of information between the two modalities and optimizes fusion. The experimental results demonstrate that lightweight and effective multimodal prompt injection network (LENet) outperforms most current state-of-the-art lightweight methods on two public datasets, achieving comparable accuracy to that of traditional methods. It has only 10.81 M parameters, with 2.72 GFLOPS. Our code and results are available athttps://github.com/LYZ00918/LENet.

Compared with the progress made on human activity classification, much less success has been achieved on human interaction understanding (HIU). Apart from the latter task is much more challenging, the main causation is that recent approaches learn human interactive relations via shallow graphical representations, which are inadequate to model complicated human interactive-relations. This paper proposes a deep consistency-aware framework aiming at tackling the grouping and labelling inconsistencies in HIU. This framework consists of three components, including a backbone CNN to extract image features, a factor graph network to implicitly learn higher-order consistencies among labelling and grouping variables, and a consistency-aware reasoning module to explicitly enforcing consistencies. The last module is inspired by our key observation that the consistency-aware reasoning bias can be embedded into an energy function or a particular loss function, minimizing which delivers consistent predictions. An efficient mean-field inference algorithm is proposed, such that all modules of our network could be trained in an end-to-end fashion. Experimental results demonstrate that the two proposed consistency-learning modules complement each other, and both make considerable contributions in achieving leading performance on three benchmarks of HIU. The effectiveness of the proposed approach is further validated by experiments on detecting human-object interactions.

Human interaction recognition has wide applications including intelligent surveillance, intelligent transportation and the analysis of sports videos. In recent years, benefiting from the development of action recognition based on deep learning, the performance of human interaction recognition has been boosted. This paper tackles two vital issues in recognizing human interactions, namely target missing and inadequate feature expression. To this end, we first design a data preprocessing method using skeleton estimation and multi-object tracking, which effectively reduces the chance of missing detection. Second, we propose a two-stream network composing of an appearance branch and a pose branch. The appearance branch extracts features enhanced via part affinity maps and part confidences maps, while the pose branch trains a customized Shift-GCN to extract skeletal features from people-pairs. Appearance and pose features are then fused to generate a more powerful representation of human interactions. Extensive experiments on two existing benchmarks, UT and BIT-Interaction, as well as a new dataset crafted by us, namely Campus-Interaction (CI), demonstrate the superior performance of the proposed approach over the state-of-the-arts.

Compared with the progress made on human activity classification, much less success has been achieved on human interaction understanding (HIU). Apart from the latter task is much more challenging, the main cause is that recent approaches learn human interactive relations via shallow graphical models, which is inadequate to model complicated human interactions. In this paper, we propose a consistency-aware graph network, which combines the representative ability of graph network and the consistency-aware reasoning to facilitate HIU. Our network consists of three components, a backbone CNN to extract image features, a factor graph network to learn third-order interactive relations among participants, and a consistency-aware reasoning module to enforce labeling and grouping consistencies. Our key observation is that the consistency-aware-reasoning bias for HIU can be embedded into an energy, minimizing which delivers consistent predictions. An efficient mean-field inference algorithm is proposed, such that all modules of our network could be trained jointly in an end-to-end manner. Experimental results show that our approach achieves leading performance on three benchmarks. Code is available at https://git.io/CAGNet.