Multimodal knowledge graphs are typically represented as heterogeneous graphs, serving as crucial structures for depicting data that integrates various modalities, such as text, images, and audio, within multifaceted relationships. Established knowledge representation methods have aimed to enhance the expressive capabilities of knowledge triplets in these graphs by increasing the interaction between entity and relationship embeddings. However, these methods often fall short in encapsulating the multilayered semantics of multimodal data, especially considering the complex association attributes an entity may possess under specific relations. Graph neural networks (GNNs) provide a mechanism to weight neighboring nodes of entities based on structural and multimodal information. Yet, their approach lacks precision when dealing with the intricate interactions between an entity and its neighboring nodes, especially when these interactions involve diverse data types. Addressing this shortcoming, we introduce the self-attention graph neural network model (SAGNN). This model implements self-attention mechanisms for composite entities within multimodal KGs, allowing it to aggregate information from neighboring nodes effectively. By doing so, it dynamically updates entity representations based on the varying contribution levels of neighbors, considering both textual and nontextual data. The encoder of the SAGNN model is structured with multiple graph attention mechanism layers, designed to handle the multilocal features of composite entities in a multimodal context. Consequently, it adapts to learn composite entity embeddings that effectively integrate information from various data types. The decoder, on the other hand, is geared towards capturing the graph features during the decoding of triplets, considering the structure aspect of the knowledge graph. In tasks related to knowledge graph completion, particularly in multimodal settings, empirical results reveal significant improvements using the SAGNN model across all datasets.

When dealing with crowds, occlusions, and truncations in complex scenes, existing solutions for multi-person pose estimation remain challenging. This is because all examples are randomly organized and equally treated by previous methods during training, which ignores that examples vary significantly in their difficulty levels. Once trained, hard examples are underutilized due to the high proportion of simple training examples, resulting in poor model robustness for complex scenes. To tackle this, we propose a novel training strategy termedDMH-CLfor complex pose estimation, brought from curriculum learning (CL) which mainly addresses easy examples in the early training stage and hard ones in the later stage. Different from typical CL methods, we define easy/hard examples via mining both the dataset-specific statistical difficulty and the multi-model evaluated difficulty. After that, we adopt an annealing arrangement strategy to construct learning courses from easy to hard. Furthermore, we introduce a model learning feedback indicator, i.e.,Dynamic Model Hardness (DMH)to conduct course scheduling, and to explicitly explore hard poses and utilize the knowledge learned from easy poses to better handle complex scenes as well. Our DMH-CL is model-agnostic and can be easily applied to various pose estimators including single-stage models and two-stage models, and achieves significant improvements on two challenging benchmarks especially for complex scenes. Notably, it achieves substantial performance gains of 2.6% and 4.6% for hard poses compared to the strong single-stage model PETR on CrowdPose and COCO datasets, respectively. Source codes and models are publicly available online.

Multiple human parsing (MHP) is typically treated as two sub-tasks, i.e., instance separation and body part segmentation. Existing methods usually tackle the sub-tasks by adopting a two-stage strategy, which regards MHP as an ROI-based (i.e., detect-then-segment) or grouping-based (i.e., segment-then-grouping) paradigm. However, the strong dependence between the two sub-tasks limits the potential of an MHP method, since it often requires qualified prior predictions. Besides, isolated models responsible for the two sub-tasks bring a significant computational burden. Unlike existing methods, we regard MHP as a hierarchical set prediction problem and handle two sub-tasks using several landmarks of body parts. Motivated by this, we propose a novel multiple human parser with representative sets, termed ReSParser. In ReSParser, several landmarks of body parts are hierarchically estimated, resulting in coarse-to-fine representative sets. After that, each representative set is adaptively responsible for segmenting pixels into semantically consistent regions belonging to the corresponding person. In such a manner, the ReSParser simultaneously addresses two sub-tasks in a fully convolutional fashion, thus eliminating the dependence between two sub-tasks and significantly alleviating computational complexity. Extensive experiments on two challenging benchmarks demonstrate that our proposed ReSParser is an efficient framework with a superior parsing performance, which significantly outperforms that of other ROI-free yet grouping-free methods. Besides, it achieves competitive results to that of the best two-stage methods such as RP-RCNN, but requires a much lower inference time, showing a good precision-speed trade-off. We hope the ReSParser serves as a new baseline for multiple human parsing research in the future.

Building multi-person pose estimation (MPPE) models that can handle complex foreground and uncommon scenes is an important challenge in computer vision. Aside from designing novel models, strengthening training data is a promising direction but remains largely unexploited for the MPPE task. In this article, we systematically identify the key deficiencies of existing pose datasets that prevent the power of well-designed models from being fully exploited and propose the corresponding solutions. Specifically, we find that the traditional data augmentation techniques are inadequate in addressing the two key deficiencies, imbalanced instance complexity (IC) (evaluated by our new metric IC) and insufficient realistic scenes. To overcome these deficiencies, we propose a model-agnostic full-view data generation (Full-DG) method to enrich the training data from the perspectives of both poses and scenes. By hallucinating images with more balanced pose complexity and richer real-world scenes, Full-DG can help improve pose estimators' robustness and generalizability. In addition, we introduce a plug-and-play adaptive category-aware loss (AC-loss) to alleviate the severe pixel-level imbalance between keypoints and backgrounds (i.e., around 1:600). Full-DG together with AC-loss can be readily applied to both the bottom-up and top-down models to improve their accuracy. Notably, plugging into the representative estimators HigherHRNet and HRNet, our method achieves substantial performance gains of 1.0%-2.9% AP on the COCO benchmark, and 1.0%-5.1% AP on the CrowdPose benchmark.

Skeleton-based action recognition aims to project skeleton sequences to action categories, where skeleton sequences are derived from multiple forms of pre-detected points. Compared with earlier methods that focus on exploring single-form skeletons via Graph Convolutional Networks (GCNs), existing methods tend to improve GCNs by leveraging multi-form skeletons due to their complementary cues. However, these methods (either adapting structure of GCNs or model ensemble) require the co-existence of all skeleton forms during both training and inference stages, while a typical situation in real life is the existence of only partial forms for inference. To tackle this, we present Adaptive Cross-Form Learning (ACFL), which empowers well-designed GCNs to generate complementary representation from single-form skeletons without changing model capacity. Specifically, each GCN model in ACFL not only learns action representation from the single-form skeletons, but also adaptively mimics useful representations derived from other forms of skeletons. In this way, each GCN can learn how to strengthen what has been learned, thus exploiting model potential and facilitating action recognition as well. Extensive experiments conducted on three challenging benchmarks, i.e., NTU-RGB+D 120, NTU-RGB+D 60 and UAV-Human, demonstrate the effectiveness and generalizability of our method. Specifically, the ACFL significantly improves various GCN models (i.e., CTR-GCN, MS-G3D, and Shift-GCN), achieving a new record for skeleton-based action recognition.

Despite of the recent great progress on multi-person pose estimation, existing solutions still remain challenging under the condition of "crowded scenes'', where RGB images capture complex real-world scenes with highly-overlapped people, severe occlusions and diverse postures. In this work, we focus on two main problems: 1) how to design an effective pipeline for crowded scenes pose estimation; and 2) how to equip this pipeline with the ability of relation modeling for interference resolving. To tackle these problems, we propose a new pipeline named Relation based Skeleton Graph Network (RSGNet). Unlike existing works that directly predict joints-of-target by labeling joints-of-interference as false positive, we first encourage all joints to be predicted. And then, a Target-aware Relation Parser (TRP) is designed to model the relation over all predicted joints, resulting in a target-aware encoding. This new pipeline will largely relieve the confusion of the joints estimation model when seeing identical joints with totally distinct labels (e.g., the identical hand exists in two bounding boxes). Furthermore, we introduce a Skeleton Graph Machine (SGM) to model the skeleton-based commonsense knowledge, aiming to estimate the target pose with the constraint of human body structure. Such skeleton-based constraint can help to deal with the challenges in crowded scenes from a reasoning perspective. Solid experiments on pose estimation benchmarks demonstrate that our method outperforms existing state-of-the-art methods.

Crowded scenes human pose estimation remains challenging, which requires joint comprehension of multi-persons and their keypoints in a highly complex scenario. The top-down mechanism, which is a detect-then-estimate pipeline, has become the mainstream solution for general pose estimation and obtained impressive progress. However, simply applying this mechanism to crowded scenes pose estimation results in unsatisfactory performance due to several issues, in particular involving missing keypoints in crowds and ambiguously labeling during training. To tackle above two issues, we introduce a novel method named Semantic-aware Transfer with Instance-adaptive Parsing (STIP). Specifically, our STIP first enhances the discriminative power of pixel-level representations with a semantic-aware mechanism, where it smartly decides which pixels to enhance and what semantic embeddings to add. In this way, the missing keypoints detection can be alleviated.Secondly, instead of adopting a standard regressor with fixed parameters, we propose a new instance-adaptive parsing method, where it dynamically generates instance-specific parameters for reducing adverse effects caused by ambiguously labeling. Notably, STIP is designed in a plugin fashion and it can be integrated into any top-down models, such as HRNet. Extensive experiments on two challenging benchmarks, i.e., CrowdPose and MS-COCO, demonstrate the superiority and generalizability of our approach.