Group activity recognition (GAR) plays a crucial role in computer vision, enabling the exploration and comprehension of human behavior patterns. Existing methods mainly focus on dyad-level interactions within a group, but sociological studies have highlighted the importance of individual features, subgroup-level interactions, and overall group structure for understanding group activities. Therefore, we propose a new framework, the progressive group activity reasoning model (PGAR), which models these four aspects for GAR. Initially, we construct a person-person graph (PPG) using individual features to capture dyadic interactions. Subsequently, the PPG is fed into a novel ingredient graph model (Ingredient-GNN) for capturing subgroup-level interactions. Finally, we fuse the dyad-level and subgroup-level interactions with global information of group structure, obtained through an F-Formation modeling module, to form comprehensive representations for GAR. The F-Formation modeling module decouples the group structure into position, orientation, and skeleton graphs, and subsequently performs attribute recoupling at the individual level using the designed Tri-Coupling Transformer to form a global representation of the group structure. Extensive experiments on four public datasets demonstrate that our final model effectively integrates multi-level representations for group activity understanding, with our F-Formation modeling module outperforming comparable methods that rely solely on non-visual data.

As multimedia advances, there is a growing need for machines to adeptly understand diverse social relations . Traditional methods for recognizing these relations, which are limited to a fixed number of classes, are ill-equipped for continual learning as new social interactions emerge. To address this prob-lem, we propose a pioneering Graph-based Interactive Knowledge Distillation (GI-KD) method for social relation continual learning. GI-KD, embedded in a class incremental learning structure, creates a balanced system where previously learned social relations and new knowledge are positioned at either end of the scale. The old and new knowledge is learned dynamically by adjusting the tilt of the balance. To achieve this balance, we propose a novel Libra loss function, which evaluate the relative contribution of old and new information and thus guides the adaptive fine-tuning of the model. We evaluate the GI-KD on three public social relation recognition (SRR) datasets, under different data distribution strategies. Our method shows a remarkable average 3.6% increase in incremental accuracy over current CIL techniques, effectively reducing catastrophic forgetting. Furthermore, GI-KD improves mAP and Acc by 4.6%, 5.4%, and 4.5%, respectively, compared to current CIL techniques, highlighting its strength in both continual learning and SRR.

Recognizing social relations from images is crucial for improving machine perception of social interactions. Current studies mainly focus on exploring single-type relation reasoning frameworks, such as the relation between father, mother and son in a family. However, real-world scenarios often involve complex hybrid relations, such as friendships and professional relations, which pose a challenge for current methods due to the difficulty of establishing robust logical connections between these relations. In fact, in this hybrid social relation recognition setting, the interactions extend beyond dyadic to multipartite structures. To effectively explore these multipartite interactions, we propose a novel Hypergraph Mamba (HGM) framework. Specifically, we construct two hypergraphs, i.e., Person-Person Hypergraphs (PPH) and Person-Object Hypergraphs (POH), to model these high-order multipartite interactions. The HGM module performs social relation reasoning within these hypergraph structures, which includes a Vertex Selection Algorithm to mitigate inference confusion by filtering out confounders, and a Vertex Interaction Operator to find optimal global vertex neighborhoods by capturing long-range vertex dependencies. In addition, a Multilevel Transformer is proposed to adaptively align the PPH and POH inferred knowledge and visual signals to facilitate information fusion. We validate the effectiveness of our proposed HGM model on several public datasets and perform extensive ablation studies to elucidate the reasons contributing to its superior performance. Experimental results indicate that our HGM model achieves superior accuracy in predicting social relations compared to the state-of-the-art methods. Codes and datasets are available at: https://github.com/tw-repository/HGM-SRR.

Identifying relationships between people from images is essential for studying social activities and interactions, and this has significant potential to further the understanding of human social behaviors. Existing image-based research mainly explores social relationships at the dyadic level, i.e., recognizing pairwise relationships based on visual features of persons, objects, and scenes and their logical constraints. Notably, social relational structures are hierarchically nested, i.e., individuals and dyads are nested within group structures, as indicated in the social relations model (SRM) of social psychology. However, existing computer vision-based studies fail to consider hierarchical nested structures, thus overlooking some of the most important interactions, which leads to poor relation reasoning. To improve the performance of reasoning neural networks, we propose a novel SRM framework for progressive graph reasoning (PGR) to explore social interactions. Specifically, we construct individual–dyad and dyad–group graphs to progressively explore the impact of individuals and groups on recognition of dyadic relationships. A transformer is utilized to fuse visual features and graph reasoning knowledge into a comprehensive representation of social relationships. We demonstrate the effectiveness of the proposed model based on PGR using several public datasets and perform extensive ablation studies to explore the reasons behind its superior performance. Experimental results demonstrate that our proposed model successfully predicts social relationships with higher accuracy than state-of-the-art methods. Codes and datasets are available at:https://github.com/tw-repository/PGRSRR.