Group activity recognition is a crucial yet challenging problem, whose core lies in fully exploring spatial-temporal interactions among individuals and generating reasonable group representations. However, previous methods either model spatial and temporal information separately, or directly aggregate individual features to form group features. To address these issues, we propose a novel group activity recognition network termed GroupFormer. It captures spatial-temporal contextual information jointly to augment the individual and group representations effectively with a clustered spatial-temporal transformer. Specifically, our GroupFormer has three appealing advantages: (1) A tailor-modified Transformer, Clustered Spatial-Temporal Transformer, is proposed to enhance the individual representation and group representation. (2) It models the spatial and temporal dependencies integrally and utilizes decoders to build the bridge between the spatial and temporal information. (3) A clustered attention mechanism is utilized to dynamically divide individuals into multiple clusters for better learning activity-aware semantic representations. Moreover, experimental results show that the proposed framework outperforms state-of-the-art methods on the Volleyball dataset and Collective Activity dataset. Code is available at https://github.com/xueyee/GroupFormer

Feature pyramid network (FPN) has dominated the task of object detection for many years with its congenital capability of semantic information interaction between different feature levels. Satisfactory performance can be easily achieved via the heavy backbones or tricky designs of anchors. However, the heavy structure brings about a huge increase of parameters and calculation. To address this challenge, this paper proposes a novel Hint Pyramid Learning(HPL) strategy to optimize a lightweight FPN-based detector such as one based on ResNet18. In HPL, the hint feature representation is learnt via salient semantic mining based on the teacher-student learning mechanism and self learning mechanism. Besides minimizing of the discrepancy of features belonging to the same feature pyramid layer, HPL also focuses on the relationships between feature pyramid layers. Unlike the popular mimicking algorithms such as hint learning and knowledge distillation which achieve comparable performance on classification tasks but failed to generalize to the detection tasks with FPN, HPL can be easily plugged into different FPN-based pipelines such as one-stage RetinaNet or two-stage Faster R-CNN and improves the performance straightforwardly. Without any meticulous designing, when the backbone of students is ResNet18 and the backbone of teachers is ResNet50, HPL effectively improves the performance by 2.1 AP for RetinaNet and 1.0 AP for Faster R-CNN on COCO benchmark.

Feature pyramid networks (FPN) are widely exploited for multi-scale feature fusion in existing advanced object detection frameworks. Numerous previous works have developed various structures for bidirectional feature fusion, all of which are shown to improve the detection performance effectively. We observe that these complicated network structures require feature pyramids to be stacked in a fixed order, which introduces longer pipelines and reduces the inference speed. Moreover, semantics from non-adjacent levels are diluted in the feature pyramid since only features at adjacent pyramid levels are merged by the local fusion operation in a sequence manner. To address these issues, we propose a novel architecture named RCNet, which consists of Reverse Feature Pyramid (RevFP) and Cross-scale Shift Network (CSN). RevFP utilizes local bidirectional feature fusion to simplify the bidirectional pyramid inference pipeline. CSN directly propagates representations to both adjacent and non-adjacent levels to enable multi-scale features more correlative. Extensive experiments on the MS COCO dataset demonstrate RCNet can consistently bring significant improvements over both one-stage and two-stage detectors with subtle extra computational overhead. In particular, RetinaNet is boosted to 40.2 AP, which is 3.7 points higher than baseline, by replacing FPN with our proposed model. On COCO test-dev, RCNet can achieve very competitive performance with a single-model single-scale 50.5 AP.