Visual Relationship Detection (VRD) has significantly advanced with Transformer-based architectures. However, we identify two fundamental drawbacks in conventional label assignment methods used for training Transformer-based VRD models, where ground-truth (GT) annotations are matched to model predictions. In conventional assignment, queries are trained to detect all relations rather than specializing in specific ones, resulting in ‘unspecialized’ queries. Also, each ground-truth (GT) annotation is assigned to only one prediction under conventional assignment, suppressing other near-correct predictions by labeling them as ‘no relation’. To address these issues, we introduce a novel method called Groupwise Query Spe ci a lization and Q uality-Aware Multi-Assignment (SpeaQ). Groupwise Query Specialization clusters queries and relations into exclusive groups, promoting specialization by assigning a set of relations only to a corresponding query group. Quality-Aware Multi-Assignment enhances training signals by allowing multiple predictions closely matching the GT to be positively assigned. Additionally, we introduce dynamic query reallocation, which transfers queries from high- to low-performing groups for balanced training. Experimental results demonstrate that SpeaQ+, combining SpeaQ with dynamic query reallocation, consistently improves performance across seven baseline models on five benchmarks without additional inference cost.

Deep neural networks easily suffer from weak generalization caused by overfitting on biased data. One popular remedy to alleviate this issue is sample reweighting methods that adaptively adjust the importance of biased samples. Separate from the effort to reduce bias, recent works show that the generalization power can be improved by auxiliary tasks. Inspired by the two lines of works, we extend the sample reweighting methods to auxiliary tasks. In this paper, we propose a novel auxiliary learning framework that improves the primary task by adaptively adjusting the weights of samples from multiple tasks rather than samples from a single task using a weighting function. The weighting function is optimized by meta-learning along the gradient of the loss for meta-data, which is a small unbiased validation data. We also present a task-activation score that indicates the correlation between the learning tendency of the training samples and meta-data samples. This score is utilized as a regularizer for meta-learning objective. Our framework can obtain powerful representations for the primary task on biased data by automatically identifying effective combinations of tasks. Our experiments demonstrate that our proposed method consistently outperforms all baselines and state-of-the-art methods on both corrupted labels and class imbalance settings.

Many self-supervised representation learning methods have achieved high performance in image classification tasks. However, these methods have limited performance on localization tasks such as object detection or semantic segmentation. Most self-supervised representation learning methods are optimized with only one global representation, which does not pay much attention to the spatial information in an image. We propose a simple and effective method that uses the positional relationships between the entities in an image by shuffling the convolution kernels. Our method extends current self-supervised learning and calculates the pixel-wise (dis) similarities between the output of the standard convolution kernels and that of the randomly shuffled convolution kernels. Our proposed method achieves higher performance on object detection, instance segmentation, and semantic segmentation when attached to recent self-supervised learning methods.