Transformer-based Large Language Models (LLMs) have demonstrated powerful in-context learning capabilities. However, their predictions can be disrupted by factually correct context, a phenomenon known as context hijacking, revealing a significant robustness issue. To understand this phenomenon theoretically, we explore an in-context linear classification problem based on recent advances in linear transformers. In our setup, context tokens are designed as factually correct query-answer pairs, where the queries are similar to the final query but have opposite labels. Then, we develop a general theoretical analysis on the robustness of the linear transformers, which is formulated as a function of the model depth, training context lengths, and number of hijacking context tokens. A key finding is that a well-trained deeper transformer can achieve higher robustness, which aligns with empirical observations. We show that this improvement arises because deeper layers enable more fine-grained optimization steps, effectively mitigating interference from context hijacking. This is also well supported by our numerical and real-world experiments. Our findings provide theoretical insights into the benefits of deeper architectures and contribute to enhancing the understanding of transformer architectures.

Underwater object detection tasks face significant challenges due to the complexity of aquatic environments and underwater conditions, coupled with the limited computational resources of current underwater equipment. The characteristics of online knowledge distillation, including its ability to compress models and enhance the performance and generalization capability of lightweight models, make it a highly suitable approach for underwater object detection tasks. Despite this, the key parameter setting of online knowledge distillation is still challenging in underwater image object detection, because of the difficulties in capturing the key information that differentiates different underwater objects. Therefore, we propose an online decoupled distillation framework based on prototype contrastive learning (PCD). The core idea of PCD is to facilitate knowledge transfer among a group of networks through prototype contrastive learning and to design decoupled distillation to enable the student model to learn more comprehensive and fine-grained knowledge. In the PCD model, we employ a contrastive loss to align the distributions of the student model and the teacher model in the feature space, enhancing its semantic structural properties. This enables the student model to achieve better feature representation. Innovatively, we use the average feature vector as the prototype to execute prototype contrastive learning to ensure model stability, thereby enhancing the detection capability of lightweight models in complex environments. On detectors, we designed cross-knowledge transfer and decoupled distillation loss to make the distillation of the logit output more comprehensive and effective. Our extensive experimental results demonstrate significant improvements in underwater object detection tasks and applicability to various types of dense detectors. Our PCD can improve the average precision of ResNet-based GFL detectors by 3.0–5.3, proving the effectiveness of our PCD model.

We study the capabilities of the transformer architecture with varying depth. Specifically, we designed a novel set of sequence learning tasks to systematically evaluate and comprehend how the depth of transformer affects its ability to perform memorization, reasoning, generalization, and contextual generalization. We show a transformer with only one attention layer can excel in memorization but falls short in other tasks. Then, we show that exhibiting reasoning and generalization ability requires the transformer to have at least two attention layers, while context generalization ability may necessitate three attention layers. Additionally, we identify a class of simple operations that a single attention layer can execute, and show that the complex tasks can be approached as the combinations of these simple operations and thus can be resolved by stacking multiple attention layers. This sheds light on studying more practical and complex tasks beyond our design. Numerical experiments corroborate our theoretical findings.

Despite the remarkable success of transformer-based models in various real-world tasks, their underlying mechanisms remain poorly understood. Recent studies have suggested that transformers can implement gradient descent as an in-context learner for linear regression problems and have developed various theoretical analyses accordingly. However, these works mostly focus on the expressive power of transformers by designing specific parameter constructions, lacking a comprehensive understanding of their inherent working mechanisms post-training. In this study, we consider a sparse linear regression problem and investigate how a trained multi-head transformer performs in-context learning. We experimentally discover that the utilization of multi-heads exhibits different patterns across layers: multiple heads are utilized and essential in the first layer, while usually only a single head is sufficient for subsequent layers. We provide a theoretical explanation for this observation: the first layer preprocesses the context data, and the following layers execute simple optimization steps based on the preprocessed context. Moreover, we demonstrate that such a preprocess-then-optimize algorithm can significantly outperform naive gradient descent and ridge regression algorithms. Further experimental results support our explanations. Our findings offer insights into the benefits of multi-head attention and contribute to understanding the more intricate mechanisms hidden within trained transformers.