Large Language Models (LLMs) are increasingly deployed in real-world applications that demand complex reasoning.To track progress, robust benchmarks are required to evaluate their capabilities beyond superficial pattern recognition.However, current LLM reasoning benchmarks often face challenges such as insufficient interpretability, performance saturation or data contamination.To address these challenges, we introduce GAMEBOT (GAME Battle of Tactics), a gaming arena designed for rigorous and transparent assessment of LLM reasoning capabilities.GAMEBOT decomposes complex reasoning in games into predefined modular subproblems.This decomposition allows us to design a suite of Chain-of-Thought (CoT) prompts that leverage domain knowledge to guide LLMs in addressing these subproblems before action selection.Furthermore, we develop a suite of rule-based algorithms to generate ground truth for these subproblems, enabling rigorous validation of the LLMs' intermediate reasoning steps.This approach facilitates evaluation of both the quality of final actions and the accuracy of the underlying reasoning process.GAMEBOT also naturally alleviates the risk of data contamination through dynamic games and head-to-head LLM competitions.We benchmark 17 prominent LLMs across eight games, encompassing various strategic abilities and game characteristics.Our results suggest that GAMEBOT presents a significant challenge, even when LLMs are provided with detailed CoT prompts.Project page: https://visual-ai.github.io/gamebot

Structured prediction models aim at solving tasks where the output is a complex structure, rather than a single variable. Performing knowledge distillation for such problems is non- trivial due to their exponentially large output space. Previous works address this problem by developing particular distillation strategies (e.g., dynamic programming) that are both complicated and of low run-time efficiency. In this work, we propose an approach that is much simpler in its formulation, far more efficient for training than existing methods, and even performs better than our baselines. Specifically, we transfer the knowledge from a teacher model to its student by locally matching their computations on all internal structures rather than the final outputs. In this manner, we avoid time-consuming techniques like Monte Carlo Sampling for decoding output structures, permitting parallel computation and efficient training. Besides, we show that it encourages the student model to better mimic the internal behavior of the teacher model. Experiments on two structured prediction tasks demonstrate that our approach not only halves the time cost, but also outperforms previous methods on two widely adopted benchmark datasets.1 2

Recent research has reported a performance degradation in self-supervised contrastive learning for specially designed efficient networks, such as MobileNet and EfficientNet. A common practice to address this problem is to introduce a pretrained contrastive teacher model and train the lightweight networks with distillation signals generated by the teacher. However, it is time and resource consuming to pretrain a teacher model when it is not available. In this work, we aim to establish a stronger baseline for lightweight contrastive models without using a pretrained teacher model. Specifically, we show that the optimal recipe for efficient models is different from that of larger models, and using the same training settings as ResNet50, as previous research does, is inappropriate. Additionally, we observe a common issue in contrastive learning where either the positive or negative views can be noisy, and propose a smoothed version of InfoNCE loss to alleviate this problem. As a result, we successfully improve the linear evaluation results from 36.3% to 62.3% for MobileNet-V3-Large and from 42.2% to 65.8% for EfficientNet-B0 on ImageNet, closing the accuracy gap to ResNet50 with 5× fewer parameters. We hope our research will facilitate the usage of lightweight contrastive models.1

Traditional knowledge distillation adopts a two-stage training process in which a teacher model is pre-trained and then transfers the knowledge to a compact student model. To overcome the limitation, online knowledge distillation is proposed to perform one-stage distillation when the teacher is unavailable. Recent researches on online knowledge distillation mainly focus on the design of the distillation objective, including attention or gate mechanism. Instead, in this work, we focus on the design of the global architecture and propose Tree-Structured Auxiliary online knowledge distillation (TSA), which adds more parallel peers for layers close to the output hierarchically to strengthen the effect of knowledge distillation. Different branches construct different views of the inputs, which can be the source of the knowledge. The hierarchical structure implies that the knowledge transfers from general to task-specific with the growth of the layers. Extensive experiments on 3 computer vision and 4 natural language processing datasets show that our method achieves state-of-the-art performance without bells and whistles. To the best of our knowledge, we are the first to demonstrate the effectiveness of online knowledge distillation for machine translation tasks. 1 1 Code is available at https://github.com/Linwenye/Tree-Supervised.