Recent studies have raised significant concerns regarding the reliability of current mathematical benchmarks, highlighting key limitations such as simplistic design and potential data contamination that undermine evaluation accuracy. Consequently, developing a reliable benchmark that effectively evaluates large language models' (LLMs) genuine capabilities in mathematical reasoning remains a critical challenge. To address these concerns, we propose RV-Bench, a novel evaluation methodology for Benchmarking LLMs with Random Variables in mathematical reasoning. Specifically, we develop question-generating functions to produce random variable questions (RVQs), whose background content mirrors the original benchmark problems, but with randomized variable combinations, rendering them "unseen" to LLMs. Models must completely understand the inherent question pattern to correctly answer RVQs with diverse variable combinations. Thus, an LLMs' genuine reasoning capability is reflected through its accuracy and robustness on RV-Bench. We conducted extensive experiments on over 30 representative LLMs across more than 1,000 RVQs. Our findings reveal that LLMs exhibit a proficiency imbalance between encountered and "unseen" data distributions. Furthermore, RV-Bench reveals that proficiency generalization across similar mathematical reasoning tasks is limited, but we verified that it can still be effectively elicited through test-time scaling.

Cross-workload design space exploration (DSE) is crucial in CPU architecture design. Existing DSE methods typically employ the transfer learning technique to leverage knowledge from source workloads, aiming to minimize the requirement of target workload simulation. However, these methods struggle with overfitting, data ambiguity, and workload dissimilarity. To address these challenges, we reframe the cross-workload CPU DSE task as a few-shot meta-learning problem and further introduce MetaDSE. By leveraging model agnostic meta-learning, MetaDSE swiftly adapts to new target workloads, greatly enhancing the efficiency of cross-workload CPU DSE. Additionally, MetaDSE introduces a novel knowledge transfer method called the workload-adaptive architectural mask algorithm, which uncovers the inherent properties of the architecture. Experiments on SPEC CPU 2017 demonstrate that MetaDSE significantly reduces prediction error by 44.3% compared to the state-of-theart. MetaDSE is open-sourced and available at this anonymous GitHub.

Parameter-efficient fine-tuning (PEFT), updating only a small set of parameters either inherently in the model or additionally introduced, reduces the cost of adaptation of large vision models (e.g. Vision Transformers) and avoids overfitting to few-shot samples. However, the selection of parameters to update often follows heuristic criteria, thus lacking systematic analysis and may lead to suboptimal results. In this work, we adopt the concept of skilled parameter localization (SPL) from the NLP community, which can identify the location of task-specific parameters in a fine-tuned model automatically given any task. By applying this technique to ViTs, we observe that while the task-specific (skilled) parameters scatter in the parameter space across different tasks, the out-projection bias of attention and MLP layers are often concentrated with these skilled parameters. Inspired by this, we propose Out-projection Bias Fine-Tuning, or OBFT, a simple yet effective PEFT method that conducts few-shot adaptation solely relying on the out-projection bias of attention and MLP modules in pre-trained ViTs. We demonstrate the effectiveness and efficiency of our OBFT over 10 diverse datasets: 1) OBFT achieves superior parameter efficiency than a broad spectrum of PEFT strategies; 2) by updating only 0.01% parameters of ViTs, OBFT attains comparable performance with full fine-tuning, while significantly reducing training costs, as it does not need to maintain optimizer states for most parameters.

Few-shot classification consists of a training phase where a model is learned on a relatively large dataset and an adaptation phase where the learned model is adapted to previously-unseen tasks with limited labeled samples. In this paper, we empirically prove that the training algorithm and the adaptation algorithm can be completely disentangled, which allows algorithm analysis and design to be done individually for each phase. Our meta-analysis for each phase reveals several interesting insights that may help better understand key aspects of few-shot classification and connections with other fields such as visual representation learning and transfer learning. We hope the insights and research challenges revealed in this paper can inspire future work in related directions. Code and pre-trained models (in PyTorch) are available at https://github.com/Frankluox/CloserLookAgainFewShot.