Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization.Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency.In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training.TLoRA introduces a data-driven initialization strategy that aligns the LoRA A matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance.After this, the A matrix is frozen, and only the B matrix is trained.Furthermore, TLoRA employs a sensitivitybased importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget.We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.

This paper considers a canonical problem in kernel regression: how good are the model performances when it is trained by the popular online first-order algorithms, compared to the offline ones, such as ridge and ridgeless regression? In this paper, we analyze the foundational single-pass Stochastic Gradient Descent (SGD) in kernel regression under source condition where the optimal predictor can even not belong to the RKHS, i.e. the model is misspecified. Specifically, we focus on the inner product kernel over the sphere and characterize the exact orders of the excess risk curves under different scales of sample sizes $n$ concerning the input dimension $d$. Surprisingly, we show that SGD achieves min-max optimal rates up to constants among all the scales, $without$ suffering the saturation, a prevalent phenomenon observed in (ridge) regression, except when the model is highly misspecified and the learning is in a final stage where $n\gg d^\gamma$ with any constant $\gamma >0$. The main reason for SGD to overcome the curse of saturation is the exponentially decaying step size schedule, a common practice in deep neural network training. As a byproduct, we provide the $first$ provable advantage of the scheme over the iterative averaging method in the common setting.

Autonomous Driving Systems (ADS) hold significant potential for enhancing travel convenience. Ensuring the reliability of ADS through efficient and comprehensive simulation testing has garnered substantial attention from researchers. In recent years, various search-based automated test scenario generation methods have been proposed to identify potential ADS defects. However, these methods still face challenges in balancing efficiency with comprehensive testing and suffer from a lack of diversity. To address these challenges, we propose AS-Fuzzer, an optimized ADS fuzzing method based on composite traffic scenario generation. AS- Fuzzer introduces a scenario slicing technique based on traffic road structures, allowing each sliced scenario to evolve independently and in parallel, enhancing interaction rates and balancing efficiency with comprehensive testing. A novel scenario generation method Co-Evolutionary Genetic Algorithms (CEGA), is applied within AS-Fuzzer to improve the diversity of generated scenarios, thereby exploring a wider range of ADS defects. Experimental results demonstrate that the proposed method improves test scenario generation efficiency by 120% compared to the state-of-the-art baseline method. Additionally, in the simulation testing of the proposed method, the interaction rate between the ADS vehicle and non-player characters is 2.79 times that of the baseline method, thereby further enhancing ADS testing efficiency. Furthermore, within the same time frame, the proposed method uncovered 19 types of ADS defects that other baseline methods did not explore, achieving higher ADS defect diversity.