Various parameter-efficient fine-tuning (PEFT) methods, including LoRA and its variants, have gained popularity for reducing computational costs. However, there is often an accuracy gap between PEFT approaches and full fine-tuning (FT), and this discrepancy has not yet been systematically explored. In this work, we introduce a method for selecting sparse sub-matrices that aims to minimize the performance gap between PEFT vs. full fine-tuning (FT) while also reducing both fine-tuning computational costs and memory costs. We explored both gradient-based and activation-based parameter selection methods to identify the most significant sub-matrices for downstream tasks, updating only these blocks during fine-tuning. In our experiments, we demonstrated that SMT consistently surpasses other PEFT baselines (e.g., LoRA and DoRA) in fine-tuning popular large language models such as LLaMA across a broad spectrum of tasks, while reducing the GPU memory footprint by 67% compared to FT. We also examine how the performance of LoRA and DoRA tends to plateau and decline as the number of trainable parameters increases, in contrast, our SMT method does not suffer from such issues.

This work focuses on the decentralized deep learning optimization framework. We propose Adjacent Leader Decentralized Gradient Descent (AL-DSGD), for improving final model performance, accelerating convergence, and reducing the communication overhead of decentralized deep learning optimizers. AL-DSGD relies on two main ideas. Firstly, to increase the influence of the strongest learners on the learning system it assigns weights to different neighbor workers according to both their performance and the degree when averaging among them, and it applies a corrective force on the workers dictated by both the currently best-performing neighbor and the neighbor with the maximal degree. Secondly, to alleviate the problem of the deterioration of the convergence speed and performance of the nodes with lower degrees, AL-DSGD relies on dynamic communication graphs, which effectively allows the workers to communicate with more nodes while keeping the degrees of the nodes low. Experiments demonstrate that AL-DSGD accelerates the convergence of the decentralized state-of-the-art techniques and improves their test performance especially in the communication constrained environments. We also theoretically prove the convergence of the proposed scheme. Finally, we release to the community a highly general and concise PyTorch-based library for distributed training of deep learning models that supports easy implementation of any distributed deep learning approach ((a)synchronous, (de)centralized).

Automotive radar has been extensively utilized in cars for many years as an essential sensor, primarily due to its robustness in extreme weather conditions, its capacity to measure Doppler information in the surrounding environment, and its cost-effectiveness. Recently, developments in radar technologies and the availability of open-source radar data sets have attracted more attention to radars and using them for perception tasks in deep learning based autonomous driving. However, annotating radar data for large-scale autonomous driving perception tasks is challenging, i.e., it is difficult for humans to label this data and often requires a semi-automatic approach that involves projecting labels from other sensors, such as cameras and LiDARs. The lack of high-quality labeled data has limited the performance of radar perception models. In this paper, we propose MVRAE, a Multi-View Radar AutoEncoder, which employs self-supervised learning to learn meaningful representations from multi-view radar data without any labels. Our approach is based on the intuition that a good representation for multi-view radar data, which includes range-angle, range-Doppler, and angle-Doppler views, should enable the reconstruction of one view solely from the representations of the other two views. Experimental results demonstrate that our proposed self-supervised method, that can be used as a pre-training step for autonomous driving task, allows the model to learn meaningful representations from unlabeled radar data and achieves enhanced label efficiency for downstream tasks, such as radar semantic segmentation. To the best of our knowledge, MVRAE is the first work that employs self-supervised learning and conducts systematic experiments with multi-view radar data.