Low-rank adaptation (LoRA) efficiently adapts LLMs to downstream tasks by decomposing LLMs' weight update into trainable low-rank matrices for fine-tuning.However, the random low-rank matrices may introduce massive taskirrelevant information, while their recomposed form suffers from limited representation spaces under low-rank operations.Such dense and choked adaptation in LoRA impairs the adaptation performance of LLMs on downstream tasks.To address these challenges, this paper proposes OHoRA, an orthogonal high-rank adaptation for parameter-efficient fine-tuning on LLMs.According to the information bottleneck theory, OHoRA decomposes LLMs' pre-trained weight matrices into orthogonal basis vectors via QR decomposition and splits them into two low-redundancy high-rank components to suppress task-irrelevant information.It then performs dynamic rank-elevated recomposition through Kronecker product to generate expansive task-tailored representation spaces, enabling precise LLM adaptation and enhanced generalization.OHoRA effectively operationalizes the information bottleneck theory to decompose LLMs' weight matrices into low-redundancy high-rank components and recompose them in rank-elevated manner for more task-tailored representation spaces and precise LLM adaptation.Empirical evaluation shows OHoRA's effectiveness by outperforming LoRA and its variants and achieving comparable performance to full fine-tuning with only 0.0371% trainable parameters.

The Broad Learning System (BLS) has been established as an effective flat network alternative to Deep Neural Networks (DNNs), delivering high efficiency while achieving competitive accuracy. Despite its advantages, the incremental learning methods of BLS face challenges in stability and computation when expanding with new nodes or input. We introduce two novel incremental learning algorithms based on factorization updates for BLS that optimize node and input additions to overcome these limitations. Our node addition algorithm utilizes QR decomposition and Cholesky factorization, using the update of the Cholesky factor instead of pseudo-inverse computations. For input addition, we propose an iterative Cholesky factor update algorithm. Our algorithms demonstrate not only faster computation compared to the existing BLS but also improved testing accuracy on the MNIST or Fashion-MNIST dataset. This work presents a significant step forward in the practical application and scalability of BLS in various data-dense environments.