The ever-increasing size of large language models (LLMs) presents significant challenges for deployment due to their heavy computational and memory requirements. Current model pruning techniques attempt to alleviate these issues by relying heavily on external calibration datasets to determine which parameters to prune or compress, thus limiting their flexibility and scalability across different compression ratios. Moreover, these methods often cause severe performance degradation, particularly in downstream tasks, when subjected to higher compression rates. In this paper, we propose *PruneNet*, a novel model compression method that addresses these limitations by reformulating model pruning as a policy learning process. PruneNet decouples the pruning process from the model architecture, eliminating the need for calibration datasets. It learns a stochastic pruning policy to assess parameter importance solely based on intrinsic model properties while preserving the spectral structure to minimize information loss. PruneNet can compress the LLaMA-2-7B model in just 15 minutes, achieving over 80\% retention of its zero-shot performance with a 30\% compression ratio, outperforming existing methods that retain only 75\% performance. Furthermore, on complex multitask language understanding tasks, PruneNet demonstrates its robustness by preserving up to 80\% performance of the original model, proving itself a superior alternative to conventional structured compression techniques.

Key-value (KV) cache compression has emerged as a critical technique for reducing the memory and latency overhead of autoregressive language models during inference. Prior approaches predominantly rely on query-key attention scores to rank and evict cached tokens, assuming that attention intensity correlates with semantic importance. However, this heuristic overlooks the contribution of value vectors, which directly influence the attention output. In this paper, we propose CurDKV, a novel, value-centric KV compression method that selects keys and values based on leverage scores computed from CUR matrix decomposition. Our approach approximates the dominant subspace of the attention output $\mathrm{softmax}(QK^\top)V$, ensuring that the retained tokens best preserve the model’s predictive behavior. Theoretically, we show that attention score approximation does not guarantee output preservation, and demonstrate that CUR-based selection minimizes end-to-end attention reconstruction loss. Empirically, CurDKV achieves up to $9.6$\% higher accuracy than state-of-the-art methods like SnapKV and ChunkKV under aggressive compression budgets on LLaMA and Mistral, while maintaining compatibility with FlashAttention and Grouped Query Attention. In addition to improved accuracy, CurDKV reduces generation latency by up to 40\% at high compression, offering a practical speed-accuracy tradeoff.

Abstract Fine-tuning large language models (LLMs) on downstream tasks requires substantial computational resources. Selective-PEFT, a class of parameter-efficient fine-tuning (PEFT) methodologies, aims to mitigate these computational challenges by selectively fine-tuning only a small fraction of the model parameters. Although parameter-efficient, these techniques often fail to match the performance of fully fine-tuned models, primarily due to inherent biases introduced during parameter selection. Traditional selective-PEFT techniques use a fixed set of parameters selected using different importance heuristics, failing to capture parameter importance dynamically and often leading to suboptimal performance. We introduce ID3, a novel selective-PEFT method that calculates parameter importance continually, and dynamically unmasks parameters by balancing exploration and exploitation in parameter selection. Our empirical study on 16 tasks spanning natural language understanding, mathematical reasoning, and summarization demonstrates the effectiveness of our method compared to fixed-masking selective-PEFT techniques. We analytically show that ID3 reduces the number of gradient updates by a factor of two, enhancing computational efficiency. Since ID3 is robust to random initialization of neurons and operates directly on the optimization process, it is highly flexible and can be integrated with existing additive and reparameterization-based PEFT techniques such as Adapters and LoRA, respectively.1

Knowledge distillation (KD) is a technique used to transfer knowledge from a larger ''teacher'' model into a smaller ''student'' model. Recent advancements in meta-learning-based knowledge distillation (MetaKD) emphasize that the fine-tuning of teacher models should be aware of the student's need to achieve better knowledge distillation. However, existing MetaKD methods often lack incentives for the teacher model to improve itself. In this study, we introduce MPDistil, a meta-policy distillation technique, that utilizes novel optimization strategies to foster both *collaboration* and *competition* during the fine-tuning of the teacher model in the meta-learning step. Additionally, we propose a curriculum learning framework for the student model in a competitive setup, in which the student model aims to outperform the teacher model by self-training on various tasks. Exhaustive experiments on SuperGLUE and GLUE benchmarks demonstrate the efficacy of MPDistil compared to $20$ conventional KD and advanced MetaKD baselines, showing significant performance enhancements in the student model -- e.g., a distilled 6-layer BERT model outperforms a 12-layer BERT model on five out of six SuperGLUE tasks. Furthermore, MPDistil, while applied to a large language teacher model (DeBERTa-v2-xxlarge), significantly narrows the performance gap of its smaller student counterpart (DeBERTa-12) by just $4.6$% on SuperGLUE. We further demonstrate how higher rewards and customized training curricula strengthen the student model and enhance generalizability.

Liquid biopsies based on peripheral blood offer a minimally invasive alternative to solid tissue biopsies for the detection of diseases, primarily cancers. However, such tests currently consider only the serum component of blood, overlooking a potentially rich source of biomarkers: adaptive immune receptors (AIRs) expressed on circulating B and T cells. Machine learning-based classifiers trained on AIRs have been reported to accurately identify not only cancers but also autoimmune and infectious diseases as well. However, when using the conventional "clonotype cluster" representation of AIRs, individuals within a disease or healthy cohort exhibit vastly different features, limiting the generalizability of these classifiers. This study aimed to address the challenge of classifying specific diseases from circulating B or T cells by developing a novel representation of AIRs based on similarity networks constructed from their antigen-binding regions (paratopes). Features based on this novel representation, paratope cluster occupancies (PCOs), significantly improved disease classification performance for infectious disease, autoimmune disease, and cancer. Under identical methodological conditions, classifiers trained on PCOs achieved a mean AUC of 0.893 when applied to new individuals, outperforming clonotype cluster-based classifiers (AUC 0.714) and the best-performing published classifier (AUC 0.777). Surprisingly, for cancer patients, we observed that "healthy-biased" AIRs were predicted to target known cancer-associated antigens at dramatically higher rates than healthy AIRs as a whole (Z scores >75), suggesting an overlooked reservoir of cancer-targeting immune cells that could be identified by PCOs.

Being a popular mode of text-based communication in multilingual communities, code mixing in online social media has become an important subject to study. Learning the semantics and morphology of code-mixed language remains a key challenge due to the scarcity of data, the unavailability of robust, and language-invariant representation learning techniques. Any morphologically rich language can benefit from character, subword, and word-level embeddings, aiding in learning meaningful correlations. In this article, we explore a hierarchical transformer (HIT)-based architecture to learn the semantics of code-mixed languages. HIT consists of multiheaded self-attention (MSA) and outer product attention components to simultaneously comprehend the semantic and syntactic structures of code-mixed texts. We evaluate the proposed method across six Indian languages (Bengali, Gujarati, Hindi, Tamil, Telugu, and Malayalam) and Spanish for nine tasks on 17 datasets. The HIT model outperforms state-of-the-art code-mixed representation learning and multilingual language models on 13 datasets across eight tasks. We further demonstrate the generalizability of the HIT architecture using masked language modeling (MLM)-based pretraining, zero-shot learning (ZSL), and transfer learning approaches. Our empirical results show that the pretraining objectives significantly improve the performance of downstream tasks.

With the availability of domain specific historical negotiation data, the practical applications of machine learning techniques can prove to be increasingly effective in the field of automated negotiation. Yet a large portion of the literature focuses on domain independent negotiation and thus passes the possibility of leveraging any domain specific insights from historical data. Moreover, during sequential negotiation, utility functions may alter due to various reasons including market demand, partner agreements, weather conditions, etc. This poses a unique set of challenges and one can easily infer that one strategy that fits all is rather impossible in such scenarios. In this work, we present a simple yet effective method of learning an end-to-end negotiation strategy from historical negotiation data. Next, we show that transfer learning based solutions are effective in designing adaptive strategies when underlying utility functions of agents change. Additionally, we also propose an online method of detecting and measuring such changes in the utility functions. Combining all three contributions we propose an adaptive automated negotiating agent framework that enables the automatic creation of transfer learning based negotiating agents capable of adapting to changes in utility functions. Finally, we present the results of an agent generated using our framework in different ANAC domains with 100 different utility functions each and show that our agent outperforms the benchmark score by domain independent agents by 6%.

This paper explores the network simplification problem for Gaussian full-duplex relay networks with arbitrary topology. Particularly, given an N-relay Gaussian full-duplex network, the network simplification problem seeks to find fundamental guarantees on the capacity of the best subnetwork, among a particular class of subnetworks, as a fraction of the full-network capacity. The focus of this work is the case when the selected subnetwork class is a path from the source to the destination. The main result of the paper shows that for an N-relay Gaussian networks with arbitrary topology, the best route can in the worst case guarantee an approximate fraction 1/(⌊N/2⌋ + 1) of the capacity of the full network, independently of the channel coefficients and/or operating SNR. Furthermore, this guarantee is shown to be fundamental, i.e., it is the highest worst-case guarantee that can be provided for routing in relay networks. A key step in the proof of the main result lies in the derivation of a simplification result for antenna selection in MIMO channels that may also be of independent interest. To the best of our knowledge, this is the first result that characterizes the performance of routing in comparison to physical layer cooperation techniques that approximately achieve the network capacity for general wireless network topologies. The results in this paper show that routing can, in the worst case, result in an unbounded gap from the network capacity - or reversely, physical layer cooperation can offer unbounded gains over routing.

We study the generalized degrees of freedom (gDoF) of the block-fading noncoherent MIMO channel with asymmetric distributions of link strengths, and a coherence time of T symbol durations. We first derive the optimal signaling structure for communication over this channel, which is distinct from that for the i.i.d MIMO setting. We prove that for T = 1, the gDoF is zero for MIMO channels with arbitrary link strength distributions, extending the result for MIMO with i.i.d links. We then show that selecting the statistically best antenna is gDoF-optimal for both Multiple Input Single Output (MISO) and Single Input Multiple Output (SIMO) channels. We also derive the gDoF for the 2×2 MIMO channel with different exponents in the direct and cross links. In this setting, we show that it is always necessary to use both antennas to achieve the optimal gDoF, in contrast to the results for 2 × 2 MIMO with identical link distributions. We also show that having weaker crosslinks gives gDoF gain compared to the case with identically distributed links.

We consider an arbitrary layered Gaussian relay network with L layers of N relays each, from which we select subnetworks with K relays per layer. We prove that: (i) For arbitrary L;N and K = 1, there always exists a subnetwork that approximately achieves 2/(L-1)N+4 (resp. 2/LN+2) of the network capacity for odd L (resp. even L), (ii) For L = 2; N = 3; K = 2, there always exists a subnetwork that approximately achieves 1/2 of the network capacity. We also provide example networks where even the best subnetworks achieve exactly these fractions (up to additive gaps). Along the way, we derive some results on MIMO antenna selection and capacity decomposition that may also be of independent interest.

Physical layer cooperation of a source with a relay can significantly boost the performance of a wireless connection. However, the best practical relaying scheme can vary depending on the relative strengths of the channels that connect the source, relay and destination. This paper proposes and evaluates QUILT, a system for physical-layer relaying that seamlessly adapts to the underlying network configuration to achieve competitive or better performance as compared to the best current approaches. QUILT combines on-demand, opportunistic use of Decode-Forward (DF) or Quantize-Map-Forward (QMF) followed by interleaving at the relay, with hybrid decoding at the destination that extracts information from received frames even if these are not decodable. We theoretically quantify how our design choices for QUILT affect the system performance. We also deploy QUILT on the WarpLab software radio platform, and show through over-the-air experiments up to 5 times FER improvement over the next best cooperative protocol.

We consider a source that would like to communicate with a destination over a layered Gaussian relay network.We present a computationally efficient method that enables to select a near-optimal (in terms of throughput) subnetwork of a given size connecting the source with the destination. Our method starts by formulating an integer optimization problem that maximizes the rates that the Quantize-Map-and-Forward relaying protocol can achieve over a selected subnetwork; we then relax the integer constraints to obtain a non-linear optimization over reals. For diamond networks, we prove that this optimization over reals is concave while for general layered networks we give empirical demonstrations of near-concavity, paving the way for efficient algorithms to solve the relaxed problem. We then round the relaxed solution to select a specific subnetwork. Simulations using off-the-shelf non-linear optimization algorithms demonstrate excellent performance with respect to the true integer optimum for both diamond networks as well as multi-layered networks. Even with these non-customized algorithms, significant time savings are observed vis-à-vis exhaustive integer optimization.

We consider a Gaussian diamond network where a source communicates with the destination through n non-interfering half-duplex relays. We focus on half-duplex schedules that utilize only local channel state information, i.e., each relay has access to its incoming and outgoing channel realizations. We demonstrate that random independent switching, resulting in multiple listen-transmit sub cycles at each relay, while still respecting the overall locally optimal listen-transmit fractions, enables to approximately achieve at least 3/4 of the capacity of the 2-relay diamond network. With a single listen-transmit cycle, this fraction drops from 3/4 to 1/2. We also provide simulation results that point to the same fractions of capacity being retained over networks with more than 2 relays.

This paper contributes to the design and analysis of Quantize-Map-and-Forward (QMF) relaying by optimizing its performance for small relay networks. QMF was proved to achieve the capacity of arbitrary networks within a bounded gap, as well as the optimal diversity-multiplexing tradeoff over slow fading networks. The initial QMF scheme has each relay performing the same operation, agnostic to the network topology and the channel state information (CSI); this facilitates the analysis for arbitrary networks, yet comes at a performance penalty for small networks and medium SNR regimes. This paper demonstrates the benefits we can gain for QMF if we optimize its performance by leveraging topological and channel state information. We show that for the N-relay diamond network, by taking into account topological information, we can exponentially reduce the QMF additive approximation gap from Θ(N) bits/s/Hz to Θ(log N) bits/s/Hz, while for the one-relay and two-relay networks, use of topological information and CSI can help to gain as much as 6 dB. Moreover, we explore what benefits we can realize if we jointly optimize QMF and half-duplex scheduling, as well as if we employ hybrid schemes that combine QMF and Decode-and-Forward (DF) relay operations.

We present the design and experimental evaluation of a wireless system that exploits relaying in the context of WiFi. We opt for WiFi given its popularity and wide spread use for a number of applications, such as smart homes. Our testbed consists of three nodes, a source, a relay and a destination, that operate using the physical layer procedures of IEEE802.11. We deploy three main competing strategies that have been proposed for relaying, Decode-and-Forward (DF), Amplify-and-Forward (AF) and Quantize-Map-Forward (QMF). QMF is the most recently introduced of the three, and although it was shown in theory to approximately achieve the capacity of arbitrary wireless networks, its performance in practice had not been evaluated. We present in this work experimental results---to the best of our knowledge, the first ones---that compare QMF, AF and DF in a realistic indoor setting. We find that QMF is a competitive scheme to the other two, offering in some cases up to 12% throughput benefits and up to 60% improvement in frame error-rates over the next best scheme.

We evaluate the information-theoretic achievable rates of Quantize-Map-and-Forward (QMF) relaying schemes over Gaussian N-relay diamond networks. Focusing on vector Gaussian quantization at the relays, our goal is to understand how close to the cutset upper bound these schemes can achieve in the context of diamond networks, and how much benefit is obtained by optimizing the quantizer distortions at the relays. First, with noise-level quantization, we point out that the worst-case gap from the cutset upper bound is (N + log2N) bits/s/Hz. A better universal quantization level found without using channel state information (CSI) leads to a sharpened gap of log2N + log2(1 + N) + N log2(1 + 1/N) bits/s/Hz. On the other hand, it turns out that finding the optimal distortion levels depending on the channel gains is a non-trivial problem in the general N-relay setup. We manage to solve the two-relay problem and the symmetric N-relay problem analytically, and show the improvement via numerical evaluations both in static as well as slow-fading channels.

We present a structured Quantize-Map-and-Forward (QMF) scheme for cooperative communication over wireless networks, that employs LDPC ensembles for the node operations and message-passing algorithms for decoding. We demonstrate through extensive simulation results over the full-duplex parallel relay network, that our scheme, with no transmit channel state information, offers a robust performance over fading channels and achieves the full diversity order of our network at moderate SNRs.