Donald Shenaj, Ondrej Bohdal, Taha Ceritli, Mete Ozay, Pietro Zanuttigh, Umberto Michieli. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Current network training paradigms primarily focus on either centralized or decentralized data regimes. However, in practice, data availability often exhibits a hybrid nature, where both regimes coexist. This hybrid setting presents new opportunities for model training, as the two regimes offer complementary trade-offs: decentralized data is abundant but subject to heterogeneity and communication constraints, while centralized data—though limited in volume and potentially unrepresentative—enables better curation and high-throughput access. Despite its potential, effectively combining these paradigms remains challenging, and few frameworks are tailored to hybrid data regimes. To address this, we propose a novel framework that constructs a model atlas from decentralized models and leverages centralized data to refine a global model within this structured space. The refined model is then used to reinitialize the decentralized models. Our method synergizes federated learning (to exploit decentralized data) and model merging (to utilize centralized data), enabling effective training under hybrid data availability. Theoretically, we show that our approach achieves faster convergence than methods relying solely on decentralized data, due to variance reduction in the merging process. Extensive experiments demonstrate that our framework consistently outperforms purely centralized, purely decentralized, and existing hybrid-adaptable methods. Notably, our method remains robust even when the centralized and decentralized data domains differ or when decentralized data contains noise, significantly broadening its applicability.

Large language models (LLMs) often leverage adapters, such as low-rank-based adapters, to achieve strong performance on downstream tasks.However, storing a separate adapter for each task significantly increases memory requirements, posing a challenge for resourceconstrained environments such as mobile devices.Although model merging techniques can reduce storage costs, they typically result in substantial performance degradation.In this work, we introduce HydraOpt, a new model merging technique that capitalizes on the inherent similarities between the matrices of low-rank adapters.Unlike existing methods that produce a fixed trade-off between storage size and performance, HydraOpt allows us to navigate this spectrum of efficiency and performance.Our experiments show that Hy-draOpt significantly reduces storage size (48% reduction) compared to storing all adapters, while achieving competitive performance (0.2-1.8% drop).Furthermore, it outperforms existing merging techniques in terms of performance at the same or slightly worse storage efficiency.

In recent years, machine learning (ML) models have been integrated into various applications and products to improve user experience. However, this approach raises significant concerns about the protection of private user data utilized for training the models. One limitation of vanilla privacy methods is that they can improve the robustness of the models against privacy attacks at the cost of accuracy while performing ML tasks. We propose a framework for implementing privacy models that learn privacy budgets to improve the trade-off between privacy of user data, task models, and their utility (task accuracy). The experimental results show that our framework dramatically enhances the task accuracy of ML models in image classification tasks while providing better privacy protection compared to the state-of-the-art methods.

Parameter-efficient fine-tuning (PEFT) is utilized to adapt large pre-trained machine learning (ML) models to new tasks using a small number of trainable parameters. In particular, Low-Rank Adaptation (LoRA) is one of the prominent PEFT methods. To this end, we introduce a novel method that exploits data to improve the accuracy of models fine-tuned by LoRA. Our method implements a hypernetwork that generates refinement parameters using data to update the low-rank parameters of LoRA. The experimental results validate that our method improves the accuracy of large language models (LLMs) on language understanding tasks by 3% on average compared to LoRA and its variants.

Data wrangling tasks such as obtaining and linking data from various sources, transforming data formats, and correcting erroneous records, can constitute up to 80% of typical data engineering work. Despite the rise of machine learning and artificial intelligence, data wrangling remains a tedious and manual task. We introduceAI assistants, a class of semi-automatic interactive tools to streamline data wrangling. An AI assistant guides the analyst through a specific data wrangling task by recommending a suitable data transformation that respects the constraints obtained through interaction with the analyst. We formally define the structure of AI assistants and describe how existing tools that treat data cleaning as an optimization problem fit the definition. We implement AI assistants for four common data wrangling tasks and make AI assistants easily accessible to data analysts in an open-source notebook environment for data science, by leveraging the common structure they follow. We evaluate our AI assistants both quantitatively and qualitatively through three example scenarios. We show that the unified and interactive design makes it easy to perform tasks that would be difficult to do manually or with a fully automatic tool.

Abstract Type inference refers to the task of inferring the data type of a given column of data. Current approaches often fail when data contains missing data and anomalies, which are found commonly in real-world data sets. In this paper, we propose ptype, a probabilistic robust type inference method that allows us to detect such entries, and infer data types. We further show that the proposed method outperforms existing methods.