Large Language Models (LLMs) have achieved state-of-the-art results across diverse domains, yet their development remains reliant on vast amounts of publicly available data, raising concerns about data scarcity and the lack of access to domain-specific, sensitive information. Federated Learning (FL) presents a compelling framework to address these challenges by enabling decentralized fine-tuning on pre-trained LLMs without sharing raw data. However, the compatibility and performance of pre-trained LLMs in FL settings remain largely under explored. We introduce the FlowerTune LLM Leaderboard, a first-of-its-kind benchmarking suite designed to evaluate federated fine-tuning of LLMs across four diverse domains: general NLP, finance, medical, and coding. Each domain includes federated instruction-tuning datasets and domain-specific evaluation metrics. Our results, obtained through a collaborative, open-source and community-driven approach, provide the first comprehensive comparison across 26 pre-trained LLMs with different aggregation and fine-tuning strategies under federated settings, offering actionable insights into model performance, resource constraints, and domain adaptation. This work lays the foundation for developing privacy-preserving, domain-specialized LLMs for real-world applications.

AI projects often face the challenge of limited access to meaningful amounts of training data. In traditional approaches, collecting data in a central location can be problematic, especially in industry settings with sensitive and distributed data. However, there is a solution -"moving the computation to the data" through Federated Learning.

Federated learning (FL) enables multiple parties to collaboratively train a machine learning model without sharing their data; rather, they train their own model locally and send updates to a central server for aggregation. Depending on how the data is distributed among the participants, FL can be classified into Horizontal (HFL) and Vertical (VFL). In VFL, the participants share the same set of training instances but only host a different and non-overlapping subset of the whole feature space. Whereas in HFL, each participant shares the same set of features while the training set is split into locally owned training data subsets.VFL is increasingly used in applications like financial fraud detection; nonetheless, very little work has analyzed its security. In this paper, we focus on robustness in VFL, in particular, on backdoor attacks, whereby an adversary attempts to manipulate the aggregate model during the training process to trigger misclassifications. Performing backdoor attacks in VFL is more challenging than in HFL, as the adversary i) does not have access to the labels during training and ii) cannot change the labels as she only has access to the feature embeddings. We present a first-of-its-kind clean-label backdoor attack in VFL, which consists of two phases: a label inference and a backdoor phase. We demonstrate the effectiveness of the attack on three different datasets, investigate the factors involved in its success, and discuss countermeasures to mitigate its impact.

Algorithms such as Differentially Private SGD enable training machine learning models with formal privacy guarantees. However, because these guarantees hold with respect to unrealistic adversaries, the protection afforded against practical attacks is typically much better. An emerging strand of work empirically estimates the protection afforded by differentially private training as a confidence interval for the privacy budget $\hat{\varepsilon}$ spent with respect to specific threat models. Existing approaches derive confidence intervals for $\hat{\varepsilon}$ from confidence intervals for false positive and false negative rates of membership inference attacks, which requires training an impractically large number of models to get intervals that can be acted upon. We propose a novel, more efficient Bayesian approach that brings privacy estimates within the reach of practitioners. Our approach reduces sample size by computing a posterior for $\hat{\varepsilon}$ (not just a confidence interval) from the joint posterior of the false positive and false negative rates of membership inference attacks. We implement an end-to-end system for privacy estimation that integrates our approach and state-of-the-art membership inference attacks, and evaluate it on text and vision classification tasks. For the same number of samples, we see a reduction in interval width of up to 40% compared to prior work.

Modern defenses against cyberattacks increasingly rely on proactive approaches, e.g., to predict the adversary's next actions based on past events. Building accurate prediction models requires knowledge from many organizations; alas, this entails disclosing sensitive information, such as network structures, security postures, and policies, which might often be undesirable or outright impossible.

With widespread use of mobile devices, instant messaging (IM) services have recently attracted a great deal of attention by millions of users. This has motivated news agencies to share their contents via such platforms in addition to their websites and popular social media. As a result, thousands of users nowadays follow the news agencies through their verified channels in IM services. However, user interactions with such platforms is relatively unstudied. In this paper, we provide an initial study to analyze and predict news popularity in an instant messaging service. To this aim, we focus on Telegram, a popular IM service with 200 million monthly active users. We explore the differences between news popularity analysis in Telegram and typical social media, such as Twitter, and highlight its unique characteristics. We perform our analysis on the data we collected from four diverse news agencies. Following our analysis, we study the task of news popularity prediction in Telegram and show that the performance of the prediction models can be substantially improved by learning from the data of multiple news agencies using multi-task learning. To foster research in this area, we have made the collected data publicly available.

Abstract To support users with disabilities, Android provides the accessibility services, which implement means of navigating through an app. According to the Android developer’s guide: “Accessibility services should only be used to assist users with disabilities in using Android devices and apps”. However, developers are free to use this service without any restrictions, giving them critical privileges such as monitoring user input or screen content to capture sensitive information. In this paper, we show that simply enabling the accessibility service leaves 72 % of the top finance a nd 80 % of the top social media apps vulnerable to eavesdropping attacks, leaking sensitive information such as logins and passwords. A combination of several tools and recommendations could mitigate the privacy risks: We introduce an analysis technique that detects most of these issues automatically, e.g. in an app store. We also found that these issues can be automatically fixed in almost all cases; our fixes have b een accepted by 70 % of the surveyed developers. Finally, we designed a notification mechanism which would warn users against possible misuses of the accessibility services; 50 % of users would follow these notifications.