The integration of Differential Privacy (DP) with diffusion models (DMs) presents a promising yet challenging frontier, particularly due to the substantial memorization capabilities of DMs that pose significant privacy risks. Differential privacy offers a rigorous framework for safeguarding individual data points during model training, with Differential Privacy Stochastic Gradient Descent (DP-SGD) being a prominent implementation. Diffusion method decomposes image generation into iterative steps, theoretically aligning well with DP's incremental noise addition. Despite the natural fit, the unique architecture of DMs necessitates tailored approaches to effectively balance privacy-utility trade-off. Recent developments in this field have highlighted the potential for generating high-quality synthetic data by pre-training on public data (i.e., ImageNet) and fine-tuning on private data, however, there is a pronounced gap in research on optimizing the trade-offs involved in DP settings, particularly concerning parameter efficiency and model scalability. Our work addresses this by proposing a parameter-efficient fine-tuning strategy optimized for private diffusion models, which minimizes the number of trainable parameters to enhance the privacy-utility trade-off. We empirically demonstrate that our method achieves state-of-the-art performance in DP synthesis, significantly surpassing previous benchmarks on widely studied datasets (e.g., with only 0.47M trainable parameters, achieving a more than 35% improvement over the previous state-of-the-art with a small privacy budget on the CelebA-64 dataset). Anonymous codes available at https://anonymous.4open.science/r/DP-LORA-F02F.

In many applications, regulations or best practices often lead to specific requirements in machine learning relating to four key pillars: fairness, privacy, interpretability and greenhouse gas emissions. These all sit in the broader context of sustainability in AI, an emerging practical AI topic. However, although these pillars have been individually addressed by past literature, none of these works have considered all the pillars. There are inherent trade-offs between each of the pillars (for example, utility vs fairness or utility vs privacy), making it even more important to consider them together. This paper outlines a new framework for Sustainable Machine Learning. It proposes FPIG, a general AI pipeline that allows for simultaneous consideration and a better understanding of the tradeoffs between the pillars. Based on the FPIG framework, we propose a meta-learning algorithm to estimate the four key pillars given a dataset summary, model architecture, and hyperparameters before model training. This algorithm allows users to select the optimal model architecture for a given dataset and a set of user requirements on the pillars. We illustrate the trade-offs under the FPIG model on three classical datasets and demonstrate the meta-learning approach with an example of real-world datasets and models with different interpretability, showcasing how it can aid model selection.

When trying to liquidate a large quantity of a particular stock, the price of that stock is likely to be affected by trades, thus leading to a reduced expected return if we were to sell the entire quantity at once. This leads to the problem of optimal execution, where the aim is to split the sell order into several smaller sell orders over the course of a period of time, to optimally balance stock price with market risk. This problem can be defined in terms of difference equations. Here, we show how we can reformulate this as a multi-objective problem, which we solve with a novel multi-armed bandit algorithm.

This paper considers a multi-state Log Gaussian Cox Process (`"LGCP'') on a graph, where transmissions amongst states are calibrated using a non-parametric approach. We thus consider multi-output LGCPs and introduce numerical approximations to compute posterior distributions extremely quickly and in a completely transparent and reproducible fashion. The model is tested on historical data and shows very good performance.

Current deep learning techniques for RNA classification suffer from over-fitting and lack of reproducibility. We show that by introducing robustness by design in both CNN and RNN algorithms, we are able to achieve standalone state-of-the-art accuracy. By constructing model-agnostic robustness checks and reusing features obtained from both architectures, we build a collaborative framework that improves performance and stability.

Multi-output supervised learning and multi-task learning are all instances of a broader learning paradigm where features, parameters and objectives are shared to a certain extent. Examples of such approaches include reusing features from pre-existing models in a new algorithm, performing multi-label regression or optimising for several tasks jointly. In this paper, we address this challenge by devising a generic framework based on generalised low-rank models ("GLRMs"), which include – broadly speaking– most techniques that can be expressed in terms of matrix factorisation. Importantly, while GLRMs first and foremost tackle unsupervised learning problems and supervised linear models. Here, we show that GLRMs can be extended by introducing multivariate functionals and structure regularisation terms to handle multivariate learning. This paper also proposes a coherent framework to design multi-learning strategies and covers existing algorithms. Finally, we prove the simplicity and effectiveness of our approach on empirical data.

It is well-known that, for separable data, the regularised two-class logistic regression or support vector machine re-normalised estimate converges to the maximal margin classifier as the regularisation hyper-parameter $\lambda$ goes to 0. The fact that different loss functions may lead to the same solution is of theoretical and practical relevance as margin maximisation allows more straightforward considerations in terms of generalisation and geometric interpretation. We investigate the case where this convergence property is not guaranteed to hold and show that it can be fully characterised by the distribution of error terms in the latent variable interpretation of linear classifiers. In particular, if errors follow a regularly varying distribution, then the regularised and re-normalised estimate does not converge to the maximal margin classifier. This shows that classification with fat tails has a qualitatively different behaviour, which should be taken into account when considering real-life data.

This paper explores a generalisation of factorisation machines via kernels, which we call Kernel Factorisation Machines (“KFM”). It is well-known that functions in reproducing kernel Hilbert spaces can be understood as a linear combination of features in very high-dimensional (or infinite-dimensional) spaces while being computed in a finite-dimensional space, thanks to the representer theorem. Simultaneously, it has been shown recently that the dot product operation was a key component behind the success of a number of recommender systems, while the recent literature has been preoccupied with enriching factorisation machines. There is thus a need for a framework able to interpolate between factorisation machines that tend to outperform other techniques on sparse datasets and more advanced models that perform well on large and dense datasets. One of the drawbacks of kernel methods is their high dimensionality when the number of observations is large, which is typical of recommender systems. It is thus extremely important to be able to reduce the dimensionality, which we do in two different ways: first, we find a representation of the input features in a lower-dimensional space, and, second, we consider inducing points, i.e., surrogate inputs that are optimised upon training to avoid building (kernel) interactions between each pair of observations in the dataset. In short, we propose a method that adapts kernels to the set up of high-dimensional and potentially sparse datasets. To illustrate our approach, we test it on four well-known datasets and benchmark its results against most available models. While comparisons are difficult and should be interpreted carefully, KFM is able to perform well and obtains the best performance overall. Our methodology is not limited to recommender systems and can be applied to other settings, which we illustrate on a heart disease classification task.