Abstract Distributed machine learning approaches are required when training data cannot be collected in a central location, due to storage, transmission or privacy/security constraints. An important task in any distributed machine learning context, and Federated Learning is no exception, is data value estimation or credit allocation, where the goal is to reward each participant proportionally to their contribution to the final performance of the machine learning model. However, all existing data value estimation techniques require that training be completed before the data values are obtained, and in this sense they can be considered as “a posteriori” approaches. Thus, all potential contributors must participate in the training process, regardless of the quality of their data or the final reward they can obtain. Here we present an “a priori” Shapley data value estimation technique in which, based on some statistical measures provided by the participants, the central counterpart or aggregator can obtain reasonably accurate data value estimates before actually starting the distributed learning process. To the best of our knowledge, this is the first “a priori” data value estimation approach proposed in the literature, and it can be used for the pre-selection of participants or to implement new pricing schemes. The introduced algorithms have been benchmarked using a variety of datasets and a logistic regression model, and we show that our “a priori” estimates are very accurate, compared to the centralized Shapley data values.

Removing the contribution of a part of the training data from a trained model is a task that has recently gained a lot of attention. A naïve approach is to retrain the model from scratch after excluding the unwanted training data, but the main research interest is in unlearning techniques that operate on the trained model with less computational cost than full retraining. Additionally, data contributors may no longer be available for a retraining process, for example in a distributed federated learning environment. Existing methods for unlearning in Support Vector Machine, based on forgetting samples using decremental methods, require active collaboration of all participants, and general unlearning methods that are based on gradient descent are not applicable in the SVM case since the cost function is not differentiable. We propose here a training scheme for Budget Support Vector Machines that has unlearning capabilities once the model has been trained. We benchmark the proposed unlearning method using a variety of datasets and unlearning scenarios in a federated distributed training setup and show its advantages in Accuracy, Efficiency and Effectiveness, with respect to the Sharded, Isolated, Sliced, and Aggregated approach, which consists of training weak local learners at each participant and obtaining the final decision by a voting mechanism.

In recent years, there has been remarkable growth in Federated Learning (FL) approaches because they have proven to be very effective in training large Machine Learning (ML) models and also serve to preserve data confidentiality, as recommended by the GDPR or other business confidentiality restrictions that may apply. Despite the success of FL, performance is greatly reduced when data is not distributed identically (non-ID) across participants, as local model updates tend to diverge from the optimal global solution and thus the model averaging procedure in the aggregator is less effective. Kernel methods such as Support Vector Machines (SVMs) have not seen an equivalent evolution in the area of privacy preserving edge computing because they suffer from inherent computational, privacy and scalability issues. Furthermore, non-linear SVMs do not naturally lead to federated schemes, since locally trained models cannot be passed to the aggregator because they reveal training data (they are built on Support Vectors), and the global model cannot be updated at every worker using gradient descent. In this article, we explore the use of a particular controlled complexity (“Budget”) Distributed SVM (BDSVM) in the FL scenario with non-ID data, which is the least favorable situation, but very common in practice. The proposed BDSVM algorithm is as follows: model weights are broadcasted to workers, which locally update some kernel Gram matrices computed according to a common architectural base and send them back to the aggregator, which finally combines them, updates the global model, and repeats the procedure until a convergence criterion is met. Experimental results using synthetic 2D datasets show that the proposed method can obtain maximal margin decision boundaries even when the data is non-ID distributed. Further experiments using real-world datasets with non-ID data distribution show that the proposed algorithm provides better performance with less communication requirements than a comparable Multilayer Perceptron (MLP) trained using FedAvg. The advantage is more remarkable for a larger number of edge devices. We have also demonstrated the robustness of the proposed method against information leakage, membership inference attacks, and situations with dropout or straggler participants. Finally, in experiments run on separate processes/machines interconnected via the cloud messaging service developed in the context of the EU-H2020 MUSKETEER project, BDSVM is able to train better models than FedAvg in about half the time.

In recent years there has been a noticeable increase in the number of available Big Data infrastructures. This fact has promoted the adaptation of traditional machine learning techniques to be capable of addressing large scale problems in distributed environments. Kernel methods like support vector machines (SVMs) suffer from scalability problems due to their nonparametric nature and the complexity of their training procedures. In this paper, we propose a new and efficient distributed implementation of a training procedure for nonlinear semiparametric (budgeted) SVMs called distributed iterative reweighted least squares (IRWLS). This algorithm uses k-means to select the centroids of the semiparametric model and a new distributed algorithmic implementation of the IRWLS optimization procedure to find the weights of the model. We have implemented the proposed algorithm in Apache Spark and we have benchmarked it against other state-of-the-art methods, either full SVM (p-pack SVM) or budgeted (budgeted stochastic gradient descent). Experimental results show that the proposed algorithm achieves higher accuracy while controlling the size of the final model, and also offers high performance in terms of run time and efficiency, when processing very large datasets (the computation time grows linear with the number of training patterns).

A new method for eye movement artefacts removal based on independent component analysis (ICA) and recursive least squares (RLS) is presented. The proposed algorithm combines the effective ICA capacity of separating artefacts from brain waves, together with the online interference cancellation achieved by adaptive filtering. Eye blink, saccades, eyes opening and closing produce changes of potentials at frontal areas. For this reason, the method uses as a reference the electrodes closest to the eyes Fp1, Fp2, F7 and F8, which register vertical and horizontal eye movements in the electroencephalogram (EEG) caused by these activities as an alternative of using extra dedicated electrooculogram (EOG) electrodes, which could not always be available and could be subject to larger variability. Both reference signals and EEG components are first projected into ICA domain and then the interference is estimated using the RLS algorithm. The component related to EOG artefact is automatically eliminated using channel localisations. Results from experimental data demonstrate that this approach is suitable for eliminating artefacts caused by eye movements, and the principles of this method can be extended to certain other artefacts as well, whenever a correlated reference signal is available.

In recent years the number of cores in computers has increased considerably, opening new lines of research to adapt classical techniques of machine learning to a parallel scenario. In this paper, we have developed and implemented with the multi-platform application programming interface OpenMP a method to train Semiparametric Support Vector Machines relying on Sparse Greedy Matrix Approximation (SGMA) and Iterated Re-Weighted Least Squares algorithm (IRWLS). We take advantage of the matrix formulation of SGMA and IRWLS. We recursively apply the partitioned matrix inversion lemma and other matrix decompositions to obtain a simple procedure to parallelize SVMS with good performance and computational efficiency.

Nonlinear system identification based on support vector machines (SVM) has been usually addressed by means of the standard SVM regression (SVR), which can be seen as an implicit nonlinear autoregressive and moving average (ARMA) model in some reproducing kernel Hilbert space (RKHS). The proposal of this letter is twofold. First, the explicit consideration of an ARMA model in an RKHS (SVM-ARMA2K) is proposed. We show that stating the ARMA equations in an RKHS leads to solving the regularized normal equations in that RKHS, in terms of the autocorrelation and cross correlation of the (nonlinearly) transformed input and output discrete time processes. Second, a general class of SVM-based system identification nonlinear models is presented, based on the use of composite Mercer's kernels. This general class can improve model flexibility by emphasizing the input-output cross information (SVM-ARMA4K), which leads to straightforward and natural combinations of implicit and explicit ARMA models (SVR-ARMA2K and SVR-ARMA4K). Capabilities of these different SVM-based system identification schemes are illustrated with two benchmark problems.

A truly distributed (as opposed to parallelized) support vector machine (SVM) algorithm is presented. Training data are assumed to come from the same distribution and are locally stored in a number of different locations with processing capabilities (nodes). In several examples, it has been found that a reasonably small amount of information is interchanged among nodes to obtain an SVM solution, which is better than that obtained when classifiers are trained only with the local data and comparable (although a little bit worse) to that of the centralized approach (obtained when all the training data are available at the same place). We propose and analyze two distributed schemes: a "naïve" distributed chunking approach, where raw data (support vectors) are communicated, and the more elaborated distributed semiparametric SVM, which aims at further reducing the total amount of information passed between nodes while providing a privacy-preserving mechanism for information sharing. We show the feasibility of our proposal by evaluating the performance of the algorithms in benchmarks with both synthetic and real-world datasets.

An adaptive, convex linear combination of the outputs of a standard least mean square (LMS) equalizer and a sigmoidal equalizer is proposed. This procedure results in improving the speed of the LMS equalizer while retaining the low steady-state error of the sigmoidal filter. Appropriate adaption schemes for both of the filters and for the combination parameters are established. Simulations of practical communication applications demonstrate the effectiveness of this adaptive combination.

In this paper, we propose a general technique for solving support vector classifiers (SVCs) for an arbitrary loss function, relying on the application of an iterative reweighted least squares (IRWLS) procedure. We further show that three properties of the SVC solution can be written as conditions over the loss function. This technique allows the implementation of the empirical risk minimization (ERM) inductive principle on large margin classifiers obtaining, at the same time, very compact (in terms of number of support vectors) solutions. The improvements obtained by changing the SVC loss function are illustrated with synthetic and real data examples.

An iterative block training method for support vector classifiers (SVCs) based on weighted least squares (WLS) optimization is presented. The algorithm, which minimizes structural risk in the primal space, is applicable to both linear and nonlinear machines. In some nonlinear cases, it is necessary to previously find a projection of data onto an intermediate-dimensional space by means of either principal component analysis or clustering techniques. The proposed approach yields very compact machines, the complexity reduction with respect to the SVC solution is especially notable in problems with highly overlapped classes. Furthermore, the formulation in terms of WLS minimization makes the development of adaptive SVCs straightforward, opening up new fields of application for this type of model, mainly online processing of large amounts of (static/stationary) data, as well as online update in nonstationary scenarios (adaptive solutions). The performance of this new type of algorithm is analyzed by means of several simulations.

The support vector classifier is a new tool to solve classification problems, giving the classification boundary as a linear combination of the training samples. In non-separable problems with highly overlapped classes, the achieved classifiers are oversized. In this paper, we proposed to change the support vector classifier penalty function by an hyperbolic tangent one, obtaining as a result of the training phase a reduced support vector classifier with the same performance as the original one.

Among the many techniques available for information concealment, those based on spread spectrum modulations have proven to yield improved results when robustness against attack is at a premium. In this paper, we propose a new spread spectrum-based watermarking procedure that combines space and frequency marks to provide good robustness properties against both spatial (affine) and transform-based compression attacks, without needing the original image as a reference (blind detection). It provides a mechanism for N-bit concealment and also improves the detection-of-presence process by gathering all the watermark energy into a single value (sufficient statistic for detection). The recovery of every single bit is also improved by taking into account the so-called "watermark-print" or "waterprint" instead of looking at a single correlation value. It additionally provides the means to recover synchronization under affine transformations in the blind detection scenario. These characteristics are analyzed by means of several practical examples.

In this communication we present a new algorithm for solving Support Vector Classifiers (SVC) with large training data sets. The new algorithm is based on an Iterative Re-Weighted Least Squares procedure which is used to optimize the SVc. Moreover, a novel sample selection strategy for the working set is presented, which randomly chooses the working set among the training samples that do not fulfill the stopping criteria. The validity of both proposals, the optimization procedure and sample selection strategy, is shown by means of computer experiments using well-known data sets.

The attractive possibility of applying layerwise block training algorithms to multilayer perceptrons MLP, which offers initial advantages in computational effort, is refined in this article by means of introducing a sensitivity correction factor in the formulation. This results in a clear performance advantage, which we verify in several applications. The reasons for this advantage are discussed and related to implicit relations with second-order techniques, natural gradient formulations through Fisher's information matrix, and sample selection. Extensions to recurrent networks and other research lines are suggested at the close of the article.