This paper introduces a novel approach to learn multi-task regression models with constrained architecture complexity. The proposed model, named RFF-BLR, consists of a randomised feedforward neural network with two fundamental characteristics: a single hidden layer whose units implement the random Fourier features that approximate an RBF kernel, and a Bayesian formulation that optimises the weights connecting the hidden and output layers. The RFF-based hidden layer inherits the robustness of kernel methods. The Bayesian formulation enables promoting multioutput sparsity: all tasks interplay during the optimisation to select a compact subset of the hidden layer units that serve as common non-linear mapping for every tasks. The experimental results show that the RFF-BLR framework can lead to significant performance improvements compared to the state-of-the-art methods in multitask nonlinear regression, especially in small-sized training dataset scenarios.

Adaptive learning is necessary for nonstationary environments where the learning machine needs to forget past data distribution. Efficient algorithms require a compact model update to not grow in computational burden with the incoming data and with the lowest possible computational cost for online parameter updating. Existing solutions only partially cover these needs. Here, we propose the first adaptive sparse Gaussian process (GP) able to address all these issues. We first reformulate a variational sparse GP (VSGP) algorithm to make it adaptive through a forgetting factor. Next, to make the model inference as simple as possible, we propose updating a single inducing point of the SGP model together with the remaining model parameters every time a new sample arrives. As a result, the algorithm presents a fast convergence of the inference process, which allows an efficient model update (with a single inference iteration) even in highly nonstationary environments. Experimental results demonstrate the capabilities of the proposed algorithm and its good performance in modeling the predictive posterior in mean and confidence interval estimation compared to state-of-the-art approaches.

PAC-Bayes bounds have been proposed to get risk estimates based on a training sample. In this paper the PAC-Bayes approach is combined with stability of the hypothesis learned by a Hilbert space valued algorithm. The PAC-Bayes setting is used with a Gaussian prior centered at the expected output. Thus a novelty of our paper is using priors defined in terms of the data-generating distribution. Our main result estimates the risk of the randomized algorithm in terms of the hypothesis stability coefficients. We also provide a new bound for the SVM classifier, which is compared to other known bounds experimentally. Ours appears to be the first uniform hypothesis stability-based bound that evaluates to non-trivial values.

We propose a new algorithm for sequence segmentation based on recent advances in semi-parametric sequence clustering. This approach implies the use of model-based distance measures between sequences, as well as a variant of spectral clustering specially tailored for segmentation. The method is highly flexible since it allows for the use of any probabilistic generative model for the individual segments. The performance of the proposed algorithm is demonstrated using both a synthetic dataset and a speaker segmentation task.

We review the existing alternatives for defining model-based distances for clustering sequences and propose a new one based on the Kullback-Leibler divergence. This distance is shown to be especially useful in combination with spectral clustering. For improved performance in real-world scenarios, a model selection scheme is also proposed.

This paper proposes a PAC-Bayes bound to measure the performance of Support Vector Machine (SVM) classifiers. The bound is based on learning a prior over the distribution of classifiers with a part of the training samples. Experimental work shows that this bound is tighter than the original PAC-Bayes, resulting in an enhancement of the predictive capabilities of the PAC-Bayes bound. In addition, it is shown that the use of this bound as a means to estimate the hyperparameters of the classifier compares favourably with cross validation in terms of accuracy of the model, while saving a lot of computational burden.

The fields of machine learning and mathematical programming are increasingly intertwined. Optimization problems lie at the heart of most machine learning approaches. The Special Topic on Machine Learning and Large Scale Optimization examines this interplay. Machine learning researchers have embraced the advances in mathematical programming allowing new types of models to be pursued. The special topic includes models using quadratic, linear, second-order cone, semi-definite, and semi-infinite programs. We observe that the qualities of good optimization algorithms from the machine learning and optimization perspectives can be quite different. Mathematical programming puts a premium on accuracy, speed, and robustness. Since generalization is the bottom line in machine learning and training is normally done off-line, accuracy and small speed improvements are of little concern in machine learning. Machine learning prefers simpler algorithms that work in reasonable computational time for specific classes of problems. Reducing machine learning problems to well-explored mathematical programming classes with robust general purpose optimization codes allows machine learning researchers to rapidly develop new techniques. In turn, machine learning presents new challenges to mathematical programming. The special issue include papers from two primary themes: novel machine learning models and novel optimization approaches for existing models. Many papers blend both themes, making small changes in the underlying core mathematical program that enable the develop of effective new algorithms.

A truly distributed (as opposed to parallelized) support vector machine (SVM) algorithm is presented. Training data are assumed to come from the same distribution and are locally stored in a number of different locations with processing capabilities (nodes). In several examples, it has been found that a reasonably small amount of information is interchanged among nodes to obtain an SVM solution, which is better than that obtained when classifiers are trained only with the local data and comparable (although a little bit worse) to that of the centralized approach (obtained when all the training data are available at the same place). We propose and analyze two distributed schemes: a "naïve" distributed chunking approach, where raw data (support vectors) are communicated, and the more elaborated distributed semiparametric SVM, which aims at further reducing the total amount of information passed between nodes while providing a privacy-preserving mechanism for information sharing. We show the feasibility of our proposal by evaluating the performance of the algorithms in benchmarks with both synthetic and real-world datasets.

We focus on a non-GEO scenario and investigate the effect of actual power control performance on the return capacity dynamics of a DS-CDMA system. We consider mobile users at different speeds and transmission rates yielding several classes of service. We compute analytically the ideal maximum return capacity of the system which is shown not to be fixed but limited by a hyperplane in the space of the considered quality of service (QoS) so that different resource assignment criteria can be applied. At maximum ideal capacity, it is shown that, for the different percentages of users that can be served, the more restraining the QoS the more resources that must be assigned. We then explore the appropriate efficiency of the system, i.e., the actual used resources as a function of the distribution of QoS. We finally look into the inter-beam effect by applying different degrees of diversity gains and we show the trade-off between improving the capacity efficiency which is poor due to power control error and the net capacity achieved by the overall system.

We focus on a non-GEO scenario and investigate the effect of actual power control performance on the return capacity dynamics of a DS-CDMA system. We consider mobile users at different speeds and transmission rates yielding several classes of services. We compute analytically the ideal maximum return capacity of the system which is shown not to be fixed but limited by a hyperplane in the space of the considered quality of service (QoS), so that different resource assignment criteria can be applied. At maximum ideal capacity, for the different percentages of users that can be served, it is shown that the more restraining the QoS, the more resources that must be assigned. We then explore the appropriate efficiency of the system, i.e. the actual used resources as a function of the distribution of QoS. We finally look into the inter-beam effect by applying different degrees of diversity gains and we show the trade-off between improving the capacity efficiency which is poor due to power control error and the net capacity achieved by the overall system.