One central theme in machine learning is function estimation from sparse and noisy data. An example is supervised learning where the elements of the training set are couples, each containing an input location and an output response. In the last decades, a substantial amount of work has been devoted to design estimators for the unknown function and to study their convergence to the optimal predictor, also characterizing the learning rate. These results typically rely on stationary assumptions where input locations are drawn from a probability distribution that does not change in time. In this work, we consider kernel-based ridge regression and derive convergence conditions under non stationary distributions, addressing also cases where stochastic adaption may happen infinitely often. This includes the important exploration-exploitation problems where e.g., a set of agents/robots has to monitor an environment to reconstruct a sensorial field and their movements rules are continuously updated on the basis of the acquired knowledge on the field and/or the surrounding environment.

Estimating a set of orthogonal functions from a finite set of noisy data plays a crucial role in several areas such as imaging, dictionary learning and compressed sensing. The problem turns out especially hard due to its intrinsic non-convexity. In this paper, we solve it by recasting it in the framework of multi-task learning in Hilbert spaces, where orthogonality plays a role as inductive bias. Two perspectives are analyzed. The first one is mainly theoretic. It considers a formulation of the problem where non-orthogonal function estimates are seen as noisy data belonging to an infinite-dimensional space from which orthogonal functions have to be reconstructed. We then provide results concerning the existence and the convergence of the optimizers. The second one is more oriented towards applications. It consists in a learning scheme where orthogonal functions are directly inferred from a finite amount of noisy data. It relies on regularization in reproducing kernel Hilbert spaces and on the introduction of special penalty terms promoting orthogonality among tasks. The problem is then cast in a Bayesian framework, overcoming non-convexity through an efficient Markov chain Monte Carlo scheme. If orthogonality is not certain, our scheme can also understand from data if such form of task interaction really holds.