This paper aims at the algorithmic/theoretical core of reinforcement learning (RL) by introducing the novel class of proximal Bellman mappings. These mappings are defined in reproducing kernel Hilbert spaces (RKHSs), to benefit from the rich approximation properties and inner product of RKHSs, they are shown to belong to the powerful Hilbertian family of (firmly) nonexpansive mappings, regardless of the values of their discount factors, and possess ample degrees of design freedom to even reproduce attributes of the classical Bellman mappings and to pave the way for novel RL designs. An approximate policy-iteration scheme is built on the proposed class of mappings to solve the problem of selecting online, at every time instance, the "optimal" exponent p in a p-norm loss to combat outliers in linear adaptive filtering, without training data and any knowledge on the statistical properties of the outliers. Numerical tests on synthetic data showcase the superior performance of the proposed framework over several non-RL and kernel-based RL schemes.

This paper introduces an efficient multi-linear nonparametric (kernel-based) approximation framework for data regression and imputation. Data features are assumed to reside in or close to a smooth and userunknown manifold embedded in a reproducing kernel Hilbert space. Landmark points are identified to describe concisely the point cloud of features by linear approximating patches which mimic the concept of tangent spaces to smooth manifolds. The multi-linear model effects dimensionality reduction, enables efficient computations, and extracts data patterns and their geometry without any training data or additional information. Numerical tests on highly accelerated dynamic magnetic-resonance imaging (dMRI) data demonstrate remarkable improvements in efficiency and accuracy of the proposed approach over its predecessors and popular "shallow" data modeling methods, while offering substantial computational savings with regards to a deep-image-prior scheme.

This study addresses the problem of selecting dynamically, at each time instance, the "optimal" p-norm to combat outliers in linear adaptive filtering without any knowledge on the potentially time-varying probability density function of the outliers. To this end, an online and data-driven framework is designed via kernel-based reinforcement learning (KBRL). Novel Bellman mappings on reproducing kernel Hilbert spaces (RKHSs) are introduced that need no knowledge on transition probabilities of Markov decision processes, and are nonexpansive with respect to the underlying Hilbertian norm. An approximate policy-iteration framework is finally offered via the introduction of a finite-dimensional affine superset of the fixed-point set of the proposed Bellman mappings. The well-known "curse of dimensionality" in RKHSs is addressed by building a basis of vectors via an approximate linear dependency criterion. Numerical tests on synthetic data demonstrate that the proposed framework selects the "optimal" p-norm for the outlier scenario at hand at every time instance, outperforming several non-RL and KBRL schemes.

This paper introduces a non-parametric learning framework to combat outliers in online, multi-output, and nonlinear regression tasks. A hierarchical-optimization problem underpins the learning task: Search in a reproducing kernel Hilbert space (RKHS) for a function that minimizes a sample average ℓp-norm (1 ≤ p ≤ 2) error loss defined on data contaminated by noise and outliers, under affine constraints defined as the set of minimizers of a quadratic loss on a finite number of faithful data devoid of noise and outliers (side information). To surmount the computational obstacles inflicted by the choice of loss and the potentially infinite dimensional RKHS, approximations of the ℓp-norm loss, as well as a novel twist of the criterion of approximate linear dependency are devised to keep the computational-complexity footprint of the proposed algorithm bounded over time. Numerical tests on datasets showcase the robust behavior of the advocated framework against different types of outliers, under a low computational load, while satisfying at the same time the affine constraints, in contrast to the state-of-the-art methods which are constraint agnostic.

This work exploits Riemannian manifolds to introduce a geometric framework for online state and community classification in dynamic multilayer networks where nodes are annotated with time series. A bottom-up approach is followed, starting from the extraction of Riemannian features from nodal time series, and reaching up to on- line/sequential classification of features via geodesic distances and angular information in the tangent spaces of a Riemannian manifold. As a case study, features in the Grassmann manifold are generated by fitting a kernel autoregressive-moving-average model to the nodal time series of the multilayer network. The paper highlights also numerical tests on synthetic and real brain-network data, where it is shown that the proposed geometric framework outperforms state-of- the-art deep-learning models in classification accuracy, especially in cases where the number of training data is small with respect to the number of the testing ones.

This letter fortifies the recently introduced hierarchical-optimization recursive least squares (HO-RLS) against outliers which contaminate infrequently linear-regression models. Outliers are modeled as nuisance variables and are estimated together with the linear filter/system variables via a sparsity-inducing (non-)convexly regularized least-squares task. The proposed outlier-robust HO-RLS builds on steepest-descent directions with a constant step size (learning rate), needs no matrix inversion (lemma), accommodates colored nominal noise of known correlation matrix, exhibits small computational footprint, and offers theoretical guarantees, in a probabilistic sense, for the convergence of the system estimates to the solutions of a hierarchical-optimization problem: Minimize a convex loss, which models a-priori knowledge about the unknown system, over the minimizers of the classical ensemble LS loss. Extensive numerical tests on synthetically generated data in both stationary and non-stationary scenarios showcase notable improvements of the proposed scheme over state-of-the-art techniques.

This paper puts forth a novel bi-linear modeling framework for data recovery via manifold-learning and sparse-approximation arguments and considers its application to dynamic magnetic-resonance imaging (dMRI). Each temporal-domain MR image is viewed as a point that lies onto or close to a smooth manifold, and landmark points are identified to describe the point cloud concisely. To facilitate computations, a dimensionality reduction module generates low-dimensional/compressed renditions of the landmark points. Recovery of high-fidelity MRI data is realized by solving a non-convex minimization task for the linear decompression operator and affine combinations of landmark points which locally approximate the latent manifold geometry. An algorithm with guaranteed convergence to stationary solutions of the non-convex minimization task is also provided. The aforementioned framework exploits the underlying spatio-temporal patterns and geometry of the acquired data without any prior training on external data or information. Extensive numerical results on simulated as well as real cardiac-cine MRI data illustrate noteworthy improvements of the advocated machine-learning framework over state-of-the-art reconstruction techniques.

Quadratically constrained quadratic programming (QCQP) forms an important class of optimization tasks in various engineering disciplines. Fast identification of a feasible point under low computational complexity load is critical for several approximation techniques which have been developed to solve non-convex QCQPs. This paper introduces two projection-based techniques to compute feasible points of non-convex QCQPs with low computational complexity footprints: The first one employs successive projection mappings, while the second one builds on a composition of successive and averaged projection steps. Extensive experiments on synthetically generated instances of non-convex quadratically constrained feasibility problems demonstrate that the simple successive-projection based technique compares favorably against state-of-the-art feasible point pursuit methods which capitalize on successive convex approximation, parallel projections and computationally demanding interior-point techniques.

The hybrid steepest descent method (HSDM) [Yamada, '01] was introduced as a low-computational complexity tool for solving convex variational-inequality problems over the fixed-point set of non-expansive mappings in Hilbert spaces. Motivated by results on decentralized optimization, this study introduces an HSDM variant that extends, for the first time, the applicability of HSDM to affinely constrained composite convex minimization tasks over Euclidean spaces; the same class of problems solved by the popular alternating direction method of multipliers and primal-dual methods. The proposed scheme shows desirable attributes for large-scale optimization tasks that have not been met, partly or all-together, in any other member of the HSDM family of algorithms: tunable computational complexity, a step-size parameter which stays constant over recursions, promoting thus acceleration of convergence, no boundedness constraints on iterates and/or gradients, and the ability to deal with convex losses which comprise a smooth and a non-smooth part, where the smooth part is only required to have a Lipschitz-continuous derivative. Convergence guarantees and rates are established. Numerical tests on synthetic data and on colored-image inpainting underline the rich potential of the proposed scheme for large-scale optimization tasks.

Partial correlations (PCs) of functional magnetic resonance imaging (fMRI) time series play a principal role in revealing connectivity of brain networks. To explore nonlinear behavior of the blood-oxygen-level dependent signal, the present work postulates a kernel-based nonlinear connectivity model based on which it obtains topology revealing PCs. Instead of relying on a single predefined kernel, a data-driven approach is advocated to learn the combination of multiple kernel functions that optimizes the data fit. Synthetically generated data based on both a dynamic causal and a linear model are used to validate the proposed approach in resting-state fMRI scenarios, highlighting the gains in edge detection performance when compared with the popular linear PC method. Tests on real fMRI data demonstrate that connectivity patterns revealed by linear and nonlinear models are different.

This paper advocates a novel framework for segmenting a dataset on a Riemannian manifold M into clusters lying around low-dimensional submanifolds of M. Important examples of M, for which the proposed algorithm is computationally efficient, include the sphere, the set of positive definite matrices, and the Grassmannian. The proposed algorithm constructs a data-affinity matrix by thoroughly exploiting the intrinsic geometry and then applies spectral clustering. Local geometry is encoded by sparse coding and directional information of local tangent spaces and geodesics, which is important in resolving intersecting clusters and establishing the theoretical guarantees for a simplified variant of the algorithm. To avoid complication, these guarantees assume that the underlying submanifolds are geodesic. Extensive validation on synthetic and real data demonstrates the resiliency of the proposed method against deviations from the theoretical (geodesic) model as well as its superior performance over state-of-the-art techniques.

Applications involving large-scale dictionary learning tasks motivate well online optimization algorithms for generally non-convex and non-smooth problems. In this big data context, the present paper develops an online learning framework by jointly leveraging the stochastic approximation paradigm with first-order acceleration schemes. The generally non-convex objective evaluated online at the resultant iterates enjoys quadratic rate of convergence. The generality of the novel approach is demonstrated in two online learning applications: (i) Online linear regression using the total least-squares approach; and, (ii) a semi-supervised dictionary learning approach to network-wide link load tracking and imputation of real data with missing entries. In both cases, numerical tests highlight the potential of the proposed online framework for big data network analytics.

The present paper introduces a novel online asset allocation strategy which accounts for the sensitivity of Markowitz-inspired portfolios to low-quality estimates of the mean and the correlation matrix of stock returns. The proposed methodology builds upon the total least-squares (TLS) criterion regularized with sparsity attributes, and the ability to incorporate additional convex constraints on the portfolio vector. To solve such an optimization task, the present paper draws from the rich family of splitting algorithms to construct a novel online splitting algorithm with computational complexity that scales linearly with the number of unknowns. Real-world financial data are utilized to demonstrate the potential of the proposed technique.

This paper deals with a novel class of set-theoretic adaptive sparsity promoting algorithms of linear computational complexity. Sparsity is induced via generalized thresholding operators, which correspond to nonconvex penalties such as those used in a number of sparse LMS based schemes. The results demonstrate the significant performance gain of our approach, at comparable computational cost.

This letter establishes a novel analysis of the Adaptive Projected Subgradient Method (APSM) in the intersection of the stochastic and robust estimation paradigms. Utilizing classical worst-case bounds on the noise process, drawn from the robust estimation methodology, the present study demonstrates that the hyperslab-inspired version of the APSM generates a sequence of estimates which converges to a point located, with probability one, arbitrarily close to the estimand. Numerical tests and comparisons with classical time-adaptive algorithms corroborate the theoretical findings of the study.

In this paper, a novel scheme for online, sparsity-aware learning is presented. A new theory is developed that allows for the incorporation, in a unifying way, of different thresholding rules to promote sparsity, that may even be of a nonconvex nature. The complexity of the algorithm exhibits a linear dependence on the number of free parameters.

This paper presents a wide framework for non-linear online supervised learning tasks in the context of complex valued signal processing. The (complex) input data are mapped into a complex reproducing kernel Hilbert space (RKHS), where the learning phase is taking place. Both pure complex kernels and real kernels (via the complexification trick) can be employed. Moreover, any convex, continuous and not necessarily differentiable function can be used to measure the loss between the output of the specific system and the desired response. The only requirement is the subgradient of the adopted loss function to be available in an analytic form. In order to derive analytically the subgradients, the principles of the (recently developed) Wirtinger's calculus in complex RKHS are exploited. Furthermore, both linear and widely linear (in RKHS) estimation filters are considered. To cope with the problem of increasing memory requirements, which is present in almost all online schemes in RKHS, the sparsification scheme, based on projection onto closed balls, has been adopted. We demonstrate the effectiveness of the proposed framework in a non-linear channel identification task, a non-linear channel equalization problem and a quadrature phase shift keying equalization scheme, using both circular and non circular synthetic signal sources.

This paper introduces a wide framework for online, i.e., time-adaptive, supervised multiregression tasks. The problem is formulated in a general infinite-dimensional reproducing kernel Hilbert space (RKHS). In this context, a fairly large number of nonlinear multiregression models fall as special cases, including the linear case. Any convex, continuous, and not necessarily differentiable function can be used as a loss function in order to quantify the disagreement between the output of the system and the desired response. The only requirement is the subgradient of the adopted loss function to be available in an analytic form. To this end, we demonstrate a way to calculate the subgradients of robust loss functions, suitable for the multiregression task. As it is by now well documented, when dealing with online schemes in RKHS, the memory keeps increasing with each iteration step. To attack this problem, a simple sparsification strategy is utilized, which leads to an algorithmic scheme of linear complexity with respect to the number of unknown parameters. A convergence analysis of the technique, based on arguments of convex analysis, is also provided. To demonstrate the capacity of the proposed method, the multiregressor is applied to the multiaccess multiple-input multiple-output channel equalization task for a setting with poor resources and nonavailable channel information. Numerical results verify the potential of the method, when its performance is compared with those of the state-of-the-art linear techniques, which, in contrast, use space-time coding, more antenna elements, as well as full channel information.

In this paper, a novel algorithm for bandwidth reduction in adaptive distributed learning is introduced. We deal with diffusion net works, in which the nodes cooperate with each other, by exchanging information, in order to estimate an unknown parameter vector of interest. We seek for solutions in the framework of set theoretic estimation. Moreover, in order to reduce the required bandwidth by the transmitted information, which is dictated by the dimension of the unknown vector, we choose to project and work in a lower dimension Krylov subspace. This provides the benefit of trading off dimensionality with accuracy. Full convergence properties are presented, and experiments, within the system identification task, demonstrate the robustness of the algorithmic technique.

This paper presents a novel time-adaptive estimation technique by revisiting the classical Wiener-Hopf equation. Any convex and not necessarily differentiable function can be used for enlarging the Wiener-Hopf equation in order to incorporate the often met, in practice, measurement and model inaccuracies. Unlike classical techniques, e.g., the Recursive Least Squares (RLS) algorithm, the proposed method is free of the computation of the inverse of a correlation matrix. Moreover, the method offers the means for dealing with the presence of convex constraints in an efficient way, by exploiting general convex analytic tools. To validate the pro posed estimation method, an application of increasing importance nowadays, the online sparse signal recovery task is considered. Numerical results support the introduced theoretical arguments against the sparsity-aware classical batch, and the very recently introduced RLS-based signal recovery techniques.

This paper presents a novel projection-based adaptive algorithm for sparse system identification. Sequentially observed data are used to generate an equivalent number of closed convex sets, namely hyperslabs, which quantify an associated cost criterion. Sparsity is exploited by the introduction of appropriately designed weighted ℓ1balls. The algorithm uses only projections onto hyperslabs and weighted ℓ1balls, and results into a computational complexity of order O(L) multiplications/additions and O(Llog2L) sorting operations, where L is the length of the system to be estimated. Numerical results are also given to validate the proposed method against very recently developed sparse LMS and RLS type of algorithms, which are considered to belong to the same type of algorithmic family.

We propose multi-domain adaptive filtering based on the idea of feasibility splitting - dealing with feasibility in individual domains. The proposed approach provides a useful and mathematically rigorous framework to incorporate multiple pieces of information (expressed in different domains) efficiently. Indeed, it processes such multiple pieces of information by means of the metric projection in each individual domain; this is a significant advantage over existing single-domain approaches. Also we provide a reasonable strategy to treat the case where the available prior information is inconsistent. A convergence analysis and numerical examples are presented to support the proposed method.

The goal of this paper is the development of a novel approach for the problem of Noise Removal, based on the theory of Reproducing Kernels Hilbert Spaces (RKHS). The problem is cast as an optimization task in a RKHS, by taking advantage of the celebrated semi parametric Representer Theorem. Examples verify that in the presence of gaussian noise the proposed method performs relatively well compared to wavelet based techniques and outperforms them significantly in the presence of impulse or mixed noise.

The main contribution of this paper is the development of a novel approach, based on the theory of Reproducing Kernel Hilbert Spaces (RKHS), for the problem of noise removal in the spatial domain. The proposed methodology has the advantage that it is able to remove any kind of additive noise (impulse, gaussian, uniform, etc.) from any digital image, in contrast to the most commonly used denoising techniques, which are noise dependent. The problem is cast as an optimization task in a RKHS, by taking advantage of the celebrated Representer Theorem in its semi-parametric formulation. The semi-parametric formulation, although known in theory, has so far found limited, to our knowledge, application. However, in the image denoising problem, its use is dictated by the nature of the problem itself. The need for edge preservation naturally leads to such a modeling. Examples verify that in the presence of gaussian noise the proposed methodology performs well compared to wavelet based technics and outperforms them significantly in the presence of impulse or mixed noise.

This paper presents a novel method for incorporating a-priori affine constraints in online kernel-based learning tasks. The proposed technique elaborates the generic tool of projections to form a sequence of estimates in reproducing kernel Hilbert spaces (RKHS). The method guarantees that the whole sequence of estimates lies in the given affine constraint set. To validate the algorithm, a beamforming task is considered. The numerical results show that the proposed frame provides with solutions in cases where the classical linear approach collapses, and forms proper beam-patterns as opposed to a recent unconstrained kernel-based regression method.

Very recently, an adaptive projection algorithm was introduced for the online classification task with sparsification in reproducing kernel Hilbert spaces (RKHS). This paper presents another sparsification method for the projection-based approach by generating a sequence of linear subspaces in RKHS. Projection mappings give a geometrical flavor to the design; classification is performed by metric projection mappings, sparsification is achieved by orthogonal projections, while the online system's memory and tracking requirements are attained by oblique projections. The resulting sparsification scheme shows strong similarities with the classical sliding window adaptive schemes. Validation is performed by considering the adaptive equalization problem of a nonlinear communication channel. Although here the classification scheme is considered, the method is readily extended to regression tasks. Furthermore its generality allows for a number of cost functions including non-differentiable ones.

The goal of this paper is the development of a novel efficient online kernel-based algorithm for classification. The spirit of the algorithm stems from the recently introduced adaptive projected subgradient method. This is a general convex analytic tool that employs projections onto a sequence of convex sets and it can be considered as a generalization of the celebrated APA algorithm, widely used in classical adaptive filtering.

This paper indicates that an appropriate design of metric leads to significant improvements in the adaptive projected subgradient method (APSM), which unifies a wide range of projection-based algorithms [including normalized least mean square (NLMS) and affine projection algorithm (APA)]. The key is to incorporate a priori (or a posteriori) information on characteristics of an estimandum, a system to be estimated, into the metric design. We propose a family of efficient adaptive filtering algorithms based on a parallel use of quadratic-metric projection, which assigns every point to the nearest point in a closed convex set in a quadratic-metric sense. We present two versions: (1) constant-metric and (2) variable-metric, i.e., the metric function employed is (1) constant and (2) variable among iterations. As a constant-metric version, adaptive parallel quadratic-metric projection (APQP) and adaptive parallel min-max quadratic-metric projection (APMQP) algorithms are naturally derived by APSM, being endowed with desirable properties such as convergence to a point optimal in asymptotic sense. As a variable-metric version, adaptive parallel variable-metric projection (APVP) algorithm is derived by a generalized APSM, enjoying an extended monotone property at each iteration. By employing a simple quadratic-metric, the computational complexity of the proposed algorithms is kept linear with respect to the filter length. Numerical examples demonstrate the remarkable advantages of the proposed algorithms in an application to acoustic echo cancellation.

It is well-known that the Capon beamformer is sensitive to array steering vector errors and may result into a worse performance than classical data-independent beamformers. This paper follows a different path from the well-established diagonal loading techniques and designs a robust Capon beamformer by a recent extension of the adaptive projected subgradient method. The proposed method marks a computational complexity of O(N2), where N is the number of array elements. The simulation results show that the proposed beamformer achieves excellent performance especially in cases where the diagonal loading techniques face difficulties, i.e. in cases where the interference to noise ratio (INR) is moderately larger than SNR

The adaptive projected subgradient method offers a unified mathematical perspective for the adaptive (set-membership/set-theoretic) filtering schemes. In this paper, we introduce an overview of its recent theoretical advances and successful applications to robust signal processing problems including the stereo acoustic echo canceling, the MAI suppression in DS/CDMA receivers, and the robust adaptive beamforming with array antenna systems

It is well-known that the unbiased estimate of the covariance matrix of a real vector wide sense stationary process is not necessarily positive semidefinite. By defining the real Hilbert space of all symmetric matrices, the conditions for a symmetric matrix to be positive definite, block Toeplitz, as well as to satisfy other design constraints, are formed as closed convex sets. This paper demonstrates that the problem of approximating the unbiased estimate of the covariance matrix of a real vector wide sense stationary process over the intersection of those closed convex sets in an optimal way can be resolved by the Hybrid Steepest Descent Method. An optimal solution is also provided even when inconsistent constraints are met, i.e., whenever the intersection of the closed convex sets is empty. The numerical results exhibit significant improvement of the proposed method over the standard estimates of the covariance matrix.

This paper presents a novel robust adaptive filtering scheme based on the interactive use of statistical noise information and an extension of the ideas developed originally for efficient algorithmic solutions to the convex feasibility problems. The statistical noise information is quantitatively formulated as stochastic property closed convex sets by the simple design formulae developed. The proposed adaptive algorithm is computationally efficient and robust to noise because it requires only an iterative parallel projection onto a series of closed half spaces highly expected to contain the unknown system to be identified. The numerical examples show that the proposed adaptive filtering scheme achieves low estimation error and realizes dramatically fast and stable convergence even for highly colored excited input signals in severely noisy situations.