Data silos, mainly caused by privacy and interoperability, significantly constrain collaborations among different organizations with similar data for the same purpose. Distributed learning based on divide-and-conquer provides a promising way to settle the data silos, but it suffers from several challenges, including autonomy, privacy guarantees, and the necessity of collaborations. This paper focuses on developing an adaptive distributed kernel ridge regression (AdaDKRR) by taking autonomy in parameter selection, privacy in communicating non-sensitive information, and the necessity of collaborations for performance improvement into account. We provide both solid theoretical verifications and comprehensive experiments for AdaDKRR to demonstrate its feasibility and effectiveness. Theoretically, we prove that under some mild conditions, AdaDKRR performs similarly to running the optimal learning algorithms on the whole data, verifying the necessity of collaborations and showing that no other distributed learning scheme can essentially beat AdaDKRR under the same conditions. Numerically, we test AdaDKRR on both toy simulations and two real-world applications to show that AdaDKRR is superior to other existing distributed learning schemes. All these results show that AdaDKRR is a feasible scheme to overcome data silos, which are highly desired in numerous application regions such as intelligent decision-making, pricing forecasting, and performance prediction for products.

Although robust tensor completion has been extensively studied, the effect of incorporating side information has not been explored. In this article, we fill this gap by developing a novel high-order robust tensor completion model that incorporates both latent and explicit side information. We base our model on the transformed t-product because the corresponding tensor tubal rank can characterize the inherent low-rank structure of a tensor. We study the effect of side information on sample complexity and prove that our model needs fewer observations than other tensor recovery methods when side information is perfect. This theoretically shows that informative side information is beneficial for learning. Extensive experimental results on synthetic and real data further demonstrate the superiority of the proposed method over several popular alternatives. In particular, we evaluate the performance of our solution based on two important applications, namely, link prediction in signed networks and rating prediction in recommender systems. We show that the proposed model, which manages to exploit side information in learning, outperforms other methods in the learning of such low-rank tensor data. Furthermore, when dealing with varying dimensions, we also design an online robust tensor completion with side information algorithm and validate its effectiveness using a real-world traffic dataset in the supplementary material. The source code is available athttps://github.com/yyyancy/RTCF.

Abstract. Spherical radial-basis-based kernel interpolation abounds in image sciences, including geophysical image reconstruction, climate trends description, and image rendering, due to its excellent spatial localization property and perfect approximation performance. However, in dealing with noisy data, kernel interpolation frequently behaves not so well due to the large condition number of the kernel matrix and instability of the interpolation process. In this paper, we introduce a weighted spectral filter approach to reduce the condition number of the kernel matrix and then stabilize kernel interpolation. The main building blocks of the proposed method are the well-developed spherical positive quadrature rules and high-pass spectral filters. Using a recently developed integral operator approach for spherical data analysis, we theoretically demonstrate that the proposed weighted spectral filter approach succeeds in breaking through the bottleneck of kernel interpolation, especially in fitting noisy data. We provide optimal approximation rates of the new method to show that our approach does not compromise the predicting accuracy. Furthermore, we conduct both toy simulations and two real-world data experiments with synthetically added noise in geophysical image reconstruction and climate image processing to verify our theoretical assertions and show the feasibility of the weighted spectral filter approach.

In this paper, we present a general optimization framework that leverages structured sparsity to achieve superior recovery results. The traditional method for solving the structured sparse objectives based on$\ell _{2,0}$-norm is to use the$\ell _{2,1}$-norm as a convex surrogate. However, such an approximation often yields a large performance gap. To tackle this issue, we first provide a framework that allows for a wide range of surrogate functions (including non-convex surrogates), which exhibits better performance in harnessing structured sparsity. Moreover, we develop a fixed point algorithm that solves a key underlying non-convex structured sparse recovery optimization problem to global optimality with a guaranteed super-linear convergence rate. Building on this, we consider three specific applications, i.e., outlier pursuit, supervised feature selection, and structured dictionary learning, which can benefit from the proposed structured sparsity optimization framework. In each application, how the optimization problem can be formulated and thus be relaxed under a generic surrogate function is explained in detail. We conduct extensive experiments on both synthetic and real-world data and demonstrate the effectiveness and efficiency of the proposed framework.

Training an interpretable deep net to embody its theoretical advantages is difficult but extremely important in the community of machine learning. In this article, noticing the importance of spatial sparseness in signal and image processing, we develop a constructive approach to generate a deep net to capture the spatial sparseness feature. We conduct both theoretical analysis and numerical verifications to show the power of the constructive approach. Theoretically, we prove that the constructive approach can yield a deep net estimate that achieves the optimal generalization error bounds in the framework of learning theory. Numerically, we show that the constructive approach is essentially better than shallow learning in the sense that it provides better prediction accuracy with less training time.

Ensemble learning methods, such as boosting, focus on producing a strong classifier based on numerous weak classifiers. In this paper, we develop a novel ensemble learning method called rescaled boosting with truncation (ReBooT) for binary classification by combining well-known rescaling and regularization ideas in boosting. Theoretically, we present some sufficient conditions for the convergence of ReBooT, derive an almost optimal numerical convergence rate, and deduce fast-learning rates in the framework of statistical learning theory. Experimentally, we conduct both toy simulations and four real-world data runs to show the power of ReBooT. Our results show that, compared with the existing boosting algorithms, ReBooT possesses better learning performance and interpretability in terms of solid theoretical guarantees, perfect structure constraints, and good prediction performance. History: Accepted by Ram Ramesh, Area Editor for Data Science & Machine Learning Funding: This work was supported by the National Natural Science Foundation of China [Grants 11971374, 61772374, and 61876133]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijoc.2022.1224 .

Low-rank tensor recovery in the presence of sparse but arbitrary errors is an important problem with many practical applications. In this work, we propose a general framework that recovers low-rank tensors, in which the data can be deformed by some unknown transformations and corrupted by arbitrary sparse errors. We give a unified presentation of the surrogate-based formulations that incorporate the features of rectification and alignment simultaneously, and establish worst-case error bounds of the recovered tensor. In this context, the state-of-the-art methods 'RASL' and 'TILT' can be viewed as two special cases of our work, and yet each only performs part of the function of our method. Subsequently, we study the optimization aspects of the problem in detail by deriving two algorithms, one based on the alternating direction method of multipliers (ADMM) and the other based on proximal gradient. We provide convergence guarantees for the latter algorithm, and demonstrate the performance of the former through in-depth simulations. Finally, we present extensive experimental results on public datasets to demonstrate the effectiveness and efficiency of the proposed framework and algorithms.

Training neural networks is recently a hot topic in machine learning due to its great success in many applications. Since the neural networks' training usually involves a highly nonconvex optimization problem, it is difficult to design optimization algorithms with perfect convergence guarantees to derive a neural network estimator of high quality. In this article, we borrow the well-known random sketching strategy from kernel methods to transform the training of shallow rectified linear unit (ReLU) nets into a linear least-squares problem. Using the localized approximation property of shallow ReLU nets and a recently developed dimensionality-leveraging scheme, we succeed in equipping shallow ReLU nets with a specific random sketching scheme. The efficiency of the suggested random sketching strategy is guaranteed by theoretical analysis and also verified via a series of numerical experiments. Theoretically, we show that the proposed random sketching is almost optimal in terms of both approximation capability and learning performance. This implies that random sketching does not degenerate the performance of shallow ReLU nets. Numerically, we show that random sketching can significantly reduce the computational burden of numerous backpropagation (BP) algorithms while maintaining their learning performance.

Feature selection (FS), which identifies the relevant features in a data set to facilitate subsequent data analysis, is a fundamental problem in machine learning and has been widely studied in recent years. Most FS methods rank the features in order of their scores based on a specific criterion and then select the k top-ranked features, where k is the number of desired features. However, these features are usually not the top- k features and may present a suboptimal choice. To address this issue, we propose a novel FS framework in this article to select the exact top- k features in the unsupervised, semisupervised, and supervised scenarios. The new framework utilizes thel0,2-norm as the matrix sparsity constraint rather than its relaxations, such as thel1,2-norm. Since thel0,2-norm constrained problem is difficult to solve, we transform the discretel0,2-norm-based constraint into an equivalent 0-1 integer constraint and replace the 0-1 integer constraint with two continuous constraints. The obtained top- k FS framework with two continuous constraints is theoretically equivalent to thel0,2-norm constrained problem and can be optimized by the alternating direction method of multipliers (ADMM). Unsupervised and semisupervised FS methods are developed based on the proposed framework, and extensive experiments on real-world data sets are conducted to demonstrate the effectiveness of the proposed FS framework.

Sparse representation and dictionary learning have been successfully applied to encode dense data and facilitate image classification. Though existing dictionary learning methods achieve better performance than their counterparts, the class discriminative ability of learned dictionary is still limited. This paper proposes a novel supervised dictionary learning method based on the prior of the block diagonal phenomenon, i.e., each sample should be well reconstructed by the samples in the same class while poorly reconstructed by the samples in other class. Specifically, a block diagonal regularizer is imposed on the affinity matrix to enforce the sparse representation matrix to have an approximately block diagonal structure, which makes the learned dictionary more discriminative and suitable for classification tasks. Furthermore, we present an effective optimization strategy by combining the alternating minimization with the alternating direction method of multipliers (ADMM) for the proposed framework. Experimental results on six real-world datasets show that the proposed method is more effective than state-of-the-art dictionary learning methods.

This paper focuses on generalization performance analysis for distributed algorithms in the framework of learning theory. Taking distributed kernel ridge regression (DKRR) for example, we succeed in deriving its optimal learning rates in expectation and providing theoretically optimal ranges of the number of local processors. Due to the gap between theory and experiments, we also deduce optimal learning rates for DKRR in probability to essentially reflect the generalization performance and limitations of DKRR. Furthermore, we propose a communication strategy to improve the learning performance of DKRR and demonstrate the power of communications in DKRR via both theoretical assessments and numerical experiments.

Exemplar-based image denoising algorithms have shown great potential for image restoration with a multitude of existing models. In this paper, we interpret nonlocal similar patch-based denoising as a problem of low-rank recovery. This offers a physically plausible model and unifies several existing techniques in a single low-rank recovery framework. The framework can handle complex noise models, such as zero-mean Gaussian noise, impulse noise, and any other noise that can be approximated by mixing these two kinds of noise. Moreover, we introduce a new nonconvex surrogate for the $l_{0}$ -norm and find the optimal solution of the optimization problems when the new norm is applied to low-rank recovery. The experimental results with different kinds of noise confirm the effectiveness of the proposed low-rank recovery framework and the new norm.

This paper investigates classification by dictionary learning. A novel unified framework termed self-taught semisupervised dictionary learning with nonnegative constraint is proposed for simultaneously optimizing the components of a dictionary and a graph Laplacian. Specifically, an atom graph Laplacian regularization is built by using sparse coefficients to effectively capture the underlying manifold structure. It is more robust to noisy samples and outliers because atoms are more concise and representative than training samples. A nonnegative constraint imposed on the sparse coefficients guarantees that each sample is in the middle of its related atoms. In this way, the dependency between samples and atoms is made explicit. Furthermore, a self-taught mechanism is introduced to effectively feed back the manifold structure induced by atom graph Laplacian regularization and the supervised information hidden in unlabeled samples in order to learn a better dictionary. An efficient algorithm, combining a block coordinate descent method with the alternating direction method of multipliers, is derived to optimize the unified framework. Experimental results on several benchmark datasets show the effectiveness of the proposed model.

In this paper, we focus on the multi-view clustering problem. A novel multi-view clustering framework, called multi-view subspace clustering based on tensor Schatten-p norm (MVSC-TSP), is proposed for clustering task. In our method, the tensor Schatten-p norm, which is based on tensor singular value decomposition, is utilized to explore the global low-rank structure of multi-view self-representations. Since 0<; p<; 1, using tensor Schatten-p norm to relax the tensor multi-rank is more effective than the commonly used tensor nuclear norm. Furthermore, we present a new generalized tensor soft thresholding algorithm to solve the tensor Schatten-p norm minimization problem. Based on this, the proposed non-convex optimal problem can be efficiently solved by the alternating direction method of multipliers. Experimental results on image clustering demonstrate that the proposed method is superior to the state-of-the-art methods in term of various evaluation metrics.

Recently, group sparse representation which is based on a hypothesis about correlation of coefficient variables has attracted much attention due to its effectiveness and robustness in dictionary learning. Traditional group sparse representation methods use ℓ2,1norm to enforce the estimation of models with joint sparsity patterns, which often leads to over-punishment phenomenon. To solve this issue, we replace ℓ2,1with non-convex surrogate of ℓ2,1, and give a general solver for the corresponding optimization algorithm. Experimental results confirm the effectiveness of our proposed method.

In this paper, we propose a novel unified optimization framework for semi-supervised dictionary learning, which optimizes a graph Laplacian component and the dictionary simultaneously. In the framework, the graph Laplacian is defined on the atoms and the corresponding sparse codings. Since the atoms are more concise and representative than the original training samples, the constructed graph Laplacian can not only effectively capture the manifold structure of training samples, but also be more robust to noise and outliers. Moreover, the dictionary and the graph Laplacian can facilitate each other during the learning iterations. We derive an efficient algorithm by combining the block coordinate descent method with the alternating direction method of multipliers to solve the unified optimization problem. Extensive experimental evaluation on several challenging datasets demonstrates the superior performance of the proposed method.

Graph based Semi-Supervised Learning (G-SSL) methods usually include the stages of the construction of affinity matrix and the mechanism of inferring unknown labels. However, solving each of the stages individually does not fully exploit the potential relationship between the affinity matrix and the labels of samples. In this paper, we formulate the global self-expressiveness induced affinity and Laplacian Regularized Least Squares (LapRLS) into a single optimization model, called as Self-Taught LapRLS (ST-LapRLS). In the unified model, both the given labels and the estimated labels are used to build a better affinity matrix and to facilitate the LapRLS classifer. The proposed ST-LapRLS classifier is explicit, and can be easily extended to deal with out-of-sample problem. We propose an efficient algorithm which combines the alternating direction method of multiplier and LapRLS to solve the unified optimization problem. Experiments on several Benchmark datasets show the superior performance of our method in classification applications.

Recently, structured sparsity inducing based feature selection has become a hot topic in machine learning and pattern recognition. Most of the sparsity inducing feature selection methods are designed to rank all features by certain criterion and then select the k top ranked features, where k is an integer. However, the k top features are usually not the top k features and therefore maybe a suboptimal result. In this paper, we propose a novel supervised feature selection method to directly identify the top k features. The new method is formulated as a classic regularized least squares regression model with two groups of variables. The problem with respect to one group of the variables turn out to be a 0-1 integer programming, which had been considered very hard to solve. To address this, we utilize an efficient optimization method to solve the integer programming, which first replaces the discrete 0-1 constraints with two continuous constraints and then utilizes the alternating direction method of multipliers to optimize the equivalent problem. The obtained result is the top subset with k features under the proposed criterion rather than the subset of k top features. Experiments have been conducted on benchmark data sets to show the effectiveness of proposed method.

While recent techniques for discriminative dictionary learning have attained promising results on the classification tasks, their performance is highly dependent on the number of labeled samples available for training. However, labeling samples is expensive and time consuming due to the significant human effort involved. In this paper, we present a novel semi- supervised dictionary learning method which utilizes the structural sparse relationships between the labeled and unlabeled samples. Specifically, by connecting the sparse reconstruction coefficients on both the original samples and dictionary, the unlabeled samples can be automatically grouped to the different labeled samples, and the grouped samples share a small number of atoms in the dictionary via mixed l2p- norm regularization. This makes the learned dictionary more representative and discriminative since the shared atoms are learned by using the labeled and unlabeled samples potentially from the same class. Minimizing the derived objective function is a challenging task because it is non-convex and highly non-smooth. We propose an efficient optimization algorithm to solve the problem based on the block coordinate descent method. Moreover, we have a rigorous proof of the convergence of the algorithm. Extensive experiments are presented to show the superior performance of our method in classification applications.

In this paper, we propose a block-polynomial mapping for image feature learning, which can be efficiently represented by the matrix Khatri-Rao product. The block-polynomial mapping not only captures the local discriminative information within the image structure, but is also much more efficient than the traditional kernel mapping. Moreover, we embed the proposed mapping into the manifold regularization framework for semi-supervised image classification. Experimental results demonstrate that, while maintaining a comparable classification accuracy, the proposed algorithm performs much more efficient than the state-of-the-art methods.

In this paper, we study the low-rank tensor completion problem, where a high-order tensor with missing entries is given and the goal is to complete the tensor. We propose to minimize a new convex objective function, based on log sum of exponentials of nuclear norms, that promotes the low-rankness of unfolding matrices of the completed tensor. We show for the first time that the proximal operator to this objective function is readily computable through a hybrid singular value thresholding scheme. This leads to a new solution to high-order (low-rank) tensor completion via convex relaxation. We show that this convex relaxation and the resulting solution are much more effective than existing tensor completion methods (including those also based on minimizing ranks of unfolding matrices). The hybrid singular value thresholding scheme can be applied to any problem where the goal is to minimize the maximum rank of a set of low-rank matrices.

In this work, we propose a general method for recovering low-rank three-order tensors, in which the data can be deformed by some unknown transformation and corrupted by arbitrary sparse errors. Since the unfolding matrices of a tensor are interdependent, we introduce auxiliary variables and relax the hard equality constraints by the augmented Lagrange multiplier method. To improve the computational efficiency, we introduce a proximal gradient step to the alternating direction minimization method. We have provided proof for the convergence of the linearized version of the problem which is the inner loop of the overall algorithm. Both simulations and experiments show that our methods are more efficient and effective than previous work. The proposed method can be easily applied to simultaneously rectify and align multiple images or videos frames. In this context, the state-of-the-art algorithms RASL'' and "TILT'' can be viewed as two special cases of our work, and yet each only performs part of the function of our method."