Optimizing the performance of deep neural networks (DNNs) remains a significant challenge due to the sensitivity of models to both hyperparameter selection and weight initialization. Existing approaches typically address these two factors independently, which often leads to limiting adaptability and overall effectiveness. In this paper, we present a novel meta-learning framework that jointly recommends hyperparameters and initial weights by leveraging dataset similarity. Our method begins by extracting meta-features from a collection of historical datasets. For a given query dataset, similarity is computed based on distances in the meta-feature space, and the most similar historical datasets are used to recommend the underlying parameter configurations. To capture the diverse characteristics of image datasets, we introduce two complementary types of meta-features. The first, referred to as shallow or visible meta-features, comprises five groups of statistical measures that summarize color and texture information. The second, termed deep or invisible meta-features, consists of 512 descriptors extracted from a convolutional neural network pre-trained on ImageNet. We evaluated our framework in 105 real-world image classification tasks, using 75 datasets for historical modeling and 30 for querying. Experimental results with both vision transformers and convolutional neural networks demonstrate that our approach consistently outperforms state-of-the-art baselines, underscoring the effectiveness of dataset-driven parameter recommendation in deep learning.

Sequential Model-Based Optimization (SMBO) is a highly effective strategy for hyperparameter search in machine learning. It utilizes a surrogate model that fits previous trials and approximates the hyperparameter response surface (performance). This surrogate model primarily guides the decision-making process for selecting the next set of hyperparameters. Existing classic surrogates, such as Gaussian processes and random forests, focus solely on the current task of interest and cannot incorporate trials from historical tasks. This limitation hinders their efficacy in various applications. Inspired by the state-of-the-art convolutional neural process, this paper proposes a novel meta-learning-based surrogate model for efficient and effective hyperparameter optimization. Our surrogate is trained on the meta-knowledge from a range of historical tasks, enabling it to accurately predict the hyperparameter response surface even with a limited number of trials on a new task. We tested our approach on the hyperparameter selection problem for the well-known support vector machine (SVM), residual neural network (ResNet), and vision transformer (ViT) across hundreds of real-world classification datasets. The empirical results demonstrate its superiority over existing surrogate models, highlighting the effectiveness of meta-learning in hyperparameter optimization.

Hyperparameter recommendation through meta-learning (HPR-MtL) has proven effective in a wide range of studies. At its core, HPR-MtL constructs a recommendation model using metadata extracted from historical learning tasks, such as dataset characteristics and the empirical performance of hyperparameter configurations. Existing approaches-typically based on k-nearest neighbors (KNN), linear regression, or collaborative filtering-focus primarily on modeling interactions between datasets and hyperparameters, i.e., performance observations. However, they often overlook important interactions within datasets or within hyperparameter configurations, such as similarity relationships, which are critical components embedded in the metadata. To overcome this limitation, we propose a novel hyperparameter recommendation framework formulated as a link prediction problem on a bipartite graph. In our approach, each historical dataset and each hyperparameter configuration is represented as a node, and the links between them are weighted by the observed performance. When a new dataset becomes available, it is added as a node, and the model predicts its potential links to hyperparameter nodes. To enhance prediction accuracy, we introduce a graph convolutional network (GCN) that simultaneously captures both homogeneous interactions (within datasets and within hyperparameters) and heterogeneous interactions (between datasets and hyperparameters)-a capability not found in existing methods. We evaluated our method on two popular deep learning models, ResNet and Vision Transformer (ViT), across 105 real-world classification datasets. Extensive experiments using multiple evaluation metrics demonstrate the superior performance of our approach compared to state-of-the-art hyperparameter recommendation baselines.

In this paper, we present a novel non-convex tensor completion model specifically tailored for multidimensional data. Our approach introduces a three-directional non-convex tensor rank surrogate regularized by the Minimax Concave Penalty (MCP) function. Crucially, the method processes data by simultaneously exploiting low-rank structures across its three modal directions, with the MCP function effectively mitigating the over-penalization of large singular values-a common drawback in convex nuclear norm minimization. To address the inherent challenges of this non-convex optimization, we develop an innovative approximate convex model that accurately captures the original formulation's essence. We then develop a robust convex Alternating Direction Method of Multipliers (ADMM)-based algorithm, supported by a rigorous convergence guarantee, ensuring both theoretical soundness and practical reliability. Extensive experiments on a variety of real-world datasets demonstrate the superior performance and robustness of the proposed method compared to state-of-the-art approaches.

Hyperparameter recommendation via meta-learning has shown great promise in various studies. The main challenge for meta-learning is how to develop an effective meta-learner (learning algorithm) that can capture the intrinsic relationship between dataset characteristics and the empirical performance of hyperparameters. Existing meta-learners are mostly based on traditional machine-learning models that only learn data representations with a single layer, which are incapable of learning complex features from the data and often cannot capture those properties deeply embedded in data. To address this issue, in this article, we propose hyperparameter recommendation approaches by integrating the learning model with convolutional neural networks (CNNs). Specifically, we first formulate the recommendation task as a regression problem, where dataset characteristics are treated as predictors and the historical performance of hyperparameters as responses. We establish a CNN-based learning model with feature selection capability to serve as the regressor. We then develop a convolutional denoising autoencoder (ConvDAE) that can leverage the spatial structure of the entire hyperparameter performance space and evaluate the performance of hyperparameters via denoising when the performance of partial hyperparameters is available under the multidimensional framework. To make our approach being flexible in applications, we establish a comprehensive two-branch CNN model that can utilize both dataset characteristics and partial evaluations to make effective recommendations. We conduct extensive experiments on 400 real classification problems and the well-known SVM. Our proposed approaches outperform existing meta-learning baselines as well as various search algorithms, demonstrating the high effectiveness in hyperparameter recommendations via deep learning.

How to handle large multidimensional datasets, such as hyperspectral images and video information, efficiently and effectively plays a critical role in big-data processing. The characteristics of low-rank tensor decomposition in recent years demonstrate the essentials in describing the tensor rank, which often leads to promising approaches. However, most current tensor decomposition models consider the rank-1 component simply to be the vector outer product, which may not fully capture the correlated spatial information effectively for large-scale and high-order multidimensional datasets. In this article, we develop a new novel tensor decomposition model by extending it to the matrix outer product or called Bhattacharya-Mesner product, to form an effective dataset decomposition. The fundamental idea is to decompose tensors structurally in a compact manner as much as possible while retaining data spatial characteristics in a tractable way. By incorporating the framework of the Bayesian inference, a new tensor decomposition model on the subtle matrix unfolding outer product is established for both tensor completion and robust principal component analysis problems, including hyperspectral image completion and denoising, traffic data imputation, and video background subtraction. Numerical experiments on real-world datasets demonstrate the highly desirable effectiveness of the proposed approach.

The performance of classification algorithms is mainly governed by the hyperparameter settings deployed in applications, and the search for desirable hyperparameter configurations usually is quite challenging due to the complexity of datasets. Metafeatures are a group of measures that characterize the underlying dataset from various aspects, and the corresponding recommendation algorithm fully relies on the appropriate selection of metafeatures. Metalearning (MtL), aiming to improve the learning algorithm itself, requires development in integrating features, models, and algorithm learning to accomplish its goal. In this article, we develop a multivariate sparse-group Lasso (SGLasso) model embedded with MtL capacity in recommending suitable configurations via learning. The main idea is to select the principal metafeatures by removing those redundant or irregular ones, promoting both efficiency and performance in the hyperparameter configuration recommendation. To be specific, we first extract the metafeatures and classification performance of a set of configurations from the collection of historical datasets, and then, a metaregression task is established through SGLasso to capture the main characteristics of the underlying relationship between metafeatures and historical performance. For a new dataset, the classification performance of configurations can be estimated through the selected metafeatures so that the configuration with the highest predictive performance in terms of the new dataset can be generated. Furthermore, a general MtL architecture combined with our model is developed. Extensive experiments are conducted on 136 UCI datasets, demonstrating the effectiveness of the proposed approach. The empirical results on the well-known SVM show that our model can effectively recommend suitable configurations and outperform the existing MtL-based methods and the well-known search-based algorithms, such as random search, Bayesian optimization, and Hyperband.

The recent study on tensor singular value decomposition (t-SVD) that performs the Fourier transform on the tubes of a third-order tensor has gained promising performance on multidimensional data recovery problems. However, such a fixed transformation, e.g., discrete Fourier transform and discrete cosine transform, lacks being self-adapted to the change of different datasets, and thus, it is not flexible enough to exploit the low-rank and sparse property of the variety of multidimensional datasets. In this article, we consider a tube as an atom of a third-order tensor and construct a data-driven learning dictionary from the observed noisy data along the tubes of the given tensor. Then, a Bayesian dictionary learning (DL) model with tensor tubal transformed factorization, aiming to identify the underlying low-tubal-rank structure of the tensor effectively via the data-adaptive dictionary, is developed to solve the tensor robust principal component analysis (TRPCA) problem. With the defined pagewise tensor operators, a variational Bayesian DL algorithm is established and updates the posterior distributions instantaneously along the third dimension to solve the TPRCA. Extensive experiments on real-world applications, such as color image and hyperspectral image denoising and background/foreground separation problems, demonstrate both effectiveness and efficiency of the proposed approach in terms of various standard metrics.

Hyperparameter optimization (HPO), characterized by hyperparameter tuning, is not only a critical step for effective modeling but also is the most time-consuming process in machine learning. Traditional search-based algorithms tend to require extensive configuration evaluations for each round to select the desirable hyperparameters during the process, and they are often very inefficient for the implementations on large-scale tasks. In this paper, we study the HPO problem via meta-learning (MtL) approach under the low-rank tensor completion (LRTC) framework. Our proposed approach predicts the performance for hyperparameters of new problems based on their previous performance so that the underlying suitable hyperparameters with better efficiency can be attained. Different from existing approaches, the hyperparameter performance space is instantiated under tensor framework that can preserve the spatial structure and reflect the correlations among the adjacent hyperparameters. When some partial evaluations are available for a new problem, the task of estimating the performance of the unevaluated hyperparameters can be formulated as a tensor completion (TC) problem. Toward the completion purpose, we develop an LRTC algorithm utilizing the sum of nuclear norm (SNN) model. A kernelized version is further developed to capture the nonlinear structure of the performance space. In addition, a corresponding coupled matrix factorization (CMF) algorithm is established to render the predictions solely depend on the meta-features to avoid additional hyperparameter evaluations. Finally, a strategy integrating LRTC and CMF is provided to further enhance the recommendation capacity. We test recommendation performance with our proposed methods for classical SVM and the state-of-the-art deep neural networks such as vision transformer (ViT) and residual network (ResNet), and the obtained results demonstrate the effectiveness of our approaches under various evaluation metrics by comparing with the baselines commonly used for MtL.

In the study of salient object detection, multiview features play an important role in identifying various underlying salient objects. As to current common patch-based methods, all different features are handled directly by stacking them into a high-dimensional vector to represent related image patches. These approaches ignore the correlations inhering in the original spatial structure, which may lead to the loss of certain underlying characterization such as view interaction. In this article, different from currently available approaches, a tensorial feature representation framework is developed for the salient object detection in order to better explore the complementary information of multiview features. Under the tensor framework, a tensor low-rank constraint is applied to the background to capture its intrinsic structure, a tensor group sparsity regularization is posed on the salient part, and a tensorial sliced Laplacian regularization is then introduced to enlarge the gap between the subspaces of the background and salient object. Moreover, a nonconvex tensor Log-determinant function, instead of the tensor nuclear norm, is adopted to approximate the tensor rank for effectively suppressing the confusing information resulted from underlying complex backgrounds. Further, we have deduced the closed-form solution of this nonconvex minimization problem and established a feasible algorithm whose convergence is mathematically proven. Experiments on five well-known public datasets are provided and the simulations demonstrate that our method outperforms the latest unsupervised handcrafted features-based methods in the literature. Furthermore, our model is flexible with various deep features and is competitive with the state-of-the-art approaches.

Multi-view clustering aims to capture the multiple views inherent information by identifying the data clustering that reflects distinct features of datasets. Since there is a consensus in literature that different views of a dataset share a common latent structure, most existing multi-view subspace learning methods rely on the nuclear norm to seek the low-rank representation of the underlying subspace. However, the nuclear norm often fails to distinguish the variance of features for each cluster due to its convex nature and data tends to fall in multiple non-linear subspaces for multi-dimensional datasets. To address these problems, we propose a new and novel multi-view clustering method (HL-L21-TLD-MSC) that unifies the Hyper-Laplacian (HL) and exclusive ℓ 2,1 (L21) regularization with the Tensor Log-Determinant Rank Minimization (TLD) setting. Specifically, the hyper-Laplacian regularization maintains the local geometrical structure that makes the estimation prune to nonlinearities, and the mixed ℓ 2,1 and ℓ 1,2 regularization provides the joint sparsity within-cluster as well as the exclusive sparsity between-cluster. Furthermore, a log-determinant function is used as a tighter tensor rank approximation to discriminate the dimension of features. An efficient alternating algorithm is then derived to optimize the proposed model, and the construction of a convergent sequence to the Karush-Kuhn-Tucker (KKT) critical point solution is mathematically validated in detail. Extensive experiments are conducted on ten well-known datasets to demonstrate that the proposed approach outperforms the existing state-of-the-art approaches with various scenarios, in which, six of them achieve perfect results under our framework developed in this article, demonstrating highl effectiveness for the proposed approach.

Saliency detection is an important but challenging task in the study of computer vision. In this article, we develop a new unsupervised learning approach for the saliency detection by an intrinsic regularization model, in which the Schatten-2/3 norm is integrated with the nonconvex sparse${l_{2/3}}$norm. The${l_{2/3}}$-norm is shown to be capable of detecting consistent values among sparse foreground by using image geometrical structure and feature similarity, while the Schatten-2/3 norm can capture the lower rank of background by matrix factorization. To improve effective performance of separation for Schatten-2/3-norm and${l_{2/3}}$-norm, a Laplacian regularization is adopted to the foreground for the smoothness. The proposed model essentially converts the required nonconvex optimization problem into the convex one, conducted by splitting the objective function based on singular value decomposition on one much smaller factor matrix and then optimized by using the alternating direction method of the multiplier. The convergence of the proposed algorithm is discussed in detail. Extensive experiments on three benchmark datasets demonstrate that our unsupervised learning approach is very competitive and appears to be more consistent across various salient objects than the current existing approaches.

In this paper, we study the image multiview subspace clustering problem via a nonconvex low-rank representation under the framework of tensors. Most of the recent studies of tensor based multiview subspace clustering use the tensor nuclear norm as a convex surrogate of the tensor rank, i.e., the t-SVD based multiview subspace clustering model. However, since the tensor nuclear norm is linearly proportional to the sum of singular values, the tensor rank approximation by using the tensor nuclear norm may become problematic if the ratios of the nonzero singular values are far from 1. In this paper, a nonconvex tensor log-determinant function is proposed as the objective function regularizer, aiming to achieve a better tensor low-rank approximation. Instead of directly solving the minimization problem in its original setting, the corresponding non-convex optimization is conducted in the Fourier domain, which is shown not only to be feasible but also to be quite effective. A corresponding algorithm associated with the augmented Lagrangian multipliers is established and the constructed convergent sequence to the desirable Karush--Kuhn--Tucker critical point solution is mathematically validated in detail. Extensive simulations on eight benchmark image datasets are provided, along with full comparisons with the latest existing approaches. The obtained results demonstrate that our proposed method significantly outperforms those convex approaches currently available in the literature.

This paper studies the exponential synchronization problem for a new array of nonlinearly and stochastically coupled networks via events-triggered sampling (ETS) by self-adaptive learning. The networks include the following features: 1) a Bernoulli stochastic variable is introduced to describe the random structural coupling; 2) a stochastic variable with positive mean is used to model the coupling strength; and 3) a continuous time homogeneous Markov chain is employed to characterize the dynamical switching of the coupling structure and pinned node sets. The proposed network model is capable to capture various stochastic effect of an external environment during the network operations. In order to reduce networks' workload, different ETS strategies for network self-adaptive learning are proposed under continuous and discrete monitoring, respectively. Based on these ETS approaches, several sufficient conditions for synchronization are derived by employing stochastic Lyapunov-Krasovskii functions, the properties of stochastic processes, and some linear matrix inequalities. Numerical simulations are provided to demonstrate the effectiveness of the theoretical results and the superiority of the proposed ETS approach.

This paper investigates the passivity problem for a class of uncertain stochastic fuzzy nonlinear systems with mixed delays and nonlinear noise disturbances by employing an improved free-weighting matrix approach. The fuzzy system is based on the Takagi-Sugeno model that is often used to represent the complex nonlinear systems in terms of fuzzy sets and fuzzy reasoning. To reflect more realistic dynamical behaviors of the system, the parameter uncertainties, the stochastic disturbances, and nonlinearities are considered, where the parameter uncertainties enter into all the system matrices, the stochastic disturbances are given in the form of a Brownian motion. The mixed delays comprise both discrete and distributed time-varying delays. By taking the relationship among the time delays, their lower and upper bounds into account, some less conservative linear-matrix-inequality-based delay-dependent passivity criteria are obtained without ignoring any useful terms in the derivative of Lyapunov functional. Finally, numerical examples are given to demonstrate the effectiveness and merits of the proposed methods.

This paper investigates the problem of state feedback H∞control for singular systems through delta operator approach. Firstly, a bounded real lemma corresponding to a singular continuous system under the framework of the delta operator model is obtained. Then, the existence condition and explicit expression of a desirable H∞controller for the singular delta operator system are presented. As special cases, the results of admissible control for the singular delta operator system and H∞control as well as admissible control for the singular continuous system are also derived. All required conditions in this paper are characterized in the form of strict linear matrix inequalities whose feasible solutions can be obtained easily and directly. Finally, some numerical examples are provided to illustrate the effectiveness of the obtained theoretical results in the paper.