Feature selection for high-risk disease diagnosis is critical for accurate risk assessment, yet existing methods fail to incorporate medical domain knowledge, potentially leading to suboptimal diagnostic decisions. To address this limitation, we propose recursive statistical invariants elimination (RSIE), a novel framework that systematically integrates medical domain knowledge into feature selection. RSIE comprises three complementary components: learning using statistical invariants (LUSI) for extracting reliable diagnostic patterns from domain knowledge, kernel alignment for establishing meaningful correlations between domain knowledge and dataset features, and recursive feature elimination (RFE) for systematic feature ranking and selection. We evaluate RSIE on 15 high-risk disease datasets spanning cardiovascular, neurological, and oncological conditions. Experimental results show that RSIE achieves an average accuracy of 88.74% across all datasets compared to 83.28% for traditional SVM-RFE, representing a 5.46% improvement. Specifically, the LUSI-RFE* variant achieves superior performance, reducing feature dimensions by 45% -74% while maintaining high classification accuracy. The experiments simultaneously demonstrate the stability of RSIE against high noise conditions, computational efficiency, and model interpretability for high-risk disease diagnosis.

Deep learning methodologies have shown impressive effectiveness in tackling time series tasks; nevertheless, a significant issue revolves around the inherent trade-off between the length of the look-back window (data input length) and prediction accuracy. In general, expanding the look-back window tends to positively influence predictive tasks. However, the fixed nature of the look-back window restricts both the accuracy and temporal scope of predictive models. Consequently, this paper introduces a novel approach named the Multi-scale Input Ensemble Linear Network (MIEL), aimed at leveraging historical time series data comprehensively while addressing the complexities of ultra-long time series prediction. MIEL generates multi-scale time series inputs through random forward sampling, thus extending the predictive capabilities of the model for extended time series. Simultaneously, the model integration strategy, which uses the same parameters to process data at different scales, reduces the number of parameters and memory usage when handling multi-scale data. Empirical results demonstrate that MIEL outperforms the eight most recent methodologies in terms of prediction accuracy especially for ultra-long time series prediction, while only marginally increasing memory utilization. The code has been released to github1

The$V$-matrix Support Vector Machine (VSVM) is an innovative machine learning method recently proposed by Vapnik and Izmailov, which integrates positional relationships among training samples into the model learning, yielding the decision via conditional probability. But it overlooks the distribution information hidden in the data which plays a pivotal role in the training process and neglects the utilization of testing samples. To fully exploit the distribution information of the data, this paper proposes a novel Distribution Metric Based$V$-matrix Support Vector Machine (DVSVM) building upon VSVM. DVSVM incorporates the distributional information implicit in the data by measuring the distances between samples using the Wasserstein distance. Compared to VSVM, it also additionally accounts for the positional relationships of testing samples. It is further theoretically proved that VSVM can degenerate from DVSVM under certain conditions. Experimental results on several synthetic datasets and real-world disease datasets demonstrate the superiority of DVSVM.

Domain adaptation has found widespread applications in real-life scenarios, especially when the target domain has limited labeled samples. However, most of the domain adaptation models only utilize one type of knowledge from the source domain, which is usually achieved by strong mode of convergence. To fully incorporate multiple knowledge from the source domain, for binary classification, this paper studies a novel learning paradigm for Domain Adaptation via Learning Using Statistical Invariant by simultaneously combining the strong and weak modes of convergence in a Hilbert space. The strong mode of convergence undertakes the mission of learning a least squares probability output binary classification task in a general hypothesis space, while the weak mode of convergence integrates diverse knowledge by constructing meaningful statistical invariants that embody the concept of intelligence. The utilization of weak convergence shrinks the admissible set of approximation functions, and subsequently accelerates the learning process. In this paper, several statistical invariants that represent sample, feature and parameter information from the source domain are constructed. By taking an appropriate statistical invariant, DLUSI realizes some existing methods. Experimental results on synthetic data as well as the widely used Amazon Reviews and 20 News data demonstrate the superiority of the proposed method.

To investigate the adversarial perturbations to twin support vector machines (TWSVMs) and hence increase the deviation of separating hyperplanes, we establish an optimization framework for adversarial perturbations of linear TWSVMs by taking the minimal perturbation that may cause the original label changes into account. By transforming the problem into a distance problem from point to intersecting hyperplane, we respectively obtain the explicit solutions to the model for the sample-adversarial perturbations case and for the class-universal adversarial perturbations case. The explicit solution obtained, it increases the interpretability of the conclusion and provides great convenience for calculation. Some numerical experiments are conducted on datasets MNIST and CIFAR-10 with Gaussian noise and trained SVM perturbations, which shows the efficiency of our proposed adversarial perturbations model.

Support vector machine (SVM) is a popular supervised machine learning classifier and has found extensive applied in many fields, including biological sciences, disease detection, health and clinical sciences, cancer classification, and more. However, the major challenge faced by SVM is its high-computational complexity, which becomes a bottleneck for large-scale SVM. To reduce computational complexity, we design a novel truncated squared loss function to get the novel SVM$(L_{{\rm tsl}}$-SVM), and is a challenging model due to its nonconvex and nonsmooth characteristics. To solve$L_{{\rm tsl}}$-SVM, we present new concept of proximal stationary point to establish its optimality theory. Using this theory, we then develop a novel and fast alternating direction method of multipliers in terms of low-computational complexity to address$L_{{\rm tsl}}$-SVM and our new proposed algorithm achieve global convergence. Finally, numerical experiments have verified the superior performance of our developed method in terms of classification accuracy, number of support vectors and computational speed when compared to other eight leading solvers. For instance, when solving the real dataset with more than$10^{7}$samples, our developed method only takes 18.89 s, significantly outperforming other solvers that require at least 589.8 s.

Support vector machines (SVM) have drawn wide attention for the last two decades due to its extensive applications, so a vast body of work has developed optimization algorithms to solve SVM with various soft-margin losses. To distinguish all, in this paper, we aim at solving an ideal soft-margin loss SVM:$L_{0/1}$soft-margin loss SVM (dubbed as$L_{0/1}$-SVM). Many of the existing (non)convex soft-margin losses can be viewed as one of the surrogates of the$L_{0/1}$soft-margin loss. Despite its discrete nature, we manage to establish the optimality theory for the$L_{0/1}$-SVM including the existence of the optimal solutions, the relationship between them and P-stationary points. These not only enable us to deliver a rigorous definition of$L_{0/1}$support vectors but also allow us to define a working set. Integrating such a working set, a fast alternating direction method of multipliers is then proposed with its limit point being a locally optimal solution to the$L_{0/1}$-SVM. Finally, numerical experiments demonstrate that our proposed method outperforms some leading classification solvers from SVM communities, in terms of faster computational speed and a fewer number of support vectors. The bigger the data size is, the more evident its advantage appears.

Cross-manifold clustering is an extreme challenge learning problem. Since the low-density hypothesis is not satisfied in cross-manifold problems, many traditional clustering methods failed to discover the cross-manifold structures. In this article, we propose multiple flat projections clustering (MFPC) for cross-manifold clustering. In our MFPC, the given samples are projected into multiple localized flats to discover the global structures of implicit manifolds. Thus, the intersected clusters are distinguished in various projection flats. In MFPC, a series of nonconvex matrix optimization problems is solved by a proposed recursive algorithm. Furthermore, a nonlinear version of MFPC is extended via kernel tricks to deal with a more complex cross-manifold learning situation. The synthetic tests show that our MFPC works on the cross-manifold structures well. Moreover, experimental results on the benchmark datasets and object tracking videos show excellent performance of our MFPC compared with some state-of-the-art manifold clustering methods.

In semisupervised fuzzy clustering, this article extends the traditional pairwise constraint (i.e., must-link or cannot-link) to fuzzy pairwise constraint. The fuzzy pairwise constraint allows a supervisor to provide the grade of similarity or dissimilarity between the implicit fuzzy vectors of a pair of samples. This constraint can represent more complicated relationship between the pair of samples and avoid eliminating the fuzzy characteristics. Then, we propose a semisupervised fuzzy clustering with fuzzy pairwise constraints (SSFPC). The nonconvex optimization problem in our SSFPC is solved by a modified expectation-maximization algorithm, involving to solve several indefinite quadratic programming problems (IQPPs). Further, a diagonal block coordinate decent (DBCD) algorithm is proposed for these IQPPs, whose stationary points are guaranteed, and the global solutions can be obtained under certain conditions. To suit for different applications, the SSFPC is extended into various metric spaces, e.g., the reproducing kernel Hilbert space. Experimental results on several benchmark datasets and a facial expression database demonstrate the outperformance of our SSFPC compared with some state-of-the-art clustering models

When solving many regression problems, there exist a large number of input features. However, not all features are relevant for current regression, and sometimes, including irrelevant features may deteriorate the learning performance. Therefore, it is essential to select the most relevant features, especially for high-dimensional regression. Feature selection is an effective way to solve this problem. It tries to represent original data by extracting relevant features that contain useful information. In this paper, aiming to effectively select useful features in least squares support vector regression (LSSVR), we propose a novel sparse LSSVR based on L p -norm (SLSSVR), 0 < p ≤ 1 . Different from the existing L 1 -norm LSSVR ( L 1 -LSSVR) and L p -norm LSSVR ( L p -LSSVR), SLSSVR uses a smooth approximation of the nonsmooth nonconvex L p -norm term along with an effective solving algorithm. The proposed algorithm avoids the singularity issue that may encounter in L p -LSSVR, and its convergency is also guaranteed. Experimental results support the effectiveness of SLSSVR on both feature selection ability and regression performance.

In this article, we propose a general model for plane-based clustering. The general model reveals the relationship between cluster assignment and cluster updating during clustering implementation, and it contains many existing plane-based clustering methods, e.g., k-plane clustering, proximal plane clustering, twin support vector clustering, and their extensions. Under this general model, one may obtain an appropriate clustering method for a specific purpose. The general model is a procedure corresponding to an optimization problem, which minimizes the total loss of the samples. Thereinto, the loss of a sample derives from both within-cluster and between-cluster information. We discuss the theoretical termination conditions and prove that the general model terminates in a finite number of steps at a local or weak local solution. Furthermore, we propose a distribution loss function that fluctuates with the input data and introduce it into the general model to obtain a plane-based clustering method (DPC). DPC can capture the data distribution precisely because of its statistical characteristics, and its termination that finitely terminates at a weak local solution is given immediately based on the general model. The experimental results show that our DPC outperforms the state-of-the-art plane-based clustering methods on many synthetic and benchmark data sets.

In this paper, we propose a robust linear discriminant analysis (RLDA) through Bhattacharyya error bound optimization. RLDA considers a nonconvex problem with the L1-norm operation that makes it less sensitive to outliers and noise than the L2-norm linear discriminant analysis (LDA). In addition, we extend our RLDA to a sparse model (RSLDA). Both RLDA and RSLDA can extract unbounded numbers of features and avoid the small sample size (SSS) problem, and an alternating direction method of multipliers (ADMM) is used to cope with the nonconvexity in the proposed formulations. Compared with the traditional LDA, our RLDA and RSLDA are more robust to outliers and noise, and RSLDA can obtain sparse discriminant directions. These findings are supported by experiments on artificial data sets as well as human face databases.

The twin support vector machine (TWSVM) is one of the powerful classification methods. In this brief, a TWSVM-type clustering method, called twin support vector clustering (TWSVC), is proposed. Our TWSVC includes both linear and nonlinear versions. It determines k cluster center planes by solving a series of quadratic programming problems. To make TWSVC more efficient and stable, an initialization algorithm based on the nearest neighbor graph is also suggested. The experimental results on several benchmark data sets have shown a comparable performance of our TWSVC.

In this letter, we propose an improved version of generalized eigenvalue proximal support vector machine (GEPSVM), called IGEPSVM for short. The main improvements are 1) the generalized eigenvalue decomposition is replaced by the standard eigenvalue decomposition, resulting in simpler optimization problems without the possible singularity. 2) An extra meaningful parameter is introduced, resulting in the stronger classification generalization ability. Experimental results on both the artificial datasets and several benchmark datasets show that our IGEPSVM is superior to GEPSVM in both computation time and classification accuracy.

For classification problems, the generalized eigenvalue proximal support vector machine (GEPSVM) and twin support vector machine (TWSVM) are regarded as milestones in the development of the powerful SVMs, as they use the nonparallel hyperplane classifiers. In this brief, we propose an improved version, named twin bounded support vector machines (TBSVM), based on TWSVM. The significant advantage of our TBSVM over TWSVM is that the structural risk minimization principle is implemented by introducing the regularization term. This embodies the marrow of statistical learning theory, so this modification can improve the performance of classification. In addition, the successive overrelaxation technique is used to solve the optimization problems to speed up the training procedure. Experimental results show the effectiveness of our method in both computation time and classification accuracy, and therefore confirm the above conclusion further.