Due to the rise of cardinality minimization in optimization, sparse support vector machines (SSVMs) have attracted much attention lately and show certain empirical advantages over convex SVMs. A common way to derive an SSVM is to add a cardinality function such as $\ell _{0}$ℓ0-norm to the dual problem of a convex SVM. However, this process lacks theoretical justification. This paper fills the gap by developing a local duality theory for such an SSVM formulation and exploring its relationship with the hinge-loss SVM (hSVM) and the ramp-loss SVM (rSVM). In particular, we prove that the derived SSVM is exactly the dual problem of the 0/1-loss SVM, and the linear representer theorem holds for their local solutions. The local solution of SSVM also provides guidelines on selecting hyperparameters of hSVM and rSVM. Under specific conditions, we show that a sequence of global solutions of hSVM converges to a local solution of 0/1-loss SVM. Moreover, a local minimizer of 0/1-loss SVM is a local minimizer of rSVM. This explains why a local solution induced by SSVM outperforms hSVM and rSVM in the prior empirical study. We further conduct numerical tests on real datasets and demonstrate potential advantages of SSVM by working with locally nice solutions proposed in this paper.

The hard-margin loss function has been at the core of the support vector machine research from the very beginning due to its generalization capability. On the other hand, the cardinality constraint has been widely used for feature selection, leading to sparse solutions. This paper studies the sparse SVM with the hard-margin loss that integrates the virtues of both worlds, resulting in one of the most challenging models to solve. We cast the problem as a composite optimization with the cardinality constraint. We characterize its local minimizers in terms of pseudo KKT point that well captures the combinatorial structure of the problem, and investigate a sharper P-stationary point with a concise representation for algorithm design. We further develop an inexact proximal augmented Lagrangian method (iPAL). The different parts of the inexactness measurements from the {\rm P}-stationarity are controlled at different scales in a way that the generated sequence converges both globally and at a linear rate. To make iPAL practically efficient, we propose a gradient-Newton method in a subspace for the iPAL subproblem. This is accomplished by detecting active samples and features with the help of the proximal operator of the hard margin loss and the projection of the cardinality constraint. Extensive numerical results on both simulated and real data sets demonstrate that the proposed method is fast, produces sparse solution of high accuracy, and can lead to effective reduction on active samples and features when compared with several leading solvers.

Rectified linear unit (ReLU) deep neural network (DNN) is a classical model in deep learning and has achieved great success in many applications. However, this model is characterized by too many parameters, which not only requires huge memory but also imposes unbearable computation burden. The$l_{2,0}$regularization has become a useful technique to cope with this trouble. In this article, we design a recursion Newton-like algorithm (RNLA) to simultaneously train and compress ReLU-DNNs with$l_{2,0}$regularization. First, we reformulate the multicomposite training model into a constrained optimization problem by explicitly introducing the network nodes as the variables of the optimization. Based on the penalty function of the reformulation, we obtain two types of minimization subproblems. Second, we build the first-order optimality conditions for acquiring P-stationary points of the two subproblems, and these P-stationary points enable us to equivalently derive two sequences of stationary equations, which are piecewise linear matrix equations. We solve these equations by the column Newton-like method in group sparse subspace with lower computational scale and cost. Finally, numerical experiments are conducted on real datasets, and the results demonstrate that the proposed method RNLA is effective and applicable.

Support vector machines (SVM) have drawn wide attention for the last two decades due to its extensive applications, so a vast body of work has developed optimization algorithms to solve SVM with various soft-margin losses. To distinguish all, in this paper, we aim at solving an ideal soft-margin loss SVM:$L_{0/1}$soft-margin loss SVM (dubbed as$L_{0/1}$-SVM). Many of the existing (non)convex soft-margin losses can be viewed as one of the surrogates of the$L_{0/1}$soft-margin loss. Despite its discrete nature, we manage to establish the optimality theory for the$L_{0/1}$-SVM including the existence of the optimal solutions, the relationship between them and P-stationary points. These not only enable us to deliver a rigorous definition of$L_{0/1}$support vectors but also allow us to define a working set. Integrating such a working set, a fast alternating direction method of multipliers is then proposed with its limit point being a locally optimal solution to the$L_{0/1}$-SVM. Finally, numerical experiments demonstrate that our proposed method outperforms some leading classification solvers from SVM communities, in terms of faster computational speed and a fewer number of support vectors. The bigger the data size is, the more evident its advantage appears.

Algorithms based on the hard thresholding principle have been well studied with sounding theoretical guarantees in the compressed sensing and more general sparsity-constrained optimization. It is widely observed in existing empirical studies that when a restricted Newton step was used (as the debiasing step), the hard-thresholding algorithms tend to meet halting conditions in a significantly low number of iterations and are very efficient. Hence, the thus obtained Newton hard-thresholding algorithms call for stronger theoretical guarantees than for their simple hard-thresholding counterparts. This paper provides a theoretical justification for the use of the restricted Newton step. We build our theory and algorithm, Newton Hard-Thresholding Pursuit (NHTP), for the sparsity-constrained optimization. Our main result shows that NHTP is quadratically convergent under the standard assumption of restricted strong convexity and smoothness. We also establish its global convergence to a stationary point under a weaker assumption. In the special case of the compressive sensing, NHTP effectively reduces to some of the existing hard-thresholding algorithms with a Newton step. Consequently, our fast convergence result justifies why those algorithms perform better than without the Newton step. The efficiency of NHTP was demonstrated on both synthetic and real data in compressed sensing and sparse logistic regression.

Sparse logistic regression (SLR), which is widely used for classification and feature selection in many fields, such as neural networks, deep learning, and bioinformatics, is the classical logistic regression model with sparsity constraints. In this paper, we perform theoretical analysis on the existence and uniqueness of the solution to the SLR, and we propose a greedy projected gradient-Newton (GPGN) method for solving the SLR. The GPGN method is a combination of the projected gradient method and the Newton method. The following characteristics show that the GPGN method achieves not only elegant theoretical results but also a remarkable numerical performance in solving the SLR: 1) the full iterative sequence generated by the GPGN method converges to a global/local minimizer of the SLR under weaker conditions; 2) the GPGN method has the properties of afinite identification for an optimal support set and local quadratic convergence; and 3) the GPGN method achieves higher accuracy and higher speed compared with a number of state-of-the-art solvers according to numerical experiments.

The octagonal shrinkage and clustering algorithm for regression (OSCAR), equipped with the $\ell_1$-norm and a pair-wise $\ell_{\infty}$-norm regularizer, is a useful tool for feature selection and grouping in high-dimensional data analysis. The computational challenge posed by OSCAR, for high dimensional and/or large sample size data, has not yet been well resolved due to the non-smoothness and non-separability of the regularizer involved. In this paper, we successfully resolve this numerical challenge by proposing a sparse semismooth Newton-based augmented Lagrangian method to solve the more general SLOPE (the sorted L-one penalized estimation) model. By appropriately exploiting the inherent sparse and low-rank property of the generalized Jacobian of the semismooth Newton system in the augmented Lagrangian subproblem, we show how the computational complexity can be substantially reduced. Our algorithm offers a notable computational advantage in the high-dimensional statistical regression settings. Numerical experiments are conducted on real data sets, and the results demonstrate that our algorithm is far superior, in both speed and robustness, to the existing state-of-the-art algorithms based on first-order iterative schemes, including the widely used accelerated proximal gradient (APG) method and the alternating direction method of multipliers (ADMM).

Quadratic compressive sensing, as a nonlinear extension of compressive sensing, has attracted considerable attention in optical image, X-ray crystallography, transmission electron microscopy, etc. We introduce the concept of uniform s-regularity to study the uniqueness in quadratic compressive sensing and propose a greedy algorithm for the corresponding numerical optimization. Moreover, we prove the convergence of the proposed algorithm under the uniform s-regularity condition. Finally, we present numerical results to demonstrate the efficiency of the proposed method.

In this paper, we study approximation algorithms for two supply chain network design problems, namely, the warehouse-retailer network design problem (WRND) and the stochastic transportation-inventory network design problem (STIND). These two problems generalize the classical uncapacitated facility location problem by incorporating, respectively, the warehouse-retailer echelon inventory cost and the warehouse cycle inventory together with the safety stock costs. The WRND and the STIND were initially studied, respectively, by Teo and Shu (Teo CP, Shu J (2004) Warehouse-retailer network design problem. Oper. Res. 52(3):396–408) and Shu et al. (Shu J, Teo CP, Shen ZJM (2005) Stochastic transportation-inventory network design problem. Oper. Res. 53(1):48–60), where they are formulated as set-covering problems, and column-generation algorithms were used to solve their linear programming relaxations. Both problems can be regarded as special cases of the so-called facility location with submodular facility costs proposed by Svitkina and Tardos (Svitkina Z, Tardos É (2010) Facility location with hierarchical facility costs. ACM Trans. Algorithms 6(2), Article No. 37), for which only a logarithmic-factor approximation algorithm is known. Our main contribution is to obtain efficient constant-factor approximation algorithms for the WRND and the STIND, which are capable of solving large-scale instances of these problems efficiently.