Due to the rise of cardinality minimization in optimization, sparse support vector machines (SSVMs) have attracted much attention lately and show certain empirical advantages over convex SVMs. A common way to derive an SSVM is to add a cardinality function such as $\ell _{0}$ℓ0-norm to the dual problem of a convex SVM. However, this process lacks theoretical justification. This paper fills the gap by developing a local duality theory for such an SSVM formulation and exploring its relationship with the hinge-loss SVM (hSVM) and the ramp-loss SVM (rSVM). In particular, we prove that the derived SSVM is exactly the dual problem of the 0/1-loss SVM, and the linear representer theorem holds for their local solutions. The local solution of SSVM also provides guidelines on selecting hyperparameters of hSVM and rSVM. Under specific conditions, we show that a sequence of global solutions of hSVM converges to a local solution of 0/1-loss SVM. Moreover, a local minimizer of 0/1-loss SVM is a local minimizer of rSVM. This explains why a local solution induced by SSVM outperforms hSVM and rSVM in the prior empirical study. We further conduct numerical tests on real datasets and demonstrate potential advantages of SSVM by working with locally nice solutions proposed in this paper.

The hard-margin loss function has been at the core of the support vector machine research from the very beginning due to its generalization capability. On the other hand, the cardinality constraint has been widely used for feature selection, leading to sparse solutions. This paper studies the sparse SVM with the hard-margin loss that integrates the virtues of both worlds, resulting in one of the most challenging models to solve. We cast the problem as a composite optimization with the cardinality constraint. We characterize its local minimizers in terms of pseudo KKT point that well captures the combinatorial structure of the problem, and investigate a sharper P-stationary point with a concise representation for algorithm design. We further develop an inexact proximal augmented Lagrangian method (iPAL). The different parts of the inexactness measurements from the {\rm P}-stationarity are controlled at different scales in a way that the generated sequence converges both globally and at a linear rate. To make iPAL practically efficient, we propose a gradient-Newton method in a subspace for the iPAL subproblem. This is accomplished by detecting active samples and features with the help of the proximal operator of the hard margin loss and the projection of the cardinality constraint. Extensive numerical results on both simulated and real data sets demonstrate that the proposed method is fast, produces sparse solution of high accuracy, and can lead to effective reduction on active samples and features when compared with several leading solvers.

Abstract Maximum Variance Unfolding (MVU) is among the first methods in nonlinear dimensionality reduction for data visualization and classification. It aims to preserve local data structure and in the meantime push the variance among data as big as possible. However, MVU in general remains a computationally challenging problem and this may explain why it is less popular than other leading methods such as Isomap and t-SNE. In this paper, based on a key observation that the structure-preserving term in MVU is actually the squared stress in Multi-Dimensional Scaling (MDS), we replace the term with the stress function from MDS, resulting in a model that is usable. The property of the usability guarantees the “crowding phenomenon” will not happen in the dimension reduced results. The new model also allows us to combine label information and hence we call it the supervised MVU (SMVU). We then develop a fast algorithm that is based on Euclidean distance matrix optimization. By making use of the majorization-mininmization technique, the algorithm at each iteration solves a number of one-dimensional optimization problems, each having a closed-form solution. This strategy significantly speeds up the computation. We demonstrate the advantage of SMVU on some standard data sets against a few leading algorithms including Isomap and t-SNE.

Algorithms based on the hard thresholding principle have been well studied with sounding theoretical guarantees in the compressed sensing and more general sparsity-constrained optimization. It is widely observed in existing empirical studies that when a restricted Newton step was used (as the debiasing step), the hard-thresholding algorithms tend to meet halting conditions in a significantly low number of iterations and are very efficient. Hence, the thus obtained Newton hard-thresholding algorithms call for stronger theoretical guarantees than for their simple hard-thresholding counterparts. This paper provides a theoretical justification for the use of the restricted Newton step. We build our theory and algorithm, Newton Hard-Thresholding Pursuit (NHTP), for the sparsity-constrained optimization. Our main result shows that NHTP is quadratically convergent under the standard assumption of restricted strong convexity and smoothness. We also establish its global convergence to a stationary point under a weaker assumption. In the special case of the compressive sensing, NHTP effectively reduces to some of the existing hard-thresholding algorithms with a Newton step. Consequently, our fast convergence result justifies why those algorithms perform better than without the Newton step. The efficiency of NHTP was demonstrated on both synthetic and real data in compressed sensing and sparse logistic regression.

The quality of large-scale recommendation systems has been insufficient in terms of the accuracy of prediction. One of the major reasons is caused by the sparsity of the samples, usually represented by vectors of userspsila ratings on a set of items. Combining information other than userspsila ratings can provide the learning model complementary views of the data and, thus, a more accurate prediction. In this paper, we propose efficient methods for finding the best combination weights among single kernels. The weight parameters are optimized by aligning the combination kernel to ideal kernels. We solve the kernel alignment problem by linear programming techniques.

Following our recent approach of nonsmooth analysis, we report a new set of sufficient conditions and its implications for the global asymptotic stability of delayed cellular neural networks (DCNN). The new conditions not only unify a string of previous stability results, but also yield strict improvement over them by allowing the symmetric part of the feedback matrix positive definite, hence enlarging the application domain of DCNNs. Advantages of the new results over existing ones are illustrated with examples. We also compare our results with those related results obtained via LMI approach.

In this paper, we obtain new sufficient conditions ensuring existence, uniqueness, and global asymptotic stability (GAS) of the equilibrium point for a general class of delayed neural networks (DNNs) via nonsmooth analysis, which makes full use of the Lipschitz property of functions defining DNNs. Based on this new tool of nonsmooth analysis, we first obtain a couple of general results concerning the existence and uniqueness of the equilibrium point. Then those results are applied to show that existence assumptions on the equilibrium point in some existing sufficient conditions ensuring GAS are actually unnecessary; and some strong assumptions such as the boundedness of activation functions in some other existing sufficient conditions can be actually dropped. Finally, we derive some new sufficient conditions which are easy to check. Comparison with some related existing results is conducted and advantages are illustrated with examples. Throughout our paper, spectral properties of the matrix (A + Atau) play an important role, which is a distinguished feature from previous studies. Here, A and Atau are, respectively, the feedback and the delayed feedback matrix defining the neural network under consideration.