-regularized logistic regression (L1-LR) is popular for classification problems. To accelerate its training speed for high-dimensional data, techniques named safe screening rules have been proposed recently. They can safely delete the inactive features in data so as to greatly reduce the training cost of L1-LR. The screening power of these rules is determined by their corresponding safe regions, which is also the core technique of safe screening rules. In this paper, we introduce a new safe feature elimination rule (SFER) for L1-LR. Compared to existing safe rules, the safe region of SFER is improved in two aspects: (1) a smaller sphere region is constructed by using the strong convexity of dual L1-LR twice; (2) multiple half-spaces, which correspond to the potential active constraints, are added for further contraction. Both improvements can enhance the screening ability of SFER. As for the complexity of SFER, an iterative filtering framework is given by decomposing the safe region into multiple "domes". In this way, SFER admits a closed form solution and the identified features will not be scanned repeatedly. Experiments on ten benchmark data sets demonstrate that SFER gives superior performance than existing methods on training efficiency.

To make support vector machine (SVM) applicable to large-scale data sets, safe screening rules are developed recently. The main idea is to reduce the scale of SVM by safely discarding the redundant training samples. Among existing safe screening rules, the dual screening method with variational inequalities (DVI) and the dynamic screening rule (DSR) based on duality gap are two representative strategies. DVI is efficient, while its safety may be affected by inaccurate solving algorithms. DSR is guaranteed to be safe; however, accurate feasible solutions are required for good efficiency. Based on the above-mentioned studies, in this paper, a two-stage screening (TSS) rule, which fully exploits the advantages of the above-mentioned two approaches and improves their shortcomings, is proposed. First, DVI is applied prior to training for sample screening. It reduces the scale of SVM and, meanwhile, produces a better initial feasible solution for DSR. Then, by embedding DSR into the solving algorithm, the solver becomes more accurate, and the safety of DVI can be strengthened. In the end, for safety guarantee, a postchecking step is added to search the wrongly identified samples and retrain them. To theoretically analyze the safety of DVI, an upper bound of the deviation in DVI is estimated, and a Safe-DVI is given based on it. To ensure the efficiency of TSS, the superiority of DVI over initial DSR is verified. In addition, kernel version of TSS is also given for nonlinear SVM. Numerical experiments on synthetic data sets and 12 real-world data sets verify the efficiency and safety of this TSS.

The twin support vector machine (TSVM) is widely used in classification problems, but it is not efficient enough for large-scale data sets. Furthermore, to get the optimal parameter, the exhaustive grid search method is applied to TSVM. It is very time-consuming, especially for multiparameter models. Although many techniques have been presented to solve these problems, all of them always affect the performance of TSVM to some extent. In this paper, we propose a safe screening rule (SSR) for linear-TSVM, and give a modified SSR (MSSR) for nonlinear TSVM, which contains multiple parameters. The SSR and MSSR can delete most training samples and reduce the scale of TSVM before solving it. Sequential versions of SSR and MSSR are further introduced to substantially accelerate the whole parameter tuning process. One important advantage of SSR and MSSR is that they are safe, i.e., we can obtain the same solution as the original problem by utilizing them. Experiments on eight real-world data sets and an imbalanced data set with different imbalanced ratios demonstrate the efficiency and safety of SSR and MSSR.

Twin support-vector machine (TSVM), which generates two nonparallel hyperplanes by solving a pair of smaller-sized quadratic programming problems (QPPs) instead of a single larger-sized QPP, works faster than the standard SVM, especially for the large-scale data sets. However, the traditional TSVM adopts hinge loss which easily leads to its sensitivity of the noise and instability for resampling. To enhance the performance of the TSVM, we present a novel TSVM with the pinball loss (Pin-TSVM) which deals with the quantile distance and is less sensitive to noise points. We further investigate its properties, including the noise insensitivity, between-class distance maximization, and within-class scatter minimization. In addition, we compare our Pin-TSVM with the twin parametric-margin SVM and the SVM with the pinball loss in theory. Numerical experiments on a synthetic data set and 14 benchmark data sets with different noises demonstrate the feasibility and validity of our proposed method.