We consider the projected gradient algorithm for the nonconvex best subset selection problem that minimizes a given empirical loss function under an $\ell_0$-norm constraint. Through decomposing the feasible set of the given sparsity constraint as a finite union of linear subspaces, we present two acceleration schemes with global convergence guarantees, one by same-space extrapolation and the other by subspace identification. The former fully utilizes the problem structure to greatly accelerate the optimization speed with only negligible additional cost. The latter leads to a two-stage meta-algorithm that first uses classical projected gradient iterations to identify the correct subspace containing an optimal solution, and then switches to a highly-efficient smooth optimization method in the identified subspace to attain superlinear convergence. Experiments demonstrate that the proposed accelerated algorithms are magnitudes faster than their non-accelerated counterparts as well as the state of the art.

We propose an efficient neural network for solving the second-order cone constrained variational inequality (SOCCVI). The network is constructed using the Karush-Kuhn-Tucker (KKT) conditions of the variational inequality (VI), which is used to recast the SOCCVI as a system of equations by using a smoothing function for the metric projection mapping to deal with the complementarity condition. Aside from standard stability results, we explore second-order sufficient conditions to obtain exponential stability. Especially, we prove the nonsingularity of the Jacobian of the KKT system based on the second-order sufficient condition and constraint nondegeneracy. Finally, we present some numerical experiments, illustrating the efficiency of the neural network in solving SOCCVI problems. Our numerical simulations reveal that, in general, the new neural network is more dominant than all other neural networks in the SOCCVI literature in terms of stability and convergence rates of trajectories to SOCCVI solution.