Although application examples of multilevel optimization have already been discussed since the 1990s, the development of solution methods was almost limited to bilevel cases due to the difficulty of the problem. In recent years, in machine learning, Franceschi et al. have proposed a method for solving bilevel optimization problems by replacing their lower-level problems with the $T$ steepest descent update equations with some prechosen iteration number $T$. In this paper, we have developed a gradient-based algorithm for multilevel optimization with $n$ levels based on their idea and proved that our reformulation asymptotically converges to the original multilevel problem. As far as we know, this is one of the first algorithms with some theoretical guarantee for multilevel optimization. Numerical experiments show that a trilevel hyperparameter learning model considering data poisoning produces more stable prediction results than an existing bilevel hyperparameter learning model in noisy data settings.

We theoretically study the landscape of the training error for neural networks in overparameterized cases. We consider three basic methods for embedding a network into a wider one with more hidden units, and discuss whether a minimum point of the narrower network gives a minimum or saddle point of the wider one. Our results show that the networks with smooth and ReLU activation have different partially flat landscapes around the embedded point. We also relate these results to a difference of their generalization abilities in overparameterized realization.

Matrices are often required to be well-conditioned in a wide variety of areas including signal processing. Problems to find the nearest positive definite matrix or the nearest correlation matrix that simultaneously satisfy the condition number constraint and sign constraints are presented in this paper. Both problems can be regarded as those to find a projection to the intersection of the closed convex cone corresponding to the condition number constraint and the convex polyhedron corresponding to the other constraints. Thus, we can apply a successive projection method, which is a classical algorithm for finding the projection to the intersection of multiple convex sets, to these problems. The numerical results demonstrated that the algorithm effectively solved the problems.