This study focuses on the Nesterov's accelerated quasi-Newton (NAQ) method in the context of deep neural networks (DNN) and its applications. The thesis objective is to confirm the robustness and efficiency of Nesterov's acceleration to quasi-Netwon (QN) methods by developing practical algorithms for different fields of optimization problems.

Incorporating curvature information in stochastic methods has been a challenging task. This paper proposes a momentum accelerated BFGS quasi-Newton method in both its full and limited memory forms, for solving stochastic large scale non-convex optimization problems in neural networks (NN).

With the rising need for high-performance circuits, electronic design automation has gained significant attention. Analog circuit design optimization is complex mainly due to the large search space and high non-linearity. Neural networks have shown to be effective in solving highly non-linear problems. Training algorithms play an important role in neural networks. Gradient-based algorithms are popularly used in the training of neural networks. They can be broadly classified into first and second-order methods. While first-order methods are simple and low in computational complexity, they are slow in convergence when applied to highly non-linear problems. Incorporating second order curvature information has shown to significantly improve convergence despite its high computational cost. This paper proposes a second order adaptive modified Nesterov's accelerated quasi-Newton method (amNAQ) with an adaptive momentum scheme. The performance of the proposed method is evaluated in transistor sizing of a two stage op-amp. The results indicate that the proposed method can efficiently deduce the size while satisfying the desired specifications much faster compared to the existing popular algorithms.

Recently algorithms incorporating second order curvature information have become popular in training neural networks. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to effectively accelerate the BFGS quasi-Newton method by incorporating the momentum term and Nesterov's accelerated gradient vector. A stochastic version of NAQ method was proposed for training of large-scale problems. However, this method incurs high stochastic variance noise. This paper proposes a stochastic variance reduced Nesterov's Accelerated Quasi-Newton method in full (SVR-NAQ) and limited (SVRLNAQ) memory forms. The performance of the proposed method is evaluated in Tensorflow on four benchmark problems - two regression and two classification problems respectively. The results show improved performance compared to conventional methods.

Recent studies incorporate Nesterov's accelerated gradient method for the acceleration of gradient based training. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to drastically improve the convergence speed compared to the conventional quasi-Newton method. This paper implements NAQ for non-convex optimization on Tensorflow. Two modifications have been proposed to the original NAQ algorithm to ensure global convergence and eliminate linesearch. The performance of the proposed algorithm - mNAQ is evaluated on standard non-convex function approximation benchmark problems and microwave circuit modelling problems. The results show that the improved algorithm converges better and faster compared to first order optimizers such as AdaGrad, RMSProp, Adam, and the second order methods such as the quasi-Newton method.