Optimization methods often face a trade-off between the fast convergence of second-order methods and the low computational cost of first-order methods. Motivated by the need to bridge this gap, this paper proposes a novel optimization algorithm called the Adaptive Dynamical Momentum Newton Algorithm (ADINA), based on a spatiotemporal second-order dynamical model incorporating both momentum and an adaptive learning rate. The proposed ADINA integrates the momentum acceleration and adaptive learning rate of Adam with the quadratic convergence properties of Newton method while eliminating the need for explicit Hessian computations. By dynamically adjusting step sizes according to the gradient norm, the proposed method achieves rapid convergence and high-precision solutions while maintaining computational efficiency similar to first-order methods. Extensive experiments on function optimization, regression, and classification tasks demonstrate that ADINA consistently outperforms existing first- and second-order optimization algorithms in terms of convergence speed and solution quality. Moreover, its practical effectiveness is validated through deep learning experiments conducted with TensorFlow, confirming its superiority over conventional built-in optimizers.

Incorporating curvature information in stochastic methods has been a challenging task. This paper proposes a momentum accelerated BFGS quasi-Newton method in both its full and limited memory forms, for solving stochastic large scale non-convex optimization problems in neural networks (NN).

With the rising need for high-performance circuits, electronic design automation has gained significant attention. Analog circuit design optimization is complex mainly due to the large search space and high non-linearity. Neural networks have shown to be effective in solving highly non-linear problems. Training algorithms play an important role in neural networks. Gradient-based algorithms are popularly used in the training of neural networks. They can be broadly classified into first and second-order methods. While first-order methods are simple and low in computational complexity, they are slow in convergence when applied to highly non-linear problems. Incorporating second order curvature information has shown to significantly improve convergence despite its high computational cost. This paper proposes a second order adaptive modified Nesterov's accelerated quasi-Newton method (amNAQ) with an adaptive momentum scheme. The performance of the proposed method is evaluated in transistor sizing of a two stage op-amp. The results indicate that the proposed method can efficiently deduce the size while satisfying the desired specifications much faster compared to the existing popular algorithms.

Recently algorithms incorporating second order curvature information have become popular in training neural networks. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to effectively accelerate the BFGS quasi-Newton method by incorporating the momentum term and Nesterov's accelerated gradient vector. A stochastic version of NAQ method was proposed for training of large-scale problems. However, this method incurs high stochastic variance noise. This paper proposes a stochastic variance reduced Nesterov's Accelerated Quasi-Newton method in full (SVR-NAQ) and limited (SVRLNAQ) memory forms. The performance of the proposed method is evaluated in Tensorflow on four benchmark problems - two regression and two classification problems respectively. The results show improved performance compared to conventional methods.

Recent studies incorporate Nesterov's accelerated gradient method for the acceleration of gradient based training. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to drastically improve the convergence speed compared to the conventional quasi-Newton method. This paper implements NAQ for non-convex optimization on Tensorflow. Two modifications have been proposed to the original NAQ algorithm to ensure global convergence and eliminate linesearch. The performance of the proposed algorithm - mNAQ is evaluated on standard non-convex function approximation benchmark problems and microwave circuit modelling problems. The results show that the improved algorithm converges better and faster compared to first order optimizers such as AdaGrad, RMSProp, Adam, and the second order methods such as the quasi-Newton method.