Optimization methods often face a trade-off between the fast convergence of second-order methods and the low computational cost of first-order methods. Motivated by the need to bridge this gap, this paper proposes a novel optimization algorithm called the Adaptive Dynamical Momentum Newton Algorithm (ADINA), based on a spatiotemporal second-order dynamical model incorporating both momentum and an adaptive learning rate. The proposed ADINA integrates the momentum acceleration and adaptive learning rate of Adam with the quadratic convergence properties of Newton method while eliminating the need for explicit Hessian computations. By dynamically adjusting step sizes according to the gradient norm, the proposed method achieves rapid convergence and high-precision solutions while maintaining computational efficiency similar to first-order methods. Extensive experiments on function optimization, regression, and classification tasks demonstrate that ADINA consistently outperforms existing first- and second-order optimization algorithms in terms of convergence speed and solution quality. Moreover, its practical effectiveness is validated through deep learning experiments conducted with TensorFlow, confirming its superiority over conventional built-in optimizers.

This study focuses on the Nesterov's accelerated quasi-Newton (NAQ) method in the context of deep neural networks (DNN) and its applications. The thesis objective is to confirm the robustness and efficiency of Nesterov's acceleration to quasi-Netwon (QN) methods by developing practical algorithms for different fields of optimization problems.

Incorporating curvature information in stochastic methods has been a challenging task. This paper proposes a momentum accelerated BFGS quasi-Newton method in both its full and limited memory forms, for solving stochastic large scale non-convex optimization problems in neural networks (NN).

With the rising need for high-performance circuits, electronic design automation has gained significant attention. Analog circuit design optimization is complex mainly due to the large search space and high non-linearity. Neural networks have shown to be effective in solving highly non-linear problems. Training algorithms play an important role in neural networks. Gradient-based algorithms are popularly used in the training of neural networks. They can be broadly classified into first and second-order methods. While first-order methods are simple and low in computational complexity, they are slow in convergence when applied to highly non-linear problems. Incorporating second order curvature information has shown to significantly improve convergence despite its high computational cost. This paper proposes a second order adaptive modified Nesterov's accelerated quasi-Newton method (amNAQ) with an adaptive momentum scheme. The performance of the proposed method is evaluated in transistor sizing of a two stage op-amp. The results indicate that the proposed method can efficiently deduce the size while satisfying the desired specifications much faster compared to the existing popular algorithms.

Recently algorithms incorporating second order curvature information have become popular in training neural networks. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to effectively accelerate the BFGS quasi-Newton method by incorporating the momentum term and Nesterov's accelerated gradient vector. A stochastic version of NAQ method was proposed for training of large-scale problems. However, this method incurs high stochastic variance noise. This paper proposes a stochastic variance reduced Nesterov's Accelerated Quasi-Newton method in full (SVR-NAQ) and limited (SVRLNAQ) memory forms. The performance of the proposed method is evaluated in Tensorflow on four benchmark problems - two regression and two classification problems respectively. The results show improved performance compared to conventional methods.

Recent studies incorporate Nesterov's accelerated gradient method for the acceleration of gradient based training. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to drastically improve the convergence speed compared to the conventional quasi-Newton method. This paper implements NAQ for non-convex optimization on Tensorflow. Two modifications have been proposed to the original NAQ algorithm to ensure global convergence and eliminate linesearch. The performance of the proposed algorithm - mNAQ is evaluated on standard non-convex function approximation benchmark problems and microwave circuit modelling problems. The results show that the improved algorithm converges better and faster compared to first order optimizers such as AdaGrad, RMSProp, Adam, and the second order methods such as the quasi-Newton method.

This paper describes a novel robust training algorithm based on quasi-Newton iteration. The size of training samples for each iteration is dynamically and analytically determined by variance estimates during the computation of its gradient in the mini-batch based online training methodology. Furthermore, the size of mini-batch is controlled by a parameter to ensure that the number of samples in a mini-batch changes from a portion of samples (online) to all ones (batch) as quasi-Newton iteration progressed. As a result, the iteration during online mode can be shortened compared with previous quasi-Newton-based methods in which the gradient of error function for the training step was improved.

Recently, the improved online (stochastic) quasi-Newton method was developed for neural network training improving the gradient of error function. The gradient was calculated by a training sample in the online method, but the gradient of improved online one was calculated by variable training samples which were automatically increased from one to all samples as quasi-Newton iteration progressed. That is, the improved algorithm gradually changed from online to batch methods during iteration. The algorithm was efficient, and provided high quality training solutions regardless of initial values compared with online and batch methods. This paper proposes a novel robust training algorithm based on quasi-Newton in which online and batch error functions are associated by a weighting coefficient parameter. This means that the transition from the online method to the batch one is parameterized in quasi-Newton iteration in the same concept as the above improved algorithm. Furthermore, an analogy between the proposed algorithm and Langevin one is considered. Langevin algorithm is a gradient-based continuous optimization method using Simulated Annealing concept. The proposed algorithm is employed for robust neural network training purpose. Neural network training for some benchmark problems with high-nonlinearity is presented to demonstrate the validity of proposed algorithm. The new training algorithm achieves more accurate and robust training results than the other quasi-Newton based training algorithms.

This paper describes a new technique for robust training of feedforward neural networks. The proposed algorithm is employed for the robust neural network training purpose. The quasi-Newton method was studied as one of the most efficient optimization algorithms based on the gradient descent and used as the batch training method of neural networks. On the other hand, the stochastic (online) quasi-Newton method was developed as an algorithm for the machine learning. In this paper the stochastic quasi-Newton training algorithm is improved for robust neural network training. Neural network training for some benchmark problems is presented to demonstrate the proposed algorithm. Furthermore, neural network training for microwave circuit modeling, such as the waveguide and the microstrip examples is presented, demonstrating that the proposed algorithm achieves more accurate models than both the batch and the stochastic quasi-Newton methods.

This paper describes a new technique for training microwave neural network models. The proposed technique combines quasi-Newton algorithm with a global optimization algorithm called particle swarm optimization (PSO). The quasi-Newton process for searching optimal solutions is incorporated into PSO to speed up local search, while the PSO performs global search avoid being trapped in local minima of training. The overall algorithm iterates between quasi-Newton and PSO. Neural network training for microwave circuit modeling, such as waveguide and microstrip examples is presented, demonstrating that the proposed algorithm achieves more accurate models than the conventional gradient based technique and the conventional PSOs.

This paper describes a new technique for training feedforward neural networks. We employ the proposed algorithm for robust neural network training purpose. Conventional neural network training algorithms based on the gradient descent often encounter local minima problems. Recently, some evolutionary algorithms are getting a lot more attention about global search ability but are less-accurate for complicated training task of neural networks. The proposed technique hybridizes local training algorithm based on quasi-Newton method with a recent global optimization algorithm called particle swarm optimization (PSO). The proposed technique provides higher global convergence property than the conventional global optimization technique. Neural network training for some benchmark problems is presented to demonstrate the proposed algorithm. The proposed algorithm achieves more accurate and robust training results than the quasi-Newton method and the conventional PSOs.

This paper describes the two-staged Tabu search for the non-slicing floorplan problem using the ordered tree representation called O-tree. The floorplan problem is a part of VLSI layout design problem. Furthermore, we combine ideas from the simulated annealing into the two-staged Tabu search and propose a novel hybrid algorithm for floorplan represented by O-tree. Finally, we demonstrate the validity of two-staged search and hybrid method for MCNC benchmark tests through the computer simulations.

A three-dimensional chip design strategy is of increasing significance in association with miniaturization of VLSI circuits. In the design of 3D VLSI and 3D MCM, the placement which is to locate given 3D objects without overlapping each other in the 3D space, becomes an important issue. The 3D placement under the nonoverlapping constraint is called the 3D packing in which minimizing the volume of the package is required. This paper introduces a new coding system to encode the topology of 3D packing as an extension of BSG (bounded-sliceline grid) for 2D packing. The novel coding system is referred to as three dimensional bounded-sliceplane grid (3DBSG). The topology is a set of relative relations assigned to pairs of 3D modules in such a way that one module shall be either "right-of", "rear-of' or "above" the other. Furthermore, we propose an idea to handle L-shaped 3D modules on the 3DBSG. The simulation results in which a standard simulated annealing is utilized as an optimization algorithm, are demonstrated in order to test the validity of our 3D packing method based on the 3DBSG.

This paper describes an efficient learning algorithm based on fixed-point homotopy method. The proposed algorithm has the ability to train the neural networks with high success rates for the initial guesses compared with other typical second-order training algorithms. Furthermore, the method proposed here not only has the widely convergent property but also find out multiple solutions. The validity of the proposed algorithm for the standard multilayer neural networks is demonstrated through the computer simulations. As a result, it is confirmed that our algorithm is efficient and practical for the learning of the multilayer feedforward neural networks.

This paper describes an efficient second-order algorithm for learning of the multilayer neural networks with widely and stable convergent properties. First, the algorithm based on iterative formula of the steepest descent method, which is "implicitly" employed, is introduced. We show the equivalent property between the Gauss-Newton(GN) method and the "implicit" steepest descent (ISD) method. This means that ISD method satisfy the desired targets by simultaneously combining the merits of the GN and SD techniques in order to enhance the very good properties of SD method. Next, we propose very powerful algorithm for learning multilayer feedforward neural networks, called "implicit" steepest descent with momentum (ISDM) method and show the analogy with the trapezoidal formula in the field of numerical analysis. Finally, the proposed algorithms are compared with GN method for training multilayer neural networks through the computer simulations.

We have proposed a design method of neural networks for limit cycle generator. The proposed method is based on the linear programming method. In this paper, we apply this method to the hysteresis neural networks for limit cycle generator. The hysteresis neuron has two threshold levels. Therefore the state of neuron can be flexibly controlled. As a result, the limit cycle, which can not be designed in the previous work, can be realized by the hysteresis neural networks. The behavior of this network is verified by our specialized simulator. As a result, we can easily design hysteresis neural networks for limit cycle generator.

This paper describes a new supervised learning algorithm for multilayer neural networks without explicit error backpropagation (BP). The proposed method allows the asynchronous and parallel processing by neurons. Therefore this algorithm has an advantage over the standard backpropagation algorithm in hardware implementation of trainable artificial neural networks. We demonstrate the validity of the method through computer simulations. It is shown that the algorithm is not only almost equivalent to the BP algorithm from the viewpoint of the generalization ability, but also much superior to the one from the viewpoint of the convergence speed. As a result, it is confirmed that our algorithm is efficient and practical for the supervised learning of multilayer neural networks.

A design method of neural networks for limit cycle generator is described. First, the constraint conditions for the synaptic weights, which are given by the linear inequalities, are derived from the dynamics of neural networks. Next, the linear inequalities are solved by the linear programming method. The synaptic weights and other parameters are determined by the above solutions. Furthermore, the limit cycle generator is designed by analog neural networks and simulated by Spice. Finally, we confirm that our design method is efficient and practical for the design of neuro-based limit cycle generator.

In this paper, we describe the convergence of the discrete Walsh transform (DWT) processor based on Hopfield neural networks. First, the influence of the orthonormal matrix on solving linear equations by the steepest descent (SD) method is investigated and this theory is applied to the convergence of Hopfield neural networks. Finally, it is shown both analytically and by simulation that this type of network is suitable for orthogonal transforms.

This paper describes a novel algorithm based on Steepest Descent Method (SDM) for solving a system of nonlinear algebraic equations. First, we compare SDM with Newton-Raphson Method (NRM) and propose a novel technique which is derived from the equivalent property between them. The proposed method is an efficient algorithm which not only can overcome drawbacks of NRM but also can exploit the convergence speed of NRM. We refer to this technique as orthogonalized Steepest Descent Method. We demonstrate the validity of the proposed technique for several nonlinear equations and Hopfield neural network analysis.