This study focuses on the Nesterov's accelerated quasi-Newton (NAQ) method in the context of deep neural networks (DNN) and its applications. The thesis objective is to confirm the robustness and efficiency of Nesterov's acceleration to quasi-Netwon (QN) methods by developing practical algorithms for different fields of optimization problems.

Incorporating curvature information in stochastic methods has been a challenging task. This paper proposes a momentum accelerated BFGS quasi-Newton method in both its full and limited memory forms, for solving stochastic large scale non-convex optimization problems in neural networks (NN).

With the rising need for high-performance circuits, electronic design automation has gained significant attention. Analog circuit design optimization is complex mainly due to the large search space and high non-linearity. Neural networks have shown to be effective in solving highly non-linear problems. Training algorithms play an important role in neural networks. Gradient-based algorithms are popularly used in the training of neural networks. They can be broadly classified into first and second-order methods. While first-order methods are simple and low in computational complexity, they are slow in convergence when applied to highly non-linear problems. Incorporating second order curvature information has shown to significantly improve convergence despite its high computational cost. This paper proposes a second order adaptive modified Nesterov's accelerated quasi-Newton method (amNAQ) with an adaptive momentum scheme. The performance of the proposed method is evaluated in transistor sizing of a two stage op-amp. The results indicate that the proposed method can efficiently deduce the size while satisfying the desired specifications much faster compared to the existing popular algorithms.

Recently algorithms incorporating second order curvature information have become popular in training neural networks. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to effectively accelerate the BFGS quasi-Newton method by incorporating the momentum term and Nesterov's accelerated gradient vector. A stochastic version of NAQ method was proposed for training of large-scale problems. However, this method incurs high stochastic variance noise. This paper proposes a stochastic variance reduced Nesterov's Accelerated Quasi-Newton method in full (SVR-NAQ) and limited (SVRLNAQ) memory forms. The performance of the proposed method is evaluated in Tensorflow on four benchmark problems - two regression and two classification problems respectively. The results show improved performance compared to conventional methods.

Recent studies incorporate Nesterov's accelerated gradient method for the acceleration of gradient based training. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to drastically improve the convergence speed compared to the conventional quasi-Newton method. This paper implements NAQ for non-convex optimization on Tensorflow. Two modifications have been proposed to the original NAQ algorithm to ensure global convergence and eliminate linesearch. The performance of the proposed algorithm - mNAQ is evaluated on standard non-convex function approximation benchmark problems and microwave circuit modelling problems. The results show that the improved algorithm converges better and faster compared to first order optimizers such as AdaGrad, RMSProp, Adam, and the second order methods such as the quasi-Newton method.

This paper describes a hybrid implicit-explicit block latency insertion method (HIE-block-LIM) for fast simulation of nonuniform multiconductor transmission lines. In the HIE-block-LIM, an implicit difference method is used with respect to the current variables in one direction, and an explicit method is adopted to update the other variables. The HIE-block-LIM can alleviate a time step size limitation of the existing block-LIM by taking both advantages of the explicit and implicit difference methods.

The equivalent circuit of an on-chip power distribution network (PDN) has a fine 3-D grid structure due to the vias between equipotential conductors, and the vertical couplings between power and ground lines. In addition, a through silicon via is modeled with inductive and capacitive parasitic elements and appended to the PDN. Therefore, the circuit related to the 3-D IC technology tends to be a tightly coupled large network. For the simulation of this type of network, an explicit time marching scheme has an advantage over conventional general-purpose circuit simulators such as SPICE in the computational cost. However, the explicit method has a strict numerical stability condition, which may limit the maximum time step size and increase the total amount of the cost. In this work, we propose the method which is explicit, but stable with no stability condition. Additionally, the proposed unconditionally-stable explicit method is accelerated more by combining with an order reduction technique.

This paper describes a predictor-corrector latency insertion method (LIM) for a fast transient analysis of an ill-constructed circuit. First, the basic LIM algorithm and limitations of the method are described. Next, we propose the predictor-corrector LIM with a large value of fictitious latency for the ill-constructed topologies. Finally, numerical results show that our proposed method is applicable and efficient for the fast simulation of the ill-constructed circuit.

In this paper, we propose a new FDTD (Finite-Difference Time-Domain) method using the alternating-direction explicit (ADE) method for the efficient electromagnetic field simulation. Furthermore, the modified PML (Perfectly Matched Layer) absorbing boundary condition, which is applicable to the proposed new method, is introduced. Finally, The efficiency of the ADE-FDTD method is evaluated by computer simulations.

This paper presents a modeling method for power distribution networks (PDNs) consisting of multilayered power/ground planes of the PCB/Package. Using our proposed method, multiple stacked power/ground plane pairs having holes and apertures can be modeled as an equivalent circuit. The structure of this equivalent circuit is suitable for the Latency Insertion Method (LIM), which is one of the fast transient simulation methods based on the ¿leapfrog¿ algorithm. Numerical results show that the leapfrog algorithm enables a speed-up of 105 and 486 times compared to the linear circuit simulator based on the sparse LU-decomposition and HSPICE, respectively, with the same level of accuracy.

This paper presents generating stable and sparse reluctance/inductance matrix from the inductance matrix which is extracted under insufficient discretization. So far, to generate the sparse reluctance matrix with guaranteed stability, this matrix has to be diagonally dominant M matrix. Hence, the repeated inductance extractions are necessary using a smaller grid size, in order to obtain the well-defined matrix. Alternatively, this paper provides some ideas for generating the sparse reluctance matrix, even if the extracted reluctance matrix is not diagonally dominant M matrix, precisely, the positive off-diagonal elements are even found. This eases the extraction tasks greatly. Furthermore, the sparse inductance matrix is also generated by using the practical and sophisticated double inverse methods, which is useful for the SPICE simulation, since reluctance components are not still supported in SPICE- like simulators.

This paper describes the relaxation technique for the alternating direction implicit finite-difference time-domain (ADI-FDTD) method. The ADI-FDTD method is computationally more complicate than the conventional FDTD and this method requires solving the tridiagonal matrix equation. Thus, this method may require more computational cost than the conventional method due to the large scale tri-diagonal matrix solution corresponding to a large number of meshes. In this paper, relaxation-based solution technique is discussed for the matrix solution and a simple numerical example is shown. As a result, it is confirmed that ADI-FDTD method with the relaxation technique is useful for the electromagnetic field simulation

In this paper, we focus on the verification of the PCB/package power integrity, which becomes very important for the design of state-of-art high speed digital circuits. The simulation of power distribution networks (PDNs) of the PCB/package, which can be modeled as a large number of RLC lumped components, is a time-consuming task for using the conventional circuit simulator, such as SPICE. For this problem, we propose a parallel-distributed time-domain circuit simulation algorithm based on LIM. Furthermore, an effective modeling of frequency-dependencies of the PDNs, such as skin effects and dielectric losses, to solve by LIM is proposed

This paper describes the two-staged Tabu search for the non-slicing floorplan problem using the ordered tree representation called O-tree. The floorplan problem is a part of VLSI layout design problem. Furthermore, we combine ideas from the simulated annealing into the two-staged Tabu search and propose a novel hybrid algorithm for floorplan represented by O-tree. Finally, we demonstrate the validity of two-staged search and hybrid method for MCNC benchmark tests through the computer simulations.

A fast method for timing analysis of large scale RLC networks using the RLCG-MNA formulation, which provides good properties for fast matrix solvers, is presented. The proposed method is faster than INDUCTWISE and more general than the RLP algorithm, where INDUCTWISE and RLP algorithm are known as the state-of-art simulation methods. In the numerical example, good performances of the proposed method are illustrated compared with the previous works

A fast timing analysis of plane circuits via two-layer CNN-based modeling, which is necessary for the solution of power/signal integrity problems in printed circuit boards and packages, is presented. Using the new notation expressed by the two-layer CNN, more than 1500 times faster simulation is achieved, compared with Berkeley SPICE (ngspice). In CNN community, CNNs are generally simulated by explicit numerical integration algorithms such as the forward Euler and Runge-Kutta methods. However, since the system of the two-layer CNN becomes stiff, we cannot analyze the CNN by using an explicit numerical integration algorithm. Hence, to analyze the two-layer CNN and reduce the computational cost, the leapfrog method is introduced in this paper. This procedure would open a CNN application up to electronic design automation area

A three-dimensional chip design strategy is of increasing significance in association with miniaturization of VLSI circuits. In the design of 3D VLSI and 3D MCM, the placement which is to locate given 3D objects without overlapping each other in the 3D space, becomes an important issue. The 3D placement under the nonoverlapping constraint is called the 3D packing in which minimizing the volume of the package is required. This paper introduces a new coding system to encode the topology of 3D packing as an extension of BSG (bounded-sliceline grid) for 2D packing. The novel coding system is referred to as three dimensional bounded-sliceplane grid (3DBSG). The topology is a set of relative relations assigned to pairs of 3D modules in such a way that one module shall be either "right-of", "rear-of' or "above" the other. Furthermore, we propose an idea to handle L-shaped 3D modules on the 3DBSG. The simulation results in which a standard simulated annealing is utilized as an optimization algorithm, are demonstrated in order to test the validity of our 3D packing method based on the 3DBSG.

This paper describes an efficient learning algorithm based on fixed-point homotopy method. The proposed algorithm has the ability to train the neural networks with high success rates for the initial guesses compared with other typical second-order training algorithms. Furthermore, the method proposed here not only has the widely convergent property but also find out multiple solutions. The validity of the proposed algorithm for the standard multilayer neural networks is demonstrated through the computer simulations. As a result, it is confirmed that our algorithm is efficient and practical for the learning of the multilayer feedforward neural networks.

The closed-form expressions of distributed RLC interconnects are proposed for analysis of on-chip inductance effects in order to insert optimally the repeaters. The transfer function of a circuit with driver-interconnect-load structure is approximated by the 5th order rational functions. The step responses computed by using the proposed expressions give the good agreement with the SPICE simulations.

In this paper, we propose a new method which makes transient simulation faster for the circuit including both nonlinear and linear elements. First, the method for generating the projection matrix with krylov-subspace technique is described. The order of the circuit equation is reduced by congruence transformation with the projection matrix. Next, we suggest a method which can calculate the reduced Jacobian matrix directly in the each Newton-Raphson iteration. Since this technique does not need to calculate the original size of Jacobian matrix, the calculation cost is reduced drastically. Therefore, efficient circuit simulation can be achieved. Finally, our method is applied to some example circuits: and the validity of the nonlinear circuit reduction technique is verified.

This paper presents a method for modeling EM devices, where sample data obtained by numerical EM solver is approximated into a rational matrix of complex s. The model is described in verilog-AMS, thus, the EM devices can be simulated with the digital/circuit mixed circuits described at various abstraction levels. To generate the model, the selective orthogonal matrix least-squares method is presented. The computational efficiency of the proposed approach is confirmed on a commercial simulator, compared with the numerical EM method.

This paper describes an efficient second-order algorithm for learning of the multilayer neural networks with widely and stable convergent properties. First, the algorithm based on iterative formula of the steepest descent method, which is "implicitly" employed, is introduced. We show the equivalent property between the Gauss-Newton(GN) method and the "implicit" steepest descent (ISD) method. This means that ISD method satisfy the desired targets by simultaneously combining the merits of the GN and SD techniques in order to enhance the very good properties of SD method. Next, we propose very powerful algorithm for learning multilayer feedforward neural networks, called "implicit" steepest descent with momentum (ISDM) method and show the analogy with the trapezoidal formula in the field of numerical analysis. Finally, the proposed algorithms are compared with GN method for training multilayer neural networks through the computer simulations.

We have proposed a design method of neural networks for limit cycle generator. The proposed method is based on the linear programming method. In this paper, we apply this method to the hysteresis neural networks for limit cycle generator. The hysteresis neuron has two threshold levels. Therefore the state of neuron can be flexibly controlled. As a result, the limit cycle, which can not be designed in the previous work, can be realized by the hysteresis neural networks. The behavior of this network is verified by our specialized simulator. As a result, we can easily design hysteresis neural networks for limit cycle generator.

Conventional face image recognition has been done based on the extraction techniques of many features such as shapes and relations among the positions of eyes, mouth, nose and so on. However, it is difficult to extract many features because the human face has an unclear shadow. In addition, the extracted features resemble each other among face images. Therefore, face images recognition for many persons is difficult. In this research, we try to recognize face images for many persons by using a multi-layer neural network and 2-dimensional discrete Walsh transform. This face image recognition is also able to discriminate unlearnt face images.

This paper describes a neuro-based optimization algorithm for 2D tiling problems which are to pack a checkerboard with polyominoes. First, we review the previous neuro-based parallel algorithms for filing problems without rotation. Next, we expand the algorithm, which has been proposed for 2D tiling problems without rotation, to the optimization one for 2D tiling problems with rotation, where a novel nonlinear energy function is defined. Finally, we show the simulation results and the estimation.

A design method of neural networks for limit cycle generator is described. First, the constraint conditions for the synaptic weights, which are given by the linear inequalities, are derived from the dynamics of neural networks. Next, the linear inequalities are solved by the linear programming method. The synaptic weights and other parameters are determined by the above solutions. Furthermore, the limit cycle generator is designed by analog neural networks and simulated by Spice. Finally, we confirm that our design method is efficient and practical for the design of neuro-based limit cycle generator.

In this paper, we describe the convergence of the discrete Walsh transform (DWT) processor based on Hopfield neural networks. First, the influence of the orthonormal matrix on solving linear equations by the steepest descent (SD) method is investigated and this theory is applied to the convergence of Hopfield neural networks. Finally, it is shown both analytically and by simulation that this type of network is suitable for orthogonal transforms.

This paper describes a novel algorithm based on Steepest Descent Method (SDM) for solving a system of nonlinear algebraic equations. First, we compare SDM with Newton-Raphson Method (NRM) and propose a novel technique which is derived from the equivalent property between them. The proposed method is an efficient algorithm which not only can overcome drawbacks of NRM but also can exploit the convergence speed of NRM. We refer to this technique as orthogonalized Steepest Descent Method. We demonstrate the validity of the proposed technique for several nonlinear equations and Hopfield neural network analysis.

A new algorithm, which is incorporated into the waveform relaxation analysis, to compute the transient response of coupled lossy transmission lines is presented. First, coupled lines are transformed into uncoupled lines by a transformation network. Second, each uncoupled line is modelled into an equivalent disjoint two-port network. This model, called characteristic model, consists of characteristic impedances and voltage generators in each branch. Synthesized networks are used to represent characteristic impedances and attenuation functions. Finally, the circuit is partitioned into two subcircuits and waveform relaxation analysis is carried out in sequential time windows of size equal to the delay of an uncoupled line with fastest propagation velocity. By this approach, the near end and far end subcircuits are totally decoupled within each window and as a result converged waveforms are obtained with a single iteration.>

This paper presents relaxation-based algorithms with the dynamic partitioning technique for bipolar circuit analysis. In this technique, a circuit is partitioned dynamically based on the consideration of the operating region of specified bipolar devices throughout the simulation interval. This technique has been used already in the waveform relaxation method, and, in this paper, is implemented in the Iterated Timing Analysis (ITA). The present method is applied to the transient simulation of several digital bipolar junction transistor (BJT) circuits and compared with the waveform relaxation method.>

This paper proposes a mixed mode simulation using dynamic network separation and selective trace. Since the separation technique enables the analysis of each subcircuit independently, it is possible to skip solving the latent subcircuits according to selective trace, though it is based on the direct method. Selecting the time points in accordance with activity of each subcircuit is analogous to multirate numerical integration technique used in the waveform relaxation algorithm. As a result, the matured direct/relaxation-based mixed mode simulator, including a variety of skilful techniques, is realized.>

This paper describes steady-state analysis of single- and multi-conductor transmission lines in frequency domain. First, we present Relaxation Harmonic Newton, where the harmonic balance method is applied to each node. Second, we apply a characteristic model to transmission lines for relaxation technique. Finally, we apply the present technique to the simple circuit simulation and verify its availability.>