This study focuses on the Nesterov's accelerated quasi-Newton (NAQ) method in the context of deep neural networks (DNN) and its applications. The thesis objective is to confirm the robustness and efficiency of Nesterov's acceleration to quasi-Netwon (QN) methods by developing practical algorithms for different fields of optimization problems.

Incorporating curvature information in stochastic methods has been a challenging task. This paper proposes a momentum accelerated BFGS quasi-Newton method in both its full and limited memory forms, for solving stochastic large scale non-convex optimization problems in neural networks (NN).

With the rising need for high-performance circuits, electronic design automation has gained significant attention. Analog circuit design optimization is complex mainly due to the large search space and high non-linearity. Neural networks have shown to be effective in solving highly non-linear problems. Training algorithms play an important role in neural networks. Gradient-based algorithms are popularly used in the training of neural networks. They can be broadly classified into first and second-order methods. While first-order methods are simple and low in computational complexity, they are slow in convergence when applied to highly non-linear problems. Incorporating second order curvature information has shown to significantly improve convergence despite its high computational cost. This paper proposes a second order adaptive modified Nesterov's accelerated quasi-Newton method (amNAQ) with an adaptive momentum scheme. The performance of the proposed method is evaluated in transistor sizing of a two stage op-amp. The results indicate that the proposed method can efficiently deduce the size while satisfying the desired specifications much faster compared to the existing popular algorithms.

Quantum particles possess both wave and particle natures. When quantum effect devices exploiting the wave nature and the devices described from the view point of particle nature coexist in a system, the following advantages are gained by representing former devices in the same way as we describe the latter devices: First, it is not necessary to compute probability distribution from wave functions and sample device states in accordance with the probability distribution. Secondly, the particle models of the devices are suitable models for conventional circuit simulators. In this paper, simple particle models of two quantum systems operating as wave filters are constructed by applying Nelson's stochastic quantization theory. The filter characteristics obtained from the wave functions and from the behavior of the particle models were almost equal.

Quantum particles possess both wave and particle natures. When quantum effect devices exploiting the wave nature and the devices described from the view point of particle nature coexist in a system, the following advantages are gained by representing former devices in the same way as we describe the latter devices: First, it is not necessary to compute probability distribution from wave functions and sample device states in accordance with the probability distribution. Secondly, the particle models of the devices are suitable models for conventional circuit simulators. In this paper, simple particle models of an electromagnetic wave detector and an electron lens are constructed by applying Nelson's stochastic quantization theory. The device characteristics obtained from the wave functions and from the behavior of the particle models were almost equal.

Cellular neural network (CNNs) with the space-invariant cloning template are suitable for analog hardware implementation because the sparse interconnected structure decreases the number of the synaptic circuits and each neuron has the identical set of weights. However, this condition restricts very much the capability of CNNs for associative memories. In this paper, we propose associative memories using CNNs with the space-invariant cloning template library to overcome this problem.

At present, we have proposed associative memories using multilayer perceptrons (MLPs) and sparsely interconnected neural networks (SINNs), named MLP-SINN, to improve SINNs without increasing their interconnections. MLP-SINN is more suitable for hardware implementation than SINN with a large number of interconnections. However, the capabilities of MLP and SINN are not effectively used in the conventional MLP-SINN, because they are synthesized independently. In this paper, we propose the noise supplement learning algorithm to improve MLP-SINN associative memories.

Multilayer feedforward neural network (MFNN) trained by the backpropagation (BP) algorithm is one of the most significant models in artificial neural networks. Although they have been implemented as analog, mixed analog-digital and fully digital VLSI circuits, it is still difficult to realize their hardware implementation with BP learning function. This paper describes the BP algorithm for the logic oriented neural network (LOGO-NN) which we have proposed as a kind of MFNN with quantized weights and multilevel threshold neurons. Since both weights and neuron outputs are quantized to integer values in LOGO-NNs, it is expected that LOGO-NNs with BP learning can be more effectively implemented than the common MFNNs. Finally, it is shown by simulations that the proposed BP algorithm has good performance for LOGO-NNs.

Conventional face image recognition has been done based on the extraction techniques of many features such as shapes and relations among the positions of eyes, mouth, nose and so on. However, it is difficult to extract many features because the human face has an unclear shadow. In addition, the extracted features resemble each other among face images. Therefore, face images recognition for many persons is difficult. In this research, we try to recognize face images for many persons by using a multi-layer neural network and 2-dimensional discrete Walsh transform. This face image recognition is also able to discriminate unlearnt face images.

In this paper, we describe the convergence of the discrete Walsh transform (DWT) processor based on Hopfield neural networks. First, the influence of the orthonormal matrix on solving linear equations by the steepest descent (SD) method is investigated and this theory is applied to the convergence of Hopfield neural networks. Finally, it is shown both analytically and by simulation that this type of network is suitable for orthogonal transforms.