Abstract In this journal, Cheng has proposed a backpropagation (BP) procedure called BPFCC for deep fully connected cascaded (FCC) neural network learning in comparison with a neuron-by-neuron (NBN) algorithm of Wilamowski and Yu. Both BPFCC and NBN are designed to implement the Levenberg-Marquardt method, which requires an efficient evaluation of the Gauss-Newton (approximate Hessian) matrix $$\nabla \textbf{r}^\textsf{T} \nabla \textbf{r}$$ ∇ r T ∇ r , the cross product of the Jacobian matrix $$\nabla \textbf{r}$$ ∇ r of the residual vector $$\textbf{r}$$ r in nonlinear least squares sense. Here, the dominant cost is to form $$\nabla \textbf{r}^\textsf{T} \nabla \textbf{r}$$ ∇ r T ∇ r by rank updates on each data pattern. Notably, NBN is better than BPFCC for the multiple $$q~\!(>\!1)$$ q ( > 1 ) -output FCC-learning when q rows (per pattern) of the Jacobian matrix $$\nabla \textbf{r}$$ ∇ r are evaluated; however, the dominant cost (for rank updates) is common to both BPFCC and NBN. The purpose of this paper is to present a new more efficient stage-wise BP procedure (for q-output FCC-learning) that reduces the dominant cost with no rows of $$\nabla \textbf{r}$$ ∇ r explicitly evaluated, just as standard BP evaluates the gradient vector $$\nabla \textbf{r}^\textsf{T} \textbf{r}$$ ∇ r T r with no explicit evaluation of any rows of the Jacobian matrix $$\nabla \textbf{r}$$ ∇ r .

We describe how to alleviate the problem of plateau phenomena that may arise in learning with a CANFIS neuro-fuzzy modular network. The network model consists of multiple “local-expert” MLPs (multilayer perceptrons) mediated by fuzzy membership functions. Even with such a complex modular architecture, our recently-developed second-order stagewise backpropagation procedure efficiently evaluates the Hessian matrix of a given objective function, for which we employ the sum-squared-error measure. For concreteness, we use a small curve-fitting problem that allows us to demonstrate detailed analysis based on the Hessian matrix. In particular, we describe how to use a block-diagonal approximate local Hessian matrix associated with a bottleneck local-expert MLP that does not perform very well for a designated task. Since the “bad” performance implies relatively large residuals (or errors), the local Hessian matrix tends to be indefinite; therefore, it is worth exploiting the negative curvature to escape from the plateaus.

On-line first-order backpropagation (BP) has been widely employed for optimizing a multi-layer neural network and a fuzzy neural network. When BP is applied to a TSK fuzzy system, the interpretability of fuzzy rules may be lost. We describe a simple and effective remedy for the loss by casting the posed optimization problem into a general Bolza-type optimal-control mold so as to include stage costs on top of the terminal cost; this is what we called the (on-line) hidden-node teaching. Our on-line learning scheme turns out to be useful not only in optimizing a large model, a so-called CANFIS neuro-fuzzy modular network, but also in enhancing its generalization capacity. The fundamental concept is that each (local) expert network is supervised individually by hidden-node teaching for minimizing a stage cost while all local-expert modules are encouraged to cooperate in reducing the terminal cost simultaneously. Furthermore, we show how to construct a CANFIS modular network in order to alleviate the so-called curse of dimensionality that frequently hampers the design of fuzzy systems. In simulation, our concepts have been demonstrated in the letter recognition benchmark problem as well as in small regression and XOR-classiflcation tasks.

In numerical linear algebra, the variable projection (VP) algorithm has been a standard approach to separable “mixed” linear and nonlinear least squares problems since early 1970s. Such a separable case often arises in diverse contexts of machine learning (e.g., with generalized linear discriminant functions); yet VP is not fully investigated in the literature. We thus describe in detail its implementation issues, highlighting an economical trust-region implementation of VP in the framework of a so-called block-arrow least squares (BA) algorithm for a general multiple-response nonlinear model. We then present numerical results using an exponential-mixture benchmark, seven-bit parity, and color reproduction problems; in some situations, VP enjoys quick convergence and attains high classification rates, while in some others VP works poorly. This observation motivates us to investigate original VP's strengths and weaknesses compared with other (full-functional) approaches. To overcome the limitation of VP, we suggests how VP can be modified to be a Hessian matrix-based approach that exploits negative curvature when it arises. For this purpose, our economical BA algorithm is very useful in implementing such a modified VP especially when a given model is expressed in a multi-layer (neural) network for efficient Hessian evaluation by the so-called second-order stagewise backpropagation.

In the neural-network parameter space, an attractive field is likely to be induced by singularities. In such a singularity region, first-order gradient learning typically causes a long plateau with very little change in the objective function value E (hence, a flat region). Therefore, it may be confused with ``attractive'' local minima. Our analysis shows that the Hessian matrix of E tends to be indefinite in the vicinity of (perturbed) singular points, suggesting a promising strategy that exploits negative curvature so as to escape from the singularity plateaus. For numerical evidence, we limit the scope to small examples (some of which are found in journal papers) that allow us to confirm singularities and the eigenvalues of the Hessian matrix, and for which computation using a descent direction of negative curvature encounters no plateau. Even for those small problems, no efficient methods have been previously developed that avoided plateaus.

We analyze the Hessian matrix H of the sum-squared-error measure for multilayer-perceptron (MLP) learning, showing the following intriguing results: At an early stage of learning, H is indefinite. The indefiniteness is related to the MLP structure, which also determines rank of H (per datum). Exploiting negative curvature leads to efficient learning algorithms. H can be much less ill-conditioned than the Gauss-Newton Hessian JTJ. All these new findings are obtained by our stagewise second-order backpropagation; the procedure exploits MLP's "layered symmetry" to evaluate H quickly, making exact Hessian evaluation feasible for fairly large practical problems. In fact, it works faster than rank-update methods that evaluate only JTJ. It also serves to appraise other existing learning algorithms that perform implicit Hessian-vector multiply.

The classicalB-bit parity two-class pattern classification problem can be solved by a multilayer perceptron (MLP) with onlyH ≡ [(B+1)/2]hidden nodes. By our "symmetric" arguments, we show the following: (1)Hhidden nodes may consist of one "linear" and(H-1)binary-valued step-functions (i.e., McCulloch-Pitts neurons) for a facile solution (of integer-valued weights); (2) the posed model can be transformed to a well-known triangular-shaped network with(H-1)hidden nodes having direct connections fromBinputs to the terminal node; (3) "weight-sharing" simplifies the structure and thus makes it easier to find a solution; (4) it is possible to find inO(H)an optimal set of weights for an MLP withH"tanh" hidden nodes; and (5) a new scheme is designed to supply desired outputs to at most(H-1)hidden nodes for developing insensitivity to initial weights. These findings concerning how sigmoid hidden nodes get saturated for solution are closely related to plateau phenomena that often occur in MLP-learning on parity. Since the phenomena are related to the saddle-point issue, we investigated the indefiniteness of the Hessian matrix H of the sum-squared-error measure: When H is indefinite, a learning algorithm that exploits negative curvature can efficiently find a separating hyperplane by moving away from the nearest saddle point.

We present a simple, intuitive argument based on "invariant imbedding" in the spirit of dynamic programming to derive a stagewise second-order backpropagation (BP) algorithm. The method evaluates the Hessian matrix of a general objective function efficiently by exploiting the multistage structure embedded in a given neural-network model such as a multilayer perceptron (MLP). In consequence, for instance, our stagewise BP can compute the full Hessian matrix "faster" than the standard method that evaluates the Gauss-Newton Hessian matrix alone by rank updates in nonlinear least squares learning. Through our derivation, we also show how the procedure serves to develop advanced learning algorithms; in particular, we explain how the introduction of "stage costs" leads to alternative systematic implementations of multi-task learning and weight decay.

Recent advances in computer technology allows the implementation of some important methods that were assigned lower priority in the past due to their computational burdens. Second-order backpropagation (BP) is such a method that computes the exact Hessian matrix of a given objective function. We describe two algorithms for feed-forward neural-network (NN) learning with emphasis on how to organize Hessian elements into a so-called stagewise-partitioned block-arrow matrix form: (1) stagewise BP, an extension of the discrete-time optimal-control stagewise Newton of Dreyfus 1966; and (2) nodewise BP, based on direct implementation of the chain rule for differentiation attributable to Bishop 1992. The former, a more systematic and cost-efficient implementation in both memory and operation, progresses in the same layer-by-layer (i.e., stagewise) fashion as the widely-employed first-order BP computes the gradient vector. We also show intriguing separable structures of each block in the partitioned Hessian, disclosing the rank of blocks.

We describe two stochastic non-Markovian dynamic programming (DP) problems, showing how the posed problems can be attacked by using actor-critic reinforcement learning with recurrent neural networks (RNN). We assume that the current state of a dynamical system is "completely observable", but that the rules, unknown to our decision-making agent, for the current reward and state transition depend not only on current state and action, but on possibly the "entire history" of past states and actions. This should not be confused with problems of "partially observable Markov decision processes (POMDPs)", where the current state is only deduced from either partial (observable) state alone or error-corrupted observations. Our actor-critic RNN agent is capable of finding an optimal policy, while learning neither transitional probabilities, associated rewards, nor by how much the current state space must be augmented so that the Markov property holds. The RNN's recurrent connections or context units function as an "implicit" history memory (or internal state) to develop "sensitivity" to non-Markovian dependencies, rendering the process Markovian implicitly and automatically in a "totally model-free" fashion. In particular, using two small-scale longest-path problems in a stochastic non-Markovian setting, we discuss model-free learning features in comparison with the model-based approach by the classical DP algorithm.

This paper highlights nonlinear least squares algorithms with trust-region regularization for multiple-output neural-network (NN) models, describing how special structures of the "block-angular" residual Jacobian matrix and the "block-arrow" Gauss-Newton Hessian (or Fisher information matrix) can be exploited to render a large class of NN-learning algorithms "efficient" in both memory and operation counts. In simulation, we demonstrate both direct and iterative trust-region algorithms with two distinct nonlinear models: "multilayer perceptrons (MLP)" and "complementary mixtures of NN-experts" (or neuro-fuzzy modular networks) using a relatively large real-world nonlinear regression application.

This paper describes our empirical study in neuro-fuzzy modeling for a real-world nonlinear regression application "multi-illuminant color reproduction for electric cameras", investigating a variety of neuro-fuzzy architectures. We explain all those models in the complementary modular network framework, demonstrating several representative models, and discuss strengths and weaknesses of individual models. In particular, we emphasize usefulness of neuro-fuzzy modular networks especially when the posed regression task requires multiple inputs and outputs with a small number of training data due to efficient practical implementation.

We describe color reproduction and correction of images captured by electronic cameras under multiple illumination (or lighting) conditions, relating to color device characterization for enhancing the quality of color in the obtained images. In particular, we highlight a very practical use of neuro-fuzzy modular network coactive neuro-fuzzy inference systems (CANFIS) models for this application, and discuss their strengths and weaknesses compared with other adaptive network models (e.g., multilayer perceptron (MLP)) as well as conventional lookup-table-type (TRC-matrix) methods. Our in-depth investigation based on comprehensive numerical tests with a wide variety of illumination/lighting data (180 sources of illumination) shows that the "neuro-fuzzy CANFIS with MLP local experts" possesses a remarkable generalization/approximation capacity, even under a very restricted condition where only four-illuminant data sets were permitted to be used for optimization because of efficient practical implementation subject to an industrial setting.

The well-known backpropagation (BP) derivative computation process for multilayer perceptrons (MLP) learning can be viewed as a simplified version of the Kelley-Bryson gradient formula in the classical discrete-time optimal control theory. We detail the derivation in the spirit of dynamic programming, showing how they can serve to implement more elaborate learning whereby teacher signals can be presented to any nodes at any hidden layers, as well as at the terminal output layer. We illustrate such an elaborate training scheme using a small-scale industrial problem as a concrete example, in which some hidden nodes are taught to produce specified target values. In this context, part of the hidden layer is no longer "hidden".

This paper describes a method of dogleg trust-region steps, or re(cid:173) stricted Levenberg-Marquardt steps, based on a projection pro(cid:173) cess onto the Krylov subspaces for neural networks nonlinear least squares problems. In particular, the linear conjugate gradient (CG) method works as the inner iterative algorithm for solving the lin(cid:173) earized Gauss-Newton normal equation, whereas the outer nonlin(cid:173) ear algorithm repeatedly takes so-called "Krylov-dogleg" steps, re(cid:173) lying only on matrix-vector multiplication without explicitly form(cid:173) ing the Jacobian matrix or the Gauss-Newton model Hessian. That is, our iterative dogleg algorithm can reduce both operational counts and memory space by a factor of O(n) (the number of pa(cid:173) rameters) in comparison with a direct linear-equation solver. This memory-less property is useful for large-scale problems.

This paper highlights an evolutionary computing intelligence for a computerized color recipe prediction that requires function approximation and combinatorial solution of colorants to produce color recipes for a given target color sample. We attack this real challenging problem in the color (paint) industry by using an evolutionary computing system that consists of a problem-specific knowledge and three principal constituents of soft-computing: neural networks, a fuzzy system, and a genetic algorithm. Departing from the recipe results obtained by neural networks (NN) approaches, the evolutionary system attempts to improve them in conjunction with fuzzy classification, a knowledge base and neural fitness functions. All components function synergistically in obtaining precise color recipe outputs through simulation of color paint manufacturing process. Such computational intelligence can be useful, especially when an exact mathematical model of the real-world process under consideration is not available explicitly.

This paper highlights Powell's dogleg trust-region algorithms with self-scaling quasi-Newton Hessian augmentation for neural-network (NN) nonlinear least squares problems. The dogleg algorithms approximate a restricted Levenberg-Marquardt step within the trust region of the local quadratic model in a piecewise-linear fashion. Furthermore, the second-derivative term of the Hessian is approximated by quasi-Newton iteration to obtain augmented Gauss-Newton model Hessian, which may be useful for highly nonlinear residuals when starting with a poor initial point (i.e., randomly initialized weight parameters). By small-scale examples, we illustrate how those devices come into play as a promising NN learning algorithm.

This paper highlights a sample-path-based policy-only learning algorithm for regenerative (stochastic) processes proposed by Marbach and Tsitsiklis (1998). The algorithm attempts to optimize a randomized, parameterized policy according to the average cost criteria in conjunction with the so-called infinitesimal perturbation analysis gradient estimation technique. We present our numerical studies, demonstrating this learning algorithm using small-scale problems; in particular, the parametrized policy-only agent is a neural network function approximator in the spirit of neuro-dynamic programming (or reinforcement learning).