In this paper we present three neurocontrol problems where the analytic policy gradient via back-propagation through time is used to train a simulated agent to maximise a polynomial reward function in a simulated environment. If the environment includes terminal barriers (e.g. solid walls) which terminate the episode whenever the agent touches them, then we show learning can get stuck in oscillating limit cycles, or local minima. Hence we propose to use fixed-length trajectories, and change these barriers into soft barriers, which the agent may pass through, while incurring a significant penalty cost. We demonstrate that the presence of soft barriers can have the drawback of causing exploding learning gradients. Furthermore, the strongest learning gradients often appear at inappropriate parts of the trajectory, where control of the system has already been lost. When combined with modern adaptive optimisers, this combination of exploding gradients and inappropriate learning often causes learning to grind to a halt. We propose ways to avoid these difficulties; either by careful gradient clipping, or by smoothly truncating the gradients of the soft barriers' polynomial cost functions. We argue that this enables the learning algorithm to avoid exploding gradients, and also to concentrate on the most important parts of the trajectory, as opposed to parts of the trajectory where control has already been irreversibly lost.

This paper introduces CBFkit, a Python/ROS toolbox for safe robotics planning and control under uncertainty. The toolbox provides a general framework for designing control barrier functions for mobility systems within both deterministic and stochastic environments. It can be connected to the ROS open-source robotics middleware, allowing for the setup of multi-robot applications, encoding of environments and maps, and integrations with predictive motion planning algorithms. Additionally, it offers multiple CBF variations and algorithms for robot control. The CBFKit is demonstrated on the Toyota Human Support Robot (HSR) in both simulation and in physical experiments.

This article introduces a model-based approach for training feedback controllers for an autonomous agent operating in a highly non-linear (albeit deterministic) environment. We desire the trained policy to ensure that the agent satisfies specific task objectives and safety constraints, both expressed in Discrete-Time Signal Temporal Logic (DT-STL). One advantage for reformulation of a task via formal frameworks, like DT-STL, is that it permits quantitative satisfaction semantics. In other words, given a trajectory and a DT-STL formula, we can compute the robustness , which can be interpreted as an approximate signed distance between the trajectory and the set of trajectories satisfying the formula. We utilize feedback control, and we assume a feed forward neural network for learning the feedback controller. We show how this learning problem is similar to training recurrent neural networks (RNNs), where the number of recurrent units is proportional to the temporal horizon of the agent’s task objectives. This poses a challenge: RNNs are susceptible to vanishing and exploding gradients, and naïve gradient descent-based strategies to solve long-horizon task objectives thus suffer from the same problems. To address this challenge, we introduce a novel gradient approximation algorithm based on the idea of dropout or gradient sampling. One of the main contributions is the notion of controller network dropout , where we approximate the NN controller in several timesteps in the task horizon by the control input obtained using the controller in a previous training step. We show that our control synthesis methodology can be quite helpful for stochastic gradient descent to converge with less numerical issues, enabling scalable back-propagation over longer time horizons and trajectories over higher-dimensional state spaces. We demonstrate the efficacy of our approach on various motion planning applications requiring complex spatio-temporal and sequential tasks ranging over thousands of timesteps.

Most deep neural network (DNN) verification research focuses on qualitative verification, which answers whether or not a DNN violates a safety/robustness property. This paper proposes an approach to convert qualitative verification into quantitative verification for neural networks. The resulting quantitative verification method not only can answer YES or NO questions but also can compute the probability of a property being violated. To do that, we introduce the concept of a probabilistic star (or shortly ProbStar), a new variant of the well-known star set, in which the predicate variables belong to a Gaussian distribution and propose an approach to compute the probability of a probabilistic star in high-dimensional space. Unlike existing works dealing with constrained input sets, our work considers the input set as a truncated multivariate normal (Gaussian) distribution, i.e., besides the constraints on the input variables, the input set has a probability of the constraints being satisfied. The input distribution is represented as a probabilistic star set and is propagated through a network to construct the output reachable set containing multiple ProbStars, which are used to verify the safety or robustness properties of the network. In case of a property is violated, the violation probability can be computed precisely by an exact verification algorithm or approximately by an overapproximate verification algorithm. The proposed approach is implemented in a tool named StarV and is evaluated using the well-known ACASXu networks and a rocket landing benchmark.

The paper extends the recent star reachability method to verify the robustness of recurrent neural networks (RNNs) for use in safety-critical applications. RNNs are a popular machine learning method for various applications, but they are vulnerable to adversarial attacks, where slightly perturbing the input sequence can lead to an unexpected result. Recent notable techniques for verifying RNNs include unrolling, and invariant inference approaches. The first method has scaling issues since unrolling an RNN creates a large feedforward neural network. The second method, using invariant sets, has better scalability but can produce unknown results due to the accumulation of overapproximation errors over time. This paper introduces a complementary verification method for RNNs that is both sound and complete. A relaxation parameter can be used to convert the method into a fast overapproximation method that still provides soundness guarantees. The method is designed to be used with NNV, a tool for verifying deep neural networks and learning-enabled cyber-physical systems. Compared to state-of-the-art methods, the extended exact reachability method is 10 × faster, and the overapproximation method is 100 × to 5000 × faster.

We propose a novel class of risk-aware control barrier functions (RA-CBFs) for the control of stochastic safety-critical systems. Leveraging a result from the stochastic level-crossing literature, we deviate from the martingale theory that is currently used in stochastic CBF techniques and prove that a RA-CBF based control synthesis confers a tighter upper bound on the probability of the system becoming unsafe within a finite time interval than existing approaches. We highlight the advantages of our proposed approach over the state-of-the-art via a comparative study on an mobile-robot example, and further demonstrate its viability on an autonomous vehicle highway merging problem in dense traffic.

Deep Neural Networks (DNNs) are powerful machine learning models for approximating complex functions. In this work, we provide an exact reachability analysis method for DNNs with Rectified Linear Unit (ReLU) activation functions. At its core, our set-based method utilizes a facet-vertex incidence matrix, which represents a complete encoding of the combinatorial structure of convex sets. When a safety violation is detected, our approach provides backtracking which determines the complete input set that caused the safety violation. The performance of our method is evaluated and compared to other state-of-the-art methods by using the ACAS Xu flight controller and other benchmarks.

Sampling-based methods such as Rapidly-exploring Random Trees (RRTs) have been widely used for generating motion paths for autonomous mobile systems. In this work, we extend time-based RRTs with Control Barrier Functions (CBFs) to generate, safe motion plans in dynamic environments with many pedestrians. Our framework is based upon a human motion prediction model which is well suited for indoor narrow environments. We demonstrate our approach on a high-fidelity model of the Toyota Human Support Robot navigating in narrow corridors. We show in simulation results that our proposed online method can navigate safely in the presence of moving agents with unknown dynamics.

Recent developments in simulation-based testing methods for automotive systems with machine learning components have shown promise. This work in progress paper presents our efforts in applying these methods in the evaluation and development of control and perception systems. Experimental results demonstrate a significant improvement in system performance.

Verification and validation are vital steps in the development process of autonomous systems such as mobile robots and self-driving vehicles, as they allow reasoning about system safety. In the domain of cyber-physical systems, techniques using formal requirements have been show to enable rigorous mathematical reasoning about system safety through techniques for automatic test generation and performance analysis. In this paper, we show that system-level and subsystem-level requirements can also enable fault localization in autonomous systems that use heterogeneous functional components. However, writing correct formal requirements is challenging and requires a significant investment of time, effort and most importantly, expertise. To address this issue, we propose a specification library for autonomous mobile systems called TLAM (Temporal Logic for Autonomous Mobility). Our contributions are twofold: We provide a library of parametric formal specifications at both the system-level and subsystem-level for typical subsystems in autonomous systems such as those for perception, planning and decision-making. The specification parameters encode the design trade-offs for such components. Second, we introduce a new fault localization technique based on these parametric specifications that identifies the likeliest subsystem that has a fault.

It is important that automotive engineers understand the interactions between active human maneuvering motions and vehicle dynamics, and how vehicle control affects the physical sensations of the human driver. This paper proposes a new system framework, the human model-based active driving system (HuMADS) for simulating human driver-vehicle interactions. HuMADS integrates the vehicle controller with models of vehicle dynamics and human biomechanics. It has an hierarchical closed-loop architecture for driver-vehicle control systems, including structures and contact interfaces of human and vehicle bodies. HuMADS is based on the OpenSim simulation platform. The developed system regulates the human model dynamics, such that the human model can react realistically to vehicle maneuver motions. The usability of the HuMADS is demonstrated through the simulation of coordinated gas/brake pedal operation and wheel-steering in highway driving tasks. The simulated vehicle dynamics and vehicle maneuvers are comparable with previously published experimental data of car-following driving. In addition, the proposed controllers successfully maintain the human body's balance inside the vehicle during vehicle maneuvers. We are convinced that the HuMADS has potential as a tool for the development of intelligent transportation systems and investigation of integrated safety.

Pavement crack detection is a critical task for insuring road safety. Manual crack detection is extremely time-consuming. Therefore, an automatic road crack detection method is required to boost this progress. However, it remains a challenging task due to the intensity inhomogeneity of cracks and complexity of the background, e.g., the low contrast with surrounding pavements and possible shadows with a similar intensity. Inspired by recent advances of deep learning in computer vision, we propose a novel network architecture, named feature pyramid and hierarchical boosting network (FPHBN), for pavement crack detection. The proposed network integrates context information to low-level features for crack detection in a feature pyramid way, and it balances the contributions of both easy and hard samples to loss by nested sample reweighting in a hierarchical way during training. In addition, we propose a novel measurement for crack detection named average intersection over union (AIU). To demonstrate the superiority and generalizability of the proposed method, we evaluate it on five crack datasets and compare it with the state-of-the-art crack detection, edge detection, and semantic segmentation methods. The extensive experiments show that the proposed method outperforms these methods in terms of accuracy and generalizability. Code and data can be found in https://github.com/fyangneil/pavement-crack-detection.

There is recent interest in using deep neural networks (DNNs) for controlling autonomous cyber-physical systems (CPSs). One challenge with this approach is that many autonomous CPS applications are safety-critical, and is not clear if DNNs can proffer safe system behaviors. To address this problem, we present an approach to modify existing (deep) reinforcement learning algorithms to guide the training of those controllers so that the overall system is safe. We present a novel verification-in-the-loop training algorithm that uses the formalism of barrier certificates to synthesize DNN-controllers that are safe by design. We demonstrate a proof-of-concept evaluation of our technique on multiple CPS examples.

I discuss a growing area of research in autonomous driving systems and overview what this means in terms of their testing. I also discuss implications for autonomous decision making systems of the future. (This paper is expected to serve as the basis of my plenary talk at IJCNN 2019.).

Differential features are popularly used in computer vision tasks, such as object detection. In this paper, we revisit these features from a functional approximation perspective. In particular, we view an image as a 2-D functional and investigate its Taylor series approximation. Differential features are derived from the approximation coefficients and, therefore, are naturally collected for appearance representation. Thus motivated, we propose to use the zeroth-, first-, and second-order differential features for pedestrian detection and call such features Taylor feature transform (TAFT). In practice, the TAFT features are computed by discrete sampling to address scale issues and meanwhile achieve computational efficiency. In addition, orientation insensitivity is handled by using directional versions of differentials. When applied to pedestrian detection, the TAFT is sampled on grid pixels and calculated from multiple channels following previous solutions. In our extensive experiments on the INRIA, Caltech, TUD-Brussel, and KITTI data sets, the TAFT achieves state-of-the-art results. It outperforms all handcrafted features and performs on par with many deep-learning solutions. Moreover, when a low false-positive rate is requested, the TAFT generates results that are better than or comparable to the state-of-the-art deep learning-based methods. Meanwhile, our implementation runs at 33 fps for 640×480 images without GPU, making TAFT favorable in many practical scenarios.

Model optimization and statistical inference have played a central role in various applications of computational intelligence, data analytics, and computer vision. Traditional approaches are usually based on model-centric learning. That is, even after model training, it is still required to design proper algorithms and to specify hand-crafted parameters for optimization and inference. Recently, discriminative learning has demonstrated its power for process-centric learning. Taking domain expertise and problem structure into account, problem-specific deep architectures can be formed by unfolding the model inference as an iterative process, and the parameters of the optimization process can then be learned from training data. These solutions are closely related with bilevel optimization, partial differential equation (PDE), as well as meta learning, and can provide new insights into the studies of versatile statistical and optimization models, such as sparse representation, structured regression, and conditional random fields. Moreover, generic deep network architectures are often referred to as “black-box” methods, while discriminative process-centric learning can provide a new perspective for the understanding and development of generic deep architectures. To sum up, connecting discriminative learning with model optimization and inference is not only helpful in analyzing convergence and generalization of deep architectures but also offers new perspectives for understanding and developing generic deep learning models.

This paper presents a technology for simple and non-iterative improvements of Multilayer and Deep Learning neural networks and Artificial Intelligence (AI) systems. The improvements are, in essence, shallow networks constructed on top of the existing Deep Learning architecture. Theoretical foundation of the technology is based on Stochastic Separation Theorems and the ideas of measure concentration. We show that, subject to mild technical assumptions on statistical properties of internal signals in Deep Learning AI, with probability close to one the technology enables instantaneous “learning away” of spurious and systematic errors. The method is illustrated with numerical examples.

Understanding traffic scene images taken from vehicle mounted cameras is important for high-level tasks, such as advanced driver assistance systems and autonomous driving. It is a challenging problem due to large variations under different weather or illumination conditions. In this paper, we tackle the problem of traffic scene understanding from a cross-domain perspective. We attempt to understand the traffic scene from images taken from the same location but under different weather or illumination conditions (e.g., understanding the same traffic scene from images on a rainy night with the help of images taken on a sunny day). To this end, we propose a dense correspondence-based transfer learning (DCTL) approach, which consists of three main steps: 1) extracting deep representations of traffic scene images via a fine-tuned convolutional neural network; 2) constructing compact and effective representations via cross-domain metric learning and subspace alignment for cross-domain retrieval; and 3) transferring the annotations from the retrieved best matching image to the test image based on cross-domain dense correspondences and a probabilistic Markov random field. To verify the effectiveness of our DCTL approach, we conduct extensive experiments on a challenging data set, which contains 1828 images from six weather or illumination conditions.

Image context in image is crucial for improving scene labeling. While the existing methods only exploit local context generated from a small surrounding area of an image patch or a pixel, the long-range and global contextual information is often ignored. To handle this issue, we propose a novel approach for scene labeling by multi-level contextual recurrent neural networks (RNNs). We encode three kinds of contextual cues, viz., local context, global context, and image topic context in structural RNNs to model long-range local and global dependencies in an image. In this way, our method is able to “see” the image in terms of both long-range local and holistic views, and make a more reliable inference for image labeling. Besides, we integrate the proposed contextual RNNs into hierarchical convolutional neural networks, and exploit dependence relationships at multiple levels to provide rich spatial and semantic information. Moreover, we adopt an attention model to effectively merge multiple levels and show that it outperforms average- or max-pooling fusion strategies. Extensive experiments demonstrate that the proposed approach achieves improved results on the CamVid, KITTI, SiftFlow, Stanford Background, and Cityscapes data sets.

Hierarchical neural networks have been shown to be effective in learning representative image features and recognizing object classes. However, most existing networks combine the low/middle level cues for classification without accounting for any spatial structures. For applications such as understanding a scene, how the visual cues are spatially distributed in an image becomes essential for successful analysis. This paper extends the framework of deep neural networks by accounting for the structural cues in the visual signals. In particular, two kinds of neural networks have been proposed. First, we develop a multitask deep convolutional network, which simultaneously detects the presence of the target and the geometric attributes (location and orientation) of the target with respect to the region of interest. Second, a recurrent neuron layer is adopted for structured visual detection. The recurrent neurons can deal with the spatial distribution of visible cues belonging to an object whose shape or structure is difficult to explicitly define. Both the networks are demonstrated by the practical task of detecting lane boundaries in traffic scenes. The multitask convolutional neural network provides auxiliary geometric information to help the subsequent modeling of the given lane structures. The recurrent neural network automatically detects lane boundaries, including those areas containing no marks, without any explicit prior knowledge or secondary modeling.

There has been continuously increasing interest in applying neural networks (NNs) to identification and adaptive control of practical systems that are characterized by nonlinearity, uncertainty, communication constraints, and complexity. The past few years have witnessed a variety of new developments in NN-based approaches for behavior learning, information processing, autonomous decision, and system control. Biologically inspired NN structures can significantly enhance the capabilities of information processing, control, and computational performance. New discoveries in neurocognitive psychology, sociology, and elsewhere reveal new neurological learning structures with more powerful capabilities in complex problem solving and fast decision in dynamic environments. The goal of the special issue is to consolidate recent new developments in NN structures for signal processing, autonomous decision, and adaptive control with application to complex systems. It includes contributions from a wide range of research aspects relevant to the topic, ranging from neural computing, adaptive control, cooperative control, autonomous decision systems, mathematical and computational models, neuropsychology decision and control, algorithms and simulation, to applications and/or case studies. This issue contains 24 papers and the contents of which are summarized below.

In this paper, we propose a vision-based approach for roadside vegetation detection by superpixel matching with local context. Unlike previous detection methods which seek help from additional sensors such as lidar, our algorithm only requires an off-the-shelf camera. The proposed method contains two stages. In the first stage, a superpixel database is constructed by segmenting training images into superpixels, and each superpixel patch is represented with multiple features. After that, the appearance information of vegetation or non-vegetation is encoded in the superpixel database. In the second stage, vegetation detection in each testing image is achieved by superpixel matching. The test image is segmented into superpixels and the (vegetation) label cost of each superpixel is derived by comparing with the k-nearest neighbors in the superpixel database. Furthermore, we incorporate the local context information through the feedback to refine superpixel matching. Taking this context information into account, Markov Random Field (MRF) is utilized to further improve the classification accuracy. Besides, considering the stable layout of road scene images, we utilize spatial priors of road scene to guide vegetation classification. Experiments on real-world datasets demonstrate the promise of our method.

Variations in the appearance of a tracked object, such as changes in geometry/photometry, camera viewpoint, illumination, or partial occlusion, pose a major challenge to object tracking. Here, we adopt cognitive psychology principles to design a flexible representation that can adapt to changes in object appearance during tracking. Inspired by the well-known Atkinson-Shiffrin Memory Model, we propose MUlti-Store Tracker (MUSTer), a dual-component approach consisting of short- and long-term memory stores to process target appearance memories. A powerful and efficient Integrated Correlation Filter (ICF) is employed in the short-term store for short-term tracking. The integrated long-term component, which is based on keypoint matching-tracking and RANSAC estimation, can interact with the long-term memory and provide additional information for output control. MUSTer was extensively evaluated on the CVPR2013 Online Object Tracking Benchmark (OOTB) and ALOV++ datasets. The experimental results demonstrated the superior performance of MUSTer in comparison with other state-of-art trackers.

Reliable long distance obstacle detection and motion planning is a key issue for modern intelligent vehicles, since it can help to make the decision early and design proper driving trajectory to avoid discomfort for the passengers caused by hard brake or sudden large lateral movement. Specifically, when there is vehicle parked on the roadside, we need to detect its position and pass it safely with proper distance without causing much disruption during driving. In this paper, we propose a method to detect roadside parked vehicles robustly and design a trajectory with proper lateral offset from the lane center for the host vehicle to safely pass by it. To successfully detect the roadside parked vehicles, we fuse the output from a long range lidar and radar. We pre-compute multiple path candidates with different lateral offset, and the path planner selects the most proper one based on the distance of the parked vehicle to the lane center. To deal with false alarms and missing detections, we apply temporal filtering to the detection output and history of the decision making. The speed control is carefully designed to ensure that the host vehicle passes the parked vehicle with a safe and comfortable speed. The implemented system was evaluated in numerous scenarios with vehicles parked on the roadside. The results show that the system effectively commands the host vehicle to pass by the parked vehicle safely and comfortably with proper distance and smooth trajectory.

Various sparse-representation-based methods have been proposed to solve tracking problems, and most of them employ least squares (LSs) criteria to learn the sparse representation. In many tracking scenarios, traditional LS-based methods may not perform well owing to the presence of heavy-tailed noise. In this paper, we present a tracking approach using an approximate least absolute deviation (LAD)-based multitask multiview sparse learning method to enjoy robustness of LAD and take advantage of multiple types of visual features, such as intensity, color, and texture. The proposed method is integrated in a particle filter framework, where learning the sparse representation for each view of the single particle is regarded as an individual task. The underlying relationship between tasks across different views and different particles is jointly exploited in a unified robust multitask formulation based on LAD. In addition, to capture the frequently emerging outlier tasks, we decompose the representation matrix to two collaborative components that enable a more robust and accurate approximation. We show that the proposed formulation can be effectively approximated by Nesterov's smoothing method and efficiently solved using the accelerated proximal gradient method. The presented tracker is implemented using four types of features and is tested on numerous synthetic sequences and real-world video sequences, including the CVPR2013 tracking benchmark and ALOV++ data set. Both the qualitative and quantitative results demonstrate the superior performance of the proposed approach compared with several state-of-the-art trackers.

Model-based dual heuristic dynamic programming (MB-DHP) is a popular approach in approximating optimal solutions in control problems. Yet, it usually requires offline training for the model network, and thus resulting in extra computational cost. In this brief, we propose a model-free DHP (MF-DHP) design based on finite-difference technique. In particular, we adopt multilayer perceptron with one hidden layer for both the action and the critic networks design, and use delayed objective functions to train both the action and the critic networks online over time. We test both the MF-DHP and MB-DHP approaches with a discrete time example and a continuous time example under the same parameter settings. Our simulation results demonstrate that the MF-DHP approach can obtain a control performance competitive with that of the traditional MB-DHP approach while requiring less computational resources.

A general utility function representation is proposed to provide the required derivable and adjustable utility function for the dual heuristic dynamic programming (DHP) design. Goal representation DHP (GrDHP) is presented with a goal network being on top of the traditional DHP design. This goal network provides a general mapping between the system states and the derivatives of the utility function. With this proposed architecture, we can obtain the required derivatives of the utility function directly from the goal network. In addition, instead of a fixed predefined utility function in literature, we conduct an online learning process for the goal network so that the derivatives of the utility function can be adaptively tuned over time. We provide the control performance of both the proposed GrDHP and the traditional DHP approaches under the same environment and parameter settings. The statistical simulation results and the snapshot of the system variables are presented to demonstrate the improved learning and controlling performance. We also apply both approaches to a power system example to further demonstrate the control capabilities of the GrDHP approach.

We consider the problem of construction of decision trees in cases when data is non-categorical and is inherently high-dimensional. Using conventional tree growing algorithms that either rely on univariate splits or employ direct search methods for determining multivariate splitting conditions is computationally prohibitive. On the other hand application of standard optimization methods for finding locally optimal splitting conditions is obstructed by abundance of local minima and discontinuities of classical goodness functions such as e.g. information gain or Gini impurity. In order to avoid this limitation a method to generate smoothed replacement for measuring impurity of splits is proposed. This enables to use vast number of efficient optimization techniques for finding locally optimal splits and, at the same time, decreases the number of local minima. The approach is illustrated with examples.

In adaptive dynamic programming, neurocontrol, and reinforcement learning, the objective is for an agent to learn to choose actions so as to minimize a total cost function. In this paper, we show that when discretized time is used to model the motion of the agent, it can be very important to do clipping on the motion of the agent in the final time step of the trajectory. By clipping, we mean that the final time step of the trajectory is to be truncated such that the agent stops exactly at the first terminal state reached, and no distance further. We demonstrate that when clipping is omitted, learning performance can fail to reach the optimum, and when clipping is done properly, learning performance can improve significantly. The clipping problem we describe affects algorithms that use explicit derivatives of the model functions of the environment to calculate a learning gradient. These include backpropagation through time for control and methods based on dual heuristic programming. However, the clipping problem does not significantly affect methods based on heuristic dynamic programming, temporal differences learning, or policy-gradient learning algorithms.

Combining multiple observation views has proven beneficial for tracking. In this paper, we cast tracking as a novel multi-task multi-view sparse learning problem and exploit the cues from multiple views including various types of visual features, such as intensity, color, and edge, where each feature observation can be sparsely represented by a linear combination of atoms from an adaptive feature dictionary. The proposed method is integrated in a particle filter framework where every view in each particle is regarded as an individual task. We jointly consider the underlying relationship between tasks across different views and different particles, and tackle it in a unified robust multi-task formulation. In addition, to capture the frequently emerging outlier tasks, we decompose the representation matrix to two collaborative components which enable a more robust and accurate approximation. We show that the proposed formulation can be efficiently solved using the Accelerated Proximal Gradient method with a small number of closed-form updates. The presented tracker is implemented using four types of features and is tested on numerous benchmark video sequences. Both the qualitative and quantitative results demonstrate the superior performance of the proposed approach compared to several state-of-the-art trackers.

In this paper, we propose an optimal control method based on the solution of Hamilton-Jacobi-Bellman (HJB) equation for the continuous-time nonlinear system with bounded unknown perturbation. The robust control system is converted into the corresponding optimal control system with appropriate performance index and the equivalence of the transformation is proved, i.e., the solution of the optimal control problem can globally asymptotically stabilize the robust control system. Adaptive dynamic programming (ADP) based approach is presented to iteratively approximate the optimal performance index and obtain the optimal control policy. A neural network with adaptive weights is applied to implement this approach. An example is given to illustrate the proposed method.

We consider the adaptive dynamic programming technique called Dual Heuristic Programming (DHP), which is designed to learn a critic function, when using learned model functions of the environment. DHP is designed for optimizing control problems in large and continuous state spaces. We extend DHP into a new algorithm that we call Value-Gradient Learning, VGL(λ), and prove equivalence of an instance of the new algorithm to Backpropagation Through Time for Control with a greedy policy. Not only does this equivalence provide a link between these two different approaches, but it also enables our variant of DHP to have guaranteed convergence, under certain smoothness conditions and a greedy policy, when using a general smooth nonlinear function approximator for the critic. We consider several experimental scenarios including some that prove divergence of DHP under a greedy policy, which contrasts against our proven-convergent algorithm.

We are interested in developing a multi-goal generator to provide detailed goal representations that help to improve the performance of the adaptive critic design (ACD). In this paper we propose a hierarchical structure of goal generator networks to cascade external reinforcement into more informative internal goal representations in the ACD. This is in contrast with previous designs in which the external reward signal is assigned to the critic network directly. The ACD control system performance is evaluated on the ball-and-beam balancing benchmark under noise-free and various noisy conditions. Simulation results in the form of a comparative study demonstrate effectiveness of our approach.

We derive an algorithm to exactly calculate the mixed second-order derivatives of a neural network's output with respect to its input vector and weight vector. This is necessary for the adaptive dynamic programming (ADP) algorithms globalized dual heuristic programming (GDHP) and value-gradient learning. The algorithm calculates the inner product of this second-order matrix with a given fixed vector in a time that is linear in the number of weights in the neural network. We use a "forward accumulation" of the derivative calculations which produces a much more elegant and easy-to-implement solution than has previously been published for this task. In doing so, the algorithm makes GDHP simple to implement and efficient, bridging the gap between the widely used DHP and GDHP ADP methods.

This paper focuses on the efficiency improvement of online actor-critic design base on the Levenberg-Marquardt (LM) algorithm rather than traditional chain rule. Over the decades, several generations of adaptive/approximate dynamic programming (ADP) structures have been proposed in the community and demonstrated many successfully applications. Neural network with backpropagation has been one of the most important approaches to tune the parameters in such ADP designs. In this paper, we aim to study the integration of Levenberg-Marquardt method into the regular actor-critic design to improve weights updating and learning for a quadratic convergence under certain condition. Specifically, for the critic network design, we adopt the LM method targeting improved learning performance, while for the action network, we use the neural network with backpropagation to provide an appropriate control action. A detailed learning algorithm is presented, followed by benchmark tests of pendulum swing up and balance and cart-pole balance tasks. Various simulation results and comparative study demonstrated the effectiveness of this approach.

IAM ABOUT to start my second year of service as the Editor-in-Chief (EiC) of the IEEE TRANSACTIONS ON NEURAL NETWORKS (TNN). Needless to say, my first year as the EiC has been full of excitement and challenges. Transitioning this position from my predecessor to me went very smoothly during the months of September 2009 to January 2010. During the past year, we have accumulated 50+ Associate Editors (AEs) handling roughly 600 new submissions (not counting resubmissions and revised submissions). With the help of these AEs and my predecessor, I was quickly able to learn to do my job, and as such, the transition had very few glitches. The easy part of my job is checking whether a submission is in compliance with our guidelines and where it is within the scope of the TRANSACTIONS, before it is assigned to an AE for handling. The difficult part of my job has been dealing with some papers with three or more reviewers, all of whom agreed to review them but for some reason failed to respond to repeated automatic-review reminders. AEs handling these papers have to take several extra steps to remind reviewers through phone calls or e-mails, look for replacement reviewers, or review the papers themselves. Most authors have been appreciative of the work of the AEs and reviewers, and they accept our decisions without a problem. The backlog of papers has been kept short over the last year. We have maintained an organized printing and paperacceptance schedule, with papers typically printed in the journal within 2‐3 months of acceptance. Our page budget has been kept constant in the past few years (roughly 2060 pages per year), and we expect to hold the same page count for next year.

A multi-agent framework to remote diagnostics with the emphasis on automotive vehicles is discussed. An overview of existing work on intelligent agents is given, and key features of the multi-agent framework are listed. Two real-world illustrations of the proposed multi-agent system are provided.

A learning system for detection and classification of road obstacles, such as vehicles and non-vehicles, is proposed which utilizes information from multiple sensors. An advanced range sensor guides a selection of candidate images provided by the camera for subsequent analysis. A competition based learning algorithm is used to distinguish between representations of different obstacles. High classification accuracy is demonstrated in a realistic variety of driving conditions in the presence of intentional data mislabeling in the two-class setup with state-of-art image descriptors.

In this brief, a convolutional learning system for classification of segmented objects represented in 3-D as point clouds of laser reflections is proposed. Several novelties are discussed: (1) extension of the existing convolutional neural network (CNN) framework to direct processing of 3-D data in a multiview setting which may be helpful for rotation-invariant consideration, (2) improvement of CNN training effectiveness by employing a stochastic meta-descent (SMD) method, and (3) combination of unsupervised and supervised training for enhanced performance of CNN. CNN performance is illustrated on a two-class data set of objects in a segmented outdoor environment.

Imperfect test outcomes, due to factors such as unreliable sensors, electromagnetic interference, and environmental conditions, manifest themselves as missed detections and false alarms. This paper develops near-optimal algorithms for dynamic multiple fault diagnosis (DMFD) problems in the presence of imperfect test outcomes. The DMFD problem is to determine the most likely evolution of component states, the one that best explains the observed test outcomes. Here, we discuss four formulations of the DMFD problem. These include the deterministic situation corresponding to perfectly observed coupled Markov decision processes to several partially observed factorial hidden Markov models ranging from the case where the imperfect test outcomes are functions of tests only to the case where the test outcomes are functions of faults and tests, as well as the case where the false alarms are associated with the nominal (fault free) case only. All these formulations are intractable NP-hard combinatorial optimization problems. Our solution scheme can be viewed as a two-level coordinated solution framework for the DMFD problem. At the top (coordination) level, we update the Lagrange multipliers (coordination variables, dual variables) using the subgradient method. At the bottom level, we use a dynamic programming technique (specifically, the Viterbi decoding or Max-sum algorithm) to solve each of the subproblems, one for each component state sequence. The key advantage of our approach is that it provides an approximate duality gap, which is a measure of the suboptimality of the DMFD solution. Computational results on real-world problems are presented. A detailed performance analysis of the proposed algorithm is also discussed.

In this paper, we describe an adaptive technique for states and parameter estimation involving a combination of two methods, namely the Variable Structure Filter (VSF) and the Extend Kalman Filters (EKF).

Recurrent neural networks with fixed weights have been shown in practice to successfully classify adaptively signals that vary as a function of time in the presence of additive noise and parametric perturbations. We address the question: Can this ability be explained theoretically? We provide a mathematical proof that these networks have this ability even when parametric perturbations enter the signals nonlinearly. The restrictions that we impose on the signals to be classified are that they satisfy an assumption of nondegeneracy and that noise amplitude is sufficiently small. Further, we demonstrate that the recurrent neural networks may not only classify uncertain signals adaptively but also can recover the values of uncertain parameters of the signals, up to their equivalence classes.

The author propose a neural network controller for improved fuel efficiency of the Toyota Prius hybrid electric vehicle. The approach is based on recurrent neural networks and an effective combination of off-line and on-line training methods including the extended Kalman filter and the simultaneous perturbation stochastic approximation (SPSA). The proposed approach is quite general and applicable to other control systems.

This paper considers the problem of temporally fusing classifier outputs to improve the overall diagnostic classification accuracy in safety-critical systems. Here, we discuss dynamic fusion of classifiers which is a special case of the dynamic multiple fault diagnosis (DMFD) problem [1]–[3]. The DMFD problem is formulated as a maximum a posteriori (MAP) configuration problem in tri-partite graphical models, which is NP-hard. A primal-dual optimization framework is applied to solve the MAP problem. Our process for dynamic fusion consists of four key steps: (1) data preprocessing such as noise suppression, data reduction and feature selection using data-driven techniques, (2) error correcting codes to transform the multiclass data into binary classification, (3) fault detection using pattern recognition techniques (support vector machines in this paper), and (4) dynamic fusion of classifiers output labels over time using the DMFD algorithm. An automobile engine data set, simulated under various fault conditions [4], was used to illustrate the fusion process. The results demonstrate that an ensemble of classifiers, when fused over time, reduces the classification error as compared to a single classifier and static fusion of classifiers trained over the entire batch of data. The results for sliding window dynamic fusion are also provided.

The winner-take-all (WTA) network is useful in database management, very large scale integration (VLSI) design, and digital processing. The synthesis procedure of WTA on single-layer fully connected architecture with sigmoid transfer function is still not fully explored. We discuss the use of simultaneous recurrent networks (SRNs) trained by Kalman filter algorithms for the task of finding the maximum among N numbers. The simulation demonstrates the effectiveness of our training approach under conditions of a shared-weight SRN architecture. A more general SRN also succeeds in solving a real classification application on car engine data.

The twenty-two papers in this special issue are devoted to neural networks for feedback control systems. Covers some of the following topics: reinforcement learning; applications; neurocontrol systems; discrete time systems; and network architectures and training methods.

This book consists of 13 chapters contributed mainly by European academic authors. The book opens with three overview chapters on fuzzy, neural, and evolutionary systems for system identification and control. Later chapters cover such topics as adaptive local linear modeling and control of nonlinear dynamical systems; Gaussian process approaches to nonlinear modeling and control; neuro-fuzzy model construction, design and estimation; and reinforcement learning for on-line control and optimization. The last few chapters concentrate on specific application areas, such as diagnostics, autonomous parking, and medicine. The book provides decent coverage of computational intelligence methods relevant to modeling and control. Better chapter cross referencing and proofreading would have benefited the book. Despite its shortcomings, the book is recommended to a broad audience interested in CI applications to system identification and control.

In this paper, we introduce a new approach to train recurrent neurocontrollers for real-time applications. We begin with training a recurrent neurocontroller for robustness on high-fidelity models of physical systems. For training, we use a recently developed derivative-free Kalman filter method which we enhance for controller training. After training, we fix weights of our recurrent neurocontroller and deploy it in an embedded environment. Then, we carry out additional training of the neurocontroller by adapting in real time its internal state (short-term memory), rather than its weights (long-term memory). Such real-time training is done with a new combination of simultaneous perturbation stochastic approximation (SPSA) and adaptive critic. Our critic is also a recurrent neural network (RNN), and it is trained by stochastic meta-descent (SMD) for increased efficiency. Our approach is applied to two important practical problems, electronic throttle control and hybrid electric vehicle control, with apparent performance improvement.

We consider a simple mathematical model of a disease. It includes four coupled nonlinear equations and four state variables for pathogens, plasma cells, antibodies and patient health indicator. Depending on initial values of state variables (concentrations) and without control, the model exhibits at least four possible classes of behavior. We treat the system of equations from the standpoint of feedback neurocontrol. We show that various feedback control strategies (due to injection of therapeutic agents) are possible and effective. We also discuss ways of making the "patient controller" more robust with respect to realistic uncertainties.

We are interested in training neurocontrollers for robustness on discrete-time models of physical systems. Our neurocontrollers are implemented as recurrent neural networks (RNNs). A model of the system to be controlled is known to the extent of parameters and/or signal uncertainties. Parameter values are drawn from a known distribution. For each instance of the model with specified parameters, a recurrent neurocontroller is trained by evaluating sensitivities of the model outputs to perturbations of the neurocontroller weights and incrementally updating the weights. Our training process strives to minimize a quadratic cost function averaged over many different models. In the end, the process yields a robust recurrent neurocontroller, which is ready for deployment with fixed weights. We employ a derivative-free Kalman filter algorithm proposed by Norgaard et al. and extended by Feldkamp et al. (2001) and Feldkamp et al. (2002) to neural network training. Our training algorithm combines effectiveness of a second-order training method with universal applicability to both differentiable and nondifferentiable systems. Our approach is that of model reference control, and it extends significantly the capabilities proposed by Prokhorov et al. (2001). We illustrate it with two examples.

The echo state network (ESN) has recently been proposed for modeling complex dynamic systems. The ESN is a sparsely connected recurrent neural network with most of its weights fixed a priori to randomly chosen values. The only trainable weights are those on links connected to the outputs. The ESN can demonstrate remarkable performance after seemingly effortless training. This brief paper discusses ESN in a broader context of applications of recurrent neural networks (RNN) and highlights challenges on the road to practical applications.

We study an evolution model of adaptive self-learning agents. The control system of agents is based on a neural network adaptive critic design. Each agent is a broker that predicts stock price changes and uses its predictions for action selection. The agent tries to get rich by buying and selling stocks. We demonstrate that the Baldwin effect takes place in our model, viz., originally acquired adaptive policy of an agent-broker becomes inherited in the course of the evolution. In addition, we compare agent behavioral tactics with searching behavior of simple animals.

We propose a new technique to improve the confidence in results of repeated neural network training runs under the practical constraint of a fixed computational budget. Our technique is applicable to problems for which there is a correlation between results early in the training process and results near the end of training. Targeting well-studied training problems, the technique may be most valuable when the computational time required for thorough training makes impractical performing a large number of differently initialized training sessions.

We propose a general scheme of intelligent adaptive control system based on the Petr K. Anokhin's theory of functional systems. This scheme is aimed at controlling adaptive purposeful behavior of an animat (a simulated animal) that has several natural needs (e.g., energy replenishment, reproduction). The control system consists of a set of hierarchically linked functional systems and enables predictive and goal-directed behavior. Each functional system includes a neural network based adaptive critic design. We also discuss schemes of prognosis, decision making, action selection and learning that occur in the functional systems and in the whole control system of the animat.

We demonstrate that a fixed-weight neural network can be trained with Kalman filter methods to exhibit input-output behavior that depends on which of two conditioning tasks had been performed a substantial number of time steps in the past. This behavior can also be made to survive an intervening interference task.

Superposition of sigmoid function over a finite time interval is shown to be equivalent to the linear combination of the solutions of a linearly parameterized system of logistic differential equations. Due to the linearity with respect to the parameters of the system, it is possible to design an effective procedure for parameter adjustment. Stability properties of this procedure are analyzed.

We address the problem of global Lyapunov stability of discrete-time recurrent neural networks (RNNs) in the unforced (unperturbed) setting. It is assumed that network weights are fixed to some values, for example, those attained after training. Based on classical results of the theory of absolute stability, we propose a new approach for the stability analysis of RNNs with sector-type monotone nonlinearities and nonzero biases. We devise a simple state-space transformation to convert the original RNN equations to a form suitable for our stability analysis. We then present appropriate linear matrix inequalities (LMIs) to be solved to determine whether the system under study is globally exponentially stable. Unlike previous treatments, our approach readily permits one to account for non-zero biases usually present in RNNs for improved approximation capabilities. We show how recent results of others on the stability analysis of RNNs can be interpreted as special cases within our approach. We illustrate how to use our approach with examples. Though illustrated on the stability analysis of recurrent multilayer perceptrons, the approach proposed can also be applied to other forms of time-lagged RNNs.

The ball-and-beam problem is a benchmark for testing control algorithms. In the World Congress on Neural Networks, 1994, Prof. L. Zadeh proposed a twist to the problem, which, he suggested, would require a fuzzy logic controller. This experiment uses a beam, partially covered with a sticky substance, increasing the difficulty of predicting the ball's motion. We complicated this problem even more by not using any information concerning the ball's velocity. Although it is common to use the first differences of the ball's consecutive positions as a measure of velocity and explicit input to the controller, we preferred to exploit recurrent neural networks, inputting only consecutive positions instead. We have used truncated backpropagation through time with the node-decoupled extended Kalman filter (NDEKF) algorithm to update the weights in the networks. Our best neurocontroller uses a form of approximate dynamic programming called an adaptive critic design. A hierarchy of such designs exists. Our system uses dual heuristic programming (DHP), an upper-level design. To our best knowledge, our results are the first use of DHP to control a physical system. It is also the first system we know of to respond to Zadeh's challenge. We do not claim this neural network control algorithm is the best approach to this problem, nor do we claim it is better than a fuzzy controller. It is instead a contribution to the scientific dialogue about the boundary between the two overlapping disciplines.

This paper proposes a novel technique to approximate Lyapunov functions for discrete time autonomous systems using a special form of support vector machine (SVM). We assume that a Lyapunov function can be accurately approximated by a polynomial of arbitrary degree on a finite set of points from trajectories of the closed-loop system. We transform the original problem of linearly constrained quadratic optimization into an equivalent dual problem. For computational tractability, we apply an iterative decomposition of the dual problem. We illustrate our technique on two examples.

We propose an approach to analyze trained neurocontrollers numerically for achieving Lyapunov stability of the closed-loop system. Our approach is based on training an adaptive critic to approximate a Lyapunov function. The trained critic is analyzed numerically to establish regions of stability of the closed-loop system. A simple example illustrates our approach. Our setting is the most relevant for neurocontrollers, but conclusions are applicable to other forms of controllers as well.

Three networks are compared for low false alarm stock trend predictions. Short-term trends, particularly attractive for neural network analysis, can be used profitably in scenarios such as option trading, but only with significant risk. Therefore, we focus on limiting false alarms, which improves the risk/reward ratio by preventing losses. To predict stock trends, we exploit time delay, recurrent, and probabilistic neural networks (TDNN, RNN, and PNN, respectively), utilizing conjugate gradient and multistream extended Kalman filter training for TDNN and RNN. We also discuss different predictability analysis techniques and perform an analysis of predictability based on a history of daily closing price. Our results indicate that all the networks are feasible, the primary preference being one of convenience.

We discuss a variety of adaptive critic designs (ACDs) for neurocontrol. These are suitable for learning in noisy, nonlinear, and nonstationary environments. They have common roots as generalizations of dynamic programming for neural reinforcement learning approaches. Our discussion of these origins leads to an explanation of three design families: heuristic dynamic programming, dual heuristic programming, and globalized dual heuristic programming (GDHP). The main emphasis is on DHP and GDHP as advanced ACDs. We suggest two new modifications of the original GDHP design that are currently the only working implementations of GDHP. They promise to be useful for many engineering applications in the areas of optimization and optimal control. Based on one of these modifications, we present a unified approach to all ACDs. This leads to a generalized training procedure for ACDs.

In the above paper by Mao-Jain (ibid., vol.7 (1996)), the Mahalanobis distance is used instead of Euclidean distance as the distance measure in order to acquire the hyperellipsoidal clustering. We prove that the clustering cost function is a constant under this condition, so hyperellipsoidal clustering cannot be realized. We also explains why the clustering algorithm developed in the above paper can get some good hyperellipsoidal clustering results. In reply, Mao-Jain state that the Wang-Xia failed to point out that their HEC clustering algorithm used a regularized Mahalanobis distance instead of the standard Mahalanobis distance. It is the regularized Mahalanobis distance which plays an important role in realizing hyperellipsoidal clusters. In conclusion, the comments made by Wang-Xia together with this response provide some new insights into the behavior of their HEC clustering algorithm. It further confirms that the HEC algorithm is a useful tool for understanding the structure of multidimensional data.

Describes a system that predicts significant short-term price movement in a single stock utilizing conservative strategies. We use preprocessing techniques, then train a probabilistic neural network to predict only price gains large enough to create a significant profit opportunity. Our primary objective is to limit false predictions (known in the pattern recognition literature as false alarms). False alarms are more significant than missed opportunities, because false alarms acted upon lead to losses. We can achieve false alarm rates as low as 5.7% with the correct system design and parameterization.