A function approximation method is developed which aims to approximate a function in a small neighborhood of a state that travels within a compact set. The method provides a novel approximation strategy for the efficient approximation of nonlinear functions for real-time simulations and experiments. The development is based on the theory of universal reproducing kernel Hilbert spaces over the n -dimensional Euclidean space. Several theorems are introduced which support the development of this state following (StaF) method. In particular, it is shown that there is a bound on the number of kernel functions required for the maintenance of an accurate function approximation as a state moves through a compact set. In addition, a weight update law, based on gradient descent, is introduced where arbitrarily close accuracy can be achieved provided the weight update law is iterated at a sufficient frequency, as detailed in Theorem 4. An experience-based approximation method is presented which utilizes the samples of the estimations of the ideal weights to generate a global approximation of a function. The experience-based approximation interpolates the samples of the weight estimates using radial basis functions. To illustrate the StaF method, the method is utilized for derivative estimation, function approximation, and is applied to an adaptive dynamic programming problem where it is demonstrated that the stability is maintained with a reduced number of basis functions.

An infinite-horizon optimal regulation problem for a control-affine deterministic system is solved online using a local state following (StaF) kernel and a regional model-based reinforcement learning (R-MBRL) method to approximate the value function. Unlike traditional methods such as R-MBRL that aim to approximate the value function over a large compact set, the StaF kernel approach aims to approximate the value function in a local neighborhood of the state that travels within a compact set. In this paper, the value function is approximated using a state-dependent convex combination of the StaF-based and the R-MBRL-based approximations. As the state enters a neighborhood containing the origin, the value function transitions from being approximated by the StaF approach to the R-MBRL approach. Semiglobal uniformly ultimately bounded (SGUUB) convergence of the system states to the origin is established using a Lyapunov-based analysis. Simulation results are provided for two, three, six, and ten-state dynamical systems to demonstrate the scalability and performance of the developed method.

When using a camera to estimate the pose of a moving target, measurements may only be available intermittently, due to feature tracking losses from occlusions or the limited field of view of the camera. Results spanning back to the Kalman filter have demonstrated the utility of using a predictor to update state estimates when measurements are not available, but target velocity measurements or a motion model must be known to implement a predictor for image-based pose estimation. In this paper, a novel estimator and predictor are developed to simultaneously learn a motion model, and estimate the pose, of a moving target from a moving camera. A stability analysis is provided to prove convergence of the state estimates and function approximation without requiring the restrictive persistent excitation condition. Two experiments illustrate the performance of the developed estimator and predictor. One experiment involves a stationary camera observing a mobile robot with sporadic feature tracking losses, and a second experiment involves a quadcopter moving between two mobile robots on a road network.

Online approximation of the optimal station-keeping strategy for a marine craft subject to an irrotational current is considered. An approximate policy that minimizes a user-defined cost function over an infinite time horizon is obtained using an actor-critic-identifier-based adaptive dynamic programming technique. The hydrodynamic drift dynamics are assumed to be unknown; therefore, a concurrent learning-based system identifier is developed to identify the unknown model parameters. The identified model is used to implement an adaptive model-based reinforcement learning technique to estimate the unknown value function. The developed policy guarantees uniformly ultimately bounded convergence of the vehicle to the desired station and uniformly ultimately bounded convergence of the approximated policies to the optimal polices without the requirement of persistence of excitation. The developed strategy is validated using an autonomous underwater vehicle, where the three degrees-of-freedom in the horizontal plane are regulated. The experiments are conducted in a second-magnitude spring located in central Florida.

This brief paper provides an approximate online adaptive solution to the infinite-horizon optimal tracking problem for control-affine continuous-time nonlinear systems with unknown drift dynamics. To relax the persistence of excitation condition, model-based reinforcement learning is implemented using a concurrent-learning-based system identifier to simulate experience by evaluating the Bellman error over unexplored areas of the state space. Tracking of the desired trajectory and convergence of the developed policy to a neighborhood of the optimal policy are established via Lyapunov-based stability analysis. Simulation results demonstrate the effectiveness of the developed technique.

An upper motor neuron lesion (UMNL) can be caused by various neurological disorders or trauma and leads to disabilities. Neuromuscular electrical stimulation (NMES) is a technique that is widely used for rehabilitation and restoration of motor function for people suffering from UMNL. Typically, stability analysis for closed-loop NMES ignores the modulated implementation of NMES. However, electrical stimulation must be applied to muscle as a modulated series of pulses. In this paper, a muscle activation model with an amplitude modulated control input is developed to capture the discontinuous nature of muscle activation, and an identification-based closed-loop NMES controller is designed and analyzed for the uncertain amplitude modulated muscle activation model. Semi-global uniformly ultimately bounded tracking is guaranteed. The stability of the closed-loop system is analyzed with Lyapunov-based methods, and a pulse frequency related gain condition is obtained. Experiments are performed with five able-bodied subjects to demonstrate the interplay between the control gains and the pulse frequency, and results are provided which indicate that control gains should be increased to maintain stability if the stimulation pulse frequency is decreased to mitigate muscle fatigue. For the first time, this paper brings together an analysis of the controller and modulation scheme.

An approximate online equilibrium solution is developed for an N -player nonzero-sum game subject to continuous-time nonlinear unknown dynamics and an infinite horizon quadratic cost. A novel actor-critic-identifier structure is used, wherein a robust dynamic neural network is used to asymptotically identify the uncertain system with additive disturbances, and a set of critic and actor NNs are used to approximate the value functions and equilibrium policies, respectively. The weight update laws for the actor neural networks (NNs) are generated using a gradient-descent method, and the critic NNs are generated by least square regression, which are both based on the modified Bellman error that is independent of the system dynamics. A Lyapunov-based stability analysis shows that uniformly ultimately bounded tracking is achieved, and a convergence analysis demonstrates that the approximate control policies converge to a neighborhood of the optimal solutions. The actor, critic, and identifier structures are implemented in real time continuously and simultaneously. Simulations on two and three player games illustrate the performance of the developed method.