We propose an online learning algorithm tailored for a class of machine learning models within a separable stochastic approximation framework. The central idea of our approach is to exploit the inherent separability in many models, recognizing that certain parameters are easier to optimize than others. This paper focuses on models where some parameters exhibit linear characteristics, which are common in machine learning applications. In our proposed algorithm, the linear parameters are updated using the recursive least squares (RLS) algorithm, akin to a stochastic Newton method. Subsequently, based on these updated linear parameters, the nonlinear parameters are adjusted using the stochastic gradient method (SGD). This dual-update mechanism can be viewed as a stochastic approximation variant of block coordinate gradient descent, where one subset of parameters is optimized using a second-order method while the other is handled with a first-order approach. We establish the global convergence of our online algorithm for non-convex cases in terms of the expected violation of first-order optimality conditions. Numerical experiments demonstrate that our method achieves significantly faster initial convergence and produces more robust performance compared to other popular learning algorithms. Additionally, our algorithm exhibits reduced sensitivity to learning rates and outperforms the recently proposedslimTrainalgorithm (Newman et al. 2022). For validation, the code has been made available on GitHub.

The total least squares (TLS) algorithm is a superior identification tool for low-order errors-in-variables (EIV) systems, where the estimate can be obtained by solving an eigenvector of the minimum eigenvalue of an augmented matrix. However, the TLS algorithm demonstrates inefficiency when applied to high-order EIV systems. This study introduces two innovative TLS algorithms: an iterative TLS algorithm, offering superior performance for low-order EIV models, and a two-step TLS algorithm, designed to effectively handle high-order EIV models. In comparison to the conventional TLS algorithm, these proposed methodologies present noteworthy advantages, including: 1) reduced computational costs, 2) the utilization of an iterative technique to calculate the inverse, and 3) the diversification of EIV identification methods. Simulation bench test examples are selected to show the efficacy of the proposed algorithms and transparent procedure for applications. Note to Practitioners—This paper was motivated by the problem of identifying network systems which are contaminated by noises. For network systems, the input and output data are usually contaminated by noises. Existing approaches to estimating such systems have the assumption that the noises are in little level scenarios or only the output data are contaminated by noises. This paper suggests two new total least squares approaches which can deal with systems contaminated by noises in medium level scenarios or whose input and output are both contaminated by noises. These two algorithms, using iterative technique and two-step technique, can: 1) avoid the matrix inverse calculation; 2) reduce the computational efforts; 3) increase the convergence rates. The proposed algorithms can also be extended to various fields such as inverse scattering, pattern recognition, image restoration, and computer vision.

The multipath effect often reduce global positioning system (GPS) positioning accuracy. Traditional GPS positioning methods usually ignore the multipath effect errors, resulting in poor positioning estimation accuracy. This article proposes a variable projection algorithm based on the Aitken method, which takes into consideration of the multipath effect error with the aim at improving the estimation accuracy, and uses the Aitken acceleration method to increase the convergence rate. In addition, the proposed method is robust to the step-size. Theoretical analysis and simulation examples show the effectiveness of the proposed method.

The expectation maximization (EM) algorithm has been extensively used to solve system identification problems with hidden variables. It needs to calculate a derivative equation and perform a matrix inversion in the EM-M step. The equations related to the EM algorithm may be unsolvable for some complex nonlinear systems, and the matrix inversion has heavy computational costs for large-scale systems. This article provides two expectation-based algorithms with the aim of constructing a comprehensive expectation framework concerning different kinds of time-delayed systems: 1) for a small-scale linear system, the classical EM algorithm can quickly obtain the parameter and time-delay estimates; 2) for a complex nonlinear system with low order, the proposed expectation gradient descent algorithm can avoid derivative function calculation; 3) for a large-scale system, the proposed expectation multidirection algorithm does not require eigenvalue calculation and has less computational costs. These two algorithms are developed based on the gradient descent and multidirection methods. Under such an expectation framework, different kinds of models are identified on a case-by-case basis. The convergence analysis and simulation examples show the effectiveness of the algorithms.

Multi-direction iterative (MUL-DI) algorithm is an efficient algorithm for large-scale models, and it establishes a theoretical linkage between least squares (LS) and gradient descent (GD) algorithms. However, it involves Givens transformation and dense matrix calculation in each iteration, which leads to heavy computational efforts. In this letter, a modified MULDI algorithm is proposed for separable nonlinear models with missing data. Several directions are designed using a diagonal matrix, and their corresponding step-sizes are obtained based on LS algorithm. Compared with the traditional algorithms, the algorithm proposed in this letter has the following advantages: (1) has a faster convergence rate; (2) has a simple cost function; (3) is more robust to the condition number; (4) has less computational efforts. A simulation example shows the effectiveness of the modified MUL-DI algorithm.

A robust standard gradient descent (SGD) algorithm for ARX models using the Aitken acceleration method is developed. Considering that the SGD algorithm has slow convergence rates and is sensitive to the step size, a robust and accelerative SGD (RA-SGD) algorithm is derived. This algorithm is based on the Aitken acceleration method, and its convergence rate is improved from linear convergence to at least quadratic convergence in general. Furthermore, the RA-SGD algorithm is always convergent with no limitation of the step size. Both the convergence analysis and the simulation examples demonstrate that the presented algorithm is effective.

In the traditional gradient descent (T-GD) algorithm, the convergence rate is strongly depend on the condition number of the information matrix: a larger condition number leads to a poor optimal convergence factor infimum$\mu _{\text{op}}$, which sets a convergence rate ceiling. That is, once the information matrix is fixed, the convergence factor of the T-GD algorithm reaches at most the infimum$\mu _{\text{op}}$. This article studies a varying infimum gradient descent algorithm, which can move down the infimum by using different order iterative preconditioning methods, as follows: first, for infinite iterative algorithm, the infimum becomes smaller and smaller with the increased iteration numbers; second, for finite iterative algorithm, the infimum is equal to zero, and the parameter estimates can be obtained in only one iteration; third, construct an adaptive interval between zero and$\mu _{\text{op}}$, which can establish a link between the least squares and T-GD algorithms. Based on the varying infimum gradient descent algorithm, researchers can adaptively choose preconditioning matrices for different kinds of models on a case by case basis. The convergence analysis and simulation examples show effectiveness of the proposed algorithms.

For 2-D causal systems, the variables depend both on time, and on spatial coordinates. This article develops two identification algorithms for two-dimensional causal systems. First, a maximum likelihood estimation algorithm is developed for two-dimensional causal systems when there is no missing data. Second, an expectation-maximization based auxiliary model algorithm, and an expectation-maximization based modified Kalman filtering and smoothing algorithm are derived for 2-D causal systems with missing outputs. It is demonstrated that the modified Kalman filtering, and smoothing algorithm is more effective for systems with missing outputs. The effectiveness of these two algorithms is verified by a simulation example.

A stochastic gradient (SG)-based particle filter (SG-PF) algorithm is developed for an ARX model with nonlinear communication output in this paper. This ARX model consists of two submodels, one is a linear ARX model and the other is a nonlinear output model. The process outputs (outputs of the linear submodel) transmitted over a communication channel are unmeasurable, while the communication outputs (outputs of the nonlinear submodel) are available, and both of the two-type outputs are contaminated by white noises. Based on the rich input data and the available communication output data, a SG-PF algorithm is proposed to estimate the unknown process outputs and parameters of the ARX model. Furthermore, a direct weight optimization method and the Epanechnikov kernel method are extended to modify the particle filter when the measurement noise is a Gaussian noise with unknown variance and the measurement noise distribution is unknown. The simulation results demonstrate that the SG-PF algorithm is effective.

Based on the work in Ding and Ding (2008), we develop a modified stochastic gradient (SG) parameter estimation algorithm for a dual-rate Box-Jenkins model by using an auxiliary model. We simplify the complex dual-rate Box-Jenkins model to two finite impulse response (FIR) models, present an auxiliary model to estimate the missing outputs and the unknown noise variables, and compute all the unknown parameters of the system with colored noises. Simulation results indicate that the proposed method is effective.