As a well-known machine learning algorithm, sparse Bayesian learning (SBL) can find sparse representations in linearly probabilistic models by imposing a sparsity-promoting prior on model coefficients. However, classical SBL algorithms lack the essential theoretical guarantees of global convergence. To address this issue, we propose an iterative Min-Min optimization method to solve the marginal likelihood function (MLF) of SBL based on the concave-convex procedure. The method can optimize the hyperparameters related to both the prior and noise level analytically at each iteration by re-expressing MLF using auxiliary functions. Particularly, we demonstrate that the method globally converges to a local minimum or saddle point of MLF. With rigorous theoretical guarantees, the proposed novel SBL algorithm outperforms classical ones in finding sparse representations on simulation and real-world examples, ranging from sparse signal recovery to system identification and kernel regression.

The vast majority of convergence rates analysis for stochastic gradient methods in the literature focus on convergence in expectation, whereas trajectory-wise almost sure convergence is clearly important to ensure that any instantiation of the stochastic algorithms would converge with probability one. Here we provide a unified almost sure convergence rates analysis for stochastic gradient descent (SGD), stochastic heavy-ball (SHB), and stochastic Nesterov's accelerated gradient (SNAG) methods. We show, for the first time, that the almost sure convergence rates obtained for these stochastic gradient methods on strongly convex functions, are arbitrarily close to their optimal convergence rates possible. For non-convex objective functions, we not only show that a weighted average of the squared gradient norms converges to zero almost surely, but also the last iterates of the algorithms. We further provide last-iterate almost sure convergence rates analysis for stochastic gradient methods on general convex smooth functions, in contrast with most existing results in the literature that only provide convergence in expectation for a weighted average of the iterates. The last-iterate almost sure convergence results also enable us to obtain almost sure avoidance of any strict saddle manifold by stochastic gradient methods with or without momentum. To the best of our knowledge, this is the first time such results are obtained for SHB and SNAG methods.

In modern industry, human–robot collaboration is becoming the norm. Since the robots need to share the same workspace with humans in an unstructured/semistructured environment, robot–human and robot–environment collisions are inevitable in general. To reduce the harm caused by these collisions, it is necessary to detect them in real time so that actions can be taken accordingly. In this article, we propose a general robot collision detection method based on switched momentum dynamics identification. This enables real-time collision detection without any additional sensors, which are usually required by most of the existing real-time collision detection methods. Our algorithm identifies the specific parts in robot momentum dynamics that are affected by collisions and reports a collision occurrence whenever the identified parts deviate from a known collision-free model. The identification results are further analyzed using a support vector machine classifier to locate the linkage involved in the collisions. Finally, the effectiveness of our method is verified through experiments in the PyBullet environment and on a real 6-DOF robot, showing improved robustness to noise for identical collision detection accuracies.

State of health (SOH) estimation of lithium-ion batteries (LIBs) is of critical importance for battery management systems (BMSs) of electronic devices. An accurate SOH estimation is still a challenging problem limited by diverse usage conditions between training and testing LIBs. To tackle this problem, this article proposes a transfer learning-based method for personalized SOH estimation of a new battery. More specifically, a convolutional neural network (CNN) combined with an improved domain adaptation method is used to construct an SOH estimation model, where the CNN is used to automatically extract features from raw charging voltage trajectories, while the domain adaptation method named maximum mean discrepancy (MMD) is adopted to reduce the distribution difference between training and testing battery data. This article extends MMD from classification tasks to regression tasks, which can therefore be used for SOH estimation. Three different datasets with different charging policies, discharging policies, and ambient temperatures are used to validate the effectiveness and generalizability of the proposed method. The superiority of the proposed SOH estimation method is demonstrated through the comparison with direct model training using state-of-the-art machine learning methods and several other domain adaptation approaches. The results show that the proposed transfer learning-based method has wide generalizability as well as a positive precision improvement.

This article presents a data-driven, Koopman operator-based modeling scheme for analyzing and predicting cross-coupling hysteresis effects of the piezoelectric tube scanners (PTSs) used in atomic force microscopes (AFMs). Such cross-coupling hysteresis effects between different PTS axes significantly reduce the positioning precision of AFMs. In contrast to most of the existing methods for PTS hysteresis, which involve complex nonlinear dynamics identification processes, the present study leverages the Koopman operator theory instead to treat the nonlinear hysteresis as a linear system. Therein, a Hankel extended dynamic mode decomposition (H-EDMD) algorithm is proposed to learn the finite-dimensional descriptions of the Koopman operator and the associated Koopman eigenspectrum. Moreover, the proposed H-EDMD even allows sparse sampling on the PTS systems, which is desirable in real industrial applications. Finally, extensive comparison experiments with a mainstream modified Prandtl-Ishlinskii model are conducted on an NTMDT Prima AFM to substantiate the effectiveness and superiority of the proposed H-EDMD method.

Bootstrap aggregating (Bagging) and boosting are two popular ensemble learning approaches, which combine multiple base learners to generate a composite model for more accurate and more reliable performance. They have been widely used in biology, engineering, healthcare, etc. This article proposes BoostForest, which is an ensemble learning approach using BoostTree as base learners and can be used for both classification and regression. BoostTree constructs a tree model by gradient boosting. It increases the randomness (diversity) by drawing the cut-points randomly at node splitting. BoostForest further increases the randomness by bootstrapping the training data in constructing different BoostTrees. BoostForest generally outperformed four classical ensemble learning approaches (Random Forest, Extra-Trees, XGBoost and LightGBM) on 35 classification and regression datasets. Remarkably, BoostForest tunes its parameters by simply sampling them randomly from a parameter pool, which can be easily specified, and its ensemble learning framework can also be used to combine many other base learners.

Deep neural networks (DNNs) excel at industrial fault diagnosis. Their performance heavily relies on the quality of human-annotated labels. Due to perception limitations of annotators, industrial time series samples (such as vibration and voltage signals) are frequently mislabeled in several conditions, such as samples with frequency domain feature differences and samples on class borders. Hence, an annotated industrial dataset will inevitably contain noisy labels at a certain level, leading to overfitting and poor generalization of DNNs. In this work, we introduce an industrial noisy label semisupervised learning (INL-SSL) fault diagnosis approach, addressing the problem that a certain number of samples in an industrial dataset are mislabeled. The proposed INL-SSL architecture simultaneously trains two DNNs, which cross-train on each other to filter noisy label errors. In particular, a fitted Gaussian mixture model divides time series samples of each DNN flow into an unlabeled set with samples likely to be noisy and a labeled set with samples likely to be clean. Given the labeled and unlabeled data, we proposed a time series MixMatch semisupervised learning strategy to train the diagnostic model. Ablation study verifies the benefit of the proposed time series augmentation techniques for semisupervised training. Extensive experiments on a benchmark industrial dataset of rolling element bearings (REB) reveal that the INL-SSL outperforms state-of-the-art approaches. On another self-collected REB dataset, the proposed approach also exceeds other comparison methods under noise ratios from 20% to 90%, validating the model's generalizability.

Maintenance of machinery equipment in smart manufacturing requires real-time health monitoring, strongly supported by the rapid evolution of Artificial Intelligence (AI) technologies. Most AI-based health monitoring systems are powered by advanced modeling methods and intensive high-quality monitoring data. Such monitoring systems center on high-accuracy predictive performance but cannot necessarily convey reliability, such as satisfactory resistance to strong noises, credible uncertainty analysis, and model interpretability. This article novelly proposes noise-aware sparse Gaussian processes (NASGP) within the Bayesian inference framework. NASGP are capable of consistent high-performance and credible uncertainty assessment under strong noises. Based on NASGP, we then develop an explainable generalized additive model to bridge the gap between latent inference mechanism and domain expert knowledge. The efficacy of the proposed approach is corroborated through two case studies including remaining useful life prognosis and fault diagnosis for rolling bearings.

The vast majority of convergence rates analysis for stochastic gradient methods in the literature focus on convergence in expectation, whereas trajectory-wise almost sure convergence is clearly important to ensure that any instantiation of the stochastic algorithms would converge with probability one. Here we provide a unified almost sure convergence rates analysis for stochastic gradient descent (SGD), stochastic heavy-ball (SHB), and stochastic Nesterov’s accelerated gradient (SNAG) methods. We show, for the first time, that the almost sure convergence rates obtained for these stochastic gradient methods on strongly convex functions, are arbitrarily close to their optimal convergence rates possible. For non-convex objective functions, we not only show that a weighted average of the squared gradient norms converges to zero almost surely, but also the last iterates of the algorithms. We further provide last-iterate almost sure convergence rates analysis for stochastic gradient methods on weakly convex smooth functions, in contrast with most existing results in the literature that only provide convergence in expectation for a weighted average of the iterates.

Perception and portability are critical issues for wearable gloves in hand assistive engineering. However, available wearable gloves either lack flexible sensing or are bulky. In this paper, we present a tendon-driven lightweight wearable glove with soft joint sensing, Sen-Glove. Sen-Glove is equipped with 14 soft strain sensors, which enables full bending motion monitoring of 14 joints of five fingers and greatly reduces the weight of the glove. Besides, modular design makes Sen-Glove more compact and weighs 161g in total, reducing the burden on hand. A series of mechanical tests are conducted to evaluate the characteristics of Sen-Glove. Experimental results show that Sen-Glove can withstand 500 bending cycles, assist the subject in grasping 21 multi-scale objects, and recognize 11 gestures. The classification accuracy of 11 different gestures reaches 98.6 %, which verifies the efficacy of the strain sensors.

This article develops an identification algorithm for nonlinear systems. Specifically, the nonlinear system identification problem is formulated as a sparse recovery problem of a homogeneous variant searching for the sparsest vector in the null subspace. An augmented Lagrangian function is utilized to relax the nonconvex optimization. Thereafter, an algorithm based on the alternating direction method and a regularization technique is proposed to solve the sparse recovery problem. The convergence of the proposed algorithm can be guaranteed through theoretical analysis. Moreover, by the proposed sparse identification method, redundant terms in nonlinear functional forms are removed and the computational efficiency is thus substantially enhanced. Numerical simulations are presented to verify the effectiveness and superiority of the present algorithm.

We propose a novel technique for improving the stochastic gradient descent (SGD) method to train deep networks, which we term pbSGD. The proposed pbSGD method simply raises the stochastic gradient to a certain power elementwise during iterations and introduces only one additional parameter, namely, the power exponent (when it equals to 1, pbSGD reduces to SGD). We further propose pbSGD with momentum, which we term pbSGDM. The main results of this paper present comprehensive experiments on popular deep learning models and benchmark datasets. Empirical results show that the proposed pbSGD and pbSGDM obtain faster initial training speed than adaptive gradient methods, comparable generalization ability with SGD, and improved robustness to hyper-parameter selection and vanishing gradients. pbSGD is essentially a gradient modifier via a nonlinear transformation. As such, it is orthogonal and complementary to other techniques for accelerating gradient-based optimization such as learning rate schedules. Finally, we show convergence rate analysis for both pbSGD and pbSGDM methods. The theoretical rates of convergence match the best known theoretical rates of convergence for SGD and SGDM methods on nonconvex functions.

Takagi–Sugeno–Kang (TSK) fuzzy systems are very useful machine learning models for regression problems. However, to our knowledge, there has not existed an efficient and effective training algorithm that ensures their generalization performance and also enables them to deal with big data. Inspired by the connections between TSK fuzzy systems and neural networks, we extend three powerful neural network optimization techniques, i.e., minibatch gradient descent (MBGD), regularization, and AdaBound, to TSK fuzzy systems, and also propose three novel techniques (DropRule, DropMF, and DropMembership) specifically for training TSK fuzzy systems. Our final algorithm, MBGD with regularization, DropRule, and AdaBound, can achieve fast convergence in training TSK fuzzy systems, and also superior generalization performance in testing. It can be used for training TSK fuzzy systems on datasets of any size; however, it is particularly useful for big datasets, on which currently no other efficient training algorithms exist.

The complexity and randomness of the power system with distributed energy resources have led to the difficulties for fast optimal power flow (OPF) analysis. As a remedy, in this paper, we develop an interior point Powerball algorithm to accelerate the OPF solution process. To achieve better convergence characteristics, the proposed IPPB algorithm which is based on the Powerball optimization method, improves the search directions during iterative optimization by a nonlinear transformation. Also, a Newton—Raphson Powerball algorithm is derived for a faster power flow calculation, which is a basic yet critical part of the OPF problem. Numerical case studies are conducted on benchmark power systems with different scales to validate the proposed algorithms. Performances of the proposed algorithms to address improper initial points are studied by randomly picking the initial bus voltages. Numerical study results verify the feasibility and superiority of the proposed algorithms.

The circadian oscillator, an internal time-keeping device found in most organisms, enables timely regulation of daily biological activities by maintaining synchrony with the external environment. The mechanistic basis underlying the adjustment of circadian rhythms to changing external conditions, however, has yet to be clearly elucidated. We explored the mechanism of action of nicotinamide in Arabidopsis thaliana, a metabolite that lengthens the period of circadian rhythms, to understand the regulation of circadian period. To identify the key mechanisms involved in the circadian response to nicotinamide, we developed a systematic and practical modeling framework based on the identification and comparison of gene regulatory dynamics. Our mathematical predictions, confirmed by experimentation, identified key transcriptional regulatory mechanisms of circadian period and uncovered the role of blue light in the response of the circadian oscillator to nicotinamide. We suggest that our methodology could be adapted to predict mechanisms of drug action in complex biological systems.