Human action recognition methods based on multimodal channels, especially the strategy combining RGB and skeleton information, have achieved remarkable success. However, the existing action recognition methods still face challenges in terms of insufficient multimodal fusion, imperfect multi-scale modeling, and inadequate preservation of spatial information. This paper proposes an innovative framework: by combining early bidirectional lateral connections with late fusion, deep interaction between modalities and complementarity at the prediction layer are achieved; the Multi-scale Spatiotemporal Attention Weighting (MSAW) mechanism is designed to adaptively fuse spatiotemporal features across different scales, selecting key frames in spatiotemporal sequences; the Pixel Sub-sampling Downsampling (PSD) module is proposed, which effectively retains spatial details during the strong downsampling process. A large number of experiments on the NTU RGB+D 60, NTU RGB+D 120, and Kinetics-400 datasets verified the effectiveness of the proposed method. The results show that our method demonstrates stronger competitiveness compared with the current state-of-the-art (SOTA) methods. Our code is available at:https://github.com/course-prog/MASW

Inferring 3D structures from sparse, unposed observations is challenging due to its unconstrained nature. Recent methods propose to predict implicit representations directly from unposed inputs in a data-driven manner, achieving promising results. However, these methods do not utilize geometric priors and cannot hallucinate the appearance of unseen regions, thus making it challenging to reconstruct fine geometric and textural details. To tackle this challenge, our key idea is to reformulate this ill-posed problem as conditional novel view synthesis, aiming to generate complete observations from limited input views to facilitate reconstruction. With complete observations, the poses of the input views can be easily recovered and further used to optimize the reconstructed object. To this end, we propose a novel pipeline, Pragmatist. First, we generate a complete observation of the object via a multiview conditional diffusion model. Then, we use a feed-forward large reconstruction model to obtain the reconstructed mesh. To further improve the reconstruction quality, we recover the poses of input views by inverting the obtained 3D representations and further optimize the texture using detailed input views. Unlike previous approaches, our pipeline improves reconstruction by efficiently leveraging unposed inputs and generative priors, circumventing the direct resolution of highly ill-posed problems. Extensive experiments show that our approach achieves promising performance in several benchmarks.

Online monitoring is essential for the safety of Industrial IoT (IIoT). Most existing methods seek low-dimensional representations to assess the overall operation status. However, we reveal that the existing methods face some unsolved and interrelated limitations, including coarse granularity, tight boundary, and weak extrapolation. This article proposes a federated episodic learning method for IIoT monitoring that simultaneously enhances interpretability, robustness, and extrapolation. The method centers on a dual-level normality bank (DLNB) with a normality contrastive separation network (NCSN) and an episodic training strategy (ETS), designed within a cloud-edge collaborative manner. To solve the coarse granularity issue, we propose a DLNB from both condition-level and variable-level perspectives, which facilitates fine-grained pattern matching and improves interpretability. To address the tight boundary issue, we propose an NCSN, which utilizes prior fault knowledge to construct negative samples and encourages models to focus on fault-related representations, thus improving robustness. To tackle the weak extrapolation issue, we design an ETS, which develops a client alternation policy to construct refining sets and makes inferences using patterns from adjacent working conditions. It fully exploits the relation of adjacent working conditions and improves extrapolation for unseen conditions with theoretical guarantees. Extensive experiments on two clusters validate the method’s superior interpretability, robustness, and extrapolation.

While existing causal discovery methods mostly focus on continuous time series, causal discovery for mixed time series encompassing both continuous variables (CVs) and discrete variables (DVs) is a fundamental yet underexplored problem. Together with nonlinearity and high dimensionality, mixed time series pose significant challenges for causal discovery. This study addresses the aforementioned challenges based on the following recognitions: 1) DVs may originate from latent continuous variables (LCVs) and undergo discretization processes due to measurement limitations, storage requirements, and other reasons. 2) LCVs contain fine-grained information and interact with CVs. By leveraging these interactions, the intrinsic continuity of DVs can be recovered. Thereupon, we propose a generic deep mixed time series temporal causal discovery framework. Our key idea is to adaptively recover LCVs from DVs with the guidance of CVs and perform causal discovery in a unified continuous-valued space. Technically, a new contextual adaptive Gaussian kernel embedding technique is developed for latent continuity recovery by adaptively aggregating temporal contextual information of DVs. Accordingly, two interdependent model training stages are devised for learning the latent continuity recovery with self-supervision and causal structure learning with sparsity-induced optimization. Experimentally, extensive empirical evaluations and in-depth investigations validate the superior performance of our framework.

Anomaly detection plays a vital role in monitoring industrial processes. Despite the extensive development of deep anomaly detection approaches, many assume uniform sampling rates for process variables, yet multirate phenomena are common in practice. The resultant large-scale missing data and varying dynamics of variables with different sampling rates pose a challenge to conventional anomaly detection methods. To this end, this paper developed a SAmpling-Rate-aware sequential Knowledge trAnsfer (SARKA) model for detecting the anomalies in multirate industrial processes. First, the raw multirate dataset is chunked into multiple data blocks, such that samples in each block correspond to a specific sampling rate and can be more readily processed by deep neural networks. Then, a Sampling-rate aware Slow Variational Auto-Encoder (S2VAE) model is developed, in which a sampling-rate-aware slowness principle is devised and integrated to enable both data block- and instance-wise personalized dynamic features characterization. Besides, to alleviate the scarce sample problem in the low-sampling-rate data block due to the multirate phenomena, a Sequential Knowledge Transfer (SKT) strategy is devised to convey the knowledge from the high-rate data to facilitate low-rate data modeling and improve the overall monitoring performance. Experimental results from a real-world coal mill in a thermal power plant multirate process demonstrate the effectiveness of the proposed method.Note to Practitioners—Due to the challenges resulting from large-scale missing data and complex varying dynamics, multirate process anomaly detection is a crucial task in modern industries. To tackle the challenge, this paper presents SARKA, which consists of two modules, i.e., S2VAE and SKT. Among them, S2VAE serves as the base model, which integrates a novel devised slowness principle under the probabilistic framework for improved sampling-rate-aware dynamic features characterization. Besides, SKT is elaborated to bridge the multiple S2VAE base models by conveying the modeling knowledge from those high-rate data to low-rate data, as the amount of the low-rate data is generally scarce to train a valid base model. Therefore, the base model can sufficiently capture both the static and dynamic features of the multirate process, and the SKT further enhances the modeling of the low-rate data, yielding improved performance on the overall multirate process.

Process uncertainty, which is usually caused by various factors, is generally subject to unknown complex distribution. However, many existing monitoring methods are established with a single distribution, and thus they may not accurately reflect the uncertainty within process systems. In this study, a probabilistic quality- relevant monitoring (PQM-GMM) is proposed with the Gaussian mixture model to address the aforementioned issue. Different from conventional monitoring methods, the proposed method measures the process uncertainty using multiple Gaussian distributions, which can be used to approximate any unknown complex distribution. Then, the optimization problem of the proposed PQM-GMM model is solved using the expectation maximization (EM) algorithm, which includes an augmented Lagrange multiplier in the M-step for model parameter estimation. Using the obtained results, a quality-relevant monitoring model is established with three statistics. It is noted that the proposed model can also be extended to many existing methods since they share a similar structure. Besides, the detailed information such as initial value selection, missing data problem, computation complexity is discussed. The effectiveness and superiority of the proposed method are tested using a numerical simulation example and a real low-pressure heater application. In comparison with some commonly used quality-relevant methods, the proposed model can be robustly established in the presence of corrupted data, and has a better detection sensitivity for the process anomalies in both process and quality variables. Note to Practitioners—A quality-relevant monitoring method is proposed in this study with Gaussian mixture model (GMM) for detecting the abnormal conditions of industrial processes under harsh environment. Since GMM can be used to approximate any unknown complex distribution, the process uncertainty within the collected data can be meticulously measured using the proposed PQM-GMM model. Besides, the quality-independent faults and quality-related faults can also be effectively distinguished using the designed monitoring statistics.

Recently, transfer learning (TL) approaches have been extensively applied in industrial cross-domain fault diagnosis, most of which depend on the consistency assumption of the source and target fault categories. In practice, it is common to utilize multiple source domains for transfer learning, but each of them may not include all fault categories in the target domain, which are referred to as incomplete multisource domains. For the challenge of fault diagnosis under incomplete multisource domains, a cross-domain bilateral transfer learning (CDBTL) method is proposed in this article. First, a cross-domain bilateral transfer strategy is developed, where the source and target domains are reconstructed from each other and their distribution differences are reduced by minimizing the reconstruction error to avoid negative transfer. Then, for the source domain with label information, CDBTL maximizes the between-class distance of different fault categories and minimizes the within-class distance of the same fault category to ensure the discriminative nature of its feature representation. Afterwards, the common projection matrix is learned through the mutual cooperation of projection matrices between different incomplete source domains and target domain to compensate for the missing fault categories in a single source domain. The key to discriminate CDBTL from many exiting TL algorithms is that it relaxes the restriction of consistent fault categories in the source and target domains, and skillfully integrates the knowledge of multiple incomplete source domains. Extensive experiments on Tennessee Eastman process demonstrate the superiority of CDBTL in solving cross-domain fault diagnosis problem, whose accuracy is averagely improved by 17.99% compared with eleven existing algorithms.Note to Practitioners—The changing industrial operating modes (domains) may result in different data distributions and fault categories between the historical mode (source domain) and current mode (target domain). Traditional machine learning methods usually fail to diagnose under the above domain and category inconsistencies. The key work in this article is to develop a cross-domain bilateral transfer learning (CDBTL) algorithm to realize cross-domain fault diagnosis under incomplete multisource domains. The proposed algorithm can avoid negative transfer while reducing inter-domain differences, and utilize the mutual cooperation of multiple source domains to fully cover the fault categories in the target domain. The constructed CDBTL model can be combined with various classifiers, and the learned classifiers can be directly applied to fault diagnosis in advanced manufacturing industry under incomplete multisource domains.

Industrial data often consist of continuous variables (CVs) and binary variables (BVs), both of which provide crucial information about process operating conditions. Due to the coupling between industrial systems or equipment, these hybrid variables are usually high-dimensional and highly correlated. However, existing methods generally model hybrid variables directly in the observation space and assume independence between the variables to overcome the curse of dimensionality. Thus, they are ineffective at capturing dependencies among hybrid variables, and the effectiveness of process monitoring will be compromised. To overcome the limitations, this study proposes to seek a unified subspace for hybrid variables using the probabilistic latent variable (LV) model. By introducing a low-dimensional continuous LV, the proposed method can avoid the curse of dimensionality while capturing the dependencies between hybrid variables. Nevertheless, the inference of LV is analytically intractable and thus time-consuming due to the heterogeneity of CVs and BVs. To accelerate offline learning and online inference procedures, this study originally derives an analytical Gaussian distribution to approximate the true posterior distribution of the LV, based on which an efficient expectation-maximization algorithm is developed for parameter estimation. The Gaussian approximation is simultaneously optimized with the latest parameters to achieve a high approximation accuracy. The LV is then estimated by the posterior mean of the Gaussian approximation. By mapping the heterogeneous variables into a unified subspace, the proposed method defines three monitoring statistics, which are physically interpretable and thoroughly evaluate the probability of hybrid variables being normal. The effectiveness of the proposed method in detecting anomalies in CVs and BVs is shown through a numerically simulated case and a real industrial case.

Process variables may exhibit both temporal trends and periodic responses, with their fault propagation pathways manifesting in time-domain and frequency-domain causalities, respectively. However, the differing causal perspectives of time-domain and frequency-domain methods can lead to distinct causalities, posing the causal heterogeneity challenge for root cause diagnosis (RCD). Thereupon, we reveal the mechanism of source consistency in Granger causality (GC), that is, the root cause variable provides the most significant predictive information in both time and frequency domains. Accordingly, we propose a causal source consistency analytics (CSCA) framework that achieves time-frequency synergy. First, we design a nonlinear enhancement module to extract temporal features for causal inference. Second, to extract time-domain and frequency-domain GC, we develop a parallel causality learning module, where a differentiable frequency-domain expansion operator is designed along with a temporal prediction submodule. Meanwhile, a time-frequency entropy constraint is constructed to ensure causal significance by inducing sparsity. Finally, a root cause alignment module is proposed to ensure source consistency. A predictive information quantification algorithm, formulated as an eigenvalue decomposition problem, is designed to locate the root cause. We develop an approximate exponential transformation to convert the eigenvalue decomposition into a differentiable source alignment loss. Thus, source consistency can be ensured during end-to-end inference. The validity of CSCA is illustrated through the Tennessee Eastman process and a gas turbine application. CSCA identified the root causes in both examples correctly. Furthermore, ablation studies validate that CSCA enables the time-domain and frequency-domain models to identify consistent root causes, thereby overcoming causal heterogeneity.

Causal discovery in multi-rate time series encounters greater challenges compared to regular time series. This stems from a potential problem that has not been noticed and explored in existing studies:information granularity heterogeneity, which refers to the natural difference in information granularity between fast sampling rate data (high information granularity) and slow sampling rate data (low information granularity). Such an imbalance in information granularity can hinder forecasting relationships modeling and induce biased causal learning. Therefore, we propose aMutual Information-iNspired causalDiscovery framework (MIND), aiming to derive rate-agnostic features with consistent information granularity to alleviate information granularity heterogeneity problem. Technically, MIND comprises Stage 1 (pre-training) and Stage 2 (fine-tuning and causal discovery). In Stage 1, empowered by pseudo-slow sampling rate data (generated through the interleaved down sampling strategy) and mutual information, we can eliminate the influence of sampling rates and drive rate-aware encoders (RAEs) to sense key information (i.e., rate-agnostic) that remains unchanged across varying sampling rates. In Stage 2, the well-trained RAEs can extract rate-agnostic features from real multi-rate time series, thus facilitating effective forecasting relationships modeling and yield accurate causal discovery. Empirically, MIND realizes superior performance on various multi-rate scenarios, including four simulation datasets and one real-world dataset.

Due to sensor malfunctions and data transmission corruptions, the industrial process data collected commonly contain missing values. It poses a significant challenge for data-driven approaches in aggregating temporal-spatial correlations that reflect dependencies across both variables and times, which makes it difficult to directly carry out downstream industrial operational applications. In this study, a self-supervised representation learning model is proposed to extract probabilistic temporal-spatial latent variables (LVs) from sequential data under missing value interference. The extracted LVs can be utilized for typical industrial operational applications through a unified framework. First, a novel deep dynamic probabilistic latent variable model, named Markov dynamic variational autoencoder (MD-VAE), is proposed to explicitly model the temporal-spatial dependencies between LVs. The latent posteriors are Bayesian smoothed by global sequence information for effective variational inference (VI). Second, a self-supervised learning approach, termed masked MD-VAE (M2D-VAE), is proposed to address the challenge of directly extracting temporal-spatial LVs under missing value interference. Controllable constraints with practical interpretations are introduced to balance the latent bottleneck capacity with reconstruction accuracy during model optimization. A unified framework is proposed to utilize the latent representations for typical industrial downstream tasks. Case studies conducted on a real-world multiphase flow process demonstrate the superiority of M2D-VAE in unsupervised industrial operational applications including missing value imputation and dynamic process monitoring under missing value interference.

Cross-time spatial dependence (i.e., the interaction between different variables at different time points) is indispensable for detecting anomalies in multivariate time series, as certain anomalies may have time delays in their propagation from one variable to another. However, accurately capturing cross-time spatial dependence remains a challenge. Specifically, real-world time series usually exhibits complex and incomprehensible evolutions that may be compounded by multiple temporal states (i.e., temporal patterns, such as rising, fluctuating, and peak). These temporal states mix and overlap with each other and exhibit dynamic and heterogeneous evolution laws in different time series, making the cross-time spatial dependence extremely intricate and mutable. Therefore, a cross-time spatial graph network with fuzzy embedding is proposed to disentangle latent and mixing temporal states and exploit it to meticulously learn cross-time spatial dependence. First, considering that temporal states are diversiform and their mixing modes are unknown, we introduce a fuzzy state set to uniformly characterize potential temporal states and adaptively generate corresponding membership degrees to depict how these states mix. Further, we propose a cross-time spatial graph, quantifying similarities among fuzzy states and sensing their dynamic evolutions, to flexibly learn mutable cross-time spatial dependence. Finally, we design state diversity and temporal proximity constraints to ensure the differences among fuzzy states and the evolution continuity of fuzzy states. Experiments on real-world datasets show that the proposed model outperforms the state-of-the-art models.

The industrial system usually contains not only controllable variables (CVs) but also uncontrollable variables (unCVs), e.g., weather conditions and friction. These unCVs have a direct impact on system control performance. Despite the success of current deep reinforcement learning (DRL) control algorithms, most of them neglect the impact of unCVs, which can cause the deterioration of control performance and instability of the system. To perceive and eliminate the impact of unCVs, a performance-aware self-optimizing universal controller (PASOUC) is designed in this article. The PASOUC aims at integrating the representation of unCVs and controller design to perceive and eliminate the impact of unCVs under different conditions, which goes beyond most existing control methods. Technically, a historical trajectory-inspired control performance perceptron is developed to perceive the impact of unCVs on system control performance under different conditions. Subsequently, a new performance-aware reward is designed to integrate the representation of unCVs and controller design while training the DRL controller. In addition, the domain randomization (DR) training strategy is employed to learn a universal control policy, which can access the approximate optimal trajectory under nonideal conditions. In this way, the impact of unCVs can be eliminated. To handle the low efficiency of the DR training, the policy improvement-policy proximal optimization (PI-PPO) is proposed to enhance the convergence speed of the DR training by performing explicit policy improvement. Finally, illustrative examples are presented to demonstrate the superiority of the proposed method.

Causal discovery under Granger causality framework has yielded widespread concerns in time series analysis task. Nevertheless, most previous methods are unaware of the underlying causality disappearing problem, that is, certain weak causalities are less focusable and may be lost during the modeling process, thus leading to biased causal conclusions. Therefore, we propose to introduce joint causal influences (i.e., causal influences from the union of multiple variables) as additional causal indication information to help identify weak causalities. Further, to break the limitation of existing methods that implicitly and coarsely model joint causal influences, we propose a novel hidden variable-driven causal hypergraph neural network to meticulously explore the locality and diversity of joint causal influences, and realize its explicit and fine-grained modeling. Specifically, we introduce hidden variables to construct a causal hypergraph for explicitly characterizing various fine-grained joint causal influences. Then, we customize a dual causal information transfer mechanism (encompassing a multi-level causal path and an information aggregation path) to realize the free diffusion and meticulous aggregation of joint causal influences and facilitate its adaptive learning. Finally, we design a multi-view collaborative optimization constraint to guarantee the characterization diversity of causal hypergraph and capture remarkable forecasting relationships (i.e., causalities). Experiments are conducted to demonstrate the superiority of the proposed model.

Mixed time series (MiTS) comprising both continuous variables (CVs) and discrete variables (DVs) are frequently encountered yet under-explored in time series analysis. Essentially, CVs and DVs exhibit different temporal patterns and distribution types. Overlooking these heterogeneities would lead to insufficient and imbalanced representation learning, bringing biased results. This paper addresses the problem with two insights: 1) DVs may originate from intrinsic latent continuous variables (LCVs), which lose fine-grained information due to extrinsic discretization; 2) LCVs and CVs share similar temporal patterns and interact spatially. Considering these similarities and interactions, we propose a general MiTS analysis framework MiTSformer, which recovers LCVs behind DVs for sufficient and balanced spatial-temporal modeling by designing two essential inductive biases: 1) hierarchically aggregating multi-scale temporal context information to enrich the information granularity of DVs; 2) adaptively learning the aggregation processes via the adversarial guidance from CVs. Subsequently, MiTSformer captures complete spatial-temporal dependencies within and across LCVs and CVs via cascaded self- and cross-attention blocks. Empirically, MiTSformer achieves consistent SOTA on five mixed time series analysis tasks, including classification, extrinsic regression, anomaly detection, imputation, and long-term forecasting. The code is available at https://github.com/chunhuiz/MiTSformer.

Real industrial processes often present coupled static and dynamic characteristics, leading to significant challenges for fault detection and isolation. However, traditional dynamic modeling methods may lead to the coupling problem of statics and dynamics, which provide ambiguous process status descriptions and incorrect fault isolation results. In this work, a novel full decoupling high-order dynamic mode decomposition (FDHODMD) method is developed for fault detection and isolation of dynamic processes. Different from the existing dynamic methods, the proposed FDHODMD can separate the high-order dynamic information from static information. First, the static characteristics are separated by discarding the decomposed features corresponding to smaller singular values. Then, a high-order dynamic model is established to present the temporal relationships between variables. In this way, the effect of static characteristics on dynamics analysis can be eliminated. Accordingly, from both static and dynamic perspectives, multiple statistics are designed to comprehensively detect the anomalies and provide an explicit status identification. In addition to the static fault isolation strategy, a dynamic fault isolation strategy is designed to recognize the fault variables after detecting the deviation, which divides the complex dynamic system into several individual dynamic modes to avoid the interaction of dynamic characteristics. Thereupon, the contributions of different variables to each dynamic mode can be clearly presented to recognize the fault variables. Finally, the validity of the proposed method is illustrated through both a numerical case and a real industrial process. Note to Practitioners—In industrial processes, the static and dynamic characteristics of data are often coupled. This work presents a FDHODMD method, which can be considered the high-order version of DMD, to achieve the decoupling of dynamics and statics while extracting high-order dynamic characteristics. Then the proposed fault detection strategy designs multiple statistics for monitoring the process from both dynamic and static perspectives, which ensures the safe operation of industrial processes. After detecting the anomaly, the proposed fault isolation strategy can recognize the fault variables separately dominating the dynamic and static deviation, which may help engineers locate the fault source.

Alarm flood classification and alarm prediction are significant ways to assist the on-site operators to manage alarm floods and maintain process safety. The two tasks are interdependent considering the decisive role of different alarm floods on the arising alarms. To give comprehensive consideration of both, this work proposed a hierarchical strategy for alarm flood classification and alarm prediction leveraging the structure feature of alarm floods. The structure feature aims at revealing the sparse causal dependencies among alarm variables. It is achieved by a deep learning model under the guidance of a designed objective function that probabilistically parametrizes causal dependencies with sparsity constraint. Due to its interpretable physical meaning, desirable robustness and discriminate properties are achieved, allowing a win-win situation for both tasks. Based on the structure features, the hierarchical strategy is given, where an overall classifier is built while prediction models are trained for each category. The classifier trained by structure features is predisposed to generate satisfactory early classification results, enabling timely prediction. For the prediction, the structure features are also used to incorporate temporal features to achieve better performance. Experimental results illustrate the interpretability of structure features and show the feasibility of the proposed hierarchical strategy.Note to Practitioners—During alarm floods, the alarm patterns are different than usual and the generic prediction model may fail. The focus of this study is to achieve a win-win situation for both alarm flood classification and prediction, thereby providing comprehensive information required for handling alarm floods. Considering the arising alarm is strongly affected by the type of current alarm flood, a hierarchical alarm flood classification and alarm prediction strategy is given. It exploits the essential characteristic of alarm interactions in alarm floods to generate robust and early classification results, allowing timely predictions by category. In this way, the performance of both tasks can be guaranteed. The proposed method requires labeled historical alarm flood data.

Although reinforcement learning (RL) is effective in stabilizing systems, it faces many challenges in solving the tracking problem of unknown continuous-time systems. One of the major challenges is that RL-based control can hardly satisfy both the transient and steady-state performance requirements for the tracking problem simultaneously. In this study, instead of implementing an RL controller, the RL agent acts as a planner in the closed-loop reference model. The RL-based planner concentrates on tracking performance optimization by the constrained integral RL algorithm. Meanwhile, the system is controlled by the proposed library-based adaptive controller, which contains a library of candidate functions for modeling the unknown system dynamics. A natural gradient-like adaptive law is developed to update the controller, ensuring asymptotic tracking and promoting sparsity in the controller parameter. Compared with the conventional RL-based control, the proposed framework can eliminate the tracking error while avoiding the high-frequency oscillation and peaking phenomenon. Furthermore, we theoretically demonstrate that our approach can improve the transient performance in terms of the${\cal L}_{2} $norm of the tracking error and explicitly limit the${\cal L}_{\infty} $norm of the peaking value through the Lyapunov analysis. Simulations are presented to support the theoretical findings at the end of the paper.Note to Practitioners—Practical control design is often interested in tracking non-zero reference trajectories. However, the non-optimal transient response, such as oscillation and overshoot, is the major obstacle to the development of a high-performance tracking control system. The proposed method addresses this issue by designing a constrained RL-based CRM to ensure the optimal transient performance of the system. The primary advantage is that the maximum peaking value can be conveniently tuned as a hyperparameter by users, which is extremely useful in practice. Furthermore, the proposed library-based adaptive controller can handle unknown system dynamics, where the governing equations of the dynamics can be determined through engineering experience.

Weakly supervised temporal action localization (WSTAL) aims to localize actions in untrimmed videos using video-level labels. Despite recent advances, existing approaches mainly follow alocalization-by-classificationpipeline, generally processing each segment individually, thereby exploiting only limited contextual information. As a result, the model will lack a comprehensive understanding (e.g. appearance and temporal structure) of various action patterns, leading to ambiguity in classification learning and temporal localization. Our work addresses this from a novel perspective, by exploring and exploiting the cross-video contextual knowledge within the dataset to recover the dataset-level semantic structure of action instances via weak labels only, thereby indirectly improving the holistic understanding of fine-grained action patterns and alleviating the aforementioned ambiguities. Specifically, an end-to-end framework is proposed, including aRobust Memory-Guided Contrastive Learning(RMGCL) module and aGlobal Knowledge Summarization and Aggregation(GKSA) module. First, the RMGCL module explores the contrast and consistency of cross-video action features, assisting in learning more structured and compact embedding space, thus reducing ambiguity in classification learning. Further, the GKSA module is used to efficiently summarize and propagate the cross-video representative action knowledge in a learnable manner to promote holistic action patterns understanding, which in turn allows the generation of high-confidence pseudo-labels for self-learning, thus alleviating ambiguity in temporal localization. Extensive experiments on THUMOS14, ActivityNet1.3, and FineAction demonstrate that our method outperforms the state-of-the-art methods, and can be easily plugged into other WSTAL methods.

Due to record errors, transmission interruptions, etc., low-quality process data, including outliers and missing data, commonly exist in real industrial processes, challenging the accurate modeling and reliable monitoring of the operating statuses. In this study, a novel variational Bayesian Student's-t mixture model (VBSMM) with a closed-form missing value imputation method is proposed to develop a robust process monitoring scheme for low-quality data. First, a new paradigm for the variational inference of Student's-t mixture model is proposed to develop a robust VBSMM model, which optimizes the variational posteriors in an extended feasible region. Second, conditioned on the complete and partially missing data information, a closed-form missing value imputation method is derived to address the challenges of outliers and multimodality in accurate data recovery. Then, a robust online monitoring scheme that can maintain its fault detection performance in the presence of poor data quality is developed, where a novel monitoring statistic called the expected variational distance (EVD) is first proposed to quantify the changes in operating conditions and can be easily extended to other variational mixture models. Case studies on a numerical simulation and a real-world three-phase flow facility illustrate the superiority of the proposed method in missing value imputation and fault detection of low-quality data.

Efficient monitoring of production performance is crucial for ensuring safe operations and enhancing the economic benefits of the Iron and Steel Corporation. Although basic modeling algorithms and visualization diagrams are available in many scientific platforms and industrial applications, there is still a lack of customized research in production performance monitoring. Therefore, this article proposes an interactive visual analytics approach for monitoring the heavy-plate production process (iHPPPVis). Specifically, a multicategory aggregated monitoring framework is proposed to facilitate production performance monitoring under varying working conditions. In addition, A set of visualizations and interactions are designed to enhance analysts' analysis, identification, and perception of the abnormal production performance in heavy-plate production data. Ultimately, the efficacy and practicality of iHPPPVis are demonstrated through multiple evaluations.

Zero-shot fault diagnosis can identify unseen faults by predicting attributes. However, existing methods ignore the multi-grained characteristics of attributes, namely the varying levels of detail in describing fault categories. We recognize the following considerations for the first time: (1) attributes show typical multi-grained characteristics, which could be expressed in a coarse-to-fine-grained hierarchical structure; (2) multi-grained attributes play different roles in fault diagnosis, where coarse-grained attributes indicate the rough range of faults, while fine-grained attributes facilitate the precise identification of fault types. In this paper, a fuzzy hierarchical zero-shot learning method is proposed to solve these issues. First, the attributes are divided into different layers according to the coarse-to-fine granularity via expert knowledge rather than being treated equally. Then, a knowledge transfer strategy is designed to transfer the knowledge from coarse-grained attributes to fine-grained ones, which can improve attribute prediction accuracy. Finally, a fuzzy inference strategy is developed to distinguish the effect of attributes with different granularity on fault inference. This strategy can identify the faults stepwise in a coarse-to-fine-grained order. The effectiveness of the proposed method is verified by a real thermal power plant process.

The generalized zero-shot diagnosis (GZSD) has attracted rising attention from researchers, but faces a major bottleneck: the domain shift problem (DSP), which causes most unseen faults to be misclassified as seen. In this study, we propose a novel method termed semisupervised hybrid triplet network (SSHTN) with high resistance toward DSP to achieve GZSD by first introducing auxiliary set. To leverage the unlabeled data of auxiliary set, which is different from known faults (both seen and unseen), SSHTN defines relative boundaries of seen and unseen faults in the similarity space and comprises two branches: the data–data branch (DDB) and the data–semantic branch (DSB). For DDB, it constructs homogeneous and heterogeneous data pairs to learn data–data similarity. With the support of auxiliary set, the semisupervised similarity learning mechanism is constructed. Specifically, DDB analyzes differences between data pairs mixed by seen and auxiliary fault data to narrow down the decision boundaries of seen faults. Meanwhile, similarities between data pairs formed by disturbing auxiliary set data are learned to improve the generalization of DDB. For DSB, it learns data–semantic similarity by matching fault data with corresponding descriptions and extends SSHTN discriminative ability toward unseen faults. The SSHTN obtained from the comprehensive learning of similarity by the two branches could determine the boundaries of seen and unseen faults more accurately, thereby alleviating the effect of DSP. Experiments are implemented on the Tennessee Eastman process and a real thermal power plant process, and results show the superior performance of SSHTN over state-of-the-art ZSD methods.

The fault detection capability of traditional data-driven process monitoring methods is highly dependent on the quality of process data. However, affected by measurement noise, harsh operation scenarios and other factors, the process data are inevitably contaminated by uncertainty in real processes. In this article, a vertices packaging-based interval independent component analysis (VP-I2CA) method is proposed to monitor the uncertain non-Gaussian processes. First, a variable bandwidth-kernel density estimation-based measurement error estimation method is developed to describe the uncertainty-contaminated process data in interval form using limited reliable data samples. Then, VP-I2CA is developed to estimate the demixing matrix based on hypermatrices constructed by vertices encoding, which includes all possible combinations between the bounds of interval data by explicitly considering the existence of uncertainty. In order to reduce computational complexity, the idea of data packaging is introduced to represent the hyper-independent components in interval form with a series of values by packaging all possible feature information hidden in uncertain process data. Afterwards, four monitoring statistics are constructed to monitor the systematic and nonsystematic parts of process operation variation. The proposed algorithm is verified in both a six-variable numerical simulation system and a continuous stirred tank reactor system.

Temporal data contain a wealth of valuable information, playing an essential role in various machine-learning tasks. Slow feature analysis (SFA), one of the most classic temporal feature extraction models, has been deeply explored in two decades of development. SFA extracts slowly varying features as high-level representations of temporal data. Its core idea of "slow" has been proven to be consistent with the nature of biological vision and beneficial in capturing significant temporal information for various tasks. So far, SFA has evolved into numerous improved versions and is widely applied in many fields such as computer vision, industrial control, remote sensing, signal processing, and computational biology. However, there currently lacks an insightful review of SFA. In this article, a comprehensive overview of SFA and its extensions is provided for the first time. The formulation and optimization of SFA are introduced. Two mainstream solutions, geometric interpretation, and a gradient-based training method of SFA are presented and discussed. Following that, a taxonomy of the current progress of SFA is proposed. We classify improved versions of SFA into six categories, including dual-input SFA (DISFA), online slow feature analysis (OSFA), probabilistic SFA (PSFA), multimode SFA, nonlinear SFA, and discrete labeled SFA. For each category, we illustrate its main ideas, mathematical principles, and applicable scenarios. In addition, the practical applications of SFA are summarized and presented. Finally, we bring new insights into SFA according to its research status and provide potential research directions, which may serve as a good reference for promoting future work.

Typically, industrial processes possess both temporal and spatial dependencies due to intravariable dynamics and intervariable couplings. The two dependencies have different manifestations, indicating diverse process characteristics. However, the existing methods fail to separate temporal and spatial information well, leading to inappropriate representation and inaccurate fault detection and isolation results. This study proposes an explicit representation and customized fault isolation framework to tackle temporal and spatial characteristics, so as to identify and locate anomalies affecting different dependencies. First, we design a double-level separation method for temporal and spatial information. In the first level, we construct two independent auto-encoding modules to extract temporal correlation and spatial graph structure in parallel. In the second level, we propose an information aliasing loss function to guild the two modules to distinguish between temporal and spatial characteristics, further facilitating information separation. By monitoring the explicit temporal and spatial statistics obtained by the two modules, spatiotemporal dependencies of anomalies can be determined for subsequent isolation. Furthermore, we propose a customized isolation strategy for anomalies in temporal and spatial characteristics. By quantifying changes in intravariable temporal dynamics and intervariable spatial graph structure individually, temporal impact and spatial propagation of faults can be finely characterized and isolated. Three examples are adopted to verify the performance of the proposed framework, including a numerical example, a real condensing system of the thermal power plant process, and the Tennessee Eastman benchmark process.

Industrial process data are usually high-dimensional with dynamic characteristics, and a mix of continuous and binary quantities. However, current dynamic latent variable (DLV) methods primarily focus on analyzing continuous variables (CVs), overlooking the prevalence and significance of binary variables (BVs). BVs often serve as control references, indicating operating conditions or specific states and influencing the behavior of CVs. Integrating BVs into DLV models is crucial for elucidating the correspondence between CVs and BVs and uncovering the real operating patterns of the system. The main challenge lies in effectively accommodating the statistical heterogeneity exhibited by CVs and BVs, while comprehensively investigating their contemporaneous and temporal dependencies. To address this challenge, this study proposes a novel DLV model called hybrid probabilistic slow feature analysis (HPSFA). The HPSFA algorithm is specifically designed to extract slow features (SFs) from CVs while incorporating supervision from BVs. To efficiently infer posterior distributions of SFs, a variational recursive filter (VRF) is developed using the local approximation method, providing closed-form posterior estimations. Leveraging the VRF, an efficient expectation-maximization algorithm is proposed for parameter estimation. For process monitoring, three statistics are designed based on prediction or reconstruction errors, which are separated from dynamic variations and exhibit reduced variability. This reduction in variability enables the definition of narrower control regions while maintaining the desired confidence level. The HPSFA method is thoroughly evaluated through both simulated and real industrial case studies to demonstrate its validity and superior performance over existing approaches. The experimental results show that HPSFA timely detects both static and dynamic anomalies of the hybrid variables, and achieves the highest-fault detection rate (85.89%) while maintaining a considerably low-false alarm rate (2.67%) in the practical industrial case.

Recently, a considerable literature has grown up around the few-sample fault diagnosis task, in which few samples of fault data are available for model training. The lack of fault samples usually causes significant degradation of diagnosis performance, which is referred as few-sample problem. Many federated learning-based fault diagnosis methods are proposed as a popular strategy to solve this problem. However, the vast majority of these algorithms assume that few-sample categories of the local client are well-sampled in some clients. In this sense, they are unable to cope with faults that have rare or even no sample in all clients, namely, global unseen faults. To diagnosis global unseen fault categories, a federated zero-shot fault diagnosis framework is proposed in this article. First, we propose a general approach for building a semantic knowledge base, which provides auxiliary discriminant descriptions of different faults. Second, a bidirectional alignment network is designed using two coupled variational autoencoders, enabling the fusion of data and attributes and allowing attribute descriptions to assist in fault diagnosis tasks. Third, a cloud–edge collaboration model aggregation strategy is developed, which utilizes a generative replay mechanism and integrates the knowledge of each client, thus, enhancing the generalization and generation ability of the global model. Experiments conducted on the thermal power plant group illustrate the feasibility and effectiveness of the proposed framework to categorize local and global unseen classes.

Due to the frequent changes in operating conditions, time-varying behaviors, including slow-varying dynamics and switching modes, commonly exist in industrial processes, resulting in different degrees of shifting in the process data distribution. When the data distribution shifts in a relatively wide range, conventional adaptive methods become ineffective since they are unable to distinguish normal shifts from real faults, leading to false alarms. In this study, an incremental variational Bayesian Gaussian mixture model (IncVBGMM) is proposed for developing a fine-scale adaptive monitoring scheme to efficiently accommodate the shifting data distribution caused by different degrees of time-varying behaviors. First, IncVBGMM with decremental optimization is proposed to adapt to the changing data distribution via the automatic complement of local models while reducing redundancy to optimize the mixture model. Then, a fine-scale adaptive monitoring scheme is built with physical interpretations to discern between normal shifts and real faults by joint analysis of the static and dynamic information. In addition, a novel monitoring statistic called the expectation of variational Bayesian inference distance (EVBID) is proposed, which can quantify the distance from samples to the variational monitoring model and indicate the fault effects. Case studies involving a real-world three-phase flow facility reveal that the proposed method can accurately differentiate various types of faults from normal shifts and effectively adapt to the time-varying dynamics.

Operational data from industrial processes typically consist of continuous and categorical variables. They reveal different aspects of the operational conditions, and both are useful for comprehensive fault detection and diagnosis (FDD). However, due to multiple production modes, the variables usually do not follow Gaussian or Bernoulli distributions. Furthermore, their correlations can be different across normal and various faulty classes. Thus, the main challenge is how to fuse the complementary information in the two types of variables and accurately characterize the complicated distribution for each class. This article proposes a flexible probabilistic framework that can concurrently analyze continuous and categorical variables for FDD. Our framework specifies a finite mixture model for each class. Thus, it can handle non-Gaussian and non-Bernoulli variables and capture their correlations under the conditional independence assumption. We then introduce the variational inference for parameter estimation, which makes our framework adaptive to the different distributions of various classes. Furthermore, a unified statistical index is designed, which gives our method extra capability to distinguish unknown faults. Finally, the effectiveness of our method is validated on the Tennessee Eastman (TE) process and a practical industrial plant process. The averageF1score of our method is improved by 3.4/3.9 percentage points in the single-mode/multimode situation on the TE process compared with traditional mixture discriminant analysis. In the industrial plant process, when unknown faults are added, the averageF1score of our method is 91.1% and only 1.2 percentage points lower than that on the test set without unknown faults.

Due to unexpected data transition or equipment failures, irregular data with missing values, which have both irregular sampling intervals and missing values, become very common in industrial processes and bring significant challenges for existing dynamic monitoring methods to explore temporal correlations. Therefore, this article develops an interval-aware probabilistic slow feature analysis (IA-PSFA) method along with the corresponding monitoring strategy to address the above problems for industrial processes. The IA-PSFA method incorporates functions of sampling intervals to adjust the influences of previous samples on the current one when inferring state variables. Specifically, different functions are designed such that the changing temporal correlations between adjacent samples caused by irregular sampling intervals can be tracked effectively. Parameters of the IA-PSFA model are estimated through the expectation-maximization (EM) algorithm with an interval-aware Kalman filter, which addresses the missing variable issue along with irregular sampling intervals. After that, three statistics are constructed based on the state variables, transition and emission errors, and the varying speed of the state variables, to establish comprehensive evaluations of processes. Finally, cases from the Tennessee Eastman (TE) process are provided to validate the effectiveness of the proposed method confronted with different degrees of data irregularity and missing values.

Video anomaly detection is challenging due to the lack of abnormal videos and ambiguity of anomaly definition. Context information is important to identify anomalous events and can be reflected from the salient objects and related background. In this paper, we propose a new multi-level attention network consisting of an Object-Guided Attention Module (OGAM) and a Motion-Refined Attention Module (MRAM) to fully exploit context by leveraging both frame-level and object-level semantics. Specifically, OGAM highlights features of salient objects on frame features for future frame prediction. MRAM further leverages global positions and optical flow of objects to improve the prediction loss. By multi-level attention, objects are highlighted in the predicted frame and necessary context information is preserved at both feature level and pixel level to identify anomalies. Experiments demonstrate our method achieves state-of-the-art performance on benchmark datasets including Avenue, UCSD Ped2 and ShanghaiTech, specifically obtaining a frame-level AUC of 92.6% on Avenue.

Despite rapid advances in machine learning based fault diagnosis, their identical distribution assumption of the training (source domain) and testing data (target domain) is generally challenged in industrial applications due to the variation of working conditions. In this article, a fault-prototypical adapted network (FPAN) is proposed, which enables cross-domain industrial intelligent fault diagnosis aided by deep transfer learning. First, a similarity learning-based discrimination module is designed to learn fault prototypes (FPs) that are representative for each fault and discriminative across different faults. Then, a fault prototypical-adaptation module is developed, which adapts the multiple FPs to the target dataset and enables more precise category-wise domain invariance. The two modules are trained simultaneously to extract transferrable and discriminative FPs, by which the cross-domain intelligent diagnosis can be readily achieved. Experimental results on two industrial cases illustrate that the proposed approach learns transferable feature representations that better reduce domain discrepancy, and provides improved diagnosis performance on target data. Note to Practitioners—As the distribution of source and target domain data may differ due to the varying working conditions, cross-domain intelligent fault diagnosis plays an increasingly important role in practical industrial cases. This work presents the FPAN model, in which the fault discrimination and the category-wise adaptation are naturally connected and unified by exploring representative virtual FP for each category, irrespective of which domain that the category comes from. Connected by the adaptive FPs, the similarity learning-based discrimination and the fault-prototypical adaptation modules can benefit from each other to boost the target learning. Finally, the cross-domain fault diagnosis can be performed based on the similarities with different FP representations. The proposed method is readily applicable to various intelligent fault diagnosis problems, including the mechanical cases and the industrial processes.

Missing data is quite common in the industrial field, resulting in problems in downstream applications, as most data driven methods used in these applications rely on complete and high-quality dataset to build a high-quality model. Existing methods deal with missing data individually regardless of its downstream application, treating all variables equally without considering their different roles in the downstream application. This would affect imputation performance for key variables, thus deteriorating the accuracy of the downstream model. A considerable challenge is how to refine the missing data imputation task. In this paper, a new method termed fine-tuned imputation GAN (FIGAN) is designed to achieve customized data imputation for industrial soft sensor. The major contribution of the paper lies in two aspects: 1) different from the original imputation GAN (GAIN) which treats all variables equally, FIGAN is guided by a soft sensor module so as to achieve customized data imputation by performing improved data imputation on quality-related variables. Enhanced accuracy for the final industrial soft sensor would be possible; 2) in addition, since labels of the soft sensor might also have missing data, a soft sensor with pseudo labeling is designed to conquer the problem with data imputation and label prediction being optimized interactively. Case studies on a converter steelmaking process and a penicillin fermentation process show the feasibility of the proposed FIGAN. It is noted that such customized imputation could be readily transferred to other downstream applications with missing data. Note to Practitioners—Industrial data is often incomplete and needs proper treatment. Meanwhile, downstream applications with preprocessed data vary under different industrial scenes. The focus of this study is to develop a customized data imputation method for specific downstream applications such as soft sensing. A fine-tuned imputation GAN is designed with a soft sensor module so as to guide the data imputation for good imputation on key variables of the soft sensor. Considering that missing data exists not only in measurement variables but also in soft sensor labels, a semi-supervised soft sensor is designed to handle missing data in the labels, optimized together with the imputation model. The customized data imputation can thus improve the final performance of the downstream model which is a soft sensor in this work. The customization could be transferred to other applications such as an anomaly detection model as well.

Unsupervised cross-domain fault diagnosis has been actively researched in recent years. It learns transferable features that reduce distribution inconsistency between source and target domains without target supervision. Most of the existing cross-domain fault diagnosis approaches are developed based on the consistency assumption of the source and target fault category sets. This assumption, however, is generally challenged in practice, as different working conditions can have different fault category sets. To solve the fault diagnosis problem under both domain and category inconsistencies, a multisource-refined transfer network is proposed in this article. First, a multisource-domain-refined adversarial adaptation strategy is designed to reduce the refined categorywise distribution inconsistency within each source-target domain pair. It avoids the negative transfer trap caused by conventional global-domainwise-forced alignments. Then, a multiple classifier complementation module is developed by complementing and transferring the source classifiers to the target domain to leverage different diagnostic knowledge existing in various sources. Different classifiers are complemented by the similarity scores produced by the adaptation module, and the complemented smooth predictions are used to guide the refined adaptation. Thus, the refined adversarial adaptation and the classifier complementation can benefit from each other in the training stage, yielding target-faults-discriminative and domain-refined-indistinguishable feature representations. Extensive experiments on two cases demonstrate the superiority of the proposed method when domain and category inconsistencies coexist.

Deep-learning-based soft sensors have been extensively developed for predicting key quality or performance variables in industrial processes. However, most approaches assume that data are uniformly sampled while the multiple variables are often acquired at different rates in practical processes. This article designed a progressive transfer strategy, based on which a variational progressive-transfer network (VPTN) method is proposed for the soft sensor development of industrial multirate processes. In VPTN, the multirate data are first separated into multiple data chunks where the variables within each chunk are acquired at a uniform rate. Then, a variational multichunk data modeling framework is developed to model the multiple chunks in a unified fashion through deep variational structures. The base models, including the unsupervised ones with only partial process variables and the supervised soft sensor model share a similar network structure, such that the subsequent transfer strategy can be readily implemented. Finally, a progressive transfer learning strategy is designed to transfer the model parameters from the fastest sampled data chunk to the slowest one in a progressive manner. Thus, the knowledge from various data chunks can be sequentially explored and transferred to enhance the performance of the terminal soft sensor model. Case studies on both a debutanizer column dataset and a real coal mill dataset in a thermal power plant validate the performance of the proposed method.

Modern industrial plants generally consist of multiple manufacturing units, and the local correlation within each unit can be used to effectively alleviate the effect of spurious correlation and meticulously reflect the operation status of the process system. Therefore, the local correlation, which is called spatial information here, should also be taken into consideration when developing the monitoring model. In this study, a cascaded monitoring network (MoniNet) method is proposed to develop the monitoring model with concurrent analytics of temporal and spatial information. By implementing convolutional operation to each variable, the temporal information that reveals dynamic correlation of process data and spatial information that reflects local characteristics within individual operation unit can be extracted simultaneously. For each convolutional feature, a submodel is developed and then all the submodels are integrated to generate a final monitoring model. Based on the developed model, the operation status of the newly collected sample can be identified by comparing the calculated statistics with their corresponding control limits. Similar to the convolutional neural network (CNN), the MoniNet can also expand its receptive field and capture deeper information by adding more convolutional layers. Besides, the filter selection and submodel development in MoniNet can be replaced to generalize the proposed network to many existing monitoring strategies. The performance of the proposed method is validated using two real industrial processes. The illustration results show that the proposed method can effectively detect process anomalies by concurrent analytics of temporal and spatial information.

When training samples are scarce, the semantic embedding technique, i. e., describing class labels with attributes, provides a condition to generate visual features for unseen objects by transferring the knowledge from seen objects. However, semantic descriptions are usually obtained in an external paradigm, such as manual annotation, resulting in weak consistency between descriptions and visual features. In this paper, we refine the coarse-grained semantic description for any-shot learning tasks, i. e., zero-shot learning (ZSL), generalized zero-shot learning (GZSL), and few-shot learning (FSL). A new model, namely, the semantic refinement Wasserstein generative adversarial network (SRWGAN) model, is designed with the proposed multihead representation and hierarchical alignment techniques. Unlike conventional methods, semantic refinement is performed with the aim of identifying a bias-eliminated condition for disjoint-class feature generation and is applicable in both inductive and transductive settings. We extensively evaluate model performance on six benchmark datasets and observe state-of-the-art results for any-shot learning; e. g., we obtain 70.2% harmonic accuracy for the Caltech UCSD Birds (CUB) dataset and 82.2% harmonic accuracy for the Oxford Flowers (FLO) dataset in the standard GZSL setting. Various visualizations are also provided to show the bias-eliminated generation of SRWGAN. Our code is available. 1.

Soft sensors have been extensively developed and applied in the process industry. One of the main challenges of the data-driven soft sensors is the lack of labeled data and the need to absorb the knowledge from a related source operating condition to enhance the soft sensing performance on the target application. This article introduces deep transfer learning to soft sensor modeling and proposes a deep probabilistic transfer regression (DPTR) framework. In DPTR, a deep generative regression model is first developed to learn Gaussian latent feature representations and model the regression relationship under the stochastic gradient variational Bayes framework. Then, a probabilistic latent space transfer strategy is designed to reduce the discrepancy between the source and target latent features such that the knowledge from the source data can be explored and transferred to enhance the target soft sensor performance. Besides, considering the missing values in the process data in the target operating condition, the DPTR is further extended to handle the missing data problem utilizing the strong generation and reconstruction capability of the deep generative model. The effectiveness of the proposed method is validated through an industrial multiphase flow process.

Frequent and wide changes in operation conditions are quite common in real process industry, resulting in typical wide-range nonstationary and transient characteristics along time direction. The considerable challenge is, thus, how to solve the conflict between the learning model accuracy and change complexity for analysis and monitoring of nonstationary and transient continuous processes. In this work, a novel condition-driven data analytics method is developed to handle this problem. A condition-driven data reorganization strategy is designed which can neatly restore the time-wise nonstationary and transient process into different condition slices, revealing similar process characteristics within the same condition slice. Process analytics can then be conducted for the new analysis unit. On the one hand, coarse-grained automatic condition-mode division is implemented with slow feature analysis to track the changing operation characteristics along condition dimension. On the other hand, fine-grained distribution evaluation is performed for each condition mode with Gaussian mixture model. Bayesian inference-based distance (BID) monitoring indices are defined which can clearly indicate the fault effects and distinguish different operation scenarios with meaningful physical interpretation. A case study on a real industrial process shows the feasibility of the proposed method which, thus, can be generalized to other continuous processes with typical wide-range nonstationary and transient characteristics along time direction.Note to Practitioners—Industrial processes in general have nonstationary characteristics which are ubiquitous in real world data, often reflected by a time-variant mean, a time-variant autocovariance, or both resulting from various factors. The focus of this study is to develop a universal analytics and monitoring method for wide-range nonstationary and transient continuous processes. Condition-driven concept takes the place of time-driven thought. For the first time, it is recognized that there are similar process characteristics within the same condition slice and changes in the process correlations may relate to its condition modes. Besides, the proposed method can provide enhanced physical interpretation for the monitoring results with concurrent analysis of the static and dynamic information which carry different information, analogous to the concepts of “position” and “velocity” in physics, respectively. The static information can tell the current operation condition, while the dynamic information can clarify whether the process status is switching between different steady states. It is noted that the condition-driven concept is universal and can be extended to other applications for industrial manufacturing applications.

For real industrial processes, time-varying behaviors are quite common. Consequently, industrial processes usually possess nonstationary characteristics, which makes conventional monitoring methods suffer from the model mismatch problem. In this article, an exponential analytic stationary subspace analysis (EASSA) algorithm is proposed to develop an adaptive strategy for nonstationary process monitoring. It is recognized that although covered by nonstationary trends, some underlying components of the process may remain stationary, which can be used for reliable process monitoring. For this, an EASSA algorithm is first developed to estimate the stationary sources more accurately and numerically stably. Then, a monitoring strategy is developed on the estimated stationary sources to provide reliable monitoring results. Considering that the relationships between process variables are driven to change slowly by time-varying behaviors, an update strategy and an adaptive monitoring scheme are designed to accurately track the trajectory of nonstationary processes while reducing the update frequency as much as possible. To meet the need of the EASSA algorithm, the minimum update unit has been expanded from one sample to one batch and the update conditions are given, by which the model is prevented from erroneously adapting to incipient faults to a certain extent. Case study on both a simulation process and a real thermal power plant process demonstrates that the proposed method can distinguish the real faults from normal changes while being robust to the disturbances in the nonstationary process.

In this article, a challenging fault diagnosis task is studied, in which no samples of the target faults are available for the model training. This scenario has hardly been studied in industrial research. But it is a common problem that massive fault samples are not available for the target faults, which limits the successes of conventional data-driven approaches in practical application. Here, we introduce the idea of zero-shot learning into the industry field, and tackle the zero-sample fault diagnosis task by proposing the fault description based attribute transfer method. Specifically, the method learns to determine the fault categories using the human-defined fault descriptions instead of the collected fault samples.The defined description consists of arbitrary attributes of the faults, including the fault positions, the consequences of the fault, and even the cause of the fault, etc. For the attribute knowledge of target faults, they can be prelearned and transferred from some readily available faults occurred in the same process. Afterwards, the target faults can be diagnosed based on the defined fault descriptions without the need for any additional data based training. Besides, the supervised principle component analysis is adopted in our method to extract the attribute related features to offer an effective attribute learning. We analyze and interpret the feasibility of the fault description based method theoretically. Also, the zero-sample fault diagnosis experiments are designed and conducted on the benchmark Tennessee-Eastman process and the real thermal power plant process to validate the effectiveness. The results show that it is indeed possible to diagnose target faults without their samples.

In real industrial applications, process data may get corrupted due to failure of the measurement devices or errors in data management. Besides, incipient faults, which may evolve into serious accidents, are generally more difficult to be detected because of its small magnitudes. In this article, a robust canonical variate dissimilarity analysis method is proposed to detect incipient faults for industrial processes with missing value. According to the low-rank characteristic, the low-rank matrix decomposition (LRMD) method is applied to recover the missing elements and reduce the ambient noise for process data. The output results of LRMD model consist of a low-rank component and a sparse component, which indicate main variance and residual information of the inputted data, respectively. For each component, a canonical variate analysis) model is developed to extract the temporal correlation in the process data. Based on the obtained features, a total of three monitoring statistics are established to reflect the operation status of the online sample. Among them, a statistic is used to measure the static deviation of this sample, and other two indices are applied to evaluate the dissimilarity between the past and future canonical variates. A simulated process and a real industrial process are adopted to illustrate the performance of the proposed method. Experimental results show that the proposed model can be well developed with incomplete training data and robustly detects the incipient faults for industrial applications.

Zero-shot learning (ZSL) is a successful paradigm for categorizing objects from the previously unseen classes. However, it suffers from severe performance degradation in the generalized ZSL (GZSL) setting, i.e., to recognize the test images that are from both seen and unseen classes. In this article, we present a simple but effective mechanism for GZSL and more open scenarios based on a transfer-increment strategy. On the one hand, a dual-knowledge-source-based generative model is constructed to tackle the missing data problem. Specifically, the local relational knowledge extracted from the label-embedding space and the global relational knowledge, which is the estimated data center in the feature-embedding space, are concurrently considered to synthesize the virtual exemplars. On the other hand, we further explore the training issue for the generative models under the GZSL setting. Two incremental training modes are designed to learn directly the unseen classes from the synthesized exemplars instead of the training classifiers with the seen and synthesized unseen exemplars together. It not only presents an effective unseen class learning but also requires less computing and storage resources in practical application. Comprehensive experiments are conducted based on five benchmark data sets. In comparison with the state-of-the-art methods, both the generating and training processes are considered for virtual exemplars by the proposed transfer-increment strategy, which results in a significant improvement in the conventional and GZSL tasks.

Soft sensor techniques have been applied to predict the hard-to-measure quality variables based on the easy-to-measure process variables in industry scenarios. Since the products are usually produced with prearranged processing orders, the sequential dependence among different variables can be important for the process modeling. To use this property, a dual attention-based encoder-decoder is developed in this article, which presents a customized sequence-to-sequence learning for soft sensor. We reveal that different quality variables in the same process are sequentially dependent on each other and the process variables are natural time sequences. Hence, the encoder-decoder is constructed to explicitly exploit the sequential information of both the input, that is, the process variables, and the output, that is, the quality variables. The encoder and decoder modules are specified as the long short-term memory network. In addition, since different process variables and time points impose different effects on the quality variables, a dual attention mechanism is embedded into the encoder-decoder to concurrently search the quality-related process variables and time points for a fine-grained quality prediction. Comprehensive experiments are performed based on a real cigarette production process and a benchmark multiphase flow process, which illustrate the effectiveness of the proposed encoder-decoder and its sequence to sequence learning for soft sensor.

Reliable and effective fault diagnosis methods are necessary for complex industrial processes that consists of various units. After a process fault is detected, it remains a challenging task to locate the root cause unit and determine the propagation path of the fault. In this paper, a novel method, termed Sparse Causal Residual Neural Network (SCRNN), is proposed and applied for modern industrial root cause diagnosis. The advantage of SCRNN lies in that it can not only recognize linear and nonlinear causal relationships in parallel, but also automatically determine the causality lags and deduce the time delay of causal transmission. Besides, due to the specially designed sparse constraint and optimization algorithm, the SCRNN model can realize the function of key dependent variable selection, avoiding the high computational complexity and complicated procedure brought by pairwise comparison. The feasibility of the proposed method is illustrated through the benchmark TE process.

De novo peptide sequencing via tandem mass spectrometry is one of the most powerful tools for identifying proteins, especially for novel sequences without any database information. Due to the incomplete fragmentation information and the high complexity of the experimental spectra, the accuracy and efficiency of de novo peptide sequencing is a considerable challenge. In this study, a novel residual network structure integrated with attention mechanism is proposed for de novo peptide sequencing, called RANovo. On one hand, the residual structure enables the network to go deeper, therefore more features can be extracted from the input data. On the other hand, attention mechanism is designed to adaptively recalibrate dynamic channel-wise information, which makes better use of the hidden features. Taking these advantages, the proposed method shows superior prediction accuracy on both amino acid level and peptide level in a series of experiments.

Process monitoring is a challenging task for modern industrial processes which are commonly nonstationary in nature, revealing typical non-Gaussian characteristics. Nowadays, data-driven based fault detection methods have drawn increasing attention, most of which work under an assumption that the process is subject to Gaussian distribution. But in practice, the underlying non-Gaussian characteristics may be typical in the complex process, which cannot be properly enclosed by a statistical model with a close confidence region and thus may be insensitive to fault detection. Hence, it is necessary to explore and separate the underlying Gaussian and non-Gaussian distributions in fine-grain. In this work, a Gaussian and non-Gaussian subspace decomposition method is proposed by designing a variant of stationary subspace analysis (VSSA) for nonstationary process monitoring. First, the whole time-wise nonstationary process can be neatly converted to condition-wise slices. Then, a Monte Carlo sampling based VSSA technique is designed to separate Gaussian and non-Gaussian subspaces from each other, which focuses on analyzing sample distribution rather than time series properties. Here the Gaussian subspace, which is readily characterized by a statistical model, is used for revealing similar condition slices and affiliate them into the same condition mode. And two monitoring statistics are developed to explore the Gaussian and non-Gaussian distribution structures, thus providing fine-grained distribution analytics and promoting monitoring performance. The feasibility and performance of the proposed method are demonstrated on a real thermal power plant process.

While machine-learning techniques have been widely used in smart industrial fault diagnosis, there is a major assumption that the source domain data (where the diagnosis model is trained) and the future target data (where the model is applied) must have the same distribution. However, this assumption may not hold in real industrial applications due to the changing operating conditions or mechanical wear. Recent advances have embedded the adversarial-learning mechanism into deep neural networks to reduce the distribution discrepancy between different domains to learn domain-invariant features and perform fault diagnosis. However, they only aligned the distributions of domains and neglected the fault-discriminative structure underlying the target domain, which leads to a decline in the diagnostic performance. In this article, a new method termed the fine-grained adversarial network-based domain adaptation (FANDA) is proposed to address the cross-domain industrial fault diagnosis problem. Different from the existing domain adversarial adaptation methods considering the domain discrepancy only, the features in FANDA are learned by competing against multiple-domain discriminators, which enable both a global alignment for two domains and a fine-grained alignment for each fault class across two domains. Thus, the fault-discriminative structure underlying two domains can be preserved in the adaptation process and the fault classification ability learned on the source domain can remain effective on the target data. Experiments on a mechanical bearing case and an industrial three-phase flow process case demonstrate the effectiveness of the proposed method.

Nonstationary variations widely exist in abnormal industrial processes, in which the mean values and variances of the fault nonstationary variables change with time. Thus, the stationary fault information may be buried by nonstationary fault variations resulting in high misclassification rate for fault diagnosis. Besides, the existing fault diagnosis methods do not consider underlying relations among different fault classes, which may lose important classification information. Here, it is recognized that different faults may not only share some common information but also have some specific characteristics. A fault diagnosis strategy with dual analysis of common and specific nonstationary fault variations is proposed here. The nonstationary variables and stationary variables are first separated using Augmented Dickey-Fuller (ADF) test. Then common and specific information is analyzed for fault diagnosis. Two models are developed, in which, the fault-common model is constructed by cointegration analysis (CA) to capture common nonstationary fault variations, and the fault-specific model is built to capture specific fault nonstationary variations of each fault class. With dual consideration of common and specific fault characteristics, the classification accuracy and fault diagnosis performance can be greatly improved. The performance of the proposed method is illustrated with both a well-known benchmark process and a real industrial process. Note to Practitioners-Process data analysis methods play an increasing important role in system maintenance and process monitoring in real industrial processes. The focus of this paper is to develop a fault diagnosis strategy with dual analysis of common and specific nonstationary fault variations. The proposed strategy can automatically describe the relationships between different fault classes using process data without complex mechanism knowledge. By exploring the relationship information between different faults, more accurate diagnosis models can be developed. Besides, the proposed strategy is a feasible technique for the nonstationary problem of complicated and varied industrial processes.

Dissimilarity analysis (DISSIM) has been widely used to monitor the Gaussian processes. However, its further application is hindered due to its unavailability to non-Gaussian processes whose data do not satisfy the hypothesis of the Gaussian distributions. To sensitively detect faults and enhance understanding of the non-Gaussian processes, a Gaussian feature analytics-based DISSIM (GDISSIM) method is proposed to monitor both Gaussian information and non-Gaussian information concurrently. The key lies in the separation of information with different statistical properties mixed in the process data. Hence, Gaussian-feature-based analytics is lirst proposed to devise the extraction, representation, and analysis of the Gaussian information. Besides, multiple Gaussian clusters are estimated for the remained non-Gaussian information integrated with posterior probabilities, enabling both Gaussian information and non-Gaussian information to be readily monitored. Different from the methods based on specilic assumptions or approximations, the proposed GDISSIM scheme investigates both non-Gaussian information and Gaussian information and is, therefore, delined as a line-grained monitoring method. The practical utility and feasibility of the proposed method are verilied by a numerical case and a real thermal power plant process.

In recent years, machine learning algorithms have been successfully applied to industrial processes. However, the concurrent analysis of static and dynamic representations has not been comprehensively addressed for industrial process fault classification. In this paper, an enhanced random forest algorithm with a concurrent analysis of static and dynamic nodes is proposed to address this issue for fault classification. First, the standard slow feature analysis is modified by designing a new slowness index that is more suitable for a supervised fault classification problem. Second, a feature ranking process is conducted to determine the significant features. These features, which substitute the raw variables in the nodes, are used to build the enhanced random forest. Using this scheme, the significant static and dynamic nodes are selected to enhance the discriminative ability and interpretation. Additionally, the slow features that are uncorrelated are more suitable for training the forest than the initial correlated variables, and the dynamic characteristics of industrial processes are thus comprehensively addressed. The application of the proposed method to fault classification is evaluated by both the Tennessee Eastman benchmark and a real-world three-phase flow process. The experimental results show that the proposed method outperforms the traditional learning algorithms with remarkable accuracy and F1 score that both exceed 70% for the 16-class Tennessee Eastman process and exceed 99% for the 4-class three-phase flow process. The selected significant features reveal that both the static and dynamic information play important roles in fault classification.

Assessment of operating performance for industrial processes is critical to guarantee high productivity and low cost, under routine operating condition. In the traditional research works on assessment of operating performance, the static characteristics are fully investigated, but the dynamic characteristics are seldom explored. Actually, the dynamic characteristics are important to distinguish operating performance and indicate the regulating actions of controllers. This article presents a concurrent static and dynamic assessment (ConSDA) method for operating performance in terms of industrial processes under closed-loop control. The performance levels are distinguished from both static and dynamic aspects. Canonical variate analysis and slow feature analysis are combined to fully extract the static and dynamic features of a process to well characterize each performance level. An efficient assessing scheme using the Bayesian inference based criterion is developed to provide meticulous assessing result with meaningful physical interpretability and sensitive switching identification for performance levels. The efficacy is demonstrated through application to a numerical example and a three-phase flow process. The rates of accurately distinguishing the performance levels for ConSDA is over 95% for the two applications with strong dynamic properties. Meanwhile, the highest average accuracy rates of four other assessing methods is 87.0%. The comparison illustrates the superiority of ConSDA.

Oblique random forests (ObRFs) have attracted increasing attention recently. Their popularity is mainly driven by learning oblique hyperplanes instead of expensively searching for axis-aligned hyperplanes in the standard random forest. However, most existing methods are trained in an off-line mode, which assumes that the training data are given as a batch. Efficient dual-incremental learning (DIL) strategies for ObRF have rarely been explored when new inputs from the existing classes or unseen classes come. The goal of this article is to provide an ObRF with DIL capacity to perform classification on-the-fly. First, we propose a batch multiclass ObRF (ObRF-BM) algorithm by using a broad learning system and a multi-to-binary method to obtain an optimal oblique hyperplane in a higher dimensional space and then separate the samples into two supervised clusters at each node, which provides the basis for the following incremental learning strategy. Then, the DIL strategy for ObRF-BM, termed ObRF-DIL, is developed by analytically updating the parameters of all nodes on the classification route of the increment of input samples and the increment of input classes so that the ObRF-BM model can be effectively updated without laborious retraining from scratch. Experimental results using several public data sets demonstrate the superiority of the proposed approach in comparison with several state-of-the-art methods.

The efficient mitigation of the detrimental effects of a fault in complex systems requires online fault diagnosis techniques that are able to identify the cause of an observable anomaly. However, an individual diagnosis model can only acquire a limited diagnostic effect and may be insufficient for a particular application. In this paper, a Bayesian network-based probabilistic ensemble learning (PEL-BN) strategy is proposed to address the aforementioned issue. First, an ensemble index is proposed to evaluate the candidate diagnosis models in a probabilistic manner so that the diagnosis models with better diagnosis performance can be selected. Then, based on the selected classifiers, the architecture of the Bayesian network can be constructed using the proposed three types of basic topologies. Finally, the advantages of different diagnosis models are integrated using the developed Bayesian network, and thus, the fault causes of the observable anomaly can be accurately inferred. In addition, the proposed method can effectively capture the mixed fault characteristics of multifaults (MFs) by integrating decisions derived from different diagnosis models. Hence, this method can also provide a feasible solution for diagnosing MFs in real industrial processes. A simulation process and a real industrial process are adopted to verify the performance of the proposed method, and the experimental results illustrate that the proposed PEL-BN strategy improves the diagnosis performance of single faults and is a feasible solution for MF diagnosis. Note to Practitioners-The focus of this paper is to develop a probabilistic ensemble learning strategy based on the Bayesian network (PEL-BN) to diagnose different kinds of faults in industrial processes. The PEL-BN strategy can automatically select the base classifiers to establish the architecture of the Bayesian network. In this way, the conclusions of these base classifiers can be effectively integrated to provide better diagnosis performance. In addition, the proposed method is also a feasible technique for diagnosing MFs resulted from the joint effects of multiple faults.

Due to the compensation of the control loops, industrial processes under feedback control generally reveal typical dynamic behaviors for different operation statuses. Conventional adaptive methods may update model falsely and thus result in invalid monitoring results, since they cannot effectively extract the feedback dynamic information and fail to accurately differentiate real anomalies from normal process changes. In this study, a recursive exponential slow feature analysis (ESFA) algorithm is developed for fine-scale adaptive monitoring to solve the problem of false model updating. First, an ESFA method is proposed to nonlinearly extract slow features, so that the general trend of the process variations can be better captured. On the basis of the ESFA model, a fine-scale adaptive monitoring scheme is developed to accurately capture the normal changes of industrial processes, including normal slow varying and normal shift of operation conditions. In this way, the normal slow varying can be effectively distinguished from incipient faults with unusual dynamic behaviors to avoid falsely adapting for the fault case, and the monitoring model can be correctly updated for new operation status after distinguishing real process anomalies from normal shifts of operation conditions. A simulation process and two real industrial processes are adopted to validate the performance of the proposed adaptive monitoring method. Experimental results show that the proposed method can effectively identify different operation statuses to decide whether to update the monitoring model or to raise an alarm.

Closed-loop control is commonly used in industrial processes to track setpoints or regulate process disturbances. Process dynamics resulting from closed-loop control are reflected in data mainly in two aspects, namely serial correlation and variation of response speed. Concurrent analysis of both aspects from data has not been fully investigated in the literature. In this work, a combined strategy of canonical variate analysis and slow feature analysis is proposed to monitor process dynamics resulting from closed-loop control by exploring both serial correlations and variation speed of process data. First, the canonical subspaces reflecting serial correlation are modeled by maximizing correlation between the past and future values of the process data. Then, both the serially correlated canonical subspace and its residual subspace are further explored to extract the slow features, which are representations of process variation speed. The proposed method provides a meaningful physical interpretation and in-depth process analysis with considerations of process dynamics under closed-loop control. Besides, it provides a concurrent monitoring of both process faults and operating condition deviations, resulting in fine-scale identification of different operation statuses. To demonstrate the feasibility and effectiveness, the proposed strategy is tested in a simulated typical chemical process under closed-loop control, namely the three-phase flow process.

Patient activity recognition in bed is very valuable to clinician to understand patient disease and drive clinical decisions. This paper proposes a recognition method based on the CNN (Convolutional Neural Network) to identify the action of bedridden patients. The inputs are 4 time series signals acquired from pressure sensors on the bed. Through CNN we obtain the corresponding membership of four pre-defined actions. A probability density analysis is made for setting a judgment standard, and ultimately recognizing the action. The method has been tested with real human activity signal and the results are promising.

Human activity recognition (HAR) is now valuable for bedridden patients to prevent falling, bedsore or other dangerous situation. This work proposes an intelligent broad learning system (BLS) recognition method based on the random vector functional-link neural network (RVFLNN) to identify the actions of bedridden patients. And the actions cover six types, including turning over to left, turning over to right, sitting up, lying down, stretching out for something and exiting from the bed. With the data collected from four pressure sensors that installed at four corners of an intelligent nursing bed, first, some pivotal preprocessing such as median filtering and down sampling are adopted to make a good performance. Then sparse auto encoder (SAE) is adopted for feature extraction. Finally, the RVFLNN is used for classification. Besides, for both new samples and new categories, the proposed method offers an incremental learning ability that can easily update the model with no need of model retaining. Compared with the convolutional neural network (CNN), the proposed method has superiority in training time while the accuracy is guaranteed.

Competition and demand for consistent and high-quality product have spurred the development of quality prediction methods for industrial manufacturing processes. Multiplicity of phases is, in general, common nature of many batch manufacturing processes. Considering that different phases may have different effects on qualities, one of the key issues is how to partition the whole batch process into multiple phases. In the present work, an automatic quality-relevant step-wise sequential phase partition (QSSPP) algorithm is developed for phase-based regression modeling and quality prediction. It considers the time sequence of operation phases and can capture the time-varying quality prediction relationships. Using this algorithm, phases are separated in order from quality-relevant perspective, revealing different quality prediction relationships. The phase-based regression system is set up for online quality prediction and the online prediction results are quantitatively evaluated for each phase. The feasibility and performance of the proposed algorithm are illustrated by an important manufacturing process, injection molding.

The aim of this paper is to develop a nonlinear orthogonal signal correction (OSC) algorithm using kernel-based technique, termed as kernel OSC (KOSC), and investigate its effects on multivariate calibration. As a nonlinear data pretreatment, the proposed KOSC method can better analyze the nonlinear relationships between descriptor and response variables and remove from process measurement those undesirable variations not correlated with process property from a nonlinear point of view, which well prepares the corrected process trajectory for the subsequent calibration modeling. Two data sets are employed in illustration experiment. It is found that nonlinear OSC plus nonlinear calibration algorithm seems to have the superiority over other methods to improve the interpretation ability of regression model when process data nonlinearly vary with quality.