With the continuous growth in the number of parameters of the Transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter-Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, extensive experiments are conducted using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.

Modern engineering and scientific systems are usually equipped with abundant sensors to collect large-dimensional time series for monitoring and operations. In this article, we develop a novel principal predictor analysis (PPA) framework with reduced-dimensional dynamics to obtain parsimonious predictor models of large-dimensional time series data. Principal predictors are obtained by maximizing the variance of predictions from their past values. Unlike classical principal component analysis (PCA), which reduces the dimensionality without emphasizing the prediction, PPA focuses on extracting latent variables with the maximum predictive capability. The PPA application to dynamic process monitoring is performed with predictive monitoring indices to account for variations in the predictors and the unpredicted residuals, which can be subsequently modeled with PCA. PPA-based monitoring and diagnosis are demonstrated in an illustrative closed-loop system and the industrial Dow Challenge Problem and an extension to include known first-principles relations to show their effectiveness.

Soft sensors are essential for advanced monitoring and control to prevent undesirable operations and improve product quality. However, nonlinear, autocorrelated, and cross-correlated behaviors in industrial data demand concurrent modeling of the dynamics and nonlinearities. Deep learning-based soft sensors, such as recurrent neural network (RNN) and long short-term memory (LSTM) networks, often incorporate complex structures and numerous parameters, which can lead to an overly complex model. In practical applications where training data samples are limited, a lightweight neural network with strong generalization capability is preferred. With a simple structure of feed-forward layers of 1-D convolutional neural networks (CNNs) (1-D-CNN) for time-series data modeling, this article proposes a novel lightweight dynamic CNN (LDCNN) for soft sensors. Positional embedding (PE) and simplified temporal attention mechanisms are integrated for improved dynamic modeling, while dilated convolutions and layer normalization (LN) are incorporated to significantly reduce the depth and width of the network and avoid over-parametrization. Experimental results on a real industrial case indicate that a lightweight model outperforms the traditional methods with limited training samples.

Users often assume that large language models (LLMs) share their cognitive alignment of context and intent, leading them to omit critical information in question-answering (QA) and produce ambiguous queries.Responses based on misaligned assumptions may be perceived as hallucinations.Therefore, identifying possible implicit assumptions is crucial in QA.To address this fundamental challenge, we propose Conditional Ambiguous Question-Answering (CondAmbigQA), a benchmark comprising 2,000 ambiguous queries and condition-aware evaluation metrics 1 .Our study pioneers "conditions" as explicit contextual constraints that resolve ambiguities in QA tasks through retrievalbased annotation, where retrieved Wikipedia fragments help identify possible interpretations for a given query and annotate answers accordingly.Experiments demonstrate that models considering conditions before answering improve answer accuracy by 11.75%, with an additional 7.15% gain when conditions are explicitly provided.These results highlight that apparent hallucinations may stem from inherent query ambiguity rather than purely model failure, and demonstrate the effectiveness of condition reasoning in QA, providing researchers with tools for rigorous evaluation.

Modeling nonlinear dynamical systems is a challenging task in fields such as speech processing, music generation, and video prediction. This paper introduces a hierarchical framework for Deep State Space Models (DSSMs), categorizing them by their conditional independence properties and Markov assumptions and positioning existing models within this framework, including the Stochastic Recurrent Neural Network (SRNN), Variational Recurrent Neural Network (VRNN), and Recurrent State Space Model (RSSM). We discuss different options for the inference networks and demonstrate how integrating normalizing flows can enhance model flexibility by capturing complex distributions. Our work not only clarifies the relationships among existing models but also paves the way for the development of new, more effective approaches for modeling nonlinear dynamics. In particular, we propose the Autoregressive State Space Model (ArSSM) and evaluate its effectiveness in speech and polyphonic music modeling tasks.

This paper presents Deep Dynamic Probabilistic Canonical Correlation Analysis (D2PCCA), a model that integrates deep learning with probabilistic modeling to analyze nonlinear dynamical systems. Building on the probabilistic extensions of Canonical Correlation Analysis (CCA), D2PCCA captures nonlinear latent dynamics and supports enhancements such as KL annealing for improved convergence and normalizing flows for a more flexible posterior approximation. D2PCCA naturally extends to multiple observed variables, making it a versatile tool for encoding prior knowledge about sequential datasets and providing a probabilistic understanding of the system’s dynamics. Experimental validation on real financial datasets demonstrates the effectiveness of D2PCCA and its extensions in capturing latent dynamics.

The application of metric learning in recommender systems has gained considerable attention, particularly for enabling supervised training through Euclidean distance. However, conventional collaborative translational metric learning typically represents each user and item as a single point and projects collaborative signals onto a specific translation vector within a uniform space. This approach fails to explicitly capture the high-order semantics and dynamic nature of user-item interactions derived from implicit user behavior. Given the extensive research on graph neural networks (GNNs), a natural extension is to incorporate high-order information into the generation of translation vectors. Yet, our empirical findings reveal that the performance is not as expected. In this paper, we systematically investigate the limitations of existing methods and attribute the issue to semantic confusion. Motivated by this finding, we propose a novel framework that integrates bilateral metric learning with a causal graph network. Specifically, we construct a structural causal model (SCM) to disentangle user behavior into causal features and confounders. To mitigate the influence of these confounders, we propose a conditional intervention module that generates confounder-specific conditions for the target. Moreover, inspired by this conditional intervention mechanism, we develop an additional module conditioned on user behavior to enhance the diversity of the recommended list, recognizing that diversity is also a crucial metric for evaluating recommendation quality. We conducted a series of experiments to validate the effectiveness of our proposed method. The results demonstrate that our approach achieves state-of-the-art performance in terms of both recommendation accuracy and diversity.

Monitoring nonlinear dynamic processes is challenging due to the inherent complexity and uncertainty in industrial systems. Fault detection and fault identification are critical components of process monitoring, yet existing methods often struggle to capture predictable dynamics with compact model parameters. This could compromise the fault detection performance and further impede the fault identification accuracy. This article proposes a unified modeling framework to strategically address these challenges through nonlinearity handling, most predictable dynamics extraction, and parsimonious system parameterization. Building on this enhanced capability, the proposed model integrates fault detection and identification with mutual information, thereby improving the identification accuracy. Validations on the revamped Tennessee Eastman Process and a real-world multiphase flow facility demonstrate superior fault detection metrics and fault identification capabilities compared to alternative approaches.

Large-dimensional systems are ubiquitous in industrial manufacturing systems, smart grids, and biological systems. This article develops a reduced-dimensional latent state-space identification (LaSSID) framework for systems where the output dynamics is in a low-dimensional latent space. The proposed LaSSID algorithm effectively handles collinear dynamics among output variables and improves the condition number. The method is compared to traditional methods in terms of enhanced predictability and interpretability. In addition, a modified system identification experiment is developed using the Tennessee Eastman process simulation, incorporating reduced-dimensional excitations and collinear perturbations to simulate real-world scenarios. The results of this simulated experiment and the Dow Challenge industrial dataset demonstrate the superior performance of the proposed algorithm in terms of system identification accuracy and robustness to noise.

Anomaly detection (AD) is typically regarded as an unsupervised learning task, where the training data either do not contain any anomalous samples or contain only a few unlabeled anomalous samples. In fact, in many real scenarios such as fault diagnosis and disease detection, a small number of anomalous samples labeled by domain experts are often available during the training phase, which makes semi-supervised AD (SAD) more appealing, though the related study is quite limited. Existing semi-supervised AD methods directly add optimization terms of anomalous samples to the optimization objective of unsupervised AD (UAD), where the effects of the limited labeled anomalous data on the optimization process become trivial and they cannot fully contribute to the detection task. To cover the shortage, in this work, we propose a novel semi-supervised AD method to fully use the limited labeled anomalous data and further to boost detection performance. The proposed method learns a nonlinear transformation to project normal data into a compact target distribution and simultaneously to project exposed anomalous samples into another target distribution, where the two target distributions do not overlap each other. The goal is difficult to achieve because of the scarcity of anomalous samples. To address this problem, we propose to generate a large number of intermediate samples interpolating between normal and anomalous data and project them into a third target distribution lying between the aforementioned two target distributions. Empirical results on multiple benchmarks with varying domains demonstrate the superiority of our method over existing supervised and semi-supervised AD methods.

In traditional multivariate data analysis, dimension reduction and regression have been treated as distinct endeavors. Established techniques such as principal component regression (PCR) and partial least squares (PLS) regression traditionally compute latent components as intermediary steps—although with different underlying criteria—before proceeding with the regression analysis. In this paper, we introduce an innovative regression methodology named PLS-integrated Lasso (PLS-Lasso) that integrates the concept of dimension reduction directly into the regression process. We present two distinct formulations for PLS-Lasso, denoted as PLS-Lasso-v1 and PLS-Lasso-v2, along with clear and effective algorithms that ensure convergence to global optima. PLS-Lasso-v1 and PLS-Lasso-v2 are compared with Lasso on the task of financial index tracking and show promising results.

In this paper, we define reduced dimensional predictors in latent dynamic systems in contrast to the traditional full-dimensional predictor models. Then the estimation of the new latent vector autoregressive model is developed with an objective to maximize the predicted variance for a given number of latent variables. A new dynamic predictive monitoring index that accounts for variations in the prediction residual and the predictor is developed. The residuals are modeled with a subsequent principal component analysis and a comprehensive monitoring method is developed to detect abnormal situations in industrial and operational systems. The new algorithm is tested on a simple closed-loop control system and the revamped Tennessee Eastman simulated process to show its effectiveness compared to other state-of-the-art methods.

Traffic prediction is a typical spatio-temporal data mining task and has great significance to the public transportation system. Considering the demand for its grand application, we recognize key factors for an ideal spatio-temporal prediction method: efficient, lightweight, and effective. However, the current deep model-based spatio-temporal prediction solutions generally own intricate architectures with cumbersome optimization, which can hardly meet these expectations. To accomplish the above goals, we propose an intuitive and novel framework, MLPST, a pure multi-layer perceptron architecture for traffic prediction. Specifically, we first capture spatial relationships from both local and global receptive fields. Then, temporal dependencies in different intervals are comprehensively considered. Through compact and swift MLP processing, MLPST can well capture the spatial and temporal dependencies while requiring only linear computational complexity, as well as model parameters that are more than an order of magnitude lower than baselines. Extensive experiments validated the superior effectiveness and efficiency of MLPST against advanced baselines, and among models with optimal accuracy, MLPST achieves the best time and space efficiency.

Industrial data are usually collinear, which can cause pure data-driven sparse learning to deselect physically relevant variables and select collinear surrogates. In this article, a novel two-step learning approach to retaining knowledge-informed variables (KIVs) is proposed to build inferential models. The first step is an improved knowledge-informed Lasso (KILasso) algorithm by removing penalty on the KIVs to produce a series of candidate subsets that guarantee the retention of the KIVs. The candidate subsets are then used to run the KILasso or ridge regression again to select the best sets of variables and estimate the final model. Two new algorithms are proposed and applied to datasets from an industrial boiler process and the Dow Chemical challenge problem. It is demonstrated that some important physically relevant variables are deselected by pure data-driven sparse methods, but they are retained using the proposed knowledge-informed methods with superior prediction performance.

Missing data widely exist in industrial processes and lead to difficulties in modeling, monitoring, fault diagnosis, and control. In this article, we propose a nonlinear method to handle the missing data problem in the offline modeling stage or/and the online monitoring stage of statistical process monitoring. We provide a fast incremental nonlinear matrix completion (FINLMC) method for missing data imputation, which enables us to use kernel methods such as kernel principal component analysis to monitor nonlinear multivariate processes even when there are missing data. We also provide theoretical analysis for the effectiveness of the proposed method. Experiments show that the proposed method can reduce the false alarm rate and improve the fault detection rate in nonlinear processing monitoring with missing data. The proposed FINLMC method can also be used to solve missing data in other problems such as classification and process control.

This Special Section presents the latest developments on intelligent modeling, neural networks, deep learning, and adaptive estimation, and their applications in industrial applications. Through a rigorous peer-review process, eleven articles have been accepted, which are summarized below.

Dynamic-inner canonical correlation analysis (DiCCA) extracts dynamic latent variables from high-dimensional time series data with a descending order of predictability in terms of R2. The reduced dimensional latent variables with rank-ordered predictability capture the dynamic features in the data, leading to easy interpretation and visualization. In this article, numerically efficient algorithms for DiCCA are developed to extract dynamic latent components from high-dimensional time series data. The numerically improved DiCCA algorithms avoid repeatedly inverting a covariance matrix inside the iteration loop of the numerical DiCCA algorithms. A further improvement using singular value decomposition converts the generalized eigenvector problem into a standard eigenvector problem for the DiCCA solution. Another improvement in model efficiency in this article is the dynamic model compaction of the extracted latent scores using autoregressive integrated moving average (ARIMA) models. Integrating factors, if existed in the latent variable scores, are made explicit in the ARIMA models. Numerical tests on two industrial datasets are provided to illustrate the improvements.

Motor fault diagnosis is imperative to enhance the reliability and security of industrial systems. However, since motors are often operated under nonstationary conditions, the high complexity of vibration signals raises notable difficulties for fault diagnosis. Therefore, considering the special physical characteristics of motor signals under nonstationary conditions, in this article, we propose a multiscale kernel based residual convolutional neural network (CNN) for motor fault diagnosis. Our contributions mainly fall into two aspects. First, we notice that each motor fault category has various patterns in vibration signals due to the changing operational conditions of the motor. To capture these patterns, a multiscale kernel algorithm is applied in the CNN architecture. Second, since the motor vibration signals are made up of many different components from different transfer paths, they are very complex and variable. To enable the architecture to extract fault features from deep and hierarchical representation spaces, sufficient depth of the network is needed, which will lead to the degradation problem. In the proposed method, residual learning is embedded into the multiscale kernel CNN to avoid performance degradation and build a deeper network. To validate the effectiveness of the proposed networks, a normal motor and five motors with different failures are tested. The results and comparisons with state-of-the-art methods highlight the superiority of the proposed method.

This paper considers a city with a large fleet of plug-in electric taxis (PETs) and studies the charging coordination problem of the fleet. The goal is to reduce charging cost for each PET, defined as the loss of service income caused by charging, by wisely choosing when and where to charge. Considering the fact that the fleet can contain thousands of autonomous PETs, this problem is approached in a distributed way. In detail, a two-stage decision process is designed for each PET in an online fashion upon receiving real-time information. In the first stage, a thresholding method is proposed to assist a PET driver in choosing a proper time slot for charging, with comprehensive consideration of state of charge of PET, time varying income, and queuing status at charging stations (CSs). In the second stage, a game-theoretical approach is devised for PETs to select CSs, so that the traveling and queuing time of each PET can be reduced with fairness. Extensive numerical simulations illustrate the following threefold benefits of the proposed approach: it can effectively reduce the charging cost for PETs, enhance the utilization ratio for CSs, and also flatten the unevenness of charging request for power grid.

In this paper, a dynamic modeling method for unevenly sampled data is proposed for the monitoring of bi-layer (i.e., a process layer and a quality layer) dynamic processes. First, a novel uneven data dynamic canonical correlation analysis method with an integrated dynamic time window is proposed for interlayer latent structure modeling, which captures the dynamic relations between regularly sampled process data and quality data with slow and irregular sampling. The new model is a step toward big data modeling to deal with data irregularity and diversity. Second, after extracting covariations using an interlayer model, intralayer variations are extracted using subsequent principal component analysis on the residual subspaces of the original process data and quality data, respectively. Third, a concurrent monitoring method for unevenly sampled bi-layer data is proposed. Finally, the proposed method is demonstrated using an illustrative simulation example and applied successfully to a real blast furnace iron-making process.

Projection to latent structures (PLS) and concurrent PLS are approaches for solving quality-relevant process monitoring. In this paper, a new approach called concurrent kernel PLS (CKPLS) is presented to detect faults comprehensively for nonlinear processes. The new model divides the nonlinear process and quality spaces into five subspaces: the co-varying, process-principal, process-residual, quality-principal, and quality-residual subspaces. The co-varying subspace reflects nonlinear relationship between quality variables and original process variables. The process-principal and process-residual subspaces reflect the principal variations and residuals, respectively, in the nonlinear process space. Further, the quality-principal and quality-residual subspaces reflect the principal variations and residuals, respectively, in the quality space. The proposed approach is demonstrated by a numerical simulation and an application of the Tennessee Eastman process.

The 17 papers in this special section focus on integrated optimization of industrial automation.

Process monitoring and fault diagnosis of the continuous annealing process lines (CAPLs) have been a primary concern in industry. Stable operation of the line is essential to final product quality and continuous processing of the upstream and downstream materials. In this paper, a multilevel principal component analysis (MLPCA)-based fault diagnosis method is proposed to provide meaningful monitoring of the underlying process and help diagnose faults. First, multiblock consensus principal component analysis (CPCA) is extended to MLPCA to model the large scale continuous annealing process. Secondly, a decentralized fault diagnosis approach is designed based on the proposed MLPCA algorithm. Finally, experiment results on an industrial CAPL are obtained to demonstrate the effectiveness of the proposed method.

In data-based monitoring field, the nonlinear iterative partial least squares procedure has been a useful tool for process data modeling, which is also the foundation of projection to latent structures (PLS) models. To describe the dynamic processes properly, a dynamic PLS algorithm is proposed in this paper for dynamic process modeling, which captures the dynamic correlation between the measurement block and quality data block. For the purpose of process monitoring, a dynamic total PLS (T-PLS) model is presented to decompose the measurement block into four subspaces. The new model is the dynamic extension of the T-PLS model, which is efficient for detecting quality-related abnormal situation. Several examples are given to show the effectiveness of dynamic T-PLS models and the corresponding fault detection methods.

The continuous annealing process line (CAPL) of cold rolling is an important unit to improve the mechanical properties of steel strips in steel making. In continuous annealing processes, strip tension is an important factor, which indicates whether the line operates steadily. Abnormal tension profile distribution along the production line can lead to strip break and roll slippage. Therefore, it is essential to estimate the whole tension profile in order to prevent the occurrence of faults. However, in real annealing processes, only a limited number of strip tension sensors are installed along the machine direction. Since the effects of strip temperature, gas flow, bearing friction, strip inertia, and roll eccentricity can lead to nonlinear tension dynamics, it is difficult to apply the first-principles induced model to estimate the tension profile distribution. In this paper, a novel data-based hybrid tension estimation and fault diagnosis method is proposed to estimate the unmeasured tension between two neighboring rolls. The main model is established by an observer-based method using a limited number of measured tensions, speeds, and currents of each roll, where the tension error compensation model is designed by applying neural networks principal component regression. The corresponding tension fault diagnosis method is designed using the estimated tensions. Finally, the proposed tension estimation and fault diagnosis method was applied to a real CAPL in a steel-making company, demonstrating the effectiveness of the proposed method.

In this paper, a decentralized fault diagnosis approach of complex processes is proposed based on multiblock kernel partial least squares (MBKPLS). To solve the problem posed by nonlinear characteristics, kernel partial least squares (KPLS) approaches have been proposed. In this paper, MBKPLS algorithm is first proposed and applied to monitor large-scale processes. The advantages of MBKPLS are: 1) MBKPLS can capture more useful information between and within blocks compared to partial least squares (PLS); 2) MBKPLS gives nonlinear interpretation compared to MBPLS; 3) Fault diagnosis becomes possible if number of sub-blocks is equal to the number of the variables compared to KPLS. The proposed methods are applied to process monitoring of a continuous annealing process. Application results indicate that the proposed decentralized monitoring scheme effectively captures the complex relations in the process and improves the diagnosis ability tremendously.

Although a fuzzy logic controller is generally nonlinear, a PI-type fuzzy controller that uses only control error and change in control error is not able to detect the process nonlinearity and make a control move accordingly. In this paper, a multiregion fuzzy logic controller is proposed for nonlinear process control. Based on prior knowledge, the process to be controlled is divided into fuzzy regions such as high-gain, low-gain, large-time-constant, and small-time-constant. Then a fuzzy controller is designed based on the regional information. Using an auxiliary process variable to detect the process operating regions, the resulting multiregion fuzzy logic controller can give satisfactory performance in all regions. Rule combination and controller tuning are discussed. Application of the controller to pH control is demonstrated.>