The black-box domain adaptation (BBDA) topic is developed to address the privacy and security issues where only an application programming interface (API) of the source model is available for domain adaptations. Although the BBDA topic has attracted growing research attentions, existing works mostly target the vision applications and are not directly applicable to the time-series applications possessing unique spatio-temporal characteristics. In addition, none of existing approaches have explored the strength of foundation model for black box time-series domain adaptation (BBTSDA). This paper proposes a concept of Cross-Prompt Foundation Model (CPFM) for the BBTSDA problems. CPFM is constructed under a dual branch network structure where each branch is equipped with a unique prompt to capture different characteristics of data distributions. In the domain adaptation phase, the reconstruction learning phases in the prompt and input levels are developed. All of which are built upon a time-series foundation model to overcome the spatio-temporal dynamic. Our rigorous experiments substantiate the advantage of CPFM achieving improved results with noticeable margins from its competitors in three time-series datasets of different application domains.

Services based on various forms of video streaming deliver content to clients by compressing originally captured videos using video codecs. During this process, due to limitations such as network bandwidth and display device capabilities, the original resolution is often reduced or a higher compression rate is applied, resulting in smaller data sizes being transmitted. Consequently, the video quality experienced by end users is often degraded. To address this issue, numerous AI-based Video Super-Resolution (VSR) techniques have been proposed. However, in addition to their high architectural complexity, many of these models require a large number of input frames or rely on recurrent frameworks, which further complicate both training and inference. These factors present significant challenges for deployment in real-time video services. In this paper, we propose a lightweight VSR model that achieves high performance even on low-quality, highly compressed video content. We propose an adaptive inference method that dynamically selects between lightweight and enhanced processing modules based on the input video’s resolution, thereby improving the suitability of our approach for real-time video streaming applications. Additionally, we visualize the performance of the proposed model with Grad-CAM to demonstrate its effectiveness compared to existing methods.

The data privacy constraint in online continual learning (OCL), where the data can be seen only once, complicates the catastrophic forgetting problem in streaming data. A common approach applied by the current SOTAs in OCL is with the use of memory saving exemplars or features from previous classes to be replayed in the current task. On the other hand, the prompt-based approach performs excellently in continual learning but with the cost of a growing number of trainable parameters. The first approach may not be applicable in practice due to data openness policy, while the second approach has the issue of throughput associated with the streaming data. In this study, we propose a novel prompt-based method for online continual learning that includes 4 main components: (1) single light-weight prompt generator as a general knowledge, (2) trainable scaler-and-shifter as specific knowledge, (3) pre-trained model (PTM) generalization preserving, and (4) hard-soft updates mechanism. Our proposed method achieves significantly higher performance than the current SOTAs in CIFAR100, ImageNet-R, ImageNet-A, and CUB dataset. Our complexity analysis shows that our method requires a relatively smaller number of parameters and achieves moderate training time, inference time, and throughput. For further study, the source code of our method is available at https://github.com/anwarmaxsum/PROL.

This study proposes a challenging yet practical Federated Few-Shot Class-Incremental Learning (FFSCIL) problem, where clients only hold very few samples for new classes. We develop a novel Unified Optimized Prototype Prompt (UOPP) model to simultaneously handle catastrophic forgetting, over-fitting, and prototype bias in FFSCIL. UOPP utilizes task-wise prompt learning to mitigate task interference and over-fitting, unified static-dynamic prototypes to achieve a stability-plasticity balance, and adaptive dual heads for enhanced inferences. Dynamic prototypes represent new classes in the current few-shot task and are rectified to deal with prototype bias. Our comprehensive experimental results show that UOPP significantly outperforms state-of-the-art (SOTA) methods on three datasets with improvements up to 76% on average accuracy and 90% on harmonic mean accuracy respectively. Our extensive analysis shows UOPP robustness in various numbers of local clients and global rounds, low communication costs, and moderate running time. The source code of UOPP is publicly available at https://github.com/anwarmaxsum/FFSCIL.

The prompt-based approach has demonstrated its success for continual learning problems. However, it still suffers from catastrophic forgetting due to inter-task vector similarity and unfitted new components of previously learned tasks. On the other hand, the language-guided approach falls short of its full potential due to minimum utilized knowledge and participation in the prompt tuning process. To correct this problem, we propose a novel prompt-based structure and algorithm that incorporate 4 key concepts (1) language as input for prompt generation (2) task-wise generators (3) limiting matching descriptors search space via soft task-id prediction (4) generated prompt as auxiliary data. Our experimental analysis shows the superiority of our method to existing SOTAs in CIFAR100, ImageNet-R, and CUB datasets with significant margins i.e. up to 30% final average accuracy, 24% cumulative average accuracy, 8% final forgetting measure, and 7% cumulative forgetting measure. Our historical analysis confirms our method successfully maintains the stability-plasticity trade-off in every task. Our robustness analysis shows the proposed method consistently achieves high performances in various prompt lengths, layer depths, and number of generators per task compared to the SOTAs. We provide a comprehensive theoretical analysis, and complete numerical results in appendix sections. The method code is available in https://github.com/anwarmaxsum/LEAPGEN for further study.

The existing approaches on continual learning (CL) call for a lot of samples in their training processes. Such approaches are impractical for many real-world problems having limited samples because of the overfitting problem. This article proposes a few-shot CL approach, termed flat-to-wide approach (FLOWER), where a flat-to-wide learning process finding the flat-wide minima is proposed to address the catastrophic forgetting (CF) problem. The issue of data scarcity is overcome with a data augmentation approach making use of a ball-generator concept to restrict the sampling space into the smallest enclosing ball. Our numerical studies demonstrate the advantage of FLOWER achieving significantly improved performances over prior arts notably in the small base tasks. For further study, source codes of FLOWER, competitor algorithms, and experimental logs are shared publicly in https://github.com/anwarmaxsum/FLOWER.

Federated Class Incremental Learning (FCIL) is a new direction in continual learning (CL) for addressing catastrophic forgetting and non-IID data distribution simultaneously. Existing FCIL methods call for high communication costs and exemplars from previous classes. We propose a novel rehearsal-free method for FCIL named prototypes-injected prompt (PIP) that involves 3 main ideas: a) prototype injection on prompt learning, b) prototype augmentation, and c) weighted Gaussian aggregation on the server side. Our experiment result shows that the proposed method outperforms the current state of the arts (SOTAs) with a significant improvement (up to 33%) in CIFAR100, MiniImageNet, and TinyImageNet datasets. Our extensive analysis demonstrates the robustness of PIP in different task sizes, and the advantage of requiring smaller participating local clients, and smaller global rounds. For further study, source codes of PIP, baseline, and experimental logs are shared publicly in https://github.com/anwarmaxsum/PIP.

Continual learning is a process that involves training learning agents to sequentially master a stream of tasks or classes without revisiting past data. The challenge lies in leveraging previously acquired knowledge to learn new tasks efficiently, while avoiding catastrophic forgetting. Existing methods primarily focus on single domains, restricting their applicability to specific problems. In this work, we introduce a novel approach called Cross-Domain Continual Learning (CDCL) that addresses the limitations of being limited to single supervised domains. Our method combines inter- and intra-task cross-attention mechanisms within a compact convolutional network. This integration enables the model to maintain alignment with features from previous tasks, thereby delaying the data drift that may occur between tasks, while performing unsupervised cross-domain (UDA) between related domains. By leveraging an intra-task-specific pseudo-labeling method, we ensure accurate input pairs for both labeled and unlabeled samples, enhancing the learning process. To validate our approach, we conduct extensive experiments on public UDA datasets, showcasing its positive performance on cross-domain continual learning challenges. Additionally, our work introduces incremental ideas that contribute to the advancement of this field. We make our code and models available to encourage further exploration and reproduction of our results: https://github.com/IvsucramlCDCL

Multitude of deep learning models have been proposed for node classification in graphs. However, they tend to perform poorly under labeled-data scarcity. Although Few-shot learning for graphs has been introduced to overcome this problem, the existing models are not easily adaptable for generic graph learning frameworks like Graph Neural Networks (GNNs). Our work proposes an Uncertainty Estimator framework that can be applied on top of any generic GNN backbone network (which are typically designed for supervised/semi-supervised node classification) to improve the node classification performance. A neural network is used to model the Uncertainty Estimator as a probability distribution rather than probabilistic discrete scalar values. We train these models under the classic episodic learning paradigm in the n-way, k-shot fashion, in an end-to-end setting.Our work demonstrates that implementation of the uncertainty estimator on a GNN backbone network improves the classification accuracy under Few-shot setting without any meta-learning specific architecture. We conduct experiments on multiple datasets under different Few-shot settings and different GNN-based backbone networks. Our method outperforms the baselines, which demonstrates the efficacy of the Uncertainty Estimator for Few-shot node classification on graphs with a GNN.

Few-Shot Class-Incremental Learning (FSCIL)presents an extension of the Class Incremental Learning (CIL)problem where a model is faced with the problem of data scarcity while addressing the Catastrophic Forgetting (CF)problem. This problem remains an open problem because all recent works are built upon the Convolutional Neural Networks (CNNs)performing sub-optimally compared to the transformer approaches. Our paper presents Robust Transformer Approach (ROBUSTA)built upon the Compact Convolutional Transformer (CCT). The issue of overfitting due to few samples is overcome with the notion of the stochastic classifier, where the classifier's weights are sampled from a distribution with mean and variance vectors, thus increasing the likelihood of correct classifications, and the batch-norm layer to stabilize the training process. The issue of CFis dealt with the idea of delta parameters, small task-specific trainable parameters while keeping the backbone networks frozen. A non-parametric approach is developed to infer the delta parameters for the model's predictions. The prototype rectification approach is applied to avoid biased prototype calculations due to the issue of data scarcity. The advantage of ROBUSTAis demonstrated through a series of experiments in the benchmark problems where it is capable of outperforming prior arts with big margins without any data augmentation protocols.

Artificial neural networks, celebrated for their human-like cognitive learning abilities, often encounter the well-known catastrophic forgetting (CF) problem, where the neural networks lose the proficiency in previously acquired knowledge. Despite numerous efforts to mitigate CF, it remains the significant challenge particularly in complex changing environments. This challenge is even more pronounced in cross-domain adaptation following the continual learning (CL) setting, which is a more challenging and realistic scenario that is under-explored. To this end, this article proposes a cross-domain CL approach making possible to deploy a single model in such environments without additional labelling costs. Our approach, namely continual learning approach for many processes (CLAMP), integrates a class-aware adversarial domain adaptation strategy to align a source domain and a target domain. An assessor-guided learning process is put forward to navigate the learning process of a base model assigning a set of weights to every sample controlling the influence of every sample and the interactions of each loss function in such a way to balance the stability and plasticity dilemma thus preventing the CF problem. The first assessor focuses on the negative transfer problem rejecting irrelevant samples of the source domain while the second assessor prevents noisy pseudo labels of the target domain. Both assessors are trained in the meta-learning approach using random transformation techniques and similar samples of the source domain. Theoretical analysis and extensive numerical validations demonstrate that CLAMP significantly outperforms established baseline algorithms across all experiments by at least 10% margin.

Most few-shot learning works rely on the same domain assumption between the base and the target tasks, hindering their practical applications. This paper proposes an adaptive transformer network (ADAPTER), a simple but effective solution for cross-domain few-shot learning where there exist large domain shifts between the base task and the target task. ADAPTER is built upon the idea of bidirectional cross-attention to learn transferable features between the two domains. The proposed architecture is trained with DINO to produce diverse, and less biased features to avoid the supervision collapse problem. Furthermore, the label smoothing approach is proposed to improve the consistency and reliability of the predictions by also considering the predicted labels of the close samples in the embedding space. The performance of ADAPTER is rigorously evaluated in the BSCD-FSL benchmarks in which it outperforms prior arts with significant margins.

A continual learning (CL) model is desired for remote sensing (RS) image analysis because of varying camera parameters, spectral ranges, resolutions, etc. There exist some recent initiatives to develop CL techniques in this domain but they still depend on massive labeled samples which do not fully fit RS applications because ground truths are often obtained via field-based surveys. This article addresses this problem with a proposal of unsupervised flat-wide learning approach (UNISA) for unsupervised few-shot CL approaches of RS image scene classifications which do not depend on any labeled samples for its model updates. UNISA is developed from the idea of prototype scattering and positive sampling for learning representations while the catastrophic forgetting (CF) problem is tackled with the flat-wide learning approach combined with a ball generator to address the data scarcity problem. Our numerical study with RS image scene datasets and a hyperspectral dataset confirms the advantages of our solution. For future studies and reproductions, source codes of UNISA are shared publicly inhttps://github.com/anwarmaxsum/UNISA.

The main objective of this study is to propose an enhanced wind power forecasting (EWPF) transformer model for handling power grid operations and boosting power market competition. It helps reliable large-scale integration of wind power relies in large part on accurate wind power forecasting (WPF). The proposed model is evaluated for single-step and multi-step WPF, and compared with gated recurrent unit (GRU) and long short-term memory (LSTM) models on a wind power dataset. The results of the study indicate that the proposed EWPF transformer model outperforms conventional recurrent neural network (RNN) models in terms of time-series forecasting accuracy. In particular, the results reveal a minimum performance improvement of 5% and a maximum of 20% compared to LSTM and GRU. These results indicate that the EWPF transformer model provides a promising alternative for wind power forecasting and has the potential to significantly improve the precision of WPF. The findings of this study have implications for energy producers and researchers in the field of WPF.

This paper proposes an assessor-guided learning strategy for continual learning where an assessor guides the learning process of a base learner by controlling the direction and pace of the learning process thus allowing an efficient learning of new environments while protecting against the catastrophic interference problem. The assessor is trained in a meta-learning manner with a meta-objective to boost the learning process of the base learner. It performs a soft-weighting mechanism of every sample accepting positive samples while rejecting negative samples. The training objective of a base learner is to minimize a meta-weighted combination of the cross entropy loss function, the dark experience replay (DER) loss function and the knowledge distillation loss function whose interactions are controlled in such a way to attain an improved performance. A compensated over-sampling (COS) strategy is developed to overcome the class imbalanced problem of the episodic memory due to limited memory budgets. Our approach, Assessor-Guided Learning Approach (AGLA), has been evaluated in the class-incremental and task-incremental learning problems. AGLA achieves improved performances compared to its competitors while the theoretical analysis of the COS strategy is offered. Source codes of AGLA, baseline algorithms and experimental logs are shared publicly in https://github.com/anwarmaxsum/AGLA for further study.

A deep clustering network (DCN) is desired for data streams because of its aptitude in extracting natural features thus bypassing the laborious feature engineering step. While automatic construction of deep networks in streaming environments remains an open issue, it is also hindered by the expensive labeling cost of data streams rendering the increasing demand for unsupervised approaches. This article presents an unsupervised approach of DCN construction on the fly via simultaneous deep learning and clustering termed autonomous DCN (ADCN). It combines the feature extraction layer and autonomous fully connected layer in which both network width and depth are self-evolved from data streams based on the bias-variance decomposition of reconstruction loss. The self-clustering mechanism is performed in the deep embedding space of every fully connected layer, while the final output is inferred via the summation of cluster prediction score. Furthermore, a latent-based regularization is incorporated to resolve the catastrophic forgetting issue. A rigorous numerical study has shown that ADCN produces better performance compared with its counterparts while offering fully autonomous construction of ADCN structure in streaming environments in the absence of any labeled samples for model updates. To support the reproducible research initiative, codes, supplementary material, and raw results of ADCN are made available in https://github.com/andriash001/AutonomousDCN.git.

A cross domain multistream classification is a challenging problem calling for fast domain adaptations to handle different but related streams in never-ending and rapidly changing environments. Notwithstanding that existing multistream classifiers assume no labeled samples in the target stream, they still incur expensive labeling costs since they require fully labeled samples of the source stream. This article aims to attack the problem of extreme label shortage in the cross domain multistream classification problems where only very few labeled samples of the source stream are provided before process runs. Our solution, namely, Learning Streaming Process from Partial Ground Truth (LEOPARD), is built upon a flexible deep clustering network where its hidden nodes, layers, and clusters are added and removed dynamically with respect to varying data distributions. A deep clustering strategy is underpinned by a simultaneous feature learning and clustering technique leading to clustering-friendly latent spaces. A domain adaptation strategy relies on the adversarial domain adaptation technique where a feature extractor is trained to fool a domain classifier by classifying source and target streams. Our numerical study demonstrates the efficacy of LEOPARD where it delivers improved performances compared to prominent algorithms in 15 of 24 cases. Source codes of LEOPARD are shared in https://github.com/wengweng001/LEOPARD.git to enable further study.

Graph Neural Networks (GNNs) have become the backbone for a myriad of tasks pertaining to graphs and similar topological data structures. While many works have been established in domains related to node and graph classification/regression tasks, they mostly deal with a single task. Continual learning on graphs is largely unexplored and existing graph continual learning approaches are limited to the task-incremental learning scenarios. This paper proposes a graph continual learning strategy that combines the architecture-based and memory-based approaches. The structural learning strategy is driven by reinforcement learning, where a controller network is trained in such a way to determine an optimal number of nodes to be added/pruned from the base network when new tasks are observed, thus assuring sufficient network capacities. The parameter learning strategy is underpinned by the concept of Dark Experience replay method to cope with the catastrophic forgetting problem. Our approach is numerically validated with several graph continual learning benchmark problems in both task-incremental learning and class-incremental learning settings. Compared to recently published works, our approach demonstrates improved performance in both the settings. The implementation code can be found at https://github.com/codexhammer/gcl.

Many real-world classification problems have imbalanced frequency of class labels; a well-known issue known as the "class imbalance" problem. Classic classification algorithms tend to be biased towards the majority class, leaving the classifier vulnerable to misclassification of the minority class. While the literature is rich with methods to fix this problem, as the dimensionality of the problem increases, many of these methods do not scale-up and the cost of running them become prohibitive. In this paper, we present an end-to-end deep generative classifier. We propose a domain-constraint autoencoder to preserve the latent-space as prior for a generator, which is then used to play an adversarial game with two other deep networks, a discriminator and a classifier. Extensive experiments are carried out on three different multi-class imbalanced problems and a comparison with state-of-the-art methods. Experimental results confirmed the superiority of our method over popular algorithms in handling high-dimensional imbalanced classification problems. Our code is available on https://github.com/TanmDL/SLPPL-GAN

Designing a Convolutional Neural Networks (CNN) is a complex task and requires expert knowledge to optimize the performance and network architecture. In this paper, a novel data-driven approach is proposed to determine the architecture of CNN models. The proposed Autonomous Convolutional Neural Networks (AutoCNNThe executable code and original numerical results can be downloaded from (https://tinyurl.com/AutoCNN)) algorithm introduces data driven strategies for addition of new convolutional layers, pruning of redundant filters and training cycle optimization. AutoCNN is evaluated using MNIST, MNIST-rot-back-image, Fashion MNIST and the ADHD200 datasets to measure the performance on small datasets with varied feature distributions. The results indicate that AutoCNN optimizes the CNN network architecture and helps maximise the classification performance. The data-driven network determination approach introduced in this paper was found to not only provides competitive performance similar to existing evolutionary computation based network determination algorithms in literature, but was found to be an effective optimization tool to improve the performance of existing CNN architectures. Further, the AutoCNN was found to highly immune to noise in the dataset and has proven to be effective method to transfer knowledge between related datasets. Therefore, the AutoCNN is a highly versatile CNN architecture determination tool that has a wide range of applications in the field of autonomous driving, medical image analysis, image enhancement, camera based security monitoring and image based fault detection

The existing methods to calculate the Eigenvector Centrality(EC) tend to not be robust enough for determination of EC in low time complexity or not well-scalable for large networks, hence rendering them practically unreliable/ computationally expensive. So, it is of the essence to develop a method that is scalable in low computational time. Hence, we propose a deep learning model for the identification of nodes with high Eigenvector Centrality. There have been a few previous works in identifying the high ranked nodes with supervised learning methods, but in real-world cases, the graphs are not labelled and hence deployment of supervised learning methods becomes a hazard and its usage becomes impractical. So, we devise CUL(Centrality with Unsupervised Learning) method to learn the relative EC scores in a network in an unsupervised manner. To achieve this, we develop an Encoder-Decoder based framework that maps the nodes to their respective estimated EC scores. Extensive experiments were conducted on different synthetic and real-world networks. We compared CUL against a baseline supervised method for EC estimation similar to some of the past works. It was observed that even with training on a minuscule number of training datasets, CUL delivers a relatively better accuracy score when identifying the higher ranked nodes than its supervised counterpart. We also show that CUL is much faster and has a smaller runtime than the conventional baseline method for EC computation. The code is available at https://github.com/codexhammer/CUL.

Current aerial robots are increasingly adaptive; they can morph to enable operation in changing conditions to complete diverse missions. Each mission may require the robot to conduct a different task. A conventional learning approach can handle these variations when the system is trained for similar tasks in a representative environment. However, it may result in overfitting to the new data stream or the failure to adapt, leading to degradation or a potential crash. These problems can be mitigated with an excessive amount of data and embedded model, but the computational power and the memory of the aerial robots are limited. In order to address the variations in the model, environment as well as the tasks within onboard computation limitations, we propose a deep neuromorphic controller approach with variable topologies to handle each different condition and the data stream with a feasible computation and memory allocation. The proposed approach is based on a deep neuromorphic (multi and variable layered neural network) controller with dynamic depth and progressive layer adaptation for each new data stream. This adaptive structure is combined with a switching function to form a sliding mode controller. The network parameter update rule guarantees the stability of the closed loop system by the convergence of the error dynamics to the sliding surface. Being the first implementation on an aerial robot in this context, the results illustrate the adaptation capability, stability, computational efficiency as well as the real-time validation.

The common practice of quality monitoring in industry relies on manual inspection well-known to be slow, error-prone and operator-dependent. This issue raises strong demand for automated real-time quality monitoring developed from data-driven approaches thus alleviating from operator dependence and adapting to various process uncertainties. Nonetheless, current approaches do not take into account the streaming nature of sensory information while relying heavily on handcrafted features making them application-specific. This paper proposes the online quality monitoring methodology developed from recently developed deep learning algorithms for data streams, Neural Networks with Dynamically Evolved Capacity (NADINE), namely NADINE++. It features the integration of 1-D and 2-D convolutional layers to extract natural features of time-series and visual data streams captured from sensors and cameras of the injection molding machines from our own project. Real-time experiments have been conducted where the online quality monitoring task is simulated on the fly under the prequential test-then-train fashion - the prominent data stream evaluation protocol. Comparison with the state-of-the-art techniques clearly exhibits the advantage of NADINE++ with 4.68% improvement on average for the quality monitoring task in streaming environments. To support the reproducible research initiative, codes, results of NADINE++ along with supplementary materials and injection molding dataset are made available in https://github.com/ContinualAL/NADINE-IJCNN2021.

Evolving fuzzy systems (EFS) have received increased attention from the community for the purpose of data stream modeling in an incremental, single-pass and transparent manner. To date, a wide variety of EFS approaches have been developed and successfully used in real-world applications which address structural evolution and parameter adaptation in single EFS models. We propose a specific ensemble scheme of EFS to increase their robustness in predictive performance on new stream samples. Our approach relies on an online variant of bagging in which various EFS ensemble members are generated from online bags, that is, the members are updated based on a specific probabilistic online sampling technique, and this with guaranteed convergence to classical sampling in batch bagging. The autonomous pruning of ensemble members is undertaken to omit undesired members with atypically higher errors than other members. We propose two variants, hard pruning where undesired members are deleted forever from the ensemble, and soft pruning where members receive weights to calculate the overall ensemble prediction, according to their single performance; thus, members who are undesired at a certain point of time may be dynamically recalled at a later stage. The autonomous evolution of new ensemble members is carried out whenever a drift in the stream is detected, based on a significantly worsening performance indicator, measured in terms of the Hoeffding inequality. Newer members typically represent the drifted state better and are thus up-weighed compared to older members within an advanced (weighted) calculation of the overall ensemble prediction. The new approach termed online bagged EFS (OB-EFS) was successfully evaluated and compared with single EFS models and related SoA approaches on four data streams from real-world applications (containing various noise levels, drifts and new operating conditions) and showed significantly lower prediction error trend lines.

When it comes to time-series forecasting, it is crucial to learn the intricate temporal relationship between past and future, and historical information is of paramount importance for this purpose. Traditional neuro-fuzzy systems generally resort to an empirical (offline) method to determine the number of past instances (i.e., historical information) required for a particular model, hence often unsuitable in an online time-series scenario. In this article, we propose a Bayesian neuro-fuzzy inference system (BaNFIS), where the temporal dependence on past instances is estimated with an online Bayesian probabilistic mechanism, and the uncertainty associated with real-world data is handled by the fuzzy inference system. The BaNFIS retains historical information only as per necessity and employs it in two ways: globally or locally. Moreover, an online learning method is employed here to update the BaNFIS parameters. Hence, the BaNFIS is able to capture both the system dynamics and uncertainty efficiently in an online manner. Three real-world time-series problems are employed here to evaluate the online performance of the BaNFIS compared to seven state-of-the-art neuro-fuzzy methods both under standard train–test and prequential test–train protocols. Numerical results clearly indicate that the BaNFIS provides a statistically improved prediction performance than its peers in terms of accuracy.

Finding synergy between a variety of control and estimation approaches can lead to effective solutions for controlling nonlinear dynamic systems in an efficient and systematic manner. In this paper, a novel controller design consisting of generalized regression neural networks (GRNNs) and sliding mode control (SMC) is proposed to control nonlinear multi-input and multi-output (MIMO) dynamic systems. The proposed design transforms GRNN from an offline regression model to an online adaptive controller. The suggested controller does not require any pretraining and it learns quickly from scratch. It uses a low computational complexity algorithm to provide accurate and stable performance. The proposed controller (GRNNSMC) performance is verified with a generic MIMO nonlinear dynamic system and a hexacopter model with a variable center of gravity. The results are compared with the standard PID controller. In addition, the stability of the GRNNSMC controller is verified using the Lyapunov stability method.

To investigate the robustness of complex networks in face of disturbances can help prevent potential network disasters. Percolation on networks is a potent instrument for network robustness analysis. However, existing percolation theories are primarily developed for interdependent or multilayer networks. Little attention is paid to multipartite networks which are an indispensable part of complex networks. In this article, we theoretically explore the robustness of multipartite networks under node failures. We put forward the generic percolation theory for gauging the robustness of multipartite networks with arbitrary degree distributions. Our developed theory is capable of quantifying the robustness of multipartite networks under either random or target node attacks. Our theory unravels the second order phase transition phenomenon for multipartite networks. In order to verify the correctness of the proposed theory, simulations on computer generated multipartite networks have been carried out. The experiments demonstrate that the simulation results coincide quite well with that yielded by the proposed theory.

Stream classification models for non-stationary environments often assume the immediate availability of data labels. However, in a practical scenario, it is quite natural that the data labels are available only after some temporal lag. This paper explores how a stream classifier model can be made adaptive to such label latency scenario. We propose SkipE-RNN, a self-evolutionary recurrent neural network with dynamically evolving skipped-recurrent-connection for the best utilization of previously observed label information while classifying the current data. When the data label is unavailable, SkipE-RNN uses an auto-learned mapping function to find the best match from the already known data labels and updates the network parameter accordingly. Later, upon availability of true data label, if the previously mapped label is found to be incorrect, SkipE-RNN employs a regularization technique along with the parameter updating process, so as to penalize the model. In addition, SkipE-RNN has inborn power of self-adjusting the network capacity by growing/pruning hidden nodes to cope with the evolving nature of data stream. Rigorous empirical evaluations using synthetic as well as real-world datasets reveal effectiveness of SkipE-RNN in both finitely delayed and infinitely delayed data label scenarios.

When it comes to time series forecasting, lag in the predicted sequence can be a predominant issue. Unfortunately, this is often overlooked in most of the time series literature as this does not contribute to a high prediction error (i.e. MSE). However, it leads to a rather poor forecast in terms of movement prediction in time series. In this article, we tackle this basic problem with a novel trend driven mechanism. Trend, defined as the inherent pattern of the data, is extracted here and utilized next to perform a lag-free forecasting. We propose a generic and light Dual Network Solution (DNS), where the first network predicts the trend and the second network utilizes that predicted trend along with its historical information to capture the dynamical behavior of the time series efficiently. DNS exhibits a substantially improved (≈10% better) performance compared to more complex and resource-intensive state-of-the- art algorithms in large scale regression problems. Apart from the traditional Mean Squared Error (MSE), we also propose a new Movement Prediction Metric or MPM (for detection of lag in time series) as a new complementary performance metric to evaluate the efficacy of DNS better.

With the increasing demand of product supply, manufacturers are in urgent need of online tool condition monitoring (TCM) without compromising with the maintenance cost in terms of time as well as man-power requirement. However, the existing machine learning models for TCM are mostly offline and not suitable for the non-stationary environment of the machining settings. Moreover, the access of the ground truth always imposes a shutdown of the machining process and the existing models are severely affected by such delay in receiving labelled samples. In order to tackle these issues, we propose SERMON as a novel learning model based on a pair of self-evolving mutually-operative recurrent neural networks. The proposed SERMON is well-equipped with features for automated and real-time monitoring of machine fault status even in the finite/infinite label delay scenario. The experimental evaluation of SERMON using real-world dataset on 3D-printing process demonstrates its effectiveness in online fault detection under non-stationary as well as delayed label context of the machining process. Additional comparative study on large-scale benchmark streaming datasets further exhibits the scalability power of SERMON.

Existing methodologies for tool condition monitoring (TCM) still rely on batch approaches which cannot cope with a fast sampling rate of a metal cutting process. Furthermore, they require a retraining process to be completed from scratch when dealing with a new set of machining parameters. This paper presents an online TCM approach based on Parsimonious Ensemble+ (pENsemble+). The unique feature of pENsemble+ lies in its highly flexible principle where both the ensemble structure and base-classifier structure can automatically grow and shrink on the fly based on the characteristics of data streams. Moreover, the online feature selection scenario is integrated to actively sample relevant input attributes. This paper presents advancement of a newly developed ensemble learning algorithm, pENsemble, where the online active learning scenario is incorporated to reduce the operator's labeling effort. The ensemble merging scenario is proposed which allows reduction of ensemble complexity while retaining its diversity. Experimental studies utilizing two real-world manufacturing data streams: 1) metal turning and 2) 3-D-printing processes and comparisons with well-known algorithms were carried out. Furthermore, the efficacy of pENsemble+ was examined using benchmark concept drift data streams. It has been found that pENsemble+ incurs low structural complexity and results in a significant reduction of the operator's labeling effort.

In recent times, with the incremental demand for fully autonomous systems, research interests are observed in learning machine-based intelligent, self-organizing, and evolving controllers. In this paper, a new evolving and self-organizing controller, namely generic-controller (G-controller), is proposed. The G-controller works in a fully online mode with minor expert domain knowledge. It is developed by incorporating the sliding mode control (SMC) theory with an advanced incremental learning machine, namely generic evolving neuro-fuzzy inference system. The controller starts operating from scratch with an empty set of fuzzy rule, and therefore, no offline training is required. To cope with the changing dynamic characteristics of the plant, the controller can add or prune the rules on demand. Control law and adaptation laws for the consequent parameters are derived from the SMC algorithm to establish a stable closed-loop system, where the stability of the G-controller is guaranteed by using the Lyapunov function. The uniform asymptotic convergence of tracking error to zero is witnessed through the implication of an auxiliary robustifying control term. In addition, the implementation of the multivariate Gaussian function helps the controller to handle the nonaxis parallel data from the plant and consequently, enhances the robustness against uncertainties and environmental perturbations. Finally, the controller's performance has been evaluated by observing the tracking performance in controlling simulated plants of unmanned aerial vehicle, namely bio-inspired flapping wing micro air vehicle and hexacopter for a variety of trajectories.

Existing fuzzy neural networks (FNNs) are mostly developed under a shallow network configuration having lower generalization power than those of deep structures. This article proposes a novel self-organizing deep FNN, namely deep evolving fuzzy neural network (DEVFNN). Fuzzy rules can be automatically extracted from data streams or removed if they play limited role during their lifespan. The structure of the network can be deepened on demand by stacking additional layers using a drift detection method, which not only detects the covariate drift, variations of input space, but also accurately identifies the real drift, dynamic changes of both feature space and target space. The DEVFNN is developed under the stacked generalization principle via the feature augmentation concept, where a recently developed algorithm, namely generic classifier, drives the hidden layer. It is equipped by an automatic feature selection method, which controls activation and deactivation of input attributes to induce varying subsets of input features. A deep network simplification procedure is put forward using the concept of hidden layer merging to prevent the uncontrollable growth of dimensionality of input space due to the nature of the feature augmentation approach in building a deep network structure. The DEVFNN works in the samplewise fashion and is compatible for data stream applications. The efficacy of the DEVFNN has been thoroughly evaluated using seven datasets with nonstationary properties under the prequential test-then-train protocol. It has been compared with four popular continual learning algorithms and its shallow counterpart, where the DEVFNN demonstrates improvement of classification accuracy. Moreover, it is also shown that the concept of the drift detection method is an effective tool to control the depth of the network structure, while the hidden layer merging scenario is capable of simplifying the network complexity of a deep network with negligible compromise of generalization performance.

Analysis of resting state - functional Magnetic Resonance Imaging (rs-fMRI) data has been a challenging problem due to a high homogeneity, large intra-class variability, limited samples and difference in acquisition technologies/techniques. These issues are predominant in the case of Attention Deficit Hyperactivity Disorder (ADHD). In this paper, we propose a new Deep Transformation Method (DTM) that extracts the discriminant latent feature space from rsfMRI and projects it in the subsequent layer for classification of rs-fMRI data. The hidden transformation layer in DTM projects the original rs-fMRI data into a new space using the learning policy and extracts the spatio-temporal correlations of the functional activities as a latent feature space. The subsequent convolution and decision layers transform the latent feature space into high-level features and provide accurate classification. The performance of DTM has been evaluated using the ADHD200 rs-fMRI benchmark data with crossvalidation. The results show that the proposed DTM achieves a mean classification accuracy of 70.36% and an improvement of 8.25% on the state of the art methodologies was observed. The improvement is due to concurrent analysis of the spatio-temporal correlations between the different regions of the brain and can be easily extended to study other cognitive disorders using rs-fMRI. Further, brain network analysis has been studied to identify the difference in functional activities and the corresponding regions behind cognitive symptoms in ADHD.

The feasibility of existing data stream algorithms is often hindered by the weakly supervised condition of data streams. A self-evolving deep neural network, namely Parsimonious Network (ParsNet), is proposed as a solution to various weakly-supervised data stream problems. A self-labelling strategy with hedge (SLASH) is proposed in which its auto-correction mechanism copes with the accumulation of mistakes significantly affecting the model's generalization. ParsNet is developed from a closed-loop configuration of the self-evolving generative and discriminative training processes exploiting shared parameters in which its structure flexibly grows and shrinks to overcome the issue of concept drift with/without labels. The numerical evaluation has been performed under two challenging problems, namely sporadic access to ground truth and infinitely delayed access to the ground truth. Our numerical study shows the advantage of ParsNet with a substantial margin from its counterparts in the high-dimensional data streams and infinite delay simulation protocol. To support the reproducible research initiative, the source code of ParsNet along with supplementary materials are made available at https://bit.ly/2qNW7p4.

Transferring knowledge across many streaming processes remains an uncharted territory in the existing literature and features unique characteristics: no labelled instance of the target domain, covariate shift of source and target domain, different period of drifts in the source and target domains. Autonomous transfer learning (ATL) is proposed in this paper as a flexible deep learning approach for the online unsupervised transfer learning problem across many streaming processes. ATL offers an online domain adaptation strategy via the generative and discriminative phases coupled with the KL divergence based optimization strategy to produce a domain invariant network while putting forward an elastic network structure. It automatically evolves its network structure from scratch with/without the presence of ground truth to overcome independent concept drifts in the source and target domain. Rigorous numerical evaluation has been conducted along with comparison against recently published works. ATL demonstrates improved performance while showing significantly faster training speed than its counterparts.

Autonomous construction of deep neural network (DNNs) is desired for data streams because it potentially offers two advantages: proper model's capacity and quick reaction to drift and shift. While self-organizing mechanism of DNNs remains an open issue, this task is even more challenging to be developed for standard multi-layer DNNs than that using the different-depth structures, because addition of a new layer results in information loss of previously trained knowledge. A Neural Network with Dynamically Evolved Capacity (NADINE) is proposed in this paper. NADINE features a fully open structure where its network structure, depth and width, can be automatically evolved from scratch in the online manner and without the use of problem-specific thresholds. NADINE is structured under a standard MLP architecture and the catastrophic forgetting issue during the hidden layer addition phase is resolved using the proposal of soft-forgetting and adaptive memory methods. The advantage of NADINE, namely elastic structure and online learning trait, is numerically validated using nine data stream classification and regression problems where it demonstrates performance's improvement over prominent algorithms in all problems. In addition, it is capable of dealing with data stream regression and classification problems equally well.

In this work, a simple evolving neuro-fuzzy system with less learning parameters is utilized to develop an intelligent controller namely Reduced Parsimonious Controller (RedPAC). The proposed RedPAC is a simplified version of one of the recently developed intelligent controller called Parsimonious Controller (PAC). In RedPAC, the network parameters are reduced into two steps. Firstly, unlike the conventional fuzzy logic or neuro-fuzzy-based intelligent controller, it has no premise parameters. Secondly, in contrast with PAC, the number of consequent parameters have further reduced to one parameter per rule in RedPAC. The sliding mode control (SMC) technique is utilized to adapt consequent parameters of RedPAC, where the SMC-based auxiliary robustifying control term has guaranteed the uniform asymptotic convergence of tracking error to zero. The proposed controller's performance has been evaluated by implementing it to control a quadcopter unmanned aerial vehicle (UAV) simulator namely Dronekit. In addition, trajectory tracking performance of the quadcopter is compared with three different benchmark controllers namely a linear PID, a nonlinear SMC, and an intelligent controller called PAC. RedPAC outperforms PID and SMC techniques. The results of tracking trajectories are also comparable to PAC; however, RedPAC needs comparatively less learning parameters to obtain a similar or better tracking accuracy.

This paper presents the design of a robust evolving neuro-fuzzy output feedback control of a novel tilt-rotor vertical takeoff and landing (VTOL) aircraft. Detailed six degrees of freedom nonlinear model of the VTOL is presented. The asymmetry in the longitudinal centre of gravity position and the motor-propellers with diverse thrust and torque characteristics poses unique challenges for the control. The evolving neuro-fuzzy controller described here is called Parsimonious Autonomous Controller (PAC). PAC is based on model free design and is robust against system uncertainties and input disturbances. The effectiveness of the proposed controller is demonstrated through high fidelity numerical simulations for attitude and altitude control of the VTOL.

In this paper, we propose MUSE-RNN, a multilayer self-evolving recurrent neural network model for real-time classification of streaming data. Unlike the existing approaches, MUSE-RNN offers special treatment towards capturing temporal aspects of data stream through its novel recurrent learning approach based on the teacher forcing policy. Novelties here are twofold. First, in contrast to the traditional RNN models, MUSE-RNN has intrinsic ability to self-adjust its capacity by growing and pruning hidden nodes as well as layers, to handle the ever-changing characteristics of data stream. Second, MUSERNN adopts a unique scoring-based layer adaptation mechanism, which makes it capable of recalling prior tasks, with minimum exploitation of network parameters. The performance of MUSERNN is evaluated in comparison with a number of state-of-theart techniques, using seven popular data streams and continual learning problems under prequential test-then-train protocol. Experimental results demonstrate the effectiveness of MUSERNN in stream classification scenario.

The indoor positioning system can be applied to smart factories to monitor the location of time-critical items in real-time. It is useful for planning and control in the dynamic manufacturing environment. The challenge of the localization is the non-stationary characteristics of the environment. In this paper, we present a predictive modeling method using online machine learning. The wireless signals from time-critical items can be captured constantly. The online positioning model is built and updated by using the sensor data stream.

With the recent explosion of data navigating in motion, there is a growing research interest for analyzing streaming data, and consequently, there are several recent works on data stream analytics. However, exploring the potentials of traditional recurrent neural network (RNN) in the context of streaming data classification is still a little investigated area. In this paper, we propose a novel variant of RNN, termed as FERNN, which features single-pass learning capability along with self-evolution property. The online learning capability makes FERNN fit for working on streaming data, whereas the self-organizing property makes the model adaptive to the rapidly changing environment. FERNN utilizes hyperplane activation in the hidden layer, which not only reduces the network parameters to a significant extent, but also triggers the model to work by default as per teacher forcing mechanism so that it automatically handles the vanishing/exploding gradient issues in traditional RNN learning based on back-propagation-through-time policy. Moreover, unlike the majority of the existing autonomous learning models, FERNN is free from normal distribution assumption for streaming data, making it more flexible. The efficacy of FERNN is evaluated in terms of classifying six publicly available data streams, under the prequential test-then-train protocol. Experimental results show encouraging performance of FERNN attaining state-of-the-art classification accuracy with fairly reduced computation cost.

Two of the major challenges associated with time series modelling are handling uncertainty present in the data and tracing its dynamical behaviour. A Recurrent Interval Type 2 Fuzzy Inference System or RIT2FIS is presented in this paper. RIT2FIS adopts an interval type 2 fuzzy inference mechanism for superior handling of uncertainty. The memory neurons employed in its hidden and output layer, retain the temporal information, making RIT2FIS highly proficient in tracing system dynamics at a granular level. RIT2FIS also benefits from incorporating a k-means algorithm inspired approach to cluster the data in an unsupervised manner. An 'Elbow Method' is utilized next to determine the optimal clustering which is then employed as the optimal fuzzy rule base for RIT2FIS, eliminating the necessity of expert knowledge for fuzzy initiation. The antecedent and consequent parameters of RIT2FIS are updated using a gradient descent based back-propagation through time algorithm where the learning is made self-regulatory to avoid over-fitting and ensure generalization. Performance of RIT2FIS is evaluated against popular neuro-fuzzy methods on different benchmark and real-world time series problems which distinctly indicates an improved accuracy and a parsimonious rule base.

Being motivated by ceiling inspection applications via unmanned aerial vehicles (UAVs) which require close proximity flight to surfaces, a systematic control approach enabling safe and accurate close proximity flight is proposed in this work. There are two main challenges for close proximity flights: (i) the trust characteristics varies drastically for the different distance from the ceiling which results in a complex nonlinear dynamics; (ii) the system needs to consider physical and environmental constraints to safely fly in close proximity. To address these challenges, a novel framework consisting of a constrained optimization-based force estimation and an optimization-based nonlinear controller is proposed. Experimental results illustrate that the performance of the proposed control approach can stabilize UAV down to 1 cm distance to the ceiling. Furthermore, we report that the UAV consumes up to 12.5% less power when it is operated 1 cm distance to ceiling, which is promising potential for more battery-efficient inspection flights.

The feasibility of deep neural networks (DNNs) to address data stream problems still requires intensive study because of the static and offline nature of conventional deep learning approaches. A deep continual learning algorithm, namely autonomous deep learning (ADL), is proposed in this paper. Unlike traditional deep learning methods, ADL features a flexible structure where its network structure can be constructed from scratch with the absence of initial network structure via the self-constructing network structure. ADL specifically addresses catastrophic forgetting by having a different-depth structure which is capable of achieving a trade-off between plasticity and stability. Network significance (NS) formula is proposed to drive the hidden nodes growing and pruning mechanism. Drift detection scenario (DDS) is put forward to signal distributional changes in data streams which induce the creation of a new hidden layer. Maximum information compression index (MICI) method plays an important role as a complexity reduction module eliminating redundant layers. The efficacy of ADL is numerically validated under the prequential test-then-train procedure in lifelong environments using nine popular data stream problems. The numerical results demonstrate that ADL consistently outperforms recent continual learning methods while characterizing the automatic construction of network structures.

Data stream has been the underlying challenge in the age of big data because it calls for real-time data processing with the absence of a retraining process and/or an iterative learning approach. In the realm of the fuzzy system community, data stream is handled by algorithmic development of self-adaptive neuro-fuzzy systems (SANFS) characterized by the single-pass learning mode and the open structure property that enables effective handling of fast and rapidly changing natures of data streams. The underlying bottleneck of SANFSs lies in its design principle, which involves a high number of free parameters (rule premise and rule consequent) to be adapted in the training process. This figure can even double in the case of the type-2 fuzzy system. In this paper, a novel SANFS, namely parsimonious learning machine (PALM), is proposed. PALM features utilization of a new type of fuzzy rule based on the concept of hyperplane clustering, which significantly reduces the number of network parameters because it has no rule premise parameters. PALM is proposed in both type-1 and type-2 fuzzy systems where all of which characterize a fully dynamic rule-based system. That is, it is capable of automatically generating, merging, and tuning the hyperplane-based fuzzy rule in the single-pass manner. Moreover, an extension of PALM, namely recurrent PALM, is proposed and adopts the concept of teacher-forcing mechanism in the deep learning literature. The efficacy of PALM has been evaluated through numerical study with six real-world and synthetic data streams from public database and our own real-world project of autonomous vehicles. The proposed model showcases significant improvements in terms of computational complexity and number of required parameters against several renowned SANFSs, while attaining comparable and often better predictive accuracy.

Tracking the location of mobile manufacturing objects is essential for manufacturing execution system. This paper presents a review of recent RFID-based localization and tracking technologies. Based on the literature review, we studied the feasibility of indoor localization that locates items on the manufacturing shop-floor automatically using wireless signals and RF-based localization technologies. Although the real-time locating systems (RTLS) exist to provide real-time localization, these technologies are costly to implement on a large scale. In this feasibility study, we measured received signal strength and phase shift signals from commercial off-the-shelf (COTS) wireless devices for tracking the location of manufacturing objects.

One of the issues faced by manufacturing industry is a lack of automatic localization techniques. In this research, a Radio Frequency Identification (RFID) based localization system is proposed for resource tracking. In this study, we incorporated a RFID tag at each item to be tracked, and a RFID reference tag at each location zone. The encoded IDs are read to identify the names of items and location zones. At the same time, radio signals (received signal strength and phase) are measured as RFID fingerprints. Similarity measures are studied to compare fingerprints between RFID item tags and location reference tags to track the location of the items. The kernel-based learning method was implemented as similarity measure. Different cluster labelling methods were compared and it was found that the proximity method is more efficient. The clustering method is used to overcome the issues faced by traditional RFID based localization methods.

Nowadays, the application of fully autonomous system like rotary wing unamnned air vehicles (UAVs) are increasing sharply. Due to the complex nonlinear dynamics a huge research interest is witnessed in developing learning machine based intelligent, self-organizing evolving controller for these vehicles notably to address the system's dynamic characteristics. In this work, such an evolving controller namely Generic-controller (Gcontroller) is proposed to control the altitude of a rotary wing UAV namely hexacopter. This controller can work with very minor expert domain knowledge. The evolving architecture of this controller is based on an advanced incremental learning algorithm namely Generic Evolving Neuro-Fuzzy Inference System (GENEFIS). The controller does not require any offline training, since it starts operating from scratch with an empty set of fuzzy rules, and then add or delete rules on demand. The adaptation laws for the consequent paramters are derived from the sliding model control (SMC) theory. The Lyapunov theory is used to guarantee the stability of the proposed controller. In addition, an auxiliary robustifying control term is implemented to obtain an uniform asymptotic convergence of tracking error to zero. Finally, the G-controller's performance evaluation is observed through the altitude tracking of an UAV namely hexacopter for various trajectories.

RFID technology has gained popularity to address localization problem in the manufacturing shopfloor by tracking the manufacturing object location to increase the production's efficiency. However, the signals (data) used for localization task is not easy to analyze because it is generated from the nonstationary environment. It also continuously arrive over time and yields the large-volume of data. Therefore, an advanced big data analytic is required to overcome this problem. We propose a distributed big data analytic framework based on PANFIS (Scalable PANFIS), where PANFIS is an evolving algorithm which has capability to learn data stream in the single pass mode. Scalable PANFIS can learn big data stream by processing many chunks/partitions of data stream. Scalable PANFIS is also equipped with rule' structure merging to eliminate the redundancy among rules. Scalable PANFIS is validated by measuring its performance against single PANFIS and other Spark's scalable machine learning algorithms. The result shows that Scalable PANFIS performs running time more than 20 times faster than single PANFIS. The rule merging process in Scalable PANFIS shows that there is no significant reduction of accuracy in classification task with 96.67 percent of accuracy in comparison with single PANFIS of 98.71 percent. Scalable PANFIS also generally outperforms some Spark MLib machine learnings to classify RFID data with the comparable speed in running time.

In this paper, we propose three criteria for efficient sample selection in case of data stream regression problems within an online active learning context. The selection becomes important whenever the target values, which guide the update of the regressors as well as the implicit model structures, are costly or time-consuming to measure and also in case when very fast models updates are required to cope with stream mining real-time demands. Reducing the selected samples as much as possible while keeping the predictive accuracy of the models on a high level is, thus, a central challenge. This should be ideally achieved in unsupervised and single-pass manner. Our selection criteria rely on three aspects: 1) the extrapolation degree combined with the model's nonlinearity degree , which is measured in terms of a new specific homogeneity criterion among adjacent local approximators; 2) the uncertainty in model outputs, which can be measured in terms of confidence intervals using so-called adaptive local error bars - we integrate a weighted localization of an incremental noise level estimator and propose formulas for online merging of local error bars; 3) the uncertainty in model parameters, which is estimated by the so-called A-optimality criterion, which relies on the Fisher information matrix. The selection criteria are developed in combination with evolving generalized Takagi-Sugeno (TS) fuzzy models (containing rules in arbitrarily rotated position), as it could be shown in previous publications that these outperform conventional evolving TS models (containing axis-parallel rules). The results based on three high-dimensional real-world streaming problems show that a model update based on only 10%-20% selected samples can still achieve similar accumulated model errors over time to the case when performing a full model update on all samples. This can be achieved with a negligible sensitivity on the size of the active learning latency buffer. Random sampling with the same percentages of samples selected, however, achieved much higher error rates. Hence, the intelligence in our sample selection concept leads to an economic balance between model accuracy and measurement as well computational costs for model updates.

Gradual drifts in data streams are usually hard to detect and often do not necessarily trigger the evolution of new fuzzy rules during model adaptation steps in order to represent the new, drifted data distribution(s) appropriately in the fuzzy model. Over time, they thus lead to oversized rules with untypically large local errors (typically also worsening the global model error), as representing joint local data distributions before and after a drift happened likewise. We therefore propose an incremental rule splitting concept for generalized fuzzy rules in order to autonomously compensate these negative effects of gradual drifts. Our splitting condition is based on the local error of rules measured in terms of a weighted contribution to the whole model error and on the size of the rules measured in terms of the volume of the associated clusters. We use the concept of statistical process control in order to omit an extra threshold parameter in our splitting condition. The splitting technique relies on the eigendecomposition of the rule covariance matrix to adequately manipulate the largest eigenvector and eigenvalues in order to retrieve the new centers and contours of the two split rules. Furthermore, we guarantee sufficient flexibility in adapting the shapes and consequents of the split rules to the new drifted situation in the stream by integrating a specific dynamic and smooth forgetting concept of older samples, which formed the original (nonsplit) rules. Robustness against outliers is guaranteed by the realization of a two-layer model building process, where one layer represents the cluster partition and the other layer the rule partition: Only clusters becoming significant over time are accepted as rules in the fuzzy model. The splitting concepts are integrated in the generalized smart evolving learning engine for fuzzy systems (termed as Gen-Smart-EFS) and successfully tested on two real-world application scenarios, engine test benches and rolling mills, the latter including a real-occurring gradual drift (whose position in the data is known). Results show clearly improved error trend lines over time when splitting is applied, compared to the case when it is not applied: reduction of the mean absolute model error by about one third (rolling mills) and about one half (engine test benches).

The concept of ensemble learning offers a promising avenue in learning from data streams under complex environments because it better addresses the bias and variance dilemma than its single-model counterpart and features a reconfigurable structure, which is well suited to the given context. While various extensions of ensemble learning for mining nonstationary data streams can be found in the literature, most of them are crafted under static base-classifier and revisit preceding samples in the sliding window for a retraining step. This feature causes computationally prohibitive complexity and is not flexible enough to cope with rapidly changing environments. Their complexities are often demanding because they involve a large collection of offline classifiers due to the absence of structural complexities reduction mechanisms and lack of an online feature selection mechanism. A novel evolving ensemble classifier, namely Parsimonious Ensemble (pENsemble), is proposed in this paper. pENsemble differs from existing architectures in the fact that it is built upon an evolving classifier from data streams, termed Parsimonious Classifier. pENsemble is equipped by an ensemble pruning mechanism, which estimates a localized generalization error of a base classifier. A dynamic online feature selection scenario is integrated into the pENsemble. This method allows for dynamic selection and deselection of input features on the fly. pENsemble adopts a dynamic ensemble structure to output a final classification decision where it features a novel drift detection scenario to grow the ensemble's structure. The efficacy of the pENsemble has been numerically demonstrated through rigorous numerical studies with dynamic and evolving data streams, where it delivers the most encouraging performance in attaining a tradeoff between accuracy and complexity.

Randomized neural network (RNN) is a highly feasible solution in the era of big data because it offers a simple and fast working principle in processing dynamic and evolving data streams. This paper proposes a novel RNN, namely recurrent type-2 random vector functional link network (RT2McRVFLN), which provides a highly scalable solution for data streams in a strictly online and integrated framework. It is built upon the psychologically inspired concept of metacognitive learning, which covers three basic components of human learning: what-to-learn, how-to-learn, and when-to-learn. The what-to-learn selects important samples on the fly with the use of online active learning scenario, which renders our algorithm an online semi-supervised algorithm. The how-to-learn process combines an open structure of evolving concept and a randomized learning algorithm of random vector functional link network (RVFLN). The efficacy of the RT2McRVFLN has been numerically validated through two real-world case studies and comparisons with its counterparts, which arrive at a conclusive finding that our algorithm delivers a tradeoff between accuracy and simplicity.

Existing extreme learning algorithm have not taken into account four issues: 1) complexity; 2) uncertainty; 3) concept drift; and 4) high dimensionality. A novel incremental type-2 meta-cognitive extreme learning machine (ELM) called evolving type-2 ELM (eT2ELM) is proposed to cope with the four issues in this paper. The eT2ELM presents three main pillars of human meta-cognition: 1) what-to-learn; 2) how-to-learn; and 3) when-to-learn. The what-to-learn component selects important training samples for model updates by virtue of the online certainty-based active learning method, which renders eT2ELM as a semi-supervised classifier. The how-to-learn element develops a synergy between extreme learning theory and the evolving concept, whereby the hidden nodes can be generated and pruned automatically from data streams with no tuning of hidden nodes. The when-to-learn constituent makes use of the standard sample reserved strategy. A generalized interval type-2 fuzzy neural network is also put forward as a cognitive component, in which a hidden node is built upon the interval type-2 multivariate Gaussian function while exploiting a subset of Chebyshev series in the output node. The efficacy of the proposed eT2ELM is numerically validated in 12 data streams containing various concept drifts. The numerical results are confirmed by thorough statistical tests, where the eT2ELM demonstrates the most encouraging numerical results in delivering reliable prediction, while sustaining low complexity.

Multidocument summarization has gained popularity in many real world applications because vital information can be extracted within a short time. Extractive summarization aims to generate a summary of a document or a set of documents by ranking sentences and the ranking results rely heavily on the quality of sentence features. However, almost all previous algorithms require hand-crafted features for sentence representation. In this paper, we leverage on word embedding to represent sentences so as to avoid the intensive labor in feature engineering. An enhanced convolutional neural networks (CNNs) termed multiview CNNs is successfully developed to obtain the features of sentences and rank sentences jointly. Multiview learning is incorporated into the model to greatly enhance the learning capability of original CNN. We evaluate the generic summarization performance of our proposed method on five Document Understanding Conference datasets. The proposed system outperforms the state-of-the-art approaches and the improvement is statistically significant shown by paired t -test.

The age of online data stream and dynamic environments results in the increasing demand of advanced machine learning techniques to deal with concept drifts in large data streams. Evolving fuzzy systems (EFS) are one of recent initiatives from the fuzzy system community to resolve the issue. Existing EFSs are not robust against data uncertainty, temporal system dynamics, and the absence of system order, because a vast majority of EFSs are designed in the type-1 feedforward network architecture. This paper aims to solve the issue of data uncertainty, temporal behavior, and the absence of system order by developing a novel evolving recurrent fuzzy neural network, called evolving type-2 recurrent fuzzy neural network (eT2RFNN). eT2RFNN is constructed in a new recurrent network architecture, featuring double recurrent layers. The new recurrent network architecture evolves a generalized interval type-2 fuzzy rule, where the rule premise is built upon the interval type-2 multivariate Gaussian function, whereas the rule consequent is crafted by the nonlinear wavelet function. The eT2RFNN adopts a holistic concept of evolving systems, where the fuzzy rule can be automatically generated, pruned, merged, and recalled in the single-pass learning mode. eT2RFNN is capable of coping with the problem of high dimensionality because it is equipped with online feature selection technology. The efficacy of eT2RFNN was experimentally validated using artificial and real-world data streams and compared with prominent learning algorithms. eT2RFNN produced more reliable predictive accuracy, while retaining lower complexity than its counterparts.

This paper discusses an overview on different path planning algorithms and analyzes their performance. Based on this, an efficient hybrid algorithm is proposed for path planning in both static and dynamic environments. The numerical results are presented to validate the proposed design. The method is compared to other heuristic algorithms in different scenarios. The proposed algorithm is shown to be more efficient computationally than the other comparable algorithms.

This paper describes path replanning techniques and obstacles avoidance for autonomous vehicles, in complex environments. The Vectorized Particle Swarm optimization (VPSO) method is used for searching for an optimal path. A simultaneous replanning concept is incorporated with path planning to avoid static and dynamic obstacles. This proposed simultaneous replanning vectorized particle swarm optimization (SRVPSO) algorithm reduces the computational time of the path planning. Numerical results show that the SRVPSO algorithm provides better replanning time in a dynamic environment.

Extractive summarization aims to generate a summary by ranking sentences, whose performance relies heavily on the quality of sentence features. In this paper, a document summarization framework based on convolutional neural networks is successfully developed to learn sentence features and perform sentence ranking jointly. We adapt the original CNN model to address a regression process for sentence ranking. Pre-trained word vectors are used to enhance the performance of our model. We evaluate our proposed method on the DUC 2002 and 2004 datasets covering single and multi-document summarization tasks respectively. The proposed system achieves competitive or even better performance compared with state-of-the-art document summarization systems.

Evolving intelligent system (EIS) is a machine learning algorithm, specifically designed to deal with learning from large data streams. Although the EIS research topic has attracted various contributions over the past decade, the issue of uncertainty, temporal system dynamic, and system order are relatively unexplored by existing studies. A novel EIS, namely evolving type-2 recurrent fuzzy neural network (eT2RFNN) is proposed in this paper. eT2RFNN features a novel recurrent network architecture, possessing double local recurrent connections. It generates a generalized interval type-2 fuzzy rule, where an interval type-2 multivariate Gaussian function constructs the rule premise, and the rule consequent is crafted by the nonlinear wavelet function. eT2RFNN adopts an open structure, where it can start learning process from scratch with an empty rule base. Fuzzy rules can be automatically generated according to degree of nonlinearity data stream conveys. It can performs a rule base simplification procedure by pruning and merging inactive, outdated and overlapping rules. eT2RFNN can deal with the high dimensionality problem, where an online dimensionality reduction method is integrated in the training process. The efficacy of the eT2RFNN has been numerically validated using two real-world data streams, where it provides high predictive accuracy, while retaining low complexity.

In this paper, a novel extreme learning machine based online multi-label classifier for real-time data streams is proposed. Multi-label classification is one of the actively researched machine learning paradigm that has gained much attention in the recent years due to its rapidly increasing real world applications. In contrast to traditional binary and multi-class classification, multi-label classification involves association of each of the input samples with a set of target labels simultaneously. There are no real-time online neural network based multi-label classifier available in the literature. In this paper, we exploit the inherent nature of high speed exhibited by the extreme learning machines to develop a novel online real-time classifier for multi-label data streams. The developed classifier is experimented with datasets from different application domains for consistency, performance and speed. The experimental studies show that the proposed method outperforms the existing state-of-the-art techniques in terms of speed and accuracy and can classify multi-label data streams in real-time.

Sentiment classification has been a very hot topic in the field of natural language processing (NLP) and understanding in recent years. Recurrent neural networks (RNN) is a widely used tool to deal with the classification problem of variable-length sentences. The standard RNN can only access the preceding context of a sentence. In this paper, a new architecture termed Comprehensive Attention Recurrent Neural Networks (CA-RNN) which can store preceding, succeeding and local contexts of any position in a sequence is developed. The bidirectional recurrent neural networks (BRNN) is used to access the past and future information while a convolutional layer is employed to capture local information. The standard RNN is also replaced by two recently emerged RNN variants, namely long short-term memory (LSTM) and gated recurrent unit (GRU), to enhance the effectiveness of the new architecture. Another salient feature of the proposed model is that it can be trained end-to-end without any human intervention. It is very easy to be implemented. We conduct experiments on several sentiment-labeled datasets and analysis tasks. Experiment results demonstrate that capturing comprehensive contextual information can significantly enhance the classification accuracy compared with the standard recurrent models and the new models can achieve competitive performance compared with the state-of-the-art approaches.

Evolving fuzzy classifiers (EFCs) have achieved immense success in dealing with nonstationary data streams because of their flexible characteristics. Nonetheless, most real-world data streams feature highly uncertain characteristics, which cannot be handled by the type-1 EFC. A novel interval type-2 fuzzy classifier, namely evolving type-2 classifier (eT2Class), is proposed in this paper, which constructs an evolving working principle in the framework of interval type-2 fuzzy system. The eT2Class commences its learning process from scratch with an empty or initially trained rule base, and its fuzzy rules can be automatically grown, pruned, recalled, and merged on the fly referring to summarization power and generalization power of data streams. In addition, the eT2Class is driven by a generalized interval type-2 fuzzy rule, where the premise part is composed of the multivariate Gaussian function with an uncertain nondiagonal covariance matrix, while employing a subset of the nonlinear Chebyshev polynomial as the rule consequents. The efficacy of the eT2Class has been rigorously assessed by numerous real-world and artificial study cases, benchmarked against state-of-the-art classifiers, and validated through various statistical tests. Our numerical results demonstrate that the eT2Class produces more reliable classification rates, while retaining more compact and parsimonious rule base than state-of-the-art EFCs recently published in the literature.

Most real world classification problems involve a high degree of uncertainty, unsolved by a traditional type-1 fuzzy classifier. In this paper, a novel interval type-2 classifier, namely Evolving Type-2 Classifier (eT2Class), is proposed. The eT2Class features a flexible working principle built upon a fully sequential and local working principle. This learning notion allows eT2Class to automatically grow, adapt, prune, recall its knowledge from data streams in the single-pass learning fashion, while employing loosely coupled fuzzy sub-models. In addition, eT2Class introduces a generalized interval type-2 fuzzy neural network architecture, where a multivariate Gaussian function with uncertain non-diagonal covariance matrixes constructs the rule premise, while the rule consequent is crafted by a local non-linear Chebyshev polynomial. The efficacy of eT2Class is numerically validated by numerical studies with four data streams characterizing non-stationary behaviors, where eT2Class demonstrates the most encouraging learning performance in achieving a tradeoff between accuracy and complexity.

Extreme Learning Machine (ELM) is an answer to an increasing demand for a low-cost learning algorithm to handle big data applications. Nevertheless, existing ELMs leave four uncharted problems: complexity, uncertainty, concept drifts, curse of dimensionality. To correct these issues, a novel incremental meta-cognitive ELM, namely Evolving Type-2 Extreme Learning Machine (eT2ELM), is proposed. Et2Elm is built upon the three pillars of meta-cognitive learning, namely what-to-learn, how-to-learn, when-to-learn, where the notion of ELM is implemented in the how-to-learn component. On the other hand, eT2ELM is driven by a generalized interval type-2 Fuzzy Neural Network (FNN) as the cognitive constituent, where the interval type-2 multivariate Gaussian function is used in the hidden layer, whereas the nonlinear Chebyshev function is embedded in the output layer. The efficacy of eT2ELM is proven with four data streams possessing various concept drifts, comparisons with prominent classifiers, and statistical tests, where eT2ELM demonstrates the most encouraging learning performances in terms of accuracy and complexity.

In this paper, a novel evolving fuzzy-rule-based classifier, termed parsimonious classifier (pClass), is proposed. pClass can drive its learning engine from scratch with an empty rule base or initially trained fuzzy models. It adopts an open structure and plug and play concept where automatic knowledge building, rule-based simplification, knowledge recall mechanism, and soft feature reduction can be carried out on the fly with limited expert knowledge and without prior assumptions to underlying data distribution. In this paper, three state-of-the-art classifier architectures engaging multi-input-multi-output, multimodel, and round robin architectures are also critically analyzed. The efficacy of the pClass has been numerically validated by means of real-world and synthetic streaming data, possessing various concept drifts, noisy learning environments, and dynamic class attributes. In addition, comparative studies with prominent algorithms using comprehensive statistical tests have confirmed that the pClass delivers more superior performance in terms of classification rate, number of fuzzy rules, and number of rule-base parameters.

This paper outlines our proposal for a novel metacognitive-based scaffolding classifier, namely recurrent classifier (rClass). rClass is capable of emulating three fundamental pillars of human learning in terms of what-to-learn, how-to-learn, and when-to-learn. The cognitive constituent of rClass is underpinned by a recurrent network based on a generalized version of the Takagi-Sugeno-Kang fuzzy system possessing a local feedback of the rule layer. The main basis of the what-to-learn component relies on the new active learning-based conflict measure. Meanwhile, the when-to-learn learning scenario makes use of the standard sample reserved strategy. The how-to-learn module actualizes the Schema and Scaffolding concepts of cognitive psychology. All learning principles are committed in the single-pass local learning modes and create a plug-and-play learning foundation minimizing additional pre- or post-training phases. The efficacy of rClass has been scrutinized by means of rigorous empirical studies, statistical tests, and benchmarks with state-of-the-art classifiers, which demonstrate the rClass potency in producing reliable classification rates, while retaining low complexity in terms of the rule base burden, computational load, and annotation effort.

A novel meta-cognitive-based scaffolding classifier, namely Generic-Classifier (gClass), is proposed in this paper to handle non-stationary classification problems in the single-pass learning mode. Meta-cognitive learning is a breakthrough in the machine learning where the learning process is not only directed to craft learning strategies to exacerbate the classification rates, i.e., how-to-leam aspect, but also is focused to accommodate the emotional reasoning and commonsense of human being in terms of what-to-leam and when-to-learn facets. The crux of gClass is to synergize the scaffolding learning concept, which constitutes a well-known tutoring theory in the psychological literatures, in the how-to-learn context of meta-cognitive learning, in order to boost the learner's performance in dealing with complex data. A comprehensive empirical studies in time-varying datasets is carried out, where gClass numerical results are benchmarked with other state-of-the-art classifiers. gClass is, generally speaking, capable of delivering the most encouraging numerical results where a trade-off between predictive accuracy and classifier's complexity can be achieved.

Nowadays, there is increasing demand for an integrated system usable to real-time environments under limited computational resources and minimum operator supervision. In contrast, the model is also supposed to actualize high predictive quality in order to confirm the process safety and attractive working framework allowing the user to grasp how the particular task is settled. A holistic concept of a fully data-driven modeling tool namely Generic Evolving Neuro-Fuzzy Inference System (GENEFIS) is proposed in this paper. The major spotlight of GENEFIS is in delivering a sensible tradeoff between high predictive accuracy and parsimonious rule base while reckoning tractable rule semantics. The viability of GENEFIS is numerically validated via a series of experimentations using real world and artificial datasets and is compared against state of the art of the evolving neuro-fuzzy systems (ENFSs), where GENEFIS not only showcases higher predictive accuracies but also lands on more frugal structures than other algorithms.

Most of the dynamics in real-world systems are compiled by shifts and drifts, which are uneasy to be overcome by omnipresent neuro-fuzzy systems. Nonetheless, learning in nonstationary environment entails a system owning high degree of flexibility capable of assembling its rule base autonomously according to the degree of nonlinearity contained in the system. In practice, the rule growing and pruning are carried out merely benefiting from a small snapshot of the complete training data to truncate the computational load and memory demand to the low level. An exposure of a novel algorithm, namely parsimonious network based on fuzzy inference system (PANFIS), is to this end presented herein. PANFIS can commence its learning process from scratch with an empty rule base. The fuzzy rules can be stitched up and expelled by virtue of statistical contributions of the fuzzy rules and injected datum afterward. Identical fuzzy sets may be alluded and blended to be one fuzzy set as a pursuit of a transparent rule base escalating human's interpretability. The learning and modeling performances of the proposed PANFIS are numerically validated using several benchmark problems from real-world or synthetic datasets. The validation includes comparisons with state-of-the-art evolving neuro-fuzzy methods and showcases that our new method can compete and in some cases even outperform these approaches in terms of predictive fidelity and model complexity.

This paper presents a novel evolving fuzzy rule-based classifier stemming from our recently developed algorithm for regression problem termed generic evolving neuro-fuzzy system (GENEFIS). On the one hand, the novel classifier namely GENEFIS-class is composed of two different architectures specifically zero and first orders which are dependent on the type of consequent used. On the other hand, GENEFIS-class refurbishes GENEFIS algorithm as the main learning engine to conform classification requirement. The interesting property of GENEFIS is its fully flexible rule base and its computationally efficient algorithm. GENEFIS can initiate its learning process from scratch with an empty rule base and highly narrow expert knowledge. The fuzzy rules are then flourished based on the novelty of streaming data via their statistical contribution. Conversely, the fuzzy rules, which contribute little during their lifespan, can be pruned by virtue of their contributions up to the end of training process. Meanwhile, the fuzzy rules and fuzzy sets, which are redundant, can be merged to purpose a transparent rule base. Online feature selection process coupled during the training process can be undertaken to cope with possible combinatorial rule explosion drawback. All of these are fruitful to grant significant reduction of rule base load while retaining the classification accuracy which is in line with online real-time necessity. The efficacy of GENEFIS-class was numerically validated exploiting real world and synthetic problems and compared with state-of-the-art algorithms where it generally speaking outperforms other algorithms in terms of classification performance and rule-base complexity.

In this paper, we propose a new variant for incremental, evolving fuzzy systems extraction from data data streams, termed as GEN-FLEXFIS (short for Generalized Flexible Fuzzy Inference Systems). It builds upon the FLEXFIS methodology (published by the authors before) and extends it for generalized Takagi-Sugeno (TS) fuzzy systems, which implement generalized rotated rules in arbitrary position, employing a high-dimensional kernel rather than a connection of one-dimensional components (fuzzy sets) with t-norms. The extension includes the development of the evolving clustering learning engine, termed as eVQ-A, to extract ellipsoidal clusters in arbitrary position. Furthermore, a new merging concept based on a combined adjacency-homogenuity relation between two clusters (rules) is proposed in order to prune unnecessary rules and to keep the complexity of the generalized TS fuzzy systems low. Equipped with a new projection concept for high-dimensional kernels onto one-dimensional fuzzy sets, the new approach also provides equivalent conventional TS fuzzy systems, thus maintaining interpretability when inferring new query samples. GEN-FLEXFIS will be evaluated based on high-dimensional real-world data (streaming) sets in terms of accuracy versus final model complexity, compared with conventional FLEXFIS and other well-known (evolving) fuzzy systems approaches.