Nowadays, cyber-attacks have become a common and persistent issue affecting various human activities in modern societies. Due to the continuously evolving landscape of cyber-attacks and the growing concerns around “black box” models, there has been a strong demand for novel explainable and interpretable intrusion detection systems with online learning abilities. In this paper, a novel soft prototype-based autonomous fuzzy inference system (SPAFIS) is proposed for network intrusion detection . SPAFIS learns from network traffic data streams online on a chunk-by-chunk basis and autonomously identifies a set of meaningful, human-interpretable soft prototypes to build an IF-THEN fuzzy rule base for classification. Thanks to the utilization of soft prototypes, SPAFIS can precisely capture the underlying data structure and local patterns, and perform internal reasoning and decision-making in a human-interpretable manner based on the ensemble properties and mutual distances of data. To maintain a healthy and compact knowledge base , a pruning scheme is further introduced to SPAFIS, allowing itself to periodically examine the learned solution and remove redundant soft prototypes from its knowledge base. Numerical examples on public network intrusion detection datasets demonstrated the efficacy of the proposed SPAFIS in both offline and online application scenarios, outperforming the state-of-the-art alternatives.

Fuzzy systems offer a formal and practically popular methodology for modeling nonlinear problems with inherent uncertainties, entailing strong performance and model interpretability. Particularly, semisupervised boosting is widely recognized as a powerful approach for creating stronger ensemble classification models in the absence of sufficient labeled data without introducing any modification to the employed base classifiers. However, the potential of fuzzy systems in semisupervised boosting has not been systematically explored yet. In this study, a novel semisupervised boosting algorithm devised for zero-order evolving fuzzy systems is proposed. It ensures both the consistence among predictions made by individual base classifiers at successive boosting iterations and the respective levels of confidence toward their predictions throughout the process of sample weight updating and ensemble output generation. In so doing, the base classifiers are empowered to gradually focus more on challenging samples that are otherwise hard to generalize, enabling the development of more precise integrated classification boundaries. Numerical evaluations on a range of benchmark problems are carried out, demonstrating the efficacy of the proposed semisupervised boosting algorithm for constructing ensemble fuzzy classifiers with high accuracy.

High-dimensional data classification is widely considered as a challenging task in machine learning due to the so-called “curse of dimensionality.” In this article, a novel multilayer jointly evolving and compressing fuzzy neural network (MECFNN) is proposed to learn highly compact multilevel latent representations from high-dimensional data. As a metalevel stacking ensemble system, each layer of MECFNN is based on a single jointly evolving and compressing neural fuzzy inference system (ECNFIS) that self-organizes a set of human-interpretable fuzzy rules from input data in a samplewise manner to perform approximate reasoning. ECNFISs associate a unique compressive projection matrix to each individual fuzzy rule to compress the consequent part into a tighter form, removing redundant information while boosting the diversity within the stacking ensemble. The compressive projection matrices of the cascading ECNFISs are self-updating to minimize the prediction errors via error backpropagation together with the consequent parameters, empowering MECFNN to learn more meaningful, discriminative representations from data at multiple levels of abstraction. An adaptive activation control scheme is further introduced in MECFNN to dynamically exclude less activated fuzzy rules, effectively reducing the computational complexity and fostering generalization. Numerical examples on popular high-dimensional classification problems demonstrate the efficacy of MECFNN.

This paper proposes a dynamic evolving fuzzy system (DEFS) for streaming data prediction. DEFS utilises the enhanced data potential and prediction errors of individual local models as the main criteria for fuzzy rule generation. A vital feature of the proposed system is its novel rule merging scheme that can self-adjust its tolerance towards the degree of similarity between two similar fuzzy rules according to the size of the rule base. To better handle the shifts and drifts in the data patterns, a novel rule quality measure based on both the utility values and the prediction accuracy of individual fuzzy rules is further introduced to help DEFS identify these less activated fuzzy rules with poorer descriptive capabilities and, thereby, maintaining a healthier fuzzy rule base by removing these stale rules. Very importantly, the thresholds used by DEFS are self-adaptive towards the input data. The adaptive thresholds can help DEFS to precisely capture the underlying structure and dynamically changing patterns of streaming data, enabling the system performing accurate approximation reasoning. Numerical examples based on several popular benchmark problems show the superior performance of DEFS over the state-of-the-art evolving fuzzy systems. The prediction performance of the proposed method is at least 2.88% better than the best-performing comparative EFSs on each individual regression benchmark problem considered in this study, and the average performance improvement across all the numerical experiments is approximately 30%.

Tropical cyclones (TCs), with an intensive wind pump impact, induce sea surface temperature cooling (SSTC) on the upper ocean. SSTC is a pronounced indicator to reveal TC evolution and oceanic conditions. However, there are few effective methods for accurately approximating the amplitude of the spatial structure of TC-induced SSTC. This study proposes a novel explainable machine learning framework to model and interpret the amplitude of the spatial structure of SSTC over the northwest Pacific (NWP). In particular, 12 predictors related to TC characteristics and pre-storm ocean states are considered as inputs. A composite analysis technique is used to characterize the amplitude of the spatial structure of SSTC across the TC track. Extreme gradient boosting (XGBoost) is utilized to predict the amplitude of SSTC from the 12 predictors. To better interpret the ocean-atmosphere interaction, a SHapely Additive explanations (SHAP) method is further employed to identify the contributions of predictors in determining the amplitude of the TC-induced SSTC, bringing the attribute-oriented explainability to the proposed method. Results showed that the proposed method could accurately predict the amplitude of the spatial structure of SSTC for different TC intensity groups and outperforms a numerical model. The proposed method also serves as an effective tool for reconstructing composite maps of both interannual and seasonal evolutions of SSTC spatial structure. The study offers insight into applying machine learning to model and interpret the responses of oceanic conditions triggered by extreme weather conditions (e.g., TCs).

Ensemble learning is a widely used methodology to build powerful predictors from multiple individual weaker ones. However, the vast majority of ensemble learning models are designed for offline application scenarios, the use of evolving fuzzy systems in ensemble learning for online learning from data streams has not been sufficiently explored, yet. In this paper, a novel self-adaptive fuzzy learning ensemble system is introduced for data stream prediction. The proposed ensemble system employs the very sparse random projection technique to compress the consequent parts of the learned fuzzy rules by individual base models to a more compressed form, thereby reducing redundant information and improving computational efficiency. To improve the overall prediction performance, a dynamical base model pruning scheme is introduced to the proposed ensemble system together with a novel inferencing scheme, such that less accurate base models will be removed from the ensemble structure at each learning cycle automatically and only these more accurate ones will be involved in joint decision-making. Numerical examples based on a wide range of benchmark datasets demonstrate the stronger prediction performance of the proposed ensemble system over the state-of-the-art alternatives.

It is widely recognized that learning systems have to go deeper to exchange for more powerful representational learning capabilities in order to precisely approximate nonlinear complex problems. However, the best-known computational intelligence approaches with such characteristics, namely, deep neural networks, are often criticized for lacking transparency. In this article, a novel multilayer evolving fuzzy neural network (MEFNN) with a transparent system structure is proposed. The proposed MEFNN is a metalevel stacking ensemble learning system composed of multiple cascading evolving neuro-fuzzy inference systems (ENFISs), processing input data layer-by-layer to automatically learn multilevel nonlinear distributed representations from data. Each ENFIS is an evolving fuzzy system capable of learning from new data sample by sample to self-organize a set of human-interpretable IF– THEN fuzzy rules that facilitate approximate reasoning. Adopting ENFIS as its ensemble component, the multilayer system structure of the MEFNN is flexible and transparent, and its internal reasoning and decision-making mechanism can be explained and interpreted to/by humans. To facilitate information exchange between different layers and attain stronger representation learning capability, the MEFNN utilizes error backpropagation to self-update the consequent parameters of the IF–THEN rules of each ensemble component based on the approximation error propagated backward. To enhance the capability of the MEFNN to handle complex problems, a nonlinear activation function is introduced to modeling the consequent parts of the IF–THEN rules of ENFISs, thereby empowering both the representation and the reflection of nonlinearity in the resulting fuzzy outputs. Numerical examples on a wide variety of challenging (benchmark and real-world) classification and regression problems demonstrate the superior practical performance of the MEFNN, revealing the effectiveness and validity of the proposed approach.

Evolving intelligent systems (EISs), particularly, the zero-order ones have demonstrated strong performance on many real-world problems concerning data stream classification while offering high model transparency and interpretability thanks to their prototype-based nature. Zero-order EISs typically learn prototypes by clustering streaming data online in a “one pass” manner for greater computation efficiency. However, such identified prototypes often lack optimality, resulting in less precise classification boundaries, thereby hindering the potential classification performance of the systems. To address this issue, a commonly adopted strategy is to minimize the training error of the models on historical training data or alternatively to iteratively minimize the intracluster variance of the clusters obtained via online data partitioning. This recognizes the fact that the ultimate classification performance of zero-order EISs is driven by the positions of prototypes in the data space. Yet, simply minimizing the training error may potentially lead to overfitting while minimizing the intracluster variance does not necessarily ensure the optimized prototype-based models to attain improved classification outcomes. To achieve better classification performance while avoiding overfitting for zero-order EISs, this article presents a novel multiobjective optimization approach, enabling EISs to obtain optimal prototypes via involving these two disparate but complementary strategies simultaneously. Five decision-making schemes are introduced for selecting a suitable solution to deploy from the final nondominated set of the resulting optimized models. Systematic experimental studies are carried out to demonstrate the effectiveness of the proposed optimization approach in improving the classification performance of zero-order EISs.

Large-scale (large-area), fine spatial resolution satellite sensor images are valuable data sources for Earth observation while not yet fully exploited by research communities for practical applications. Often, such images exhibit highly complex geometrical structures and spatial patterns, and distinctive characteristics of multiple land-use categories may appear at the same region. Autonomous information extraction from these images is essential in the field of pattern recognition within remote sensing, but this task is extremely challenging due to the spectral and spatial complexity captured in satellite sensor imagery. In this research, a semi-supervised deep rule-based approach for satellite sensor image analysis (SeRBIA) is proposed, where large-scale satellite sensor images are analysed autonomously and classified into detailed land-use categories. Using an ensemble feature descriptor derived from pre-trained AlexNet and VGG-VD-16 models, SeRBIA is capable of learning continuously from both labelled and unlabelled images through self-adaptation without human involvement or intervention. Extensive numerical experiments were conducted on both benchmark datasets and real-world satellite sensor images to comprehensively test the validity and effectiveness of the proposed method. The novel information mining technique developed here can be applied to analyse large-scale satellite sensor images with high accuracy and interpretability, across a wide range of real-world applications.

Adaptive boosting (AdaBoost) is a widely used technique to construct a stronger ensemble classifier by combining a set of weaker ones. Zero-order fuzzy inference systems (FISs) are very powerful prototype-based predictive models for classification, offering both great prediction precision and high user interpretability. However, the use of zero-order FISs as base classifiers in AdaBoost has not been explored yet. To bridge the gap, in this article, a novel multiclass fuzzily weighted AdaBoost (FWAdaBoost)-based ensemble system with a self-organizing fuzzy inference system (SOFIS) as the ensemble component is proposed. To better incorporate the SOFIS, FWAdaBoost utilizes the confidence scores produced by the SOFIS in both sample weight updating and ensemble output generation, resulting in more accurate classification boundaries and greater prediction precision. Numerical examples on a wide range of benchmark classification problems demonstrate the efficacy of the proposed approach.

Evolving fuzzy systems (EFSs) are widely known as a powerful tool for streaming data prediction. In this article, a novel zero-order EFS with a unique belief structure is proposed for data stream classification. Thanks to this new belief structure, the proposed model can handle the interclass overlaps in a natural way and better capture the underlying multimodel structure of data streams in the form of prototypes. Utilizing data-driven soft thresholds, the proposed model self-organizes a set of prototype-basedif–thenfuzzy belief rules from data streams for classification, and its learning outcomes are practically meaningful. With no requirement of prior knowledge in the problem domain, the proposed model is capable of self-determining the appropriate level of granularity for rule-based construction, while enabling users to specify their preferences on the degree of fineness of its knowledge base. Numerical examples demonstrate the superior performance of the proposed model on a wide range of stationary and nonstationary classification benchmark problems.

In this article, a new data-driven autonomous fuzzy clustering (AFC) algorithm is proposed for static data clustering. Employing a Gaussian-type membership function, AFC first uses all the data samples as microcluster medoids to assign memberships to each other and obtains the membership matrix. Based on this, AFC chooses these data samples that represent local models of data distribution as cluster medoids for initial partition. It then continues to optimize the cluster medoids iteratively to obtain a locally optimal partition as the algorithm output. Moreover, an online extension is introduced to AFC enabling the algorithm to cluster streaming data chunk-by-chunk in a “one pass” manner. Numerical examples based on a variety of benchmark problems demonstrate the efficacy of the AFC algorithm in both offline and online application scenarios, proving the effectiveness and validity of the proposed concept and general principles.

The antecedent and consequent parts of a first-order evolving intelligent system (EIS) determine the validity of the learning results and overall system performance. Nonetheless, the state-of-the-art techniques mostly stress on the novelty from the system identification point of view but pay less attention to the optimality of the learned parameters. Using the recently introduced autonomous learning multiple model (ALMMo) system as the implementation basis, this article introduces a particle swarm-based approach for the EIS optimization. The proposed approach is able to simultaneously optimize the antecedent and consequent parameters of ALMMo and effectively enhance the system performance by iteratively searching for optimal solutions in the problem spaces. In addition, the proposed optimization approach does not adversely influence the "one pass" learning ability of ALMMo. Once the optimization process is complete, ALMMo can continue to learn from new data to incorporate unseen data patterns recursively without full retraining. The experimental studies with a number of real-world benchmark problems validate the proposed concept and general principles. It is also verified that the proposed optimization approach can be applied to other types of EISs with similar operating mechanisms.

In order to tackle high dimensional, complex problems, learning models have to go deeper. In this article, a novel multilayer ensemble learning model with first-order evolving fuzzy systems as its building blocks is introduced. The proposed approach can effectively learn from streaming data on a sample-by-sample basis and self-organizes its multilayered system structure and meta-parameters in a feedforward, noniterative manner. Benefiting from its multilayered distributed representation learning ability, the ensemble system not only demonstrates the state-of-the-art performance on various problems, but also offers high level of system transparency and explainability. Theoretical justifications and experimental investigation show the validity and effectiveness of the proposed concept and general principles.

A novel self-organizing fuzzy proportional-integral-derivative (SOF-PID) control system is proposed in this paper. The proposed system consists of a pair of control and reference models, both of which are implemented by a first-order autonomous learning multiple model (ALMMo) neuro-fuzzy system. The SOF-PID controller self-organizes and self-updates the structures and meta-parameters of both the control and reference models during the control process "on the fly". This gives the SOF-PID control system the capability of quickly adapting to entirely new operating environments without a full re-training. Moreover, the SOF-PID control system is free from user- and problem-specific parameters and is entirely data-driven. Simulations and real-world experiments with mobile robots demonstrate the effectiveness and validity of the proposed SOF-PID control system.

Traditionally, in supervised machine learning, (a significant) part of the available data (usually 50%-80%) is used for training and the rest—for validation. In many problems, however, the data are highly imbalanced in regard to different classes or does not have good coverage of the feasible data space which, in turn, creates problems in validation and usage phase. In this paper, we propose a technique for synthesizing feasible and likely data to help balance the classes as well as to boost the performance in terms of confusion matrix as well as overall. The idea, in a nutshell, is to synthesize data samples in close vicinity to the actual data samples specifically for the less represented (minority) classes. This has also implications to the so-called fairness of machine learning. In this paper, we propose a specific method for synthesizing data in a way to balance the classes and boost the performance, especially of the minority classes. It is generic and can be applied to different base algorithms, for example, support vector machines, k-nearest neighbour classifiers deep neural, rule-based classifiers, decision trees, and so forth. The results demonstrated that (a) a significantly more balanced (and fair) classification results can be achieved and (b) that the overall performance as well as the performance per class measured by confusion matrix can be boosted. In addition, this approach can be very valuable for the cases when the number of actual available labelled data is small which itself is one of the problems of the contemporary machine learning.

In this paper, a new deep rule-based approach using high-level ensemble feature descriptor is proposed for aerial scene classification. By creating an ensemble of three pre-trained deep convolutional neural networks for feature extraction, the proposed approach is able to extract more discriminative representations from the local regions of aerial images. With a set of massively parallel IF...THEN rules built upon the prototypes identified through a self-organizing, nonparametric, transparent and highly human-interpretable learning process, the proposed approach is able to produce the state-of-the-art classification results on the unlabeled images outperforming the alternatives. Numerical examples on benchmark datasets demonstrate the strong performance of the proposed approach.

Satellite scene images contain multiple sub-regions of different land use categories; however, traditional approaches usually classify them into a particular category only. In this paper, a new approach is proposed for automatically analyzing the semantic content of sub-regions of satellite images. At the core of the proposed approach is the recently introduced deep rule-based image classification method. The proposed approach includes a self-organizing set of transparent zero order fuzzy IF-THEN rules with human-interpretable prototypes identified from the training images and a pre-trained deep convolutional neural network as the feature descriptor. It requires a very short, nonparametric, highly parallelizable training process and can perform a highly accurate analysis on the semantic features of local areas of the image with the generated IF-THEN rules in a fully automatic way. Examples based on benchmark datasets demonstrate the validity and effectiveness of the proposed approach.

In this paper, we introduce a new form of describing fuzzy sets (FSs) and a new form of fuzzy rule-based (FRB) systems, namely, empirical fuzzy sets (εFSs) and empirical fuzzy rule-based (εFRB) systems. Traditionally, the membership functions (MFs), which are the key mathematical representation of FSs, are designed subjectively or extracted from the data by clustering projections or defined subjectively. εFSs, on the contrary, are described by the empirically derived membership functions (εMFs). The new proposal made in this paper is based on the recently introduced Empirical Data Analytics (EDA) computational framework and is closely linked with the density of the data. This allows to keep and improve the link between the objective data and the subjective labels, linguistic terms, and classes definition. Furthermore, εFSs can deal with heterogeneous data combining categorical with continuous and/or discrete data in a natural way. εFRB systems can be extracted from data including data streams and can have dynamically evolving structure. However, they can also be used as a tool to represent expert knowledge. The main difference from the traditional FSs and FRB systems is that the expert does not need to define the MF per variable; instead, possibly multimodal, densities will be extracted automatically from the data and used as εMFs in a vector form for all numerical variables. This is done in a seamless way whereby the human involvement is only required to label the classes and linguistic terms. Moreover, even this intervention is optional. Thus, the proposed new approach to define and design the FSs and FRB systems puts the human “in the driving seat.” Instead of asking experts to define features and MFs correspondingly, to parameterize them, to define algorithm parameters, to choose types of MFs, or to label each individual item, it only requires (optionally) to select prototypes from data and (again, optionally) to label them. Numerical examples as well as a naïve empirical fuzzy (εF) classifier are presented with an illustrative purpose. Due to the very fundamental nature of the proposal, it can have a very wide area of applications resulting in a series of new algorithms such as εF classifiers, εF predictors, εF controllers, and so on. This is left for the future research.

In this letter, we propose a new approach for remote sensing scene classification by creating an ensemble of the recently introduced massively parallel deep (fuzzy) rule-based (DRB) classifiers trained with different levels of spatial information separately. Each DRB classifier consists of a massively parallel set of human-interpretable, transparent zero-order fuzzy IF...THEN... rules with a prototype-based nature. The DRB classifier can self-organize “from scratch” and self-evolve its structure. By employing the pretrained deep convolution neural network as the feature descriptor, the proposed DRB ensemble is able to exhibit human-level performance through a transparent and parallelizable training process. Numerical examples using benchmark data set demonstrate the superior accuracy of the proposed approach together with human-interpretable fuzzy rules autonomously generated by the DRB classifier.

Based on a critical analysis of data analytics and its foundations, we propose a functional approach to estimate data ensemble properties, which is based entirely on the empirical observations of discrete data samples and the relative proximity of these points in the data space and hence named empirical data analysis (EDA). The ensemble functions include the nonparametric square centrality (a measure of closeness used in graph theory) and typicality (an empirically derived quantity which resembles probability). A distinctive feature of the proposed new functional approach to data analysis is that it does not assume randomness or determinism of the empirically observed data, nor independence. The typicality is derived from the discrete data directly in contrast to the traditional approach, where a continuous probability density function is assumed a priori. The typicality is expressed in a closed analytical form that can be calculated recursively and, thus, is computationally very efficient. The proposed nonparametric estimators of the ensemble properties of the data can also be interpreted as a discrete form of the information potential (known from the information theoretic learning theory as well as the Parzen windows). Therefore, EDA is very suitable for the current move to a data-rich environment, where the understanding of the underlying phenomena behind the available vast amounts of data is often not clear. We also present an extension of EDA for inference. The areas of applications of the new methodology of the EDA are wide because it concerns the very foundation of data analysis. Preliminary tests show its good performance in comparison to traditional techniques.

In this paper, an approach to autonomous learning of a multimodel system from streaming data, named ALMMo, is proposed. The proposed approach is generic and can easily be applied also to probabilistic or other types of local models forming multimodel systems. It is fully data driven and its structure is decided by the nonparametric data clouds extracted from the empirically observed data without making any prior assumptions concerning data distribution and other data properties. All metaparameters of the proposed system are obtained directly from the data and can be updated recursively, which improves memory and calculation efficiencies of the proposed algorithm. The structural evolution mechanism and online data cloud quality monitoring mechanism of the ALMMo system largely enhance the ability of handling shifts and/or drifts in the streaming data pattern. Numerical examples of the use of ALMMo system for streaming data analytics, classification, and prediction are presented as a proof of the proposed concept.

Evolving fuzzy systems (EFSs) are now well developed and widely used, thanks to their ability to self-adapt both their structures and parameters online. Since the concept was first introduced two decades ago, many different types of EFSs have been successfully implemented. However, there are only very few works considering the stability of the EFSs, and these studies were limited to certain types of membership functions with specifically predefined parameters, which largely increases the complexity of the learning process. At the same time, stability analysis is of paramount importance for control applications and provides the theoretical guarantees for the convergence of the learning algorithms. In this paper, we introduce the stability proof of a class of EFSs based on data clouds, which are grounded at the AnYa type fuzzy systems and the recently introduced empirical data analytics (EDA) methodological framework. By employing data clouds, the class of EFSs of AnYa type considered in this paper avoids the traditional way of defining membership functions for each input variable in an explicit manner and its learning process is entirely data driven. The stability of the considered EFS of AnYa type is proven through the Lyapunov theory, and the proof of stability shows that the average identification error converges to a small neighborhood of zero. Although, the stability proof presented in this paper is specially elaborated for the considered EFS, it is also applicable to general EFSs. The proposed method is illustrated with Box-Jenkins gas furnace problem, one nonlinear system identification problem, Mackey-Glass time series prediction problem, eight real-world benchmark regression problems as well as a high-frequency trading prediction problem. Compared with other EFSs, the numerical examples show that the considered EFS in this paper provides guaranteed stability as well as a better approximation accuracy.

In this paper, a new type of feedforward non-parametric deep learning network with automatic feature extraction is proposed. The proposed network is based on human-understandable local aggregations extracted directly from the images. There is no need for any feature selection and parameter tuning. The proposed network involves nonlinear transformation, segmentation operations to select the most distinctive features from the training images and builds RBF neurons based on them to perform classification with no weights to train. The design of the proposed network is very efficient (computation and time wise) and produces highly accurate classification results. Moreover, the training process is parallelizable, and the time consumption can be further reduced with more processors involved. Numerical examples demonstrate the high performance and very short training process of the proposed network for different applications.

In this paper, a fast, transparent, self-evolving, deep learning fuzzy rule-based (DLFRB) image classifier is proposed. This new classifier is a cascade of the recently introduced DLFRB classifier called MICE and an auxiliary SVM. The DLFRB classifier serves as the main engine and can identify a number of human interpretable fuzzy rules through a very short, transparent, highly parallelizable training process. The SVM based auxiliary plays the role of a conflict resolver when the DLFRB classifier produces two highly confident labels for a single image. Only the fundamental image transformation techniques (rotation, scaling and segmentation) and feature descriptors (GIST and HOG) are used for pre-processing and feature extraction, but the proposed approach significantly outperforms the state-of-art methods in terms of both time and precision. Numerical experiments based on a handwritten digits recognition problem are used to demonstrate the highly accurate and repeatable performance of the proposed approach.

In this paper, we propose an approach to data analysis, which is based entirely on the empirical observations of discrete data samples and the relative proximity of these points in the data space. At the core of the proposed new approach is the typicality—an empirically derived quantity that resembles probability. This nonparametric measure is a normalized form of the square centrality (centrality is a measure of closeness used in graph theory). It is also closely linked to the cumulative proximity and eccentricity (a measure of the tail of the distributions that is very useful for anomaly detection and analysis of extreme values). In this paper, we introduce and study two types of typicality, namely its local and global versions. The local typicality resembles the well-known probability density function (pdf), probability mass function, and fuzzy set membership but differs from all of them. The global typicality, on the other hand, resembles well-known histograms but also differs from them. A distinctive feature of the proposed new approach, empirical data analysis (EDA), is that it is not limited by restrictive impractical prior assumptions about the data generation model as the traditional probability theory and statistical learning approaches are. Moreover, it does not require an explicit and binary assumption of either randomness or determinism of the empirically observed data, their independence, or even their number (it can be as low as a couple of data samples). The typicality is considered as a fundamental quantity in the pattern analysis, which is derived directly from data and is stated in a discrete form in contrast to the traditional approach where a continuous pdf is assumed a priori and estimated from data afterward. The typicality introduced in this paper is free from the paradoxes of the pdf. Typicality is objectivist while the fuzzy sets and the belief-based branch of the probability theory are subjectivist. The local typicality is expressed in a closed analytical form and can be calculated recursively, thus, computationally very efficiently. The other nonparametric ensemble properties of the data introduced and studied in this paper, namely, the square centrality, cumulative proximity, and eccentricity, can also be updated recursively for various types of distance metrics. Finally, a new type of classifier called naïve typicality-based EDA class is introduced, which is based on the newly introduced global typicality. This is only one of the wide range of possible applications of EDA including but not limited for anomaly detection, clustering, classification, control, prediction, control, rare events analysis, etc., which will be the subject of further research.

It is well known that clustering is an unsupervised machine learning technique. However, most of the clustering methods need setting several parameters such as number of clusters, shape of clusters, or other user- or problem-specific parameters and thresholds. In this paper, we propose a new clustering approach which is fully autonomous, in the sense that it does not require parameters to be pre-defined. This approach is based on data density automatically derived from their mutual distribution in the data space. It is called ADD clustering (Autonomous Data Density based clustering). It is entirely based on the experimentally observable data and is free from restrictive prior assumptions. This new method exhibits highly accurate clustering performance. Its performance is compared on benchmarked data sets with other competitive alternative approaches. Experimental results demonstrate that ADD clustering significantly outperforms other clustering methods yet does not require restrictive user- or problem-specific parameters or assumptions. The new clustering method is a solid basis for further applications in the field of data analytics.

In this paper, a novel empirical data analysis approach (abbreviated as EDA) is introduced which is entirely data-driven and free from restricting assumptions and pre-defined problem- or user-specific parameters and thresholds. It is well known that the traditional probability theory is restricted by strong prior assumptions which are often impractical and do not hold in real problems. Machine learning methods, on the other hand, are closer to the real problems but they usually rely on problem- or user-specific parameters or thresholds making it rather art than science. In this paper we introduce a theoretically sound yet practically unrestricted and widely applicable approach that is based on the density in the data space. Since the data may have exactly the same value multiple times we distinguish between the data points and unique locations in the data space. The number of data points, k is larger or equal to the number of unique locations, l and at least one data point occupies each unique location. The number of different data points that have exactly the same location in the data space (equal value), f can be seen as frequency. Through the combination of the spatial density and the frequency of occurrence of discrete data points, a new concept called multimodal typicality, τMMis proposed in this paper. It offers a closed analytical form that represents ensemble properties derived entirely from the empirical observations of data. Moreover, it is very close (yet different) from the histograms, from the probability density function (pdf) as well as from fuzzy set membership functions. Remarkably, there is no need to perform complicated pre-processing like clustering to get the multimodal representation. Moreover, the closed form for the case of Euclidean, Mahalanobis type of distance as well as some other forms (e.g. cosine-based dissimilarity) can be expressed recursively making it applicable to data streams and online algorithms. Inference/estimation of the typicality of data points that were not present in the data so far can be made. This new concept allows to rethink the very foundations of statistical and machine learning as well as to develop a series of anomaly detection, clustering, classification, prediction, control and other algorithms.

In this paper, a novel autonomous data-driven clustering approach, called AD_clustering, is presented for live data streams processing. This newly proposed algorithm is a fully unsupervised approach and entirely based on the data samples and their ensemble properties, in the sense that there is no need for user-predefined or problem-specific assumptions and parameters, which is a problem most of the current clustering approaches suffer from. Moreover, the proposed approach automatically evolves its structure according to the experimentally observable streaming data and is able to recursively update its self-defined parameters using only the current data sample; meanwhile, it discards all the previously processed data samples. Experimental results based on benchmark datasets exhibit the higher performance of the proposed fully autonomous approach compared with the comparative approaches requiring user- and problem-specific parameters to be predefined. This new clustering algorithm is a promising tool for further applications in the field of real-time streaming data analytics.

An estimation method of pseudo‐random (PN) codes in the periodic long code direct sequence spread spectrum signals using a pair of spreading code and scrambling code [i.e. long scrambling code direct sequence spread spectrum (LSC‐DSSS)] is investigated in this study. Via the investigation of properties of triple correlation function (TCF) of m ‐sequences, the existence of common peaks in the TCFs of different m ‐sequences is proved, and the corresponding relationship between common peaks and primitive polynomials is further investigated. Four theorems are proposed as supplements of triple correlation theory and a novel estimation algorithm of the PN codes in LSC‐DSSS signals is put forward on the basis of the theorems. With certain carrier frequency and chip rate of spreading code, this algorithm first eliminates the influence of information codes through delay‐and‐multiply operation. Then the TCF of signal is calculated, and the two PN codes in signal are successfully estimated finally by searching and using the common peak coordinates in the TCF. Simulation results show that the proposed algorithm exhibits excellent performance in estimating PN codes in LSC‐DSSS signals.