Several models have been developed to address learning in the presence of label noise, with independent and identically distributed label noise models being the most common. However, these models are unrealistic and perhaps overly simplistic for many real-world applications. Instance-dependent noise models are more realistic but limit themselves under the assumption of a single generative process for the label noise across the entire dataset. To address such limitations, we propose a more grounded and practical model of label noise that generalizes instance-dependent noise onto a multiple-labeler paradigm, which is primarily sourced via crowd-sourcing methods. Within our labeler-dependent noise model, label noise is manifested in two modalities: the natural error of good-faith labelers and that of adversarial labels provided by malicious agents. We also describe an adversarial attack vector that more accurately reflects label noise found in real-world scenarios and demonstrate that - under the proposed multimodal noisy label model - even DivideMix, a modern approach for learning from noisy labels, is degraded significantly by this more realistic adversarial attack vector. As such, we propose a multi-stage, labeler-aware, model-agnostic framework that reliably filters both adversarial and good-faith noisy labels by leveraging the knowledge of which data partitions are labeled by which labeler. We show that the proposed framework remains robust and maintains high model accuracy even in the presence of increasingly growing amounts of adversarial data.

MOTIVATION: This study examines the query performance of the NBC++ (Incremental Naive Bayes Classifier) program for variations in canonicality, k-mer size, databases, and input sample data size. We demonstrate that both NBC++ and Kraken2 are influenced by database depth, with macro measures improving as depth increases. However, fully capturing the diversity of life, especially viruses, remains a challenge. RESULTS: NBC++ can competitively profile the superkingdom content of metagenomic samples using a small training database. NBC++ spends less time training and can use a fraction of the memory than Kraken2 but at the cost of long querying time. Major NBC++ enhancements include accommodating canonical k-mer storage (leading to significant storage savings) and adaptable and optimized memory allocation that accelerates query analysis and enables the software to be run on nearly any system. Additionally, the output now includes log-likelihood values for each training genome, providing users with valuable confidence information. AVAILABILITY AND IMPLEMENTATION: Source code and Dockerfile are available at http://github.com/EESI/Naive_Bayes.

Abstract Anomaly detection is a method employed to identify data points or patterns that significantly deviate from expected or normal behaviour within a dataset. This approach aims to detect observations regarded as unusual, erroneous, anomalous, rare, or potentially indicative of fraudulent or malicious activity. Open‐set recognition, also referred to as open‐set identification or open‐set classification, is a pattern recognition task that extends traditional classification by addressing the presence of unknown or novel classes during the testing phase. This approach highlights a strong connection between anomaly detection and open‐set recognition, as both seek to identify samples originating from unknown classes or distributions. Open‐set recognition methods frequently involve modelling both known and unknown classes during training, allowing for the capture of the distribution of known classes while explicitly addressing the space of unknown classes. Techniques in open‐set recognition may include outlier detection, density estimation, or configuring decision boundaries to better differentiate between known and unknown classes. This special issue calls for original contributions introducing novel datasets, innovative architectures, and advanced training methods for tasks related to visual anomaly detection and open‐set recognition.

As larger and more comprehensive datasets become standard in contemporary machine learning, it becomes increasingly more difficult to obtain reliable, trustworthy label information with which to train sophisticated models. To address this problem, crowdsourcing has emerged as a popular, inexpensive, and efficient data mining solution for performing distributed label collection. However, crowdsourced annotations are inherently untrustworthy, as the labels are provided by anonymous volunteers who may have varying, unreliable expertise. Worse yet, some participants on commonly used platforms such as Amazon Mechanical Turk may be adversarial, and provide intentionally incorrect label information without the end user's knowledge. We discuss three conventional models of the label generation process, describing their parameterizations and the model-based approaches used to solve them. We then propose OpinionRank, a model-free, interpretable, graph-based spectral algorithm for integrating crowdsourced annotations into reliable labels for performing supervised or semi-supervised learning. Our experiments show that OpinionRank performs favorably when compared against more highly parameterized algorithms. We also show that OpinionRank is scalable to very large datasets and numbers of label sources, and requires considerably fewer computational resources than previous approaches.

Continual (or “incremental”) learning approaches are employed when additional knowledge or tasks need to be learned from subsequent batches or from streaming data. However these approaches are typically adversary agnostic, i.e., they do not consider the possibility of a malicious attack. In our prior work, we explored the vulnerabilities of Elastic Weight Consolidation (EWC) to the perceptible misinformation. We now explore the vulnerabilities of other regularization-based as well as generative replay-based continual learning algorithms, and also extend the attack to imperceptible misinformation. We show that an intelligent adversary can take advantage of a continual learning algorithm's capabilities of retaining existing knowledge over time, and force it to learn and retain deliberately introduced misinformation. To demonstrate this vulnerability, we inject backdoor attack samples into the training data. These attack samples constitute the misinformation, allowing the attacker to capture control of the model at test time. We evaluate the extent of this vulnerability on both rotated and split benchmark variants of the MNIST dataset under two important domain and class incremental learning scenarios. We show that the adversary can create a “false memory” about any task by inserting carefully-designed backdoor samples to the test instances of that task thereby controlling the amount of forgetting of any task of its choosing. Perhaps most importantly, we show this vulnerability to be very acute and damaging: the model memory can be easily compromised with the addition of backdoor samples into as little as 1% of the training data, even when the misinformation is imperceptible to human eye.

Artificial neural networks are well-known to be susceptible to catastrophic forgetting when continually learning from sequences of tasks. Various continual (or "incremental") learning approaches have been proposed to avoid catastrophic forgetting, but they are typically adversary agnostic, i.e., they do not consider the possibility of a malicious attack. In this effort, we explore the vulnerability of Elastic Weight Consolidation (EWC), a popular continual learning algorithm for avoiding catastrophic forgetting. We show that an intelligent adversary can take advantage of EWC's continual learning capabilities to cause gradual and deliberate forgetting by introducing small amounts of misinformation to the model during training. We demonstrate such an adversary's ability to assume control of the model via injection of backdoor attack samples on both permuted and split benchmark variants of the MNIST dataset. Importantly, once the model has learned the adversarial misinformation, the adversary can then control the amount of forgetting of any task. Equivalently, the malicious actor can create a "false memory" about any task by inserting carefully-designed backdoor samples to any fraction of the test instances of that task. Perhaps most damaging, we show this vulnerability to be very acute; the model memory can be easily compromised with the addition of backdoor samples into as little as 1% of the training data of even a single task.

Adversarial machine learning has recently risen to prominence due to increased concerns over the vulnerability of machine learning algorithms to malicious attacks. While the impact of malicious poisoning attacks on some popular algorithms, such as deep neural networks, has been well researched, the vulnerability of other approaches has not yet been properly established. In this effort, we explore the vulnerability of unconstrained least squares importance fitting (uLSIF), an algorithm used for computing the importance ratio for covariate shift domain adaptation problems. The uLSIF algorithm is an accurate and efficient technique to compute the importance ratio; however, we show that the approach is susceptible to a poisoning attack, where an intelligent adversary – having full or partial access to the training data – can inject well crafted malicious samples into the training data, resulting in an incorrect estimation of the importance values. Through strategically designed synthetic as well as real world datasets, we demonstrate that importance ratio estimation through uLSIF algorithm can be easily compromised with the insertion of even modest number of attack points into the training data. We also show that incorrect estimation of importance values can then cripple the performance of a subsequent covariate shift adaptation.

As the prevalence and everyday use of machine learning algorithms, along with our reliance on these algorithms grow dramatically, so do the efforts to attack and undermine these algorithms with malicious intent, resulting in a growing interest in adversarial machine learning. A number of approaches have been developed that can render a machine learning algorithm ineffective through poisoning or other types of attacks. Most attack algorithms typically use sophisticated optimization approaches, whose objective function is designed to cause maximum damage to accuracy or performance of the algorithm with respect to some task. In this effort, we show that while such an objective function is indeed brutally effective in causing maximum damage on an embedded feature selection task, it often results in an attack mechanism that can be easily detected with an embarrassingly simple novelty or outlier detection algorithm. We then propose an equally simple yet elegant solution by adding a regularization term to the attacker's objective function that penalizes outlying attack points.

A common, yet rarely addressed, real-world problem in computational intelligence applications is learning from non-stationary streaming data, where the underlying distribution of the data changes over time. This problem, also referred to as concept drift, is made even more challenging if, after initially receiving a small set of labeled data, the streaming data only consists of unlabeled data, requiring the learner to adapt to changing underlying distribution without the benefit of labeled data. This particular scenario is typically referred to as learning in initially labeled nonstationary environment, or as extreme verification latency (EVL), pointing to the fact that the label verification of the test data is indefinitely delayed. In our prior work, we have noted that current EVL algorithms - including the algorithm COMPOSE that we have developed - are largely unable to track changing distributions if the data drawn from those distributions are even mildly imbalanced. In this work, we integrate COMPOSE with 13 different resampling based modified algorithms, and compare accuracy, F1 score, and execution time. The results differed from what we originally expected and provided unique insight on how to choose a data rebalancing approach for different types of drift.

Feature subset selection can be used to sieve through large volumes of data and discover the most informative subset of variables for a particular learning problem. Yet, due to memory and other resource constraints (e.g., CPU availability), many of the state-of-the-art feature subset selection methods cannot be extended to high dimensional data, or data sets with an extremely large volume of instances. In this brief, we extend online feature selection (OFS), a recently introduced approach that uses partial feature information, by developing an ensemble of online linear models to make predictions. The OFS approach employs a linear model as the base classifier, which allows the $l_{0}$ -norm of the parameter vector to be constrained to perform feature selection leading to sparse linear models. We demonstrate that the proposed ensemble model typically yields a smaller error rate than any single linear model, while maintaining the same level of sparsity and complexity at the time of testing.

Increasingly, many machine learning applications are now associated with very large data sets whose sizes were almost unimaginable just a short time ago. As a result, many of the current algorithms cannot handle, or do not scale to, today's extremely large volumes of data. Fortunately, not all features that make up a typical data set carry information that is relevant or useful for prediction, and identifying and removing such irrelevant features can significantly reduce the total data size. The unfortunate dilemma, however, is that some of the current data sets are so large that common feature selection algorithms-whose very goal is to reduce the dimensionality-cannot handle such large data sets, creating a vicious cycle. We describe a sequential learning framework for feature subset selection (SLSS) that can scale with both the number of features and the number of observations. The proposed framework uses multiarm bandit algorithms to sequentially search a subset of variables, and assign a level of importance for each feature. The novel contribution of SLSS is its ability to naturally scale to large data sets, evaluate such data in a very small amount of time, and be performed independently of the optimization of any classifier to reduce unnecessary complexity. We demonstrate the capabilities of SLSS on synthetic and real-world data sets.

Nonstationary streaming data are characterized by changes in the underlying distribution between subsequent time steps. Learning in such environments becomes even more challenging when labeled data are available only at the initial time step, and the algorithm is provided unlabeled data thereafter, a scenario referred to as extreme verification latency. Our previously introduced COMPOSE framework works very well in such settings. COMPOSE is a semi-supervised approach that iteratively labels strategically chosen instances of the next time step using the instances it labeled in the previous time step. COMPOSE originally assumed a significant distribution overlap at consecutive time steps, allowing instances lying in the center of the feature space to be used as the most representative labeled instances from current time step to help label the new data at the next time step. Such an assumption is also inherent in importance weighting based domain adaptation, but only for a single time step with mismatched train and test data distributions. We explore importance weighting not for a single time step matching training / test distributions, but rather matching distributions between two consecutive time steps, and estimate the posterior distribution of the unlabeled data using importance weighted least squares probabilistic classifier. The estimated labels are then iteratively used as the training data for the next time step. We call this algorithm as LEVELIW, Learning Extreme VErification Latency with Importance Weighting. Our primary goal in doing so is to determine if and when importance weighting provides an advantage over COMPOSE's core support extraction, and whether it provides an alternate solution with reduced parameter sensitivity. Several datasets are used to compare the two approaches, which produced some unique insights.

We propose a new non-parametric level set model for automatic image clustering and segmentation based on non-negative matrix factorization (NMF). We show that NMF: (i) clusters the image into distinct homogeneous regions and (ii) provides the local spatial distribution of each region within the image. Furthermore, NMF has a controllable resolution and can discover homogeneous regions as small as one pixel. Coupled with the level-set approach, NMF is an efficient method for image segmentation. The proposed model is unsupervised and relies on local histogram modeling to define an energy functional, whose optimization leads to the final segmentation. A unique and desirable feature of the proposed method is that it does not incorporate any spurious model parameters; hence, the optimization is performed only w.r.t level set functions. We apply the proposed Non-parametrIc Unsupervised SegmentatioN approach (geNIUS) to synthetic and real images and compare it to three state-of-the-art parametric and non-parametric level set approaches: the localized Gaussian distribution fitting model (LGDF) [1], the local histogram fitting (LHF) model [2], and our recent work: NMF-LSM in [3]. The proposed geNIUS model results in a superior accuracy and more efficient implementation, which is a result of its free-model parameter feature.

This paper proposes an algorithm that uses inductive learning and rough set theory (ILRS) to analyze the clinical data available in a patient file (records). A typical patient file has unstructured (both descriptive and quantitative) information that is also uncertain and sometimes incomplete. Successful clinical treatments depend on correct medical diagnosis which determines the correct set of variables or features causing a certain pathology. Clinical applications are by no means the only applications that require decision-making with reasoning from a large and incomplete amount of information. We show that the proposed ILRS technique is able to reduce the available number of features into a smaller core set that precisely describes the information system. We can also quantitatively evaluate the level of dependence of the considered pathology, or decision feature, on a given set of condition features or attributes. Moreover, we show that the proposed algorithm is able to cope with uncertain and incomplete information. We consider a case study of an incomplete information system obtained during cannulation of radial and dorsalis pelis arteries. We show how ILRS succeeds to remove redundancy and determine the most significant condition attributes for a given set of decision attributes from contaminated data with uncertainty. A multi-class classification with preference relations is presented through a set of decision rules. Unlike statistical analysis of clinical data, the reliability of the proposed ILRS algorithm is independent of the data size.

Dynamical systems are often required to satisfy certain constraints arising from basic physical laws, mathematical properties or geometric considerations. Incorporating constraints improves the performance of state estimation and increases the accuracy compared to unconstrained estimation.

Nonstationary environments, where underlying distributions change over time, are becoming increasingly common in real-world applications. A specific example of such an environment is concept drift, where the joint probability distributions of observed data drift over time. Such environments call for a model that can update its parameters to adapt to the changing environment. An extreme case of this scenario, referred to as extreme verification latency, is where labeled data are only available at initialization, with unlabeled data becoming available in a streaming fashion thereafter. In such a scenario, the classifier must update its hypothesis based on only unlabeled data drawn from the drifting distributions. In our prior work, we described a framework, called COMPOSE, that works well in this type of environment, provided that the data distributions experience limited (or gradual) drift. Limited drift assumption is common in many concept drift algorithms yet - surprisingly - there is little or no formal definition of this assumption. In this contribution, we describe a mechanism to formally quantify limited drift. We define two metrics, one that represents the normalized class separation drift, and the other that uses the ratio of between-class separations and within class drift through time. We test these metrics on both synthetic and real world problems, and argue that the latter can be more suitably used.

Selection of most informative features that leads to a small loss on future data are arguably one of the most important steps in classification, data analysis and model selection. Several feature selection (FS) algorithms are available; however, due to noise present in any data set, FS algorithms are typically accompanied by an appropriate cross-validation scheme. In this brief, we propose a statistical hypothesis test derived from the Neyman-Pearson lemma for determining if a feature is statistically relevant. The proposed approach can be applied as a wrapper to any FS algorithm, regardless of the FS criteria used by that algorithm, to determine whether a feature belongs in the relevant set. Perhaps more importantly, this procedure efficiently determines the number of relevant features given an initial starting point. We provide freely available software implementations of the proposed methodology.

Feature subset selection is an important step towards producing a classifier that relies only on relevant features, while keeping the computational complexity of the classifier low. Feature selection is also used in making inferences on the importance of attributes, even when classification is not the ultimate goal. For example, in bioinformatics and genomics feature subset selection is used to make inferences between the variables that best describe multiple populations. Unfortunately, many feature selection algorithms require the subset size to be specified a priori, but knowing how many variables to select is typically a nontrivial task. Other approaches rely on a specific variable subset selection framework to be used. In this work, we examine an approach to feature subset selection works with a generic variable selection algorithm, and our approach provides statistical inference on the number of features that are relevant, which may be unknown to the generic variable selection algorithm. This work extends our previous implementation of a Neyman-Pearson feature selection (NPFS) hypothesis test, which acts as a meta-subset selection algorithm. Specifically, we examine the conservativeness of the NPFS approach by biasing the hypothesis test, and examine other heuristics for NPFS. We include results from carefully designed synthetic datasets. Furthermore, we demonstrate the NPFS's ability to perform on data of a massive scale.

Learning in nonstationary environments, also called concept drift, requires an algorithm to track and learn from streaming data, drawn from a nonstationary (drifting) distribution. When data arrive continuously, a concept drift algorithm is required to maintain an up-to-date hypothesis that evolves with the changing environment. A more difficult problem that has received less attention, however, is learning from so-called initially labeled nonstationary environments, where the the environment provides only unlabeled data after initialization. Since the labels to such data never become available, learning in such a setting is also referred to as extreme verification latency, where the algorithm must only use unlabeled data to keep the hypothesis current. In this contribution, we analyze COMPOSE, a framework recently proposed for learning in such environments. One of the central processes of COMPOSE is core support extraction, where the algorithm predicts which data instances will be useful and relevant for classification in future time steps. We compare two different options, namely Gaussian mixture model based maximum a posteriori sampling and a-shape compaction, for core support extraction, and analyze their effects on both accuracy and computational complexity of the algorithm. Our findings point to-as is the case in most engineering problems - a trade-off: that a-shapes are more versatile in most situations, but they are far more computationally complex, especially as the dimensionality of the dataset increases. Our proposed GMM procedure allows COMPOSE to operate on datasets of substantially larger dimensionality without affecting its classification performance.

The ability to learn incrementally from streaming data - either in an online or batch setting - is of crucial importance for a prediction algorithm to learn from environments that generate vast amounts of data, where it is impractical or simply unfeasible to store all historical data. On the other hand, learning from streaming data becomes increasingly difficult when the probability distribution generating the data stream evolves over time, which renders the classification model generated from previously seen data suboptimal or potentially useless. Ensemble systems that employ multiple classifiers may be used to mitigate this effect, but even in such cases some classifiers (experts) become less knowledgeable for predicting on different domains than others as the distribution drifts. Further complication results when labeled data from a prediction (target) domain is not immediately available; hence, causing prediction on the target domain to yield sub-optimal results. In this work, we provide upper bounds on the loss, which hold with high probability, of a multiple expert system trained in such a nonstationary environment with verification latency. Furthermore, we show why a single model selection strategy can lead to undesirable results when learning in such nonstationary streaming settings. We present our analytical results with experiments on simulated as well as real-world data sets, comparing several different ensemble approaches to a single model.

A novel method of classifying data drawn from a nonstationary distribution with drifting mean and variance is presented. The novelty of the approach is based on splitting the problem of tracking a nonstationary distribution into separate classification and time series state estimation problems. State space models for drift in both the mean and variance are presented, which are then successfully tracked using a Kaiman filter and a particle filter for the linear and non-linear parts respectively. Preliminary results, which show the promising potential of the approach, are also presented, along with concluding remarks for potential uses of the proposed approach.

An increasing number of real-world applications are associated with streaming data drawn from drifting and nonstationary distributions that change over time. These applications demand new algorithms that can learn and adapt to such changes, also known as concept drift. Proper characterization of such data with existing approaches typically requires substantial amount of labeled instances, which may be difficult, expensive, or even impractical to obtain. In this paper, we introduce compacted object sample extraction (COMPOSE), a computational geometry-based framework to learn from nonstationary streaming data, where labels are unavailable (or presented very sporadically) after initialization. We introduce the algorithm in detail, and discuss its results and performances on several synthetic and real-world data sets, which demonstrate the ability of the algorithm to learn under several different scenarios of initially labeled streaming environments. On carefully designed synthetic data sets, we compare the performance of COMPOSE against the optimal Bayes classifier, as well as the arbitrary subpopulation tracker algorithm, which addresses a similar environment referred to as extreme verification latency. Furthermore, using the real-world National Oceanic and Atmospheric Administration weather data set, we demonstrate that COMPOSE is competitive even with a well-established and fully supervised nonstationary learning algorithm that receives labeled data in every batch.

The papers in this special issue encompass a broad spectrum of scenarios and problems, including some of the new challenges and fundamental problems that are encountered in learning in nonstationary and evolving environments.

Increasing number of practical applications that involve streaming nonstationary data have led to a recent surge in algorithms designed to learn from such data. One challenging version of this problem that has not received as much attention, however, is learning streaming nonstationary data when a small initial set of data are labeled, with unlabeled data being available thereafter. We have recently introduced the COMPOSE algorithm for learning in such scenarios, which we refer to as initially labeled nonstationary streaming data. COMPOSE works remarkably well, however it requires limited (gradual) drift, and cannot address special cases such as introduction of a new class or significant overlap of existing classes, as such scenarios cannot be learned without additional labeled data. Scenarios that provide occasional or periodic limited labeled data are not uncommon, however, for which many of COMPOSE's restrictions can be lifted. In this contribution, we describe a new version of COMPOSE as a proof-of-concept algorithm that can identify the instances whose labels - if available - would be most beneficial, and then combine those instances with unlabeled data to actively learn from streaming nonstationary data, even when the distribution of the data experiences abrupt changes. On two carefully designed experiments that include abrupt changes as well as addition of new classes, we show that COMPOSE.AL significantly outperforms original COMPOSE, while closely tracking the optimal Bayes classifier performance.

Multiple classifier systems tend to suffer from outvoting when new concept classes need to be learned incrementally. Out-voting is primarily due to existing classifiers being unable to recognize the new class until there is a sufficient number of new classifiers that can influence the ensemble decision. This problem of learning new classes was explicitly addressed in Learn++.NC, our previous work, where ensemble members dynamically adjust their own weights by consulting with each other based on their individual and collective confidence in classifying each concept class. Learn++.NC works remarkably well for learning new concept classes while requiring few ensemble members to do so. Learn++.NC cannot cope with the class imbalance problem, however, as it was not designed to do so. Yet, class imbalance is a common and important problem in machine learning, made even more challenging in an incremental learning setting. In this paper, we extend Learn++.NC so that it can incrementally learn new concept classes even if their instances are drawn from severely imbalanced class distributions. We show that the proposed algorithm is quite robust compared to other state-of-the-art algorithms.

It is very important and challenging to teach modern topics at the freshman level. This paper describes a biometrics module that fits into any introductory freshman engineering course. The project has broad learning outcomes, namely, enhanced application of math skills, software implementation skills, interest in biometrics and comprehension of ethical issues. Assessment results based on the analysis of student surveys related to the learning outcomes and vertical integration show that the project was successful.

Learning in nonstationary environments, also known as learning concept drift, is concerned with learning from data whose statistical characteristics change over time. Concept drift is further complicated if the data set is class imbalanced. While these two issues have been independently addressed, their joint treatment has been mostly underexplored. We describe two ensemble-based approaches for learning concept drift from imbalanced data. Our first approach is a logical combination of our previously introduced Learn++.NSE algorithm for concept drift, with the well-established SMOTE for learning from imbalanced data. Our second approach makes two major modifications to Learn++.NSE-SMOTE integration by replacing SMOTE with a subensemble that makes strategic use of minority class data; and replacing Learn++.NSE and its class-independent error weighting mechanism with a penalty constraint that forces the algorithm to balance accuracy on all classes. The primary novelty of this approach is in determining the voting weights for combining ensemble members, based on each classifier's time and imbalance-adjusted accuracy on current and past environments. Favorable results in comparison to other approaches indicate that both approaches are able to address this challenging problem, each with its own specific areas of strength. We also release all experimental data as a resource and benchmark for future research.

The field of forensics aims to understand the physical biomarkers that make each person unique. Recently, it has been discovered that one of the traits that makes us unique from one another are the composition of the microbial communities found throughout our bodies. For example, identical twins who share the same set of DNA may have vastly different microbial communities in or on various body sites. It was recently discovered that microbial communities can be exploited for forensic identification by clustering samples from individual's skin and objects that they may have previously touched. Typically, this is done by using basic multi-dimensional scaling analysis using phylogenetic distances. In this work, we circumvent the use of phylogenetic distances by using the raw community abundances, and we present an application of kernels for metagenomic data analysis. In addition, we show that strategic selection of features can improve classification accuracy.

Many traditional supervised machine learning approaches, either on-line or batch based, assume that data are sampled from a fixed yet unknown source distribution. Most incremental learning algorithms also make the same assumption, even though new data are presented over periods of time. Yet, many real-world problems are characterized by data whose distribution change over time, which implies that a classifier may no longer be reliable on future data, a problem commonly referred to as concept drift or learning in nonstationary environments. The issue is further complicated when the problem requires prediction from data obtained at a future time step, for which the labels are not yet available. In this work, we present a transductive learning methodology that uses probabilistic models to aid in computing ensemble classifier voting weights. Assuming the drift is limited in nature, the proposed approach exploits a probabilistic estimate to determine the class responsibility of components in a Gaussian mixture model (GMM), generated from labeled and unlabeled data. A general error bound is provided based on the ensemble decision, the probabilistic estimate of the GMM, and the true labeling function, which, unfortunately is never actually known.

Abstract—Semi-supervised learning (SSL) in non-stationary environments has received relatively little attention in machine learning, despite a growing number of applications that can benefit from a properly configured SSL algorithm. Previous works in learning non-stationary data have analyzed such cases where both labeled and unlabeled instances are received at every time step and/or in regular intervals; however, to the best of our knowledge, no work has investigated the case where labeled instances are received only at the initial time step, followed by unlabeled instances provided in subsequent time steps. In this proof-of-concept work, we propose a new framework for learning in a non-stationary environment that provides only unlabeled data after the initial time step, to which we refer to as initially labeled environment. The proposed framework generates labels for previously unlabeled data at each time step to be combined with incoming unlabeled data – possibly from a drifting distribution- using a compacted polytope sample extraction algorithm. We have conducted two experiments to demonstrate the feasibility and reliability of the approach. This proof-of-concept is presented in two dimensions; however, the algorithm can be extended to higher dimensions with appropriate modifications. Keywords-alpha shape; concept drift; non-stationary environment; shape offsets; semi-supervised learning I.

It is very important that biometrics education, particularly at the undergraduate level, keeps pace with the rapidly growing global market. This paper describes a senior level project in speech biometrics that fits in a variety of courses in order to reach out to many students. The project has broad learning outcomes, namely, enhanced application of math skills, software implementation skills, interest in biometrics, ability to carry out open-ended design and communication skills. Assessment results based on the analysis of the success of the students (refereed publications and enrolling in graduate programs), student surveys related to the learning outcomes and a target versus control group survey show that the project was successful.

Learning in non-stationary environments is an increasingly important problem in a wide variety of real-world applications. In non-stationary environments data arrives incrementally, however the underlying generating function may change over time. While there is a variety of research into such environments, the research mainly consists of detecting concept drift (and then relearning the model), or developing classifiers which adapt to drift incrementally. We introduce Heuristic Up datable Weighted Random Subspaces (HUWRS), a new technique based on the Random Subspace Method that detects drift in individual features via the use of Hellinger distance, a distributional divergence metric. Through the use of subspaces, HUWRS allows for a more fine-grained approach to dealing with concept drift which is robust to feature drift even without class labels. We then compare our approach to two state of the art algorithms, concluding that for a wide range of datasets and window sizes HUWRS outperforms the other methods.

Learning in nonstationary environments, also called learning concept drift, has been receiving increasing attention due to increasingly large number of applications that generate data with drifting distributions. These applications are usually associated with streaming data, either online or in batches, and concept drift algorithms are trained to detect and track the drifting concepts. While concept drift itself is a significantly more complex problem than the traditional machine learning paradigm of data coming from a fixed distribution, the problem is further complicated when obtaining labeled data is expensive, and training must rely, in part, on unlabelled data. Independently from concept drift research, semi-supervised approaches have been developed for learning from (limited) labeled and (abundant) unlabeled data; however, such approaches have been largely absent in concept drift literature. In this contribution, we describe an ensemble of classifiers based approach that takes advantage of both labeled and unlabeled data in addressing concept drift: available labeled data are used to generate classifiers, whose voting weights are determined based on the distances between Gaussian mixture model components trained on both labeled and unlabeled data in a drifting environment.

IAM ABOUT to start my second year of service as the Editor-in-Chief (EiC) of the IEEE TRANSACTIONS ON NEURAL NETWORKS (TNN). Needless to say, my first year as the EiC has been full of excitement and challenges. Transitioning this position from my predecessor to me went very smoothly during the months of September 2009 to January 2010. During the past year, we have accumulated 50+ Associate Editors (AEs) handling roughly 600 new submissions (not counting resubmissions and revised submissions). With the help of these AEs and my predecessor, I was quickly able to learn to do my job, and as such, the transition had very few glitches. The easy part of my job is checking whether a submission is in compliance with our guidelines and where it is within the scope of the TRANSACTIONS, before it is assigned to an AE for handling. The difficult part of my job has been dealing with some papers with three or more reviewers, all of whom agreed to review them but for some reason failed to respond to repeated automatic-review reminders. AEs handling these papers have to take several extra steps to remind reviewers through phone calls or e-mails, look for replacement reviewers, or review the papers themselves. Most authors have been appreciative of the work of the AEs and reviewers, and they accept our decisions without a problem. The backlog of papers has been kept short over the last year. We have maintained an organized printing and paperacceptance schedule, with papers typically printed in the journal within 2‐3 months of acceptance. Our page budget has been kept constant in the past few years (roughly 2060 pages per year), and we expect to hold the same page count for next year.

We introduce an ensemble of classifiers-based approach for incremental learning of concept drift, characterized by nonstationary environments (NSEs), where the underlying data distributions change over time. The proposed algorithm, named Learn(++). NSE, learns from consecutive batches of data without making any assumptions on the nature or rate of drift; it can learn from such environments that experience constant or variable rate of drift, addition or deletion of concept classes, as well as cyclical drift. The algorithm learns incrementally, as other members of the Learn(++) family of algorithms, that is, without requiring access to previously seen data. Learn(++). NSE trains one new classifier for each batch of data it receives, and combines these classifiers using a dynamically weighted majority voting. The novelty of the approach is in determining the voting weights, based on each classifier's time-adjusted accuracy on current and past environments. This approach allows the algorithm to recognize, and act accordingly, to the changes in underlying data distributions, as well as to a possible reoccurrence of an earlier distribution. We evaluate the algorithm on several synthetic datasets designed to simulate a variety of nonstationary environments, as well as a real-world weather prediction dataset. Comparisons with several other approaches are also included. Results indicate that Learn(++). NSE can track the changing environments very closely, regardless of the type of concept drift. To allow future use, comparison and benchmarking by interested researchers, we also release our data used in this paper.

Using a machine learning algorithm for a given application often requires tuning design parameters of the classifier to obtain optimal classification performance without overfitting. In this contribution, we present an evolutionary algorithm based approach for multi-objective optimization of the sensitivity and specificity of a v-SVM. The v-SVM is often preferred over the standard C-SVM due to smaller dynamic range of the v parameter compared to the unlimited dynamic range of the C parameter. Instead of looking for a single optimization result, we look for a set of optimal solutions that lie along the Pareto optimality front. The traditional advantage of using the Pareto optimality is of course the flexibility to choose any of the solutions that lies on the Pareto optimality front. However, we show that simply maximizing sensitivity and specificity over the Pareto front leads to parameters that appear to be mathematically optimal yet still cause overfitting. We propose a multiple objective optimization approach with three objective functions to find additional parameter values that do not cause overfitting.

Learning in a non-stationary environment and in the presence of class imbalance has been receiving more recognition from the computational intelligence community, but little work has been done to create an algorithm or a framework that can handle both issues simultaneously. We have recently introduced a new member to the Learn++family of algorithms, Learn++.NSE, which is designed to track non-stationary environments. However, this algorithm does not work well when there is class imbalance as it has not been designed to handle this problem. On the other hand, SMOTE - a popular algorithm that can handle class imbalance - is not designed to learn in nonstationary environments because it is a method of over sampling the data. In this work we describe and present preliminary results for integrating SMOTE and Learn++.NSE to create an algorithm that is robust to learning in a non-stationary environment and under class imbalance.

We have recently introduced an incremental learning algorithm, Learn++.NSE, designed to learn in nonstationary environments, and has been shown to provide an attractive solution to a number of concept drift problems under different drift scenarios. However, Learn++.NSE relies on error to weigh the classifiers in the ensemble on the most recent data. For balanced class distributions, this approach works very well, but when faced with imbalanced data, error is no longer an acceptable measure of performance. On the other hand, the well-established SMOTE algorithm can address the class imbalance issue, however, it cannot learn in nonstationary environments. While there is some literature available for learning in nonstationary environments and imbalanced data separately, the combined problem of learning from imbalanced data coming from nonstationary environments is underexplored. Therefore, in this work we propose two modified frameworks for an algorithm that can be used to incrementally learn from imbalanced data coming from a nonstationary environment.

Metagenomic studies inherently involve sampling genetic information from an environment potentially containing thousands of distinctly different microbial organisms. This genetic information is sequenced producing many short fragments (<;500 base pair (bp)); each is tentatively a small representative of the DNA coding structure. Any of the fragments may belong to any of the organisms in the sample, but the relationship is unknown a priori. Furthermore, most of these organisms have not been identified and correspondingly are not represented in any of the publicly available search databases. Our goal is to be able to predict the taxonomic classification of an organism based on the fragments obtained from an environmental sample that may include many (some previously unidentified) organisms. To elucidate the diversity and composition of the sample, we first use a supervised naive Bayes classifier to score the fragments of known genomes, followed by an unsupervised clustering to group fragments from similar organisms together. We are then free to analyze each cluster separately. This is challenging since we are not interested in similar sequences, but sequences that come from similar genomes, which are known to vary widely intra-genomically. Our dataset comprises of an extremely challenging scenario involving clustering fragments at the phyla level, where none of the phyla have been previously seen or identified. We present two variations of our proposed approach, one based on ART and K-means. We show that ART can cluster 500bp fragments from 17 novel phyla at an overall isolation/grouping that is 10% better than K-means and nearly 7 times over chance.

We have recently introduced an incremental learning algorithm, called Learn++.NSE, designed for Non-Stationary Environments (concept drift), where the underlying data distribution changes over time. With each dataset drawn from a new environment, Learn++.NSE generates a new classifier to form an ensemble of classifiers. The ensemble members are combined through a dynamically weighted majority voting, where voting weights are determined based on classifiers' age-adjusted accuracy on current and past environments. Unlike other ensemble-based concept drift algorithms, Learn++.NSE does not discard prior classifiers, allowing potentially cyclical environments to be learned more effectively. While Learn++.NSE has been shown to work well on a variety of concept drift problems, a potential shortcoming of this approach is the cumulative nature of the ensemble size. In this contribution, we expand our analysis of the algorithm to include various ensemble pruning methods to introduce controlled forgetting. Error or age-based pruning methods have been integrated into the algorithm to prevent potential out-voting from irrelevant classifiers or simply to save memory over an extended period of time. Here, we analyze the tradeoff between these precautions and the desire to handle recurring contexts (cyclical data). Comparisons are made using several scenarios that introduce various types of drift.

We have previously introduced an incremental learning algorithm Learn(++), which learns novel information from consecutive data sets by generating an ensemble of classifiers with each data set, and combining them by weighted majority voting. However, Learn(++) suffers from an inherent "outvoting" problem when asked to learn a new class omega(new) introduced by a subsequent data set, as earlier classifiers not trained on this class are guaranteed to misclassify omega(new) instances. The collective votes of earlier classifiers, for an inevitably incorrect decision, then outweigh the votes of the new classifiers' correct decision on omega(new) instances--until there are enough new classifiers to counteract the unfair outvoting. This forces Learn(++) to generate an unnecessarily large number of classifiers. This paper describes Learn(++).NC, specifically designed for efficient incremental learning of multiple new classes using significantly fewer classifiers. To do so, Learn (++).NC introduces dynamically weighted consult and vote (DW-CAV), a novel voting mechanism for combining classifiers: individual classifiers consult with each other to determine which ones are most qualified to classify a given instance, and decide how much weight, if any, each classifier's decision should carry. Experiments on real-world problems indicate that the new algorithm performs remarkably well with substantially fewer classifiers, not only as compared to its predecessor Learn(++), but also as compared to several other algorithms recently proposed for similar problems.

Nearest neighbour classifiers and related kernel methods often perform poorly in high dimensional problems because it is infeasible to include enough training samples to cover the class regions densely. In such cases, test samples often fall into gaps between training samples where the nearest neighbours are too distant to be good indicators of class membership. One solution is to project the data onto a discriminative lower dimensional subspace. We propose a gap-resistant nonparametric method for finding such subspaces: first the gaps are filled by building a convex model of the region spanned by each class - we test the affine and convex hulls and the bounding disk of the class training samples - then a set of highly discriminative directions is found by building and decomposing a scatter matrix of weighted displacement vectors from training examples to nearby rival class regions. The weights are chosen to focus attention on narrow margin cases while still allowing more diversity and hence more discriminability than the 1D linear Support Vector Machine (SVM) projection. Experimental results on several face and object recognition datasets show that the method finds effective projections, allowing simple classifiers such as nearest neighbours to work well in the low dimensional reduced space.

In high-dimensional classification problems it is infeasible to include enough training samples to cover the class regions densely. Irregularities in the resulting sparse sample distributions cause local classifiers such as Nearest Neighbors (NN) and kernel methods to have irregular decision boundaries. One solution is to "fill in the holes" by building a convex model of the region spanned by the training samples of each class and classifying examples based on their distances to these approximate models. Methods of this kind based on affine and convex hulls and bounding hyperspheres have already been studied. Here we propose a method based on the bounding hyperdisk of each class - the intersection of the affine hull and the smallest bounding hypersphere of its training samples. We argue that in many cases hyperdisks are preferable to affine and convex hulls and hyperspheres: they bound the classes more tightly than affine hulls or hyperspheres while avoiding much of the sample overfitting and computational complexity that is inherent in high-dimensional convex hulls. We show that the hyperdisk method can be kernelized to provide nonlinear classifiers based on non-Euclidean distance metrics. Experiments on several classification problems show promising results.

We outline an incremental learning algorithm designed for nonstationary environments where the underlying data distribution changes over time. With each dataset drawn from a new environment, we generate a new classifier. Classifiers are combined through dynamically weighted majority voting, where voting weights are determined based on classifiers¿ age and accuracy on current and past environments. The most recent and relevant classifiers are weighted higher, allowing the algorithm to appropriately adapt to drifting concepts. This algorithm does not discard prior classifiers, allowing efficient learning of potentially cyclical environments. The algorithm learns incrementally, i.e., without access to previous data. Finally, the algorithm can use any supervised classifier as its base model, including those not normally capable of incremental learning. We present the algorithm and its performance using different base learners in different environments with varying types of drift.

We describe an ensemble of classifiers based approach for incrementally learning from new data drawn from a distribution that changes in time, i.e., data obtained from a nonstationary environment. Specifically, we generate a new classifier using each additional dataset that becomes available from the changing environment. The classifiers are combined by a modified weighted majority voting, where the weights are dynamically updated based on the classifierspsila current and past performances, as well as their age. This mechanism allows the algorithm to track the changing environment by weighting the most recent and relevant classifiers higher. However, it also utilizes old classifiers by assigning them appropriate voting weights should a cyclical environment renders them relevant again. The algorithm learns incrementally, i.e., it does not need access to previously used data. The algorithm is also independent of a specific classifier model, and can be used with any classifier that fits the characteristics of the underlying problem. We describe the algorithm, and compare its performance using several classifier models, and on different environments as a function of time for several values of rate-of-change.

It has been widely accepted that the classification accuracy can be improved by combining outputs of multiple classifiers. However, how to combine multiple classifiers with various (potentially conflicting) decisions is still an open problem. A rich collection of classifier combination procedures -- many of which are heuristic in nature -- have been developed for this goal. In this brief, we describe a dynamic approach to combine classifiers that have expertise in different regions of the input space. To this end, we use local classifier accuracy estimates to weight classifier outputs. Specifically, we estimate local recognition accuracies of classifiers near a query sample by utilizing its nearest neighbors, and then use these estimates to find the best weights of classifiers to label the query. The problem is formulated as a convex quadratic optimization problem, which returns optimal nonnegative classifier weights with respect to the chosen objective function, and the weights ensure that locally most accurate classifiers are weighted more heavily for labeling the query sample. Experimental results on several data sets indicate that the proposed weighting scheme outperforms other popular classifier combination schemes, particularly on problems with complex decision boundaries. Hence, the results indicate that local classification-accuracy-based combination techniques are well suited for decision making when the classifiers are trained by focusing on different regions of the input space.

As a natural consequence of steady increase of average population age in developed countries, Alzheimer's disease is becoming an increasingly important public health concern. The financial and emotional toll of the disease is exacerbated with lack of standard diagnostic procedures available at the community clinics and hospitals, where most patients are evaluated. In our recent preliminary results, we have reported that the event related potentials (ERPs) of the electroencephalogram can be used to train an ensemble-based classifier for automated diagnosis of Alzheimer's disease. In this study, we present an updated alternative approach by combining complementary information provided by ERPs obtained from several parietal region electrodes. The results indicate that ERPs obtained from parietal region of the cortex carry substantial complementary diagnostic information. Specifically, the diagnostic ability of such an approach is substantially better, compared to the performance obtained by using data from any of the individual electrodes alone. Furthermore, the diagnostic performance of the proposed approach compares very favorably to that obtained at community clinics and hospitals.

We discuss an ensemble-of-classifiers based algorithm for the missing feature problem. The proposed approach is inspired in part by the random subspace method, and in part by the incremental learning algorithm, Learn++. The premise is to generate an adequately large number of classifiers, each trained on a different and random combination of features, drawn from an iteratively updated distribution. To classify an instance with missing features, only those classifiers whose training data did not include the currently missing feature are used. These classifiers are combined by using a majority voting combination rule to obtain the final classification of the given instance. We had previously presented preliminary results on a similar approach, which could handle up to 10% missing data. In this study, we expand our work to include different types of rules to update the distribution, and also examine the effect of the algorithm's primary free parameter (the number of features used to train the ensemble of classifiers) on the overall classification performance. We show that this algorithm can now accommodate up to 30% of features missing without a significant drop in performance.

This paper introduces Learn++, an ensemble of classifiers based algorithm originally developed for incremental learning, and now adapted for information/data fusion applications. Recognizing the conceptual similarity between incremental learning and data fusion, Learn++ follows an alternative approach to data fusion, i.e., sequentially generating an ensemble of classifiers that specifically seek the most discriminating information from each data set. It was observed that Learn++ based data fusion consistently outperforms a similarly configured ensemble classifier trained on any of the individual data sources across several applications. Furthermore, even if the classifiers trained on individual data sources are fine tuned for the given problem, Learn++ can still achieve a statistically significant improvement by combining them, if the additional data sets carry complementary information. The algorithm can also identify-albeit indirectly-those data sets that do not carry such additional information. Finally, it was shown that the algorithm can consecutively learn both the supplementary novel information coming from additional data of the same source, and the complementary information coming from new data sources without requiring access to any of the previously seen data.

With the rapid increase in the population of elderly individuals affected by Alzheimer's disease, the need for an accurate, inexpensive and non-intrusive diagnostic biomarker that can be made available to community healthcare providers presents itself as a major public health concern. The feasibility of EEG as such a biomarker has gained a renewed attention as several recent studies, including our previous efforts, reported promising results. In this paper we present our preliminary results on using wavelet coefficients of event related potentials along with an ensemble of classifiers combined with majority vote and decision templates

As the population of our elderly suffering from Alzheimer's disease increases rapidly, the need for an accurate, inexpensive and non-intrusive diagnostic procedure that can be made available to local community clinics becomes an increasingly critical public health concern. We propose multiresolution analysis of the electroencephalogram (EEG) followed by an ensemble based classification designed to fuse data from different EEG channels. Several classifier combination rules, including competence based weighted combination have been implemented to evaluate their data fusion performance, with particular emphasis on diagnosing the disease at its earliest stages. Diagnostic performance of the proposed approach has been very promising.

We have recently introduced Learn++, an incremental learning algorithm, inspired by the multiple classifiers structure of AdaBoost. Both algorithms generate an ensemble of classifiers trained on bootstrapped replicates of the training data, and the classifiers are then combined through a voting process. Learn++, however, generates additional ensembles as new data become available, and uses a different distribution update rule to resample the data. While AdaBoost was originally designed to improve the performance of a weak classifier, whether it can still achieve incremental learning through its ensemble structure is still an open question. In this paper, we compare the incremental learning ability of AdaBoost.M1 and Learn++under very hostile nonstationary learning environments, which may introduce new concept classes. We also compare the algorithms under several combination rules to determine which of the three key components -ensemble structure, resampling procedure, or the combination rule -has the primary impact on incremental learning in nonstationary environments.

The diagnosis of Alzheimer's disease at an early stage is a major concern due to the growing number of the elderly population affected, as well as the lack of a standard and effective diagnosis procedure available to community healthcare providers. Recent studies have used wavelets and other signal processing methods to analyze EEG signals in an attempt to find a non-invasive biomarker for Alzheimer's disease and had varying degrees of success. These studies have traditionally used automated classifiers such as neural networks; however the use of an ensemble of classifiers has not been previously explored and may prove to be beneficial. In this study, multiresolution wavelet analysis is performed on event related potentials of the EEG which are then used with the ensemble of classifiers based Learn++ algorithm. We describe the approach, and present our promising preliminary results.

We have previously introduced Learn++, an ensemble based incremental learning algorithm for acquiring new knowledge from data that later become available, even when such data introduce new classes. In this paper, we describe a modification to this algorithm, where the voting weights of the classifiers are updated dynamically based on the location of the test input in the feature space. The new algorithm provides improved performance, stronger immunity to catastrophic forgetting and finer balance to the stability-plasticity dilemma than its predecessor, particularly when new classes are introduced. The modified algorithm and its performance, as compared to Adaboost.Ml and the original Learn++, on real and benchmark datasets are presented.

An ensemble based algorithm, Learn++. MT2, is introduced as an enhanced alternative to our previously reported incremental learning algorithm, Learn++. Both algorithms are capable of incrementally learning novel information from new datasets that consecutively become available, without requiring access to the previously seen data. In this contribution, we describe Learn++. MT2, which specifically targets incrementally learning from distinctly unbalanced data, where the amount of data that become available varies significantly from one database to the next. The problem of unbalanced data within the context of incremental learning is discussed first, followed by a description of the proposed solution. Initial, yet promising results indicate considerable improvement on the generalization performance and the stability of the algorithm.

A new learning algorithm is introduced that can accommodate data with missing features. The algorithm uses an ensemble of classifiers approach. The classifiers in the ensemble are trained with random subsets of the total number of available features. The approach takes advantage of the basic assumption that an unknown subset of the features is in fact adequate for the classification, or in other words, that are redundant, and possibly irrelevant features in the data. This assumption is in general true for most practical applications. We empirically show that if a certain number of networks produce a particular classification performance using all of the features, then the same classification performance can be reached even if some features are missing, as long as the same number of usable networks can be generated with the missing features. The proposed approach has its roots in the incremental learning algorithm, Learn/sup ++/ which seeks to learn new information that is provided by additional datasets that may later become available, even when such data introduce new classes. We have modified the Learn/sup ++/ algorithm for addressing the missing feature problem. The proposed algorithm showed surprisingly remarkable performance on three real-world applications, with up to 10% of the features missing in the validation / field data.

An incremental learning algorithm based on weighted majority voting of an ensemble of classifiers is introduced for supervised neural networks, where the voting weights are updated dynamically based on the current test input of unknown class. The algorithm's dynamic voting weight update feature is an enhancement to our previously introduced incremental learning algorithm, Learn++. The algorithm is capable of incrementally learning new information from additional datasets that may later become available, even when the new datasets include instances from additional classes that were not previously seen. Furthermore, the algorithm retains formerly acquired knowledge without requiring access to datasets used earlier, attaining a delicate balance on the stability-plasticity dilemma. The algorithm creates additional ensembles of classifiers based on an iteratively updated distribution function on the training data that favors training with increasingly difficult to learn, previously not learned and/or unseen instances. The final classification is made by weighted majority voting of all classifier outputs in the ensemble, where the voting weights are determined dynamically during actual testing, based on the estimated performance of each classifier on the current test data instance. We present the algorithm in its entirety, as well as its promising simulation results on two real world applications.

Nondestructive evaluation of the gas pipeline system is most commonly performed using magnetic flux leakage (MFL) techniques. A major segment of this network employs seamless pipes. The data obtained From MFL inspection of seamless pipes is contaminated by various sources of noise, including seamless pipe noise due to material properties of the pipe, lift-off variation of MFL sensor due to motion of the pipe and system noise due to on-board electronics. The noise can considerably reduce the detectability of defect signals in MFL data. This paper presents a new technique for improving the signal-to-noise-ratio in MFL data obtained from seamless pipes. The approach utilizes normalized least mean squares adaptive noise filtering coupled with wavelet shrinkage denoising to minimize the effects of various sources of noise. Results from application of the approach to data from field tests are presented. It is shown that the proposed algorithm is computationally efficient and data-independent.

Gas sensing systems for detection and identification of odorant molecules are of crucial importance in an increasing number of applications. Such applications include environmental monitoring, food quality assessment, airport security, and detection of hazardous gases. We describe a gas sensing system for detecting and identifying volatile organic compounds (VOC), and discuss the unique problems associated with the separability of signal patterns obtained by using such a system. We then present solutions for enhancing the separability of VOC patterns to enable classification. A new incremental learning algorithm that allows new odorants to be learned is also introduced.

We introduce Learn++, an algorithm for incremental training of neural network (NN) pattern classifiers. The proposed algorithm enables supervised NN paradigms, such as the multilayer perceptron (MLP), to accommodate new data, including examples that correspond to previously unseen classes. Furthermore, the algorithm does not require access to previously used data during subsequent incremental learning sessions, yet at the same time, it does not forget previously acquired knowledge. Learn++ utilizes ensemble of classifiers by generating multiple hypotheses using training data sampled according to carefully tailored distributions. The outputs of the resulting classifiers are combined using a weighted majority voting procedure. We present simulation results on several benchmark datasets as well as a real-world classification task. Initial results indicate that the proposed algorithm works rather well in practice. A theoretical upper bound on the error of the classifiers constructed by Learn++ is also provided.

We introduce a supervised learning algorithm that gives neural network classification algorithms the capability of learning incrementally from new data without forgetting what has been learned in earlier training sessions. Schapire's (1990) boosting algorithm, originally intended for improving the accuracy of weak learners, has been modified to be used in an incremental learning setting. The algorithm is based on generating a number of hypotheses using different distributions of the training data and combining these hypotheses using a weighted majority voting. This scheme allows the classifier previously trained with a training database, to learn from new data when the original data is no longer available, even when new classes are introduced. Initial results on incremental training of multilayer perceptron networks on synthetic as well as real-world data are presented in this paper.

Neural networks, particularly the multilayer perceptron, have been used extensively in automated signal classification systems with classification accuracy as the figure of merit. Three important issues that can enhance the utility of these systems are (i) incremental learning, (ii) confidence or reliability measures and (iii) performance improvement through continual learning. This paper investigates these issues using a fuzzy ARTMAP network. A hypothesis testing based algorithm is developed for computing reliability measures, which are fed back to the network for retraining and performance improvement. Implementation results on ultrasonic data are presented.

The objective of classification is to generate a nonlinear multidimensional decision boundary that partitions the pattern space into prescribed classes. However, these algorithms are successful only when the data is well distributed in their domain. In practice, patterns from different classes can be closely packed with significant overlap. Prior to classification, the data is generally preprocessed so that the intercluster to intracluster distance ratio is maximized. This paper discusses limitations of conventional approaches for preprocessing based on Fisher's linear discriminant, and proposes an intuitive nonlinear cluster transformation (NCT) that can be used for increasing the intercluster distances within a set of data points. A generalized regression neural network (GRNN) is used to learn the functional mapping between original clusters and transformed clusters. The performance of this proposed method was tested on a benchmark database and then on a real world database of patterns generated for odor identification. Initial results using NCT have been very promising.