ABSTRACT In text classification, creating an adversarial example means subtly perturbing a few words in a sentence without changing its meaning, causing it to be misclassified by a classifier. A concerning observation is that a significant portion of adversarial examples generated by existing methods change only one word. This single‐word perturbation vulnerability represents a significant weakness in classifiers, which malicious users can exploit to efficiently create a multitude of adversarial examples. This paper studies this problem and makes the following key contributions: (1) We introduce a novel metric to quantitatively assess a classifier's robustness against single‐word perturbation . (2) We present the SP‐Attack, designed to exploit the single‐word perturbation vulnerability, achieving a higher attack success rate, better preserving sentence meaning, while reducing computation costs compared to state‐of‐the‐art adversarial methods. (3) We propose SP‐Defence, which aims to improve by applying data augmentation in learning. Experimental results on 4 datasets and 2 masked language models show that SP‐Defence improves by 14.6% and 13.9% and decreases the attack success rate of SP‐Attack by 30.4% and 21.2% on two classifiers respectively, and decreases the attack success rate of existing attack methods that involve multiple‐word perturbations.

Time series anomaly detection is a vital task in many domains, including patient monitoring in healthcare, forecasting in finance, and predictive maintenance in energy industries. This has led to a proliferation of anomaly detection methods, including deep learning-based methods. Benchmarks are essential for comparing the performances of these models as they emerge, in a fair, rigorous, and reproducible approach. Although several benchmarks for comparing models have been proposed, these usually rely on a one-time execution over a limited set of datasets, with comparisons restricted to a few models. We propose OrionBench– an end-user centric, continuously maintained benchmarking framework for unsupervised time series anomaly detection models. Our framework provides universal abstractions to represent models, extensibility to add new pipelines and datasets, hyperparameter standardization, pipeline verification, and frequent releases with published updates of the benchmark. We demonstrate how to use OrionBench, and the performance of pipelines across 17 releases published over the course of four years. We also walk through two real scenarios we experienced with OrionBench that highlight the importance of continuous benchmarking for unsupervised time series anomaly detection.

Explanations of machine learning (ML) model predictions generated by Explainable AI (XAI) techniques such as SHAP are essential for people using ML outputs for decision-making. We explore the potential of Large Language Models (LLMs) to transform these explanations into human-readable, narrative formats that align with natural communication. We address two key research questions: (1) Can LLMs reliably transform traditional explanations into high-quality narratives? and (2) How can we effectively evaluate the quality of narrative explanations? To answer these questions, we introduce Explingo, which consists of two LLM-based subsystems, a Narrator and Grader. The Narrator takes in ML explanations and transforms them into natural-language descriptions. The Grader scores these narratives on a set of metrics including accuracy, completeness, fluency, and conciseness.Our experiments demonstrate that LLMs can generate high-quality narratives that achieve high scores across all metrics, particularly when guided by a small number of human-labeled and bootstrapped examples. We also identified areas that remain challenging, in particular for effectively scoring narratives in complex domains. The findings from this work have been integrated into an open-source tool that makes narrative explanations available for further applications.

The flexible nature of large language models allows them to be used for diverse applications. Recent studies have showcased numerous abilities of these models, including performing time series forecasting. In this paper, we present a novel study of large language models used for the challenging task of time series anomaly detection. This problem entails two novel aspects for LLMs specifically: first, the model needs to be able to identify part of an input sequence (or multiple parts) as anomalous; and second, the model needs to work with time series data rather than with text input. We introduce SIGLLM, a framework for time series anomaly detection using large language models. Our framework includes a time-series-to-text conversion module, as well as end-to-end pipelines that prompt language models to perform time series anomaly detection. We investigate two paradigms for testing the abilities of large language models to perform the detection task. First, we present a prompt-based detection method that directly asks a language model to indicate which elements of the input are anomalies. Second, we leverage the forecasting capability of a large language model to guide the anomaly detection process. We evaluated our framework on 11 datasets spanning various sources and 10 pipelines. We show that the forecasting method significantly outperformed the prompting method in all 11 datasets with respect to the F1 score. Moreover, while large language models are capable of finding anomalies, state-of-the-art deep learning models are still superior in performance, achieving 30% improvement.

Anomaly detection on time series data is increasingly common across various industrial domains that monitor metrics in order to prevent potential accidents and economic losses. However, a scarcity of labeled data and ambiguous definitions of anomalies can complicate these efforts. Recent unsupervised machine learning methods have made remarkable progress in tackling this problem using either single-timestamp predictions or time series reconstructions. While traditionally considered separately, these methods are not mutually exclusive and can offer complementary perspectives on anomaly detection. This paper first highlights the successes and limitations of prediction-based and reconstruction-based methods with visualized time series signals and anomaly scores. We then propose AER (Auto-encoder with Regression), a joint model that combines a vanilla auto-encoder and an LSTM regressor to incorporate the successes and address the limitations of each method. Our model can produce bi-directional predictions while simultaneously reconstructing the original time series by optimizing a joint objective function. Furthermore, we propose several ways of combining the prediction and reconstruction errors through a series of ablation studies. Finally, we compare the performance of the AER architecture against two prediction-based methods and three reconstruction-based methods on 12 well-known univariate time series datasets from NASA, Yahoo, Numenta, and UCR. The results show that AER has the highest averaged F1 score across all datasets (a 23.5% improvement compared to ARIMA) while retaining a runtime similar to its vanilla auto-encoder and regressor components. Our model is available in Orion1, an opensource benchmarking tool for time series anomaly detection.

The detection of anomalies in time series data is a critical task with many monitoring applications. Existing systems often fail to encompass an end-to-end detection process, to facilitate comparative analysis of various anomaly detection methods, or to incorporate human knowledge to refine output. This precludes current methods from being used in real-world settings by practitioners who are not ML experts. In this paper, we introduce Sintel, a machine learning framework for end-to-end time series tasks such as anomaly detection. The framework uses state-of-the-art approaches to support all steps of the anomaly detection process. Sintel logs the entire anomaly detection journey, providing detailed documentation of anomalies over time. It enables users to analyze signals, compare methods, and investigate anomalies through an interactive visualization tool, where they can annotate, modify, create, and remove events. Using these annotations, the framework leverages human knowledge to improve the anomaly detection pipeline. We demonstrate the usability, efficiency, and effectiveness of Sintel through a series of experiments on three public time series datasets, and through a real-world use case with spacecraft experts. Sintel's framework, code, and datasets are open-sourced at https://github.com/sintel-dev/

Detecting anomalies in time-varying multivariate data is crucial in various industries for the predictive maintenance of equipment. Numerous machine learning (ML) algorithms have been proposed to support automated anomaly identification. However, a significant amount of human knowledge is still required to interpret, analyze, and calibrate the results of automated analysis. This paper investigates current practices used to detect and investigate anomalies in time series data in industrial contexts and identifies corresponding needs. Through iterative design and working with nine experts from two industry domains (aerospace and energy), we characterize six design elements required for a successful visualization system that supports effective detection, investigation, and annotation of time series anomalies. We summarize an ideal human-AI collaboration workflow that streamlines the process and supports efficient and collaborative analysis. We introduce MTV (MultivariateTime SeriesVisualization), a visual analytics system to support such workflow. The system incorporates a set of novel visualization and interaction designs to support multi-faceted time series exploration, efficient in-situ anomaly annotation, and insight communication. Two user studies, one with 6 spacecraft experts (with routine anomaly analysis tasks) and one with 25 general end-users (without such tasks), are conducted to demonstrate the effectiveness and usefulness of MTV.

Machine learning (ML) is increasingly applied to Electronic Health Records (EHRs) to solve clinical prediction tasks. Although many ML models perform promisingly, issues with model transparency and interpretability limit their adoption in clinical practice. Directly using existing explainable ML techniques in clinical settings can be challenging. Through literature surveys and collaborations with six clinicians with an average of 17 years of clinical experience, we identified three key challenges, including clinicians' unfamiliarity with ML features, lack of contextual information, and the need for cohort-level evidence. Following an iterative design process, we further designed and developed VBridge, a visual analytics tool that seamlessly incorporates ML explanations into clinicians' decision-making workflow. The system includes a novel hierarchical display of contribution-based feature explanations and enriched interactions that connect the dots between ML features, explanations, and data. We demonstrated the effectiveness of VBridge through two case studies and expert interviews with four clinicians, showing that visually associating model explanations with patients' situational records can help clinicians better interpret and use model predictions when making clinician decisions. We further derived a list of design implications for developing future explainable ML tools to support clinical decision-making.

Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts - who often have no expertise in ML or data science - are asked to use ML predictions to make high-stakes decisions. Multiple ML usability challenges can appear as result, such as lack of user trust in the model, inability to reconcile human-ML disagreement, and ethical concerns about oversimplification of complex problems to a single algorithm output. In this paper, we investigate the ML usability challenges that present in the domain of child welfare screening through a series of collaborations with child welfare screeners. Following the iterative design process between the ML scientists, visualization researchers, and domain experts (child screeners), we first identified four key ML challenges and honed in on one promising explainable ML technique to address them (local factor contributions). Then we implemented and evaluated our visual analytics tool, Sibyl, to increase the interpretability and interactivity of local factor contributions. The effectiveness of our tool is demonstrated by two formal user studies with 12 non-expert participants and 13 expert participants respectively. Valuable feedback was collected, from which we composed a list of design implications as a useful guideline for researchers who aim to develop an interpretable and interactive visualization tool for ML prediction models deployed for child welfare screeners and other similar domain experts.

In many real life situations, including job and loan applications, gatekeepers must make justified and fair real-time decisions about a person’s fitness for a particular opportunity. In this paper, we aim to accomplish approximate group fairness in an online stochastic decision-making process, where the fairness metric we consider is equalized odds. Our work follows from the classical learning-from-experts scheme, assuming a finite set of classifiers (human experts, rules, options, etc) that cannot be modified. We run separate instances of the algorithm for each label class as well as sensitive groups, where the probability of choosing each instance is optimized for both fairness and regret. Our theoretical results show that approximately equalized odds can be achieved without sacrificing much regret. We also demonstrate the performance of the algorithm on real data sets commonly used by the fairness community.

While the open-source software development model has led to successful large-scale collaborations in building software systems, data science projects are frequently developed by individuals or small teams. We describe challenges to scaling data science collaborations and present a conceptual framework and ML programming model to address them. We instantiate these ideas in Ballet, the first lightweight framework for collaborative, open-source data science through a focus on feature engineering, and an accompanying cloud-based development environment. Using our framework, collaborators incrementally propose feature definitions to a repository which are each subjected to software and ML performance validation and can be automatically merged into an executable feature engineering pipeline. We leverage Ballet to conduct a case study analysis of an income prediction problem with 27 collaborators, and discuss implications for future designers of collaborative projects.

Time series anomalies can offer information relevant to critical situations facing various fields, from finance and aerospace to the IT, security, and medical domains. However, detecting anomalies in time series data is particularly challenging due to the vague definition o f a nomalies and said data's frequent lack of labels and highly complex temporal correlations. Current state-of-the-art unsupervised machine learning methods for anomaly detection suffer from scalability and portability issues, and may have high false positive rates. In this paper, we propose TadGAN, an unsupervised anomaly detection approach built on Generative Adversarial Networks (GANs). To capture the temporal correlations of time series distributions, we use LSTM Recurrent Neural Networks as base models for Generators and Critics. TadGAN is trained with cycle consistency loss to allow for effective time-series data reconstruction. We further propose several novel methods to compute reconstruction errors, as well as different approaches to combine reconstruction errors and Critic outputs to compute anomaly scores. To demonstrate the performance and generalizability of our approach, we test several anomaly scoring techniques and report the best-suited one. We compare our approach to 8 baseline anomaly detection methods on 11 datasets from multiple reputable sources such as NASA, Yahoo, Numenta, Amazon, and Twitter. The results show that our approach can effectively detect anomalies and outperform baseline methods in most cases (6 out of 11). Notably, our method has the highest averaged F1 score across all the datasets. Our code is open source and is available as a benchmarking tool.

An estimated 180 papers focusing on deep learning and EHR were published between 2010 and 2018. Despite the common workflow structure appearing in these publications, no trusted and verified software framework exists, forcing researchers to arduously repeat previous work. In this paper, we propose Cardea, an extensible open-source automated machine learning framework encapsulating common prediction problems in the health domain and allows users to build predictive models with their own data. This system relies on two components: Fast Healthcare Interoperability Resources (FHIR) - a standardized data structure for electronic health systems - and several AU TOML frameworks for automated feature engineering, model selection, and tuning. We augment these components with an adaptive data assembler and comprehensive data- and modelauditing capabilities. We demonstrate our framework via 5 prediction tasks on MIMIC-III and KAGGLE datasets, which highlight Cardea's human competitiveness, flexibility in problem definition, extensive feature generation capability, adaptable automatic data assembler, and its usability.

As machine learning is applied more widely, data scientists often struggle to find or create end-to-end machine learning systems for specific tasks. The proliferation of libraries and frameworks and the complexity of the tasks have led to the emergence of "pipeline jungles" - brittle, ad hoc ML systems. To address these problems, we introduce the Machine Learning Bazaar, a new framework for developing machine learning and automated machine learning software systems. First, we introduce ML primitives, a unified API and specification for data processing and ML components from different software libraries. Next, we compose primitives into usable ML pipelines, abstracting away glue code, data flow, and data storage. We further pair these pipelines with a hierarchy of AutoML strategies - Bayesian optimization and bandit learning. We use these components to create a general-purpose, multi-task, end-to-end AutoML system that provides solutions to a variety of data modalities (image, text, graph, tabular, relational, etc.) and problem types (classification, regression, anomaly detection, graph matching, etc.). We demonstrate 5 real-world use cases and 2 case studies of our approach. Finally, we present an evaluation suite of 456 real-world ML tasks and describe the characteristics of 2.5 million pipelines searched over this task suite.

A vine copula model is a flexible high-dimensional dependence model which uses only bivariate building blocks. However, the number of possible configurations of a vine copula grows exponentially as the number of variables increases, making model selection a major challenge in development. In this work, we formulate a vine structure learning problem with both vector and reinforcement learning representation. We use neural network to find the embeddings for the best possible vine model and generate a structure. Throughout experiments on synthetic and real-world datasets, we show that our proposed approach fits the data better in terms of loglikelihood. Moreover, we demonstrate that the model is able to generate high-quality samples in a variety of applications, making it a good candidate for synthetic data generation.

To relieve the pain of manually selecting machine learning algorithms and tuning hyperparameters, automated machine learning (AutoML) methods have been developed to automatically search for good models. Due to the huge model search space, it is impossible to try all models. Users tend to distrust automatic results and increase the search budget as much as they can, thereby undermining the efficiency of AutoML. To address these issues, we design and implement ATMSeer, an interactive visualization tool that supports users in refining the search space of AutoML and in analyzing the results. To guide the design of ATMSeer, we derive a workflow of using AutoML based on interviews with machine learning experts. A multi-granularity visualization is proposed to enable users to monitor the AutoML process, analyze the searched models, and refine the search space in real time. We demonstrate the utility and usability of ATMSeer through two case studies, expert interviews, and a user study with 13 end users.

Content of text data are often influenced by contextual factors which often evolve over time (e.g., content of social media are often influenced by topics covered in the major news streams).Existing language models do not consider the influence of such related evolving topics, and thus are not optimal.In this paper, we propose to incorporate such topicalinfluence into a language model to both improve its accuracy and enable cross-stream analysis of topical influences.Specifically, we propose a novel language model called Topical Influence Language Model (TILM), which is a novel extension of a neural language model to capture the influences on the contents in one text stream by the evolving topics in another related (or possibly same) text stream.Experimental results on six different text stream data comprised of conference paper titles show that the incorporation of evolving topical influence into a language model is beneficial and TILM outperforms multiple baselines in a challenging task of text forecasting.In addition to serving as a language model, TILM further enables interesting analysis of topical influence among multiple text streams.

Modeling the probability distribution of rows in tabular data and generating realistic synthetic data is a non-trivial task. Tabular data usually contains a mix of discrete and continuous columns. Continuous columns may have multiple modes whereas discrete columns are sometimes imbalanced making the modeling difficult. Existing statistical and deep neural network models fail to properly model this type of data. We design CTGAN, which uses a conditional generative adversarial network to address these challenges. To aid in a fair and thorough comparison, we design a benchmark with 7 simulated and 8 real datasets and several Bayesian network baselines. CTGAN outperforms Bayesian methods on most of the real datasets whereas other deep learning methods could not.

We present an automated learning system that continuously gathers domain data from open repositories, develops a deep learning model, uses the model to make detections, publishes unreported malicious domains, leverages threat intelligence to label the detected domains, and periodically updates the detection models. The results presented in this paper show that the system not only extends the detection coverage of threat intelligence feeds, but also that it reduces the delay in detection. We also leverage deep learning models to generate new, unregistered domains that are likely to be used by attackers in the future.

Many businesses ("") across industries hire technology service providers ( "providers" ) to develop and maintain software applications. The provider in turn hires a team, distributed across the globe and filling out reports focused on what is happening locally. This results in hundreds of reports covering the provider ’s portfolio of projects, each with dozens of fields. The task of sorting through and making decisions based on the ever-growing volume of data inevitably falls on the shoulders of a project manager.

In this paper, we present Auto-Tuned Models, or ATM, a distributed, collaborative, scalable system for automated machine learning. Users of ATM can simply upload a dataset, choose a subset of modeling methods, and choose to use ATM's hybrid Bayesian and multi-armed bandit optimization system. The distributed system works in a load-balanced fashion to quickly deliver results in the form of ready-to-predict models, confusion matrices, cross-validation results, and training timings. By automating hyperparameter tuning and model selection, ATM returns the emphasis of the machine learning workflow to its most irreducible part: feature engineering. We demonstrate the usefulness of ATM on 420 datasets from OpenML and train over 3 million classifiers. Our initial results show ATM can beat human-generated solutions for 30% of the datasets, and can do so in 1/100th of the time.

In this paper, we describe a system for sequential hyperparameter optimization that scales to work with complex pipelines and large datasets. Currently, the state-of-the-art in hyperparameter optimization improves on randomized and grid search by using sequential Bayesian optimization to explore the space of hyperparameters in a more informed way. These methods, however, are not scalable, as the entire data science pipeline still must be evaluated on all the data. By designing a sub sampling based approach to estimate pipeline performance, along with a distributed evaluation system, we provide a scalable solution, which we illustrate using complex image and text data pipelines. For three pipelines, we show that we are able to gain similar performance improvements, but by computing on substantially less data.

We present AnonML, a system for privacy-preserving model generation over a network of peers. Our goal is to allow a group of users to combine enough data to generate useful machine learning models without revealing private information. In our setting, each peer has a single row of featurized data according to a shared schema, and an aggregator would like to train a binary classification model on the union of all peers' data. Our system horizontally and vertically partitions the set of all peers' data and assembles a differentially-private histogram for each partition. An ensemble classifier can then be trained on the set of noisy partitions. AnonML can be used with or without differentially private data perturbation. Without perturbation, the resulting classifiers achieve performance competitive with centrally-generated models. With local differential privacy, a strong theoretical guarantee, AnonML is capable of producing useful models for practical prediction problems.

Feature engineering is a critical step in a successful data science pipeline. This step, in which raw variables are transformed into features ready for inclusion in a machine learning model, can be one of the most challenging aspects of a data science effort. We propose a new paradigm for feature engineering in a collaborative framework and instantiate this idea in a platform, FeatureHub. In our approach, independent data scientists collaborate on a feature engineering task, viewing and discussing each others' features in real-time. Feature engineering source code created by independent data scientists is then integrated into a single predictive machine learning model. Our platform includes an automated machine learning backend which abstracts model training, selection, and tuning, allowing users to focus on feature engineering while still receiving immediate feedback on the performance of their features. We use a tightly-integrated forum, native feature discovery APIs, and targeted compensation mechanisms to facilitate and incentivize collaboration among data scientists. This approach can reduce the redundancy from independent or competitive data scientists while decreasing time to task completion. In experimental results, automatically generated models using crowdsourced features show performance within 0.03 or 0.05 points of winning submissions, with minimal human oversight.

In this paper, we introduce "prediction engineering" as a formal step in the predictive modeling process. We define a generalizable 3 part framework - Label, Segment, Featurize (L-S-F) - to address the growing demand for predictive models. The framework provides abstractions for data scientists to customize the process to unique prediction problems. We describe how to apply the L-S-F framework to characteristic problems in 2 domains and demonstrate an implementation over 5 unique prediction problems defined on a dataset of crowdfunding projects from DonorsChoose.org. The results demonstrate how the L-S-F framework complements existing tools to allow us to rapidly build and evaluate 26 distinct predictive models. L-S-F enables development of models that provide value to all parties involved (donors, teachers, and people running the platform).

The goal of this paper is to build a system that automatically creates synthetic data to enable data science endeavors. To achieve this, we present the Synthetic Data Vault (SDV), a system that builds generative models of relational databases. We are able to sample from the model and create synthetic data, hence the name SDV. When implementing the SDV, we also developed an algorithm that computes statistics at the intersection of related database tables. We then used a state-of-the-art multivariate modeling approach to model this data. The SDV iterates through all possible relations, ultimately creating a model for the entire database. Once this model is computed, the same relational information allows the SDV to synthesize data by sampling from any part of the database. After building the SDV, we used it to generate synthetic data for five different publicly available datasets. We then published these datasets, and asked data scientists to develop predictive models for them as part of a crowdsourced experiment. By analyzing the outcomes, we show that synthetic data can successfully replace original data for data science. Our analysis indicates that there is no significant difference in the work produced by data scientists who used synthetic data as opposed to real data. We conclude that the SDV is a viable solution for synthetic data generation.

In this paper, we designed a formal language, called Trane, for describing prediction problems over relational datasets, implemented a system that allows data scientists to specify problems in that language. We show that this language is able to describe several prediction problems and even the ones on KAGGLE-a data science competition website. We express 29 different KAGGLE problems in this language. We designed an interpreter, which translates input from the user, specified in this language, into a series of transformation and aggregation operations to apply to a dataset in order to generate labels that can be used to train a supervised machine learning classifier. Using a smaller subset of this language, we developed a system to automatically enumerate, interpret and solve prediction problems. We tested this system on the Walmart Store Sales Forecasting dataset found on KAGGLE, enumerated 1077 prediction problems and built models that attempted to solve them, for which we produced 235 AUC scores. Considering that only one out of those 1077 problems was the focus of a 2.5 month long competition on KAGGLE, we expect this system to deliver a thousandfold increase in data scientist's productivity.

Physiological signals such as blood pressure might contain key information to predict a medical condition, but are challenging to mine. Wavelets possess the ability to unveil location-specific features within signals but there exists no principled method to choose the optimal scales and time shifts. We present a scalable, robust system to find the best wavelet parameters using Gaussian processes (GPs). We demonstrate our system by assessing wavelets as predictors for the occurrence of acute hypotensive episodes (AHEs) using over 1 billion blood pressure beats. We obtain an AUROC of 0.79 with wavelet features only, and the false positive rate when the true positive rate is fixed at 0.90 is reduced by 14% when the wavelet feature is used in conjunction with other statistical features. Furthermore, the use of GPs reduces the selection effort by a factor of 3 compared with a naive grid search.

In this paper, we present the concept of data science foundry for data from Massive Open Online Courses. In the foundry we present a series of software modules that transform the data into different representations. Ultimately, each online learner is represented using a set of variables that capture his/her online behavior. These variables are captured longitudinally over an interval. Using this representation we then build a predictive analytics stack that is able to predict online learners behavior as the course progresses in real time. To demonstrate the efficacy of the foundry, we attempt to solve an important prediction problem for Massive Open Online Courses (MOOCs): who is likely to stopout? Across a multitude of courses, with our complex per-student behavioral variables, we achieve a predictive accuracy of 0.7 AUCROC and higher for a one-week-ahead prediction problem. For a two-to-three-weeks-ahead prediction problem, we are able to achieve 0.6 AUCROC. We validate, via transfer learning, that these predictive models can be used in real time. We also demonstrate that we can protect the models using privacy-preserving mechanisms without losing any predictive accuracy.

In this paper, we develop the Data Science Machine, which is able to derive predictive models from raw data automatically. To achieve this automation, we first propose and develop the Deep Feature Synthesis algorithm for automatically generating features for relational datasets. The algorithm follows relationships in the data to a base field, and then sequentially applies mathematical functions along that path to create the final feature. Second, we implement a generalizable machine learning pipeline and tune it using a novel Gaussian Copula process based approach. We entered the Data Science Machine in 3 data science competitions that featured 906 other data science teams. Our approach beats 615 teams in these data science competitions. In 2 of the 3 competitions we beat a majority of competitors, and in the third, we achieved 94% of the best competitor's score. In the best case, with an ongoing competition, we beat 85.6% of the teams and achieved 95.7% of the top submissions score.

We introduce Evolutionary Feature Synthesis (EFS), a regression method that generates readable, nonlinear models of small to medium size datasets in seconds. EFS is, to the best of our knowledge, the fastest regression tool based on evolutionary computation reported to date. The feature search involved in the proposed method is composed of two main steps: feature composition and feature subset selection. EFS adopts a bottom-up feature composition strategy that eliminates the need for a symbolic representation of the features and exploits the variable selection process involved in pathwise regularized linear regression to perform the feature subset selection step. The result is a regression method that is competitive against neural networks, and outperforms both linear methods and Multiple Regression Genetic Programming, up to now the best regression tool based on evolutionary computation.

We examine the process of engineering features for developing models that improve our understanding of learners' online behavior in MOOCs. Because feature engineering relies so heavily on human insight, we engage the crowd for feature proposals and guidance on how to operationalize them. When we examined our crowd-sourced features in the context of predicting stopout, not only were they impressively nuanced, but they also integrated more than one interaction mode between the learner and platform and described how the learner was relatively performing.

A daunting challenge faced by program performance autotuning is input sensitivity, where the best autotuned configuration may vary with different input sets. This paper presents a novel two-level input learning algorithm to tackle the challenge for an important class of autotuning problems, algorithmic autotuning. The new approach uses a two-level input clustering method to automatically refine input grouping, feature selection, and classifier construction. Its design solves a series of open issues that are particularly essential to algorithmic autotuning, including the enormous optimization space, complex influence by deep input features, high cost in feature extraction, and variable accuracy of algorithmic choices. Experimental results show that the new solution yields up to a 3x speedup over using a single configuration for all inputs, and a 34x speedup over a traditional one-level method for addressing input sensitivity in program optimizations.

Program autotuning has been shown to achieve better or more portable performance in a number of domains. However, autotuners themselves are rarely portable between projects, for a number of reasons: using a domain-informed search space representation is critical to achieving good results; search spaces can be intractably large and require advanced machine learning techniques; and the landscape of search spaces can vary greatly between different problems, sometimes requiring domain specific search techniques to explore efficiently.

We present DevoII, an improved cell-based developmental model for wind farm layout optimization. To address the shortcomings of discretization, DevoII's gene regulatory networks control cells that act in a continuous rather than discretized grid space. We find that DevoII is competitive, and in some cases, superior with respect to state-of-the-art global, stochastic search approaches when a suite of algorithms is evaluated on different wind scenarios. The modularity of the genetic regulatory network computational paradigm in terms of isolating its search algorithm, the regulatory network simulation and the cell simulation, allowed this improvement to largely focus upon cell simulation. This indicates a robustness property of the paradigm's design. As well, wflo highlights how developmental models can be considered more efficient than other optimization methods because of their "optimize once, use-many" adaptability.

We propose graphical models as a new means of understanding genetic programming dynamics. Herein, we describe how to build an unbiased graphical model from a population of genetic programming trees. Graphical models both express information about the conditional dependency relations among a set of random variables and they support probabilistic inference regarding the likelihood of a random variable's outcome. We focus on the former information: by their structure, graphical models reveal structural dependencies between the nodes of genetic programming trees. We identify graphical model properties of potential interest in this regard - edge quantity and dependency among nodes expressed in terms of family relations. Using a simple symbolic regression problem we generate a graphical model of the population each generation. Then we interpret the graphical models with respect to conventional knowledge about the influence of subtree crossover and mutation upon tree structure.

In this paper we examine the challenge of producing ensembles of regression models for large datasets. We generate numerous regression models by concurrently executing multiple independent instances of a genetic programming learner. Each instance may be configured with different parameters and a different subset of the training data. Several strategies for fusing predictions from multiple regression models are compared. To overcome the small memory size of each instance, we challenge our framework to learn from small subsets of training data and yet produce a prediction of competitive quality after fusion. This decreases the running time of learning which produces models of good quality in a timely fashion. Finally, we examine the quality of fused predictions over the progress of the computation.

Optimizing a wind farm layout is a very complex problem that involves many local and global constraints such as inter-turbine wind interference or terrain peculiarities. Existing methods are either inefficient or, when efficient, take days or weeks to execute. Solutions are contextually sensitive to the specific values of the problem variables; when one value is modified, the algorithm has to be re-run from scratch. This paper proposes the use of a developmental model to generate farm layouts. Controlled by a gene regulatory network, virtual cells have to populate a simulated environment that represents the wind farm. When the cells' behavior is learned, this approach has the advantage that it is re-usable in different contexts; the same initial cell is responsive to a variety of environments and the layout generation takes few minutes instead of days.

This work provides a system modeling approach to describe a virtualized data center environment running a business process. This model allows the collection of simulation data at different workload rates that can be used as a dataset for reasoning about quality of service and energy efficiency issues related to the process. The model overcomes the issue of obtaining real data from data center administrators and collecting relevant information needed for studying the process behavior. The model is flexible and can be used to model data centers of different dimensions and characteristics. It can also be used for "what-if" analysis about the system configuration, predicting the outcome of a modification over energy efficiency and quality of service.

The design of a wind farm involves several complex optimization problems. We consider the multi-objective optimization problem of maximizing the energy output under the consideration of wake effects and minimizing the cost of the turbines and land area used for the wind farm. We present an efficient particle swarm optimization algorithm that computes a set of trade-off solutions for the given task. Our algorithm can be easily integrated into the layout process for developing wind farms and gives designers new insights into the trade-off between energy output and land area.

We present an improved estimation of distribution (EDA) genetic programming (GP) algorithm which does not rely upon a prototype tree. Instead of using a prototype tree, Operator-Free Genetic Programming learns the distribution of ancestor node chains, "n-grams", in a fit fraction of each generation's population. It then uses this information, via sampling, to create trees for the next generation. Ancestral n-grams are used because an analysis of a GP run conducted by learning depth first graphical models for each generation indicated their emergence as substructures of conditional dependence. We are able to show that our algorithm, without an operator and a prototype tree, achieves, on average, performance close to conventional tree based crossover GP on the problem we study. Our approach sets a direction for pattern-based EDA GP which off ers better tractability and improvements over GP with operators or EDAs using prototype trees.

We have acquired panelist data that provides hedonic (liking) ratings for a set of 40 flavors each composed of the same 7 ingredients at different concentration levels. Our goal is to use this data and predict other flavors, composed of the same ingredients in new combinations, which the panelist will like. We describe how we first employ Pareto-Genetic Programming (GP) to generate a surrogate for the human panelist from the 40 observations. This surrogate, in fact an ensemble of GP symbolic regression models, can predict liking scores for flavors outside the observations and provide a confidence in the prediction. We then employ a multi-objective particle swarm optimization (MOPSO) to design a well and consistently liked flavor suite for a panelist. The MOPSO identifies flavors that are well liked, i.e., high liking score, and consistently-liked, i.e., of maximum confidence. Further, we generate flavors that are well and consistently liked by a cluster of panelists, by giving the MOPSO slightly different objectives.

This paper presents a novel approach for knowledge mining from a sparse and repeated measures dataset. Genetic programming based symbolic regression is employed to generate multiple models that provide alternate explanations of the data. This set of models, called an ensemble, is generated for each of the repeated measures separately. These multiple ensembles are then utilized to generate information about, (a) which variables are important in each ensemble, (b) cluster the ensembles into different groups that have similar variables that drive their response variable, and (c) measure sensitivity of response with respect to the important variables. We apply our methodology to a sensory science dataset. The data contains hedonic evaluations (liking scores), assigned by a diverse set of human testers, for a small set of flavors composed from seven ingredients. Our approach: (1) identifies the important ingredients that drive the liking score of a panelist and (2) segments the panelists into groups that are driven by the same ingredient, and (3) enables flavor scientists to perform the sensitivity analysis of liking scores relative to changes in the levels of important ingredients.

Particle swarm optimization (PSO) has been extensively used in recent years for the optimization of nonlinear optimization problems. Two of the most popular variants of PSO are PSO-W (PSO with inertia weight) and PSO-C (PSO with constriction factor). Typically particles in swarm use information from global best performing particle, gbest and their own personal best, pbest. Recently, studies have focused on incorporating influences of other particles other than gbest. In this paper, we develop a methodology to share information between two particles using a Laplacian operator designed from Laplace probability density function. The properties of this operator are analyzed. Two particles share their positional information in the search space and a new particle is formed. The particle, called as Laplacian particle, replaces the worst performing particle in the swarm. Using this new operator, this paper introduces two algorithms namely Laplace Crossover PSO with inertia weight (LXPSO-W) and Laplace Crossover PSO with constriction factor (LXPSO-C). The performance of the newly designed algorithms is evaluated with respect to PSO-W and PSO-C using 15 benchmark test problems. The empirical results show that the new approach improves performance measured in terms of efficiency, reliability and robustness.

In this paper, an adaptive learning algorithm is developed using particle swarm to identify and mitigate the non-line of sight (NLOS) signals in ranging measurements. Training data is generated using the IEEE 802.15.4a UWB channel model for different conditions. Multiple metrics derived from this data are fused to identify the NLOS signals. Specifically, kurtosis, mean excess delay and root mean square delay are used as metrics for fusion. The fusion strategy is derived using PSO, considering the correlation between multiple classifiers. We compare the fusion methodology achieved by PSO for the correlated data set to the likelihood ratio based fusion methodology assuming independence. Further, the paper presents an NLOS mitigation approach derived using PSO. A scalar called “error mitigation ratio (EMR)” is defined. The EMR transform a NLOS measurement into an equivalent LOS measurement. The PSO identifies the EMR using the training data. Application of PSO generated EMR enhances the positioning accuracy and is demonstrated in this paper for indoor wireless channel. This mitigation approach enables us to arrive at a position for the unknown node even when one of the measurements is identified as NLOS. Finally, PSO is used for multilateration to combine measurements from three nodes. Comparisons are done with the linearized least square method.

In this paper, a novel approach is presented for ultra wide band signal design. The bottom up approach designs the signals that utilize the ultra wide band spectrum efficiently. 1st-6thderivative Gaussian pulses are linearly combined using a particle swarm optimization algorithm to form one single pulse. A binary PSO determines the order of the derivative of the pulses that are combined. The continuous PSO determines the time duration and amplitudes for different pulses in the composite pulse. The power spectral density (PSD) of the resultant pulse conforms to the FCC spectral mask and effectively exploits the allowable bandwidth and power. The particle swarm optimization algorithm achieves multiple orthogonal pulses. The newly designed pulses achieve higher spectral efficiencies which is shown theoretically and in simulations. The proposed method presents a flexible and effective way for generating UWB pulses that satisfy the FCC mask. The method can be generalized to design UWB pulses for any given spectral mask.

In this paper threshold design and hierarchy management of serial sensor networks employed for distributed detection is accomplished using a hybrid of ant colony optimization and particle swarm optimization. The particle swarm optimization determines the optimal thresholds, decision rules for the sensors. The ant colony optimization algorithm determines the hierarchy of sensor decision communication, affecting the accuracy. The problem of hierarchy management is known as ldquowho reports to whom?rdquo problem in sensor networks. The new algorithm is tested on a suite of 10 heterogeneous sensors. Probabilistic measures including probability of error and Bayesian risk are adopted to evaluate the performance of the sensor network. The new sensor management methodology is compared to (a) static hierarchy network, (b) a network with the best sensor at the top of the hierarchy and (c) incrementally best hierarchy. Results show 40% performance improvements in terms of Bayesian risk value.

A new particle swarm optimization (PSO) algorithm that is more effective for discrete, multi-valued optimization problems is presented. The new algorithm is probabilistically driven since it uses probabilistic transition rules to move from one discrete value to another in the search for an optimum solution. Properties of the binary discrete particle swarms are discussed. The new algorithm for discrete multi-values is designed with the similar properties. The algorithm is tested on a suite of benchmarks and comparisons are made between the binary PSO and the new discrete PSO implemented for ternary, quaternary systems. The results show that the new algorithm's performance is close and even slightly better than the original discrete, binary PSO designed by Kennedy and Eberhart. The algorithm can be used in any real world optimization problems, which have a discrete, bounded field

This paper presents an evolutionary approach to the sensor management of a biometric security system that improves robustness. Multiple biometrics are fused at the decision level to support a system that can meet more challenging and varying accuracy requirements as well as address user needs such as ease of use and universality better than a single biometric system or static multimodal biometric system. The decision fusion rules are adapted to meet the varying system needs by particle swarm optimization, which is an evolutionary algorithm. This paper focuses on the details of this new sensor management algorithm and demonstrates its effectiveness. The evolutionary nature of adaptive, multimodal biometric management (AMBM) allows it to react in pseudoreal time to changing security needs as well as user needs. Error weights are modified to reflect the security and user needs of the system. The AMBM algorithm selects the fusion rule and sensor operating points to optimize system performance in terms of accuracy.

This paper presents a modification of the particle swarm optimization algorithm (PSO) intended to combat the problem of premature convergence observed in many applications of PSO. The proposed new algorithm moves particles towards nearby particles of higher fitness, instead of attracting each particle towards just the best position discovered so far by any particle. This is accomplished by using the ratio of the relative fitness and the distance of other particles to determine the direction in which each component of the particle position needs to be changed. The resulting algorithm (FDR-PSO) is shown to perform significantly better than the original PSO algorithm and some of its variants, on many different benchmark optimization problems. Empirical examination of the evolution of the particles demonstrates that the convergence of the algorithm does not occur at an early phase of particle evolution, unlike PSO. Avoiding premature convergence allows FDR-PSO to continue search for global optima in difficult multimodal optimization problems.