The Fragile Earth Workshop is a recurring event in ACM's KDD Conference on research in knowledge discovery and data mining that gathers the research community to find and explore how data science can measure and progress climate and social issues, following the United Nations Sustainable Development Goals (SDGs) framework.

We address the contextual bandit problem for high-dimensional combinatorial action spaces involving a class of structured discrete constrained optimization problems. In this setting, key quantities within the optimization formulation, such as performance indicators contributing to the objective or constraints, must be estimated from data during the trial sequence of optimized actions. These problems frequently arise in the many operations management domains including IT resource allocation and retail assortment price optimization. We propose a novel, practical, and transparent approach based on general-purpose regression oracles, leveraging Inverse Gap Weighting (IGW) for seamless integration within an optimization framework. IGW sampling is efficiently managed by: (a) a column generation reformulation of the underlying Mixed Integer Programming (MIP) model which allows for flexible lower-level predictors, causal coherence, and efficient representation of large action spaces;(b) a diverse solution pool generation to balance the exploration-exploitation trade-off in large action spaces. To address non-smoothness in the reward function due to optimization constraints, we incorporate a risk-averse phased learning strategy. We validate our approach on a real-world auto-scaling problem in IT automation, achieving a significant reduction in cumulative regret through skillful exploration, with additional gains from risk-averse methods that effectively manage constraint violations.

The Fragile Earth Workshop is a recurring event in ACM's KDD Conference on research in knowledge discovery and data mining that gathers the research community to find and explore how data science can measure and progress climate and social issues, following the United Nations Sustainable Development Goals (SDGs) framework.

The Fragile Earth Workshop is a recurring event in ACM's KDD Conference on research in knowledge discovery and data mining that gathers the research community to find and explore how data science can measure and progress climate and social issues, fol- lowing the United Nations Sustainable Development Goals (SDGs) framework.

We address the task of probabilistic anomaly attribution in the black-box regression setting, where the goal is to compute the probability distribution of the attribution score of each input variable, given an observed anomaly. The training dataset is assumed to be unavailable. This task differs from the standard XAI (explainable AI) scenario, since we wish to explain the anomalous deviation from a black-box prediction rather than the black-box model itself.

The Fragile EarthWorkshop is a recurring event that gathers the research community to find and explore howdata science can measure and progress climate and social issues, following the framework of the United Nations Sustainable Development Goals (SDGs).

Fragile Earth 2021, our annual workshop is taking place as part of the Earth Day events at ACM's KDD 2021 Conference on research in Machine Learning and its applications. The 5th edition of Fragile Earth will bring together the research community, industry, and policymakers to develop radically new technological foundations for advancing and meeting the Sustainable Development Goals in a way that ensures equitable and inclusive progress.

Phenotyping, the process of measuring plant traits, plays a central role in plant breeding. However, traditional approaches are labor-intensive, time-consuming, costly, and error prone. Accurate, automated, high-throughput phenotyping can relieve a huge burden in the breeding pipeline. In this paper, we propose computer vision systems and approaches to annotate, detect, and count panicles (heads), a key phenotype, from aerial images of Sorghum crops. The annotation system allows the users to label panicles in Sorghum aerial images. This annotated data is used for learning by the panicle detection and counting algorithms. The proposed approaches were used with aerial imagery of 18 varieties of Sorghum crop collected at 6 different dates in the Midwestern United States. The detector has an AUC of over 0.98 and the counter has a mean absolute error of 2.66 without adapting to variety and 1.88 when using variety specific information. Our approaches are being adopted into a high-throughput phenotyping pipeline for accelerating Sorghum breeding.

Modeling how marketing actions in various channels influence or cause consumer purchase decisions is crucial for marketing decision-making. Marketing campaigns stimulate consumer awareness, interest and help drive interactions such as the browsing of product web pages, ultimately impacting an individual's purchase decision. In addition, some successful campaigns stimulate word-of-mouth and social trends among consumers, and such collective behavior of consumers result in concurrent and correlated responses over a short term. Though each consumer's response should be attributed with both the same individual's experiences and the collective factors, unobservability of most word-of-mouth events makes the estimation challenging. The authors propose a new continuous-time predictive model for time-dependent response rates of each consumer, which can incorporate both the individual and the collective factors without explicit word-of-mouth observations. The individual factor is modeled as staircase functions associated with the experienced events by each consumer, and provides a clear psychological interpretation about how marketing advertising communications impact short-term and mid-term memories of consumers. The collective factor is modeled with aggregate response frequencies for mutually-interacting groups that are automatically estimated from data. The key idea to mine the mutually-interacting groups exists in a three-step estimator, which initially performs a Poisson regression without the collective factor, then does clustering of the residual time-series in the initial regression, and finally performs another Poisson regression involving the collective factor. The proposed collective factor robustly incorporates the underlying trends even when causality from one consumer's event spikes to another consumer's response is weak. High predictive accuracy of the proposed approach is empirically validated using real-world data provided by an online retailer in Europe.

The problem of optimally managing the collections process by taxation authorities is one of prime importance, not only for the revenue it brings but also as a means to administer a fair taxing system. The analogous problem of debt collections management in the private sector, such as banks and credit card companies, is also increasingly gaining attention. With the recent successes in the applications of data analytics and optimization to various business areas, the question arises to what extent such collections processes can be improved by use of leading edge data modeling and optimization techniques. In this paper, we propose and develop a novel approach to this problem based on the framework of constrained Markov Decision Process (MDP), and report on our experience in an actual deployment of a tax collections optimization system at New York State Department of Taxation and Finance (NYS DTF).

We develop and evaluate an approach to causal modeling based on time series data, collectively referred to as "grouped graphical Granger modeling methods." Graphical Granger modeling uses graphical modeling techniques on time series data and invokes the notion of "Granger causality" to make assertions on causality among a potentially large number of time series variables through inference on time-lagged effects. The present paper proposes a novel enhancement to the graphical Granger methodology by developing and applying families of regression methods that are sensitive to group information among variables, to leverage the group structure present in the lagged temporal variables according to the time series they belong to. Additionally, we propose a new family of algorithms we call group boosting, as an improved component of grouped graphical Granger modeling over the existing regression methods with grouped variable selection in the literature (e.g group Lasso). The introduction of group boosting methods is primarily motivated by the need to deal with non-linearity in the data. We perform empirical evaluation to confirm the advantage of the grouped graphical Granger methods over the standard (non-grouped) methods, as well as that specific to the methods based on group boosting. This advantage is also demonstrated for the real world application of gene regulatory network discovery from time-course microarray data.

Attribution of climate change to causal factors has been based predominantly on simulations using physical climate models, which have inherent limitations in describing such a complex and chaotic system. We propose an alternative, data centric, approach that relies on actual measurements of climate observations and human and natural forcing factors. Specifically, we develop a novel method to infer causality from spatial-temporal data, as well as a procedure to incorporate extreme value modeling into our method in order to address the attribution of extreme climate events, such as heatwaves. Our experimental results on a real world dataset indicate that changes in temperature are not solely accounted for by solar radiance, but attributed more significantly to CO2 and other greenhouse gases. Combined with extreme value modeling, we also show that there has been a significant increase in the intensity of extreme temperatures, and that such changes in extreme temperature are also attributable to greenhouse gases. These preliminary results suggest that our approach can offer a useful alternative to the simulation-based approach to climate modeling and attribution, and provide valuable insights from a fresh perspective.

We consider the task of performing anomaly detection in highly noisy multivariate data. In many applications involving real-valued time-series data, such as physical sensor data and economic metrics, discovering changes and anomalies in the way variables depend on one another is of particular importance. Our goal is to robustly compute the “correlation anomaly” score of each variable by comparing the test data with reference data, even when some of the variables are highly correlated (and thus collinearity exists). To remove seeming dependencies introduced by noise, we focus on the most significant dependencies for each variable. We perform this “neighborhood selection” in an adaptive manner by fitting a sparse graphical Gaussian model. Instead of traditional covariance selection procedures, we solve this problem as maximum likelihood estimation of the precision matrix (inverse covariance matrix) under the L1 penalty. Then the anomaly score for each variable is computed by evaluating the distances between the fitted conditional distributions within the Markov blanket for that variable, for the (two) data sets to be compared. Using real-world data, we demonstrate that our matrix-based sparse structure learning approach successfully detects correlation anomalies under collinearities and heavy noise.

We propose a family of novel cost-sensitive boosting methods for multi-class classification by applying the theory of gradient boosting to p-norm based cost functionals. We establish theoretical guarantees including proof of convergence and convergence rates for the proposed methods. Our theoretical treatment provides interpretations for some of the existing algorithms in terms of the proposed family, including a generalization of the costing algorithm, DSE and GBSE-t, and the Average Cost method. We also experimentally evaluate the performance of our new algorithms against existing methods of cost sensitive boosting, including AdaCost, CSB2, and AdaBoost.M2 with cost-sensitive weight initialization. We show that our proposed scheme generally achieves superior results in terms of cost minimization and, with the use of higher order p-norm loss in certain cases, consistently outperforms the comparison methods, thus establishing its empirical advantage.

The need for mining causality, beyond mere statistical correlations, for real world problems has been recognized widely. Many of these applications naturally involve temporal data, which raises the challenge of how best to leverage the temporal information for causal modeling. Recently graphical modeling with the concept of “Granger causality”, based on the intuition that a cause helps predict its effects in the future, has gained attention in many domains involving time series data analysis. With the surge of interest in model selection methodologies for regression, such as the Lasso, as practical alternatives to solving structural learning of graphical models, the question arises whether and how to combine these two notions into a practically viable approach for temporal causal modeling. In this paper, we examine a host of related

Secure coprocessors have traditionally been used as a keystone of a security subsystem, eliminating the need to protect the rest of the subsystem with physical security measures. With technological advances and hardware miniaturization they have become increasingly powerful. This opens up the possibility of using them for non traditional use. This paper describes a solution for privacy preserving data sharing and mining using cryptographically secure but resource limited coprocessors. It uses memory light data mining methodologies along with a light weight database engine with federation capability, running on a coprocessor. The data to be shared resides with the enterprises that want to collaborate. This system will allow multiple enterprises, which are generally not allowed to share data, to do so solely for the purpose of detecting particular types of anomalies and for generating alerts. We also present results from experiments which demonstrate the value of such collaborations.

We introduce a new approach to the problem of link prediction for network structured domains, such as the Web, social networks, and biological networks. Our approach is based on the topological features of network structures, not on the node features. We present a novel parameterized probabilistic model of network evolution and derive an efficient incremental learning algorithm for such models, which is then used to predict links among the nodes. We show some promising experimental results using biological network data sets.

Most existing approaches to outlier detection are based on density estimation methods. There are two notable issues with these methods: one is the lack of explanation for outlier flagging decisions, and the other is the relatively high computational requirement. In this paper, we present a novel approach to outlier detection based on classification, in an attempt to address both of these issues. Our approach isbased on two key ideas. First, we present a simple reduction of outlier detection to classification, via a procedure that involves applying classification to a labeled data set containing artificially generated examples that play the role of potential outliers. Once the task has been reduced to classification, we then invoke a selective sampling mechanism based on active learning to the reduced classification problem. We empirically evaluate the proposed approach using a number of data sets, and find that our method is superior to other methods based on the same reduction to classification, but using standard classification methods. We also show that it is competitive to the state-of-the-art outlier detection methods in the literature based on density estimation, while significantly improving the computational complexity and explanatory power.

The issues of cross channel integration and customer life time value modeling are two of the most important topics surrounding customer relationship management (CRM) today. In the present paper, we describe and evaluate a novel solution that treats these two important issues in a unified framework of Markov Decision Processes (MDP). In particular, we report on the results of a joint project between IBM Research and Saks Fifth Avenue to investigate the applicability of this technology to real world problems. The business problem we use as a testbed for our evaluation is that of optimizing direct mail campaign mailings for maximization of profits in the store channel. We identify a problem common to cross-channel CRM, which we call the Cross-Channel Challenge, due to the lack of explicit linking between the marketing actions taken in one channel and the customer responses obtained in another. We provide a solution for this problem based on old and new techniques in reinforcement learning. Our in-laboratory experimental evaluation using actual customer interaction data show that as much as 7 to 8 per cent increase in the store profits can be expected, by employing a mailing policy automatically generated by our methodology. These results confirm that our approach is valid in dealing with the cross channel CRM scenarios in the real world.

Cost-sensitive learning addresses the issue of classification in the presence of varying costs associated with different types of misclassification. In this paper, we present a method for solving multi-class cost-sensitive learning problems using any binary classification algorithm. This algorithm is derived using hree key ideas: 1) iterative weighting; 2) expanding data space; and 3) gradient boosting with stochastic ensembles. We establish some theoretical guarantees concerning the performance of this method. In particular, we show that a certain variant possesses the boosting property, given a form of weak learning assumption on the component binary classifier. We also empirically evaluate the performance of the proposed method using benchmark data sets and verify that our method generally achieves better results than representative methods for cost-sensitive learning, in terms of predictive performance (cost minimization) and, in many cases, computational efficiency.

We propose and evaluate a family of methods for converting classifier learning algorithms and classification theory into cost-sensitive algorithms and theory. The proposed conversion is based on cost-proportionate weighting of the training examples, which can be realized either by feeding the weights to the classification algorithm (as often done in boosting), or by careful subsampling. We give some theoretical performance guarantees on the proposed methods, as well as empirical evidence that they are practical alternatives to existing approaches. In particular, we propose costing, a method based on cost-proportionate rejection sampling and ensemble aggregation, which achieves excellent predictive performance on two publicly available datasets, while drastically reducing the computation required by other methods.

We empirically evaluate the performance of various reinforcement learning methods in applications to sequential targeted marketing. In particular we propose and evaluate a progression of reinforcement learning methods, ranging from the "direct" or "batch" methods to "indirect" or "simulation based" methods, and those that we call "semidirect" methods that fall between them. We conduct a number of controlled experiments to evaluate the performance of these competing methods. Our results indicate that while the indirect methods can perform better in a situation in which nearly perfect modeling is possible, under the more realistic situations in which the system's modeling parameters have restricted attention, the indirect methods' performance tend to degrade. We also show that semi-direct methods are effective in reducing the amount of computation necessary to attain a given level of performance, and often result in more profitable policies.

Recently, there has been increasing interest in the issues of cost-sensitive learning and decision making in a variety of applications of data mining. A number of approaches have been developed that are effective at optimizing cost-sensitive decisions when each decision is considered in isolation. However, the issue of sequential decision making, with the goal of maximizing total benefits accrued over a period of time instead of immediate benefits, has rarely been addressed. In the present paper, we propose a novel approach to sequential decision making based on the reinforcement learning framework. Our approach attempts to learn decision rules that optimize a sequence of cost-sensitive decisions so as to maximize the total benefits accrued over time. We use the domain of targeted' marketing as a testbed for empirical evaluation of the proposed method. We conducted experiments using approximately two years of monthly promotion data derived from the well-known KDD Cup 1998 donation data set. The experimental results show that the proposed method for optimizing total accrued benefits out performs the usual targeted-marketing methodology of optimizing each promotion in isolation. We also analyze the behavior of the targeting rules that were obtained and discuss their appropriateness to the application domain.