Abstract The F-measure, also known as the F1-score, is widely used to assess the performance of classification algorithms. However, some researchers find it lacking in intuitive interpretation, questioning the appropriateness of combining two aspects of performance as conceptually distinct as precision and recall, and also questioning whether the harmonic mean is the best way to combine them. To ease this concern, we describe a simple transformation of the F-measure, which we call $$F^*$$ F ∗ (F-star), which has an immediate practical interpretation.

Financial applications of data science provide a perfect illustration of the power of the shift from subjective decision-making to data- and evidence-driven decision-making. In the space of some fifty years, an entire sector of industry has been totally revolutionised. Such applications come in three broad areas: actuarial and insurance, consumer banking, and investment banking. Actuarial and insurance work was one of the earliest adopters of data science ideas, dating from long before the term had been coined, and even before the computer had been invented. But these areas have fallen behind the latest advances in data science technology - which means there is considerable potential for applying modern data analytic ideas. Consumer banking has been described as one the first and major success stories of the data revolution. Dating from the 1960s, when the first credit cards were launched, techniques for analysing the massive data sets of consumer financial transactions have driven much of the development of data mining and data science ideas. But new model types, and new sources of data, are leading to a rich opportunity for significant developments. In investment banking the "efficient market hypothesis" of classic economics says that it is impossible to predict the financial markets. But this is false - though very nearly true. That means that there is an opportunity to use advanced data analytic methods to exploit the tiny gap between conventional theory and what actually happens. Other data science issues, such as data quality, ethics, and security, along with the need to understand the limitations of models, become particularly pointed in the context of financial applications.

Sensor networks can be naturally represented as graphical models, where the edge set encodes the presence of sparsity in the correlation structure between sensors. Such graphical representations can be valuable for information mining purposes as well as for optimizing bandwidth and battery usage with minimal loss of estimation accuracy. We use a computationally efficient technique for estimating sparse graphical models which fits a sparse linear regression locally at each node of the graph via the Lasso estimator. Using a recently suggested online, temporally adaptive implementation of the Lasso, we propose an algorithm for streaming graphical model selection over sensor networks. With battery consumption minimization applications in mind, we use this algorithm as the basis of an adaptive querying scheme. We discuss implementation issues in the context of environmental monitoring using sensor networks, where the objective is short-term forecasting of local wind direction. The algorithm is tested against real UK weather data and conclusions are drawn about certain tradeoffs inherent in decentralized sensor networks data analysis.

Sensor management in information-rich and dynamic environments can be posed as a sequential action selection problem with side information. To study such problems we employ the dynamic multi-armed bandit with covariates framework. In this generalization of the multi-armed bandit, the expected rewards are time-varying linear functions of the covariate vector. The learning goal is to associate the covariate with the optimal action at each instance, essentially learning to partition the covariate space adaptively. Applications of sensor management are frequently in environments in which the precise nature of the dynamics is unknown. In such settings, the sensor manager tracks the evolving environment by observing only the covariates and the consequences of the selected actions. This creates difficulties not encountered in static problems, and changes the exploitation–exploration dilemma. We study the relationship between the different factors of the problem and provide interesting insights. The impact of the environment dynamics on the action selection problem is influenced by the covariate dimensionality. We present the surprising result that strategies that perform very little or no exploration perform surprisingly well in dynamic environments.

Data mining techniques use 'score functions' to quantify how well a model fits a given data set. Parameters are estimated by optimising the fit, as measured by the chosen score function, and model choice is guided by the size of the scores for the different models. Since different score functions summarise the fit in different ways, it is important to choose a function which matches the objectives of the data mining exercise. For predictive classification problems, a wide variety of score functions exist, including measures such as precision and recall, the F measure, misclassification rate, the area under the ROC curve (the AUC), and others. The first four of these require a 'classification threshold' to be chosen, a choice which may not be easy, or may even be impossible, especially when the classification rule is to be applied in the future. In contrast, the AUC does not require the specification of a classification threshold, but summarises performance over the range of possible threshold choices. However, unfortunately, and despite the widespread use of the AUC, it has a previously unrecognised fundamental incoherence lying at the core of its definition. This means that using the AUC can lead to poor model choice and unecessary misclassifications. The AUC is set in context, its deficiency explained and the implications illustrated - with the bottom line being that the AUC should not be used. A family of coherent alternative scores is described. The ideas are illustrated with examples from bank loans, fraud, face recognition, and health screening.

Variable selection for regression is a classical statistical problem, motivated by concerns that too many covariates invite overfitting. Existing approaches notably include a class of convex optimisation techniques, such as the Lasso algorithm. Such techniques are invariably reliant on assumptions that are unrealistic in streaming contexts, namely that the data is available off-line and the correlation structure is static. In this paper, we relax both these constraints, proposing for the first time an online implementation of the Lasso algorithm with exponential forgetting. We also optimise the model dimension and the speed of forgetting in an online manner, resulting in a fully automatic scheme. In simulations our scheme improves on recursive least squares in dynamic environments, while also featuring model discovery and changepoint detection capabilities.

Brownboost is an adaptive, continuous time boosting algorithm based on the Boost-by-Majority (BBM) algorithm. Though it has been little studied at the time of writing, it is believed that it should prove especially robust with respect to noisy data sets. This would make it a very useful boosting algorithm for real-world applications. More familiar algorithms such as Adaboost, or its successor Logitboost, are known to be especially susceptible to overfitting the training data examples. This can lead to a poor generalization error in the presence of class noise, since weak hypotheses induced at later iterations to fit the noisy examples will tend to be given undue influence in the final combined hypothesis. Brownboost allows us to specify an expected base-line error rate in advance, corresponding to our prior beliefs about the proportion of noise in the training data, and thus avoid overfitting. The original derivation of Brownboost is restricted to binary classification problems. In this paper we propose a natural multiclass extension to the basic algorithm, incorporating error-correcting output codes and a multiclass gain measure. We test two-class and multiclass versions of the algorithm on a number of real and simulated data sets with artificial class noise, and show that Brownboost consistently outperforms Adaboost in these situations.

The authors consider the question of uncertainty of detected patterns in data mining. In particular, we develop statistical tests for patterns found in continuous data, indicating the significance of these patterns in terms of the probability that they have occurred by chance. We examine the performance of these tests on patterns detected in several large data sets, including a data set describing the locations of earthquakes in California and another describing flow cytometry measurements on phytoplankton.

In this note we use examples from the literature to illustrate some poor practices in assessing the performance of supervised classification rules, and we suggest guidelines for better methodology. We also describe a new assessment criterion that is suitable for the needs of many practical problems.

Article Free Access Share on The impact of changing populations on classifier performance Authors: Mark G. Kelly Department of Mathematics, Imperial College, 180 Queen's Gate, London, SW7 2BZ, UK Department of Mathematics, Imperial College, 180 Queen's Gate, London, SW7 2BZ, UKView Profile , David J. Hand Department of Mathematics, Imperial College, 180 Queen's Gate, London, SW7 2BZ, UK Department of Mathematics, Imperial College, 180 Queen's Gate, London, SW7 2BZ, UKView Profile , Niall M. Adams Department of Mathematics, Imperial College, 180 Queen's Gate, London, SW7 2BZ, UK Department of Mathematics, Imperial College, 180 Queen's Gate, London, SW7 2BZ, UKView Profile Authors Info & Claims KDD '99: Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data miningAugust 1999 Pages 367–371https://doi.org/10.1145/312129.312285Published:01 August 1999Publication History 125citation965DownloadsMetricsTotal Citations125Total Downloads965Last 12 Months110Last 6 weeks10 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Data analysis is an interdisciplinary science. Traditionally its development has been driven by the areas of application, but nowadays its development is also stimulated by the ever-changing possibilities promised by progress in computer technology. Huge data sets and non-numerical data, such as text data, image data, and metadata, present both challenges and opportunities for modern data analysts. These in turn lead to new types of problems and require the development of new types of models. Intelligent data analysis also requires that one take proper advantage of the largely complementary abilities of humans and computers. Interactive graphics, an important tool for modern intelligent data analysis, nicely illustrates this: the production of such graphics, and the ability to manipulate them in real time, requires advanced computational facilities; but the ability to interpret them requires the capacity to synthesise possessed only by the human eye and mind. Intelligent data analysis also requires one to have a proper strategy for analysis. Analysis without strategy is surely one of the hallmarks of unintelligent data analysis. Likewise, a key to intelligent data analysis is the ability to recognise what is important in a problem—what counts and what doesn't count.

This paper outlines some of the ways in which the widespread availability of computers may be used to lead to better statistical data analysis. Attention is focused on the necessary properties of the interface between the user and that part of the program which does the actual analysis. An illustration is given using a program developed by the authors.