Classifier ensembles benefit in terms of both accuracy and size reduction from the diversity of their members. It has become popular to analyze diversity using so-called kappa-error plots. In 2013, Kuncheva presented a bound on kappa-error points that identifies regions of the diagram that can never be occupied. She also claimed that there is an unoccupied space in the vicinity of these bounds where new ensembles might be engineered. This article presents a more in-depth study of the areas that may and may not be occupied by kappa-error points. It is shown that the region that Kuncheva and some followers claimed to be available may in fact be unavailable in many practical applications. Several properties are proved that constructively characterize the contents of kappa-error optimal contingency tables describing class assignment differences between raters in multiclass problems with arbitrary class proportions. Formulae for the exact bounds of the plots are provided. Examples with equal class sizes are also analyzed to illustrate and visualize how the availability of these areas depends on the number of classes. The analysis reveals and discusses further weaknesses of the kappa statistic as a measure of model diversity. Recommendations are provided for researchers who wish to base model selection on kappa or to prepare, analyze, and interpret kappa-error plots based on the location of point clouds.

Kappa-error diagrams are a visual tool for diversity analysis of classifier ensembles. In 2013, Kuncheva presented a bound on kappa-error points that shows some areas of the diagrams that can never be occupied. She also addressed the question of availability of an unoccupied space in the vicinity of the bounds. This article sheds a new light on kappa-error diagrams. A more detailed analysis of the bounds and more properties of kappa-error diagrams are presented. In particular, they show that the area, Kuncheva and some followers claimed available, may be unavailable in many applications. The analysis reveals more weaknesses of kappa statistic as a measure of model diversity which should be kept in mind when founding model selection on kappa or preparing kappa-error diagrams, analyzing them and drawing conclusions.

Meta-learning helps us find solutions to computational intelligence (CI) challenges in automated way. Meta-learning algorithm presented in this paper is universal and may be applied to any type of CI problems. The novelty of our proposal lies in complexity controlled testing combined with very useful learning machines generators. The simplest and the best solutions are strongly preferred and are explored earlier. The learning algorithm is augmented by meta-knowledge repository which accumulates information about progress of the search through the space of candidate solutions. The approach facilitates using human experts knowledge to restrict the search space and provide goal definition, gaining meta-knowledge in an automated manner.

There are many data mining systems derived from machine learning, neural network, statistics and other fields. Most of them are dedicated to some particular algorithms or applications. Unfortunately, their architectures are still too naive to provide satisfactory background for advanced meta-learning problems. In order to efficiently perform sophisticated meta-level analysis, we need a very versatile, easily expandable system (in many independent aspects), which uniformly deals with different kinds of models and models with very complex structures of models (not only committees but also much more hierarchic models). Meta-level techniques must provide mechanisms facilitating optimization of computation time and memory consumption. This article presents requirements and their motivations for an advanced data mining system, efficient not only in model construction for given data, but also in meta-learning. Some particular solutions to significant problems are presented. The newly proposed advanced meta-learning architecture has been implemented in our new data analysis system.

With growing amount of data gathered nowadays, need for efficient data mining methodologies is getting more and more common. There is a large number of different classification algorithms, but choosing the best one for given data is still a difficult task. Thanks to different data mining contests we can gather lots of meta level information about classification problems and strategies leading to optimal (or close to optimal) solutions. One of the contests was organized in parallel with the ICAISC'06 conference held in Zakopane. We took part in it, and our model classified the test data with the highest accuracy. The process which led to the winner model was not simpletrequired multiaspect analysis of the data and different algorithms (from the point of view of suitability to the data). This article presents our road to the winner model with numerous comments on both successful and unsuccessful efforts. It also presents our model testing methodology, which always plays important role in the pursuit of accurate and well generalizing models

Classification techniques applicable to real life data are more and more often complex hybrid systems comprising feature selection. To augment their efficiency we propose two feature selection algorithms, which take advantage of a decision tree criterion. Large computational experiments have been done to test the possibilities of the two methods and to compare their results with other techniques of similar goal.

We explore some new types of committees in search of hybrid models successful in many different classification benchmarks. To provide a reliable comparison of the ensembles we restrict the task to some constant configuration of committee members for each benchmark. We were looking for new types of committees which, in such configuration, would be as much accurate and stable as possible. The paper focuses on some ideas of heterogenous committees with different ways of their members' competence estimation. Heterogenous committee members adapt in different ways and are able to solve different problems. Measuring the competence of committee members helps in making competent and accurate decisions.

A new methodology of extraction, optimization, and application of sets of logical rules is described. Neural networks are used for initial rule extraction, local or global minimization procedures for optimization, and Gaussian uncertainties of measurements are assumed during application of logical rules. Algorithms for extraction of logical rules from data with real-valued features require determination of linguistic variables or membership functions. Contest-dependent membership functions for crisp and fuzzy linguistic variables are introduced and methods of their determination described. Several neural and machine learning methods of logical rule extraction generating initial rules are described, based on constrained multilayer perceptron, networks with localized transfer functions or on separability criteria for determination of linguistic variables. A tradeoff between accuracy/simplicity is explored at the rule extraction stage and between rejection/error level at the optimization stage. Gaussian uncertainties of measurements are assumed during application of crisp logical rules, leading to "soft trapezoidal" membership functions and allowing to optimize the linguistic variables using gradient procedures. Numerous applications of this methodology to benchmark and real-life problems are reported and very simple crisp logical rules for many datasets provided.

Neural networks are used for initial determination of linguistic variables and extraction of logical rules from data. Hierarchical sets of rules for different accuracy/rejection trade-offs are obtained. Sets of logical rules are optimized by maximization of their predictive power. To avoid global minimization methods for crisp logical rules Gaussian uncertainties of inputs are assumed. Analytical formulas reproducing Monte Carlo results for such inputs are derived, leading to a "soft trapezoidal" membership functions instead of rectangular functions used for crisp logical rules. Such approach increases accuracy, gives probabilities of classification instead of the yes/no answers, and allows one to optimize sets of rules using gradient procedures. A few illustrative applications to benchmark and real life problems show the effectiveness of this approach.