Ordinal Classification (OC) addresses those classification tasks where the labels exhibit a natural order. Unlike nominal classification, which treats all classes as mutually exclusive and unordered, OC takes the ordinal relationship into account, producing more accurate and relevant results. This is particularly critical in applications where the magnitude of classification errors has significant consequences. Despite this, OC problems are often tackled using nominal methods, leading to suboptimal solutions. Although decision trees are among the most popular classification approaches, ordinal tree-based approaches have received less attention when compared to other classifiers. This work provides a comprehensive survey of ordinal splitting criteria, standardising the notations used in the literature to enhance clarity and consistency. Three ordinal splitting criteria, Ordinal Gini (OGini), Weighted Information Gain, and Ranking Impurity, are compared to the nominal counterparts of the first two (Gini and information gain), by incorporating them into a decision tree classifier. An extensive repository considering 45 publicly available OC datasets is presented, supporting the first experimental comparison of ordinal and nominal splitting criteria using well-known OC evaluation metrics. The results have been statistically analysed, highlighting that OGini stands out as the best ordinal splitting criterion to date, reducing the mean absolute error achieved by Gini by more than 3.02 % . To promote reproducibility, all source code developed, a detailed guide for reproducing the results, the 45 OC datasets, and the individual results for all the evaluated methodologies are provided.

Ordinal classification, where labels follow a natural order, has gained increasing attention, particularly in the deep learning community due to its relevance in tasks such as age estimation, medical grading, and quality assessment. Despite the growing number of deep ordinal classification methods, a comprehensive experimental analysis of their core ordinal components remains lacking. This work presents a systematic evaluation of deep ordinal classifiers by analysing the impact of three key modelling choices: the loss function, output layer, and labelling strategy. To analyse their effects, we adopt a unified architecture and evaluate one nominal and 19 ordinal configurations, resulting from combination of two loss functions, two output layers, and five labelling strategies. These configurations are assessed on 12 diverse ordinal image datasets using six performance metrics, including both ordinal and nominal measures. Results show that ordinal output layers consistently outperform softmax, and that soft labelling generally improves generalisation. While categorical cross-entropy achieves better average performance, especially on nominal metrics, no configuration performs best across all datasets. Statistical analyses indicate significant interactions between losses, outputs, labelling strategies, and datasets, highlighting the need to adapt methodological choices to specific tasks. These findings provide valuable guidance for designing robust deep ordinal classification models.

The use of Convolutional Neural Network (CNN) models for image classification tasks has gained significant popularity. However, the lack of interpretability in CNN models poses challenges for debugging and validation. To address this issue, various explanation methods have been developed to provide insights into CNN models. This paper focuses on the validity of these explanation methods for ordinal regression tasks, where the classes have a predefined order relationship. Different modifications are proposed for two explanation methods to exploit the ordinal relationships between classes: Grad-CAM based on Ordinal Binary Decomposition (GradOBD-CAM) and Ordinal Information Bottleneck Analysis (OIBA). The performance of these modified methods is compared to existing popular alternatives. Experimental results demonstrate that GradOBD-CAM outperforms other methods in terms of interpretability for three out of four datasets, while OIBA achieves superior performance compared to IBA. • Addressing interpretability challenges in CNN models for ordinal regression. • Modification of explanation methods for ordinal regression tasks. • Superior performance of GradOBD-CAM for interpretability in 3 out of 4 datasets. • Enhanced interpretability with OIBA compared to IBA. • Evaluation using comprehensive ordinal degradation score metrics.

dlordinal is a new Python library that unifies many recent deep ordinal classification methodologies available in the literature. Developed using PyTorch as underlying framework, it implements the top performing state-of-the-art deep learning techniques for ordinal classification problems. Ordinal approaches are designed to leverage the ordering information present in the target variable. Specifically, it includes loss functions, various output layers, dropout techniques, soft labelling methodologies, and other classification strategies, all of which are appropriately designed to incorporate the ordinal information. Furthermore, as the performance metrics to assess novel proposals in ordinal classification depend on the distance between target and predicted classes in the ordinal scale, suitable ordinal evaluation metrics are also included. dlordinal is distributed under the BSD-3-Clause license and is available at https://github.com/ayrna/dlordinal.

Time-series classification (TSC) covers the supervised learning problem where input data is provided in the form of series of values observed through repeated measurements over time, and whose objective is to predict the category to which they belong. When the class values are ordinal, classifiers that take this into account can perform better than nominal classifiers. Time-series ordinal classification (TSOC) is the field bridging this gap, yet unexplored in the literature. There are a wide range of time-series problems showing an ordered label structure, and TSC techniques that ignore the order relationship discard useful information. Hence, this article presents the first benchmarking of TSOC methodologies, exploiting the ordering of the target labels to boost the performance of current TSC state of the art. Both convolutional- and deep-learning-based methodologies (among the best performing alternatives for nominal TSC) are adapted for TSOC. For the experiments, a selection of 29 ordinal problems has been made. In this way, this article contributes to the establishment of the state of the art in TSOC. The results obtained by ordinal versions are found to be significantly better than current nominal TSC techniques in terms of ordinal performance metrics, outlining the importance of considering the ordering of the labels when dealing with this kind of problems.

Real-world classification problems may disclose different hierarchical levels where the categories are displayed in an ordinal structure. However, no specific deep learning (DL) models simultaneously learn hierarchical and ordinal constraints while improving generalization performance. To fill this gap, we propose the introduction of two novel ordinal-hierarchical DL methodologies, namely, the hierarchical cumulative link model (HCLM) and hierarchical-ordinal binary decomposition (HOBD), which are able to model the ordinal structure within different hierarchical levels of the labels. In particular, we decompose the hierarchical-ordinal problem into local and global graph paths that may encode an ordinal constraint for each hierarchical level. Thus, we frame this problem as simultaneously minimizing global and local losses. Furthermore, the ordinal constraints are set by two approaches [ordinal binary decomposition (OBD) and cumulative link model (CLM)] within each global and local function. The effectiveness of the proposed approach is measured on four real-use case datasets concerning industrial, biomedical, computer vision, and financial domains. The extracted results demonstrate a statistically significant improvement to state-of-the-art nominal, ordinal, and hierarchical approaches.

In this paper we present a novel methodology, referenced as ORFEO (Ordinal classifier and Regressor Fusion for Estimating an Ordinal categorical target), to enhance the performance in ordinal classification problems for which the latent variable is observable. ORFEO is an artificial neural network model incorporating two outputs, one for ordinal classification, using the cumulative link model, and one for regression, using a linear model. Both outputs are simultaneously optimised considering a loss function that linearly combines both classification and regression losses. The main motivation behind developing the proposed approach is to enhance the performance of a standard ordinal classifier. This improvement is facilitated by considering the regression output, which allows the model to differentiate between patterns within the same category. The ORFEO model is applied to two problems in the field of marine and ocean engineering: short-term prediction of both significant wave height and flux of energy. Both problems are addressed considering four different coastal zones of the United States of America, using 13 datasets formed by buoys measurements and reanalysis data. A comprehensive comparison against 20 methodologies, including regression and nominal/ordinal classification approaches is performed, by using diverse nominal and ordinal performance metrics. Ranks achieved indicate that ORFEO outperforms all the compared methodologies in terms of all the performance measures, demonstrating the efficacy and robustness of the proposal. Finally, a statistical analysis is conducted, concluding that there are statistically significant differences across ordinal and nominal performance metrics in favour of the proposed ORFEO model.

In this study, we present EBANO (Ensemble BAsed on uNimodal Ordinal classifiers), which is a novel ensemble approach of ordinal classifiers that includes four soft labelling approaches along with an ordinal logistic regression model. These models are integrated within the ensemble using a new aggregation methodology that automatically weights each individual classifier using a randomised search algorithm. In addition, the proposed EBANO methodology is applied to tackle short-term prediction of Significant Wave Height (SWH). Thus, we employ EBANO using a diverse set of eight datasets derived from reanalysis data and buoy-recorded SWH measurements. To approach the problem from an ordinal classification perspective, the SWH values are discretised into five ordered classes by applying hierarchical clustering. EBANO is compared with each of the individual classifiers integrated in the proposed ensemble along with a different ensemble technique termed HESCA. Both the average results and the ranks obtained show the superiority of EBANO over the compared methodologies, being more pronounced in the metrics that account for the imbalance present in the datasets considered. Finally, a statistical analysis is performed, confirming the statistical significance of the observed differences in all comparisons. This analysis underscores the effectiveness of EBANO in addressing the problem of SWH prediction, showcasing its excellence.

Predictive Maintenance (PdM) methods aim to facilitate the scheduling of maintenance work before equipment failure. In this context, detecting early faults in automated teller machines (ATMs) has become increasingly important since these machines are susceptible to various types of unpredictable failures. ATMs track execution status by generating massive event-log data that collect system messages unrelated to the failure event. Predicting machine failure based on event logs poses additional challenges, mainly in extracting features that might represent sequences of events indicating impending failures. Accordingly, feature learning approaches are currently being used in PdM, where informative features are learned automatically from minimally processed sensor data. However, a gap remains to be seen on how these approaches can be exploited for deriving relevant features from event-log-based data. To fill this gap, we present a predictive model based on a convolutional kernel (MiniROCKET and HYDRA) to extract features from the original event-log data and a linear classifier to classify the sample based on the learned features. The proposed methodology is applied to a significant real-world collected dataset. Experimental results demonstrated how one of the proposed convolutional kernels (i.e. HYDRA) exhibited the best classification performance (accuracy of 0.759 and AUC of 0.693). In addition, statistical analysis revealed that the HYDRA and MiniROCKET models significantly overcome one of the established state-of-the-art approaches in time series classification (InceptionTime), and three non-temporal ML methods from the literature. The predictive model was integrated into a container-based decision support system to support operators in the timely maintenance of ATMs.

The sanitary emergency caused by COVID-19 has compromised countries and generated a worldwide health and economic crisis. To provide support to the countries' responses, numerous lines of research have been developed. The spotlight was put on effectively and rapidly diagnosing and predicting the evolution of the pandemic, one of the most challenging problems of the past months. This work contributes to the existing literature by developing a two-step methodology to analyze the transmission rate, designing models applied to territories with similar pandemic behavior characteristics. Virus transmission is considered as bacterial growth curves to understand the spread of the virus and to make predictions about its future evolution. Hence, an analytical clustering procedure is first applied to create groups of locations where the virus transmission rate behaved similarly in the different outbreaks. A curve decomposition process based on an iterative polynomial process is then applied, obtaining meaningful forecasting features. Information of the territories belonging to the same cluster is merged to build models capable of simultaneously predicting the 14-day incidence in several locations using Evolutionary Artificial Neural Networks. The methodology is applied to Andalusia (Spain), although it is applicable to any region across the world. Individual models trained for a specific territory are carried out for comparison purposes. The results demonstrate that this methodology achieves statistically similar, or even better, performance for most of the locations. In addition to being extremely competitive, the main advantage of the proposal lies in its complexity cost reduction. The total number of parameters to be estimated is reduced up to 93.51% for the short term and 93.31% for the mid-term forecasting, respectively. Moreover, the number of required models is reduced by 73.53% and 58.82% for the short- and mid-term forecasting horizons.

Abstract Automatic classification tasks on structured data have been revolutionized by Convolutional Neural Networks (CNNs), but the focus has been on binary and nominal classification tasks. Only recently, ordinal classification (where class labels present a natural ordering) has been tackled through the framework of CNNs. Also, ordinal classification datasets commonly present a high imbalance in the number of samples of each class, making it an even harder problem. Focus should be shifted from classic classification metrics towards per-class metrics (like AUC or Sensitivity) and rank agreement metrics (like Cohen’s Kappa or Spearman’s rank correlation coefficient). We present a new CNN architecture based on the Ordinal Binary Decomposition (OBD) technique using Error-Correcting Output Codes (ECOC). We aim to show experimentally, using four different CNN architectures and two ordinal classification datasets, that the OBD+ECOC methodology significantly improves the mean results on the relevant ordinal and class-balancing metrics. The proposed method is able to outperform a nominal approach as well as already existing ordinal approaches, achieving a mean performance of $${{\,\mathrm{\textit{RMSE}}\,}}= 1.0797$$ RMSE = 1.0797 for the Retinopathy dataset and $${{\,\mathrm{\textit{RMSE}}\,}}= 1.1237$$ RMSE = 1.1237 for the Adience dataset averaged over 4 different architectures.

Activation functions lie at the core of every neural network model from shallow to deep convolutional neural networks. Their properties and characteristics shape the output range of each layer and, thus, their capabilities. Modern approaches rely mostly on a single function choice for the whole network, usually ReLU or other similar alternatives. In this work, we propose two new activation functions and analyze their properties and compare them with 17 different function proposals from recent literature on six distinct problems with different characteristics. The objective is to shed some light on their comparative performance. The results show that the proposed functions achieved better performance than the most commonly used ones.

Abstract Nowadays, decision support systems (DSSs) are widely used in several application domains, from industrial to healthcare and medicine fields. Concerning the industrial scenario, we propose a DSS oriented to the aesthetic quality control (AQC) task, which has quickly established itself as one of the most crucial challenges of Industry 4.0. Taking into account the increasing amount of data in this domain, the application of machine learning (ML) and deep learning (DL) techniques offers great opportunities to automatize the overall AQC process. State-of-the-art is mainly oriented to approach this problem with a nominal DL classification method which does not exploit the ordinal structure of the AQC task, thus not penalizing the error among distant AQC classes (which is a relevant aspect for the real use case). The paper introduces a DL ordinal methodology for the AQC classification. Differently from other deep ordinal methods, we combined the standard categorical cross-entropy with the cumulative link model and we imposed the ordinal constraint via the thresholds and slope parameters. Experimental results were performed for solving an AQC task on a novel image dataset originated from a specific company’s demand (i.e., aesthetic assessment of wooden stocks). We demonstrated how the proposed methodology is able to reduce misclassification errors (up to 0.937 quadratic weight kappa loss) among distant classes while overcoming other state-of-the-art deep ordinal models and reducing the bias factor related to the item geometry. The proposed DL approach was integrated as the main core of a DSS supported by Internet of Things (IoT) architecture that can support the human operator by reducing up to 90% the time needed for the qualitative analysis carried out manually in this specific domain.

Currently, the use of deep learning for solving ordinal classification problems, where categories follow a natural order, has not received much attention. In this paper, we propose an unimodal regularisation based on the beta distribution applied to the cross-entropy loss. This regularisation encourages the distribution of the labels to be a soft unimodal distribution, more appropriate for ordinal problems. Given that the beta distribution has two parameters that must be adjusted, a method to automatically determine them is proposed. The regularised loss function is used to train a deep neural network model with an ordinal scheme in the output layer. The results obtained are statistically analysed and show that the combination of these methods increases the performance in ordinal problems. Moreover, the proposed beta distribution performs better than other distributions proposed in previous works, achieving also a reduced computational cost.

3D image scans are an assessment tool for neurological damage in Parkinson’s disease (PD) patients. This diagnosis process can be automatized to help medical staff through Decision Support Systems (DSSs), and Convolutional Neural Networks (CNNs) are good candidates, because they are effective when applied to spatial data. This paper proposes a 3D CNN ordinal model for assessing the level or neurological damage in PD patients. Given that CNNs need large datasets to achieve acceptable performance, a data augmentation method is adapted to work with spatial data. We consider the Ordinal Graph-based Oversampling via Shortest Paths (OGO-SP) method, which applies a gamma probability distribution for inter-class data generation. A modification of OGO-SP is proposed, the OGO-SP-β algorithm, which applies the beta distribution for generating synthetic samples in the inter-class region, a better suited distribution when compared to gamma. The evaluation of the different methods is based on a novel 3D image dataset provided by the Hospital Universitario ‘Reina Sofía’ (Córdoba, Spain). We show how the ordinal methodology improves the performance with respect to the nominal one, and how OGO-SP-β yields better performance than OGO-SP.

Time-series clustering is the process of grouping time series with respect to their similarity or characteristics. Previous approaches usually combine a specific distance measure for time series and a standard clustering method. However, these approaches do not take the similarity of the different subsequences of each time series into account, which can be used to better compare the time-series objects of the dataset. In this article, we propose a novel technique of time-series clustering consisting of two clustering stages. In a first step, a least-squares polynomial segmentation procedure is applied to each time series, which is based on a growing window technique that returns different-length segments. Then, all of the segments are projected into the same dimensional space, based on the coefficients of the model that approximates the segment and a set of statistical features. After mapping, a first hierarchical clustering phase is applied to all mapped segments, returning groups of segments for each time series. These clusters are used to represent all time series in the same dimensional space, after defining another specific mapping process. In a second and final clustering stage, all the time-series objects are grouped. We consider internal clustering quality to automatically adjust the main parameter of the algorithm, which is an error threshold for the segmentation. The results obtained on 84 datasets from the UCR Time Series Classification Archive have been compared against three state-of-the-art methods, showing that the performance of this methodology is very promising, especially on larger datasets.

The adjustment of the hyperparameters and network structure of Convolutional Neural Networks (CNNs) composes an important step towards building effective, but still efficient learning models. The selection of the best configuration is a problem-dependent task that involves to explore an enormous and complex search space. Due to this reason, the use of heuristic-based search fits perfectly within this task, seeking to obtain a near to optimal solution in a complex and large exploratory space. This paper presents SCRODeep, a self-adapting algorithm based on a statistically-driven Coral Reef Optimisation algorithm (SCRO), for the selection of the most adequate CNNs architecture in a particular domain. This metaheuristic has been designed to navigate through a search space where the architecture (defining the particular set of layers, including convolutional or pooling layers), and the hyperparameters of the network (i.e. activation functions, number of units or the kernel initializer, among others) are represented, but where the connections weights and bias are inferred using typical CNNs optimisation algorithms. In contrast to other approaches, where the use of a metaheuristic implies in turn to fix a series of hyperparameters (i.e. the mutation probability in a genetic algorithm), our approach follows a self-parametrisation perspective, thus removing the necessity of fixing these values. The method has been tested in the design of CNNs for image classification, showing that SCRODeep is able to find competitive solutions, while the complexity of the architectures found is constrained.

Nominal time series classification has been widely developed over the last years. However, to the best of our knowledge, ordinal classification of time series is an unexplored field, and this paper proposes a first approach in the context of the shapelet transform (ST). For those time series dataset where there is a natural order between the labels and the number of classes is higher than 2, nominal classifiers are not capable of achieving the best results, because the models impose the same cost of misclassification to all the errors, regardless the difference between the predicted and the ground-truth. In this sense, we consider four different evaluation metrics to do so, three of them of an ordinal nature. The first one is the widely known Information Gain (IG), proved to be very competitive for ST methods, whereas the remaining three measures try to boost the order information by refining the quality measure. These three measures are a reformulation of the Fisher score, the Spearman's correlation coefficient (ρ), and finally, the Pearson's correlation coefficient (R2). An empirical evaluation is carried out, considering 7 ordinal datasets from the UEA & UCR time series classification repository, 4 classifiers (2 of them of nominal nature, whereas the other 2 are of ordinal nature) and 2 performance measures (correct classification rate, CCR, and average mean absolute error, AMAE). The results show that, for both performance metrics, the ST quality metric based on R2is able to obtain the best results, specially for AMAE, for which the differences are statistically significant in favour of R2.

Data augmentation is rapidly gaining attention in machine learning. Synthetic data can be generated by simple transformations or through the data distribution. In the latter case, the main challenge is to estimate the label associated to new synthetic patterns. This paper studies the effect of generating synthetic data by convex combination of patterns and the use of these as unsupervised information in a semi-supervised learning framework with support vector machines, avoiding thus the need to label synthetic examples. We perform experiments on a total of 53 binary classification datasets. Our results show that this type of data over-sampling supports the well-known cluster assumption in semi-supervised learning, showing outstanding results for small high-dimensional datasets and imbalanced learning problems.

Ordinal input variables are common in many supervised and unsupervised machine learning problems. We focus on ordinal classification problems, where the target variable is also categorical and ordinal. In order to represent categorical input variables for measuring distances or applying continuous mapping functions, they have to be transformed to numeric values. This paper evaluates five different methods to do so. Two of them are commonly applied by practitioners, the first one based on binarising the ordinal input variable using standard indicator variables (NomBin), and the second one based on directly mapping each category to a consecutive natural number (Num). Furthermore, three novel proposals are evaluated in this paper: 1) an ordinal binarisation based on considering the order of the input variable (OrdBin), 2) the analysis of pairwise distances between input patterns to recover the latent variable generating the ordinal one (NumLVR), and 3) the refinement of the standard numeric transformation by recovering the distance between sets of patterns of consecutive categories (NumCDR). A thorough empirical evaluation is made, considering 12 datasets, 5 performance metrics and 4 classifiers (2 of them of nominal nature and 2 of ordinal nature). The results show that the Nom-Bin representation method leads to the worst results, and that both Num and NumCDR methods obtain very good performance, although NumCDR results are consistently better for almost all performance metrics and classifiers considered.

Melanoma is a type of cancer that usually occurs on the skin. Early detection is crucial for ensuring five-year survival (which varies between 15% and 99% depending on the melanoma stage). Melanoma severity is typically diagnosed by invasive methods (e.g. a biopsy). In this paper, we propose an alternative system combining image analysis and machine learning for detecting melanoma presence and severity. The 86 features selected consider the shape, colour, pigment network and texture of the melanoma. As opposed to previous studies that have focused on distinguishing melanoma and non-melanoma images, our work considers a finer-grain classification problem using five categories: benign lesions and 4 different stages of melanoma. The dataset presents two main characteristics that are approached by specific machine learning methods: 1) the classes representing melanoma severity follow a natural order, and 2) the dataset is imbalanced, where benign lesions clearly outnumber melanoma ones. Different nominal and ordinal classifiers are considered, one of them being based on an ordinal cascade decomposition method. The cascade method is shown to obtain good performance for all classes, while respecting and exploiting the order information. Moreover, we explore the alternative of applying a class balancing technique, presenting good synergy with the ordinal and nominal methods.

Ordinal regression problems are those machine learning problems where the objective is to classify patterns using a categorical scale which shows a natural order between the labels. Many real-world applications present this labelling structure and that has increased the number of methods and algorithms developed over the last years in this field. Although ordinal regression can be faced using standard nominal classification techniques, there are several algorithms which can specifically benefit from the ordering information. Therefore, this paper is aimed at reviewing the state of the art on these techniques and proposing a taxonomy based on how the models are constructed to take the order into account. Furthermore, a thorough experimental study is proposed to check if the use of the order information improves the performance of the models obtained, considering some of the approaches within the taxonomy. The results confirm that ordering information benefits ordinal models improving their accuracy and the closeness of the predictions to actual targets in the ordinal scale.

Thickness of the melanoma is the most important factor associated with survival in patients with melanoma. It is most commonly reported as a measurement of depth given in millimeters (mm) and computed by means of pathological examination after a biopsy of the suspected lesion. In order to avoid the use of an invasive method in the estimation of the thickness of melanoma before surgery, we propose a computational image analysis system from dermoscopic images. The proposed feature extraction is based on the clinical findings that correlate certain characteristics present in dermoscopic images and tumor depth. Two supervised classification schemes are proposed: a binary classification in which melanomas are classified into thin or thick, and a three-class scheme (thin, intermediate, and thick). The performance of several nominal classification methods, including a recent interpretable method combining logistic regression with artificial neural networks (Logistic regression using Initial variables and Product Units, LIPU), is compared. For the three-class problem, a set of ordinal classification methods (considering ordering relation between the three classes) is included. For the binary case, LIPU outperforms all the other methods with an accuracy of 77.6%, while, for the second scheme, although LIPU reports the highest overall accuracy, the ordinal classification methods achieve a better balance between the performances of all classes.

The imbalanced nature of some real-world data is one of the current challenges for machine learning researchers. One common approach oversamples the minority class through convex combination of its patterns. We explore the general idea of synthetic oversampling in the feature space induced by a kernel function (as opposed to input space). If the kernel function matches the underlying problem, the classes will be linearly separable and synthetically generated patterns will lie on the minority class region. Since the feature space is not directly accessible, we use the empirical feature space (EFS) (a Euclidean space isomorphic to the feature space) for oversampling purposes. The proposed method is framed in the context of support vector machines, where the imbalanced data sets can pose a serious hindrance. The idea is investigated in three scenarios: 1) oversampling in the full and reduced-rank EFSs; 2) a kernel learning technique maximizing the data class separation to study the influence of the feature space structure (implicitly defined by the kernel function); and 3) a unified framework for preferential oversampling that spans some of the previous approaches in the literature. We support our investigation with extensive experiments over 50 imbalanced data sets.

The classification of patterns into naturally ordered labels is referred to as ordinal regression or ordinal classification. Usually, this classification setting is by nature highly imbalanced, because there are classes in the problem that are a priori more probable than others. Although standard over-sampling methods can improve the classification of minority classes in ordinal classification, they tend to introduce severe errors in terms of the ordinal label scale, given that they do not take the ordering into account. A specific ordinal over-sampling method is developed in this paper for the first time in order to improve the performance of machine learning classifiers. The method proposed includes ordinal information by approaching over-sampling from a graph-based perspective. The results presented in this paper show the good synergy of a popular ordinal regression method (a reformulation of support vector machines) with the graph-based proposed algorithms, and the possibility of improving both the classification and the ordering of minority classes. A cost-sensitive version of the ordinal regression method is also introduced and compared with the over-sampling proposals, showing in general lower performance for minority classes.

The classification of patterns into naturally ordered labels is referred to as ordinal regression. This paper proposes an ensemble methodology specifically adapted to this type of problem, which is based on computing different classification tasks through the formulation of different order hypotheses. Every single model is trained in order to distinguish between one given class (k) and all the remaining ones, while grouping them in those classes with a rank lower than k , and those with a rank higher than k. Therefore, it can be considered as a reformulation of the well-known one-versus-all scheme. The base algorithm for the ensemble could be any threshold (or even probabilistic) method, such as the ones selected in this paper: kernel discriminant analysis, support vector machines and logistic regression (LR) (all reformulated to deal with ordinal regression problems). The method is seen to be competitive when compared with other state-of-the-art methodologies (both ordinal and nominal), by using six measures and a total of 15 ordinal datasets. Furthermore, an additional set of experiments is used to study the potential scalability and interpretability of the proposed method when using LR as base methodology for the ensemble.

Ordinal classification refers to classification problems in which the classes have a natural order imposed on them because of the nature of the concept studied. Some ordinal classification approaches perform a projection from the input space to one-dimensional (latent) space that is partitioned into a sequence of intervals (one for each class). Class identity of a novel input pattern is then decided based on the interval its projection falls into. This projection is trained only indirectly as part of the overall model fitting. As with any other latent model fitting, direct construction hints one may have about the desired form of the latent model can prove very useful for obtaining high-quality models. The key idea of this letter is to construct such a projection model directly, using insights about the class distribution obtained from pairwise distance calculations. The proposed approach is extensively evaluated with 8 nominal and ordinal classifiers methods, 10 real-world ordinal classification data sets, and 4 different performance measures. The new methodology obtained the best results in average ranking when considering three of the performance metrics, although significant differences are found for only some of the methods. Also, after observing other methods of internal behavior in the latent space, we conclude that the internal projections do not fully reflect the intraclass behavior of the patterns. Our method is intrinsically simple, intuitive, and easily understandable, yet highly competitive with state-of-the-art approaches to ordinal classification.

The current European debt crisis has drawn considerable attention to credit-rating agencies' news about sovereign ratings. From a technical point of view, credit rating constitutes a typical ordinal regression problem because credit-rating agencies generally present a scale of risk composed of several categories. This fact motivated the use of an ordinal regression approach to address the problem of sovereign credit rating in this paper. Therefore, the ranking of different classes will be taken into account for the design of the classifier. To do so, a novel model is introduced in order to replicate sovereign rating, based on the negative correlation learning framework. The methodology is fully described in this paper and applied to the classification of the 27 European countries' sovereign rating during the 2007-2010 period based on Standard and Poor's reports. The proposed technique seems to be competitive and robust enough to classify the sovereign ratings reported by this agency when compared with other existing well-known ordinal and nominal methods.

In this paper, two neural network threshold ensemble models are proposed for ordinal regression problems. For the first ensemble method, the thresholds are fixed a priori and are not modified during training. The second one considers the thresholds of each member of the ensemble as free parameters, allowing their modification during the training process. This is achieved through a reformulation of these tunable thresholds, which avoids the constraints they must fulfill for the ordinal regression problem. During training, diversity exists in different projections generated by each member is taken into account for the parameter updating. This diversity is promoted in an explicit way using a diversity-encouraging error function, extending the well-known negative correlation learning framework to the area of ordinal regression, and inheriting many of its good properties. Experimental results demonstrate that the proposed algorithms can achieve competitive generalization performance when considering four ordinal regression metrics.

This paper proposes a novel algorithm for ordinal classification based on combining ensemble techniques and discriminant analysis. The proposal is applied to a real application of liver transplantation, where the objective is to predict survival rates of the graft. Ordinal classification is used for this problem because the classes are defined by the following temporal order: 1) failure of the graft before the first 15 days after transplantation, 2) failure between 15 days and 3 months, 3) failure between 3 months and one year, and 4) no failure presented (taking into account that the patient follow-up is up to one year after the transplantation). When compared to other state-of-the-art classifiers like AdaBoost, EBC(SVM) or KDLOR, the proposed algorithm is shown to be competitive. The models obtained could allow medical experts to predict survival rates without knowing exactly the number of days the transplanted organ survived.

This paper presents a structural distance-based crossover for neural network classifiers, which is applied as part of a Memetic Algorithm (MA) for evolving simultaneously the structure and weights of neural network models applied to multiclass problems. Previous researchers have shown that this simultaneous evolution is a way to avoid the noisy fitness evaluation. The MA incorporates a crossover operator that shows to be useful for ameliorating the permutation problem of the network representation (i.e. different genotypes can be used to represent the same neural network phenotype), increasing the structural diversity of the individuals and improving the accuracy of the results. Instead of a recombination probability, the crossover operator considers a similarity parameter (the minimum structural distance), which allows to maintain a trade-off between global and local search. The neural network models selected in this work are the product-unit neural networks (PUNNs), due to their increasing relevance in those classification problems which show a high order relationship between the input variables. The proposed MA is intended to reduce the possible overtraining problems which can raise in some datasets for this kind of models. The evolutionary system is applied to eight classification benchmarks and the results of an analysis of variance contrast (ANOVA) show the effectiveness of the structural-based crossover operator and the capacity of our algorithm to obtain evolved PUNNs with a higher classification accuracy than those obtained using other evolutionary techniques. On the other hand, the results obtained are compared with popular effective machine learning classification methods, resulting in a competitive performance.

There are many metrics available to measure the goodness of a classifier when working with ordinal datasets. These measures are divided into product-moment and association metrics. In this paper, the behavior of several metrics is studied in different situations. In addition, two new measures associated with an ordinal classifier are defined: the maximum and the minimum mean absolute error of all the classes. From the results of this comparison, a pair of metrics is selected (one associated to the overall error and another one to the error of the class with lowest level of classification) to guide the evolution of a multi-objective evolutionary algorithm, obtaining good results in generalization on ordinal datasets.

This paper evaluates the performance of different classifiers when predicting wind speed from synoptic pressure patterns. The prediction problem has been formulated as a classification problem, where the different classes are associated to four values in an ordinal scale. The problem is relevant for long term wind speed prediction and also for wind speed reconstruction in areas (mainly wind farms) where there are not direct wind measures available. The results obtained in this paper present the Support Vector Machine as the best tested classifier for this task. In addition, the use of the intrinsic ordering information of the problem is shown to improve classifier performance.

In this paper we apply and test a recent ordinal algorithm for classification (Kernel Discriminant Learning Ordinal Regression, KDLOR), in order to recognize a group of geographically close spatial units with a similar prevalence pattern significantly high (or low), which are called hot-spots (or cold-spots). Different spatial analysis techniques have been used for studying geographical distribution of a specific illness in mental health-care because it could be useful to organize the spatial distribution of health-care services. Ordinal classification is used in this problem because the classes are: spatial unit with depression, spatial unit which could present depression and spatial unit where there is not depression. It is shown that the proposed method is capable of preserving the rank of data classes in a projected data space for this database. In comparison to other standard methods like C4.5, SVMRank, Adaboost, and MLP nominal classifiers, the proposed KDLOR algorithm is shown to be competitive.

Ordinal classification problems are an active research area in the machine learning community. Many previous works adapted state-of-art nominal classifiers to improve ordinal classification so that the method can take advantage of the ordinal structure of the dataset. However, these method improvements often rely upon a complex mathematical basis and they usually are attached to the training algorithm and model. This paper presents a novel method for generally adapting classification and regression models, such as artificial neural networks or support vector machines. The ordinal classification problem is reformulated as a regression problem by the reconstruction of a numerical variable which represents the different ordered class labels. Despite the simplicity and generality of the method, results are competitive in comparison with very specific methods for ordinal regression.

This paper proposes a hybrid multilogistic methodology, named logistic regression using initial and radial basis function (RBF) covariates. The process for obtaining the coefficients is carried out in three steps. First, an evolutionary programming (EP) algorithm is applied, in order to produce an RBF neural network (RBFNN) with a reduced number of RBF transformations and the simplest structure possible. Then, the initial attribute space (or, as commonly known as in logistic regression literature, the covariate space) is transformed by adding the nonlinear transformations of the input variables given by the RBFs of the best individual in the final generation. Finally, a maximum likelihood optimization method determines the coefficients associated with a multilogistic regression model built in this augmented covariate space. In this final step, two different multilogistic regression algorithms are applied: one considers all initial and RBF covariates (multilogistic initial-RBF regression) and the other one incrementally constructs the model and applies cross validation, resulting in an automatic covariate selection [simplelogistic initial-RBF regression (SLIRBF)]. Both methods include a regularization parameter, which has been also optimized. The methodology proposed is tested using 18 benchmark classification problems from well-known machine learning problems and two real agronomical problems. The results are compared with the corresponding multilogistic regression methods applied to the initial covariate space, to the RBFNNs obtained by the EP algorithm, and to other probabilistic classifiers, including different RBFNN design methods [e.g., relaxed variable kernel density estimation, support vector machines, a sparse classifier (sparse multinomial logistic regression)] and a procedure similar to SLIRBF but using product unit basis functions. The SLIRBF models are found to be competitive when compared with the corresponding multilogistic regression methods and the RBFEP method. A measure of statistical significance is used, which indicates that SLIRBF reaches the state of the art.

This work presents an Evolutionary Artificial Neural Network (EANN) approach based on the Pareto Differential Evolution (PDE) algorithm where the crossover operator is determined using a Gaussian distribution associated with the best models in the evolutionary population. The crossover operator used in a real-coded genetic algorithm is based on confidence intervals. The PDE is used to localize the most promising search regions for locating the best individuals. Confidence intervals use mean localization and standard deviation estimators that are highly recommendable when the distribution of the random variables is Gaussian. It has always been an issue to find good ANN architecture in both multiclassification problems and in the field of ANNs. EANNs provide a better method to optimize simultaneously both network performance (based on the Correct Classification Rate, C) and the network performance of each class (Minimum Sensitivity, MS). The proposal with respect to methodology performance is evaluated using a well characterized set of multiclassification benchmark problems. The results show that crossover performance based on confidence intervals is less dependent on the problem than crossover performance based on a random selection of three parents in the PDE.

The selection of a particular neural network model belonging to the Pareto front is a problem that exists in all multi-objective algorithms. This paper proposes a novel solution to this problem based on a linear combination of the outputs of the two extremes in the Pareto front, which form an ensemble. The decision support TOPSIS method is used to determine which linear combination creates the best ensemble. This analysis selects the most representative individual that performs better in generalization than the extremes of the Pareto front do.

This paper studies the suitability of Extreme Learning Machines (ELM) for resolving bioinformatic and biomedical classification problems. In order to test their overall performance, an experimental study is presented based on five gene microarray datasets found in bioinformatic and biomedical domains. The Fast Correlation-Based Filter (FCBF) was applied in order to identify salient expression genes among the thousands of genes in microarray data that can directly contribute to determining the class membership of each pattern. The results confirm that the ELM classifier is a promising candidate for improving Accuracy and Minimum Sensitivity.

This paper proposes a multiclassification algorithm using multilayer perceptron neural network models. It tries to boost two conflicting main objectives of multiclassifiers: a high correct classification rate level and a high classification rate for each class. This last objective is not usually optimized in classification, but is considered here given the need to obtain high precision in each class in real problems. To solve this machine learning problem, we use a Pareto-based multiobjective optimization methodology based on a memetic evolutionary algorithm. We consider a memetic Pareto evolutionary approach based on the NSGA2 evolutionary algorithm (MPENSGA2). Once the Pareto front is built, two strategies or automatic individual selection are used: the best model in accuracy and the best model in sensitivity (extremes in the Pareto front). These methodologies are applied to solve 17 classification benchmark problems obtained from the University of California at Irvine (UCI) repository and one complex real classification problem. The models obtained show high accuracy and a high classification rate for each class.

This paper proposes a hybrid multilogistic model, named multilogistic regression using initial and radial basis function covariates (MLRIRBF). The process for obtaining the coefficients is carried out in several steps. First, an evolutionary programming (EP) algorithm is applied, aimed to produce a RBF neural network (RBFNN) with a reduced number of RBF transformations and the simplest structure possible. Then, the input space is transformed by adding the nonlinear transformations of the input variables given by the RBFs of the best individual in the last generation. Finally, a maximum likelihood optimization method determines the coefficients associated with a multilogistic regression model built on this transformed input space. In this final step, two different multilogistic regression algorithms are applied, one that considers all initial and RBF covariates (MLRIRBF) and another one that incrementally constructs the model and applies cross-validation, resulting in an automatic covariate selection (MLRIRBF*). The methodology proposed is tested using six benchmark classification problems from well-known machine learning problems. The results are compared with the corresponding multilogistic regression methodologies applied over the initial input space, to the RBFNNs obtained by the EP algorithm (RBFEP) and to other competitive machine learning techniques. The MLRIRBF* models are found to be better than the corresponding multilogistic regression methodologies and the RBFEP method for almost all datasets, and obtain the highest mean accuracy rank when compared to the rest of methods in all datasets.

This paper proposes a novelty neural network model by using generalized kernel functions for the hidden layer of a feed forward network (Generalized Radial Basis Functions, GRBF), where the architecture, weights and node typology are learned through an evolutionary programming algorithm. This new kind of model is compared with the corresponding models with standard hidden nodes: Product Unit Neural Networks (PUNN), Multilayer Perceptrons (MLP) and the RBF neural networks. The methodology proposed is tested using six benchmark classification datasets from well-known machine learning problems. Generalized basis functions are found to present a better performance than the other standard basis functions for the task of classification.

In this paper, we propose a Memetic Algorithm (MA) for classifier optimization based on a clustering method that applies the k-means algorithm over a specific derived space. In this space, each classifier or individual is represented by the set of the accuracies of the classifier for each class of the problem. The proposed sensitivity clustering is able to obtain groups of individuals that perform similarly for the different classes. Then, a representative of each group is selected and it is improved by a local search procedure. This method is applied in specific stages of the evolutionary process. The sensitivity clustering process is compared to a clustering process applied over the n-dimensional space that represent the behaviour of the classifier over each training pattern, where $n$ is the number of patterns. This second method clearly results in a higher computational cost. The comparison is performed in ten imbalanced datasets, including the minimun sensitivity results (i.e. the accuracy for the worst classified class). The results indicate that, although in general the differences are not significant, the sensitivity clustering obtains the best perfomance for almost all datasets both in accuracy and minimum sensitivity, involving a lower computational demand.

This paper proposes a multi-classification pattern algorithm using multilayer perceptron neural network models which try to boost two conflicting main objectives of a classifier, a high correct classification rate and a high classification rate for each class. To solve this machine learning problem, we consider a Memetic Pareto Evolutionary approach based on the NSGA2 algorithm (MPENSGA2), where we defined two objectives for determining the goodness of a classifier: the cross-entropy error function and the variation coefficient of its sensitivities, because both measures are continuous functions, making the convergence more robust. Once the Pareto front is built, we use an automatic selection methodology of individuals: the best model in accuracy (upper extreme in the Pareto front). This methodology is applied to solve six benchmark classification problems, obtaining promising results and achieving a high classification rate in the generalization set with an acceptable level of accuracy for each class.

This paper proposes a neural network model for wind speed prediction, a very important task in wind parks management. Currently, several physical-statistical and artificial intelligence (AI) wind speed prediction models are used to this end. A recently proposed hybrid model is based on hybridizations of global and mesoscale forecasting systems, with a final downscaling step using a multilayer perceptron (MLP). In this paper, we test an alternative neural model for this final step of downscaling, in which projection hyperbolic tangent units (HTUs) are used within feed forward neural networks. The architecture, weights and node typology of the HTU-based network are learnt using a hybrid evolutionary programming algorithm. This new methodology is tested over a real problem of wind speed forecasting, in which we show that our method is able to improve the performance of previous MLPs, obtaining an interpretable model of final regression for each turbine in the wind park.

Performance evaluation is decisive when improving classifiers. Accuracy alone is insufficient because it cannot capture the myriad of contributing factors differentiating the performances of two different classifiers and approaches based on a multi-objective perspective are hindered by the growing of the Pareto optimal front as the number of classes increases. This paper proposes a new approach to deal with multi-class problems based on the accuracy (C) and minimum sensitivity (S) given by the lowest percentage of examples correctly predicted to belong to each class. From this perspective, we compare different fitness functions (accuracy, C , entropy, E , sensitivity, S , and area, A ) in an evolutionary scheme. We also present a two stage evolutionary algorithm with two sequential fitness functions, the entropy for the first step and the area for the second step. This methodology is applied to solve six benchmark classification problems. The two-stage approach obtains promising results and achieves a high classification rate level in the global dataset with an acceptable level of accuracy for each class.

The main objective of this work is to automatically design neural network models with sigmoidal basis units for classification tasks, so that classifiers are obtained in the most balanced way possible in terms of CCR and Sensitivity (given by the lowest percentage of examples correctly predicted to belong to each class). We present a Memetic Pareto Evolutionary NSGA2 (MPENSGA2) approach based on the Pareto-NSGAII evolution (PNSGAII) algorithm. We propose to augmente it with a local search using the improved Rprop—IRprop algorithm for the prediction of growth/no growth of L. monocytogenes as a function of the storage temperature, pH, citric (CA) and ascorbic acid (AA). The results obtained show that the generalization ability can be more efficiently improved within a framework that is multi-objective instead of a within a single-objective one.

Logistic Regression (LR) has become a widely used and accepted method to analyse binary or multiclass outcome variables, since it is a flexible tool that can predict the probability for the state of a dichotomous variable. A recently proposed LR method is based on the hybridisation of a linear model and Evolutionary Product-Unit Neural Network (EPUNN) models for binary classification. This produces a high number of coefficients, so two different methods for reducing the number of initial or PU covariates are proposed in this paper, both based on the Wald test. The first method is a two-step Backward Search (BS) method and the second is based on the standard Simulated Annealing (SA) heuristic. In this study, we used aerial imagery taken in mid-May to evaluate the potential of two different combinations of LR and EPUNN (LR using PUs (LRPU), as well as LR using Initial covariates and PUs (LRIPU)) and the two proposed methods for selecting variables in the final models (BS and SA) for discriminating Ridolfia segetum patches (one of the most dominant, competitive and persistent weed in sunflower crops) in one naturally infested field of southern Spain. Then, we compared the performance of these methods to six recent classification models, our proposals obtaining a competitive performance and a lower number of coefficients.

We propose a classification method based on a special class of feed-forward neural network, namely product-unit neural networks. They are based on multiplicative nodes instead of additive ones, where the nonlinear basis functions express the possible strong interactions among the variables. We apply an evolutionary algorithm to determine the basic structure of the product-unit model and to estimate the coefficients of the model. The empirical results show that the proposed model is very promising in terms of classification accuracy, yielding a state-of-the-art performance.

This paper presents a hybrid evolutionary algorithm (EA) to solve nonlinear-regression problems. Although EAs have proven their ability to explore large search spaces, they are comparatively inefficient in fine tuning the solution. This drawback is usually avoided by means of local optimization algorithms that are applied to the individuals of the population. The algorithms that use local optimization procedures are usually called hybrid algorithms. On the other hand, it is well known that the clustering process enables the creation of groups (clusters) with mutually close points that hopefully correspond to relevant regions of attraction. Local-search procedures can then be started once in every such region. This paper proposes the combination of an EA, a clustering process, and a local-search procedure to the evolutionary design of product-units neural networks. In the methodology presented, only a few individuals are subject to local optimization. Moreover, the local optimization algorithm is only applied at specific stages of the evolutionary process. Our results show a favorable performance when the regression method proposed is compared to other standard methods.

One of the most common and versatile techniques for coping with constraints consists of the penalty of the solutions whose variables do not fulfill the constraints. The genetic algorithm (GA) is one of the main tools used for the optimization of functions with constraints. In this context the crossover operator must tend to generate individuals within or near the feasible region in order to converge to useful solutions. In this work we make an analysis of the influence of the crossover operator in this kind of problems. We have used a test set that includes functions with linear and nonlinear constraints. The results confirm the importance of the crossover operator.

In this paper we propose a crossover operator for evolutionary algorithms with real values that is based on the statistical theory of population distributions. The operator is based on the theoretical distribution of the values of the genes of the best individuals in the population. The proposed operator takes into account the localization and dispersion features of the best individuals of the population with the objective that these features would be inherited by the offspring. Our aim is the optimization of the balance between exploration and exploitation in the search process. In order to test the efficiency and robustness of this crossover, we have used a set of functions to be optimized with regard to different criteria, such as, multimodality, separability, regularity and epistasis. With this set of functions we can extract conclusions in function of the problem at hand. We analyze the results using ANOVA and multiple comparison statistical tests. As an example of how our crossover can be used to solve artificial intelligence problems, we have applied the proposed model to the problem of obtaining the weight of each network in a ensemble of neural networks. The results obtained are above the performance of standard methods.

This paper presents a cooperative coevolutive approach for designing neural network ensembles. Cooperative coevolution is a recent paradigm in evolutionary computation that allows the effective modeling of cooperative environments. Although theoretically, a single neural network with a sufficient number of neurons in the hidden layer would suffice to solve any problem, in practice many real-world problems are too hard to construct the appropriate network that solve them. In such problems, neural network ensembles are a successful alternative. Nevertheless, the design of neural network ensembles is a complex task. In this paper, we propose a general framework for designing neural network ensembles by means of cooperative coevolution. The proposed model has two main objectives: first, the improvement of the combination of the trained individual networks; second, the cooperative evolution of such networks, encouraging collaboration among them, instead of a separate training of each network. In order to favor the cooperation of the networks, each network is evaluated throughout the evolutionary process using a multiobjective method. For each network, different objectives are defined, considering not only its performance in the given problem, but also its cooperation with the rest of the networks. In addition, a population of ensembles is evolved, improving the combination of networks and obtaining subsets of networks to form ensembles that perform better than the combination of all the evolved networks. The proposed model is applied to ten real-world classification problems of a very different nature from the UCI machine learning repository and proben1 benchmark set. In all of them the performance of the model is better than the performance of standard ensembles in terms of generalization error. Moreover, the size of the obtained ensembles is also smaller.

This paper presents COVNET, a new cooperative coevolutionary model for evolving artificial neural networks. This model is based on the idea of coevolving subnetworks that must cooperate to form a solution for a specific problem, instead of evolving complete networks. The combination of this subnetworks is part of a coevolutionary process. The best combinations of subnetworks must be evolved together with the coevolution of the subnetworks. Several subpopulations of subnetworks coevolve cooperatively and genetically isolated. The individual of every subpopulation are combined to form whole networks. This is a different approach from most current models of evolutionary neural networks which try to develop whole networks. COVNET places as few restrictions as possible over the network structure, allowing the model to reach a wide variety of architectures during the evolution and to be easily extensible to other kind of neural networks. The performance of the model in solving three real problems of classification is compared with a modular network, the adaptive mixture of experts and with the results presented in the bibliography. COVNET has shown better generalization and produced smaller networks than the adaptive mixture of experts and has also achieved results, at least, comparable with the results in the bibliography.