An Ordinal Classification (OC) problem corresponds to a special type of classification characterised by the presence of a natural order relationship among the classes. This type of problem, that can be found in a number of real-world applications, has motivated the design and development of many ordinal methodologies over the last years. However, it is important to highlight that the development of the OC field suffers from one main disadvantage: the lack of a comprehensive set of datasets on which novel approaches to the literature are benchmarked. In order to approach this objective, this manuscript from the University of Córdoba (UCO), which has previous experience on the OC field, provides the literature with a publicly available repository of tabular data for a robust validation of novel OC approaches, namely TOC-UCO (Tabular Ordinal Classification repository of the UCO). Specifically, this repository includes a set of tabular ordinal datasets that have been preprocessed under a common framework and that have a reasonable number of patterns and an appropriate class distribution. We also provide the sources and preprocessing steps of each dataset, along with details on how to benchmark a novel approach using the TOC-UCO repository. For this, indices for different randomised train-test partitions are provided to facilitate the reproducibility of the experiments. • Introduction of a novel ordinal classification repository: TOC-UCO . • Extension of the number of datasets with a high number of classes. • Analysis of the previous ordinal classification benchmarking repository. • In-depth comparison between TOC-UCO and the previous repository. • Presentation of a baseline experimentation on the new TOC-UCO archive.

Ordinal Classification (OC) addresses those classification tasks where the labels exhibit a natural order. Unlike nominal classification, which treats all classes as mutually exclusive and unordered, OC takes the ordinal relationship into account, producing more accurate and relevant results. This is particularly critical in applications where the magnitude of classification errors has significant consequences. Despite this, OC problems are often tackled using nominal methods, leading to suboptimal solutions. Although decision trees are among the most popular classification approaches, ordinal tree-based approaches have received less attention when compared to other classifiers. This work provides a comprehensive survey of ordinal splitting criteria, standardising the notations used in the literature to enhance clarity and consistency. Three ordinal splitting criteria, Ordinal Gini (OGini), Weighted Information Gain, and Ranking Impurity, are compared to the nominal counterparts of the first two (Gini and information gain), by incorporating them into a decision tree classifier. An extensive repository considering 45 publicly available OC datasets is presented, supporting the first experimental comparison of ordinal and nominal splitting criteria using well-known OC evaluation metrics. The results have been statistically analysed, highlighting that OGini stands out as the best ordinal splitting criterion to date, reducing the mean absolute error achieved by Gini by more than 3.02 % . To promote reproducibility, all source code developed, a detailed guide for reproducing the results, the 45 OC datasets, and the individual results for all the evaluated methodologies are provided.

Ordinal classification, where labels follow a natural order, has gained increasing attention, particularly in the deep learning community due to its relevance in tasks such as age estimation, medical grading, and quality assessment. Despite the growing number of deep ordinal classification methods, a comprehensive experimental analysis of their core ordinal components remains lacking. This work presents a systematic evaluation of deep ordinal classifiers by analysing the impact of three key modelling choices: the loss function, output layer, and labelling strategy. To analyse their effects, we adopt a unified architecture and evaluate one nominal and 19 ordinal configurations, resulting from combination of two loss functions, two output layers, and five labelling strategies. These configurations are assessed on 12 diverse ordinal image datasets using six performance metrics, including both ordinal and nominal measures. Results show that ordinal output layers consistently outperform softmax, and that soft labelling generally improves generalisation. While categorical cross-entropy achieves better average performance, especially on nominal metrics, no configuration performs best across all datasets. Statistical analyses indicate significant interactions between losses, outputs, labelling strategies, and datasets, highlighting the need to adapt methodological choices to specific tasks. These findings provide valuable guidance for designing robust deep ordinal classification models.

The use of Convolutional Neural Network (CNN) models for image classification tasks has gained significant popularity. However, the lack of interpretability in CNN models poses challenges for debugging and validation. To address this issue, various explanation methods have been developed to provide insights into CNN models. This paper focuses on the validity of these explanation methods for ordinal regression tasks, where the classes have a predefined order relationship. Different modifications are proposed for two explanation methods to exploit the ordinal relationships between classes: Grad-CAM based on Ordinal Binary Decomposition (GradOBD-CAM) and Ordinal Information Bottleneck Analysis (OIBA). The performance of these modified methods is compared to existing popular alternatives. Experimental results demonstrate that GradOBD-CAM outperforms other methods in terms of interpretability for three out of four datasets, while OIBA achieves superior performance compared to IBA. • Addressing interpretability challenges in CNN models for ordinal regression. • Modification of explanation methods for ordinal regression tasks. • Superior performance of GradOBD-CAM for interpretability in 3 out of 4 datasets. • Enhanced interpretability with OIBA compared to IBA. • Evaluation using comprehensive ordinal degradation score metrics.

dlordinal is a new Python library that unifies many recent deep ordinal classification methodologies available in the literature. Developed using PyTorch as underlying framework, it implements the top performing state-of-the-art deep learning techniques for ordinal classification problems. Ordinal approaches are designed to leverage the ordering information present in the target variable. Specifically, it includes loss functions, various output layers, dropout techniques, soft labelling methodologies, and other classification strategies, all of which are appropriately designed to incorporate the ordinal information. Furthermore, as the performance metrics to assess novel proposals in ordinal classification depend on the distance between target and predicted classes in the ordinal scale, suitable ordinal evaluation metrics are also included. dlordinal is distributed under the BSD-3-Clause license and is available at https://github.com/ayrna/dlordinal.

Time-series classification (TSC) covers the supervised learning problem where input data is provided in the form of series of values observed through repeated measurements over time, and whose objective is to predict the category to which they belong. When the class values are ordinal, classifiers that take this into account can perform better than nominal classifiers. Time-series ordinal classification (TSOC) is the field bridging this gap, yet unexplored in the literature. There are a wide range of time-series problems showing an ordered label structure, and TSC techniques that ignore the order relationship discard useful information. Hence, this article presents the first benchmarking of TSOC methodologies, exploiting the ordering of the target labels to boost the performance of current TSC state of the art. Both convolutional- and deep-learning-based methodologies (among the best performing alternatives for nominal TSC) are adapted for TSOC. For the experiments, a selection of 29 ordinal problems has been made. In this way, this article contributes to the establishment of the state of the art in TSOC. The results obtained by ordinal versions are found to be significantly better than current nominal TSC techniques in terms of ordinal performance metrics, outlining the importance of considering the ordering of the labels when dealing with this kind of problems.

Real-world classification problems may disclose different hierarchical levels where the categories are displayed in an ordinal structure. However, no specific deep learning (DL) models simultaneously learn hierarchical and ordinal constraints while improving generalization performance. To fill this gap, we propose the introduction of two novel ordinal-hierarchical DL methodologies, namely, the hierarchical cumulative link model (HCLM) and hierarchical-ordinal binary decomposition (HOBD), which are able to model the ordinal structure within different hierarchical levels of the labels. In particular, we decompose the hierarchical-ordinal problem into local and global graph paths that may encode an ordinal constraint for each hierarchical level. Thus, we frame this problem as simultaneously minimizing global and local losses. Furthermore, the ordinal constraints are set by two approaches [ordinal binary decomposition (OBD) and cumulative link model (CLM)] within each global and local function. The effectiveness of the proposed approach is measured on four real-use case datasets concerning industrial, biomedical, computer vision, and financial domains. The extracted results demonstrate a statistically significant improvement to state-of-the-art nominal, ordinal, and hierarchical approaches.

In this paper we present a novel methodology, referenced as ORFEO (Ordinal classifier and Regressor Fusion for Estimating an Ordinal categorical target), to enhance the performance in ordinal classification problems for which the latent variable is observable. ORFEO is an artificial neural network model incorporating two outputs, one for ordinal classification, using the cumulative link model, and one for regression, using a linear model. Both outputs are simultaneously optimised considering a loss function that linearly combines both classification and regression losses. The main motivation behind developing the proposed approach is to enhance the performance of a standard ordinal classifier. This improvement is facilitated by considering the regression output, which allows the model to differentiate between patterns within the same category. The ORFEO model is applied to two problems in the field of marine and ocean engineering: short-term prediction of both significant wave height and flux of energy. Both problems are addressed considering four different coastal zones of the United States of America, using 13 datasets formed by buoys measurements and reanalysis data. A comprehensive comparison against 20 methodologies, including regression and nominal/ordinal classification approaches is performed, by using diverse nominal and ordinal performance metrics. Ranks achieved indicate that ORFEO outperforms all the compared methodologies in terms of all the performance measures, demonstrating the efficacy and robustness of the proposal. Finally, a statistical analysis is conducted, concluding that there are statistically significant differences across ordinal and nominal performance metrics in favour of the proposed ORFEO model.

This paper presents a method for providing explainability in the integration of artificial intelligence (AI) and data mining techniques when dealing with meteorological prediction. Explainable artificial intelligence (XAI) refers to the transparency of AI systems in providing explanations for their predictions and decision-making processes, and contribute to improve prediction accuracy and enhance trust in AI systems. The focus of this paper relies on the interpretability challenges in ordinal classification problems within weather forecasting. Ordinal classification involves predicting weather phenomena with ordered classes, such as temperature ranges, wind speed, precipitation levels, and others. To address this challenge, a novel and general explicable forecasting framework, that combines inductive rules and fuzzy logic, is proposed in this work. Inductive rules, derived from historical weather data, provide a logical and interpretable basis for forecasting; while fuzzy logic handles the uncertainty and imprecision in the weather data. The system predicts a set of probabilities that the incoming sample belongs to each considered class. Moreover, it allows the expert decision-making process to be strengthened by relying on the transparency and physical explainability of the model, and not only on the output of a black-box algorithm. The proposed framework is evaluated using two real-world weather databases related to wind speed and low-visibility events due to fog. The results are compared to both ML classifiers and specific methods for ordinal classification problems, achieving very competitive results in terms of ordinal performance metrics while offering a higher level of explainability and transparency compared to existing approaches.

In this study, we present EBANO (Ensemble BAsed on uNimodal Ordinal classifiers), which is a novel ensemble approach of ordinal classifiers that includes four soft labelling approaches along with an ordinal logistic regression model. These models are integrated within the ensemble using a new aggregation methodology that automatically weights each individual classifier using a randomised search algorithm. In addition, the proposed EBANO methodology is applied to tackle short-term prediction of Significant Wave Height (SWH). Thus, we employ EBANO using a diverse set of eight datasets derived from reanalysis data and buoy-recorded SWH measurements. To approach the problem from an ordinal classification perspective, the SWH values are discretised into five ordered classes by applying hierarchical clustering. EBANO is compared with each of the individual classifiers integrated in the proposed ensemble along with a different ensemble technique termed HESCA. Both the average results and the ranks obtained show the superiority of EBANO over the compared methodologies, being more pronounced in the metrics that account for the imbalance present in the datasets considered. Finally, a statistical analysis is performed, confirming the statistical significance of the observed differences in all comparisons. This analysis underscores the effectiveness of EBANO in addressing the problem of SWH prediction, showcasing its excellence.

Predictive Maintenance (PdM) methods aim to facilitate the scheduling of maintenance work before equipment failure. In this context, detecting early faults in automated teller machines (ATMs) has become increasingly important since these machines are susceptible to various types of unpredictable failures. ATMs track execution status by generating massive event-log data that collect system messages unrelated to the failure event. Predicting machine failure based on event logs poses additional challenges, mainly in extracting features that might represent sequences of events indicating impending failures. Accordingly, feature learning approaches are currently being used in PdM, where informative features are learned automatically from minimally processed sensor data. However, a gap remains to be seen on how these approaches can be exploited for deriving relevant features from event-log-based data. To fill this gap, we present a predictive model based on a convolutional kernel (MiniROCKET and HYDRA) to extract features from the original event-log data and a linear classifier to classify the sample based on the learned features. The proposed methodology is applied to a significant real-world collected dataset. Experimental results demonstrated how one of the proposed convolutional kernels (i.e. HYDRA) exhibited the best classification performance (accuracy of 0.759 and AUC of 0.693). In addition, statistical analysis revealed that the HYDRA and MiniROCKET models significantly overcome one of the established state-of-the-art approaches in time series classification (InceptionTime), and three non-temporal ML methods from the literature. The predictive model was integrated into a container-based decision support system to support operators in the timely maintenance of ATMs.

The sanitary emergency caused by COVID-19 has compromised countries and generated a worldwide health and economic crisis. To provide support to the countries' responses, numerous lines of research have been developed. The spotlight was put on effectively and rapidly diagnosing and predicting the evolution of the pandemic, one of the most challenging problems of the past months. This work contributes to the existing literature by developing a two-step methodology to analyze the transmission rate, designing models applied to territories with similar pandemic behavior characteristics. Virus transmission is considered as bacterial growth curves to understand the spread of the virus and to make predictions about its future evolution. Hence, an analytical clustering procedure is first applied to create groups of locations where the virus transmission rate behaved similarly in the different outbreaks. A curve decomposition process based on an iterative polynomial process is then applied, obtaining meaningful forecasting features. Information of the territories belonging to the same cluster is merged to build models capable of simultaneously predicting the 14-day incidence in several locations using Evolutionary Artificial Neural Networks. The methodology is applied to Andalusia (Spain), although it is applicable to any region across the world. Individual models trained for a specific territory are carried out for comparison purposes. The results demonstrate that this methodology achieves statistically similar, or even better, performance for most of the locations. In addition to being extremely competitive, the main advantage of the proposal lies in its complexity cost reduction. The total number of parameters to be estimated is reduced up to 93.51% for the short term and 93.31% for the mid-term forecasting, respectively. Moreover, the number of required models is reduced by 73.53% and 58.82% for the short- and mid-term forecasting horizons.

Abstract Automatic classification tasks on structured data have been revolutionized by Convolutional Neural Networks (CNNs), but the focus has been on binary and nominal classification tasks. Only recently, ordinal classification (where class labels present a natural ordering) has been tackled through the framework of CNNs. Also, ordinal classification datasets commonly present a high imbalance in the number of samples of each class, making it an even harder problem. Focus should be shifted from classic classification metrics towards per-class metrics (like AUC or Sensitivity) and rank agreement metrics (like Cohen’s Kappa or Spearman’s rank correlation coefficient). We present a new CNN architecture based on the Ordinal Binary Decomposition (OBD) technique using Error-Correcting Output Codes (ECOC). We aim to show experimentally, using four different CNN architectures and two ordinal classification datasets, that the OBD+ECOC methodology significantly improves the mean results on the relevant ordinal and class-balancing metrics. The proposed method is able to outperform a nominal approach as well as already existing ordinal approaches, achieving a mean performance of $${{\,\mathrm{\textit{RMSE}}\,}}= 1.0797$$ RMSE = 1.0797 for the Retinopathy dataset and $${{\,\mathrm{\textit{RMSE}}\,}}= 1.1237$$ RMSE = 1.1237 for the Adience dataset averaged over 4 different architectures.

Activation functions lie at the core of every neural network model from shallow to deep convolutional neural networks. Their properties and characteristics shape the output range of each layer and, thus, their capabilities. Modern approaches rely mostly on a single function choice for the whole network, usually ReLU or other similar alternatives. In this work, we propose two new activation functions and analyze their properties and compare them with 17 different function proposals from recent literature on six distinct problems with different characteristics. The objective is to shed some light on their comparative performance. The results show that the proposed functions achieved better performance than the most commonly used ones.

Abstract Nowadays, decision support systems (DSSs) are widely used in several application domains, from industrial to healthcare and medicine fields. Concerning the industrial scenario, we propose a DSS oriented to the aesthetic quality control (AQC) task, which has quickly established itself as one of the most crucial challenges of Industry 4.0. Taking into account the increasing amount of data in this domain, the application of machine learning (ML) and deep learning (DL) techniques offers great opportunities to automatize the overall AQC process. State-of-the-art is mainly oriented to approach this problem with a nominal DL classification method which does not exploit the ordinal structure of the AQC task, thus not penalizing the error among distant AQC classes (which is a relevant aspect for the real use case). The paper introduces a DL ordinal methodology for the AQC classification. Differently from other deep ordinal methods, we combined the standard categorical cross-entropy with the cumulative link model and we imposed the ordinal constraint via the thresholds and slope parameters. Experimental results were performed for solving an AQC task on a novel image dataset originated from a specific company’s demand (i.e., aesthetic assessment of wooden stocks). We demonstrated how the proposed methodology is able to reduce misclassification errors (up to 0.937 quadratic weight kappa loss) among distant classes while overcoming other state-of-the-art deep ordinal models and reducing the bias factor related to the item geometry. The proposed DL approach was integrated as the main core of a DSS supported by Internet of Things (IoT) architecture that can support the human operator by reducing up to 90% the time needed for the qualitative analysis carried out manually in this specific domain.

Currently, the use of deep learning for solving ordinal classification problems, where categories follow a natural order, has not received much attention. In this paper, we propose an unimodal regularisation based on the beta distribution applied to the cross-entropy loss. This regularisation encourages the distribution of the labels to be a soft unimodal distribution, more appropriate for ordinal problems. Given that the beta distribution has two parameters that must be adjusted, a method to automatically determine them is proposed. The regularised loss function is used to train a deep neural network model with an ordinal scheme in the output layer. The results obtained are statistically analysed and show that the combination of these methods increases the performance in ordinal problems. Moreover, the proposed beta distribution performs better than other distributions proposed in previous works, achieving also a reduced computational cost.

Classification and regression techniques are two of the main tasks considered by the Machine Learning area. They mainly depend on the target variable to predict. In this context, ordinal classification represents an intermediate task, which is focused on the prediction of nominal variables where the categories follow a specific intrinsic order given by the problem. Nevertheless, the integration of different algorithms able to solve ordinal classification problems is often unavailable in most of existing Machine Learning software, which hinders the use of new approaches. Therefore, this paper focuses on the incorporation of an ordinal classification algorithm (NSLVOrd) in one of the most complete ordinal regression frameworks, “Ordinal Regression and Classification Algorithms framework (ORCA)” by using both fuzzy rules and the JFML library. The use of NSLVOrd in the ORCA tool as well as a case study with a real database are shown where the obtained results are promising.

3D image scans are an assessment tool for neurological damage in Parkinson’s disease (PD) patients. This diagnosis process can be automatized to help medical staff through Decision Support Systems (DSSs), and Convolutional Neural Networks (CNNs) are good candidates, because they are effective when applied to spatial data. This paper proposes a 3D CNN ordinal model for assessing the level or neurological damage in PD patients. Given that CNNs need large datasets to achieve acceptable performance, a data augmentation method is adapted to work with spatial data. We consider the Ordinal Graph-based Oversampling via Shortest Paths (OGO-SP) method, which applies a gamma probability distribution for inter-class data generation. A modification of OGO-SP is proposed, the OGO-SP-β algorithm, which applies the beta distribution for generating synthetic samples in the inter-class region, a better suited distribution when compared to gamma. The evaluation of the different methods is based on a novel 3D image dataset provided by the Hospital Universitario ‘Reina Sofía’ (Córdoba, Spain). We show how the ordinal methodology improves the performance with respect to the nominal one, and how OGO-SP-β yields better performance than OGO-SP.

Time-series clustering is the process of grouping time series with respect to their similarity or characteristics. Previous approaches usually combine a specific distance measure for time series and a standard clustering method. However, these approaches do not take the similarity of the different subsequences of each time series into account, which can be used to better compare the time-series objects of the dataset. In this article, we propose a novel technique of time-series clustering consisting of two clustering stages. In a first step, a least-squares polynomial segmentation procedure is applied to each time series, which is based on a growing window technique that returns different-length segments. Then, all of the segments are projected into the same dimensional space, based on the coefficients of the model that approximates the segment and a set of statistical features. After mapping, a first hierarchical clustering phase is applied to all mapped segments, returning groups of segments for each time series. These clusters are used to represent all time series in the same dimensional space, after defining another specific mapping process. In a second and final clustering stage, all the time-series objects are grouped. We consider internal clustering quality to automatically adjust the main parameter of the algorithm, which is an error threshold for the segmentation. The results obtained on 84 datasets from the UCR Time Series Classification Archive have been compared against three state-of-the-art methods, showing that the performance of this methodology is very promising, especially on larger datasets.

The adjustment of the hyperparameters and network structure of Convolutional Neural Networks (CNNs) composes an important step towards building effective, but still efficient learning models. The selection of the best configuration is a problem-dependent task that involves to explore an enormous and complex search space. Due to this reason, the use of heuristic-based search fits perfectly within this task, seeking to obtain a near to optimal solution in a complex and large exploratory space. This paper presents SCRODeep, a self-adapting algorithm based on a statistically-driven Coral Reef Optimisation algorithm (SCRO), for the selection of the most adequate CNNs architecture in a particular domain. This metaheuristic has been designed to navigate through a search space where the architecture (defining the particular set of layers, including convolutional or pooling layers), and the hyperparameters of the network (i.e. activation functions, number of units or the kernel initializer, among others) are represented, but where the connections weights and bias are inferred using typical CNNs optimisation algorithms. In contrast to other approaches, where the use of a metaheuristic implies in turn to fix a series of hyperparameters (i.e. the mutation probability in a genetic algorithm), our approach follows a self-parametrisation perspective, thus removing the necessity of fixing these values. The method has been tested in the design of CNNs for image classification, showing that SCRODeep is able to find competitive solutions, while the complexity of the architectures found is constrained.

Nominal time series classification has been widely developed over the last years. However, to the best of our knowledge, ordinal classification of time series is an unexplored field, and this paper proposes a first approach in the context of the shapelet transform (ST). For those time series dataset where there is a natural order between the labels and the number of classes is higher than 2, nominal classifiers are not capable of achieving the best results, because the models impose the same cost of misclassification to all the errors, regardless the difference between the predicted and the ground-truth. In this sense, we consider four different evaluation metrics to do so, three of them of an ordinal nature. The first one is the widely known Information Gain (IG), proved to be very competitive for ST methods, whereas the remaining three measures try to boost the order information by refining the quality measure. These three measures are a reformulation of the Fisher score, the Spearman's correlation coefficient (ρ), and finally, the Pearson's correlation coefficient (R2). An empirical evaluation is carried out, considering 7 ordinal datasets from the UEA & UCR time series classification repository, 4 classifiers (2 of them of nominal nature, whereas the other 2 are of ordinal nature) and 2 performance measures (correct classification rate, CCR, and average mean absolute error, AMAE). The results show that, for both performance metrics, the ST quality metric based on R2is able to obtain the best results, specially for AMAE, for which the differences are statistically significant in favour of R2.

Ordinal regression, also named ordinal classification, studies classification problems where there exist a natural order between class labels. This structured order of the labels is crucial in all steps of the learning process in order to take full advantage of the data. ORCA (Ordinal Regression and Classification Algorithms) is a Matlab/Octave framework that implements and integrates different ordinal classification algorithms and specifically designed performance metrics. The framework simplifies the task of experimental comparison to a great extent, allowing the user to: (i) describe experiments by simple configuration files; (ii) automatically run different data partitions; (iii) parallelize the executions; (iv) generate a variety of performance reports and (v) include new algorithms by using its intuitive interface. Source code, binaries, documentation, descriptions and links to data sets and tutorials (including examples of educational purpose) are available at https://github.com/ayrna/orca.

As you open this January issue of the IEEE Transactions on Neural Networks and Learning Systems (TNNLS), I hope everyone enjoyed a great holiday season and is excited for the new year of 2019. I am very delighted and honored to report several key metrics of IEEE TNNLS to the community.

Weather and atmospheric patterns are often persistent. The simplest weather forecasting method is the so-called persistence model, which assumes that the future state of a system will be similar (or equal) to the present state. Machine learning (ML) models are widely used in different weather forecasting applications, but they need to be compared to the persistence model to analyse whether they provide a competitive solution to the problem at hand. In this paper, we devise a new model for predicting low-visibility in airports using the concepts of mixture of experts. Visibility level is coded as two different ordered categorical variables: cloud height and runway visual height. The underlying system in this application is stagnant approximately in 90% of the cases, and standard ML models fail to improve on the performance of the persistence model. Because of this, instead of trying to simply beat the persistence model using ML, we use this persistence as a baseline and learn an ordinal neural network model that refines its results by focusing on learning weather fluctuations. The results show that the proposal outperforms persistence and other ordinal autoregressive models, especially for longer time horizon predictions and for the runway visual height variable.

Ordinal input variables are common in many supervised and unsupervised machine learning problems. We focus on ordinal classification problems, where the target variable is also categorical and ordinal. In order to represent categorical input variables for measuring distances or applying continuous mapping functions, they have to be transformed to numeric values. This paper evaluates five different methods to do so. Two of them are commonly applied by practitioners, the first one based on binarising the ordinal input variable using standard indicator variables (NomBin), and the second one based on directly mapping each category to a consecutive natural number (Num). Furthermore, three novel proposals are evaluated in this paper: 1) an ordinal binarisation based on considering the order of the input variable (OrdBin), 2) the analysis of pairwise distances between input patterns to recover the latent variable generating the ordinal one (NumLVR), and 3) the refinement of the standard numeric transformation by recovering the distance between sets of patterns of consecutive categories (NumCDR). A thorough empirical evaluation is made, considering 12 datasets, 5 performance metrics and 4 classifiers (2 of them of nominal nature and 2 of ordinal nature). The results show that the Nom-Bin representation method leads to the worst results, and that both Num and NumCDR methods obtain very good performance, although NumCDR results are consistently better for almost all performance metrics and classifiers considered.

Melanoma is a type of cancer that usually occurs on the skin. Early detection is crucial for ensuring five-year survival (which varies between 15% and 99% depending on the melanoma stage). Melanoma severity is typically diagnosed by invasive methods (e.g. a biopsy). In this paper, we propose an alternative system combining image analysis and machine learning for detecting melanoma presence and severity. The 86 features selected consider the shape, colour, pigment network and texture of the melanoma. As opposed to previous studies that have focused on distinguishing melanoma and non-melanoma images, our work considers a finer-grain classification problem using five categories: benign lesions and 4 different stages of melanoma. The dataset presents two main characteristics that are approached by specific machine learning methods: 1) the classes representing melanoma severity follow a natural order, and 2) the dataset is imbalanced, where benign lesions clearly outnumber melanoma ones. Different nominal and ordinal classifiers are considered, one of them being based on an ordinal cascade decomposition method. The cascade method is shown to obtain good performance for all classes, while respecting and exploiting the order information. Moreover, we explore the alternative of applying a class balancing technique, presenting good synergy with the ordinal and nominal methods.

Ordinal regression problems are those machine learning problems where the objective is to classify patterns using a categorical scale which shows a natural order between the labels. Many real-world applications present this labelling structure and that has increased the number of methods and algorithms developed over the last years in this field. Although ordinal regression can be faced using standard nominal classification techniques, there are several algorithms which can specifically benefit from the ordering information. Therefore, this paper is aimed at reviewing the state of the art on these techniques and proposing a taxonomy based on how the models are constructed to take the order into account. Furthermore, a thorough experimental study is proposed to check if the use of the order information improves the performance of the models obtained, considering some of the approaches within the taxonomy. The results confirm that ordering information benefits ordinal models improving their accuracy and the closeness of the predictions to actual targets in the ordinal scale.

Thickness of the melanoma is the most important factor associated with survival in patients with melanoma. It is most commonly reported as a measurement of depth given in millimeters (mm) and computed by means of pathological examination after a biopsy of the suspected lesion. In order to avoid the use of an invasive method in the estimation of the thickness of melanoma before surgery, we propose a computational image analysis system from dermoscopic images. The proposed feature extraction is based on the clinical findings that correlate certain characteristics present in dermoscopic images and tumor depth. Two supervised classification schemes are proposed: a binary classification in which melanomas are classified into thin or thick, and a three-class scheme (thin, intermediate, and thick). The performance of several nominal classification methods, including a recent interpretable method combining logistic regression with artificial neural networks (Logistic regression using Initial variables and Product Units, LIPU), is compared. For the three-class problem, a set of ordinal classification methods (considering ordering relation between the three classes) is included. For the binary case, LIPU outperforms all the other methods with an accuracy of 77.6%, while, for the second scheme, although LIPU reports the highest overall accuracy, the ordinal classification methods achieve a better balance between the performances of all classes.

The imbalanced nature of some real-world data is one of the current challenges for machine learning researchers. One common approach oversamples the minority class through convex combination of its patterns. We explore the general idea of synthetic oversampling in the feature space induced by a kernel function (as opposed to input space). If the kernel function matches the underlying problem, the classes will be linearly separable and synthetically generated patterns will lie on the minority class region. Since the feature space is not directly accessible, we use the empirical feature space (EFS) (a Euclidean space isomorphic to the feature space) for oversampling purposes. The proposed method is framed in the context of support vector machines, where the imbalanced data sets can pose a serious hindrance. The idea is investigated in three scenarios: 1) oversampling in the full and reduced-rank EFSs; 2) a kernel learning technique maximizing the data class separation to study the influence of the feature space structure (implicitly defined by the kernel function); and 3) a unified framework for preferential oversampling that spans some of the previous approaches in the literature. We support our investigation with extensive experiments over 50 imbalanced data sets.

In this letter, we explore the idea of modeling slack variables in support vector machine (SVM) approaches. The study is motivated by SVM+, which models the slacks through a smooth correcting function that is determined by additional (privileged) information about the training examples not available in the test phase. We take a closer look at the meaning and consequences of smooth modeling of slacks, as opposed to determining them in an unconstrained manner through the SVM optimization program. To better understand this difference we only allow the determination and modeling of slack values on the same information--that is, using the same training input in the original input space. We also explore whether it is possible to improve classification performance by combining (in a convex combination) the original SVM slacks with the modeled ones. We show experimentally that this approach not only leads to improved generalization performance but also yields more compact, lower-complexity models. Finally, we extend this idea to the context of ordinal regression, where a natural order among the classes exists. The experimental results confirm principal findings from the binary case.

The classification of patterns into naturally ordered labels is referred to as ordinal regression or ordinal classification. Usually, this classification setting is by nature highly imbalanced, because there are classes in the problem that are a priori more probable than others. Although standard over-sampling methods can improve the classification of minority classes in ordinal classification, they tend to introduce severe errors in terms of the ordinal label scale, given that they do not take the ordering into account. A specific ordinal over-sampling method is developed in this paper for the first time in order to improve the performance of machine learning classifiers. The method proposed includes ordinal information by approaching over-sampling from a graph-based perspective. The results presented in this paper show the good synergy of a popular ordinal regression method (a reformulation of support vector machines) with the graph-based proposed algorithms, and the possibility of improving both the classification and the ordering of minority classes. A cost-sensitive version of the ordinal regression method is also introduced and compared with the over-sampling proposals, showing in general lower performance for minority classes.

The classification of patterns into naturally ordered labels is referred to as ordinal regression. This paper proposes an ensemble methodology specifically adapted to this type of problem, which is based on computing different classification tasks through the formulation of different order hypotheses. Every single model is trained in order to distinguish between one given class (k) and all the remaining ones, while grouping them in those classes with a rank lower than k , and those with a rank higher than k. Therefore, it can be considered as a reformulation of the well-known one-versus-all scheme. The base algorithm for the ensemble could be any threshold (or even probabilistic) method, such as the ones selected in this paper: kernel discriminant analysis, support vector machines and logistic regression (LR) (all reformulated to deal with ordinal regression problems). The method is seen to be competitive when compared with other state-of-the-art methodologies (both ordinal and nominal), by using six measures and a total of 15 ordinal datasets. Furthermore, an additional set of experiments is used to study the potential scalability and interpretability of the proposed method when using LR as base methodology for the ensemble.

Ordinal classification refers to classification problems in which the classes have a natural order imposed on them because of the nature of the concept studied. Some ordinal classification approaches perform a projection from the input space to one-dimensional (latent) space that is partitioned into a sequence of intervals (one for each class). Class identity of a novel input pattern is then decided based on the interval its projection falls into. This projection is trained only indirectly as part of the overall model fitting. As with any other latent model fitting, direct construction hints one may have about the desired form of the latent model can prove very useful for obtaining high-quality models. The key idea of this letter is to construct such a projection model directly, using insights about the class distribution obtained from pairwise distance calculations. The proposed approach is extensively evaluated with 8 nominal and ordinal classifiers methods, 10 real-world ordinal classification data sets, and 4 different performance measures. The new methodology obtained the best results in average ranking when considering three of the performance metrics, although significant differences are found for only some of the methods. Also, after observing other methods of internal behavior in the latent space, we conclude that the internal projections do not fully reflect the intraclass behavior of the patterns. Our method is intrinsically simple, intuitive, and easily understandable, yet highly competitive with state-of-the-art approaches to ordinal classification.

In this paper, two neural network threshold ensemble models are proposed for ordinal regression problems. For the first ensemble method, the thresholds are fixed a priori and are not modified during training. The second one considers the thresholds of each member of the ensemble as free parameters, allowing their modification during the training process. This is achieved through a reformulation of these tunable thresholds, which avoids the constraints they must fulfill for the ordinal regression problem. During training, diversity exists in different projections generated by each member is taken into account for the parameter updating. This diversity is promoted in an explicit way using a diversity-encouraging error function, extending the well-known negative correlation learning framework to the area of ordinal regression, and inheriting many of its good properties. Experimental results demonstrate that the proposed algorithms can achieve competitive generalization performance when considering four ordinal regression metrics.

This paper proposes a novel algorithm for ordinal classification based on combining ensemble techniques and discriminant analysis. The proposal is applied to a real application of liver transplantation, where the objective is to predict survival rates of the graft. Ordinal classification is used for this problem because the classes are defined by the following temporal order: 1) failure of the graft before the first 15 days after transplantation, 2) failure between 15 days and 3 months, 3) failure between 3 months and one year, and 4) no failure presented (taking into account that the patient follow-up is up to one year after the transplantation). When compared to other state-of-the-art classifiers like AdaBoost, EBC(SVM) or KDLOR, the proposed algorithm is shown to be competitive. The models obtained could allow medical experts to predict survival rates without knowing exactly the number of days the transplanted organ survived.

This paper presents a structural distance-based crossover for neural network classifiers, which is applied as part of a Memetic Algorithm (MA) for evolving simultaneously the structure and weights of neural network models applied to multiclass problems. Previous researchers have shown that this simultaneous evolution is a way to avoid the noisy fitness evaluation. The MA incorporates a crossover operator that shows to be useful for ameliorating the permutation problem of the network representation (i.e. different genotypes can be used to represent the same neural network phenotype), increasing the structural diversity of the individuals and improving the accuracy of the results. Instead of a recombination probability, the crossover operator considers a similarity parameter (the minimum structural distance), which allows to maintain a trade-off between global and local search. The neural network models selected in this work are the product-unit neural networks (PUNNs), due to their increasing relevance in those classification problems which show a high order relationship between the input variables. The proposed MA is intended to reduce the possible overtraining problems which can raise in some datasets for this kind of models. The evolutionary system is applied to eight classification benchmarks and the results of an analysis of variance contrast (ANOVA) show the effectiveness of the structural-based crossover operator and the capacity of our algorithm to obtain evolved PUNNs with a higher classification accuracy than those obtained using other evolutionary techniques. On the other hand, the results obtained are compared with popular effective machine learning classification methods, resulting in a competitive performance.

There are many metrics available to measure the goodness of a classifier when working with ordinal datasets. These measures are divided into product-moment and association metrics. In this paper, the behavior of several metrics is studied in different situations. In addition, two new measures associated with an ordinal classifier are defined: the maximum and the minimum mean absolute error of all the classes. From the results of this comparison, a pair of metrics is selected (one associated to the overall error and another one to the error of the class with lowest level of classification) to guide the evolution of a multi-objective evolutionary algorithm, obtaining good results in generalization on ordinal datasets.

This paper evaluates the performance of different classifiers when predicting wind speed from synoptic pressure patterns. The prediction problem has been formulated as a classification problem, where the different classes are associated to four values in an ordinal scale. The problem is relevant for long term wind speed prediction and also for wind speed reconstruction in areas (mainly wind farms) where there are not direct wind measures available. The results obtained in this paper present the Support Vector Machine as the best tested classifier for this task. In addition, the use of the intrinsic ordering information of the problem is shown to improve classifier performance.

In this paper we apply and test a recent ordinal algorithm for classification (Kernel Discriminant Learning Ordinal Regression, KDLOR), in order to recognize a group of geographically close spatial units with a similar prevalence pattern significantly high (or low), which are called hot-spots (or cold-spots). Different spatial analysis techniques have been used for studying geographical distribution of a specific illness in mental health-care because it could be useful to organize the spatial distribution of health-care services. Ordinal classification is used in this problem because the classes are: spatial unit with depression, spatial unit which could present depression and spatial unit where there is not depression. It is shown that the proposed method is capable of preserving the rank of data classes in a projected data space for this database. In comparison to other standard methods like C4.5, SVMRank, Adaboost, and MLP nominal classifiers, the proposed KDLOR algorithm is shown to be competitive.

Ordinal classification problems are an active research area in the machine learning community. Many previous works adapted state-of-art nominal classifiers to improve ordinal classification so that the method can take advantage of the ordinal structure of the dataset. However, these method improvements often rely upon a complex mathematical basis and they usually are attached to the training algorithm and model. This paper presents a novel method for generally adapting classification and regression models, such as artificial neural networks or support vector machines. The ordinal classification problem is reformulated as a regression problem by the reconstruction of a numerical variable which represents the different ordered class labels. Despite the simplicity and generality of the method, results are competitive in comparison with very specific methods for ordinal regression.

This paper proposes a hybrid multilogistic methodology, named logistic regression using initial and radial basis function (RBF) covariates. The process for obtaining the coefficients is carried out in three steps. First, an evolutionary programming (EP) algorithm is applied, in order to produce an RBF neural network (RBFNN) with a reduced number of RBF transformations and the simplest structure possible. Then, the initial attribute space (or, as commonly known as in logistic regression literature, the covariate space) is transformed by adding the nonlinear transformations of the input variables given by the RBFs of the best individual in the final generation. Finally, a maximum likelihood optimization method determines the coefficients associated with a multilogistic regression model built in this augmented covariate space. In this final step, two different multilogistic regression algorithms are applied: one considers all initial and RBF covariates (multilogistic initial-RBF regression) and the other one incrementally constructs the model and applies cross validation, resulting in an automatic covariate selection [simplelogistic initial-RBF regression (SLIRBF)]. Both methods include a regularization parameter, which has been also optimized. The methodology proposed is tested using 18 benchmark classification problems from well-known machine learning problems and two real agronomical problems. The results are compared with the corresponding multilogistic regression methods applied to the initial covariate space, to the RBFNNs obtained by the EP algorithm, and to other probabilistic classifiers, including different RBFNN design methods [e.g., relaxed variable kernel density estimation, support vector machines, a sparse classifier (sparse multinomial logistic regression)] and a procedure similar to SLIRBF but using product unit basis functions. The SLIRBF models are found to be competitive when compared with the corresponding multilogistic regression methods and the RBFEP method. A measure of statistical significance is used, which indicates that SLIRBF reaches the state of the art.

The selection of a particular neural network model belonging to the Pareto front is a problem that exists in all multi-objective algorithms. This paper proposes a novel solution to this problem based on a linear combination of the outputs of the two extremes in the Pareto front, which form an ensemble. The decision support TOPSIS method is used to determine which linear combination creates the best ensemble. This analysis selects the most representative individual that performs better in generalization than the extremes of the Pareto front do.

This paper studies the suitability of Extreme Learning Machines (ELM) for resolving bioinformatic and biomedical classification problems. In order to test their overall performance, an experimental study is presented based on five gene microarray datasets found in bioinformatic and biomedical domains. The Fast Correlation-Based Filter (FCBF) was applied in order to identify salient expression genes among the thousands of genes in microarray data that can directly contribute to determining the class membership of each pattern. The results confirm that the ELM classifier is a promising candidate for improving Accuracy and Minimum Sensitivity.

This paper proposes a multiclassification algorithm using multilayer perceptron neural network models. It tries to boost two conflicting main objectives of multiclassifiers: a high correct classification rate level and a high classification rate for each class. This last objective is not usually optimized in classification, but is considered here given the need to obtain high precision in each class in real problems. To solve this machine learning problem, we use a Pareto-based multiobjective optimization methodology based on a memetic evolutionary algorithm. We consider a memetic Pareto evolutionary approach based on the NSGA2 evolutionary algorithm (MPENSGA2). Once the Pareto front is built, two strategies or automatic individual selection are used: the best model in accuracy and the best model in sensitivity (extremes in the Pareto front). These methodologies are applied to solve 17 classification benchmark problems obtained from the University of California at Irvine (UCI) repository and one complex real classification problem. The models obtained show high accuracy and a high classification rate for each class.

This paper proposes a hybrid multilogistic model, named multilogistic regression using initial and radial basis function covariates (MLRIRBF). The process for obtaining the coefficients is carried out in several steps. First, an evolutionary programming (EP) algorithm is applied, aimed to produce a RBF neural network (RBFNN) with a reduced number of RBF transformations and the simplest structure possible. Then, the input space is transformed by adding the nonlinear transformations of the input variables given by the RBFs of the best individual in the last generation. Finally, a maximum likelihood optimization method determines the coefficients associated with a multilogistic regression model built on this transformed input space. In this final step, two different multilogistic regression algorithms are applied, one that considers all initial and RBF covariates (MLRIRBF) and another one that incrementally constructs the model and applies cross-validation, resulting in an automatic covariate selection (MLRIRBF*). The methodology proposed is tested using six benchmark classification problems from well-known machine learning problems. The results are compared with the corresponding multilogistic regression methodologies applied over the initial input space, to the RBFNNs obtained by the EP algorithm (RBFEP) and to other competitive machine learning techniques. The MLRIRBF* models are found to be better than the corresponding multilogistic regression methodologies and the RBFEP method for almost all datasets, and obtain the highest mean accuracy rank when compared to the rest of methods in all datasets.

This paper proposes a novelty neural network model by using generalized kernel functions for the hidden layer of a feed forward network (Generalized Radial Basis Functions, GRBF), where the architecture, weights and node typology are learned through an evolutionary programming algorithm. This new kind of model is compared with the corresponding models with standard hidden nodes: Product Unit Neural Networks (PUNN), Multilayer Perceptrons (MLP) and the RBF neural networks. The methodology proposed is tested using six benchmark classification datasets from well-known machine learning problems. Generalized basis functions are found to present a better performance than the other standard basis functions for the task of classification.

In this paper, we propose a Memetic Algorithm (MA) for classifier optimization based on a clustering method that applies the k-means algorithm over a specific derived space. In this space, each classifier or individual is represented by the set of the accuracies of the classifier for each class of the problem. The proposed sensitivity clustering is able to obtain groups of individuals that perform similarly for the different classes. Then, a representative of each group is selected and it is improved by a local search procedure. This method is applied in specific stages of the evolutionary process. The sensitivity clustering process is compared to a clustering process applied over the n-dimensional space that represent the behaviour of the classifier over each training pattern, where $n$ is the number of patterns. This second method clearly results in a higher computational cost. The comparison is performed in ten imbalanced datasets, including the minimun sensitivity results (i.e. the accuracy for the worst classified class). The results indicate that, although in general the differences are not significant, the sensitivity clustering obtains the best perfomance for almost all datasets both in accuracy and minimum sensitivity, involving a lower computational demand.

This paper proposes a neural network model for wind speed prediction, a very important task in wind parks management. Currently, several physical-statistical and artificial intelligence (AI) wind speed prediction models are used to this end. A recently proposed hybrid model is based on hybridizations of global and mesoscale forecasting systems, with a final downscaling step using a multilayer perceptron (MLP). In this paper, we test an alternative neural model for this final step of downscaling, in which projection hyperbolic tangent units (HTUs) are used within feed forward neural networks. The architecture, weights and node typology of the HTU-based network are learnt using a hybrid evolutionary programming algorithm. This new methodology is tested over a real problem of wind speed forecasting, in which we show that our method is able to improve the performance of previous MLPs, obtaining an interpretable model of final regression for each turbine in the wind park.

Performance evaluation is decisive when improving classifiers. Accuracy alone is insufficient because it cannot capture the myriad of contributing factors differentiating the performances of two different classifiers and approaches based on a multi-objective perspective are hindered by the growing of the Pareto optimal front as the number of classes increases. This paper proposes a new approach to deal with multi-class problems based on the accuracy (C) and minimum sensitivity (S) given by the lowest percentage of examples correctly predicted to belong to each class. From this perspective, we compare different fitness functions (accuracy, C , entropy, E , sensitivity, S , and area, A ) in an evolutionary scheme. We also present a two stage evolutionary algorithm with two sequential fitness functions, the entropy for the first step and the area for the second step. This methodology is applied to solve six benchmark classification problems. The two-stage approach obtains promising results and achieves a high classification rate level in the global dataset with an acceptable level of accuracy for each class.

The main objective of this work is to automatically design neural network models with sigmoidal basis units for classification tasks, so that classifiers are obtained in the most balanced way possible in terms of CCR and Sensitivity (given by the lowest percentage of examples correctly predicted to belong to each class). We present a Memetic Pareto Evolutionary NSGA2 (MPENSGA2) approach based on the Pareto-NSGAII evolution (PNSGAII) algorithm. We propose to augmente it with a local search using the improved Rprop—IRprop algorithm for the prediction of growth/no growth of L. monocytogenes as a function of the storage temperature, pH, citric (CA) and ascorbic acid (AA). The results obtained show that the generalization ability can be more efficiently improved within a framework that is multi-objective instead of a within a single-objective one.

Logistic Regression (LR) has become a widely used and accepted method to analyse binary or multiclass outcome variables, since it is a flexible tool that can predict the probability for the state of a dichotomous variable. A recently proposed LR method is based on the hybridisation of a linear model and Evolutionary Product-Unit Neural Network (EPUNN) models for binary classification. This produces a high number of coefficients, so two different methods for reducing the number of initial or PU covariates are proposed in this paper, both based on the Wald test. The first method is a two-step Backward Search (BS) method and the second is based on the standard Simulated Annealing (SA) heuristic. In this study, we used aerial imagery taken in mid-May to evaluate the potential of two different combinations of LR and EPUNN (LR using PUs (LRPU), as well as LR using Initial covariates and PUs (LRIPU)) and the two proposed methods for selecting variables in the final models (BS and SA) for discriminating Ridolfia segetum patches (one of the most dominant, competitive and persistent weed in sunflower crops) in one naturally infested field of southern Spain. Then, we compared the performance of these methods to six recent classification models, our proposals obtaining a competitive performance and a lower number of coefficients.

We propose a classification method based on a special class of feed-forward neural network, namely product-unit neural networks. They are based on multiplicative nodes instead of additive ones, where the nonlinear basis functions express the possible strong interactions among the variables. We apply an evolutionary algorithm to determine the basic structure of the product-unit model and to estimate the coefficients of the model. The empirical results show that the proposed model is very promising in terms of classification accuracy, yielding a state-of-the-art performance.