We evaluate how well large language model embeddings represent continuous numerical values across different precisions and ranges. Using linear models and principal component analysis on models from major providers, we show that while embeddings can reconstruct numbers with high fidelity (R2 ≥ 0.95), they introduce substantial noise, with principal components explaining less than 40% of embedding variance. Performance degrades with increasing decimal precision and mixed-sign values, revealing fundamental limitations in how these models encode numerical information.

The principal benefit of unsupervised representation learning is that a pre-trained model can be fine-tuned where data or labels are scarce. Existing approaches for graph representation learning are domain specific, maintaining consistent node and edge features across the pre-training and target datasets. This has precluded transfer to multiple domains. We present Topology Only Pre-Training (ToP), a graph pre-training method based on node and edge feature exclusion. We show positive transfer on evaluation datasets from multiple domains, including domains not present in pre-training data, running directly contrary to assumptions made in contemporary works. On 75% of experiments, ToP models perform significantly ([Formula: see text]) better than a supervised baseline. Performance is significantly positive on 85.7% of tasks when node and edge features are used in fine-tuning. We further show that out-of-domain topologies can produce more useful pre-training than in-domain. Under ToP we show better transfer from non-molecule pre-training, compared to molecule pre-training, on 79% of molecular benchmarks. Against the limited set of other generalist graph models ToP performs strongly, including against models with many orders of magnitude larger. These findings show that ToP opens broad areas of research in both transfer learning on scarcely populated graph domains and in graph foundation models. Supplementary Information: The online version contains supplementary material available at 10.1007/s10618-026-01210-1.

Abstract A key aspect of temporal domains is the ability to make predictions multiple time-steps into the future, a process known as multi-step forecasting (MSF). At the core of this process is selecting a forecasting strategy; however, with no existing frameworks to map out the space of strategies, practitioners are left with ad-hoc methods for strategy selection. In this work, we propose Stratify , a parameterised framework that addresses multi-step forecasting, unifying existing strategies and introducing novel, improved strategies. We evaluate Stratify on 18 benchmark datasets, five function classes, and short to long forecast horizons (10, 20, 40, 80) in the univariate setting. In over 84% of 1080 experiments, novel strategies in Stratify improved performance compared to all existing ones. Importantly, we find that no single strategy consistently outperforms others in all task settings, highlighting the need for practitioners to explore the Stratify space to carefully search and select forecasting strategies based on task-specific requirements. Our results are the most comprehensive benchmarking of known and novel forecasting strategies. We share the code ( https://github.com/zs18656/stratify_unifying_MSF ) to reproduce our results.

Wake steering is a form of wind farm flow control in which upstream turbines are deliberately yawed to misalign with the incoming wind in order to reduce the impact of wakes on downstream turbines. This technique can give a net increase in the power generated by an array of turbines compared to standard greedy control where each turbine acts for its own benefit by aligning with the incoming wind. However, optimising the set-points of multiple turbines under varying wind conditions can be prohibitively complex for traditional, white-box models. Reinforcement Learning (RL) agents learn optimal long-term behaviours through “trial-and-error”, making them suited to controlling arrays of wind turbines under changing wind conditions for maximum farm power. Related works applying RL to this problem have tended to concentrate on either single wind directions or ranges up to around ± 1 0 ∘ . Here, the Deep Deterministic Policy Gradient algorithm has been used to train RL agents to control a nine-turbine array to implement wake steering under multiple wind directions between ± 4 5 ∘ . While the agents were trained on steady-state (time-averaged) wind flow data, the performance of the final agent was tested on “quasi-dynamic” wind flow with varying wind direction. Under these conditions, the final agent achieved an average of 7% more power than greedy control per direction. This agent was then used to control the wind farm under a smaller subset of directions including many not seen during training, gaining on average 17% additional farm power per direction compared to greedy control. • Reinforcement learning agents were trained to maximise wind farm power. • Agents were trained to control the farm under a wide range of wind conditions. • A robust training configuration evolved through hyperparameter investigations. • Agent achieved on average 7% increase in farm power on conditions used in training. • Agent achieved on average 17% increase in farm power on new conditions.

Preventing a Machine Learning (ML) predictor from making an unreliable prediction is extremely important in sensitive application domains, such as health contexts. In this sense, strategies based on the reject option have been increasingly explored. However, few studies explore the ability of meta-learning to inspect the errors of a base predictor under analysis, in such a way to generalize when the predictor is confident or not. Therefore, the current paper proposes a novel solution for ML with reject option based on the combination of meta-learning and novelty detection. The proposal addresses two distinct situations where a prediction should be rejected. First, novel detection is adopted to identify out-of-distribution instances, i.e., instances that significantly differ from those ones adopted to train the base predictor. Second, meta-learning is adopted to detect instances in regions of data where the base model has shown poor predictive performance during its evaluation. Such instances mainly lie in areas of class overlap or noisy regions in the training data. The results in experiments on synthetic and real data showed the superiority of the solution compared to those based only on meta-learning (aka without novelty detection) and those based on classifier confidence.

Understanding the difficulty of individual instances in a classification problem is important to define the limits of learning performance in the problem. Previous works are devoted to measuring Instance Hardness (IH), while solutions for explaining IH are still not deeply investigated. In this paper, we rely on using assessor models and eXplanaible AI (XAI) techniques to predict and explain IH. Many XAI techniques have been developed in the literature to explain the predictions of Machine Learning (ML) models. In our work, we are focused on explaining the difficulty of instances. Given a classification dataset, we trained and evaluated a pool of diverse ML models to measure the IH of each instance. Then, we trained an assessor model to predict the IH based on the instances’ features. Once the assessor is built, its predictions (i.e., the expected IH) can be explained using XAI techniques. In our experiments, we produced Partial Dependence Plots (PDP) to inspect the marginal effect of specific features on the IH predicted by the assessor. From the PDPs, we could check how IH is distributed along the instances’ features in a problem, and more specifically, we could visualize areas of high expected predictive difficulty.

Abstract This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.

Item Response Theory (IRT) is a tool developed in psychometrics to measure latent abilities of human respondents based on their responses to items with different levels of difficulty. Recently, IRT has been applied to evaluation in AI, by treating the algorithms as respondents and the AI tasks as items. Particularly in machine learning, IRT has been applied for evaluation of classifiers based on their predictions to each test instance. Based on a matrix of responses (classifiers vs instances), the IRT model estimates the latent difficulty and discrimination of each instance, as well as the ability of each classifier, in such a way that a classifier receives high ability value when it tends to correctly classify the most difficult instances. The IRT models previously adopted for evaluation in classification are not directly applied for regression, since they rely on dichotomous responses (i.e., a response has to be either correct or incorrect). In this paper we propose a new IRT model, particularly designed for dealing with nonnegative unbounded responses, which is adequate for modelling the absolute errors of regression algorithms. In the proposed model, responses follow a gamma distribution, parameterised according to respondents' abilities and items' difficulty and discrimination parameters. The proposed parameterisation results in item characteristic curves with more flexible shapes compared to the logistic curves widely adopted in IRT. The proposed model was evaluated with diverse regression algorithms and two benchmark datasets, one synthetic and one real. Useful insights were derived by inspecting regions in these datasets that present different levels of difficulty and discrimination.

Item Response Theory (IRT) aims to assess latent abilities of respondents based on the correctness of their answers in aptitude test items with different difficulty levels. In this paper, we propose the $\beta^3$-IRT model, which models continuous responses and can generate a much enriched family of Item Characteristic Curves. In experiments we applied the proposed model to data from an online exam platform, and show our model outperforms a more standard 2PL-ND model on all datasets. Furthermore, we show how to apply $\beta^3$-IRT to assess the ability of machine learning classifiers.This novel application results in a new metric for evaluating the quality of the classifier’s probability estimates, based on the inferred difficulty and discrimination of data instances.

Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax. Experiments demonstrate improved probabilistic predictions according to multiple measures (confidence-ECE, classwise-ECE, log-loss, Brier score) across a wide range of datasets and classifiers. Parameters of the learned Dirichlet calibration map provide insights to the biases in the uncalibrated model.

For optimal decision making under variable class distributions and misclassification costs a classifier needs to produce well-calibrated estimates of the posterior probability. Isotonic calibration is a powerful non-parametric method that is however prone to overfitting on smaller datasets; hence a parametric method based on the logistic curve is commonly used. While logistic calibration is designed for normally distributed per-class scores, we demonstrate experimentally that many classifiers including Naive Bayes and Adaboost suffer from a particular distortion where these score distributions are heavily skewed. In such cases logistic calibration can easily yield probability estimates that are worse than the original scores. Moreover, the logistic curve family does not include the identity function, and hence logistic calibration can easily uncalibrate a perfectly calibrated classifier. In this paper we solve all these problems with a richer class of calibration maps based on the beta distribution. We derive the method from first principles and show that fitting it is as easy as fitting a logistic curve. Extensive experiments show that beta calibration is superior to logistic calibration for Naive Bayes and Adaboost.

We introduce a powerful technique to make classifiers more reliable and versatile. Background Check equips classifiers with the ability to assess the difference of unlabelled test data from the training data. In particular, Background Check gives classifiers the capability to (i) perform cautious classification with a reject option, (ii) identify outliers, and (iii) better assess the confidence in their predictions. We derive the method from first principles and consider four particular relationships between background and foreground distributions. One of these assumes an affine relationship with two parameters, and we show how this bivariate parameter space naturally interpolates between the above capabilities. We demonstrate the versatility of the approach by comparing it experimentally with published special-purpose solutions for outlier detection and confident classification on 41 benchmark datasets. Results show that Background Check can match and in many cases surpass the performances of specialised approaches.

Symbolic data analysis deals with complex data types, capable of modeling internal data variability and imprecise data. This paper introduces two Fuzzy Learning Vector Quantization algorithms for interval symbolic data. One algorithm employs an interval Euclidean distance. The second uses a weighted interval Euclidean distance to try and achieve a better performance of classification when the data set is composed of classes with varying sizes, shapes and structures. The algorithms are evaluated for their performances with synthetic and real data sets. This paper aims at contributing to the area of Supervised Learning within Symbolic Data Analysis.

This paper presents learning vector quantization classifiers with adaptive distances. The classifiers furnish discriminant class regions from the input data set that are represented by prototypes. In order to compare prototypes and patterns, the classifiers use adaptive distances that change at each iteration and are different from one class to another or from one prototype to another. Experiments with real and synthetic data sets demonstrate the usefulness of these classifiers.