Modern evaluation of Legal QA systems is shifting from terminal accuracy toward processaware analyses of model reasoning.We propose a diagnostic framework grounded in monotonic pedagogical scaffolding, where language models receive gold-standard, caserelevant information across stages aligned with the canonical legal framework FIRAC -Facts, Issue, Rules, Application, Conclusion.By strictly adding solution-relevant content at each step, we introduce a controlled monotonic intervention that allows for the evaluation of reasoning trajectories rather than isolated outcomes.This longitudinal design enables the introduction of two transition-based diagnostics: Errorsto-Success (E2S) quantifies the guidance required to reach correctness, while Success-to-Errors (S2E) measures the fragility of that correctness under additional structure.These local patterns define a global robustness criterion termed Stable Accuracy, which credits a response only if the model maintains correctness throughout all scaffolding stages and enforces a higher bar for correctness by distinguishing sustained reasoning from transient patterns.We instantiate the framework on 3,123 Brazilian Bar Exam questions paired with expertannotated explanations.Our findings reveal model instability patterns hidden from accuracy-only metrics and demonstrate that terminal accuracy systematically overestimates legal reasoning competence.To test the robustness of our diagnostics, we also evaluate a majority-vote aggregation across multiple reasoning samples, finding that the observed instability patterns persist under this stronger inference setting.Furthermore, principal component analysis indicates that legal domains cluster into distinct regions, suggesting systematic differences in reasoning demands across domains.While focused on the legal domain, our evaluation protocol is generalizable to any task with a staged reasoning structure.

The integration of Large Language Models (LLMs) into medicine presents both great opportunities and significant challenges, particularly in ensuring these models are accurate, reliable, and safe. While LLMs have shown impressive capabilities in understanding and generating human language, their application in the medical domain requires careful evaluation due to the critical nature of medical applications which are inherently linked to patient life and health. Current evaluations of LLMs in medicine are often fragmented and insufficient, with a lack of standardized performance metrics, limited use of real patient data, and insufficient attention to important applications, such as documentation, education, and research. Furthermore, traditional NLP-based evaluations are often inadequate for assessing the text generated by LLMs. Therefore, a robust evaluation is essential to ensure the responsible and effective use of LLMs in medical settings, and to address the inherent challenges associated with their implementation. This paper explores the various dimensions of LLM evaluation in the medical domain, proposes a new taxonomy for categorizing medical applications, and discusses directions for future research in this critical area.

In public procurement, establishing reference prices is essential to guide competitors in setting product prices. Group-purchased products, which are not standardized by default, are necessary to estimate reference prices. Text clustering techniques can be used to group similar items based on their descriptions, enabling the definition of reference prices for specific products or services. However, selecting an appropriate representation for text is challenging. This paper introduces a framework for text cleaning, extraction, and representation. We test eight distinct sentence representations tailored for public procurement item descriptions. Among these representations, we propose an approach that captures the most important components of item descriptions. Through extensive evaluation of a dataset comprising over 2 million items, our findings show that using sophisticated supervised methods to derive vectors for unsupervised tasks offers little advantages over leveraging unsupervised methods. Our results also highlight that domain-specific contextual knowledge is crucial for representation improvement.

This paper explores using algorithmic recourse as a tool in mental healthcare. Algorithmic recourse provides explanations and recommendations to individuals who want to reverse a machine learning prediction and has been widely used in various domains such as finance and marketing. However, its application in mental healthcare has been restricted. This paper addresses this by examining the potential benefits and challenges of using algorithmic recourse in mental healthcare, specifically in how changing one's behavior may affect their quality of life and well-being. The paper proposes a new classification-based framework for algorithmic recourse in mental healthcare. The proposed framework considers both observed and latent variables to account for the individuality of individuals and provides a more comprehensive understanding of mental health outcomes. The research results can provide valuable insights for future work and help bridge the gap between machine learning and mental healthcare.

In recent years, vision transformers have become the state-of-the-art architecture for computer vision tasks in the medical domain. However, their robustness to health out- of-distribution (OOD) data remains an open question. In this paper, we focus on the scenario of diabetic retinopathy diagnosis and conduct an empirical analysis of the robustness of vision transformers to health OOD data. Our experiments are conducted on a large medical image dataset of diabetic retinopathy, and we evaluate the models' performance on both in-distribution and OOD data. Our results show that vision transformers are highly susceptible to health OOD data, leading to high-confidence predictions even when the data is far from the distribution seen during training. This work provides a comprehensive analysis of the robustness of vision transformers to health OOD data in diabetic retinopathy diagnosis, highlighting the importance of considering this aspect when using these models for medical image analysis.

Model interpretability/explainability is increasingly a concern when applying machine learning to real-world problems. In this article, we are interested in explaining regression models by exploiting prototypes, which are exemplar cases in the problem domain. Previous works focused on finding prototypes that are representative of all training data but ignore the model predictions, i.e., they explain the data distribution but not necessarily the predictions. We propose a two-level model-agnostic method that considers prototypes to provide global and local explanations for regression problems and that account for both the input features and the model output. M-PEER (Multiobjective Prototype-basEd Explanation for Regression) is based on a multi-objective evolutionary method that optimizes both the error of the explainable model and two other “semantics”-based measures of interpretability adapted from the context of classification, namely, model fidelity and stability. We compare the proposed method with the state-of-the-art method based on prototypes for explanation—ProtoDash—and with other methods widely used in correlated areas of machine learning, such as instance selection and clustering. We conduct experiments on 25 datasets, and results demonstrate significant gains of M-PEER over other strategies, with an average of 12% improvement in the proposed metrics (i.e., model fidelity and stability) and 17% in root mean squared error (RMSE) when compared to ProtoDash.

Length of stay (LoS) in the intensive care unit (ICU) is a general outcome measure used as an indicator of both quality of care and resource use. Existing LoS prediction methods usually model the problem as a binary classification or regression problem. This paper proposes to address this problem under a sequential learning framework using an embedded representation of patients. First, we exploit electronic health records (EHR) to learn patient distributed representation. Second, we formulate the LoS prediction task as a Deep Bayesian contextual bandit problem, which sequentially predicts LoS based on contextual distributed information about patients, with the goal of maximizing overall prediction outcomes. Experimental results show that our model has superior performance than other sequential learning approaches in the LoS task using the publicly available MIMIC-III ICU dataset. We also evaluate the robustness of our framework under different patient representations.

The outgrowing application of machine learning methods has raised a discussion in the artificial intelligence community on model transparency. In the center of this discussion is the question of model explanation and interpretability. The genetic programming (GP) community has systematically pointed out as one of the major advantages of GP the fact that it produces models that can be interpreted by humans. However, as other interpretable supervised models, the more complex the model becomes, the less interpretable it is. This work focuses on post-hoc interpretability of GP for symbolic regression. This approach does not explain the process followed by a model to reach a decision. Instead, it justifies the predictions it makes. The proposed approach, named Explanation by Local Approximation (ELA), is simple and model agnostic: it finds the nearest neighbors of the point we want to explain and performs a linear regression using this subset of points. The coefficients of this linear regression are then used to generate a local explanation to the model. Results show that the errors of ELA are similar to those of the regression performed with all points. It also shows that simple visualizations can provide insights to the users about the most relevant attributes.

Recommender Systems are tools designed to help users find relevant information from the myriad of content available online. They work by actively suggesting items that are relevant to users according to their historical preferences or observed actions. Among recommender systems, top- N recommenders work by suggesting a ranking of N items that can be of interest to a user. Although a significant number of top- N recommenders have been proposed in the literature, they often disagree in their returned rankings, offering an opportunity for improving the final recommendation ranking by aggregating the outputs of different algorithms. Rank aggregation was successfully used in a significant number of areas, but only a few rank aggregation methods have been proposed in the recommender systems literature. Furthermore, there is a lack of studies regarding rankings’ characteristics and their possible impacts on the improvements achieved through rank aggregation. This work presents an extensive two-phase experimental analysis of rank aggregation in recommender systems. In the first phase, we investigate the characteristics of rankings recommended by 15 different top- N recommender algorithms regarding agreement and diversity. In the second phase, we look at the results of 19 rank aggregation methods and identify different scenarios where they perform best or worst according to the input rankings’ characteristics. Our results show that supervised rank aggregation methods provide improvements in the results of the recommended rankings in six out of seven datasets. These methods provide robustness even in the presence of a big set of weak recommendation rankings. However, in cases where there was a set of non-diverse high-quality input rankings, supervised and unsupervised algorithms produced similar results. In these cases, we can avoid the cost of the former in favor of the latter.

Recommender systems help users to overcome the information overload problem by selecting relevant items according to their preferences. This paper deals with the problem of rank aggregation in recommender systems, where we want to generate a single consensus ranking from a given set of input rankings generated by different recommendation algorithms. This problem is NP-hard, and hence the use of meta-heuristics to solve it is appealing. Although accurate suggestions are mandatory for effective recommender systems, other recommendation quality measures need to be taken into account for delivering high-quality suggestions. This paper proposes Multi-objective Evolutionary Rank Aggregation (MERA), a genetic programming algorithm following the concepts of SPEA2 that considers three measures when suggesting items to users, namely mean average precision, diversity, and novelty. The method was tested in 3 realworld recommendation datasets, and the results show MERA can indeed find a balance for these metrics while generating a diverse set of solutions to the problem. MERA was able to return solutions with improvements of up to 15% in diversity (for the Movielens 1M dataset) and 7% in novelty (for the Filmtrust dataset) while maintaining, or even improving, the values of precision.

Recommender systems are methods built to actively suggest personalized items to users based on their explicit declared preferences (ratings of movies in Netflix), or implicitly observed actions (purchase history). Although a great number of recommendation methods have been previously proposed in the literature, in many problems these methods present a high degree of disagreement in their recommendations. In this scenario, rank aggregation methods are an interesting solution. They can help finding a consensus on which items should be recommended to the user by taking into account the opinion of all available methods. In this direction, this paper proposes ERA (Evolutionary Rank Aggregation), a genetic programming method that outputs an aggregated ranking function built from information extracted from individual input rankings. ERA was tested in four large scale datasets, and obtained better results than other rank aggregation methods in three datasets, improving the results of mean average ranking precision in up to 9.5%.

Recommender systems are quickly becoming ubiquitous in applications such as e-commerce, social media channels, and content providers, among others, acting as an enabling mechanism designed to overcome the information overload problem by improving browsing and consumption experience. A typical task in many recommender systems is to output a ranked list of items, so that items placed higher in the rank are more likely to be interesting to the users. Interestingness measures include how accurate, novel, and diverse are the suggested items, and the objective is usually to produce ranked lists optimizing one of these measures. Suggesting items that are simultaneously accurate, novel, and diverse is much more challenging, since this may lead to a conflicting-objective problem, in which the attempt to improve a measure further may result in worsening other measures. In this article, we propose new approaches for multiobjective recommender systems based on the concept of Pareto efficiency—a state achieved when the system is devised in the most efficient manner in the sense that there is no way to improve one of the objectives without making any other objective worse off. Given that existing multiobjective recommendation algorithms differ in their level of accuracy, diversity, and novelty, we exploit the Pareto-efficiency concept in two distinct manners: (i) the aggregation of ranked lists produced by existing algorithms into a single one, which we call Pareto-efficient ranking, and (ii) the weighted combination of existing algorithms resulting in a hybrid one, which we call Pareto-efficient hybridization. Our evaluation involves two real application scenarios: music recommendation with implicit feedback (i.e., Last.fm) and movie recommendation with explicit feedback (i.e., MovieLens). We show that the proposed Pareto-efficient approaches are effective in suggesting items that are likely to be simultaneously accurate, diverse, and novel. We discuss scenarios where the system achieves high levels of diversity and novelty without compromising its accuracy. Further, comparison against multiobjective baselines reveals improvements in terms of accuracy (from 10.4% to 10.9%), novelty (from 5.7% to 7.5%), and diversity (from 1.6% to 4.2%).

Daily deals sites (DDSs), such as Groupon and LivingSocial, attract millions of customers in the hunt for products and services at significantly reduced prices. A typical approach to increase revenue is to send email messages featuring the deals of the day. Such daily messages, however, are usually not centered on the customers, instead, all registered users typically receive similar messages with almost the same deals. Traditional recommendation algorithms are innocuous in DDSs because: (i) most of the users are sporadic bargain hunters, and thus past preference data is extremely sparse, (ii) deals have a short living period, and thus data is extremely volatile, and (iii) user taste and interest may undergo temporal drifts. In order to address such particularly challenging scenario, we propose new algorithms for daily deals recommendation based on the explore-then-exploit strategy.Users are split into exploration and exploitation sets -- in the exploration set the users receive non-personalized messages and a co-purchase network is updated with user feedback for purchases of the day, while in the exploitation set the updated network is used for recommending personalized messages for the remaining users.A thorough evaluation of our algorithms using real data obtained from a large daily deals website in Brazil in contrast to state-of-the-art recommendation algorithms show gains in precision ranging from 18% to 34%.

Recommender systems are quickly becoming ubiquitous in many Web applications, including e-commerce, social media channels, content providers, among others. These systems act as an enabling mechanism designed to overcome the information overload problem by improving browsing and consumption experience. Crucial to the performance of a recommender system is the accuracy of the user profiles used to represent the interests of the users. In this proposal, we analyze three different aspects of user profiling: (i) selecting the most informative events from the interaction between users and the system, (ii) combining different recommendation algorithms to (iii) including trust-aware information in user profiles to improve the accuracy of recommender systems.

Performing accurate suggestions is an objective of paramount importance for effective recommender systems. Other important and increasingly evident objectives are novelty and diversity, which are achieved by recommender systems that are able to suggest diversified items not easily discovered by the users. Different recommendation algorithms have particular strengths and weaknesses when it comes to each of these objectives, motivating the construction of hybrid approaches. However, most of these approaches only focus on optimizing accuracy, with no regard for novelty and diversity. The problem of combining recommendation algorithms grows significantly harder when multiple objectives are considered simultaneously. For instance, devising multi-objective recommender systems that suggest items that are simultaneously accurate, novel and diversified may lead to a conflicting-objective problem, where the attempt to improve an objective further may result in worsening other competing objectives. In this paper we propose a hybrid recommendation approach that combines existing algorithms which differ in their level of accuracy, novelty and diversity. We employ an evolutionary search for hybrids following the Strength Pareto approach, which isolates hybrids that are not dominated by others (i.e., the so called Pareto frontier). Experimental results on two recommendation scenarios show that: (i) we can combine recommendation algorithms in order to improve an objective without significantly hurting other objectives, and (ii) we allow for adjusting the compromise between accuracy, diversity and novelty, so that the recommendation emphasis can be adjusted dynamically according to the needs of different users.

Content-targeted advertising, the task of automatically associating ads to a Web page, constitutes a key Web monetization strategy nowadays. Further, it introduces new challenging technical problems and raises interesting questions. For instance, how to design ranking functions able to satisfy conflicting goals such as selecting advertisements (ads) that are relevant to the users and suitable and profitable to the publishers and advertisers? In this paper we propose a new framework for associating ads with web pages based on Genetic Programming (GP). Our GP method aims at learning functions that select the most appropriate ads, given the contents of a Web page. These ranking functions are designed to optimize overall precision and minimize the number of misplacements. By using a real ad collection and web pages from a newspaper, we obtained a gain over a state-of-the-art baseline method of 61.7% in average precision. Further, by evolving individuals to provide good ranking estimations, GP was able to discover ranking functions that are very effective in placing ads in web pages while avoiding irrelevant ones.