Benchmarking is a fundamental practice in machine learning (ML) for comparing the performance of classification algorithms. However, traditional evaluation methods often overlook a critical aspect: the joint consideration of dataset complexity and an algorithm’s ability to generalize. Without this dual perspective, assessments may favor models that perform well on easy instances while failing to capture their true robustness. To address this limitation, this study introduces a novel evaluation methodology that combines Item Response Theory (IRT) with the Glicko-2 rating system, originally developed to measure player strength in competitive games. IRT assesses classifier ability based on performance over difficult instances, while Glicko-2 updates performance metrics—such as rating, deviation, and volatility—via simulated tournaments between classifiers. This combined approach provides a more robust and difficulty-aware measure of algorithm capability. A case study using the OpenML-CC18 benchmark showed that only 16.66% of the datasets are truly challenging and that a reduced subset with 50% of the original datasets offers comparable evaluation power. Among the algorithms tested, Random Forest achieved the highest ability score. The results highlight the importance of improving benchmark design by focusing on dataset quality and adopting evaluation strategies that reflect both difficulty and classifier proficiency.

Fitness Landscape Analysis (FLA) is one of the most effective approaches to understanding how the best solutions to an optimization problem are distributed within its search space. Several studies have been conducted with the aim of investigating the fitness landscape of AutoML (Automated Machine Learning) problems, in which the search space is composed of alternative pipelines for building ML models. However, FLA in this context presents significant challenges due to the inherent complexity of AutoML problems. Moreover, previous research indicates that traditional FLA metrics used to characterize the fitness landscapes are not suitable for problems involving AutoML. In this paper, we explore the relationship between the complexity characteristics of ML datasets and FLA metrics for AutoML. In fact, there is evidence in the literature that these characteristics actually exert a significant influence on AutoML’s search space. To date, previous authors have not yet investigated how the complexity characteristics of datasets actually impact the fitness landscape in AutoML. In our analysis, we identified statistically significant correlations between them in such a way that complexity measures can serve as more cost-effective proxies for FLA metrics.

Supervised machine learning algorithms usually require sufficient labeled data to perform well. However, obtaining this information can be challenging due to monetary and time constraints. As a possible solution, recent works have proposed the combination of active and semi-supervised learning techniques. Active semi-supervised learning investigates methods to efficiently construct predictive models by incorporating unlabeled data, which is either labeled by a domain expert or pseudolabeled by a model. Despite already being studied in other problems, to the best of our knowledge, active semi-supervised learning has not been applied in the context of multi-target regression, a predictive task where multiple continuous targets must be predicted. In this work, we investigate active semi-supervised learning for multi-target regression. More specifically, we propose, MASSTER, Multi-target Active Semi-Supervised Training for Regression, a novel ensemble method that identifies the most relevant instance-target pairs based on the variance in their predictions. Experiments using 8 benchmark datasets reveal that our method for active learning provides superior results in most of the cases when compared to the current state-of-the-art active learning method for multi-target regression. Further, as its semi-supervised component, our method incorporates a variation of both self-learning (MASSTER-SL) and co-training (MASSTER-CT). Both variants presented better metrics in earlier epochs when compared to MASSTER-AL version with less labeled data especially in smaller datasets.

Preventing a Machine Learning (ML) predictor from making an unreliable prediction is extremely important in sensitive application domains, such as health contexts. In this sense, strategies based on the reject option have been increasingly explored. However, few studies explore the ability of meta-learning to inspect the errors of a base predictor under analysis, in such a way to generalize when the predictor is confident or not. Therefore, the current paper proposes a novel solution for ML with reject option based on the combination of meta-learning and novelty detection. The proposal addresses two distinct situations where a prediction should be rejected. First, novel detection is adopted to identify out-of-distribution instances, i.e., instances that significantly differ from those ones adopted to train the base predictor. Second, meta-learning is adopted to detect instances in regions of data where the base model has shown poor predictive performance during its evaluation. Such instances mainly lie in areas of class overlap or noisy regions in the training data. The results in experiments on synthetic and real data showed the superiority of the solution compared to those based only on meta-learning (aka without novelty detection) and those based on classifier confidence.

Traditional Machine Learning (ML) approaches often emphasize evaluating models using global metrics over a dataset, frequently overlooking the nuances of learning data. Analyzing how hard it is to classify each instance, also known as instance hardness, furnishes such information, offering insights into reasons behind particular misclassifications. This paper introduces an unsupervised Deep Boltzmann Machine model integrated with an interpretability module that provides various latent traits related to instance hardness and classifier predictive performance. Such knowledge can facilitate in-depth analyses of the learning dataset's instances and the predictive power or ability of ML algorithms. Herein, we illustrate our approach by assessing five datasets with over 230 learning algorithms.

Anomaly detection (AD) techniques are adopted to identify instances with patterns that significantly differ from the general behavior of a dataset. The development of new techniques, such as those based on deep learning, and the higher data availability have increased the use of AD techniques in challenging tasks such as in the detection of failures in industrial equipment’s. Generally, an AD technique generates an anomaly score for each instance, later used to classify it as anomalous or normal, based on a threshold above which the instance is considered anomalous. A problem that is commonly observed in practice is the presence of spurious peaks in the anomaly score signal and other irregularities that may cause, for example, a high number of false positives in AD. In this paper, we investigated the use of low-pass filters in order to smooth the anomaly scores derived by a Sparse Autoencoder (SAE) model adopted for AD. In our experiments, we investigated the usefulness of the low-pass filters considering two different approaches: (1) directly applied on the anomaly scores; and (2) applied on the classification signal returned by the AD model. The experiments were performed on a case study of AD in a metro’s air production unit. Generally, the filter applied directly on the anomaly score maximized true positives. In turn, the filter applied after classification minimized false positives. It was observed that in general the use of LPF was essential to detect sequences of anomalies. Thus, how to apply low-pass filters in AD must be defined according to specific application goals.

Understanding the difficulty of individual instances in a classification problem is important to define the limits of learning performance in the problem. Previous works are devoted to measuring Instance Hardness (IH), while solutions for explaining IH are still not deeply investigated. In this paper, we rely on using assessor models and eXplanaible AI (XAI) techniques to predict and explain IH. Many XAI techniques have been developed in the literature to explain the predictions of Machine Learning (ML) models. In our work, we are focused on explaining the difficulty of instances. Given a classification dataset, we trained and evaluated a pool of diverse ML models to measure the IH of each instance. Then, we trained an assessor model to predict the IH based on the instances’ features. Once the assessor is built, its predictions (i.e., the expected IH) can be explained using XAI techniques. In our experiments, we produced Partial Dependence Plots (PDP) to inspect the marginal effect of specific features on the IH predicted by the assessor. From the PDPs, we could check how IH is distributed along the instances’ features in a problem, and more specifically, we could visualize areas of high expected predictive difficulty.

Soccer is a hugely popular sport globally and boasts a billion-dollar industry. ML (Machine Learning) algorithms have been adopted to predict outcomes in soccer. However, soccer matches are sometimes difficult to predict even by well informed experts and in other cases have surprising results. In the same sense, ML predictions may be unreliable and inaccurate. In this paper, we investigate a new approach in the ML literature, called assessors, adopted in our work to monitor the quality of predictions of a ML base model along a championship in order to identify the most reliable ones. Given a match, the assessor is used to predict whether the base model is able to correctly predict that match's result. Unreliable predictions are then rejected by the assessor. Our goal is to optimize the accuracy of accepted predictions while controlling the rejection rate. We conducted experiments on real data to identify the championships, teams, and rounds where the proposed rejection option approach performs best. The innovative proposed approach was actually useful to identify high-quality predictions, thus enhancing the reliability of match outcome predictions.

Thermal Power Plants (TPPs) produce electricity by burning fuels such as coal and oil. In Brazil, TPPs are crucial to guarantee the energy supply in periods of critical climatic conditions, as a complement to hydroelectric generation. Failures in a TPP system can cause unscheduled interruption for a long period of time, which can harm the electricity supply and result in excessive costs. The current paper addresses the problem of detecting anomalies in sensor data used to monitor the condition of TPP equipments. In the developed work, we combined clustering and statistical techniques, evaluated to detect anomalies in the cooling water system of a Brazilian TPP company. Experiments were performed in real cases of unplanned shutdown in the TPP in order to verify whether the anomaly detection method could anticipate the observed failures. The developed clustering-based method was able to detect the anomalous cases several minutes in advance, at same time obtaining fewer false alarms compared to a baseline method in literature.

Label noise detection has been widely studied in Machine Learning because of its importance in improving training data quality. Satisfactory noise detection has been achieved by adopting ensembles of classifiers. In this approach, an instance is assigned as mislabeled if a high proportion of members in the pool misclassifies it. Previous authors have empirically evaluated this approach; nevertheless, they mostly assumed that label noise is generated completely at random in a dataset. This is a strong assumption since other types of label noise are feasible in practice and can influence noise detection results. This work investigates the performance of ensemble noise detection under two different noise models: the Noisy at Random (NAR), in which the probability of label noise depends on the instance class, in comparison to the Noisy Completely at Random model, in which the probability of label noise is entirely independent. In this setting, we investigate the effect of class distribution on noise detection performance since it changes the total noise level observed in a dataset under the NAR assumption. Further, an evaluation of the ensemble vote threshold is conducted to contrast with the most common approaches in the literature. In many performed experiments, choosing a noise generation model over another can lead to different results when considering aspects such as class imbalance and noise level ratio among different classes.

The assessment of machine learning algorithms in a particular task is usually done by means of empirical evaluation on real world observational data. However, sometimes there is no previously annotated data available. Synthetic datasets have gained attention as an alternative for efficient classifier evaluation, since they are accessible and high parameterizable for a given learning task. The characterization of such databases can be done by means of descriptors on the learning object at hand, e.g., complexity measures, which extract statistical and geometric characteristics from the data sets, for a given classification problem, in order to estimate their complexity. Such complexity measures can be used to guide the production of synthetic datasets, so it reinforces one or more dataset features. The present work proposes the use of a many-objective algorithm for the generation of synthetic data considering four measures of complexity that will be balanced at the same time. The results showed that the proposal is able to optimize conflicting objectives, generating datasets of specific complexities.

Item Response Theory (IRT) is a tool developed in psychometrics to measure latent abilities of human respondents based on their responses to items with different levels of difficulty. Recently, IRT has been applied to evaluation in AI, by treating the algorithms as respondents and the AI tasks as items. Particularly in machine learning, IRT has been applied for evaluation of classifiers based on their predictions to each test instance. Based on a matrix of responses (classifiers vs instances), the IRT model estimates the latent difficulty and discrimination of each instance, as well as the ability of each classifier, in such a way that a classifier receives high ability value when it tends to correctly classify the most difficult instances. The IRT models previously adopted for evaluation in classification are not directly applied for regression, since they rely on dichotomous responses (i.e., a response has to be either correct or incorrect). In this paper we propose a new IRT model, particularly designed for dealing with nonnegative unbounded responses, which is adequate for modelling the absolute errors of regression algorithms. In the proposed model, responses follow a gamma distribution, parameterised according to respondents' abilities and items' difficulty and discrimination parameters. The proposed parameterisation results in item characteristic curves with more flexible shapes compared to the logistic curves widely adopted in IRT. The proposed model was evaluated with diverse regression algorithms and two benchmark datasets, one synthetic and one real. Useful insights were derived by inspecting regions in these datasets that present different levels of difficulty and discrimination.

Algorithm selection is a challenging task in machine learning. Meta-learning treats algorithm selection as a supervised learning task, in which training examples (i.e., meta-examples) are generated from experiments performed with a set of candidate algorithms in several datasets. The small availability of real datasets in some domains can make it difficult to generate good sets of meta-examples. An alternative is the use of synthetic datasets. Unfortunately, not all synthetic datasets can be considered equally relevant and representative compared to real datasets. Thus simply adopting a high number of arbitrary synthetic datasets increases the computational cost of performing experiments, without necessarily improving the quality of meta-learning. In this paper, we treat the selection of relevant synthetic datasets for meta-learning as an One-Class Classification (OCC) problem. In this problem, it is assumed the availability of instances associated to a single class of interest (the positive class) and a large set of unlabelled instances (the unknown class). The objective is to classify which unlabelled instances most likely belong to the positive class. In our context, OCC techniques are used to select the most relevant synthetic datasets (unknown class), by considering the real datasets (positive class) available. In our work, we conducted experiments in a case study in which we adopted a data manipulation procedure to produce synthetic datasets and two OCC techniques for dataset selection. The results revealed that it was actually possible to select a reduced number of synthetic datasets while maintaining or even increasing meta-learning performance.

Item Response Theory (IRT) aims to assess latent abilities of respondents based on the correctness of their answers in aptitude test items with different difficulty levels. In this paper, we propose the $\beta^3$-IRT model, which models continuous responses and can generate a much enriched family of Item Characteristic Curves. In experiments we applied the proposed model to data from an online exam platform, and show our model outperforms a more standard 2PL-ND model on all datasets. Furthermore, we show how to apply $\beta^3$-IRT to assess the ability of machine learning classifiers.This novel application results in a new metric for evaluating the quality of the classifier’s probability estimates, based on the inferred difficulty and discrimination of data instances.

Deep learning techniques have been successfully applied to image processing tasks. Nevertheless, these techniques can be sensitive to the occurrence of small training sets, which is commonly faced when model training requires labelled instances. Transfer learning has been adopted to overcome this limitation by leveraging richer information in auxiliary domains to enhance the learning process in a target domain. In this paper, we proposed a new solution for transfer learning to label images (i.e., the target domain) by reusing labelled textual data (i.e., the auxiliary domain). A convolutional encoder is used to find latent features for images, while a probabilistic generative model is used to find semantic topics (traits) for texts. An ensemble of classifiers is then used to predict semantic topics to new input images according to their latent features. Experiments were performed to evaluate whether the latent features in both domains can actually be related and also to verify the use of the predicted semantic topics to classify images. Promising results were achieved compared to different baselines.

Item response theory (IRT) is widely used to measure latent abilities of subjects (specially for educational testing) based on their responses to items with different levels of difficulty. The adaptation of IRT has been recently suggested as a novel perspective for a better understanding of the results of machine learning experiments and, by extension, other artificial intelligence experiments. For instance, IRT suits classification tasks perfectly, where instances correspond to items and classifiers correspond to subjects. By adopting IRT, item (i.e., instance) characteristic curves can be estimated using logistic models, for which several parameters characterise each dataset instance: difficulty, discrimination and guessing. IRT looks promising for the analysis of instance hardness, noise, classifier dominances, etc. However, some caveats have been found when trying to interpret the IRT parameters in a machine learning setting, especially when we include some artificial classifiers in the pool of classifiers to be evaluated: the optimal and pessimal classifiers, a random classifier and the majority and minority classifiers. In this paper we perform a series of experiments with a range of datasets and classification methods to fully understand how IRT works and what their parameters really mean in the context of machine learning. This better understanding will hopefully pave the way to a myriad of potential applications in machine learning and artificial intelligence.

BACKGROUND: Drug-target networks are receiving a lot of attention in late years, given its relevance for pharmaceutical innovation and drug lead discovery. Different in silico approaches have been proposed for the identification of new drug-target interactions, many of which are based on kernel methods. Despite technical advances in the latest years, these methods are not able to cope with large drug-target interaction spaces and to integrate multiple sources of biological information. RESULTS: We propose KronRLS-MKL, which models the drug-target interaction problem as a link prediction task on bipartite networks. This method allows the integration of multiple heterogeneous information sources for the identification of new interactions, and can also work with networks of arbitrary size. Moreover, it automatically selects the more relevant kernels by returning weights indicating their importance in the drug-target prediction at hand. Empirical analysis on four data sets using twenty distinct kernels indicates that our method has higher or comparable predictive performance than 18 competing methods in all prediction tasks. Moreover, the predicted weights reflect the predictive quality of each kernel on exhaustive pairwise experiments, which indicates the success of the method to automatically reveal relevant biological sources. CONCLUSIONS: Our analysis show that the proposed data integration strategy is able to improve the quality of the predicted interactions, and can speed up the identification of new drug-target interactions as well as identify relevant information for the task. AVAILABILITY: The source code and data sets are available at www.cin.ufpe.br/~acan/kronrlsmkl/.

In this work, we propose a framework to automatically generate effective PSO designs by adopting Grammatical Evolution (GE). In the proposed framework, GE searches for adequate structures and parameter values (e.g., acceleration constants, velocity equations and different particles' topology) in order to evolve the PSO design. For this, a high-level Backus--Naur Form (BNF) grammar was developed, representing the search space of possible PSO designs. In order to verify the performance of the proposed method, we performed experiments using 16 diverse continuous optimization problems, with different levels of difficulty. In the performed experiments, we identified the parameters and components that most affected the PSO performance, as well as identified designs that could be reused across different problems. We also demonstrated that the proposed method generates useful designs which achieved competitive solutions when compared to well succeeded algorithms from the literature.

During the software testing process many test suites can be generated in order to evaluate and assure the quality of the products. In some cases the execution of all suites cannot fit the available resources (time, people, etc). Hence, automatic Test Case (TC) selection could be used to reduce the suites based on some selection criterion. This process can be treated as an optimization problem, aiming to find a subset of TCs which optimizes one or more objective functions (i.e., selection criteria). The majority of search-based works focus on single-objective selection. In this light, we developed mechanisms for functional TC selection which considers two objectives simultaneously: maximize requirements coverage while minimizing cost in terms of TC execution effort. These mechanisms were implemented by deploying multi-objective techniques based on Particle Swarm Optimization (PSO). Due to the drawbacks of original binary version of PSO we implemented five binary PSO algorithms and combined them with a multi-objective versions of PSO in order to create new optimization strategies applied to TC selection. The experiments were performed on two real test suites, revealing the feasibility of the proposed strategies and the differences among them.

Collaborative Filtering is the most commonly used technique in Recommender Systems, based on the users ratings in order to identify similar profiles and suggest them items. However, because it depends essentially on direct similarity measures between users or items, it usually suffers from the sparsity problem. Upon this situation, a good alternative is using global similarities to enrich the users neighborhood by transitively connecting them together, even when they do not share any common ratings. In this paper, we investigated the use of both local and global similarity measures with the maximin distance algorithm, along with tie-breaking criteria for neighbors with equal similarity. Our experiments showed that the maximin distance algorithm in fact produces many equally similar global neighbors, and that the criteria set for deciding between them severely improved the results of the recommendation process.

The Support Vector Machine algorithm is sensitive to the choice of parameter settings. If these are not set correctly, the algorithm may have a substandard performance. It has been shown that meta-learning can be used to support the selection of SVM parameters. However, it is very dependent on the quality of the dataset and the meta-features used to characterize the dataset. As alternative for this problem, a recent technique called Active Testing characterized a dataset based on the pairwise performance differences between possible solutions. This approach selects the most useful cross-validation tests. Each new cross-validation test will contribute information to a better estimate of dataset similarity, and thus better predict which algorithms are most promising on the new dataset. In this paper we propose the application of Active Testing for the SVM parameter problem. We test it on the problem of setting the RBF kernel parameters for classification problems and we compare its similarity strategy with based on data characteristics. The results showed the variants of Active Testing that rely on cross-validation tests to estimate dataset similarity provides better solutions than those that rely on data characteristics.

Several approaches have been applied to the task of algorithm selection. In this context, Meta-Learning provides an efficient solution by adopting a supervised strategy. Despite its promising results, Meta-Learning requires an adequate number of instances to produce a rich set of meta-examples. Recent approaches to generate synthetic or manipulated datasets have been adopted with success in the context of Meta-Learning. These proposals include the datasetoids approach, a simple data manipulation technique that generates new datasets from existing ones. Although such proposals can actually produce relevant datasets, they can eventually produce redundant, or even irrelevant, problem instances. Active Meta-Learning arises in this context to select only the most informative instances for meta-example generation. In this work, we investigate the Active Meta-Learning combined with datasetoids, focusing on using the Random forest algorithm in meta-learning. Our experiments revealed that it is possible to reduce the computational cost of generating meta-examples and obtain a significant gain in Meta-Learning performance.

The emergence of online social networks has generated an enormous amount of data containing users' opinions about the most varied subjects. Aiming to identify opinion orientation, Sentiment Analysis techniques have been proposed, mainly based on text classification methods. We propose a different perspective to treat this problem, based on a user centric approach. We adopt a graph representation in which nodes represent users and connections represent relationships in a social network. Then, we apply collective classification techniques which use link information to infer opinions of users who have not posted their opinion about the subject under analysis. Preliminary experiments on a Twitter corpus of political preferences have shown promising results.

Support Vector Machines (SVMs) have become a well succeed technique due to the good performance it achieves on different learning problems. However, the performance depends on adjustments on its model. The automatic SVM parameter selection is a way to deal with this. This approach is considered an optimization problem whose goal is to find suitable configuration of parameters which attends some learning problem. This work proposes the use of Particle Swarm Optimization (PSO) to treat the SVM parameter selection problem. As the design of learning systems is inherently a multi-objective optimization problem, a multi-objective PSO (MOPSO) was used to maximize the success rate and minimize the number of support vectors of the model. Moreover, we propose the combination of Meta-Learning (ML) with MOPSO to the cited problem. ML is used to recommend SVM parameters, to a given input problem, based on well-succeeded parameters adopted in previous similar problems. In this combination, initial solutions provided by ML are possibly located in good regions in the search space. Hence, using a reduced number of candidate search points, the search process, to find an adequate solution, would be less expensive. We highlight that, the combination of search algorithms with ML was just studied in the single objective field and the use of MOPSO in this context has not been investigated. In our work, we implemented a prototype in which MOPSO was used to select the values of two SVM parameters for classification problems. In the performed experiments, the proposed solution (MOPSO using ML or Hybrid MOPSO) was compared to a MOPSO with random initialization, obtaining paretos with higher quality on a set of 40 classification problems.

Link prediction is a task in Social Network Analysis that consists of predicting connections that are most likely to appear considering previous observed links in a social network. The majority of works in this area only performs the task by exploring the state of the network at a specific moment to make the prediction of new links, without considering the behavior of links as time goes by. In this light, we investigate if temporal information can bring any performance gain to the link prediction task. A traditional approach for link prediction uses a chosen topological similarity metric on non-connected pairs of nodes of the network at present time to obtain a score that is going to be used by an unsupervised or a supervised method for link prediction. Our approach initially consists of building time series for each pair of non-connected nodes by computing their similarity scores at different past times. Then, we deploy a forecasting model on these time series and use their forecasts as the final scores of the pairs. Our preliminary results using two link prediction methods (unsupervised and supervised) on co-authorship networks revealed satisfactory results when temporal information was considered.

Support Vector Machine (SVM) is a supervised technique, which achieves good performance on different learning problems. However, adjustments on its model are essentials to the SVM work well. Optimization techniques have been used to automatize this process finding suitable configurations of parameters which attends some learning problems. This work utilizes Particle Swarm Optimization (PSO) applied to the SVM parameter selection problem. As the learning systems are essentially a multi-objective problem, a multi-objective PSO (MOPSO) was used to maximize the success rate and minimize the number of support vectors of the model. Nevertheless, we propose the combination of Meta-Learning (ML) with a modified MOPSO which uses the crowding distance mechanism (MOPSO-CDR). In this combination, solutions provided by ML are possibly located in good regions in the search space. Hence, using a reduced number of successful candidates, the search process would converge faster and be less expensive. In our work, we implemented a prototype in which MOPSO-CDR was used to select the values of two SVM parameters for classification problems. In the performed experiments, the proposed solution (MOPSO-CDR using ML) was compared to the MOPSO-CDR with random initialization, obtaining pareto fronts with higher quality on a set of 40 classification problems.

The emergence of online social networks in the past few years has generated an enormous amount of information about potentially any subject. Valuable data containing users' opinions and thoughts are available on those repositories and several sentiment analysis techniques have been proposed that address the problem of understanding the opinion orientation of the users' postings. In this paper, we take a different perspective to the problem through a user centric approach, which uses a graph to model users (and their postings) and applies link mining techniques to infer opinions of users. Preliminary experiments on a Twitter corpus have shown promising results.

The impact of a publication is often measured by the number of citations it received, this number being taken as a proxy for the relevance of published work. However, a higher citation index does not necessarily mean that a publication necessarily had a positive feedback from citing authors, as a citation can represent a negative criticism. In order to overcome this limitation, we used sentiment analysis to rate citations as positive, neutral or negative. Adjectives are initially extracted from the citations, with the SentiWordNet lexicon being used to rate the degree of positivity and negativity for each adjective. Relevance scores were then computed to rank citations according to the sentiment expressed in the text corresponding to each citation. As expected for accurate information retrieval systems, higher precision rates were observed in the initial points of the curve. The SRC (0.6728) computed using number of raw citations is lower than the SRC (0.7397) observed by the ranking generated using sentiment scores (Table 3). Conclusion: This result indicates that child articles with higher values of relevance score were in general the ones expressing positive opinion about their parents. Therefore, the ranking generated by sentiment scores had an improved accuracy.

Although software testing is a central task in the software lifecycle, it is sometimes neglected due to its high costs. Tools to automate the testing process minor its costs, however they generate large test suites with redundant Test Cases (TC). Automatic TC Selection aims to reduce a test suite based on some selection criterion. This process can be treated as an optimization problem, aiming to find a subset of TCs which optimizes one or more objective functions (i.e., selection criteria). The majority of search-based works focus on single-objective selection. In this light, we developed a mechanism for functional TC selection which considers two objectives simultaneously: maximize requirements' coverage while minimizing cost in terms of TC execution effort. This mechanism was implemented as a multi-objective optimization process based on Particle Swarm Optimization (PSO). We implemented two multi-objective versions of PSO (BMOPSO and BMOPSO-CDR). The experiments were performed on two real test suites, revealing very satisfactory results (attesting the feasibility of the proposed approach). We highlight that execution effort is an important aspect in the testing process, and it has not been used in a multi-objective way together with requirements coverage for functional TC selection.

Several meta-learning approaches have been developed for the problem of algorithm selection. In this context, it is of central importance to collect a sufficient number of datasets to be used as meta-examples in order to provide reliable results. Recently, some proposals to generate datasets have addressed this issue with successful results. These proposals include datasetoids, which is a simple manipulation method to obtain new datasets from existing ones. However, the increase in the number of datasets raises another issue: in order to generate meta-examples for training, it is necessary to estimate the performance of the algorithms on the datasets. This typically requires running all candidate algorithms on all datasets, which is computationally very expensive. In a recent paper, active meta-learning has been used to address this problem. An uncertainty sampling method for the k-NN algorithm using a least confidence score based on a distance measure was employed. Here we extend that work, namely by investigating three hypotheses: 1) is there advantage in using a frequency-based least confidence score over the distance-based score? 2) given that the meta-learning problem used has three classes, is it better to use a margin-based score? and 3) given that datasetoids are expected to contain some noise, are better results achieved by starting the search with all datasets already labeled? Some of the results obtained are unexpected and should be further analyzed. However, they confirm that active meta-learning can significantly reduce the computational cost of meta-learning with potential gains in accuracy.

Link prediction is an important task in Social Network Analysis. This problem refers to predicting the emergence of future relationships between nodes in a social network. Our work focuses on a supervised machine learning strategy for link prediction. Here, the target attribute is a class label indicating the existence or absence of a link between a node pair. The predictor attributes are metrics computed from the network structure, describing the given pair. The majority of works for supervised prediction only considers unweighted networks. In this light, our aim is to investigate the relevance of using weights to improve supervised link prediction. Link weights express the `strength' of relationships and could bring useful information for prediction. However, the relevance of weights for unsupervised strategies of link prediction was not always verified (in some cases, the performance was even harmed). Our preliminary results on supervised prediction on a co-authorship network revealed satisfactory results when weights were considered, which encourage us for further analysis.

Meta-Learning has been applied to acquire useful knowledge to predict learning performance. Each training example in Meta-Learning (i.e. each meta-example) is related to a learning problem and stores features of the problem plus the performance obtained by a set of candidate algorithms when evaluated on the problem. Based on a set of such meta-examples, a meta-learner will be used to predict algorithm performance for new problems. The generation of a set of meta-examples can be expensive, since for each problem it is necessary to perform an empirical evaluation of the candidate algorithms. In a previous work, we proposed the Active Meta-Learning, in which Active Learning was used to reduce the set of meta-examples by selecting only the most relevant problems for meta-example generation. In the current work, we proposed the combination of different Uncertainty Sampling methods for Active Meta-Learning, considering that each individual method will provide useful information that can be combined in order to have a better assessment of problem relevance for meta-example generation. In our experiments, we observed a gain in Meta-Learning performance when the proposed method was compared to the individual active methods being combined.

Information Extraction (IE) aims to extract from textual documents only the fragments which correspond to datafields required by the user. In this paper, we present new experiments evaluating a hybrid machine learning approach for IE that combines text classifiers and Hidden MarkovModels (HMM). In this approach, a text classifier technique generates an initial output, which is refined by an HMM, taking into account dependences in the order of the data to be extracted. The proposal was evaluated to extract information from bibliographic references. Experiments performed on a corpus of 6000 references have shown an improvement in performance compared to benchmarking IE approaches adopted in previous work.

Meta-Learning has been used to predict the performance of learning algorithms based on descriptive features of the learning problems. Each training example in this context, i.e. each meta-example, stores the features of a given problem and information about the empirical performance obtained by the candidate algorithms on that problem. The process of constructing a set of meta-examples may be expensive, since for each problem avaliable for meta-example generation, it is necessary to perform an empirical evaluation of the candidate algorithms. Active Meta-Learning has been proposed to overcome this limitation by selecting only the most informative problems in the meta-example generation. In this work, we proposed an Active Meta-Learning method which combines Uncertainty Sampling and Outlier Detection techniques. Experiments were performed in a case study, yielding significant improvement in the Meta-Learning performance.

We present a novel framework that applies a meta-learning approach to clustering algorithms. Given a dataset, our meta-learning approach provides a ranking for the candidate algorithms that could be used with that dataset. This ranking could, among other things, support non-expert users in the algorithm selection task. In order to evaluate the framework proposed, we implement a prototype that employs regression support vector machines as the meta-learner. Our case study is developed in the context of cancer gene expression micro-array datasets.

Meta-learning has been used to relate the performance of algorithms and the features of the problems being tackled. The knowledge in meta-learning is acquired from a set of meta-examples which are generated from the empirical evaluation of the algorithms on problems in the past. In this work, active learning is used to reduce the number of meta-examples needed for meta-learning. The motivation is to select only the most relevant problems for meta-example generation, and consequently to reduce the number of empirical evaluations of the candidate algorithms. Experiments were performed in two different case studies, yielding promising results.

In this work, we proposed to use the Zoomed Ranking approach to rank and select time series models. Zoomed Ranking, originally proposed to generate a ranking of candidate algorithms, is employed to solve a given classification problem based on performance information from previous problems. The problem of model selection in Zoomed Ranking was solved in two distinct phases. In the first phase, we selected a subset of problems from the instances base that were similar to the new problem at hand. This selection is made using the k-Nearest Neighbor algorithm, whose distance function uses the characteristics of the series. In the second phase, the ranking of candidate models was generated based on performance information (accuracy and execution time) of the models in the series selected from the previous phase. Our experiments using the Zoomed Ranking revealed encouraging results.