Evolutionary machine learning research aims to develop classifier systems that can solve complex and hard tasks. This paper addresses the problem of inferring a regular expression from a given set of strings for automating the task of information extraction. To the best of our knowledge, this paper is the first to propose the extension of accuracy-based classifier system XCS to learn the regular expressions for text extraction. This new system named as XCSREA includes tree-like code fragments to learn regular expressions. The genetic algorithm in action sets uses two-point crossover with uniform mutation and Roulette wheel parent selection method. Seven different datasets, each with three different lengths, are used to compare the performance of the proposed model with standard genetic programming (GP) approach. The experimental results demonstrate that XCSREA outperforms standard GP approach when sufficiently large numbers of classifiers are used.

The TREC 2012 Crowdsourcing track asked participants to crowdsource relevance assessments with the goal of replicating costly expert judgements with relatively fast, inexpensive, but less reliable judgements from anonymous online workers. The track used 10 "ad-hoc" queries, highly specific and complex (as compared to web search). The crowdsourced assessments were evaluated against expert judgments made by highly trained and capable human analysts in 1999 as part of ad hoc track collection construction. Since most crowdsourcing approaches submitted to the TREC 2012 track produced assessment sets nowhere close to the expert judgements, we decided to analyze crowdsourcing mistakes made on this task using data we collected via Amazon's Mechanical Turk service. We investigate two types of crowdsourcing approaches: one that asks for nominal relevance grades for each document, and the other that asks for preferences on many (not all) pairs of documents.

High quality relevance judgments are essential for the evaluation of information retrieval systems. Traditional methods of collecting relevance judgments are based on collecting binary or graded nominal judgments, but such judgments are limited by factors such as inter-assessor disagreement and the arbitrariness of grades. Previous research has shown that it is easier for assessors to make pairwise preference judgments. However, unless the preferences collected are largely transitive, it is not clear how to combine them in order to obtain document relevance scores. Another difficulty is that the number of pairs that need to be assessed is quadratic in the number of documents. In this work, we consider the problem of inferring document relevance scores from pairwise preference judgments by analogy to tournaments using the Elo rating system. We show how to combine a linear number of pairwise preference judgments from multiple assessors to compute relevance scores for every document.