Abstract Question‐driven automatic text summarization is a popular technique to produce concise and informative answers to specific questions using a document collection. Both query‐based and question‐driven summarization may not produce reliable summaries nor contain relevant information if they do not take advantage of extractive and abstractive summarization mechanisms to improve performance. In this article, we propose a novel extractive and abstractive hybrid framework designed for question‐driven automatic text summarization. The framework consists of complimentary modules that work together to generate an effective summary: (1) discovering appropriate non‐redundant sentences as plausible answers using an open‐domain multi‐hop question answering system based on a convolutional neural network, multi‐head attention mechanism and reasoning process; and (2) a novel paraphrasing generative adversarial network model based on transformers rewrites the extracted sentences in an abstractive setup. Experiments show this framework results in more reliable abstractive summary than competing methods. We have performed extensive experiments on public datasets, and the results show our model can outperform many question‐driven and query‐based baseline methods (an R1, R2, RL increase of 6%–7% for over the next highest baseline).

Juntao Yu, Silviu Paun, Maris Camilleri, Paloma Garcia, Jon Chamberlain, Udo Kruschwitz, Massimo Poesio. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.

Loss of privacy and encounters with misinformation are two challenges individuals are likely to encounter in their search for information on the web. These challenges have potential negative impacts, especially in search domains such as health search. Existing information retrieval (IR) systems offer users little (if any) guidance as to how to reduce the likelihood of such negative impacts. The sum of these problems provides motivation for experiments to identify elements of existing IR environments that might provide low-cost options to the user for improved search outcomes.

One of the key steps in language resource creation is the identification of the text segments to be annotated, or markablesin our case, the (potentially nested) noun phrases in coreference resolution (or mentions).In this paper, we present a method for identifying markables for coreference annotation that combines high-performance automatic markable detectors with checking with a Game-With-A-Purpose (GWAP) and aggregation using a Bayesian annotation model.The method was evaluated both on news data and data from a variety of other genres and results in an improvement on F 1 of mention boundaries of over seven percentage points when compared with a state-of-the-art, domain-independent automatic mention detector, and almost three points over an in-domain mention detector.One of the key contributions of our proposal is its applicability to the case in which markables are nested, as is the case with coreference markables; but the GWAP and several of the proposed markable detectors are task-and language-independent and are thus applicable to a variety of other annotation scenarios.

Chatbots that can automatically answer customer requests have become a common feature on e-commerce Web sites. There are many factors that might affect overall customer satisfaction with such services. We explore how adding language style to e-commerce chatbots can be used to increase user satisfaction, perceived product value, user interest in a product, and user engagement with a chatbot service. We conducted an experimental pilot study, where two chatbots were used to sell theatre tickets: one communicating in modern English and one in a Shakespearean-style dialect. 169 participants interacted with a randomly-assigned version of the chatbot. The results indicate that the bot talking in modern English showed a significantly higher user satisfaction, whereas the Shakespearean-styled chatbot showed higher user engagement and perceived product value. It was also found that the modern chatbot version was more often referred to as being 'easy to use', whereas the Shakespearean chatbot version was more often referred to as being 'fun to use'.

Knowledge workers (such as healthcare information professionals, patent agents and legal researchers) need to create and execute search strategies that are accurate, repeatable and transparent. The traditional solution offered by most database vendors is to use proprietary line-by-line 'query builders'. However, these offer limited support for error checking or query optimisation, and their output can often be compromised by errors and inefficiencies. Using the healthcare domain for context, we demonstrate a new approach to search strategy formulation in which concepts are expressed as objects on a two-dimensional canvas, and relationships are articulated using direct manipulation. This approach eliminates many sources of syntactic error, makes the query semantics more transparent, and offers new ways to optimise, save and share search strategies and best practices

Games for text annotation / labelling are becoming more common, but it's difficult to find a mechanics that fits. In this work we discuss a clicker game that can support text annotation. We believe this type of game is uniquely suited to addressing some of the challenges faced by games featuring text annotation as a core task.

In this paper we present Wormingo, 1 a new Game-with-a-Purpose for anaphoric annotation. It introduces the motivation-annotation paradigm which uses linguistic puzzles and other widely known gamification techniques and word game mechanics to motivate players to carry out anaphoric annotation tasks. In a preliminary experiment, the game was tested on 270 players recruited through the Reddit platform, achieving promising results.

In this paper we present WordClicker, a clicker game for text annotation. We believe the mechanics of 'Ville type Free-To-Play (F2P) games in general, and clicker games in particular, is particularly suited for GWAPs (Games-With-A-Purpose). WordClicker was developed as one component of a suite of GWAPs meant to cover all aspects of language interpretation, from tokenization to anaphoric interpretation. As such, WordClicker is intended to have a dual function as part of this suite of GWAPs: both for parts-of-speech annotation and for teaching players about parts of speech so that they can go on and play GWAPs for more complex syntactic annotation. Therefore, game-based language learning platforms also had a strong influence on its design.

Within traditional games design, incorporating progressive difficulty is considered of fundamental importance. But despite the widespread intuition that progression could have clear benefits in Games-With-A-Purpose (GWAPs)–e.g., for training non-expert annotators to produce more complex judgements– progression is not in fact a prominent feature of GWAPs; and there is even less evidence on its effects. In this work we present an approach to progression in GWAPs that generalizes to different annotation tasks with minimal, if any, dependency on gold annotated data. Using this method we observe a statistically significant increase in accuracy over randomly showing items to annotators.

Knowledge workers such as patent agents, recruiters and legal researchers undertake work tasks in which search forms a core part of their duties. In these instances, the search task often involves formulation of complex queries expressed as Boolean strings. However, creating effective Boolean queries remains an ongoing challenge, often compromised by errors and inefficiencies. In this paper, we demonstrate a new approach to structured searching in which concepts are expressed as objects on a two-dimensional canvas. Interactive query suggestions are provided via an NLP services API, and support is offered for optimising, translating and sharing search strategies as executable artefacts. This eliminates many sources of error, makes the query semantics more transparent, and offers an open-access platform for sharing reproducible search strategies and best practices.

The availability of large scale annotated corpora for coreference is essential to the development of the field.However, creating resources at the required scale via expert annotation would be too expensive.Crowdsourcing has been proposed as an alternative; but this approach has not been widely used for coreference.This paper addresses one crucial hurdle on the way to make this possible, by introducing a new model of annotation for aggregating crowdsourced anaphoric annotations.The model is evaluated along three dimensions: the accuracy of the inferred mention pairs, the quality of the post-hoc constructed silver chains, and the viability of using the silver chains as an alternative to the expert-annotated chains in training a state of the art coreference system.The results suggest that our model can extract from crowdsourced annotations coreference chains of comparable quality to those obtained with expert annotation.

The analysis of crowdsourced annotations in natural language processing is concerned with identifying (1) gold standard labels, (2) annotator accuracies and biases, and (3) item difficulties and error patterns. Traditionally, majority voting was used for 1, and coefficients of agreement for 2 and 3. Lately, model-based analysis of corpus annotations have proven better at all three tasks. But there has been relatively little work comparing them on the same datasets. This paper aims to fill this gap by analyzing six models of annotation, covering different approaches to annotator ability, item difficulty, and parameter pooling (tying) across annotators and items. We evaluate these models along four aspects: comparison to gold labels, predictive accuracy for new annotations, annotator characterization, and item difficulty, using four datasets with varying degrees of noise in the form of random (spammy) annotators. We conclude with guidelines for model selection, application, and implementation.

Crowdsourcing and citizen science have established themselves in the mainstream of research methodology in recent years, employing a variety of methods to solve problems using human computation. An approach described here, termed "groupsourcing", uses social networks to present problems and collect solutions. This paper details a method for archiving social network messages and investigates messages containing an image classification task in the domain of marine biology. In comparison to other methods, groupsourcing offers a high accuracy, data-driven and low cost approach.

Increasingly social networks are being used for citizen science, where members of the public contribute knowledge to scientific endeavours. Tasks can be presented and solved using human computation, termed groupsourcing, with users benefiting from community tuition and experts gaining knowledge from the crowd. This paper gives details of a prototype that utilises groupsourcing to solve image classification tasks, to support social learning and to facilitate knowledge discovery in the domain of marine biology.

We are witnessing a paradigm shift in Human Language Technology (HLT) that may well have an impact on the field comparable to the statistical revolution: acquiring large-scale resources by exploiting collective intelligence. An illustration of this new approach is Phrase Detectives , an interactive online game with a purpose for creating anaphorically annotated resources that makes use of a highly distributed population of contributors with different levels of expertise. The purpose of this article is to first of all give an overview of all aspects of Phrase Detectives, from the design of the game and the HLT methods we used to the results we have obtained so far. It furthermore summarizes the lessons that we have learned in developing this game which should help other researchers to design and implement similar games.