Most existing topic models rely on bag-of-words (BOW) representation, which limits their ability to capture word order information and leads to challenges with out-of-vocabulary (OOV) words in new documents. Contextualized word embeddings, however, show superiority in word sense disambiguation and effectively address the OOV issue. In this work, we introduce a novel neural topic model called the Contextlized Word Topic Model (CWTM), which integrates contextualized word embeddings from BERT. The model is capable of learning the topic vector of a document without BOW information. In addition, it can also derive the topic vectors for individual words within a document based on their contextualized word embeddings. Experiments across various datasets show that CWTM generates more coherent and meaningful topics compared to existing topic models, while also accommodating unseen words in newly encountered documents.

To seek reliable information sources for news events, we introduce a novel task of expert recommendation, which aims to identify trustworthy sources based on their previously quoted statements. To achieve this, we built a novel dataset, called NewsQuote, consisting of 23,571 quote-speaker pairs sourced from a collection of news articles. We formulate the recommendation task as the retrieval of experts based on their likelihood of being associated with a given query. We also propose a multi-layer ranking framework employing Large Language Models to improve the recommendation performance. Our results show that employing an in-context learning based LLM ranker and a multi-layer ranking-based filter significantly improve both the predictive quality and behavioural quality of the recommender system.

Abstract In this paper we present an ethnographic study of the work of histopathologists as they grapple with the twin innovations of transitioning to digital biopsy images and the prospective adoption of an AI-based clinical decision support system (CDSS). We explore how they are adapting to the former and their expectations of the latter. The study’s ethnomethodologically-informed ethnography approach brings to light some key issues regarding the nature of diagnostic work, and accountability and trust that are central to the successful adoption of technological innovations in clinical settings.

Human-in-the-loop topic modelling incorporates users' knowledge into the modelling process, enabling them to refine the model iteratively.Recent research has demonstrated the value of user feedback, but there are still issues to consider, such as the difficulty in tracking changes, comparing different models and the lack of evaluation based on real-world examples of use.We developed a novel, interactive human-in-the-loop topic modeling system with a user-friendly interface that enables users compare and record every step they take, and a novel topic words suggestion feature to help users provide feedback that is faithful to the ground truth.Our system also supports not only what traditional topic models can do, i.e., learning the topics from the whole corpus, but also targeted topic modelling, i.e., learning topics for specific aspects of the corpus.In this article, we provide an overview of the system and present the results of a series of user studies designed to assess the value of the system in progressively more realistic applications of topic modelling.

Research on automated social media rumour verification, the task of identifying the veracity of questionable information circulating on social media, has yielded neural models achieving high performance, with accuracy scores that often exceed 90%. However, none of these studies focus on the real-world generalisability of the proposed approaches, that is whether the models perform well on datasets other than those on which they were initially trained and tested. In this work we aim to fill this gap by assessing the generalisability of top performing neural rumour verification models covering a range of different architectures from the perspectives of both topic and temporal robustness. For a more complete evaluation of generalisability, we collect and release COVID-RV, a novel dataset of Twitter conversations revolving around COVID-19 rumours. Unlike other existing COVID-19 datasets, our COVID-RV contains conversations around rumours that follow the format of prominent rumour verification benchmarks, while being different from them in terms of topic and time scale, thus allowing better assessment of the temporal robustness of the models. We evaluate model performance on COVID-RV and three popular rumour verification datasets to understand limitations and advantages of different model architectures, training datasets and evaluation scenarios. We find a dramatic drop in performance when testing models on a different dataset from that used for training. Further, we evaluate the ability of models to generalise in a few-shot learning setup, as well as when word embeddings are updated with the vocabulary of a new, unseen rumour. Drawing upon our experiments we discuss challenges and make recommendations for future research directions in addressing this important problem.

The need for AI systems to provide explanations for their behaviour is now widely recognised as key to their adoption. In this article, we examine the problem of trustworthy AI and explore what delivering this means in practice, with a focus on healthcare applications. Work in this area typically treats trustworthy AI as a problem of Human–Computer Interaction involving the individual user and an AI system. However, we argue here that this overlooks the important part played by organisational accountability in how people reason about and trust AI in socio-technical settings. To illustrate the importance of organisational accountability, we present findings from ethnographic studies of breast cancer screening and cancer treatment planning in multidisciplinary team meetings to show how participants made themselves accountable both to each other and to the organisations of which they are members. We use these findings to enrich existing understandings of the requirements for trustworthy AI and to outline some candidate solutions to the problems of making AI accountable both to individual users and organisationally. We conclude by outlining the implications of this for future work on the development of trustworthy AI, including ways in which our proposed solutions may be re-used in different application settings.

Opinion summarisation synthesises opinions expressed in a group of documents discussing the same topic to produce a single summary.Recent work has looked at opinion summarisation of clusters of social media posts.Such posts are noisy and have unpredictable structure, posing additional challenges for the construction of the summary distribution and the preservation of meaning compared to online reviews, which has been so far the focus of opinion summarisation.To address these challenges we present WassOS, an unsupervised abstractive summarization model which makes use of the Wasserstein distance.A Variational Autoencoder is used to get the distribution of documents/posts, and the distributions are disentangled into separate semantic and syntactic spaces.The summary distribution is obtained using the Wasserstein barycenter of the semantic and syntactic distributions.A latent variable sampled from the summary distribution is fed into a GRU decoder with a transformer layer to produce the final summary.Our experiments on multiple datasets including Twitter clusters, Reddit threads, and reviews show that WassOS almost always outperforms the state-of-the-art on ROUGE metrics and consistently produces the best summaries with respect to meaning preservation according to human evaluations.

Miguel Arana-Catania, Elena Kochkina, Arkaitz Zubiaga, Maria Liakata, Robert Procter, Yulan He. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Lixing Zhu, Zheng Fang, Gabriele Pergola, Robert Procter, Yulan He. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Abstract We introduce the task of microblog opinion summarization (MOS) and share a dataset of 3100 gold-standard opinion summaries to facilitate research in this domain. The dataset contains summaries of tweets spanning a 2-year period and covers more topics than any other public Twitter summarization dataset. Summaries are abstractive in nature and have been created by journalists skilled in summarizing news articles following a template separating factual information (main story) from author opinions. Our method differs from previous work on generating gold-standard summaries from social media, which usually involves selecting representative posts and thus favors extractive summarization models. To showcase the dataset’s utility and challenges, we benchmark a range of abstractive and extractive state-of-the-art summarization models and achieve good performance, with the former outperforming the latter. We also show that fine-tuning is necessary to improve performance and investigate the benefits of using different sample sizes.

Iman Munire Bilal, Bo Wang, Maria Liakata, Rob Procter, Adam Tsakalidis. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Social media tend to be rife with rumours while new reports are released piecemeal during breaking news. Interestingly, one can mine multiple reactions expressed by social media users in those situations, exploring their stance towards rumours, ultimately enabling the flagging of highly disputed rumours as being potentially false. In this work, we set out to develop an automated, supervised classifier that uses multi-task learning to classify the stance expressed in each individual tweet in a conversation around a rumour as either supporting, denying or questioning the rumour. Using a Gaussian Process classifier, and exploring its effectiveness on two datasets with very different characteristics and varying distributions of stances, we show that our approach consistently outperforms competitive baseline classifiers. Our classifier is especially effective in estimating the distribution of different types of stance associated with a given rumour, which we set forth as a desired characteristic for a rumour-tracking system that will show both ordinary users of Twitter and professional news practitioners how others orient to the disputed veracity of a rumour, with the final aim of establishing its actual truth value.

Assisted living technologies may help people live independently while also—potentially—reducing health and care costs. But they are notoriously difficult to implement at scale and many devices are abandoned following initial adoption. We report findings from a study of global positioning system (GPS) tracking devices intended to support the independent living of people with cognitive impairment. Our aims were threefold: to understand (through ethnography) such individuals’ lived experience of GPS tracking; to facilitate (through action research) the customization and adaptation of technologies and care services to provide effective, ongoing support; and to explore the possibilities for a co-production methodology that would enable people with cognitive impairment and their families to work with professionals and technical designers to shape these devices and services to meet their particular needs in a sustainable way. We found that the articulation work needed for maintaining the GPS technology in “working order” was extensive and ongoing. This articulation work does not merely supplement formal procedures, a lot of it is needed to get round them, but it is also often invisible and thus its importance goes largely unrecognized. If GPS technologies are to be implemented at scale and sustainably, methods must be found to capitalize on the skills and tacit knowledge held within the care network (professional and lay) to resolve problems, improve device design, devise new service solutions, and foster organizational learning.

Social media and user-generated content (UGC) are increasingly important features of journalistic work in a number of different ways. However, their use presents major challenges, not least because information posted on social media is not always reliable and therefore its veracity needs to be checked before it can be considered as fit for use in the reporting of news. We report on the results of a series of in-depth ethnographic studies of journalist work practices undertaken as part of the requirements gathering for a prototype of a social media verification 'dashboard' and its subsequent evaluation. We conclude with some reflections upon the broader implications of our findings for the design of tools to support journalistic work.

Existing target-specific sentiment recognition methods consider only a single target per tweet, and have been shown to miss nearly half of the actual targets mentioned.We present a corpus of UK election tweets, with an average of 3.09 entities per tweet and more than one type of sentiment in half of the tweets.This requires a method for multi-target specific sentiment recognition, which we develop by using the context around a target as well as syntactic dependencies involving the target.We present results of our method on both a benchmark corpus of single targets and the multi-target election corpus, showing state-of-the art performance in both corpora and outperforming previous approaches to multi-target sentiment task as well as deep learning models for singletarget sentiment.

The increase of interest in using social media as a source for research has motivated tackling the challenge of automatically geolocating tweets, given the lack of explicit location information in the majority of tweets. In contrast to much previous work that has focused on location classification of tweets restricted to a specific country, here we undertake the task in a broader context by classifying global tweets at the country level, which is so far unexplored in a real-time scenario. We analyze the extent to which a tweet's country of origin can be determined by making use of eight tweet-inherent features for classification. Furthermore, we use two datasets, collected a year apart from each other, to analyze the extent to which a model trained from historical tweets can still be leveraged for classification of new tweets. With classification experiments on all 217 countries in our datasets, as well as on the top 25 countries, we offer some insights into the best use of tweet-inherent features for an accurate country-level classification of tweets. We find that the use of a single feature, such as the use of tweet content alone-the most widely used feature in previous work-leaves much to be desired. Choosing an appropriate combination of both tweet content and metadata can actually lead to substantial improvements of between 20 and 50 percent. We observe that tweet content, the user's self-reported location and the user's real name, all of which are inherent in a tweet and available in a real-time scenario, are particularly useful to determine the country of origin. We also experiment on the applicability of a model trained on historical tweets to classify new tweets, finding that the choice of a particular combination of features whose utility does not fade over time can actually lead to comparable performance, avoiding the need to retrain. However, the difficulty of achieving accurate classification increases slightly for countries with multiple commonalities, especially for English and Spanish speaking countries.

Rumour stance classification, the task that determines if each tweet in a collection discussing a rumour is supporting, denying, questioning or simply commenting on the rumour, has been attracting substantial interest. Here we introduce a novel approach that makes use of the sequence of transitions observed in tree-structured conversation threads in Twitter. The conversation threads are formed by harvesting users’ replies to one another, which results in a nested tree-like structure. Previous work addressing the stance classification task has treated each tweet as a separate unit. Here we analyse tweets by virtue of their position in a sequence and test two sequential classifiers, Linear-Chain CRF and Tree CRF, each of which makes different assumptions about the conversational structure. We experiment with eight Twitter datasets, collected during breaking news, and show that exploiting the sequential structure of Twitter conversations achieves significant improvements over the non-sequential methods. Our work is the first to model Twitter conversations as a tree structure in this manner, introducing a novel way of tackling NLP tasks on Twitter conversations.

We report findings from a study of call centre staff working to deliver a telecare service designed to enable older people to ‘age in place’. We show the steps they routinely take to produce a care system on behalf of their clients and their families that is both workable within the constraints of available resources and fit-for-purpose. In doing so, we have seen how call centre staff share with one another their experiences and solutions to problems, carry out liaison work with networks of lay carers, and generally act as the ‘glue’ providing the all-important link between otherwise fragmented services. We conclude with some thoughts on the significant technical and organizational challenges if the ‘ageing in place’ vision is to be realized in a practical, secure, dependable and cost-effective way.

Social media platforms provide an increasingly popular means for individuals to share content online. Whilst this produces undoubted societal benefits, the ability for content to be spontaneously posted and reposted creates an ideal environment for rumour and false/malicious information to spread rapidly. When this occurs it can cause significant harm and can be characterised as a “digital wildfire.” In this article, we demonstrate that the propagation and regulation of digital wildfires form important topics for research and conduct an overview of existing work in this area. We outline the relevance of a range of work from the computational and social sciences, including a series of insights into the propagation of rumour and false/malicious information. We argue that significant research gaps remain—for instance, there is an absence of systematic studies on the effects of digital wildfires and there is a need to combine empirical research with a consideration of how the responsible governance of social media can be determined. We propose an agenda for research that establishes a methodology to explore in full the propagation and regulation of unverified content on social media. This agenda promotes high-quality interdisciplinary research that will also inform policy debates.

We report findings from a study that set out to explore the experience of older people living with assisted living technologies and care services. We find that successful 'ageing in place' is socially and collaboratively accomplished - 'co-produced' - day-to-day by the efforts of older people, and their formal and informal networks of carers (e.g. family, friends, neighbours). First, we reveal how 'bricolage' allows care recipients and family members to customise assisted living technologies to individual needs. We argue that making customisation easier through better design must be part of making assisted living technologies 'work'. Second, we draw attention to the importance of formal and informal carers establishing and maintaining mutual awareness of the older person's circumstances day-to-day so they can act in a concerted and coordinated way when problems arise. Unfortunately, neither the design of most current assisted living technologies, nor the ways care services are typically configured, acknowledges these realities of ageing in place. We conclude that rather than more 'advanced' technologies, the success of ageing in place programmes will depend on effortful alignments in the technical, organisational and social configuration of support.

This paper explores some of the problems encountered in using a data archive to build tools for training radiologists to interpret breast screening images. We detail our experiences of taking images and case notes created as part of the work of breast cancer screening and using them as resources for training. Four instances of the use of the archive in training are described in detail and the problems they reveal are discussed. We formulate some general lessons for the mobility and re-use of rich ensembles of data and artefacts drawn from complex professional settings. We argue for a richer representation of the context from which the data was taken than can be achieved through making selected relations explicit in metadata. We also conclude that facilities for correcting and elaborating data should be available at the point of use, and not separated out as distinct activities.

Ontologies – a form of structured and logically related knowledge or classification hierarchy embedded in a computer system – are regarded by many scientists as having enormous promise for the consistent use and re-use of data. To realise this promise, however, is not straightforward. In this paper, based on ethnographic observation, we argue that the challenges for ontology building are ‘social’ as much as they are technical. By this we mean the routine work undertaken in the building process and the problems and difficulties entailed can be understood in terms of the practices of knowledge workers and the practical nature of ‘sorting things out’. Getting a better sense of how, in practise, this work gets done gives a sense of the main challenges of building successful ontologies and how this impacts on the design of tool support. In considering the practices of one group in particular, we try to show how, for members, the technical problems of determining what classification structure is appropriate, and what its boundaries might be, depend substantially on assumptions about the ‘community’ and its interests and purposes. This ‘turn to the social’ has ramifications for the understanding of ontology building and use. Specifically, ‘modelling’ approaches to ontology building tell us little about the practical organisation of the work and how this relates to the prospect of successful sharing. Ethnographic enquiry may reveal important issues that are otherwise missed.

Abstract By making research content more reusable, and providing a social infrastructure that facilitates sharing, the human aspects of the scholarly knowledge cycle may be accelerated and ‘time‐to‐discovery’ reduced. We propose that the key to this is the sharing of methods and processes. We present myExperiment, a social web site for discovering, sharing and curating Scientific Workflows and experiment plans, and describe how myExperiment facilitates the management and sharing of research workflows, supports a social model for content curation tailored to the researcher and community, and supports Open Science by exposing content and functionality to the users' tools and applications. Based on this, we introduce the notion of the Research Object—the work objects that are built, transformed and published in the course of scientific experiments—and suggest that by encapsulating methods with results we can achieve research that is more reusable and repeatable and hence rapid and robust. Copyright © 2010 John Wiley & Sons, Ltd.

We describe the experience of using a combination of requirements engineering techniques (scenarios, storyboards, observation and workshops) in an e-science application to develop a geographical analysis tool for epidemiologists. Problems encountered were: eliciting tacit knowledge; and creating new visions and working practices for our users. The combination of techniques worked well, although observation of working practice was not so effective in this scientific domain, where activity is mainly cognitive.

The three-year UK NeuroGrid project aims to develop a Grid-based collaborative research environment to support the data and compute needs for a UK Neurosciences community. This paper describes the challenges in developing this architecture and details initial results from the development of its first prototype to support psychosis, dementia and stroke research and the social challenges of such a collaborative research project. The paper discusses approaches being taken to explore the collaborative science process to inform the requirements for follow on prototypes and methods utilized to develop an effective project team.

Large-scale clinical studies in neuro-imaging are hampered by several factors including variances in acquisition techniques, quality assurance and access to remote datasets. The Neurogrid project will build on the experience of other UK e-science projects to assemble a grid infrastructure, and apply this to three exemplar areas: stroke, dementia and psychosis, to conduct collaborative neuroscience research.

This paper reports findings from an ethnographic study of the work of Adult and Care of the Elderly Community Mental Health Teams in the context of the deployment of an Electronic Medical Record. Our findings highlight the importance of informal discussions and provisional judgments as part of the process by which teams achieve consensual clinical management decisions over time. We show how paper-based documentation supports this collaborative work by affording both the revision of preliminary clinical management options and the accretion of contributions by team members with different clinical perspectives and expertise. Finally, we consider the implications both for teamwork and the Integrated Care Record (ICR) as clinical documentation becomes increasingly held and distributed electronically.

This paper presents detailed examples of document use and re-use, through an ethnographic study of the knowledge work associated with road safety audit in a civil engineering consultancy The paper incorporates some detailed observation of practices, conversations, and other activities occurring around document re-use in everyday work. It outlines some aspects of the everyday use and re-use of engineering documents in the practical accomplishment of everyday knowledge work as the first stage in considering how these activities can be technologically supported.

The effects of computer-mediated communication on social loafing in brainstorming tasks and social compensation in decision-making tasks are examined. In the first experiment, subjects performed a brainstorming task in either nominal, face-to-face or computer-mediated brainstorming group conditions. Production blocking, in which brainstorming group members interfere with each other's output, was minimised, but the nominal group still out-performed the other groups. In the second experiment, subjects performed a group decision task in face-to-face and computer mediated communication conditions. Social compensation in the presence of social loafing was seen to occur in the first condition, but not in the second. The paper concludes by discussing some of the consequences of both experiments for the future role of computer-mediated communication in group work.

In this paper we describe an experimental evaluation of multimedia packages intended for use as in-house training aids within a large UK bank. We focus on the influence of different kinds of media content and of network quality of service upon subjects' memory for, and comprehension of, the material. In particular, we observe that degraded quality of service has a greater influence on subjects' uptake of emotive/affective content than on their uptake of factual content. The results have implications for the more general application of multimedia communication.

The advent of the virtual library is usually presented as a positive development for library users. However, much of the research and development work being carried out in this field tends to reinforce the perception of the use of information resources as a predominantly solitary activity. We argue that this narrow view of the virtual library may be counterproductive to its aims. Recent studies have emphasized that information retrieval (IR) in the conventional library is often a highly collaborative activity, involving library users peers and experts in IR such as librarians. Failure to take this into account in the move to digitally based resource discovery and access may result in users of the virtual library being disadvantaged through lack of timely and effective access to sources of assistance. Our focus here is on the ways in which, in the conventional library setting, users consult with reference librarians for the resolution of their IR problems. We describe an investigation of consultation and collaboration issues as seen from the perspective of librarians and users contexts and analysed within the framework of genre. Drawing upon this analysis, we then describe the design of a prototype network, multimedia-based system which is intended to support collaboration between librarians and IR system users in the virtual library.

Standards are one of the key components of information technology today. Yet, despite their public and widespread implications, the underlying standardisation processes are something of a "black box" to outsiders. This begs the question of how standards are determined, and by whom. This paper looks at the lowest and most technical level of this process, where the basic technological decisions are made. The results of a survey of standards professionals and standards users are presented and analysed. We examine the composition of international committees charged with defining and maintaining standards for electronic mail systems and report the views of senior committee members, including their opinions of the process, how it could be improved, and their reactions to increased user representation. We then present and analyse corporate users' views on participation in standards setting. Finally, the implications of our findings for standards setting are discussed and recommendations are made for addressing the problems revealed.

Groupware is designed to provide opportunities for physically dispersed computer users to co-operate in a manner akin to a face-to-face meeting. Little is understood, however, of the factors that might influence its success. One possible factor is ‘floor control’, or turntaking, which is an important feature of face-to-face meetings. The paper describes experiments designed to examine the importance of turn-taking in computer-mediated communications, in comparison with face-to-face conversations, and considers means whereby turntaking behaviour, and hence the effectiveness of groupware, can be improved.