We take first steps in exploring whether Large Language Models (LLMs) can be adapted to dialogic learning practices, specifically pair programming — LLMs have primarily been implemented as programming assistants, not fully exploiting their dialogic potential. We used new dialogue data from real pair-programming interactions between students, prompting state-of-the-art LLMs to assume the role of a student, when generating a response that continues the real dialogue. We asked human annotators to rate human and AI responses on the criteria through which we operationalise the LLMs’ suitability for educational dialogue: Coherence, Collaborativeness, and whether they appeared human. Results show model differences, with Llama-generated responses being rated similarly to human answers on all three criteria. Thus, for at least one of the models we investigated, the LLM utterance-level response generation appears to be suitable for pair-programming dialogue.

The aim of this research was to discover if artificial neural networks can be used to classify pressure sensor data generated by medical detection dogs as they sniff biological samples. A detection dog can be trained to recognise the odour emitted by one of a wide range of diseases such as prostate cancer, malaria or, potentially, COVID-19. The dog searches a row of sample pots and indicates a positive sample by sitting in front of it. This offers a non-invasive means of diagnosing the specific cancer or disease that the dog has been trained to recognise. For this study, pressure sensors were attached to the sample pots to generate time series data pertaining to the dog’s searching behaviour as they press their nose against the sample pot to sniff its content. Automatic classification could provide a second form of indication, to support or refute the dog’s explicit signal (to sit at a positive sample), which is not always correct. Ultimately, classification software could eliminate the need for the dog to perform an indication gesture, making the dog’s task easier and training quicker. Four different neural network architectures were evaluated: multilayer perceptron (MLP), a convolutional neural network (CNN), a fully convolutional network (FCN) and ResNet (a deep convolutional neural network). Each model was trained to classify the pressure data generated by medical detection dogs. To achieve a useful level of accuracy, it was found that the models needed to be trained using only those data samples where the dog had correctly classified the scent sample. Model hyperparameters were tuned to improve accuracy. We found that the best performing model was MLP. When tested on previously unseen data, where the dog was not always correct, the classification performance of the MLP approached that of the medical detection dogs. For our particular dataset, the model’s true positive rate (i.e. recall) was 59%, matching that of the dogs. The model’s true negative rate was 79%, compared to the dogs’ 91%.

A basic step in any annotation effort is the measurement of the Inter Annotator Agreement (IAA).An important factor that can affect the IAA is the presence of annotator bias.In this paper we introduce a new interpretation and application of the Item Response Theory (IRT) to detect annotators' bias.Our interpretation of IRT offers an original bias identification method that can be used to compare annotators' bias and characterise annotation disagreement.Our method can be used to spot outlier annotators, improve annotation guidelines and provide a better picture of the annotation reliability.Additionally, because scales for IAA interpretation are not generally agreed upon, our bias identification method is valuable as a complement to the IAA value which can help with understanding the annotation disagreement.

Students who study problem solving and programming (in a language such as Python) at University level encounter a range of challenges, from low-level issues with code that won't compile to misconceptions about the threshold concepts and skills. The current study complements existing findings on errors, misconceptions, difficulties and challenges obtained from students after-the-fact through instruments such as questionnaires and interviews. In our study, we analysed the posts from students of a large cohort (\textasciitilde1500) of first-year University distance learning students to an online 'Python help forum' - recording issues and discussions as the students encountered specific challenges. Posts were coded in terms of topics, and subsequently thematically grouped into Python-related, problem solving/generic programming related, and module specific. We discuss the set of topics and rank these in terms of the number of forum discussions in which they occur (as a proxy for their prevalence). The top challenges we identified concern student understanding and use of a mix of programming environments (in particular, Python IDLE for offline programming and CodeRunner for programming quizzes) and code fragment problems. Apart from these, Python-specific topics include, among others, collections, functions, error messages, iteration, outputting results, indentation, variables and imports. We believe that the results provide a good insight into the challenges that students encounter \em as they learn to program. In future work we intend to study the discussions in further detail in terms of theories of conceptual change.

Inter-Annotator Agreement (IAA) is used as a means of assessing the quality of NLG evaluation data, in particular, its reliability.According to existing scales of IAA interpretationsee, for example, Lommel et al. (2014), Liu et al. (2016), Sedoc et al. (2018) and Amidei et al. (2018a) -most data collected for NLG evaluation fail the reliability test.We confirmed this trend by analysing papers published over the last 10 years in NLG-specific conferences (in total 135 papers that included some sort of human evaluation study).Following Sampson and Babarczy (2008), Lommel et al. (2014), Joshi et al. (2016) and Amidei et al. ( 2018b), such phenomena can be explained in terms of irreducible human language variability.Using three case studies, we show the limits of considering IAA as the only criterion for checking evaluation reliability.Given human language variability, we propose that for human evaluation of NLG, correlation coefficients and agreement coefficients should be used together to obtain a better assessment of the evaluation data reliability.This is illustrated using the three case studies.

Rating and Likert scales are widely used in evaluation experiments to measure the quality of Natural Language Generation (NLG) systems.We review the use of rating and Likert scales for NLG evaluation tasks published in NLG specialized conferences over the last ten years (135 papers in total).Our analysis brings to light a number of deviations from good practice in their use.We conclude with some recommendations about the use of such scales.Our aim is to encourage the appropriate use of evaluation methodologies in the NLG community.

Human evaluations are broadly thought to be more valuable the higher the inter-annotator agreement. In this paper we examine this idea. We will describe our experiments and analysis within the area of Automatic Question Generation. Our experiments show how annotators diverge in language annotation tasks due to a range of ineliminable factors. For this reason, we believe that annotation schemes for natural language generation tasks that are aimed at evaluating language quality need to be treated with great care. In particular, an unchecked focus on reduction of disagreement among annotators runs the danger of creating generation goals that reward output that is more distant from, rather than closer to, natural human-like language. We conclude the paper by suggesting a new approach to the use of the agreement metrics in natural language generation evaluation tasks.

In the last few years Automatic Question Generation (AQG) has attracted increasing interest.In this paper we survey the evaluation methodologies used in AQG.Based on a sample of 37 papers, our research shows that the systems' development has not been accompanied by similar developments in the methodologies used for the systems' evaluation.Indeed, in the papers we examine here, we find a wide variety of both intrinsic and extrinsic evaluation methodologies.Such diverse evaluation practices make it difficult to reliably compare the quality of different generation systems.Our study suggests that, given the rapidly increasing level of research in the area, a common framework is urgently needed to compare the performance of AQG systems and NLG systems more generally.

Most work on automatic generation of narratives, and more specifically suspenseful narrative, has focused on detailed domain-specific modelling of character psychology and plot structure.Recent work on the automatic learning of narrative schemas suggests an alternative approach that exploits such schemas for modelling and measuring suspense.We propose a domain-independent model for tracking suspense in a story which can be used to predict the audience's suspense response on a sentence-by-sentence basis at the content determination stage of narrative generation.The model lends itself as the theoretical foundation for a suspense module that is compatible with alternative narrative generation theories.The proposal is evaluated by human judges' normalised average scores correlate strongly with predicted values.

This paper introduces a novel method for measuring non-cooperation in dialogue. The key idea is that linguistic non-cooperation can be measured in terms of the extent to which dialogue participants deviate from conventions regarding the proper introduction and discharging of conversational obligations (e.g., the obligation to respond to a question). Previous work on non cooperation has focused mainly on non-linguistic task-related non-cooperation or modelled non-cooperation in terms of special rules describing non-cooperative behaviours. In contrast, we start from rules for normal/correct dialogue behaviour - i.e., a dialogue game - which in principle can be derived from a corpus of cooperative dialogues, and provide a quantitative measure for the degree to which participants comply with these rules. We evaluated the model on a corpus of political interviews, with encouraging results. The model predicts accurately the degree of cooperation for one of the two dialogue game roles (interviewer) and also the relative cooperation for both roles (i.e., which interlocutor in the conversation was most cooperative). Being able to measure cooperation has applications in many areas from the analysis - manual, semi and fully automatic - of natural language interactions to human-like virtual personal assistants, tutoring agents, sophisticated dialogue systems, and role-playing virtual humans.

Question generation (QG) is the problem of automatically generating questions from inputs such as declarative sentences. The Shared Evaluation Task Challenge (QG-STEC) Task B that took place in 2010 evaluated several state-of-the-art QG systems. However, analysis of the evaluation results was affected by low inter-rater reliability. We adapted Nonaka & Takeuchi’s knowledge creation cycle to the task of improving the evaluation annotation guidelines with a preliminary test showing clearly improved inter-rater reliability.

The development of students' critical thinking skills is an important part of a university education. This paper focuses on helping Computing and Technology (C&T) students improve their argument analysis and construction skills. At the heart of the current approach sits the use of a diagram-based technique for scaffolding argument analysis and construction. The technique, known as argument mapping, has been pioneered in Philosophy degrees. We discuss the specific challenges posed by using this technique in the context of distance learning for C&T students of a large (>1000 students) entry-level gateway module. Our approach includes innovative interactive computer-based formative assessment of argument mapping. After describing the approach, we evaluate it in terms of data from formative and summative assessment, detailed analysis of student work, and feedback from both students and module tutors.

This PhD project aims to investigate the impact of affective embodied conversational agents (ECAs) on the learning of pupils. Based on the idea that there is a link between emotions and learning, we are developing an affective tutoring system in two domains: Information Technology and Business Studies. We will evaluate the system in a classroom setting over several weeks, with each student being assigned to one of the following two conditions: ECA with cognitive strategy only versus ECA with combined affective and cognitive strategy.

Pointing combined with verbal referring is one of the most paradigmatic human multimodal behaviours. The aim of this paper is foundational: to uncover the central notions that are required for a computational model of multimodal referring acts that include a pointing gesture. The paper draws on existing work on the generation of referring expressions and shows that in order to extend that work with pointing, the notion of salience needs to play a pivotal role. The paper starts by investigating the role of salience in the generation of referring expressions and introduces a distinction between two opposing approaches: salience-first and salience-last accounts. The paper then argues that these differ not only in computational efficiency, as has been pointed out previously, but also lead to incompatible empirical predictions. The second half of the paper shows how a salience-first account nicely meshes with a range of existing empirical findings on multimodal reference. A novel account of the circumstances under which speakers choose to point is proposed that directly links salience with pointing. Finally, this account is placed within a multi-dimensional model of salience for multimodal reference.

For the past ten years or more, most work in the field of Natural Language Generation (NLG) has shied away from considerations regarding the processes underlying human language production. Rather, the focus has been on systems that automatically produce language—usually text—from non-linguistic representations, with the main objective being generation of a text that faithfully captures the meaning of those nonlinguistic representations (see, e.g., Reiter and Dale’s 2000 textbook on NLG). There is, however, also a different take on NLG “as not just competent performance by a computer but the development of a computational theory of the human capacity for language and processes that engage it” (McDonald 1987, page 642). Guhe’s research monograph, based on his 2003 Ph.D. thesis, is firmly situated in the latter tradition. One of his main goals is to work out a computational architecture for Levelt’s (1989) psycholinguistically motivated model of language production. According to Levelt’s model, speaking involves three main activities: conceptualizing (deciding what to say), formulating (deciding how to say it), and articulating (saying it). Guhe’s book focuses on the mental activity of conceptualizing. Conceptualizing is a recalcitrant object of study, partly because of the problem of the “initial spark”; the decision to say something appears to be the result of volitional conscious decisions, which largely elude scientific study. Guhe avoids this problem by investigating conceptualization in settings where the main intention is already fixed: a speaker witnesses several events unfold and is instructed to describe what happens (while it happens). The research challenge then is to figure out how “subintentions” for individual speech acts come about. The benefit of using an on-line generation setting is that it provides information on both what a speaker says at a given point in time and what is being reported, that is, the data that drive the speaker’s utterances. The book consists of the usual preface and introduction, followed by four parts (A, B, C, and Results), a list of the book’s theses, and an appendix that includes, among other things, a glossary, bibliography, name index, and subject index. Part A of the book is titled “Conceptualization.” It starts with an introduction to the field of language production, with particular reference to Levelt’s (1989) model. The notion of conceptualization as a “quasi-module,” partly using Fodor’s (1983) criteria, is presented and four subtasks of conceptualization are discussed:

Abstract This paper describes CLIME, a web-based legal advisory system with a multilingual natural language interface. CLIME is a ‘proof-of-concept’ system which answers queries relating to ship-building and ship-operating regulations. Its core knowledge source is a set of such regulations encoded as a conceptual domain model and a set of formalised legal inference rules. The system supports retrieval of regulations via the conceptual model, and assessment of the legality of a situation or activity on a ship according to the legal inference rules. The focus of this paper is on the natural language aspects of the system, which help the user to construct semantically complex queries using WYSIWYM technology, allow the system to produce extended and cohesive responses and explanations, and support the whole interaction through a hybrid synchronous/asynchronous dialogue structure. Multilinguality (English and French) is viewed simply as interface localisation: the core representations are language-neutral, and the system can present extended or local interactions in either language at any time. The development of CLIME featured a high degree of client involvement, and the specification, implementation and evaluation of natural language components in this context are also discussed.