The Generalized Referring Expression Comprehension (GREC) task extends classic REC by generating image bounding boxes for objects referred to in natural language expressions, which may indicate zero, one, or multiple targets. This generalization enhances the practicality of REC models for diverse real-world applications. However, the presence of varying numbers of targets in samples makes GREC a more complex task, both in terms of training supervision and final prediction selection strategy. Addressing these challenges, we introduce RECANTFormer, a one-stage method for GREC that combines a decoder-free (encoder-only) transformer architecture with DETR-like Hungarian matching. Our approach consistently outperforms baselines by significant margins in three GREC datasets.

Co-speech gestures enhance both human-human and human-robot interactions. This paper examines the efficacy of a data-driven approach for generating synchronised co-speech gestures in three social robots to improve social interactions. Building on a sequence-to-sequence model, which maps speech to gestures [21], this work uses the Talking With Hands 16.2M dataset [11] to generate natural gestures for face-to-face conversations. Additionally, we address synchronisation issues identified in the original study. The model’s generality is tested on three robots—NAO, Pepper, and ARI. Objective and subjective evaluations, confirm that a data-driven approach effectively generates synchronised co-speech gestures.

In public and social spaces shared by humans and robots, it is essential for both parties to be aware of each other’s goals and to communicate their intentions effectively. This work in progress aims to develop a motion planner that not only finds the shortest path to a goal while adhering to social norms and spatial constraints but also generates communicative gestures and movements, such as hesitations or prompting motions. Extensive datasets are required to teach robots socially compliant navigation and effective communication with their human counterparts. Here, we outline our data collection efforts to train such a planner and to inform the broader community about potential interactions between humans and robots in these scenarios. The primary goal of this research is to establish a foundation for user-centered design of interaction strategies and avoidance mechanisms during social navigation.

Natural co-speech gestures are essential components to improve the experience of Human-robot interaction (HRI). However, current gesture generation approaches have many limitations of not being natural, not aligning with the speech and content, or the lack of diverse speaker styles. Therefore, this work aims to repoduce the work by [5] generating natural gestures in simulation based on tri-modal inputs and apply this to a robot. During evaluation, “motion variance” and “Frechet Gesture Distance (FGD)” is employed to evaluate the performance objectively. Then, human participants were recruited to subjectively evaluate the gestures. Results show that the movements in that paper have been successfully transferred to the robot and the gestures have diverse styles and are correlated with the speech. Moreover, there is a significant likeability and style difference between different gestures.

While research in multi-party spoken conversation with intelligent embodied agents has made significant progress in sub-tasks like speaker identification and non-verbal cues, there’s a gap in fully autonomous applications users can directly interact with. This lack translates to the absence of a standard methodology for evaluating multi-party conversational speech agents that considers both task-based system performance and user experience.

Ambiguous Candidate Identification (ACI) in multimodal dialogue is the task of identifying all potential objects that a user's utterance could be referring to in a visual scene, in cases where the reference cannot be uniquely determined.End-to-end models are the dominant approach for this task, but have limited real-world applicability due to unrealistic inference-time assumptions such as requiring predefined catalogues of items.Focusing on a more generalized and realistic ACI setup, we demonstrate that a modular approach, which first emphasizes language-only reasoning over dialogue context before performing vision-language fusion, significantly outperforms end-to-end trained baselines.To mitigate the lack of annotations for training the language-only module (student), we propose a pseudo-labelling strategy with a prompted Large Language Model (LLM) as the teacher.

Social Robots in human environments need to be able to reason about their physical surroundings while interacting with people. Furthermore, human proxemics behaviours around robots can indicate how people perceive the robots and can inform robot personality and interaction design. Here, we introduce Charlie, a situated robot receptionist that can interact with people using verbal and non-verbal communication in a dynamic environment, where users might enter or leave the scene at any time. The robot receptionist is stationary and cannot navigate. Therefore, people have full control over their personal space as they are the ones approaching the robot. We investigated the influence of different apparent robot personalities on the proxemics behaviours of the humans. The results indicate that different types of robot personalities, specifically introversion and extroversion, can influence human proxemics behaviours. participants maintained shorter distances with the introvert robot receptionist, compared to the extrovert robot. Interestingly, we observed that human-robot proxemics were not the same as typical human-human interpersonal distances, as defined in the literature. We therefore propose new proxemics zones for human-robot interaction.

Angus Addlesee, Weronika Sieińska, Nancie Gunson, Daniel Hernandez Garcia, Christian Dondrup, Oliver Lemon. Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue. 2023.

Possible applications for Social Robots in health-care settings, that could have a tremendous social impact in helping alleviating staff workload, are those of a patient-facing role such as robot receptionist, providing assistance to patients and visitors. Examples of functions that such robots would need to be able to execute are greeting visitors, reception check-in/out of patients, answering common questions they may have, showing them where to sit, helping them locate missing objects, providing directions to facilities, guiding them to different locations, etc. In this paper we describe current progress towards developing a multimodal conversational AI system integrated in a Social Conversational Robot (an ARI robot) that will act as a receptionist in a hospital waiting room. We present the developed architecture of the system and report on an initial experimental validation study carried out in laboratory conditions with the ARI robot.

Nancie Gunson, Daniel Hernandez Garcia, Weronika Sieińska, Angus Addlesee, Christian Dondrup, Oliver Lemon, Jose L. Part, Yanchao Yu. Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2022.

We will demonstrate a prototype multimodal conversational AI system that will act as a receptionist in a hospital waiting room, combining visually-grounded dialogue with social conversation. The system supports visual object conversation in the waiting room (e.g. looking for available seats or personal belongings), task-based dialogues regarding navigation and check-in procedures in the hospital, as well as access to the latest news, and a quiz game about coronavirus. The prototype system therefore demonstrates how to weave together a wide range of natural, daily conversations with end users that vary in complexity; from complex visual dialogue to chitchat and quiz games, to task-oriented domain-specific conversations. We are currently able to demonstrate the system via a web-based interface. It will soon be deployed on the ARI robot in a hospital waiting room.

Recent developments in computer vision and conversational systems have provided the AI community with novel perspectives towards improving the cognitive capabilities of engaging socially assistive robots. We show how to develop conversational skills for a hospital receptionist robot that incorporates social conversation based on visual information as well as task-based dialog. Fusing the traditional modular conversational system architecture with recent developments in computer vision and scene graph research, our agent (called ‘ViCA’) supports both visual question answering and social conversational capabilities based on the visual scene. In particular, our agent can provide guidance to users by locating visible objects in the room and can engage in social dialog using visual prompts, such as the user’s clothing or possessions. We conduct a comprehensive online evaluation study with 21 participants, showcasing that the ViCA system is perceived as both helpful and entertaining.

Public perceptions of Robotics and Artificial Intelligence (RAI) are important in the acceptance, uptake, government regulation and research funding of this technology. Recent research has shown that the public's understanding of RAI can be negative or inaccurate. We believe effective public engagement can help ensure that public opinion is better informed. In this paper, we describe our first iteration of a high throughput in-person public engagement activity. We describe the use of a light touch quiz-format survey instrument to integrate in-the-wild research participation into the engagement, allowing us to probe both the effectiveness of our engagement strategy, and public perceptions of the future roles of robots and humans working in dangerous settings, such as in the off-shore energy sector. We critique our methods and share interesting results into generational differences within the public's view of the future of Robotics and AI in hazardous environments. These findings include that older peoples' views about the future of robots in hazardous environments were not swayed by exposure to our exhibit, while the views of younger people were affected by our exhibit, leading us to consider carefully in future how to more effectively engage with and inform older people.

We present and evaluate a deployed conversational AI system that acts as a host of a working public building on a university campus. The system combines open-domain social chat with task-based conversation regarding navigation in the building, live resource updates (e.g. available computers), and events in the building. We investigated the impact of open-domain social chat on task completion and user preferences by comparing the combined system with a task-only version. We find that there is no significant difference in task completion or several aspects of user preference between the two systems, but that users would be significantly happier to talk to the task-only system in the future. This suggests that the "walk-up" public setting and workplace nature of the environment creates a markedly different use case to the in-home, and more individual and private "companion/assistant" setting which is commonly assumed for systems like Alexa. We discuss the implications for the design of conversational systems in other public settings.

We will demonstrate a deployed conversational AI system that acts as a host of a smartbuilding on a university campus.The system combines open-domain social conversation with task-based conversation regarding navigation in the building, live resource updates (e.g.available computers) and events in the building.We are able to demonstrate the system on several platforms: Google Home devices, Android phones, and a Furhat robot.

Robotics for hazardous environments is currently an important area of research, with the ambition of reducing human risk in potentially devastating situations. Here, we are presenting a Modular Control Framework (MCF) for a low-cost robot with limited sensory resources to address this issue. As a proof of concept, we emulate 3 scenarios - (1) adaptive planning for obstruction avoidance (road block), (2) object identification and support-case-based behaviour adjustment (search and rescue) and (3) autonomous navigation through the environment with reporting of structural status (patrol and monitoring). These were implemented and validated using a Cozmo robot in a small-scale Lego environment. We found that our system can reroute in 90%, can help an injured person 80% and report about failing equipment in 80% of all tested cases, where most of the fails were caused by the object detection used. Our MCF is implemented using ROS, making it easy to use and adjust for other robotic platforms.

Most of today's task-based spoken dialogue systems perform poorly if the user goal is not within the system's task domain. On the other hand, chatbots cannot perform tasks involving robot actions but are able to deal with unforeseen user input. To overcome the limitations of each of these separate approaches and be able to exploit their strengths, we present and evaluate a fully autonomous robotic system using a novel combination of task-based and chat-style dialogue in order to enhance the user experience with human-robot dialogue systems. We employ Reinforcement Learning (RL) to create a scalable and extensible approach to combining chat and task-based dialogue for multimodal systems. In an evaluation with real users, the combined system was rated as significantly more “pleasant” and better met the users' expectations in a hybrid task+chat condition, compared to the task-only condition, without suffering any significant loss in task completion.

The eldercare sector is a promising deployment area for robotics where robots can support staff and help to bridge the predicted staff-shortage. A requirement analysis showed that one field of robot-deployment could be supporting physical therapy of older adults with advanced dementia. To explore this possibility, a long-term autonomous robot was deployed as a walking group assistant at a care site for the first time. The robot accompanied two weekly walking groups for a month, offering visual and acoustic stimulation. Therapists' experience, the robot's influence on the dynamic of the group and the therapists' estimation of the robot's utility were assessed by a mixed methods design consisting of observations, interviews and rating scales. Findings suggest that a robot has the potential to enhance motivation, group coherence and also mood within the walking group. Furthermore, older adults show curiosity and openness towards the robot. However, robustness and reliability of the system must be high, otherwise technical problems quickly turn the robot from a useful assistant into a source of additional workload and exhaustion for therapists.

In this work, we propose the combination of a state-of-the-art sampling-based local planner with so-called Velocity Costmaps to achieve human-aware robot navigation. Instead of introducing humans as “special obstacles” into the representation of the environment, we restrict the sample space of a “Dynamic Window Approach” local planner to only allow trajectories based on a qualitative description of the future unfolding of the encounter. To achieve this, we use a Bayesian temporal model based on a Qualitative Trajectory Calculus to represent the mutual navigation intent of human and robot, and translate these descriptors into sample space constraints for trajectory generation. We show how to learn these models from demonstration and evaluate our approach against standard Gaussian cost models in simulation and in real-world using a non-holonomic mobile robot. Our experiments show that our approach exceeds the performance and safety of the Gaussian models in pass-by and path crossing situations.

We present a pilot study to identify hesitation signals in Human-Robot Spatial Interaction which we aim to employ to evaluate the quality of the robots executed behaviour. The presented study focuses on head-on encounters between a human and a robot in pass-by scenarios. Our results indicate that these hesitation signals can be found and therefore present a form a implicit feedback.

This paper presents a new approach to mobile robot mapping in long-term scenarios. So far, the environment models used in mobile robotics have been tailored to capture static scenes and dealt with the environment changes by means of `memory decay'. While these models keep up with slowly changing environments, their utilization in dynamic, real world environments is difficult. The representation proposed in this paper models the environment's spatio-temporal dynamics by its frequency spectrum. The spectral representation of the time domain allows to identify, analyse and remember regularly occurring environment processes in a computationally efficient way. Knowledge of the periodicity of the different environment processes constitutes the model predictive capabilities, which are especially useful for long-term mobile robotics scenarios. In the experiments presented, the proposed approach is applied to data collected by a mobile robot patrolling an indoor environment over a period of one week. Three scenarios are investigated, including intruder detection and 4D mapping. The results indicate that the proposed method allows to represent arbitrary timescales with constant (and low) memory requirements, achieving compression rates up to 106. Moreover, the representation allows for prediction of future environment states with ~ 90% precision.

In this paper we propose to augment a wellestablished Qualitative Trajectory Calculus (QTC) by incorporating social distances into the model to facilitate a richer and more powerful representation of Human-Robot Spatial Interaction (HRSI). By combining two variants of QTC that implement different resolutions and switching between them based on distance thresholds we show that we are able to both reduce the complexity of the representation and at the same time enrich QTC with one of the core HRSI concepts: proxemics. Building on this novel integrated QTC model, we propose to represent the joint spatial behaviour of a human and a robot employing a probabilistic representation based on Hidden Markov Models. We show the appropriateness of our approach by encoding different HRSI behaviours observed in a human-robot interaction study and show how the models can be used to represent and classify these behaviours using social distance-augmented QTC.