The integration of conversational agents into daily life has become increasingly common. However, sustaining deeply engaging and natural interactions remains challenging due to a lack of multimodal datasets capturing personal and emotional nuances. In this paper, we introduce MERCI (Multimodal dataset for Emotionally-aware peRsonalised Conversational In-teractions), a dataset derived from user-robot dialogues involving thirty participants who completed user profile questionnaires covering ten personal topics (e.g., hobbies, music). A conver-sational system called PERCY then engaged with each partici-pant in open-domain conversations, leveraging GPT-4, real-time facial-expression and sentiment analysis to generate contextu-ally appropriate, empathetic responses. MERCI contains 1860 utterances, equating to about 12.5 hours of aligned audio, three-view video, transcripts with timestamps, emotion labels, and sentiment scores. This dataset serves as a reproducible test-bed for tasks such as emotion-aware response generation, multimodal affect recognition, and personalised policy learning. Baseline performance results have been established using advanced models such as BERT, T5, BART, and GPT-3.5/4/4o-mini across gener-ation, regression, and classification. Evaluations through human and automated methods have demonstrated strong naturalness, relevance, and consistency in responses while indicating areas for enhanced personalisation and empathic depth. We expect that MERCI will enhance the development of emotionally intelligent, user-centric conversational AI applications, potentially ranging from social robotics to mental health support.

Human-robot interaction encompasses various applications, from simple tasks to advanced complex functionalities. This research introduces a specialised robotics end effector specially modelled with auditory processing for drumming along-side humans. This system interprets auditory vocal input to generate precise and adaptive drumming patterns. Utilising precision motor control and real-time signal processing to replicate drumming with human-like characteristics with precision. This work demonstrates collaboration between performing arts and robotics, demonstrating the potential of intelligent systems to interact with human creativity in real-time. The specialised end effector utilises precise motor control with motor drivers and a belt mechanism to regulate the speed and direction of the drumstick, enabling rapid, high-speed strokes to strike the drum 4 to 5 times per second. By integrating audio perception with mechanical execution, this research advances the state of the art and establishes a new framework for incorporating robotics into the creative and artistic domain.

Human-robot interaction plays a vital role in pushing the capabilities of socially interactive robots by enabling them to deliver content with high emotional intelligence. This research focuses on a supervisory fuzzy framework for constantly evaluating and improving the content delivered by the robot utilizing multimodal inputs and advanced intelligent algorithms. The main reason for using fuzzy logic is that it mimics human decision-making by providing a percentage-based measure of closeness. In this project, ARI Robot is being used with an LLM integration, which enables the user to communicate with the robot. Different algorithms were integrated for the classification of multimodal inputs, BERT (Bidirectional Encoder Representations from Transformers) for the classification of content, Wav2Vec 2.0 for classifying the tone of the user while interacting with the robot, and OpenFace for classifying the facial expression of the user. All of these inputs are then supervised by a fuzzy system with predefined rules to evaluate the content delivered and provide feedback for refinement. The proposed framework ensures an overall evaluation of content delivery, providing intelligent feedback to the ARI robot to improve interaction quality. By integrating these advanced models with fuzzy logic, the system mimics human-like judgment in assessing the interaction of verbal and non-verbal indications, making the way for emotionally intelligent robots in a social world.

Generative AI is transforming the way humans interact with robots by integrating language-driven comprehension with embodied execution. While recent research leveraging large language models (LLMs) to enhance communication between humans and machines has shown significant progress, the exploration of integrating LLMs with robots to assist users in creating real-world artistic works remains challenging. This research introduces a novel GENAI-driven intelligent robotic arm drawing system. We fine-tuned GPT-3.5 Turbo to better capture meaningful information from conversations with users and output precise drawing commands, which are directly fed into the fine-tuned Stable Diffusion model to generate desired images. A UFactory xArm equipped with an end-effector holding a drawing pen is deployed and controlled by a speed control algorithm to perform accurate drawing movements based on AI-generated images. Our proposed framework facilitates enhanced HRI experiences with a focus on drawing tasks, effectively embodying the capabilities of generative AI in the field of artistic creation and enabling users to engage deeply in the process of using AI for art creation in real-world environments.