We introduce an open-source ROS 2 architecture for situated social robots, along with a simulator that allows mixed-reality development and interactions. The architecture is a hybrid symbolic/subsymbolic system that integrates explicit ontology semantics for perception, reasoning, and execution, with LLMs. It features multimodal social perception by leveraging the open source ROS4HRI framework; LLMs (both edge- and cloud-based) to facilitate natural language interaction between the user and system; KnowledgeCore, an open-source knowledge base, to reason about facts in the world; and an intent-based controller to supervise the execution of parallel/sequential tasks and skills. We demonstrate our system architecture with a social robot running the mixed-reality system.

This paper presents a new approach for representing spatially-grounded semantics in interactive robots. The method combines spatial and symbolic data to improve robot interactions in human-occupied environments. A key feature is a voxel-based data structure optimized for dynamic and sparse information, along with a global lookup table to manage and track spatially-grounded entities and their relationships. The implementation, which is integrated into a ROS 2-based framework, allows for seamless querying through semantic web APIs such as SPARQL. Initial tests demonstrate the efficiency of this system in supporting advanced scenarios in human-robot interaction. All the repositories developed as part of this contribution can be found at github.com/RepresentationMaps.

This tutorial aims at providing an up-to-date picture of the state-of-art regarding using the Robot Operating System (ROS) to build robots with socio-cognitive capabilities. The tutorial will briefly introduce the ROS4HRI framework, and show how it can be used to build a complete social robot architecture, from human perception to expressive social interaction. We will illustrate the full software integration required to implement an autonomous social robot using a combination of open-source ROS-based social perception modules, a semantic knowledge base, a Large Language Model (LLM), and multi-modal expressiveness. Participants will be able to follow along using a simple social interaction simulator, as well as their own webcams. The organisers will also provide a new PAL Robotics TIAGo Pro stand-alone head to demonstrate the same system running on actual hardware.

This paper presents the TIAGo Head, a new tabletop social robot from PAL Robotics, focusing on its capabilities as an HRI platform. We detail the robots’ hardware, highlighting its sensors/actuators and on-board computing; and its software architecture, including social perception, expressive face, a knowledge base, and integration with large language models (LLMs) for natural conversations. We also describe a use-case in a receptionist scenario where TIAGo Head dynamically interacts with travelers by displaying news and conversing.

In social robots, visual attention expresses awareness of the scenario components and dynamics. As in humans, their attention should be driven by a combination of different attention mechanisms. In this article, we introduce multi-modal saliency maps, i.e., spatial representations of saliency that dynamically integrate multiple attention sources depending on the context. We provide the mathematical formulation of the model and an open source software implementation. Finally, we present an initial exploration of its potential in social interaction scenarios with humans and evaluate its implementation.

Multi-modal social perception usually involves several independent software modules, detecting for instance faces, voices, body skeletons. Those features need then to be matched to each other, to create a complete model of a person. While the problem is simple in one-to-one interactions, multi-party interactions require to optimize a probabilistic graph in order to find the most likely persons--features associations, while ensuring practical properties like stability over time. This paper presents an open-source algorithm that searches over all possible partitions of the relationship graph to identify the best partition. We playfully call this algorithm Mr. Potato, after the eponymous children' game.

We present a software tool, fully integrated with ROS, that enables robots to perceive people full body in 3D. The system works either with a simple RGB camera, or a RGB-D camera for better 3D absolute position estimation. The system is based on Google Mediapipe, and runs at > 8Hz on CPU. The consistency of the human kinematic model is ensured by relying on a URDF-defined kinematic model, that could be adjusted to each person's anthropometric characteristics.