We integrated state-of-the-art vision–language models (VLMs) into a hat shop robot, enabling four visually grounded human shop-worker behaviors: 1) offering personalized greetings based on customers' visual aspects, 2) recognizing their shopping activities and delivering timely encouragement to try hats, 3) offering feedback on hat fit, and 4) responding to customer inquiries that contain deictic references to hats. We strategically utilized both local and cloud-based VLMs to develop the robot’s visual perception, leveraging their complementary capabilities. We report key insights from our development process, including our experience collaborating with a domain expert on robot interaction development. Results from an 8-day field trial at a hat shop show that the robot’s advanced visual perception capabilities made a positive impression on both customers and the shop manager. Customers were excited by the robot’s visually grounded remarks, and the manager adopted a complementary working style that leveraged these new capabilities.

We developed an autonomous hatshop robot for encouraging customers to try on hats by providing comments that appropriately fit their actions, and in such a way also indirectly exerting social pressure. To enable it to offer such a service smoothly in a real shop, we developed a large system (around 150k lines of code with 23 ROS packages) integrated with various technologies, like people tracking, shopping activity recognition and navigation. The robot needed to move in narrow corridors, detect customers, and recognise their shopping activities. We employed an iterative development process, repeating trial-and-error integration with the robot in the actual shop, while also collecting real-world data during field-testing. This process enabled us to improve our shopping activity recognition system by collecting real-world data, and to adapt our software modules to the target shop environment. We report the lessons learnt during our system development process. The results of our 11-day field trial show that our robot was able to provide its services reasonably well. Many customers expressed a positive impression of the robot and its services.

We developed a security guard robot that is specifically designed to manage queues of people and conducted a field trial at an actual public event to assess its effectiveness. However, the acceptance of robot instructions or admonishments poses challenges in real-world applications. Our primary objective was to achieve an effective and socially acceptable queue-management solution. To accomplish this, we took inspiration from human security guards whose role has already been well received in society. Our robot, whose design embodied the image of a professional security guard, focused on three key aspects: duties, professional behavior, and appearance. To ensure its competence, we interviewed professional security guards to deepen our understanding of the responsibilities associated with queue management. Based on their insights, we incorporated features of ushering, admonishing, announcing, and question answering into the robot’s functionality. We also prioritized the modeling of professional ushering behavior. During a 10-day field trial at a children’s amusement event, we interviewed both the visitors who interacted with the robot and the event staff. The results revealed that visitors generally complied with its ushering and admonishments, indicating a positive reception. Both visitors and event staff expressed an overall favorable impression of the robot and its queue-management services. These findings suggest that our proposed security guard robot shows great promise as a solution for effective crowd handling in public spaces.

We developed an admonishing service for a shopworker robot and conducted a field trial to investigate the impressions of a shop’s staff and customers. Applying the admonishing service in a real-world robot is difficult due to the high risk of rejection by society. We wanted to achieve an acceptable admonishing service while simultaneously avoiding the impression of a forceful request from a machine. We proposed a harmonized design that provided friendly and admonishing services. First, we interviewed a shop’s staff to learn their strategies for both friendly and admonishing services. From our evaluation of the interview results, we derived three design principles: friendly impressions, zero erroneous admonishments, and polite requests. Based on the design principles, we implemented our harmonized design on a social robot that guides customers to product locations and admonishes those who are not wearing face masks. We conducted a 13-day field trial in a retail shop and interviewed the customers and shopworkers to learn their impressions of our robot. The results of the field trial imply that our harmonized design approach is successful. Both the customers and the shop staff had overall positive impressions of the robot, its admonishing and friendly services, and expressed an intention to use it in the future. Furthermore, we studied the robot’s autonomous service-providing capability in the field and conducted an evaluation with hired participants to deepen our study of the robot’s mask recognition capability.