Robots are increasingly present in human spaces, such as for conducting deliveries in hospitals, interacting with visitors at museums, and stocking items in warehouses. To ensure the seamless integration of robots into these spaces, a new role in human-robot interaction is emerging—the robot wrangler, namely an individual who is responsible for setting up, overseeing, and troubleshooting the robot. To understand the needs of this stakeholder, we conducted a scoping review that uncovered a typology of robot wrangling across the research literature, and discovered that wrangling is an umbrella term that collapses a highly complex and heterogeneous space of activities, often rendering this labor difficult to characterize and support. To further clarify and understand robot wrangling, we then reflected on our own firsthand and imagined experiences as robot wranglers within our own respective domains. Guided by the scoping review and our reflections, we devise a series of design implications for supporting wranglers directly as individuals and as members of a wider service ecology.

We integrated state-of-the-art vision–language models (VLMs) into a hat shop robot, enabling four visually grounded human shop-worker behaviors: 1) offering personalized greetings based on customers' visual aspects, 2) recognizing their shopping activities and delivering timely encouragement to try hats, 3) offering feedback on hat fit, and 4) responding to customer inquiries that contain deictic references to hats. We strategically utilized both local and cloud-based VLMs to develop the robot’s visual perception, leveraging their complementary capabilities. We report key insights from our development process, including our experience collaborating with a domain expert on robot interaction development. Results from an 8-day field trial at a hat shop show that the robot’s advanced visual perception capabilities made a positive impression on both customers and the shop manager. Customers were excited by the robot’s visually grounded remarks, and the manager adopted a complementary working style that leveraged these new capabilities.

We introduce a novel immersive social teleoperation interface to perform social interaction intuitively and semi-automatic locomotion simultaneously. Teleoperated robots in complex, human-centric social environments present a significant challenge on simultaneously managing intricate navigation while engaging in natural social interaction. This dual task imposes a high cognitive load, hindering the fluidity and quality of social interaction. As existing systems typically prioritize either task-oriented teleoperation control or non-moving social interaction, failing to integrate dynamic locomotion with social engagement effectively. Our system combines a head-mounted display with a wide-angle, 270-degree video feed to support extensive situation awareness and a strong sense of presence to overcome these limitations by fostering an immersive and socially aware experience. Our interface performs low-level navigation of the robot by pointing at a place to go and selecting a person to follow. The operator can focus on high-level goals (social interactions). We evaluated our interface through a rigorous field experiment, using a testbed (remote guide) scenario developed through iterative pilot studies in a real-world shopping mall. Our findings demonstrate that the operator's task performance improves statistically significantly. We report other findings and discuss limitations and future improvements. In short, our novel interface, integrating immersive visualization with autonomous navigation, enables operators to achieve a more intuitive and engaging social interaction in dynamic remote social environments.

We developed an autonomous hatshop robot for encouraging customers to try on hats by providing comments that appropriately fit their actions, and in such a way also indirectly exerting social pressure. To enable it to offer such a service smoothly in a real shop, we developed a large system (around 150k lines of code with 23 ROS packages) integrated with various technologies, like people tracking, shopping activity recognition and navigation. The robot needed to move in narrow corridors, detect customers, and recognise their shopping activities. We employed an iterative development process, repeating trial-and-error integration with the robot in the actual shop, while also collecting real-world data during field-testing. This process enabled us to improve our shopping activity recognition system by collecting real-world data, and to adapt our software modules to the target shop environment. We report the lessons learnt during our system development process. The results of our 11-day field trial show that our robot was able to provide its services reasonably well. Many customers expressed a positive impression of the robot and its services.

We developed a security guard robot that is specifically designed to manage queues of people and conducted a field trial at an actual public event to assess its effectiveness. However, the acceptance of robot instructions or admonishments poses challenges in real-world applications. Our primary objective was to achieve an effective and socially acceptable queue-management solution. To accomplish this, we took inspiration from human security guards whose role has already been well received in society. Our robot, whose design embodied the image of a professional security guard, focused on three key aspects: duties, professional behavior, and appearance. To ensure its competence, we interviewed professional security guards to deepen our understanding of the responsibilities associated with queue management. Based on their insights, we incorporated features of ushering, admonishing, announcing, and question answering into the robot’s functionality. We also prioritized the modeling of professional ushering behavior. During a 10-day field trial at a children’s amusement event, we interviewed both the visitors who interacted with the robot and the event staff. The results revealed that visitors generally complied with its ushering and admonishments, indicating a positive reception. Both visitors and event staff expressed an overall favorable impression of the robot and its queue-management services. These findings suggest that our proposed security guard robot shows great promise as a solution for effective crowd handling in public spaces.

We developed an admonishing service for a shopworker robot and conducted a field trial to investigate the impressions of a shop’s staff and customers. Applying the admonishing service in a real-world robot is difficult due to the high risk of rejection by society. We wanted to achieve an acceptable admonishing service while simultaneously avoiding the impression of a forceful request from a machine. We proposed a harmonized design that provided friendly and admonishing services. First, we interviewed a shop’s staff to learn their strategies for both friendly and admonishing services. From our evaluation of the interview results, we derived three design principles: friendly impressions, zero erroneous admonishments, and polite requests. Based on the design principles, we implemented our harmonized design on a social robot that guides customers to product locations and admonishes those who are not wearing face masks. We conducted a 13-day field trial in a retail shop and interviewed the customers and shopworkers to learn their impressions of our robot. The results of the field trial imply that our harmonized design approach is successful. Both the customers and the shop staff had overall positive impressions of the robot, its admonishing and friendly services, and expressed an intention to use it in the future. Furthermore, we studied the robot’s autonomous service-providing capability in the field and conducted an evaluation with hired participants to deepen our study of the robot’s mask recognition capability.

We studied the programming and debugging processes of an autonomous mobile social robot with a focus on the programmers. This process is time-consuming in a populated environment where a mobile social robot is designed to interact with real pedestrians. From our observations, we identified two types of time-wasting behaviors among programmers: cherry-picking and a shortage of coverage in their testing. We developed a new tool, a test generator framework, to help avoid these testing time-wasters. This framework generates new testing scenarios to be used in a simulator by blending a user-prepared test with pre-stored pedestrian patterns. Finally, we conducted a user study to verify the effects of our test generator. The results showed that our test generator significantly reduced the programming and debugging time needed for autonomous mobile social robots.

Interacting physically with robots and sharing environment with them leads to situations where humans and robots have to cross each other in narrow corridors. In these cases, the robot has to make space for the human to pass. From observation of human-human crossing behaviours, we isolated two main factors in this avoiding behaviour: body rotation and sliding motion. We implemented a robot controller able to vary these factors and explored how this variation impacted on people's perception. Results from a within-participants study involving 23 participants show that people prefer a robot rotating its body when crossing them. Additionally, a sliding motion is rated as being warmer. These results show the importance of social avoidance when interacting with humans.

We modeled a robot's approaching behavior for giving admonishment. We started by analyzing human behaviors. We conducted a data collection in which a guard approached others in two ways: 1) for admonishment, and 2) for a friendly purpose. We analyzed the difference between the admonishing approach and the friendly approach. The approaching trajectories in the two approaching types are similar; nevertheless, there are two subtle differences. First, the admonishing approach is slightly faster (1.3 m/sec) than the friendly approach (1.1 m/sec). Second, at the end of the approach, there is a `shortcut' in the trajectory. We implemented this model of the admonishing approach into a robot. Finally, we conducted a field experiment to verify the effectiveness of the model. A robot is used to admonish people who were using a smartphone while walking. The result shows that significantly more people yield to admonishment from a robot using the proposed method than from a robot using the friendly approach method.

We are facilitating software development for mobile social robots that operate `in the wild': that is, in real daily environments such as shopping malls. One fundamental difficulty in development is testing unfinished programs on a real robot and in the presence of people, because this process can take a significantly time-consuming. To ease the software development, we developed a simulator that allows the simulation of interactions among people, and interactions between people and the robot. With a user study, we analyzed how people's working process changed and evaluated the amount of time they saved in software development with the simulator.

People engage in social coordination without explicitly communicating when they are conflicting over spatial resources, e.g., a shop clerk who yields to customers the best place to view products. In this study, we proposed a method that achieves such social coordination with a robot. Our idea is that the social coordination between two agents can be represented as utility-maximizing behavior for joint utility rather than just by a single agent utility. That is, given that each agent's reasonable behavior can be represented as utility-maximizing behavior for single agent utility, we model each agent's plans for himself as well as for the partner agent. Moreover, superiority relationships exist in this joint-utility computation. Since each agent knows such superiority relationships, social coordination can be modeled as utility-yielding behavior based on informed superiority. We specifically focus on looking-together situations for which we developed a utility model. With simulations, we investigate whether the above joint-utility-based modeling successfully reproduces social coordination in looking-together situations. We conducted an experiment in a situation where a tele-operated robot and a customer together look at products in a shop environment. Our experimental results show that our proposed method enables the robot to socially coordinate spatial resources, yielding significantly more thoughtful, less-self-centered, and appropriate impressions than the alternate robot.

We aim to develop a walking partner robot with the capability to select small-talk topics that are associative to visual scenes. We first collected video sequences from five different locations and prepared a dataset about small-talk topics associated to visual scenes. Then we developed a technique to associate the visual scenes with the small-talk topics. We converted visual scenes into lists of words using an off-the-shelf vision library and formed a topic space with a Latent Dirichlet Allocation (LDA) method in which a list of words is transformed to a topic vector. Finally, the system selects the most similar utterance in the topic vectors. We tested our developed technique with a dataset, which successfully selected 72% appropriate utterances, and conducted a user study outdoors where participants took a walk with a small robot on their shoulder and engaged in small talk. We confirmed that the participants more highly perceived the robot with our developed technique because it selected appropriate utterances than a robot that randomly selected utterances. Further, they also felt that the former type of robot is a better walking partner.

We conducted a field study to investigate the social acceptance of social robots by stores, particularly for attracting passersby, which today's robots can autonomously perform. From interviews with ten store managers, we identified two main reasons they want to employ such social robots in their stores: robots offer cheap labor and provide unique value that humans cannot. They believe that robots are good at attracting the attention of visitors without causing or receiving stress. We also conducted three case studies in which we observed how store managers employed social robots in their stores. Each store manager requested different designs in the preparation phase. After deployment, we found that the managers were generally satisfied with the services autonomously offered by the robots, which successfully encouraged people to stop. For two out of three stores the robots successfully encouraged visitors to visit. The store managers were satisfied with the results and expressed a desire to use the robots again.

We developed an information-providing robot for a shopping mall. It moves around and hands over flyers about the mall and when people are interested or approach the robot to obtain some information, it starts the information-providing service. During the information-providing service, it provides directions and recommendations of shops in the mall. The robot is developed to be fully autonomous, but due to low performance in automatic speech recognition, this function is overtaken by a human operator. In our field trial, the robot worked for 53.4 hours (over 11 days) without any major failure (83.5% of time autonomously operated). We conducted interviews with visitors, and revealed that the majority of them found the information from the robot useful, and wanted to use it again.

Walking side by side is a common situation when we go from one place to another with another person while talking. Our previous study reported a basic mechanism for side-by-side walking, but in the previous model it was crucial that each agent knew where he or she was going, i.e. the route to the destination. However, we have recognized the need to model the situation where one of the agents does not know the destination. The extended model presented in this work has two states: leader-follower state and collaborative state. Depending on whether the follower agent has obtained a reliable estimate of the route to follow, the walking agents transition between the two states. The model is calibrated with trajectories acquired from pairs of people walking side by side, and then it is tested in a human-robot interaction scenario. The results demonstrate that the new extended model achieves better side-by-side performance than a standard method without knowledge of the subgoal.

This paper models the concept of the "territory" of shops. First, we interviewed three shopkeepers and found that they perceived the space near their shop as their territory and that they interpreted some types of behaviors as invasive. Second, we confirmed that potential visitors share this notion of territory. We also confirmed that the size of the territory depends on the characteristics of a shop's facade. While there is little territory in front of walls, there is more territory in front of shelves and entrances. Our robot traversed around two real shopping malls that included 50 shops and took 3-D scans of their environment shapes. Each shop's facade was analyzed and the shop territory was computed. The computation results match people's perception. The recognition rate accuracy reached 93.5% for the territory areas. User evaluations in a virtual shop environment confirmed that a robot with a territory model behaves better than one without it.

In this paper, we discuss how networked robot architectures can facilitate the development, deployment, management and adaptation of distributed robotic applications. Our aim is to modularize applications by factoring out environment, task-, domain-, and robot-specific knowledge components and representing them explicitly in a formal knowledge base that is shared between the robots and service applications. Robot control decisions can then be formulated in terms of inference tasks that are evaluated based on this knowledge during task execution. The explicit and modular knowledge representation allows human operators with different areas of expertise to adapt the respective parts of the knowledge independently. We implemented this concept by integrating knowledge representation methods of the RoboEarth project with the distributed task execution capabilities of the Ubiquitous Network Robot Platform.

When robots serve in urban areas such as shopping malls, they will often be required to approach people in order to initiate service. This paper presents a technique for human-robot interaction that enables a robot to approach people who are passing through an environment. For successful approach, our proposed planner first searches for a target person at public distance zones anticipating his/her future position and behavior. It chooses a person who does not seem busy and can be reached from a frontal direction. Once the robot successfully approaches the person within the social distance zone, it identifies the person's reaction and provides a timely response by coordinating its body orientation. The system was tested in a shopping mall and compared with a simple approaching method. The result demonstrates a significant improvement in approaching performance; the simple method was only 35.1% successful, whereas the proposed technique showed a success rate of 55.9%.

We aim to develop a shopping companion robot that can share experience with users. In this study, we focused on the shared memory acquired when a robot walks together with a user. We developed a computational model of memory recall of visited locations in a shopping mall. The model was developed with data collection from 30 participants. We found that shop size, color intensity of facade, relative visibility, and time elapsed are the influencing features for recall. The model was used in a scenario of a shopping companion robot. The robot, Robovie, autonomously follows a user while inferring the user's memory recall of shops in the visited route. When the user asks the location of other shops, Robovie replied with destination description, referring to the known locations inferred with the model of the user's memory recall. With this scenario, we verified the effectiveness of the developed computational model of memory recall. The evaluation experiment revealed that the model outputs shops that the participants are likely to recall, and makes the directions given easier to understand.

This paper presents a computational model for side-by-side walking for human-robot interaction (HRI). In this work we address the importance of future motion utility (motion anticipation) of the two walking partners.

Environment attributes are perceived or remembered differently according to the perspective used. In this study, two different perspectives, a survey perspective and a route perspective, are explained and discussed. This paper proposes an approach for modeling human environments from a route perspective, which is the perspective used when a human navigates through the environment. The process for route perspective semi-autonomous data extraction and modeling by a mobile robot equipped with a laser sensor and a camera is detailed. Finally, as an example of a route perspective application, a route direction robot was developed and tested in a real mall environment. Experimental results show the advantages of the proposed route perspective model compared with a survey perspective approach. Moreover, the route model is comparable to the performance of an expert person giving route guidance in the mall.

In daily conversation, we sometimes observe a deictic interaction scene that refers to a region in a space, such as saying "please put it over there" with pointing. How can such an interaction be possible with a robot? Is it enough to simulate people's behaviors, such as utterance and pointing? Instead, we highlight the importance of simulating human cognition. In the first part of our study, we empirically demonstrate the importance of simulating human cognition of regions when a robot engages in a deictic interaction by referring to a region in a space. The experiments indicate that a robot with simulated cognition of regions improves efficiency of its deictic interaction. In the second part, we present a method for a robot to computationally simulate cognition of regions.

This paper proposes to share gesture contents among heterogeneous robots. In this paper, we classify gestures in two types; pointing gesture and track gesture. Criteria of classification is factors which are essential for each gesture. Pointing gestures are used for pointing somewhere around a robot. Trajectory of a gesture is important for track gesture. Track gestures can keep its essential factors by moving track horizontally or vertically. We made gesture translation algorithms for each classification and achieved sharing semantic information of gesture contents. We carried out quantitative evaluation with simulations, and qualitative evaluation with questionnaire.

This paper proposes a model of approach behavior with which a robot can initiate conversation with people who are walking. We developed the model by learning from the failures in a simplistic approach behavior used in a real shopping mall. Sometimes people were unaware of the robot's presence, even when it spoke to them. Sometimes, people were not sure whether the robot was really trying to start a conversation, and they did not start talking with it even though they displayed interest. To prevent such failures, our model includes the following functions: predicting the walking behavior of people, choosing a target person, planning its approaching path, and nonverbally indicating its intention to initiate a conversation. The approach model was implemented and used in a real shopping mall. The field trial demonstrated that our model significantly improves the robot's performance in initiating conversations.

This paper reports the challenges of developing multiple social robots that operate in a shopping mall. We developed a networked robot system that coordinates multiple social robots and sensors to provide efficient service to customers. It directs the tasks of robots based on their positions and people's walking behavior, manages the paths of robots, and coordinates the conversation-performance between two robots. Laser range finders were distributed in the environment to estimate people's positions. The system estimates such human walking behaviors as ¿stopping¿ or ¿idle walking¿ to direct robots to provide appropriate tasks to appropriate people. Each robot interacts with people to provide recommendation information and route information about shops. The system sometimes simultaneously uses two robots to lead people from one place to another. The field trial, which was conducted in a shopping mall where four robots interacted with 414 people, revealed the effectiveness of the network robot system for guiding people around a shopping mall as well as increasing their interest.

The paper proposes a new middleware named semantic sensor network which connects the variety of physically grounded applications to the information of the real world. Semantic sensor network infers and describes the state of an environment using logical expressions. The target which semantic sensor network describes is an environment where wireless sensor nodes are attached to daily items. The remarkable achievement of semantic sensor network is to employ explicitly the concept of a class and an instance on the sensor network so as to construct efficiently an interpretation model for sensor data and queries. We have developed two physically grounded applications on semantic sensor network; one is a GUI and the other is a robotic system. They use semantic sensor network to obtain information about the environment