Enabling blind visitors to explore museum floors while feeling the facility’s atmosphere and increasing their autonomy and enjoyment are imperative for giving them a high-quality museum experience. We designed a science museum exploration system for blind visitors using an autonomous navigation robot. Blind users can control the robot to navigate them toward desired exhibits while playing short audio descriptions along the route. They can also browse detailed explanations on their smartphones and call museum staff if interactive support is needed. Our real-world user study at a science museum during its opening hour revealed that blind participants could explore the museum safely and independently at their own pace. The study also showed that the sighted visitors who saw the participants walking with the robot accepted the assistive robot well. We finally conducted focus group sessions with the blind participants and discussed further requirements toward a more independent museum experience.

Indoor navigation systems with prebuilt maps have shown great potential in navigating blind people even in unfamiliar buildings. However, blind people cannot always benefit from them in every building, as prebuilt maps are expensive to build. This paper explores a map-less navigation system for blind people to reach destinations in unfamiliar buildings, which is implemented on a robot. We first conducted a participatory design with five blind people, which revealed that intersections and signs are the most relevant information in unfamiliar buildings. Then, we prototyped PathFinder, a navigation system that allows blind people to determine their way by detecting and conveying information about intersections and signs. Through a participatory study, we improved the interface of PathFinder, such as the feedback for conveying the detection results. Finally, a study with seven blind participants validated that PathFinder could assist users in navigating unfamiliar buildings with increased confidence compared to their regular aid.

Autonomous navigation robots have a considerable potential to offer a new form of mobility aid to people with visual impairments. However, to deploy such robots in public buildings, it is imperative to receive acceptance from not only robot users but also people that use the buildings and managers of those facilities. Therefore, we conducted three studies to investigate the acceptance and concerns of our prototype robot, which looks like a regular suitcase. First, an online survey revealed that people could accept the robot navigating blind users. Second, in the interviews with facility managers, they were cautious about the robot’s camera and the privacy of their customers. Finally, focus group sessions with legally blind participants who experienced the robot navigation revealed that the robot may cause trouble when it collides with those who may not be aware of the user’s blindness. Still, many participants liked the design of the robot which assimilated into the surroundings.

We present ReCog, a mobile app that enables blind users to recognize objects by training a deep network with their own photos of such objects. This functionality is useful to differentiate personal objects, which cannot be recognized with pre-trained recognizers and may lack distinguishing tactile features. To ensure that the objects are well-framed in the captured photos, ReCog integrates a camera-aiming guidance that tracks target objects and instructs the user through verbal and sonification feedback to appropriately frame them.

The aim of our study is to develop a method by which a social robot can greet passersby and get their attention without causing them to suffer discomfort. Social robots now function in a number of customer service roles, such as receptionists, guides, and exhibitors. However, sudden greetings from a robot can startle passersby. Therefore, we developed a method that allows social robots to adapt their mannerisms situationally based the results of related work. Our proposed method, user-centered reinforcement learning, enables robots to greet passersby without causing them discomfort (p<; 0.01). Our field experiment in an office entrance demonstrated that our method meets this requirement.

Some electric-powered wheelchairs are recently redefined as personal mobility devices. Their users are not only elderly or handicapped people, but also passengers with large baggage or pedestrians going from station to destination, i.e., last-mile transport. Consequently, people with different operation skills and expectations on personal mobility would become new users of this kind of devices. Safe and comfort travel in human co-existing environment such as sidewalks and airports is a social expectation for personal mobility. In order to realize this, understanding the operation skill of each user by a practical and simple method is essential. This paper thus introduced a skill level classification method by machine learning using only joystick data as input. In order to determine the number of skill level clusters, basic 26 features of joystick operation data are used for unsupervised clustering (single-linkage). We then made evaluation indexes by using speed, speed control, and direction control. For a five-level classification by using gradient boosting as supervised learning, we achieved a 67% accuracy (tolerance: 0) and a 98% accuracy (tolerance: 1). Further analysis of the feature importance of gradient boosting revealed key features to a good operation. Results also show that skill level differed among people with different driving experiences.

Social robots have been providing a number of customer services lately. For example, they can take the place of people as receptionists. When people interact they predict each others decision-making from the others actions and take suitable actions for the sake of the other. However, taking such actions is difficult for modern social robots. Therefore, our initial aim is to solve the problem of how to predict decision-making in human-robot engagement. Choosing a reception system as a case study to approach this goal, we created a new model to predict who will use the system. The model contains a state transition function based on observation studies. We evaluated the model though a controlled experiment and a simulation experiment, using field data on prediction performance and mental effect. The experiment results lead us to believe that the model can predict the decisions of others sufficiently well. We also found that a pedestrian who will not talk with a robot receptionist suffers a negative emotion when the pedestrian is greeted by a robot, but a method we propose prevents this. We suggest that the word suitable is related to negative emotion, a part of usability. By using our model in the future we will attempt to find a method that will enable a robot to learn suitable actions by itself.

For many automated navigation applications, the underlying localization algorithm must be able to continuously produce both accurate and stable results by using a spectrum of redundant sensing technologies. To this end, various sensors have been used for localization, such as Wi-Fi, Bluetooth, GPS, LiDAR and cameras. In particular, a class of vision-based localization techniques using Structure from Motion (SfM) has been shown to produce very accurate position estimates in the real-world with moderate assumptions about the motion of the camera and the amount of visual texture in the environment. However, when these assumptions are violated, SfM techniques can fail catastrophically (i.e., cannot generate any estimate). Recently, a deep convolutional neural network (CNN) has been applied to images to robustly regress 6-DOF camera poses at the cost of lower accuracy than SfM. In this work, we propose improving image-based localization accuracy of deep CNN by combining Bluetooth radio-wave signal readings. In our experiments, we show that our proposed dual-stream CNN can robustly regress 6-DOF poses from images and radiowave signals better than one sensing modality alone. More importantly, we show that when both modes are used, the localization accuracy of the proposed deep CNN is comparable to that of SfM and significantly more robust than SfM.

State space models, such as Kalman filters or Particle filters, have been applied to improve the accuracy of radio-wave-based localization. However, these models can drift radically when assumptions of the models are violated, and they do not have a mechanism to fix errors. Therefore, we propose an approach to apply supervised learning to pedestrian localization, which is based on the Inference Machines framework. During training, we collect localization ground truths using computer vision while also collecting Bluetooth signals to train a state space model for localization, which can recover from model drift. During testing, our proposed approach uses only Bluetooth signals. Our experimental results show that our approach can improve the accuracy of Bluetooth-based localization with a small number of training examples. Moreover, our multi-modal supervision can also be used to estimate additional parameters, such as device rotation, from Bluetooth signals that do not have such information.

Great progress has been made in computer vision-based localization systems. However, some systems tend to work well only in certain visually feature-rich environments. It is often the case that feature-based matching techniques can have a hard time dealing with scenes with only a few features or a large number of repetitive features. In these situations, computer vision-based localization may fail to estimate camera position or may yield a large localization error. We approach this problem from a systems perspective, where we are required to obtain accurate localization of blind travellers using a smartphones app for localization. In particular, we assume that the environment is already instrumented with Bluetooth low energy (BLE) signals to provide rough proximity information, and we propose to integrate it with visual information to perform efficient structure-from-motion and camera localization. Our multi-model sensing approach can accelerate localization speed and obtain more accuracy in challenging environments when compared to traditional baseline approaches. We also show that our approach can accelerate the time for reconstructing large 3D models. Our framework is released as an open source project. It can be used by different mobile operating systems, enabling the development of navigation applications on mobile platforms.

Wearable computing technologies are advancing rapidly and enabling users to easily record daily activities for applications such as life-logging or health monitoring. Recognizing hand and object interactions in these videos will help broaden application domains, but recognizing such interactions automatically remains a difficult task. Activity recognition from the first-person point-of-view is difficult because the video includes constant motion, cluttered backgrounds, and sudden changes of scenery. Recognizing hand-related activities is particularly challenging due to the many temporal and spatial variations induced by hand interactions. We present a novel approach to recognize hand-object interactions by extracting both local motion features representing the subtle movements of the hands and global hand shape features to capture grasp types. We validate our approach on multiple egocentric action datasets and show that state-of-the-art performance can be achieved by considering both local motion and global appearance information.

While there are tools that allow distributed teams to collaboratively edit in real time, little work examines this practice among real teams doing real work. Even less is known about how teams from different countries make use of real-time collaborative editing tools. The current work highlights results from a qualitative user study of real-world Japanese and U.S. distributed work teams who used LiveDeck, a real-time slide editing and whiteboarding tool. Through the implementation of various novel features used as probes, differences in behavior and attitudes between team members were uncovered. Differences in the use of slide navigation options, anonymity features, and pop-up 'emotes' representing nonverbal gestures are discussed.

Presentation documents play important roles in many fields, such as business and education. The principal purpose of presentation documents is to convey information visually, so recognizing the visual layout is essential for understanding those documents. However it is inherently difficult for the blind people to recognize a visual layout, because there are numerous types of charts in presentation documents. As the first step to solve such problems, this study focuses on diagrams in which objects or groups of objects are bound by arrows. Such diagrams usually show relationships among the objects. If such relationships could be recognized by screen readers, it would make them accessible. However, the presentation authoring applications do not have functions for embedding these relationships among objects. Therefore this paper proposes a visual analysis method for diagram structure in presentation documents to automatically create metadata. It generates metadata which describes the relationships of objects, and the source-destination relationships of arrows. Then a novel interface utilizing the metadata was prototyped to present the visual structure of presentation documents in a tree view. This allows blind users to understand presentation documents easily, because it represents the visual structure that current screen readers cannot expose. In addition, they are familiar with the tree view interface, so they can use it without training. Finally, an evaluation shows that our method for automatically creating the metadata can be applied to various types of diagrams in presentation documents.