Lifestyle support through robotics is an increasingly promising field, with expectations for robots to take over or assist with chores like floor cleaning, table setting and clearing, and fetching items. The growth of AI, particularly foundation models, such as large language models (LLMs) and visual language models (VLMs), is significantly shaping this sector. LLMs, by facilitating natural interactions and providing vast general knowledge, are proving invaluable for robotic tasks. This paper focuses on the benefits of LLMs for "bring-me" tasks, where robots fetch specific items for users, often based on ambiguous instructions. Our previous efforts utilized an ontology extended to handle environmental data to resolve such ambiguities, but faced limitations when unresolvable ambiguities required user intervention for clarity. Here, we enhance our approach by integrating LLMs for providing additional commonsense knowledge, pairing it with ontological data to mitigate the issue of hallucinations and reduce the need for user queries, thus improving system usability. We present a system that merges these knowledge bases and assess its efficacy on "bring-me" tasks, aiming to provide a more seamless and efficient robotic assistance experience.

This paper describes a method of domain adap-tive training for semantic segmentation using multiple source datasets that are not necessarily relevant to the target dataset. We propose a soft pseudo-label generation method by integrating predicted object probabilities from multiple source models. The prediction of each source model is weighted based on the estimated domain similarity between the source and the target datasets to emphasize contribution of a model trained on a source that is more similar to the target and generate reasonable pseudo-labels. We also propose a training method using the soft pseudo-labels considering their entropy to fully exploit information from the source datasets while suppressing the influence of possibly misclassified pixels. The experiments show comparative or better performance than our previous work and another existing multi-source domain adaptation method, and applicability to a variety of target environments.

Autonomous mobile robots are now being perceived as natural objects in real environments such as warehouses or restaurants. Naturally, These robots must consider the indeterminate behaviors of people, especially those unaware of their surroundings, such as those texting while walking (awareness). Although the risk of nonawareness has been studied extensively, much less work has been done on developing the pedestrian awareness model for an autonomous mobile robot. This study uses three factors: viewing angle, distance, and updating cycle to replicate the pedestrian awareness model mathematically. We collect data from five participants with high and low-workload texting tasks to identify their parameters. In this data collection, one pedestrian is walking freely in a $6-\mathrm{m}^{2}$ room with three walking experimenters. Meanwhile, they carry out a task that requires repeating a given sentence, and gazing information and head position are measured. In consequence, the distance probability of pedestrian awareness can be approximated by the sigmoid. Other parameters are estimated by the time ratio of awareness time to the overall time from the gazing information. Due to the difficulty of producing the generic parameters, we verified the proposed method using the calculated parameters of each participant. The simulation evaluation result indicates that our model leads to fewer errors than the simple social force model by comparing it with the collected walking trajectories. The result indicates that the proposed model could reproduce nonawareness pedestrians with higher accuracy, and our proposed model contributes to the mobile robot and the more realistic pedestrian simulation.

For autonomous driving of construction machinery such as construction crane, it is necessary to detect objects (e.g. person) in a construction site using the camera hung from the crane’s boom. Since the camera is attached to the boom, it is in motion as the boom moves from one place to another. In this research, we propose a camera motion compensation and person detection technique in a construction site for moving camera. The motion parameters are estimated using point-to-point features correspondences with RANSAC-like algorithm. The algorithm selects the best model with few randomly selected sample points. After compensating the motion effect, a Bayesian interface along with the Yolo model (called Yollo-Bayes model) is used for person detection. Two types of Bayes models are proposed to detect missing person by the Yolo. Bayes model for moving person is used to detect the person when he/she is in motion and Yolo fail to detect him/her. On the other hand, Bayes model for nonmoving person is used when the person is not in motion and Yolo miss to detect him/her. The motion estimation accuracy is confirmed by creating ground truth images with translation and rotation changes. To evaluate the performance of the Yolo-Bayes model, extensive experimental evaluations have been done with a lot of video images collected from the real construction site. It reveals that the proposed Yolo-Bayes model outperforms Yolo model. The proposed model is more effective when the number of training sample is small (Fβ– score is 5.77% higher than Yolo) or the model is tested on a completely new dataset (Fβ– score is 13.75% higher than Yolo).

This paper describes a method of online refinement of a scene recognition model for robot navigation considering traversable plants, flexible plant parts which a robot can push aside while moving. In scene recognition systems that consider traversable plants growing out to the paths, misclassification may lead the robot to getting stuck due to the traversable plants recognized as obstacles. Yet, misclassification is inevitable in any estimation methods. In this work, we propose a framework that allows for refining a semantic segmentation model on the fly during the robot’s operation. We introduce a few-shot segmentation based on weight imprinting for online model refinement without fine-tuning. Training data are collected via observation of a human’s interaction with the plant parts. We propose novel robust weight imprinting to mitigate the effect of noise included in the masks generated by the interaction. The proposed method was evaluated through experiments using real-world data and shown to outperform an ordinary weight imprinting and provide competitive results to fine-tuning with model distillation while requiring less computational cost.

We present a novel compact deep multi-task learning model to handle various autonomous driving perception tasks in one forward pass. The model performs multiple views of semantic segmentation, depth estimation, light detection and ranging (LiDAR) segmentation, and bird’s eye view projection simultaneously without being supported by other models. We also provide an adaptive loss weighting algorithm to tackle the imbalanced learning issue that occurred due to plenty of given tasks. Through data pre-processing and intermediate sensor fusion techniques, the model can process and combine multiple input modalities retrieved from RGB cameras, dynamic vision sensors (DVS), and LiDAR placed at several positions on the ego vehicle. Therefore, a better understanding of a dynamically changing environment can be achieved. Based on the ablation study, the model variant trained with our proposed method achieves a better performance. Furthermore, a comparative study is also conducted to clarify its performance and effectiveness against the combination of some recent models. As a result, our model maintains better performance even with much fewer parameters. Hence, the model can inference faster with less GPU memory utilization. Moreover, the result tends to be consistent in 3 different CARLA simulation datasets and 1 real-world nuScenes-lidarseg dataset. To support future research, we share codes and other files publicly athttps://github.com/oskarnatan/compact-perception.

With the aim of contributing to the development of a robotic attendant system, this study proposes the concept of assessing the risk of collision using awareness estimation. The proposed approach enables an attendant robot to assess a person's risk of colliding with an obstacle by estimating whether he/she is aware of it based on behavior, and to take the requisite preventative action. To implement the proposed concept, we design a model that can simultaneously estimate a person's awareness of obstacles and predict his/her trajectory based on a convolutional neural network. When trained on a dataset of collision-related behaviors generated from people trajectory datasets, the model can detect objects of which the person is not aware and with which he/she at risk of colliding. The proposed method was evaluated in an empirical environment, and the results verified its effectiveness.

Currently, research and development of lifestyle support robots in daily life is being actively conducted. Health-case is one such function robots. In this research, we develop a fatigue estimation system using a camera that can easily be mounted on robots. Measurements taken in a real environment have to be consider noises caused by changes in light and the subject's movement. This fatigue estimation system is based on a robust feature extraction method. As an indicator of fatigue, LF/HF-ratio was calculated from the power spectrum of RR interval in the electrocardiogram or the blood volume pulse (BVP). The BVP can be detected from the fingertip by using the photoplethysmography (PPG). In this study, we used a contactless PPG: remote-PPG (rPPG) detected by the luminance change of the face image. Some studies show facial expression features extracted from facial video are also useful for fatigue estimation. dimension reduction of past method using LLE spoiled the information in the large dimention of feature. We also developed a fatigue estimation method with such features using a camera for the healthcare robots. It used facial landmark points, line-of-sight vector, and size of the ellipse fitted with eyes and mouth landmark points. Therefore, proposed method simply use time-varying shape information of face like size of eyes, or gaze direction. We verified the performance of proposed features by the fatigue state classification using Support Vector Machine (SVM).

This paper describes a method of generating attending behaviors adaptively to the user state. The method classifies the user state based on user information such as the relative position and the orientation. For each classified state, the method executes the corresponding policy for behavior generation, which has been trained using a deep reinforcement learning, namely DDPG (deep deterministic policy gradient). We use as a state space of DDPG a distance-transformed local map with person information, and define reward functions suitable for respective user states. We conducted attending experiments both in a simulated and a real environment to show the effectiveness of the proposed method.

This paper describes a method of building a semantic map of a greenhouse for a robot path planning. Existing mapping methods only consider whether there are obstacles in a certain region. They are not sufficient for path planning in greenhouses where traversable regions are often covered by branches and leaves which are also recognized as obstacles. We propose a mapping method which generates a map with semantic information on the types of obstacles. By integrating RGB-D based visual SLAM (Simultaneous Localization And Mapping) and semantic segmentation by a deep neural network, we obtain a 3D map with semantic labels. In order to deal with the uncertainty of observations, we introduce a Bayesian label updating strategy which effectively utilizes the fact that the robot traverses a region. Through evaluations, we confirmed that the proposed method can perform a more accurate semantic labeling than the one only using SegNet.

This paper describes an estimation method of a person's awareness of an obstacle. We assume that the person's awareness influences the person's motion, and construct a model of the relationship between the awareness and the motion using HCRF. We extract a sequence of motion features from the person trajectory, and then classify whether the person is aware of the obstacle or not using the model. Awareness estimation experiments are conducted in order to validate the method and evaluate its performance. Since the method uses only the position and the velocity of the person, it can be applicable to mobile robots.

The transfer of task knowledge is ubiquitous in our daily lives, where various types of interaction occur. Such an interactive task knowledge transfer, however, requires that an instructor and a learner to be at the same place and time. If we use a robot to mediate between them, such limitations can be eliminated. This paper focuses on human-to-robot teaching, in which a robot instructor interactively teaches a human learner how to achieve a task. We develop an ambiguity-driven formulation of interactive teaching based on the Dempster-Shafer theory. We implemented an experimental system for blocks world tasks as a proof-of-concept and show our preliminary results.

This paper describes a person identification method using a smartphone and laser range finders (LRFs) for a mobile service robot. The robot is equipped with LRFs and the target person holds a smartphone. The method first detects the foot strike timings of the target person using the smartphone and those of all people by using the LRFs. By finding the person whose foot strike timings captured by the LRFs are similar to those obtained by the smartphone, the robot can identify the target person. Person identification experiments and person following experiments are conducted in order to validate the method. Since the method only requires a person to simply hold a smartphone, it can be easily applied to daily situations.

This research aims to develop a robot which can adaptively attend a specific person according to the person's behavior. Transition of the person's state is modeled with finite state machine (FSM), and the robot recognizes events for the state transitions and selects an appropriate attending action for each state. We implemented attending actions for the person's walking and the sitting actions. When a person is walking, the robot takes a following action. When the person is sitting, the robot moves to a waiting position determined by considering the comfort of the person and the others. We carried out attending experiments using a real robot to show the effectiveness of the proposed approach.

We have developed a new type of robot which accompanies the healthcare professionals making medical rounds in patient's bedrooms at hospitals. This novel robot mainly executes two tasks: the carrying armamentarium of round medical supplies and the recording electronic health data on rounds. An omni-directional mobile mechanism and a human tracking control system to follow a specified medical professionals realize smooth transfer movement from the nurses' station to patient's bedroom. Electronic health data on rounds is automatically recorded by using a CCD camera and a voice recorder. When the robot is connected via a cable to a medical data server in nurses' station, the robot transmits the patients' data to the server and receives new patients' information.

Recent increase of advanced appliances at home requires lots of user's operations for setting them in a desired state. Although many appliances have been developed which can adapt to the state of a room and a person, they are mainly based on simple state values such as the room temperature. More intelligent assistance will be needed for controlling electronic appliances such as a TV or an audio system. This paper describes a method of modeling human activities and predicting them for controlling appliances. We have developed an experimental system in a room with furniture and appliances. The system observes human motions and appliance operations and compiles them into actions, and then constructs a model of possible action sequences, which is further used for human action prediction. We test two prediction methods: a probabilistic action graph-based and an SVM (support vector machine)-based. We evaluate the methods using actual observation data for over fifty days in total. We also implement an on-line appliance control system as a proof-of-concept.

This paper describes an approach to realize a cameraman robot. Here a robot is given a task to follow a certain trajectory for taking the video of an actor in a designated time. The trajectory is relative to the actor, so that the robot has to take into account the actor's movement. We map the given trajectory to a new one in the state-time space based on the prediction of the actor's pose. We then build a trajectory tracking system using a 3D time-space wavefront potential considering the robot kinematic, trajectory cost, obstacles, and visibility constraints. This potential is then used for generating the robot motion control by using a modified random tree search algorithm with the control law. Simulation results show the effectiveness and feasibility of our approach.

This paper describes a robot-mediated information guide system that introduces audio-visual information to users interactively. The developed information guide system is composed of a touch panel interface, a mediator robot and backyard control systems. The touch panel interface functions to display audio-visual contents, while the mediator robot attracts people to this guide system and help user's operation. Many robots in literature are equipped with interactive audio-visual interface, however in most cases the audio-visual interface is a front end of the robot for presentation. We propose a new communication framework in which the robot behaves as a mediator to make a bridge between users and audio-visual interface by generating speeches and gestures. We hypothesize that entrainment of the mediator robot with users and contents could promote user's immersion in the contents. We conducted experiments with the proposed information guide system to examine the effect of robot mediation in a realistic interaction scenario that actually guides city sites for visitors. The developed information guide system was demonstrated in the science museum of the city in May 2014 and will be used for public events in future.

This paper describes a viewpoint planning algorithm for a guard robot in an indoor environment. The viewpoint planner is used for the guard robot to watch a certain object such as human continuously. Rather than continuously follows the object, moving the guard robot using the viewpoint planner has many benefits such as reducing the movement and the energy used by the robot. Our viewpoint planner exploits the topology feature of the environment, which is extracted using a skeletonization technique to get a set of viewpoints. We search for escaping gaps from which the target may go out of the robot's sight, and make the movement model of the target and the robot to determine the predicted time of the worst case escape of the target. We then plan the action for the robot based on the geodesic model and escaping gaps. Simulation results using 3D simulator are provided to show the effectiveness and feasibility of our algorithm.

This paper deals with a path planning problem in the dynamic and cluttered environments. The presence of moving obstacles and kinodynamic constraints of the robot increases the complexity of path planning problem. We model the environment and motion of dynamic obstacles in 3D time-space. We propose the utilization of the arrival time field for examining the most promising area in those obstacles-occupied 3D time-space for approaching the goal. The arrival time field is used for guiding the expansion of a randomized tree search in a favorable way, considering kinodynamic constraints of the robot. The quality and the optimality of the path are taken into account by performing heuristic methods on the randomized tree. Simulation results are also provided to prove the feasibility, possibility, and effectiveness of our algorithm.

This paper describes a road boundary estimation method for autonomous navigation. We consider navigation in a campus environment, where roads (or traversable regions) are not necessarily modeled by a typical road model with a pair of parallel lines, but have a variety of shapes. We therefore use a set of flexible road models with model transition mechanisms for a robust road boundary estimation. This new modeling is incorporated into our multiple sensory feature-based road boundary estimation framework using a particle filter. The proposed method has been successfully applied to various scenes in our campus to realize autonomous navigation.

This paper deals with a navigation of a mobile robot in a campus environment using a hand-drawn line drawing building map. Hand-drawn maps often include various types of uncertainty such as incorrect size/position and missing objects, thereby making it difficult to establish correspondence between objects in the map and sensory data. We solve this problem using a SLAM approach with an input hand-drawn map being an initial estimate. The proposed method combines a FastSLAM with a particle swarm optimization for map refinement. The method has been successfully applied to a stereo-based localization in a real scene.

This paper focuses on the construction of a novel belt tactile interface and telepresence system intended for mobile robot control. The robotic system consists of a mobile robot and a wearable master robot. The elaborated algorithms allow the robot to precisely recognize the shape, boundaries, movement direction, speed, and distance to the obstacle by means of the laser range finders. The designed tactile belt interface receives the detected information and maps it through the vibrotactile patterns. We designed the patterns in such a way that they convey the obstacle parameters in a very intuitive, robust, and unobtrusive manner. The robot movement direction and speed are governed by the tilt of the user's torso. The sensors embedded into the belt interface measure the user orientation and gestures precisely. Such an interface lets to deeply engage the user into the teleoperation process and to deliver them the tactile perception of the remote environment at the same time. The key point is that the user gets the opportunity to use own arms, hands, fingers for operation of the robotic manipulators and another devices installed on the mobile robot platform. The experimental results of user study revealed the effectiveness of the designed vibration patterns for obstacle parameter presentation. The accuracy in 100% for detection of the moving object by participants was achieved. We believe that the developed robotic system has significant potential in facilitating the navigation of mobile robot while providing a high degree of immersion into remote space.

Pedestrian detection is one of the key technologies for autonomous driving systems and driving assistance systems. To predict the possibility of a future collision, these systems have to accurately recognize pedestrians as far away as possible. Moreover, the function to detect not only people walking but also people who are standing near the road is also required. This paper proposes a method for recognizing pedestrians by using a high-definition LIDAR. Two novel features are introduced to improve the classification performance. One is the slice feature, which represents the profile of a human body by widths at the different height levels. The other is the distribution of the reflection intensities of points measured on the target. This feature can contribute to the pedestrian identification because each substance has its own unique reflection characteristics in the near-infrared region of the laser beam. Our approach applies a support vector machine (SVM) to train a classifier from these features. The classifier discriminates the clusters of the laser range data that are the pedestrian candidates, generated by pre-processing. A quantitative evaluation in a road environment confirms the effectiveness of the proposed method.

This paper describes a stereo-based person tracking method for a person following robot. Many previous works on person tracking use laser range finders which can provide very accurate range measurements. Stereo-based systems have also been popular, but most of them are not used for controlling a real robot. We previously developed a tracking method which uses depth templates of person shape applied to a dense depth image. The method, however, sometimes failed when complex occlusions occurred. In this paper, we propose an accurate, stable tracking method using overlapping silhouette templates which consider how persons overlap in the image. Experimental results show the effectiveness of the proposed method.

Mapping is an activity of making a useful description of an environment. Not only geometric information such as free space shape but also semantic information such as object names are sometimes important. We call such a map making environment information summarization because how to summarize may change depending on the purpose of the map and the context. One important aspect of such summarization is the deadline, which imposes a time constraint on the robot's mapping activity. We therefore develop an observation planning method for the summarization with deadlines. The method can cope with various types of deadlines specified in the form of loss function. Experimental results shows various robot behaviors are generated by only changing the deadline specification.

Mapping is an activity of making a useful description of an environment. Not only geometric information such as free space but also object placements are important if the map is used for human-robot communication. We call such a map making environment information summarization because how to summarize may change depending on the purpose of the map. Environment information summarization usually includes searching for specified objects in the environment. It is, therefore, crucial to make a good observation plan for efficient summarization. We develop an observation planning method which uses object appearance models for appropriately handling a trade-off between visual data quality and vision cost. Experimental results using a vision-based humanoid robot show the effectiveness of the proposed planning method.

This paper describes a vision-based interface for instructing robot motion easily. An interface that makes the robot move in the same way as user's motion is effective for an intuitive motion instruction. Such an interface can be realized by estimating the pose (position and orientation) of the interface and executing move commands for making the robot take the same pose.We estimate the pose of the interface by a monocular SLAM method, which is based on visual features and the extended Kalman filter. By additionally using an orientation sensor and an accelerometer, the reliability and the accuracy of pose estimation are improved. From the estimated pose, the target values of the robot position and the head orientation are set and the robot moves to achieve them. We implemented an experimental system which run in real-time (30 [Hz]) and successfully applied it to controlling a humanoid robot.

This paper describes a view-based localization method in outdoor environments. An important issue in view-based localization is to cope with the change of object views due to changes of weather and seasons. We have developed a two-stage SVM-based localization method which exhibits a high localization performance with few parameter tunings. In this paper, we extend the method in the following two ways: (1) adding new object models and visual features to deal with various urban scenes and (2) introducing a Markov localization strategy to utilize the history of movements. The new method can achieve a 100% localization performance in an urban route under a wide variety of conditions. The comparison with local feature-based methods is also discussed.

This paper discusses a sketch interface that can be used to guide a mobile robot along a specified path in its unfamiliar place. With the sketch interface, the user draws a rough map to give navigation tasks to robots. Because sketched maps often suffer from various inaccuracies and large errors in landmarks, we discuss what kinds of uncertainties in the rough maps would mainly have effects on navigating a robot. The effects of such inaccuracies on robot navigation are analyzed in simulated environments. A quantitative navigability measure of rough maps is then developed based on the analysis. Experimental results are also presented for validating the navigability measure.

This paper describes a view planning of multiple cameras for tracking multiple persons for surveillance purposes. When only a few active cameras are used to cover a wide area, planning their views is an important issue in realizing a competent surveillance system. We develop a multi-start local search (MLS)-based planning method which iteratively selects fixation points of the cameras by which the expected number of tracked persons is maximized. Considering the fact that a person's motion can be estimated with its intermittent observations, we set a criterion which encourages frequent shifts of fixation points and develop a procedure for generating promising initial solutions for MLS. The method is shown to outperform the other approaches. We then modify the method such that it dynamically divides the cameras into mutually independent groups and determines fixation points within each group. The modified method is comparable to the original one with a much lower planning cost.

We describe a method of mobile robot localization based on a rough map using stereo vision, which uses multiple visual features to detect and segment the buildings in the robot's field of view. The rough map is an inaccurate map with large uncertainties in the shapes, the dimensions and the locations of objects so that it can be built easily. The robot fuses odometry and vision information using extended Kalman filters to update the robot pose and the associated uncertainty based on the recognition of buildings in the map. We use multi-hypothesis Kalman filter to generate and track Gaussian pose hypotheses. An experimental result shows the feasibility of our localization method in an outdoor environment.

This paper describes a method of probabilistic obstacle map building based on Bayesian estimation. Most active or passive obstacle sensors observe only the most frontal objects and any objects behind them are occluded. Since the observation of distant places includes large depth errors, a conventional method, which does not consider the sensor occlusion often, generate erroneous maps. We introduce a probabilistic observation model, which determines the visible objects. We first estimate probabilistic visibility from the current viewpoint by a Markov chain model based on the knowledge of the average sizes of obstacles and free areas. Then the likelihood of the observations based on the probabilistic visibility are estimated and then the posterior probability of each map grid are updated by Bayesian update rule. Experimental results show that more precise map building can be built by this method.

This paper deals with generation of 3D environment models. The model is expected to be used for location recognition by robots and users. For such a use, very precise models are not necessary. We therefore develop a method of generating 3D environment models relatively simply and fast. We use an omnidirectional stereo as a primary sensor and additionally use a laser range finder. The model is composed of layered contours of free spaces, with textures extracted from images. Results of modeling and application of the model to robot localization are presented

This paper describes a unique approach of applying a pattern classification technique to robot path planning. A collision-free path connecting a start and a goal point provides information on the division of the space. In the case of 2D path planning, for example, the path divides the space into two regions. This suggests a dual problem of first dividing the whole space into such two regions and then picking up the boundary as a path. We develop a method of solving this dual problem using support vector machine (SVM). SVM generates a nonlinear separating surface based on the margin maximization principle. This property is suitable for the purpose of usual path planning problems, that is, generating a safe and smooth path. The details of the path planning methods in 2D and 3D spaces are described with several planning results. Future possibilities of combining the proposed concept with other path planning methodologies are also discussed

This paper describes a panoramic view-based navigation in outdoor environments. We have been developing a two-phase navigation method. In the training phase, the robot acquires image sequences along the desired route and automatically learns the route visually. In the subsequent autonomous navigation phase, the robot moves by localizing itself by comparing input images with the learned route representation. To be robust to changes of weather and seasons, an object-based comparison is adopted. Our previous method applied a support vector machine (SVM) algorithm to object recognition and localization and exhibited a satisfactory performance but was sometimes sensitive to the variation of the robot's heading. This paper thus extends the method to use panoramic images. By searching the image for the region which matches the model image the most, a new method can considerably improve the localization performance and provide the robot with globally correct directions to move

This paper describes a choice strategy to ease user's burdens for an interactive object recognition system when the system obtains multiple object candidates as a recognition result. First, we propose several methods to display the recognition result so as to make recognition of the candidate easy and hierarchical methods to reduce candidates per choice. We verify their effectiveness by subjective tests. Next, we propose a strategy to minimize time spent for choices. We divide the time into that for displayed candidates and that for speech dialog, and formulate each time to evaluate the strategy quantitatively. Last, we compare the strategy based on choice time with a subjective strategy

Personal service robots are expected to help people in their everyday life in the near future. Such robots must be able to not only move around but also perform various operations such as carrying a user-specified object or turning a TV on. Robots working in houses and offices have to deal with a vast variety of environments and operations. Since it is almost impossible to give the robots complete knowl edge in advance, on-site robot teaching will be important. We are developing a novel teaching framework called task model-based interactive teaching. A task model describes what knowledge is necessary for achieving a task. A robot examines the task model to determine missing pieces of knowledge, and asks the user to teach them. By leading the interaction with the user in this way, the user can teach important (focal) point easily and efficiently. This paper deals with a task of moving to a destination at a different floor; the task includes not only the movement but also the operation of recognizing and pushing elevator buttons. Experimental results show the feasibility of the proposed teaching framework.

This paper describes a view-based localization method using support vector machines in outdoor environments. We have been developing a two-phase vision-based navigation method. In the training phase, the robot acquires image sequences along the desired route and automatically learns the route visually. In the subsequent autonomous navigation phase, the robot moves by localizing itself based on the comparison between input images and the learned route representation. Our previous localization method uses an object recognition method which is robust to changes of weather and the seasons; however it has many parameters and threshold values to be manually adjusted. This paper, therefore, applies a support vector machine (SVM) algorithm to this object recognition problem. SVM is also applied to discriminating locations based on the recognition results. This two-stage SVM-based localization approach exhibits a satisfactory performance for real outdoor image data without any manual adjustment of parameters and threshold values.

This paper describes an ego-motion estimation method by integrating multiple scan matching results. The method considers both the uncertainty of scan matching results and that of estimated ego-motions, and not only estimates the latest ego-motion but also updates previous ego-motions. The estimation process is formulated as an iterative one using Kalman filter. We implement the method by using an omnidirectional stereo-based scan matching method. Experimental results show the effectiveness of the proposed method.

This paper describes a calibration method of an omnidirectional stereo system. The system uses a pair of vertically-aligned catadioptric omnidirectional cameras, each of which is composed of a perspective camera and a hyperboloidal mirror, thus providing a single projection point. We divide the calibration into two steps. The first step estimates the image center and the aspect ratio by fitting an ellipse to the mirror boundary in the image. The second step estimates the focal length and the camera pose (position and orientation) including scale by using a calibration pattern and epipolar geometry. Experimental results show the effectiveness of the proposed calibration method.

This paper describes the navigation of a mobile robot in unknown static environments using an omnidirectional stereo and a laser range finder. The robot detects obstacles by the sensors, estimates the ego-motion, integrates the sensor data to generate a probabilistic occupancy map, and plans a safe motion. This paper focuses on the ego-motion estimation and the data integration for map generation. We extend our previous methods to increase the robustness of navigation. Experimental results show the feasibility of our navigation method.

This paper describes a mobile robot navigation method in dynamic environments. The method uses a real-time omnidirectional stereo, which can obtain panoramic range information of 360 degrees. From this panoramic range information, the robot estimates its ego-motion by comparing the current and the previous observations in order to integrate observations obtained at different positions. The uncertainty in the estimation is also calculated. Next, the robot recognizes and tracks moving obstacles. Finally, the robot plans a collision free path by a heuristic planner in space-time considering the velocity uncertainty of observed obstacles. Experimental results show the effectiveness of our method.

This paper describes a view-based outdoor navigation method. In the method, a user first guides a robot along a route. During this guided movement, the robot learns a sequence of images and a rough geometry of the route. The robot then moves autonomously along the route with localizing itself based on the comparison between the learned images and input images. Since appearances of objects in images may vary much according to changes of seasons and weather in outdoor scenes, a simple image comparison does not work. We, therefore, propose a comparison method in which the robot first recognizes objects in images using object models which allow for appearance variations, and then compares recognition results of learned and input images. We also developed a method which automatically selects key images used for the comparison from an image sequence. Successful autonomous navigation experiments in our campus under various conditions show the feasibility of the method.

This paper describes a method of parallel scheduling of planning and action to realize an efficient and reactive robotic system in dynamic environments. The method uses partial planning result, which comes from an iterative refinement planning, to determine feasible actions to be executed in parallel with the planning process. The method is applied to a multiple-camera multiple-person tracking problem. Simulation results show the effectiveness of the method.

This paper describes a map generation method using an omnidirectional stereo and a laser range finder. Omnidirectional stereo has an advantage of 3D range acquisition, while it may suffer from a low reliability and accuracy in range data. Laser range finders have advantage of reliable acquisition of data, while they usually obtain only 2D range information. By integrating these two sensors, a reliable map can be generated. Since the two sensors may detect different parts of an object, a separate probabilistic grid map is first generated by temporal integration of data from each sensor The resultant two maps are then integrated using a logical integration rule. An ego-motion estimation method is also described, which is necessary for integration of sensor data obtained at different positions. Experimental results on autonomous navigation in unknown environments show the feasibility of the method.

Proposes a novel concept of an intelligent navigator that can give a driver timely advice on safe and efficient driving. From both the current traffic conditions obtained from visual data and the driver's goals and preferences in driving, it autonomously generates advice and gives it to the driver. Not only can operational-level advice be generated, such as emergency braking due to an abrupt deceleration of the vehicle in front, but also tactical-level advice, such as lane changing due to a congested situation ahead. Two main components of the intelligent navigator - the advice generation system and the road scene recognition system - are explained. A three-level reasoning architecture is proposed for generating advice in dynamic and uncertain traffic environments. Online experiments using the prototype system show the potential feasibility of the proposed concept.

Proposes a method to schedule parallel execution of planning and action of a mobile robot. The method considers the following two types of parallelism: (1) acting while planning, if a partial planning result can be used to determine feasible actions, such actions can be executed while the planning process is still going; and (2) planning while acting, if the result of the current action is (at least partially) predictable, the planning for the next action can start in advance of the completion of the current action. The proposed method uses the notion of planning-action consistency to guide the scheduling. The method has been successfully applied to a mobile robot navigation problem under sensor uncertainty.

This paper proposes a method to autonomously extract stable visual landmarks from sensory data. Given a 2D occupancy map, a mobile robot first extracts vertical line features which are distinct and on vertical planar surfaces, because they are expected to be observed reliably from various viewpoints. Since the feature information such as the position and the length includes uncertainties due to errors of vision and motion of the robot, the robot then reduces the uncertainty by matching the planar surface containing the features to the map. As a result, the robot obtains modeled stable visual landmarks from the extracted features. These processes are performed online in order to adapt to actual changes of lighting and scene depending on the robot's view. Experimental results in various scenes show the validity of the proposed method.

This paper describes a real-time omnidirectional stereo system and its application to obstacle detection and tracking for a mobile robot. The stereo system uses two omnidirectional cameras aligned vertically. The images from the cameras are converted into panoramic images, which are then examined for stereo matching along vertical epipolar lines. A PC cluster system composed of 6 PCs can generate omnidirectional range data of 720/spl times/100 pixels with disparity range of 80 (about 5 frames per second). For obstacle detection, a map of static obstacles is first generated. The candidates for moving obstacles are then extracted by comparing the current observation with the map. The temporal correspondence between the candidates are established based on their estimated position and velocity which are calculated using Kalman filter-based tracking. Experimental results for a real scene are described.

This paper describes a method of tracking a person with 3D translation and rotation by integrating optical flow and depth. The target region is first extracted based on the probability of each pixel belonging to the target person. The target state (3D position, posture, motion) is estimated based on the shape and the position of the target region in addition to optical flow and depth. Multiple target states are maintained when the image measurements give rise to ambiguities about the target state. Experimental results with real image sequences show the effectiveness of our method.

In soccer games, understanding the movement of players and the ball is essential for the analysis of matches or tactics. In this paper, we present a system to track the players and ball and to estimate their positions from video images. Our system tracks the players by extracting their shirt and pants regions and can cope with the posture change and occlusion by considering their colors, positions, and velocities in the image. The system extracts ball candidates by using the color and motion information, and determines the ball among them based on motion continuity. To determine the player who is holding the ball, the position of players on the field and the 3D position of the ball are estimated. The ball position is estimated by fitting a physical model of movement in 3D space to the observed ball trajectory. Experimental results on real image sequences show the effectiveness of the system.

Describes a method of modeling the motion uncertainty of moving obstacles and its application to mobile robot motion planning. The method explicitly considers three sources of motion uncertainty: path ambiguity, velocity uncertainty, and observation uncertainty. The model is represented by a probabilistic distribution over possible position an a path of a moving obstacle. Using this model, the best robot motion is selected which minimizes the expected time of reaching the destination. By considering not the range but the distribution of the uncertainty, more efficient behaviors of the robot are realized.

Proposes a method to autonomously select stable visual landmarks from observed features by stereo vision and a given 2D obstacle map. The robot selects as stable landmarks vertical line segments which are distinct and on a vertical plane, because they are expected to be observed reliably from various viewpoints. Due to the vision and motion error of the robot, the observed feature positions include uncertainty. This uncertainty can be reduced by matching the detected vertical plane which includes the features to a known plane in the map. The position of a selected feature is modeled by a probabilistic distribution on the known plane. The selection and modeling process is performed online to adapt to an actual lighting and background condition which varies depending on viewpoints. When the robot moves, it uses several, less uncertain landmarks to estimate its motion. Experimental results in real scenes show the validity of the proposed method.

This paper describes a method to track a person by integrating two cues, optical flow and uniform brightness regions, where optical flow cannot be obtained. This method works even if tracking with either optical flow or uniform brightness regions may fail. The proposed method has been implemented on a real time image processor with multiple DSPs and successfully tracked a target person using real image sequences.

This paper proposes a novel method of scheduling parallel execution of planning and action for a vision-motion planning problem. A planning process can be viewed as a process of gradually reducing the plan candidates towards the final commitment to one plan. Using criteria on deciding if an action sequence is consistent with the remaining plan candidates (consistency criterion), and on when to commit to the final plan (commitment criterion), an appropriate action sequence is selected and executed while the planning process is still continuing. Preliminary experimental results including the comparison with a sequential method show that the proposed method is promising.

Describes a method of planning of observation and motion for a mobile robot to interpret a road intersection scene and to reach the intersection as soon as possible considering the uncertainty of interpretation. From a monocular color image, candidate regions are extracted for objects which are related to the intersection type. The probabilities of the regions coming from the objects are calculated from probabilistic models of the objects. From the probabilities of the regions and the relation between the intersection type and the objects, the current probability of intersection types is calculated. If the intersection type is ambiguous, the robot plans the observation and motion which minimize the expectation of the cost (time) to complete the task. The robot takes these actions and tries to determine the intersection type again. This process is iterated until the task is completed. The experimental result is shown for an actual intersection scene.

This paper describes a method of systematically generating visual sensing strategies based on knowledge of the assembly task to be performed. Since visual sensing is usually performed with limited resources, visual sensing strategies should be planned so that only necessary information is obtained efficiently. The generation of the appropriate visual sensing strategy entails knowing what information to extract, where to get it, and how to get it. This is facilitated by the knowledge of the task, which describes what objects are involved in the operation, and how they are assembled. In the proposed method, using the task analysis based on face contact relations between objects, necessary information for the current operation is first extracted. Then, visual features to be observed are determined using the knowledge of the sensor, which describes the relationship between a visual feature and information to be obtained. Finally, feasible visual sensing strategies are evaluated based on the predicted success probability, and the best strategy is selected. Our method has been implemented using a laser range finder as the sensor. Experimental results show the feasibility of the method, and point out the importance of task-oriented evaluation of visual sensing strategies.

This paper deals with planning of visual navigation strategies for a mobile robot under uncertainty of both visual data and dead reckoning data. When a robot passes through a narrow space, it has to move slowly while carefully observing surrounding objects and estimating the distances to the objects precisely. On the other hand, the robot can move fast in a widely open space. A novel method is proposed which can adaptively determine the speed of the robot and the visual landmarks, thereby realizing an efficient navigation of the robot. Experimental results show the feasibility of the method.

This paper describes a method of determining contours of moving objects in a cluttered scene by integrating optical flow and edges. If the motion of an object is similar to that of the background, the contour is not determined only by optical flow. If the background of a scene is cluttered, the contour is not determined only from edges because many edges may be extracted in the background and no edges may be extracted on some parts of the contour. In the proposed method, the contour is determined by using optical flow and edges in a long sequence. The whole contour of a moving object is eventually obtained by accumulating edges near motion boundaries over an image sequence. The method can also determine the occlusion relation of two overlapping objects by checking if edges exist on the predicted contours of objects. Experimental results for synthetic and real images show the usefulness of the method.

This paper describes a method of recognition of intersection scenes by attentive observation considering the uncertainty of recognition, for mobile robots on roads. From a monocular color image, homogeneous color regions are extracted. Probabilities of the regions coming from specific objects are calculated. From these probabilities and the relationship between these objects and intersection types, the current probability distribution of intersection types is calculated. If the entropy of the probability distribution is lower than a certain threshold, the robot adopts the best hypothesis. Otherwise, the robot selects and observes the part which can minimize the expectation of the entropy attentively. These actions are iterated until the entropy becomes lower than the threshold. The experimental results are shown for actual intersection scenes including a white mark, a curve mirror and so on.

In vision-guided robotic operations, vision is used for extracting necessary information for achieving the task. Since visual sensing is usually performed with limited resources, visual sensing strategies should be planned so that only necessary information is obtained efficiently. This paper describes a method of systematically generating visual sensing strategies based on knowledge of the task to be performed. The generation of the appropriate visual sensing strategy entails knowing what information to extract, where to get it, and how to get it. This is facilitated by the knowledge of the task, which describes what objects are involved in the operation, and how they are assembled. Our method has been implemented using a laser range finder as the sensor. Experimental results show the feasibility of the method, and point out the importance of task-oriented evaluation of visual sensing strategies.>

It is generally very difficult, if not impossible, for a robot to perform fine manipulation tasks without the benefit of some form of sensory feedback during actual task execution. As a result, sensing planning is an important component in assembly task planning. This paper describes a method of generating visual sensing strategies based on knowledge of the task to be performed. The generation of the appropriate visual sensing strategy entails knowing what information to extract and where to get it. This is facilitated by the knowledge of the task, which describes how objects are assembled. This knowledge, coupled with known sensor modeling, results in an abstract template of sensing strategy called the sensing task model. By instantiating the appropriate sensing task model at planning time, the sensing strategy is efficiently generated. Our method has been implemented using a laser range finder as the sensor. Experimental results involving typical assembly tasks show the feasibility of the method.

This paper describes real time object tracking by extracting optical flows from a sequence of images. Optical flows are calculated based on the generalized gradient model. Our method detects a moving object, and tracks it from the optical flow data. Assuming that pixels corresponding to the same object have similar flow vectors, we extract a connected region with similar flow vectors. We apply this method to multiple object tracking. When two objects overlap, foreground object is recognized, and each object is tracked without confusion. Optical flow extraction and object tracking are executed in realtime by a special image processor.

The ability of manipulating flexible objects, such as rubber belts and paper sheets, is important in automated manufacturing systems. This paper describes a novel approach to assembly of flexible objects. The operation dealt with in this paper is to assemble a rubber belt with fixed pulleys. By analyzing possible states of the belt based on the empirical knowledge of the belt, one can derive a method to have not only the action planning but also the visual verification planning. The authors have implemented a belt assembly system using two manipulators and a laser range finder as the sensor, and succeeded in performing the belt-pulley assembly. Extension of the authors' approach to other kinds of assembly of flexible objects is also discussed.

This paper describes a new method of modeling an environment in terms of obstacles and free spaces from a set of 3D segments obtained by stereo vision. Since live stereo vision provides only the position of segments, it is necessary to determine whether a region formed by the segments is an obstacle or a free space. The ambiguities and the uncertainties in the obtained data must be considered in modeling. The final output of the proposed method is a set of possible situations of the environment and their probabilities; each situation consists of the description of obstacles and critical regions between the obstacles. Experimental results for a real scene are described.>

This paper describes a navigation method for a mobile robot by attentive observation. A 3D scene description is constructed with the constrained Delaunay triangulation and a path to a given destination is determined from the description. If any reliable path is not found, the unknown image region on the most promising path is observed attentively. The newly obtained data is integrated to the original description and a path is searched for in the new description. The experimental results are shown.>

This paper describes an approach to automatic landmark selection for navigation of a vision guided vehicle. The system works in two phases. In the first phase, the vehicle is driven by a human through a corridor. During this movement, a vision system extracts features such as doors or staircases. The category and the location of extracted features are recorded in a map. In the second phase, given a destination by a human, the system examines the map and selects a sequence of landmarks which guide the vehicle to the destination in the minimum time. Both the uncertainty of the motion of dead reckoning and that of visual data are considered in landmark selection. Preliminary experiments are performed for a autonomous vehicle which moves in the authors' building.

The authors describe a framework for planning of vision and motion for a mobile robot. For planning in a real world, the uncertainty and the cost of visual recognition are important issues. A robot has to consider a tradeoff between the cost of visual recognition and the effect of information obtained by recognition. A problem is to generate a sequence of vision and motion operations based on sensor information which is an integration of the current information and the predicted next sensor data. The problem is solved by recursive prediction of sensor information and the recursive search of operations. As an example of sensor modeling, a model of stereo vision is described in which correspondence of wrong pairs of features as well as quantization error are considered. Using the framework, a robot can successfully generate a plan for real-world problem.>

A new framework of a hierarchical vision-motion planning for a mobile robot under uncertainty is proposed. An optimal plan is generated by two-level planning: global route selection and local path planning. In global route selection, a sequence of observation points to acquire sufficient information to reach a destination is determined. Both the cost and the uncertainty of vision are considered in this planning. In local path planning, given two successive observation points, trajectories, moving speeds, and reference points for robot localization are determined so that a robot can reach the second point safely with a minimum cost. Both the error in localization and that in motion control are considered in this planning. A local path planner is repeatedly invoked in global route selection to determine an actual path between observation points. Our hierarchical planner can generate an optimal plan for a mobile robot planning problem. I. Introduction A mobile robot with vision moving to ...

The authors propose a framework of unified planning of vision and motion with uncertainty. They use a probabilistic model to represent uncertainty and use statistical decision theory to make a unified plan of vision and motion. They describe a method of predicting the information acquired by a sensing operation and formulate the planning problem in a recurrence formula. They analyze vision-motion planning problem in a simple example and conclude that the combination of dynamic programming and hill-climbing is useful. Simulation results show the validity of the approach.>

In this paper, about the annoyance of musical signal at concert hall, we describe the preliminary experiment using digital signal processing technique and method of measurement of A-weighted sound pressure level (LAeq). To consider the time variance of musical signal in frequency domain, we adopted digital signal processing technique. Concretely, the musical signal of which duration is 3min 40sec, was analyzed by DFT with every 200ms, and 4096 points FFT operation. Next, measurement of LAeq was used to observe the fluctuation of level of musical signal in time domain. At last, we considered the relationship between this fluctuation and the result of FFT operation. And at the same time, also the same analysis was applied about traffic noise, then we described macroscopic consideration for the physical and psychological difference of musical sound and traffic noise.