Lifelong multiagent path finding (LMAPF) requires continuous task assignment and collision-free path planning for large robot fleets. However, existing LMAPF methods assume synchronous unit-time actions, whereas real robots execute movements asynchronously with nonuniform durations. This mismatch can lead to execution-time collisions and makes frequent replanning challenging in large-scale systems. We study LMAPF with asynchronous actions and present a framework that ensures safety under asynchronous execution and real-time scalability through coordinated path planning and action scheduling, complemented by a simple greedy task allocation. To ensure safety, the path planner eliminates cycle conflicts to prevent structural deadlocks, while a lightweight action scheduler enforces vertex precedence during execution, resolving following conflicts without explicit temporal modeling. To achieve real-time performance, the planner performs partial replanning by reusing the residual paths of nonidle agents, which are converted into a synchronous form through the scheduler’s path resynchronization, enabling consistent and efficient replanning. Experiments in warehouse environments with up to 1000 robots demonstrate that our framework more than doubles throughput compared with state-of-the-art LMAPF methods, while maintaining safe operations under asynchronous actions.

Fall detection is one of the most important functions for a healthcare robot system because falls are very dangerous for older people and might lead to death if failed to provide prompt and adequate treatment. In this paper, we propose an efficient fall detection method based on 3-axis accelerometer and depth sensor fusion. LSTM networks are applied to handle temporal information. Simple low-level motion and pose features are obtained from each sensor data, and then fed into the LSTM networks that can learn high-level feature representations to classify falls from other daily life activities. Also, various learning tricks are combined to improve the performance. Experimental results show that the proposed fall detection method outperforms existing methods.

Studies on the production of emotions have been conducted to create robotic facial expressions. The reported methodologies for generating emotions for a robot have focused on recognizing a user’s emotions using devices, such as cameras and microphones, and then generating the reactive emotions of a robot according to the user’s emotions. However, these methodologies may have some limitations in delivering emotions in the robot that match the robot’s utterances to users. In this paper, we propose a methodology for producing robotic emotions suitable for a robot’s utterances based on texts so that it can be applied to various fields such as robotic dialogue and reading services. To produce human-like emotions in a robot, our methodology applied patterns of human emotional changes as well as the resilience theory that humans have an ability to recover their emotional states considering their own personality over time. We measured the performance of the model for analyzing texts and observed that there is a linear correlation between the predicted emotions and the annotated ones from humans. Furthermore, when we carried out the experiments based on scenarios, our methodology could produce human-like patterns of emotional changes and the robot could recover its emotional state on its own.

In this paper, we introduce an ontology-based home care service robot that can provide personalized care for people who are in the early stage of dementia. The hardware and software framework encompassed in the proposed service robot was developed to carry out care services in their daily life at home. Specifically, to generate adaptive task plans in diverse caring situation, context reasoner and ontological model of dementia are included. Ontology includes various concepts that are related with the knowledge of caring dementia patient: dementia, dementia symptom, environment of around patient, and situation during patient's daily life. To evaluate if the proposed service robot could provide appropriate care service or not, experimental care scenario for helping a person with dementia take medicine was tried in the lab environment. Although tasks of the robot required for the experiment are rather simple, we have demonstrated that the robot could provide a personalized service that may be beneficial to dementia patient, family members and caregivers. In the future, we will add more care knowledge in the ontology and further develop a variety of care services. Additionally, we are going to test the care service robot in a real environment with actual dementia patient.

In this paper, a robust lane detection method based on vanishing point estimation is proposed. Estimating a vanishing point can be helpful in detecting lanes, because parallel lines converge on the vanishing point in a projected 2-D image. However, it is not easy to estimate the vanishing point correctly in an image with a complex background. Thus, a robust vanishing point estimation method is proposed that uses a probabilistic voting procedure based on intersection points of line segments extracted from an input image. The proposed voting function is defined with line segment strength that represents relevance of the extracted line segments. Next, candidate line segments for lanes are selected by considering geometric constraints. Finally, the host lane is detected by using the proposed score function, which is designed to remove outliers in the candidate line segments. Also, the detected host lane is refined by using inter-frame similarity that considers location consistency of the detected host lane and the estimated vanishing point in consecutive frames. Furthermore, in order to reduce computational costs in the vanishing point estimation process, a method using a lookup table is proposed. Experimental results show that the proposed method efficiently estimates the vanishing point and detects lanes in various environments.

Social skills training of children with autism spectrum disorders (ASD) is not an easy problem enough to typically spend a lot of time and repetitive efforts. As means of assistance, we propose and examine the feasibility of a robotic behavioral intervention system to facilitate social development of the children with ASD and relieve operational burden during training process. To this end, based on well-known behavioral treatment protocols, the robot system incorporates robotic stimulation design, recognition modules for human activities, and reinforcement procedure in the interaction scheme design. Using these configuration, it repeatedly perform a role of training eye contact and reading emotions targeted at preschoolers with a high functioning level in the planned training process. Through showing an advantage in a comparative analysis with control group taught by humans, we verified that the proposed system can contribute to evoke positive response of children with ASD and provide labor-saving effect on the clinical environment.

Particle filter is a widely used framework for object tracking, but it is vulnerable when its observation model is based on visual appearance. In this paper, we propose a modified particle filtering that makes use of foreground regions and their pixel-based confidences that are likely to be foreground; the foreground regions are used for preventing generations of particle in the background and the pixel-based confidences are enable to enhance the similarity between foreground and observation models. We evaluate the performance on five datasets and show that the proposed approach outperforms a number of state-of-the-art object tracking methods.

In this paper, we propose a novel method for unusual human activity detection in crowded scenes. Specifically, rather than detecting or segmenting humans, we devised an efficient method, called a motion influence map, for representing human activities. The key feature of the proposed motion influence map is that it effectively reflects the motion characteristics of the movement speed, movement direction, and size of the objects or subjects and their interactions within a frame sequence. Using the proposed motion influence map, we further developed a general framework in which we can detect both global and local unusual activities. Furthermore, thanks to the representational power of the proposed motion influence map, we can localize unusual activities in a simple manner. In our experiments on three public datasets, we compared the performances of the proposed method with that of other state-of-the-art methods and showed that the proposed method outperforms these competing methods.

In this paper, we propose a region-based object recognition (RBOR) method to identify objects from complex real-world scenes. First, the proposed method performs color image segmentation by a simplified pulse-coupled neural network (SPCNN) for the object model image and test image, and then conducts a region-based matching between them. Hence, we name it as RBOR with SPCNN (SPCNN-RBOR). Hereinto, the values of SPCNN parameters are automatically set by our previously proposed method in terms of each object model. In order to reduce various light intensity effects and take advantage of SPCNN high resolution on low intensities for achieving optimized color segmentation, a transformation integrating normalized Red Green Blue (RGB) with opponent color spaces is introduced. A novel image segmentation strategy is suggested to group the pixels firing synchronously throughout all the transformed channels of an image. Based on the segmentation results, a series of adaptive thresholds, which is adjustable according to the specific object model is employed to remove outlier region blobs, form potential clusters, and refine the clusters in test images. The proposed SPCNN-RBOR method overcomes the drawback of feature-based methods that inevitably includes background information into local invariant feature descriptors when keypoints locate near object boundaries. A large number of experiments have proved that the proposed SPCNN-RBOR method is robust for diverse complex variations, even under partial occlusion and highly cluttered environments. In addition, the SPCNN-RBOR method works well in not only identifying textured objects, but also in less-textured ones, which significantly outperforms the current feature-based methods.

In this paper, we propose with a robot-assisted behavioral intervention system to easily improve children's social capability. In particular, the system for children with autism spectrum disorders (ASD) is basically achieved through the discrete trial teaching (DTT) protocol with three task modes of therapy, encouragement, and pause in social training scenarios. In child-robot interaction architecture, the robot firstly offers therapeutic training elements of mutual greeting and interplay game, and evaluates the level of children's reactivity by recognition modules for frontal face and touch features. Thence, the system in the decision-making process determines the task mode to perform subsequent action by grasping behavioral state of the children, and then it copes with individual response appropriately by using the robotic stimuli with the combination of kinesic acts and displayable contents. From the experiments of clinical trials with children with non-ASD and ASD in each robotic stimulus, the system showed the potential to increase their attention and activeness for social training, and we believe that the proposed system has some positive effect on developing children's social skills.

This paper presents a fast and efficient computational approach to higher order spectral graph matching. Exploiting the redundancy in a tensor representing the affinity between feature points, we approximate the affinity tensor with the linear combination of Kronecker products between bases and index tensors. The bases and index tensors are highly compressed representations of the approximated affinity tensor, requiring much smaller memory than in previous methods, which store the full affinity tensor. We compute the principal eigenvector of the approximated affinity tensor using the small bases and index tensors without explicitly storing the approximated tensor. To compensate for the loss of matching accuracy by the approximation, we also adopt and incorporate a marginalization scheme that maps a higher order tensor to matrix as well as a one-to-one mapping constraint into the eigenvector computation process. The experimental results show that the proposed method is faster and requires smaller memory than the existing methods with little or no loss of accuracy.

In this paper, we propose a multiple 3D camera-based human tracking method which is robust to illumination changes and occlusions at indoor environments. To overcome the difficulties due to illumination change, several types of image features are used in a collaborative fashion, for which brightness intensity, hue, local binary pattern (LBP) and depth from 3D camera are considered. In addition, our method also exploits multiple camera views to resolve the occlusion between objects. Our algorithm first implements the background subtraction to extract moving objects from each camera view and then executes the human identification process to determine whether the human is previously confirmed. The proposed algorithm estimates the vertical axes of the humans detected in multiple calibrated camera views, which leads to generating the cross points of the detected human objects. Finally, the cross points (the location of the human objects) are fed into adaptive particle filter based on spatio-temporal information to track the human objects. The performance of the proposed algorithm is examined through experiments performed in varying indoor illumination and occlusion conditions.

An automatic parameter setting method of a simplified pulse coupled neural network (SPCNN) is proposed here. Our method successfully determines all the adjustable parameters in SPCNN and does not need any training and trials as required by previous methods. In order to achieve this goal, we try to derive the general formulae of dynamic threshold and internal activity of the SPCNN according to the dynamic properties of neurons, and then deduce the sub-intensity range expression of each segment based on the general formulae. Besides, we extract information from an input image, such as the standard deviation and the optimal histogram threshold of the image, and attempt to build a direct relation between the dynamic properties of neurons and the static properties of each input image. Finally, the experimental segmentation results of the gray natural images from the Berkeley Segmentation Dataset, rather than synthetic images, prove the validity and efficiency of our proposed automatic parameter setting method of SPCNN.

This paper presents a new scan matching method for mobile robot localization. The proposed method is based on a spectral technique, which finds consistent correspondences between two range scans without any initial alignment while considering how well their associated pairwise geometric relationships are satisfied. Based on the scan matching method, we also suggest a strategy for global localization in indoor environments, which is applicable to both grid maps and topological maps having metric information. The feasibility of the proposed methods is demonstrated by experimental results.

In this paper we propose a new method for on-line object segmentation through human-robot interaction. Particularly, we define three types of human gestures for object learning by the size of target objects; holding small objects, pointing at medium ones and contacting two corners of large ones. The regions of interest where objects are likely to be located are interpreted from those gestures and represented as rectangles in captured images. For object segmentation, we suggest a marker-based watershed segmentation method which segregates an object within a region of interest in real-time performance. Experimental results show that the segmentation quality of our method is as good as that of the GrabCut algorithm, but the computational time of ours is so much faster that it is appropriate for practical applications.

Stereo vision is a well-known ranging method because it resembles the basic mechanism of the human eye. However, the computational complexity and large amount of data access make real-time processing of stereo vision challenging because of the inherent instruction cycle delay within conventional computers. In order to solve this problem, the past 20 years of research have focused on the use of dedicated hardware architecture for stereo vision. This paper proposes a fully pipelined stereo vision system providing a dense disparity image with additional sub-pixel accuracy in real-time. The entire stereo vision process, such as rectification, stereo matching, and post-processing, is realized using a single field programmable gate array (FPGA) without the necessity of any external devices. The hardware implementation is more than 230 times faster when compared to a software program operating on a conventional computer, and shows stronger performance over previous hardware-related studies.

In this paper, we suggest a new method of human augmented mapping for indoor environments using only a stereo camera. Through user's help, a robot with a stereo camera can investigate the environment without failure and even more efficiently. Moreover, the user can share the information about the environment with the robot and add semantic information to the environmental map. We employ PCA features for visual landmarks and a hybrid map for map representation. Particularly, we define two types of nodes, U/R-nodes and divide the map building into three processes, User's Guidance, Robot's Map Revision, and Robot's Map Completion. We implemented a human augmented mapping system with a stereo camera and demonstrated it in rectangular-shaped corridors. From the comparison with a manually-built map, we showed the feasibility of the environmental map generated by our proposed method.

This paper proposes a novel global localization approach that uses hybrid maps of objects and spatial layouts. We model indoor environments using the following visual cues from a stereo camera: local invariant features for object recognition and their 3D positions for object location representation. We also use a 2D laser range finder. Therefore, we can build a hybrid local node for a topological map that is composed of an object location map and a spatial layout map. Based on this modeling, we suggest a coarse-to-fine strategy for the global localization. The coarse pose is obtained by means of object recognition and point cloud fitting, and then its fine pose is estimated with a probabilistic scan matching algorithm. With real experiments, we show that our proposed method can be an effective global localization algorithm.

We propose an effective facial features detection method for human– robot interaction (HRI) with indoor mobile robot. In case of human-robot interaction, its vision system has to manage such difficult problems as pose variations, illumination changes, and complex backgrounds, which is caused by the mobility of robot. In this paper, in order to overcome such difficult problems, we suggest a facial feature detection method based on local image area and direct pixel-intensity distributions, in which we propose two novel concepts; the directional template for evaluating intensity distributions and the edge-like blob map with multiple strength intensity. Using this proposed blob map, we show that the locations of major facial features – two eyes and a mouth – can be successfully estimated. Without the information of facial area boundary, final candidate face region is determined by both obtained locations of facial features and weighted correlations with stored facial templates. Our approach is also a flexible algorithm that can be applicable to both color and gray image. In case of color image, faster detection of both facial features and face is feasible by using the chromatic property of facial color. Experimental results from many color images and well-known gray level face database images prove the usefulness of proposed algorithm in human– robot interaction applications.

This paper proposes a formal selection framework of multiple navigation behaviors for a service robot. In our approach, modeling, analysis, and performance evaluation are carried out based on the generalized stochastic Petri nets (GSPN). By adopting probabilistic approach, our framework helps the robot to select the most desirable navigation behavior in run time according to environmental conditions. Moreover, after a mission, the robot evaluates prior navigation performance from accumulated data, and uses the results for the improvement of future operations. Also, GSPN has several advantages over classic automata or direct use of Markov process. The basic ideas of the framework were introduced in our previous work (Gunhee Kim et al., 2005). Thus, this paper focuses on experimental verification by implementing the framework into the guide robot Jinny. We conduct the experiments about real guidance tasks with visitors in the National Science Museum of Korea. The results show that the proposed strategy is useful to select an appropriate navigation behavior in a dynamic space.

This paper describes a new method for indoor environment mapping and localization with stereo camera. For environmental modeling, we directly use the depth and color information in image pixels as visual features. Furthermore, only the depth and color information at horizontal centerline in image is used, where optical axis passes through. The usefulness of this method is that we can easily build a measure between modeling and sensing data only on the horizontal centerline. That is because vertical working volume between model and sensing data can be changed according to robot motion. Therefore, we can build a map about indoor environment as compact and efficient representation. Also, based on such nodes and sensing data, we suggest a method for estimating mobile robot positioning with random sampling stochastic algorithm. With basic real experiments, we show that the proposed method can be an effective visual navigation algorithm.