Expressive human pose and shape estimation (EHPS) unifies body, hands, and face motion capture with numerous applications. Despite encouraging progress, current state-of-the-art methods focus on training innovative architectural designs on confined datasets. In this work, we investigate the impact of scaling up EHPS towards a family of generalist foundation models. 1) For data scaling, we perform a systematic investigation on 40 EHPS datasets, encompassing a wide range of scenarios that a model trained on any single dataset cannot handle. More importantly, capitalizing on insights obtained from the extensive benchmarking process, we optimize our training scheme and select datasets that lead to a significant leap in EHPS capabilities. Ultimately, we achieve diminishing returns at 10 M training instances from diverse data sources. 2) For model scaling, we take advantage of vision transformers (up to ViT-Huge as the backbone) to study the scaling law of model sizes in EHPS. To exclude the influence of algorithmic design, we base our experiments on two minimalist architectures: SMPLer-X, which consists of an intermediate step for hand and face localization, and SMPLest-X, an even simpler version that reduces the network to its bare essentials and highlights significant advances in the capture of articulated hands. With Big Data and the large model, the foundation models exhibit strong performance across diverse test benchmarks and excellent transferability to even unseen environments. Moreover, our finetuning strategy turns the generalist into specialist models, allowing them to achieve further performance boosts. Notably, our foundation models consistently deliver state-of-the-art results on seven benchmarks such as AGORA, UBody, EgoBody, and our proposed SynHand dataset for comprehensive hand evaluation.

As the space of causal sequence modeling architectures continues to grow, the need to develop a general framework for their analysis becomes increasingly important. With this aim, we draw insights from classical signal processing and control theory, to develop a quantitative measure of memory utilization: the internal mechanisms through which a model stores past information to produce future outputs. This metric, which we call effective state-size (ESS), is tailored to the fundamental class of systems with input-invariant and input-varying linear operators, encompassing a variety of computational units such as variants of attention, convolutions, and recurrences. Unlike prior work on memory utilization, which either relies on raw operator visualizations (e.g. attention maps), or simply the total memory capacity (i.e. cache size) of a model, our metrics provide highly interpretable and actionable measurements. In particular, we show how ESS can be leveraged to improve initialization strategies, inform novel regularizers and advance the performance-efficiency frontier through model distillation. Furthermore, we demonstrate that the effect of context delimiters (such as end-of-speech tokens) on ESS highlights cross-architectural differences in how large language models utilize their available memory to recall information. Overall, we find that ESS provides valuable insights into the dynamics that dictate memory utilization, enabling the design of more efficient and effective sequence models.

This paper proposes a method for generating maps in indoor environments that include transparent objects by using a stereo polarization camera and projector. Conventional sensors like LiDAR and stereo cameras struggle with glass, as they rely on diffuse reflection, while glass allows light to pass through. In contrast, polarization cameras can measure light polarization and estimate surface normals, enabling depth estimation by combining polarization and RGB information. However, when measuring transparent objects, reflected and transmitted light cancel each other out, reducing polarization contrast, and the RGB information causes the depth estimation to output the depth of objects behind the glass. To address this issue, this paper proposes a novel method that (1) improves the S/N ration in polarization measument via diffuse reflection on non-glass regions and (2) masks out the RGB color from polarimetric depth estimation to not compute depth map of objects behind the glass to obtain depth images that include glass surfaces. Additionally, (3) in the mapping part, depth estimation is repeated at multiple locations, and the results are integrated using self-localization to generate a complete environmental map. Experiments in an indoor environment confirmed the effectiveness of the proposed method, enabling glass-inclusive depth estimation and successful map generation on a mobile robot.

Self-efficacy is crucial for the effective application of assistive technology and rehabilitation. This study proposes a novel approach to assess the impact of motor interventions on motor self-efficacy, relevant for human-robot interaction in rehabilitation, by focusing on the perceived reachable space. Twelve healthy adults underwent an arm movement restriction intervention using a robotic arm (KINARM), and changes in the perceived reachable space and muscle activity were measured before and after the intervention. The results indicated a reduction in the perceived reachable space and an adaptive decrease in muscle activity for unreachable targets following motor restriction. This suggests that the perceived reachable space can serve as an objective proxy for task-specific motor self-efficacy, which is valuable for evaluating user adaptation to robotic interfaces. Furthermore, these findings imply that in rehabilitation using interactive robots, a patient’s effort levels may be influenced by their perception of task achievability.

Many elderly individuals experience a decline in motor function. To provide appropriate rehabilitation programs, a sufficient and convenient evaluation method is necessary. In this study, we focused on the sit-to-stand (STS) motion, a crucial activity in daily life, and used handrail to obtain force data safely and easily. Previous studies have proposed methods for estimating scores such as the Timed Up and Go test from forces applied to the hand, hip, and foot, classifying elderly individuals into several motor function categories. However, these indicators are insufficient for evaluating the function of specific muscles or joints in the lower extremities individually. The primary objective of this study was joint torque, which more directly represents the function of specific muscles and joints. We measured the time-series data of forces acting on the body during STS motions and developed a model using Long Short-Term Memory to estimate lower limb joint torques. As a result, knee and hip joint torques were accurately estimated from force applied to hand, hip and foot. Furthermore, this method demonstrated the potential for early detection of joint disorders. This approach allows for a detailed assessment of the state of the knee and hip joints simply by standing up while holding a handrail.

We present a novel acoustic camera simulator (ACSim) that generates realistic sonar images by incorporating recursive ray tracing and sonar artifact modeling and provides various ground truth labels, enabling benchmarking and learning purposes. The 2D forward-looking sonar (FLS), also known as the acoustic camera, produces high-quality 2D images. Conducting real-world underwater experiments is challenging, making realistic sonar image simulation a necessary alternative. However, existing simulators often lack sufficient realism or are limited to specific scenes and phenomena. As a result, training on simulations and testing on real sonar images (i.e., sim-to-real) remains an open problem for deep learning-based applications. Our work introduces a novel sonar simulator with a customized rendering engine. We use recursive ray tracing to model multipath reflections in arbitrary scenes and propose physics-based shading for intensity computation. We propose a resampling method for anti-aliasing and model significant artifacts such as rolling shutter distortions and cross-talk noise. The simulator provides various ground truths for benchmarking and deep learning applications. We tested several tasks by training on synthetic images and demonstrated that the models also work on real images. We developed a Blender add-on for an enhanced user interface and will make the simulator open-source to advance future research.

We approach designing a state-space model for deep learning applications through its dual representation, the transfer function, and uncover a highly efficient sequence parallel inference algorithm that is state-free: unlike other proposed algorithms, state-free inference does not incur any significant memory or computational cost with an increase in state size. We achieve this using properties of the proposed frequency domain transfer function parametrization, which enables direct computation of its corresponding convolutional kernel’s spectrum via a single Fast Fourier Transform. Our experimental results across multiple sequence lengths and state sizes illustrates, on average, a 35% training speed improvement over S4 layers – parametrized in time-domain – on the Long Range Arena benchmark, while delivering state-of-the-art downstream performances over other attention-free approaches. Moreover, we report improved perplexity in language modeling over a long convolutional Hyena baseline, by simply introducing our transfer function parametrization. Our code is available at https://github.com/ruke1ire/RTF.

Vision-based self-localization is a crucial technology for enabling autonomous robot navigation in GPS-deprived environments. However, standard frame cameras are subject to motion blur and suffer from a limited dynamic range. This research focuses on efficient feature tracking for self-localization by using event-based cameras. Such cameras do not provide regular snapshots of the environment but asynchronously collect events that correspond to a small delta of illumination in each pixel independently, thus addressing the issue of motion blur during fast motion and high dynamic range. Specifically, we propose a continuous real-time asynchronous event-based feature tracking pipeline, named RATE. This pipeline integrates (i) a corner detector node utilizing a time slice of the Surface of Active Events to initialize trackers continuously, along with (ii) a tracker node with a proposed "tracking manager", consisting of a grid-based distributor to reduce redundant trackers and to remove feature tracks of poor quality. Evaluations using public datasets reveal that our method maintains a stable number of tracked features, and performs real-time tracking efficiently while maintaining or even improving tracking accuracy compared to state-of-the-art event-only tracking methods. Our ROS implementation is released as open-source: https://github.com/mikihiroikura/RATE

Mobile robots functioning in human environments should behave with a secure and socially-compliant manner. Although many studies have revealed the effectiveness of Deep Reinforcement Learning (DRL) in robot navigation, most of them can only handle the presence of human as independent individuals. Failing to consider groups may lead to the robot getting stuck or behaving rudely, and omitting to separately handle obstacles from pedestrians will cause low efficiency. In this work, we present a novel all-aware neural network that utilizes DRL to process groups, obstacles, and individuals simultaneously. The proposed solution employs a new Group–Robot Interaction (GRI) subnetwork to encode the mutual effects between groups and the robot, and a modified Obstacle–Robot Unilateral interaction (ORU) subnetwork is presented to avoid obstacle collisions caused by sensing noises or motion uncertainties. In addition, the influences of a pedestrian, obstacle, and group on other pedestrians or groups, that indirectly affect the robot, are also integrated into the Human–Robot Interaction (HRI) subnetwork or GRI subnetwork respectively by using map tensors. Finally, the GRI, ORU, and HRI subnetworks are aggregated into a planning subnetwork to train and derive an all-aware robot navigation policy based on DRL. Evaluation results in both real-world and simulation experiments show that the proposed approach outperforms the current cutting-edge methods.

2D forward-looking sonar is a crucial sensor for underwater robotic perception. A well-known problem in this field is estimating missing information in the elevation direction during sonar imaging. There are demands to estimate 3D information per image for 3D mapping and robot navigation during fly-through missions. Recent learning-based methods have demonstrated their strengths, but there are still drawbacks. Supervised learning methods have achieved high-quality results but may require further efforts to acquire 3D ground-truth labels. The existing self-supervised method requires pretraining using synthetic images with 3D supervision. This study aims to realize stable self-supervised learning of elevation angle estimation without pretraining using synthetic images. Failures during self-supervised learning may be caused by motion degeneracy problems. We first analyze the motion field of 2D forward-looking sonar, which is related to the main supervision signal. We utilize a modern learning framework and prove that if the training dataset is built with effective motions, the network can be trained in a self-supervised manner without the knowledge of synthetic data. Both simulation and real experiments validate the proposed method.

Mobile robot navigation in a human-populated environment has been of great interest to the research community in recent years, referred to as crowd navigation. Currently, offline reinforcement learning (RL)-based method has been introduced to this domain, for its ability to alleviate the sim2real gap brought by online RL which relies on simulators to execute training, and its scalability to use the same dataset to train for differently customized rewards. However, the performance of the navigation policy suffered from the distributional shift between the training data and the input during deployment, since when it gets an input out of the training data distribution, the learned policy has the risk of choosing an erroneous action that leads to catastrophic failure such as colliding with a human. To realize risk sensitivity and improve the safety of the offline RL agent during deployment, this work proposes a multipolicy control framework that combines offline RL navigation policy with a risk detector and a force-based risk-avoiding policy. In particular, a Lyapunov density model is learned using the latent feature of the offline RL policy and works as a risk detector to switch the control to the risk-avoiding policy when the robot has a tendency to go out of the area supported by the training data. Experimental results showed that the proposed method was able to learn navigation in a crowded scene from the offline trajectory dataset and the risk detector substantially reduces the collision rate of the vanilla offline RL agent while maintaining the navigation efficiency outperforming the state-of-the-art methods.

Retrieving the missing dimension information in acoustic images from 2D forward-looking sonar is a well-known problem in the field of underwater robotics. There are works attempting to retrieve 3D information from a single image which allows the robot to generate 3D maps with fly-through motion. However, owing to the unique image formulation principle, estimating 3D information from a single image faces severe ambiguity problems. Classical methods of multi-view stereo can avoid the ambiguity problems, but may require a large number of viewpoints to generate an accurate model. In this work, we propose a novel learning-based multi-view stereo method to estimate 3D information. To better utilize the information from multiple frames, an elevation plane sweeping method is proposed to generate the depth-azimuth-elevation cost volume. The volume after regularization can be considered as a probabilistic volumetric representation of the target. Instead of performing regression on the elevation angles, we use pseudo front depth from the cost volume to represent the 3D information which can avoid the 2D-3D problem in acoustic imaging. High-accuracy results can be generated with only two or three images. Synthetic datasets were generated to simulate various underwater targets. We also built the first real dataset with accurate ground truth in a large scale water tank. Experimental results demonstrate the superiority of our method, compared to other state-of-the-art methods.

We detail a novel class of implicit neural models. Leveraging time-parallel methods for differential equations, Multiple Shooting Layers (MSLs) seek solutions of initial value problems via parallelizable root-finding algorithms. MSLs broadly serve as drop-in replacements for neural ordinary differential equations (Neural ODEs) with improved efficiency in number of function evaluations (NFEs) and wall-clock inference time. We develop the algorithmic framework of MSLs, analyzing the different choices of solution methods from a theoretical and computational perspective. MSLs are showcased in long horizon optimal control of ODEs and PDEs and as latent models for sequence generation. Finally, we investigate the speedups obtained through application of MSL inference in neural controlled differential equations (Neural CDEs) for time series classification of medical data.

Effective control and prediction of dynamical systems require appropriate handling of continuous-time and discrete, event-triggered processes. Stochastic hybrid systems (SHSs), common across engineering domains, provide a formalism for dynamical systems subject to discrete, possibly stochastic, state jumps and multi-modal continuous-time flows. Despite the versatility and importance of SHSs across applications, a general procedure for the explicit learning of both discrete events and multi-mode continuous dynamics remains an open problem. This work introduces Neural Hybrid Automata (NHAs), a recipe for learning SHS dynamics without a priori knowledge on the number, mode parameters, and inter-modal transition dynamics. NHAs provide a systematic inference method based on normalizing flows, neural differential equations, and self-supervision. We showcase NHAs on several tasks, including mode recovery and flow learning in systems with stochastic transitions, and end-to-end learning of hierarchical robot controllers.

In this paper, we propose a novel method for 3D measurement of large structures that have sparse features. The proposed method uses a structured-light approach based on a spherical camera and an omnidirectional ring laser. The spherical camera can capture omnidirectional images which enable it to view all sparse feature points existing in the target environment. The omnidirectional ring laser can enable dense 3D measurement of cross-sections of the environment via the structured light method. Structure from Motion (SfM) is used to measure the motion of the camera to integrate the laser cross sections to obtain a dense 3D model.However, the result of SfM does not contain real-world scale information. The novelty of this research lies in a new method to obtain the real-world scale. The real-world scale is determined by comparing a mesh generated from the resultant SfM point cloud and the integrated laser sections in terms of each scale.In a simulated environment, the proposed method was found to be accurate up to 1 mm. It was also able to accurately measure the 3D shape of a real environment.

In this study, we present a method for all-around depth estimation from multiple omnidirectional images for indoor environments. In particular, we focus on plane-sweeping stereo as the method for depth estimation from the images. We propose a new icosahedron-based representation and ConvNets for omnidirectional images, which we name "CrownConv" because the representation resembles a crown made of origami. CrownConv can be applied to both fisheye images and equirect- angular images to extract features. Furthermore, we propose icosahedron-based spherical sweeping for generating the cost volume on an icosahedron from the extracted features. The cost volume is regularized using the three-dimensional CrownConv, and the final depth is obtained by depth regression from the cost volume. Our proposed method is robust to camera alignments by using the extrinsic camera parameters; therefore, it can achieve precise depth estimation even when the camera alignment differs from that in the training dataset. We evaluate the proposed model on synthetic datasets and demonstrate its effectiveness. As our proposed method is computationally efficient, the depth is estimated from four fisheye images in less than a second using a laptop with a GPU. Therefore, it is suitable for real-world robotics applications. Our source code is available at https://github.com/matsuren/crownconv360depth.

Continuous deep learning architectures have recently re-emerged as Neural Ordinary Differential Equations (Neural ODEs). This infinite-depth approach theoretically bridges the gap between deep learning and dynamical systems, offering a novel perspective. However, deciphering the inner working of these models is still an open challenge, as most applications apply them as generic black-box modules. In this work we ``open the box'', further developing the continuous-depth formulation with the aim of clarifying the influence of several design choices on the underlying dynamics.

The infinite-depth paradigm pioneered by Neural ODEs has launched a renaissance in the search for novel dynamical system-inspired deep learning primitives; however, their utilization in problems of non-trivial size has often proved impossible due to poor computational scalability. This work paves the way for scalable Neural ODEs with time-to-prediction comparable to traditional discrete networks. We introduce hypersolvers, neural networks designed to solve ODEs with low overhead and theoretical guarantees on accuracy. The synergistic combination of hypersolvers and Neural ODEs allows for cheap inference and unlocks a new frontier for practical application of continuous-depth models. Experimental evaluations on standard benchmarks, such as sampling for continuous normalizing flows, reveal consistent pareto efficiency over classical numerical methods.

Spherical cameras, which can acquire all-round information, are effective to estimate rotation for robotic applications. Recently, Convolutional Neural Networks have shown great robustness in solving such regression problems. However they are designed for planar images and cannot deal with the non-uniform distortion present in spherical images, when expressed in the planar equirectangular projection. This can lower the accuracy of motion estimation. In this research, we propose an Equirectangular-Convolutional Neural Network (E-CNN) to solve this issue. This novel network regresses 3D spherical camera rotation by uniformizing distorted optical flow patterns in the equirectangular projection. We experimentally show that this results in consistently lower error as opposed to learning from the distorted optical flow.

Comparing to perspective cameras, the all-round 3D measurement of the environment can be done by spherical cameras in a more efficient way. However, the measurement using binocular spherical stereo has two singularity points at the epipoles of each spherical camera, where the measurement result gets extremely sensitive to the error when getting close to the epipoles and along the epipolar directions. This affects the accuracy of 3D reconstruction along with the epipolar directions. A three-way measurement method using three spherical cameras with trinocular spherical stereo setup is proposed in this paper to achieve accurate all-round 3D measurement. The improved accuracy of 3D measurement by the implementation of weighted reprojection error optimization was verified in experiments.

In this paper, we propose a novel motion planning method for off-road unmanned ground vehicles, based on three-dimensional (3-D) terrain map information. Previous studies on the motion planning of a vehicle traveling on rough terrain dealt only with a relatively small environment. Furthermore, unique vehicle characteristics were not considered, and it was also impossible to incorporate regulations, such as maintaining driving speed and suppressing posture change. The proposed method enables vehicles to adaptively generate a path by considering vehicle characteristics and the regulations, in a large-scale environment, with rough terrain. A random sampling based scheme was applied to carrying out global path planning, based on a 3-D environmental model. Experimental results showed that the proposed off-road motion planner could generate an appropriate path, which satisfies vehicle characteristics and predefined regulations.

This paper presents a dense method of real-time registration of RGB-D image pair. So far, we have proposed the “see-through system”, in which multiple images acquired from RGB-D sensors are integrated to present images that is useful for confirming the shape or the positions of objects behind obstacles. However, errors of positional relation of sensors result in see-through images in which some objects are doubled or appear at incorrect positions. It is difficult to align the images because they are captured from distant viewpoints and there are few shared field of view. In the proposed method, positional relation of two RGB-D sensors is corrected using a new IRLS framework, fast and robust minimization strategy.

Conventional techniques for frame-to-frame camera motion estimation rely on tracking a set of sparse feature points. However, images taken from spherical cameras have high distortion which can induce mistakes in feature point tracking, offsetting the advantage of their large fields-of-view. Hence, in this research, we attempt a novel approach of using dense optical flow for distortion-robust spherical camera motion estimation. Dense optical flow incorporates smoothing terms and is free of local outliers. It encodes the camera motion as well as dense 3D information. Our approach decomposes dense optical flow into epipolar geometry and the dense disparity map, and reprojects this disparity map to estimate 6 DoF camera motion. The approach handles spherical image distortion in a natural way. We experimentally demonstrate its accuracy and robustness.

Localization is an important task for mobile service robots in indoor spaces. In this research, we propose a novel technique for indoor localization using a spherical camera. Spherical cameras can obtain a complete view of the surroundings allowing the use of global environmental information. We take advantage of this in order to estimate camera position and the orientation with respect to a known 3D line map of an indoor environment, using a single image. We robustly extract 2D line information from the spherical image via spherical-gradient filtering and match it to 3D line information in the line map. Our method requires no information about the 3D-2D line correspondences. In order to avoid a complicated six degrees of freedom (6 DoF) search for position and orientation, we use a Manhattan world assumption to decompose the line information in the image. The 6 DoF localization process is divided into two phases. First, we estimate the orientation by extracting the three principle directions from the image. Then, the position is estimated by robustly matching the distribution of lines between the image and the 3D model via a spherical Hough representation. This decoupled search can robustly localize a spherical camera using a single image, as we demonstrate experimentally.

In camera-equipped teleoperated robots, it is often tedious for the operator to manage both the viewpoint and the shaky/unstable navigation, leading to disorientation. Our proposal is to create a virtual, freely rotatable camera that is decoupled from the robot's rotation. It is implemented using a complete spherical camera and removing its rotation in-image with a novel algorithm based on aligning the dense spherical optical flow field along the epipolar direction. Finally, any area on the rotation-less image sequence can be undistorted, resulting in the desired decoupled camera. We illustrate the concept by showing the effect on some videos taken from a spherical camera under different robot motions.

This paper introduces a new methodology of robotic hammering inspection for the maintenance of social infrastructures. In particular, the estimation of material defect conditions, such as delamination depth of concrete, is focused upon. Development of an automated diagnosis methodology is necessary for the maintenance of superannuated social infrastructures. The hammering test, which is an efficient inspection method, has attracted considerable attention in the context of automated inspection using robots. In this study, to apply the hammering test to robotic inspection, in which material conditions of infrastructures must be diagnosed in detail, an estimation method of the defect conditions is proposed, and an integration technique of plural classifiers for improving the inspection accuracy is introduced. Furthermore, an inspection system that can decrease the influence of the mechanical running-noise is implemented. Our experimental results using concrete test pieces demonstrate the effectiveness of the proposed method; the accuracy of the defect detection and defect condition estimation was validated.

Indoor robot localization systems using wireless signal measurements have gained popularity in recent years, as wireless Local Area Networks can be found practically everywhere. In this field, a popular approach is the use of fingerprinting techniques, such as Gaussian Processes. In our approach, we improve Gaussian Processes based mapping using path loss models as priors. Path loss models encode information regarding the signal propagation phenomena into the mapping. Our approach first fits training data to a simple path loss model, and then trains a zero-mean Gaussian Process with the mismatches between the models and the data. Signal strength mean predictions are done using both the path loss model and the Gaussian Process output, while variances are calculated by bounding the Gaussian Process variance using the path loss models. Notably, the main improvement generated by our approach is not an enhanced mean value prediction, but rather a better model variance prediction. This translates into better likelihood estimations, leading to higher localization accuracy. Experiments using data acquired in an indoor environment and our approach as the perceptual likelihood of a dual Monte Carlo localization algorithm are used to demonstrate this improvement. Furthermore, this idea can be extrapolated to other fingerprinting techniques and to applications other than wireless-based localization.

We propose a prediction method for lane changes in other vehicles. According to previous research, over 90 % of car crashes are caused by human mistakes, and lane changes are the main factor. Therefore, if an intelligent system can predict a lane change and alarm a driver before another vehicle crosses the center line, this can contribute to reducing the accident rate. The main contribution of this work is to propose a new feature describing the relationship of a vehicle to adjacent vehicles. We represent the new feature using a dynamic characteristic potential field that changes the distribution depending on the relative number of adjacent vehicles. The new feature addresses numerous situations in which lane changes are made. Adding the new feature can be expected to improve prediction performance. We trained the prediction model and evaluated the performance using a real traffic dataset with over 900 lane changes, and we confirmed that the proposed method outperforms previous methods in terms of both accuracy and prediction time.

It is important to understand the mechanism of human standing-up motion to improve the declined physical ability of the elderly people. This study employs the concept of muscle synergies (modular structure of coordinative muscle activation) to understand how humans coordinate their muscles to achieve the standing-up motion. Neuro-musculoskeletal model was developed to represent human body to generate standing-up motion. Using the developed model, forward dynamic simulation was used to analyze how humans utilized the muscle synergies to realize the motion. Results showed that the developed model could generate the standing-up motion with four muscle synergies rather than controlling individual muscles. Moreover, further analysis showed that three different strategies of the standing-up motion could be generated only by changing the start time of the particular muscle synergy.

Three-dimensional (3D) measurement is an important means for robots to acquire information about their environment. Structure from Motion is one of these 3D measurement methods. The 3D reconstruction of objects in the environment can be obtained from pictures captured with single camera in Structure from Motion. Furthermore, the camera motion can be obtained simultaneously. Because of its simplicity, Structure from Motion has been implemented in various ways. However there is an essential problem in that the scale of the measured objects cannot be computed by Structure from Motion. In order to compute the absolute scale, other information is required. However this is difficult for robots in an unknown situation. In this paper, we propose a method that can reconstruct the absolute scale of objects using refraction. Refraction changes the light ray path between the objects and the camera. This method is implemented using only a refractive plate and single camera. The results of simulations show the effectiveness of the proposed method in both air and other media (e.g., water).

We present a novel rough terrain traversability analysis method for mobile robot navigation. We focused on the scenario of mobile robot operation in a disaster environment with limited sensor data. The robot builds a map in real time and analyzes the terrain traversability of its surrounding environment. The proposed method is based on fuzzy inference so that it can handle uncertainties in the sensor data. Two values associated with the terrain traversability, roughness and slope, are calculated from an elevation map built by a laser range finder mounted on the mobile robot. These two values are inputted to the fuzzy inference, and the traversability is analyzed. Based on the traversability output from the fuzzy inference, a vector field histogram (VFH) is generated. The mobile robot course is determined according to the VFH. We demonstrated our algorithm on an artificial environment. The experimental results showed that the mobile robot was able to reach the target position safely while avoiding untraversable areas.

An automated diagnosis methodology is necessary for the maintenance of superannuated social infrastructures. In this context, the hammering test is an efficient inspection method, and it has been widely used because of the resulting accuracy and efficiency of operation. While robotic automation of the hammering inspection method is highly desirable, the development of an automatic diagnostic algorithm that can operate at actual inspection sites is essential. Furthermore, portability of the diagnostic algorithm is also highly desirable. In this study, in order to construct reliable detectors and to improve their portability for the performance of the hammering test, we propose a boosting-based defect detector that is robust against variations in environmental conditions. In particular, we present the construction of a noise-robust classifier with a refinement of the feature values extracted from hammering sounds and an updating rule of template vectors of its evaluation function. Our experimental results in a concrete tunnel demonstrate the effectiveness of the proposed method; the accuracy of the classifier at an actual site and adaptivity to environmental noise are confirmed.

The standing-up motion is one of the most important activities of daily livings. In order to understand the strategy to achieve the standing-up motion, muscle synergy analysis is applied to the measured data during human standing-up motion. In addition, musculoskeletal model which consists of three body segments and nine muscles in lower limb is developed to ensure that the standing-up motion can be generated by muscle synergies. As a result, three muscle synergies have been extracted from the human standing-up motion, and each synergy strongly corresponded to characteristic kinematic events: momentum flexion, momentum transfer, and posture stabilization. Results of forward dynamic simulation show that the standing-up motion can be achieved by controlling time-varying weighting coefficient of three muscle synergies instead of controlling individual nine muscles.

In this paper, novel reinforcement learning method with intrinsic motivation for reproducibility of the past successful experience is presented. The experience is extracted as skill, which is composed of action sequence and abstract knowledge about observed sensor input. Utilizing the collected skills, reproduction of the successful experience is attempted in novel and unknown environment. Consistent exploration and active reduction of search space are realized by learning with intrinsic motivation for reproducibility of experience. Simulation experiments in grid world demonstrate that proposed method significantly accelerate speed of learning.

Although standing-up motion is an important activity of daily living, it remains unclear how people perform the motion in different situations. As described in this paper, muscle synergy analysis is applied to standing-up motions performed at different circumstances, such as two different heights and at three different speeds. Results elucidated three invariant groups of synchronized muscle activations: The first synergy pulls the ankle and raises the hip. The second synergy extends the upper body. The third synergy stabilizes posture. Results also show that people controlled the activation coefficient of each synergy differently during all motions. The slower the standing-up motion is, the longer each synergy activates to adapt to the slower motion speed. Results of this study show that people use the same group of synchronized muscle activation and only control the activation coefficient to achieve adaptive standing-up motion.

As the population continues to age, the number of elderly people requiring healthcare is increasing. In order to improve their physical function, they need to get physical training. There are activities which require integrated arm movements and lower body movements. However there are no quantitative testing methods of the degree of recovery for the coordination between arm movements and lower body movements. In this study, we focus on dart throwing motion as arm movements in lower body movements and suggest the quantitative evaluation of the coordination between arm movements and lower body movements in dart throwing motion. Normalized correlation coefficient (NCC) between arm and lower body was computed at different throwing distances. In addition the standard deviation of the NCC was computed in order to investigate the stability of the joint correlation evaluation. This analysis shows that the correlation between elbow and ankle, or between elbow and knee, are increased at throwing long distance. We suggest that the NCC between elbow angle and right knee angle may be used for the evaluation of the joint correlation between arm movements and lower body movements in dart throwing motion.

In this paper, we propose a 3-D measurement method of underwater objects using a fish-eye stereo camera. Sensing in aquatic environments is important to maintain underwater structures and research aquatic lives. A 3-D measurement method of objects in water using a fish-eye stereo camera enables underwater robots to execute its task safely with a wide field of view. However, images taken by fish-eye camera have large distortion following projection model. Also, sensing in aquatic environments meets the difficulty that image distortion occurs by refraction of light due to the difference of refractive indices of air, watertight container and water. As to these problems, we introduce a method to correct fish-eye image and a ray tracing method to remove the effect of refraction. Experimental results show the effectiveness of the proposed method.

In this paper, we propose an efficient estimation method of an omnidirectional camera movement. The proposed method is based on Structure from Motion utilizing a constraint of parallel lines. In an environment having manmade structures, parallel lines can be extracted from an omnidirectional image easily and constantly, because of its wide field of view. Parallel lines provides a valuable constraint for camera movement estimation. The proposed method can estimate 3-D camera movements by solving one degree of freedom problem three times without regard to the number of viewpoints. Experimental results show the effectiveness of our proposed method.

View occluders influence the image quality of a subject when occluders exist between a camera and a subject. For example, a blurred fence image interrupts a subject when an image of a scene is captured by a camera through a fence. In this paper, we propose an automatic removal method of foreground occluders from images using multiple focusing. Our method automatically detects foreground occluder regions by using two images with and without flashlight. The influence of foreground occluders is estimated and overlapping effects of foreground occluders are removed by using multiple focus images. Experimental results show the effectiveness of the proposed method.

In order for robots to safely move in human-robot coexisting environment, they must be able to predict their surrounding people's behavior. In this study, a pedestrian behavior model that produces humanlike behavior was developed. The model takes into account the pedestrian's intention. Based on the intention, the model pedestrian sets its subgoal and moves toward the subgoal according to virtual forces affected by other pedestrian and environment. The proposed model was verified through pedestrian observation experiments.

Recently healthcare of the elderly people has become a serious issue in medical and rehabilitation areas. In order to know their functional mobility and provide sufficient medical treatment, it is important to measure their body state precisely and objectively. Therefore we developed a wearable and wireless sensor of gyroscope and accelerometer (PocketIMU2) as an easy and precise measurement of human motions. In the sensor, we employed a small and high accurate LiNbO3 crystal to achieve joint angle computation with simple integration of angular velocity. In the current paper, we evaluate the accuracy of the sensor in two important basic motion, such as a walking and sit-to-stand motions. Computed joint angles of ankle, knee, and hip are compared to the reference data measured from a optical motion capture system in term of coefficients of correlation and root mean square error. As a result, coefficient of correlation showed very high value for all joint angles, and root mean square error was adequately small. This strongly supports the usage of our developed gyroscope and accelerometer sensor for monitoring human body movement for medical usage.

In this paper, we propose an assisting system of touch panel operation for people with visual disability. In the system, a user specifies the target button on the panel by verbal input. The system detects the button and user's fingertip by analyzing images obtained through stereo camera. Navigation is made by indicating the direction of the fingertip on the panel through headphones with sound. To construct an efficient navigation method, comparisons were made experimentally concerning to indication of the finger motion direction, choice of navigation sound, and indication of the distance. The effectiveness of the proposed method was verified through experiments.

When an image of a scene is captured by a camera through a fence, a blurred fence image interrupts objects in the scene. In this paper, we propose a method for a fence removal from the image using multiple focusing. Most of previous methods interpolate the interrupted regions by using information of surrounding textures. However, these methods fail when information of surrounding textures is not rich. On the other hand, there are methods that acquire multiple images for image restoration and composite them to generate a new clear image. The latter approach is adopted because it is robust and accurate. Multi-focus images are acquired and "defocusing'' information is utilized to generate a clear image. Experimental results show the effectiveness of the proposed method.

A lot of plumbings such as gas pipes and water pipes exist in public utilities, factories, power plants and so on. It is difficult for humans to inspect them directly because they are long and narrow. Therefore, automated inspection by robots equipped with camera is desirable, and great efforts have been done to solve this problem. However, many of existing inspection robots have to rotate the camera to record images in piping because a conventional camera with a narrow view can see only one direction while piping has a cylindrical geometry. The use of an omni-directional camera that can take images of 360° in surroundings at a time is effective for the solution of the problem. However, the shape measurement is difficult only with the omni-directional camera. Then, in this paper, we propose a reconstruction method of piping shape by using an omni-directional camera and an omni-directional laser with a light section method and a structure from motion analysis. The validity of the proposed method is shown through experiments.

In this paper, we propose a view planning method that plans view points for efficient 3D map building by a mobile robot equipped with two LRFs (laser range finders). The robot searches for effective view points by predicting new measurement domains in the next view points, and then measures distances to obstacles around the robot with LRFs. The 3D map is generated by integrating range information obtained from multiple measurements. Experimental results show the effectiveness of the proposed method.

Sensing in aquatic environments meets the difficulty that, when a camera is set in air behind a watertight glass plate, image distortion occurs by refraction of light at the boundary surface between air and water. This paper proposes an aquatic sensing method to execute three dimensional measurement of objects in water by taking refraction of light into account. The proposed method is based on a monocular stereo technique using a difference of appearance depending on optical paths. Optimization of the angles of refracting surfaces which are key elements of the sensing device is given to realize the accurate measurement. Experimental results show the effectiveness of the proposed method.

In this paper, a new method for 3-D measurement of objects in water is proposed. When observing objects in water through a camera contained in a waterproof housing or observing objects in an aquarium tank filled with preserving liquid, we should solve a problem of light refraction at the boundary surfaces of refractive index discontinuity which gives image distortion. The proposed method uses a space encoding method which does not have a problem of corresponding point detection as a stereo vision system has, and is faster than spot light projection or slit light projection methods. A ray tracing technique solves the problem of image distortion caused by refractive index discontinuity. It should be noted that monochromatic light projection onto objects gives more accurate measurement than white light projection because the refractive index depends on the wavelength of the light. Then, in order to measure colored objects, we should project red, green and blue light patterns onto them separately. Experimental results show the validity of the proposed method.

Map information is important for path planning and self-localization when mobile robots accomplish autonomous tasks. In unknown environments, mobile robots should generate an environment map by themselves. Then, we propose a method for 3D environment modeling by a mobile robot. For environmental measurement, we use a single omni-directional camera. We propose a new estimation method of camera motions for improvement in measurement robustness and accuracy. The method takes advantage of a wide field of view of an omni-directional camera. Experimental results showed the effectiveness of the proposed method.

This paper describes a method for removing adherent noises from image sequences. In outdoor environments, it is often the case that scenes taken by a camera are deteriorated because of adherent noises such as waterdrops on the surface of the lens-protecting glass of the camera. To solve this problem, our method takes advantage of image sequences captured with a moving camera whose motion is unknown. Our method estimates a camera motion only from image sequences, and makes a spatio-temporal image to extract the regions of adherent noises by examining differences of track slopes in cross section images between adherent noises and other objects. Finally, regions of noises are eliminated by replacing with image data corresponding to object regions. Experimental results show the effectiveness of our method.

In this paper, we propose a region extraction method using chromakey with a two-tone checker pattern background. The proposed method solves the problem in conventional chromakey techniques that foreground objects become transparent if they have the same color with the background. The adjacency condition between two-tone regions of the background and the geometrical information of the background grid lines are utilized for extracting foreground objects. Experimental results show the effectiveness of the proposed method.

In this paper, we propose a new three-dimensional (3-D) measurement method of objects in unknown liquid with a stereo vision system. When applying vision sensors to measuring objects in liquid, light refraction is an important problem. Therefore, we estimate refractive indices of unknown liquids by using images of water surface, restore images that are free from refractive effects of the light, and measure 3-D shapes of objects in liquids in consideration of refractive effects. The effectiveness of the proposed method is shown through experiments.

This paper describes a method for removing adherent noises from image sequences. In outdoor environments, it is often the case that scenes taken by a camera are deteriorated because of adherent noises such as waterdrops on the surface of the lens-protecting glass of the camera. To solve this problem, our method takes advantage of image sequences captured with a moving camera. The method makes a spatio-temporal image to extract the regions of adherent noises by examining differences of track slopes in cross section images between adherent noises and other objects. Finally, regions of noises are eliminated by replacing with image data corresponding to object regions. Experimental results show the effectiveness of our method.

Colors of objects observed in underwater environments are different from those in air. This is because the light intensity decreases with the distance from objects in water by light attenuation. Robots on the ground or in air usually recognize surrounding environments by using images acquired with cameras. The same is/will be true of underwater robots. However, recognition methods in air based on image processing techniques may become invalid in water because of light attenuation. Therefore, we propose a color registration method of underwater images. The proposed method estimates underwater environments where images are acquired, in other words, parameters essential to color registration, by using more than two images. After estimating parameters, color registration is executed with consideration of light attenuation. The effectiveness of the proposed method is verified through experiments.

In this paper, we propose a new method for reducing assembly time in printed circuit board (PCB) assembly by prioritizing efficient simultaneous pickup operation of placement machines. Despite using the same placement machine, the efficiency of the schedule results in major different assembly time. Therefore, it is important to optimize the scheduling of component mounting. There are three major problems of the scheduling: (1) component feeder location (affects efficiency of pickup operation), (2) mounting sequencing (affects total distance of the mounting tour) , and (3) simultaneous pickup (affects efficiency of pickup operation). To solve these problems, this paper proposes the following approaches. We solve (1) and (3) in a heuristic way by using a random multi-start local search. We solve (2) greedily with putting the result of the feeder array to effective use. The effectiveness of the proposed method was shown through simulations.

In this paper, we propose robust sensing method against bubble noises in aquatic environments with a stereo vision system. Usually, three-dimensional (3-D) measurement by robot vision techniques is executed under the assumptions that cameras and objects are in aerial environments. However, an image distortion occurs when vision sensors measure objects in liquid. It is caused by the refraction of the light on the boundary between the air and the liquid, and the distorted image brings errors in a triangulation for the range measurement. Additionally, it is often the case that there exist air bubbles in the field of view when we observe aquatic environments. Therefore, it becomes difficult to acquire clear images because of these view-disturbing noises. As to the former problem, accurate 3-D coordinates of objects' surfaces in liquid are measured by taking for calculating the refraction effect. As to the latter problem, bubble noises are eliminated from a moving image to divide objects in the image into still backgrounds, moving objects, and bubble noise by an image processing technique. Experimental results showed the effectiveness of our proposed method

In this paper, we propose a new method that can inspect visual and non-visual features of objects simultaneously by using image and sound signal processing techniques. A method for discriminating a property of an object with the use of generated sound when striking it with a hammer is called a hammering test. This method can investigate non-visual features of objects such as inner structure of objects, e.g., the existence of defects and cracks inside objects. However, this method depends on human experience and skills. In addition, if we perform this test over a wide area of objects, it is required to manually record hammering positions one by one. To solve these problems, this paper proposes a hammering test system consisting of two video cameras that can acquire image and sound signals of a hammering scene. The shape of the object (visual feature) is measured by the image signal processing from the result of 3-D measurement of each hammering position, and the thickness or material (non-visual feature) is estimated by the sound signal processing in time and frequency domains. The validity of proposed method is shown through experiments

In this paper, we propose a three-dimensional (3-D) measurement method of objects in aquatic environments whose refractive indices or boundary shape of the refraction are unknown with a laser range finder. When applying vision sensors to measuring objects in liquid, we meet the problem of an image distortion. It is caused by the refraction of the light on the boundary between the air and the liquid, and the distorted image brings errors in a triangulation for the range measurement. Therefore, 3-D measurement of objects in liquid requires a geometrical analysis that takes refraction effects into account. In the analysis, it is indispensable to know the boundary shapes that have discontinuity of refractive indices, and refractive index of liquid. Our proposed method takes refraction effects into account and estimates these unknown parameters to measure accurate 3-D shapes of objects in liquid. Experimental results have shown the effectiveness of the proposed method.

In this paper, we propose a new method that can remove view-disturbing noises from stereo images. One of the thorny problems in outdoor surveillance by a camera is that adherent noises such as waterdrops on the protecting glass surface lens disturb the view from the camera. Therefore, we propose a method for removing adherent noises from stereo images taken with a stereo camera system. Our method is based on the stereo measurement and utilizes disparities between stereo image pair. Positions of noises in images can be detected by comparing disparities measured from stereo images with the distance between the stereo camera system and the glass surface. True disparities of image regions hidden by noises can be estimated from the property that disparities are generally similar with those around noises. Finally, we can remove noises from images by replacing the above regions with textures of corresponding image regions obtained by the disparity referring. Experimental results show the effectiveness of the proposed method.

In this paper, we propose a three-dimensional (3D) measurement method of objects' shapes in liquid by using the light stripe projection method. Usually, 3D measurement by robot vision techniques is executed under the assumptions that cameras and objects are in aerial environments. However, an image distortion occurs when vision sensors measure objects in liquid. It is caused by the refraction of the light on the boundary between the air and the liquid, and the distorted image brings errors in a triangulation for the range measurement. Our proposed method can measure accurate 3D coordinates of objects' surfaces in liquid taken for calculating the refraction effect. The effectiveness of the proposed method is shown through experiments. By considering the refraction of the light, the accuracy of the 3D measurement of objects in water becomes same as that when there is no water, although the accuracy is bad when the refraction of the light is not considered. The accuracy of the 3D measurement is about 1 mm for objects located about 400 mm from the laser range finder. The measurement speed can be also reduced as compared with the case of using a spot laser beam.

In this paper, we propose a new method that can remove view-disturbing waterdrops from images by processing images taken with a pan-tilt camera system. In rainy days, it is often the case that images taken by the camera are hard to see because of adherent waterdrops on the surface of the protecting glass of the camera. In our method, an image of a distant prospect is taken at first and another image is taken after changing the direction of eyeshot. The new image is transformed with the projective transformation and compared with the first one to detect the region where waterdrops may exist. We can distinguish which image portion belongs to waterdrops by considering the distance between two waterdrop candidate regions. Finally, the region where waterdrops exist can be eliminated to merge two images. Experimental results show the effectiveness of the proposed method.

In this paper, we propose a new path and viewpoint planning method for a mobile robot with multiple observation strategies. When a mobile robot works in the constructed environments such as indoor, it is very effective and reasonable to attach landmarks on the environment for the vision-based navigation. In that case, it is important for the robot to decide its motion automatically. Therefore, we propose a motion planning method that optimizes the efficiency of the task, the danger of colliding with obstacles, and the accuracy and the ease of the observation according to the situation and the performance of the robots. We also introduce multiple landmark-observation strategies to optimize the motion depending on the number and the configuration of visible landmarks in each place.

In this paper, we propose a new method that can remove view-disturbing noises from images of dynamic scenes. One of the thorny problems in outdoor surveillance by a camera is that adherent noises such as waterdrops or mud blobs on the protecting glass surface lens disturb the view from the camera. Therefore, we propose a method for removing adherent noises from images of dynamic scenes taken by changing the direction of a pan-tilt camera, which is often used for surveillance. Our method is based on the comparison of two images, a reference image and a second image taken by a different camera angle. The latter image is transformed by a projective transformation and subtracted from the reference image to extract the regions of adherent noises and moving objects. The regions of adherent noises in the reference image are identified by examining the shapes and distances of regions existing in the subtracted image. Finally, regions of adherent noises can be eliminated by merging two images. Experimental results show the effectiveness of our proposed method.

In this paper, we propose a three-dimensional (3-D) measurement method of objects in liquid with a laser range finder. When applying vision sensors to measuring objects in liquid, we meet the problem of an image distortion. It is caused by the refraction of the light on the boundary between the air and the liquid, and the distorted image brings errors in a triangulation for the range measurement. Our proposed method can measure the accurate 3-D coordinates of objects surfaces in liquid taken for calculating the refraction effect. The effectiveness of the proposed technique is shown through experiments. The accuracy of the 3-D measurement is 0.7 mm for objects located about 250 mm from the laser range finder when considering the refraction of the light, although that is 2.9 mm without the consideration of it.

In this paper, we propose a new method for the restoration of deteriorated images by using multiple cameras. In outdoor environment, it is often the case that scenes taken by the cameras are hard to see because of adherent noises on the surface of the lens-protecting glass of the cameras. Our proposed method analyses multiple camera images describing the same scene, and synthesizes an image in which adherent noises are eliminated.

In this paper, we propose a motion-planning method of multiple mobile robots for cooperative transportation of a large object in a three-dimensional environment. This task has various kinds of problems, such as obstacle avoidance and stable manipulation. All of these problems cannot be solved at once, since it would result in a dramatic increase of the computational time. Accordingly, we divided the motion planner into a global path planner and a local manipulation planner, designed them, and integrated them. The aim was to integrate a gross motion planner and a fine motion planner. Concerning the global path planner, we reduced the dimensions of the configuration space (C-space) using the feature of transportation by mobile robots. We used the potential field to find the solution by searching in this smaller-dimension reconstructed C-space. In the global path planner, the constraints of the object manipulation are considered as the cost function and the heuristic function in the A/sup */ search. For the local manipulation planner, we developed a manipulation technique, which is suitable for mobile robots by position control. We computed the conditions in which the object becomes unstable during manipulation and generated each robot's motion, considering the robots' motion errors and indefinite factors from the planning stage. We verified the effectiveness of our proposed motion planning method through simulations.

We propose a motion planning method for cooperative transportation of a large object by multiple mobile robots in a 3 dimensional environment. This task has various kinds of problems, such as path planning, manipulation and so on. All of these problems can't be solved at once, since computational time is exploded. Accordingly, we divide a motion planner into a local manipulation planner and a global path (motion) planner, and design these two planners respectively, and we integrate two planners. Namely, we aim at integrating a gross motion planner and a fine motion planner. As to the local manipulation planner, we build a manipulation technique, which is suitable for mobile robots by position control. We compute conditions, in which the object becomes unstable during manipulation, and generate each robot's motion considering the robots' motion errors and indefinite factors from the planning stage. As to the global path planner, we reduce the dimensions of the configuration space (C-space) using the feature of transportation by mobile robots. We can find a solution with searching in this smaller dimensional C-space using the potential field defined in the C-space, and constraints of the object manipulation are considered as the potential function. We verify the effectiveness of our proposed motion planning method through simulations and experiments.

In this paper, we propose a method of an object manipulation by multiple mobile robots using sticks as tools. In the conventional cooperative work by multiple mobile robots, manipulation technique based on force-control has been proposed. However, mobile robots are moving by position-control, and motion errors can easily arise. Then, we build the manipulation technique, which is suitable for mobile robots by position-control. We propose the manipulation method without using sensor information, and consider the motion errors of mobile robots and the indefinite element of environment from the planning stage. We compute the conditions in which the object gets unstable during manipulation, and generate the motion of each mobile robot with these analyses. We verify the effectiveness of our proposed motion planning method through simulations and experiments.

We are going to analyze on the constraint among contacting points between robots and an object that is manipulated by the robot group. Constraint can be obtained by solving pseudo inverse of matrix filled by force data measured by all robots supporting one object. The result provides additional information for obtaining a relative positioning information to the object to ordinary constraint based on geometrical reasons. This analysis on the constraint has been verified by sensing system. The model explained here can be used for estimating contacting points, fitting coordinate system among robots, and making active sensing strategy for detecting the change of environment.

In transporting various objects by multiple mobile robots we need to change the formation of robots and posture of an object. This planning depends on the mass and center of gravity of an object. Here we discuss the accuracy of the estimation of the mass and the position of the center of gravity of an object. Particularly we propose a method for estimating the height of the center of gravity which cannot be estimated by only one measurement. Validity of this strategy is verified by means of a real robot system.