Building LiDAR generative models holds promise as powerful data priors for restoration, scene manipulation, and scalable simulation in autonomous mobile robots. In recent years, approaches using diffusion models have emerged, significantly improving training stability and generation quality. Despite their success, diffusion models require numerous iterations of running neural networks to generate high-quality samples, making the increasing computational cost a potential barrier for robotics applications. To address this challenge, this paper presents R2Flow, a fast and high-fidelity generative model for LiDAR data. Our method is based on rectified flows that learn straight trajectories, simulating data generation with significantly fewer sampling steps compared to diffusion models. We also propose an efficient Transformer-based model architecture for processing the image representation of LiDAR range and reflectance measurements. Our experiments on unconditional LiDAR data generation using the KITTI-360 dataset demonstrate the effectiveness of our approach in terms of both efficiency and quality.

As global efforts to explore the Moon intensify, the need for high-quality 3D lunar maps becomes increasingly critical—particularly for long-distance missions such as NASA’s Endurance mission concept, in which a rover aims to traverse 2,000 km across the South Pole–Aitken basin. Kaguya TC (Terrain Camera) images, though globally available at 10 m/pixel, suffer from altitude inaccuracies caused by stereo matching errors and JPEG-based compression artifacts. This paper presents a method to improve the quality of 3D maps generated from Kaguya TC images, focusing on mitigating the effects of compression-induced noise in disparity maps. We analyze the compression behavior of Kaguya TC imagery, and identify systematic disparity noise patterns, especially in darker regions. In this paper, we propose an approach to enhance 3D map quality by reducing residual noise in disparity images derived from compressed images. Our experimental results show that the proposed approach effectively reduces elevation noise, enhancing the safety and reliability of terrain data for future lunar missions.

3D LiDAR sensors are indispensable for the robust vision of autonomous mobile robots. However, deploying LiDAR-based perception algorithms often fails due to a domain gap from the training environment, such as inconsistent angular resolution and missing properties. Existing studies have tackled the issue by learning inter-domain mapping, while the transferability is constrained by the training configuration and the training is susceptible to peculiar lossy noises called ray-drop. To address the issue, this paper proposes a generative model of LiDAR range images applicable to the data-level domain transfer. Motivated by the fact that LiDAR measurement is based on point-by-point range imaging, we train an implicit image representation-based generative adversarial networks along with a differentiable ray-drop effect. We demonstrate the fidelity and diversity of our model in comparison with the point-based and image-based state-of-the-art generative models. We also showcase upsampling and restoration applications. Furthermore, we introduce a Sim2Real application for LiDAR semantic segmentation. We demonstrate that our method is effective as a realistic ray-drop simulator and outperforms state-of-the-art methods.

Terrain knowledge around unmanned ground vehicles (UGVs) is vital for autonomous navigation. Having global understanding of the surroundings of UGVs is important, although the field of view from UGVs is very limited. Thus, we utilize an aerial vehicle to provide a large terrain map from sequential aerial images. In this paper, we present multiple techniques to accelerate the process of terrain classification so that it can run onboard on the aerial platform. The main techniques used to accelerate the process is a "knowledge distillation" of a deep neural net to a shallower one, and a super pixel implementation. We evaluated our system on Jetson TX1 with actual images collected from a weather balloon which confirmed the effectiveness of the proposed system.

We aim to develop a supporting system which enhances the ability of human's short-term visual memory in an intelligent space where the human and a service robot coexist. Particularly, this paper focuses on how we can interpret and record diverse and complex life events on behalf of humans, from a multi-perspective viewpoint. We propose a novel method named "fourth-person captioning", which generates natural language descriptions by summarizing visual contexts complementarily from three types of cameras corresponding the first-, second-, and third-person viewpoint. We first extend the latest image captioning technique and design a new model to generate a sequence of words given the multiple images. Then we provide an effective training strategy that needs only annotations supervising images from a single viewpoint in a general caption dataset and unsupervised triplet instances in the intelligent space. As the three types of cameras, we select a wearable camera on the human, a robot-mounted camera, and an embedded camera, which can be defined as the first-, second-, and third-person viewpoint, respectively. We hope our work will accelerate a cross-modal interaction bridging the human's egocentric cognition and multi-perspective intelligence.

Mutual subspace method (MSM), which is one of image-based approaches, showed strong discrimination capability in gait recognition. In general, 2D image matrices are transformed into 1D image vectors to be used as input into MSM, and then principal component analysis (PCA) is applied to 1D vectors to generate a subspace. However, due to the high dimensionalities of 1D vectors, the evaluation accuracy of the covariance matrix in PCA is not high enough. This results in a decrease in performance, especially in case that speed difference between gallery and probe dataset is big. Thus in this paper we propose a method, which expands the MSM-based method, to recognize people with higher accuracy. The proposed method divides the human body area into multiple areas, followed by adaptive choice of areas that have high discrimination capability. Moreover, the proposed method utilizes the frieze pattern, which is one of gait features, as an additional input into MSM. The use of divided areas and the frieze pattern allows us to evaluate the covariance matrix with higher accuracy. In experiments we applied the proposed method to challenging databases with speed variations, and we show the effectiveness of the proposed method.

This paper introduces humanoid-oriented movement writing (HOM), focusing on a notation in which body postures allow easy visual interpretation by both humans and humanoid robots. HOM Writing, derived from Sutton Movement Writing and Shorthand [1], is a natural modality for encoding the movements that humans perform during various work activities. Beyond its use as a record of human movement, the intent is to use the writing as a modality of communicating to robots what movements to execute. Humanoid robots could directly map the key postures represented in the notation to their own postures, imitating the postures captured in the description, and calculating intermediate postures by interpolation. A motion generator would ensure the motor control needed to create continuous movements. This paper focuses on the generation of the activity movement scripts, and addresses two modalities of producing the scripts: 1) `hand-coded' by a human, using an editor, and 2) automatic extraction from video, e.g. from video-recordings of a human performing an activity. A software tool developed to allow easy script writing, the HOM Editor, is described and illustrated in hand-coding of a sequence of movements. The automatic generation of scripts from video is done using the pose estimation system by Yang and Ramanan [2], which takes an image and produces the joint coordinates of the limb parts; this is illustrated with a task of moving and arranging chairs. The posture extraction from a video provided by a single camera may often lead to occlusions of body parts during activities in which objects are manipulated. We show the advantages of using an additional camera, which significantly increases the correct posture estimation, and discuss how to further improve the automatic generation of scripts.

3D modeling of real objects by a 3D laser scanner has become popular in many applications, such as reverse engineering of petrochemical plants, civil engineering and construction, and digital preservation of cultural properties. Despite the development of lightweight and high-speed laser scanners, the complicated measurement procedure and long measurement time are still heavy burdens for widespread use of laser scanning. To solve these problems, a robotic 3D scanning system using multiple robots has been proposed. This system, named CPS-SLAM, consists of a parent robot with a 3D laser scanner and child robots with target markers. A large-scale 3D model is acquired by an on-board 3D laser scanner on the parent robot from several positions determined precisely by a localization technique, named the Cooperative Positioning System (CPS), that uses multiple robots. Therefore, this system can build a 3D model without complicated post-processing procedures such as ICP. In addition, this system is an open-loop SLAM system and a very precise 3D model can be obtained without closed loops. This paper proposes an automatic planning technique for a laser measurement by using CPS-SLAM. Planning a proper scanning strategy depending on a target structure makes it possible to perform laser scanning efficiently and accurately even for a large-scale and complex environment. The proposed technique plans an efficient scanning strategy automatically by taking account of several criteria, such as visibility between robots, error accumulation, and efficient traveling. We conducted computer simulations and outdoor experiments to verify the performance of the proposed technique.

We present two multi-modal panoramic 3D outdoor (MPO) datasets for semantic place categorization with six categories: forest, coast, residential area, urban area and indoor/outdoor parking lot. The first dataset consists of 650 static panoramic scans of dense (9,000,000 points) 3D color and reflectance point clouds obtained using a FARO laser scanner with synchronized color images. The second dataset consists of 34,200 real-time panoramic scans of sparse (70,000 points) 3D reflectance point clouds obtained using a Velodyne laser scanner while driving a car. The datasets were obtained in the city of Fukuoka, Japan and are publicly available in [1], [2]. In addition, we compare several approaches for semantic place categorization with best results of 96.42% (dense) and 89.67% (sparse).

In this paper, for the purpose of monitoring people moving on the ground from a low-altitude aerial vehicle, we propose a method for stable image stabilization using planar information of the ground. In existing methods the homography-based method has been popularly used for image stabilization. However, in case that captured images consist of non-planar areas, the performance of the homography-based method decreases. The essential-based method works well in this scene, but the accuracy gets worse in case that images mainly include planar areas. Thus in the proposed method, we utilize the Geometric Robust Information Criterion (GRIC) model to choose a method from the homography-based method or the essential-based method. After the selection of the method, we extract a planar area in the scene by fitting a plane using RANSAC, followed by detection of people on the ground with high accuracy. Experimental results confirm that the effectiveness of the proposed method.

In recent years, several low-cost 3D laser scanners are being brought to the market and 3D laser scanning is becoming widely used in many fields. For example, 3D modeling of architectural structures or digital preservation of cultural heritages are typical applications for 3D laser scanning. Despite of the development of light-weight and high-speed laser scanners, however, the complicated measurement procedure and long measurement time are still a heavy burden for the widespread use of laser scanning. We have proposed a robotic 3D scanning system using multiple robots named CPS-SLAM, which consists of parent robots with a 3D laser scanner and child robots with target markers. In this system, a large-scale 3D model can be acquired by an on-board 3D laser scanner on a parent robot from several positions determined precisely by the localization technique using multiple robots named Cooperative Positioning System, CPS. Therefore, this system enables to build a 3D model without complicated post-processing procedures such as ICP. In addition, this system is an open-loop SLAM system and a quite precise 3D model can be obtained without closed loops. This paper proposes an automatic planning technique of a laser measurement for CPS-SLAM. By planning a proper scanning strategy depending on a target structure, it is possible to perform laser scanning efficiently and accurately even for a large-scale and complex environment. Proposed technique plans an efficient scanning strategy automatically by taking account of several criteria, such as visibility between robots, error accumulation, and efficient traveling. We conducted computer simulations and outdoor experiments to verify the performance of the proposed technique.

Person recognition from gait images is generally not robust to changes in appearance, such as variations of the walking direction. In general conventional methods have focused on training a model to transform gait features or gait images to those at a different viewpoint, but the performance gets worse in case the model is not trained at a viewpoint of a subject. In this paper we propose a novel gait recognition approach which differs a lot from existing approaches in that the subject's sequential 3D models and his/her motion are directly reconstructed from captured images, and arbitrary viewpoint images are synthesized from the reconstructed 3D models for the purpose of gait recognition robust to changes in the walking direction. Moreover, we propose a gait feature, named Frame Difference Frieze Pattern (FDFP), which is robust to high frequency noise. The efficiency of the proposed method is demonstrated through experiments using a database that includes 41 subjects.

We present a novel approach for outdoor place categorization using synchronized texture and depth images obtained using a laser scanner. Categorizing outdoor places according to type is useful for autonomous driving or service robots, which work adaptively according to the surrounding conditions. However, place categorization is not straight forward due to the wide variety of environments and sensor performance limitations. In the present paper, we introduce a two-dimensional local ternary pattern (2D-LTP) descriptor using a pair of synchronized texture and depth images. The proposed 2D-LTP describes the local co-occurrence of a synchronized and complementary image pair with ternary patterns. In the present study, we construct histograms of a 2D-LTP as a feature of an outdoor place and apply singular value decomposition (SVD) to deal with the high dimensionality of the place. The novel descriptor, i.e., the 2D-LTP, exhibits a higher categorization performance than conventional image descriptors with outdoor place experiments.

This paper introduces the concept of first-person animal activity recognition, the problem of recognizing activities from a view-point of an animal (e.g., a dog). Similar to first-person activity recognition scenarios where humans wear cameras, our approach estimates activities performed by an animal wearing a camera. This enables monitoring and understanding of natural animal behaviors even when there are no people around them. Its applications include automated logging of animal behaviors for medical/biology experiments, monitoring of pets, and investigation of wildlife patterns. In this paper, we construct a new dataset composed of first-person animal videos obtained by mounting a camera on each of the four pet dogs. Our new dataset consists of 10 activities containing a heavy/fair amount of ego-motion. We implemented multiple baseline approaches to recognize activities from such videos while utilizing multiple types of global/local motion features. Animal ego-actions as well as human-animal interactions are recognized with the baseline approaches, and we discuss experimental results.

We propose a novel method for recognizing people in aerial surveillance videos. Aerial surveillance images cover a wide area at low resolution. In order to detect objects (e.g., pedestrians) from such videos, conventional methods either utilize appearance information from raw videos or extract blob information from background subtraction results. However, people seen in low resolution images have less appearance information, and hence are very difficulty to classify based on their appearance or blob size. In addition, due to heavy camera movements caused by aerial vehicle ego-motion and wind, the system is expected to generate many noisy false detections including parallax. The idea presented in this paper is to detect and classify objects from aerial videos based on their motion: we analyze a trajectory of each object candidate, deciding whether it is a person-of-interest or simple noise based on how it moved. After objects are tracked by a Kalman filter-based tracking, we represent their motion as multi-scale histograms of ‘orientation changes’, which efficiently captures movements displayed by objects. Random forest classifiers are applied to our new representation to make the decision. The experimental results illustrate that our approach recognizes objects-of-interest (i.e., humans) even when there exist a large number of false detection/tracking, and it does it more reliably compared to the approaches with previous paradigm.

For creating a three-dimensional (3D) model of a real object using a laser scanner and a camera, texture mapping is an effective technique to enhance the reality. However, in case that the positions of the camera and the laser scanner differ from each other, some textureless regions (holes) may exist on the object surface where the appearance information is missing due to the occlusion or out-of-sight of the camera. In this paper, we propose a new texture completion technique utilizing laser reflectivity for hole-free texture mapping. The laser reflectivity, which denotes the power of a reflected laser light/pulse, is obtained as by-product of the range information at laser scanning. Since the laser reflectivity captures the appearance property of the target as a camera image, it is reasonable that the regions with similar reflectance properties have similar color textures. Based on this idea, texture information in these holes is copied and pasted from the other texture regions according to the similarity and the order determined by the texture and laser reflectivity. To verify the performance of the proposed technique, we carried out texture completion experiments in real scenes.

In this paper, we propose a new technique for adding color to a surface of a 3D geometrical model utilizing laser reflectivity. A time-of-flight laser scanner obtains a range image from the sensor toward the target by measuring the round-trip time of a laser pulse. At the same time, for most laser scanners, the reflectance image (which is the strength of the reflected light) is available as a by-product of the range value. The proposed technique first assigns appearance information to a 3D model by colorizing a reflectance image based on the similarity of color and reflectance images. Then the color information is transferred to the corresponding range image, and the colorized 3D model is obtained. We carried out experiments using a laser scanner and showed the performance of the proposed technique in several conditions.

Conventional methods of gait analysis for person identification use features extracted from a sequence of camera images taken during one or more gait cycles. An implicit assumption is made that the walking direction does not change. However, cameras deployed in real-world environments (and often placed at corners) capture images of humans who walk on paths that, for a variety of reasons, such as turning corners or avoiding obstacles, are not straight but curved. This change of the direction of the velocity vector causes a decrease in performance for conventional methods. In this paper we address this aspect, and propose a method that offers improved identification results for people walking on curved trajectories. The large diversity of curved trajectories makes the collection of complete real world data infeasible. The proposed method utilizes a 4D gait database consisting of multiple 3D shape models of walking subjects and adaptive virtual image synthesis. Each frame, for the duration of a gait cycle, is used to estimate a walking direction for the subject, and consequently a virtual image corresponding to this estimated direction is synthesized from the 4D gait database. The identification uses affine moment invariants as gait features. Experiments using the 4D gait database of 21 subjects show that the proposed method has a higher recognition performance than conventional methods.

Two denoising techniques using reflectivity for noisy range images are proposed: range image smoothing by trilateral filter and range image inpainting by belief propagation. The trilateral filter makes use of reflectivity as well as spatial and intensity information so that geometric features, such as jump and roof edges, are preserved while smoothing. The range image inpainting technique based on belief propagation recovers a deteriorated range image using not only the adjacent range values but also the continuity of the reflectance image. We conduct simulations and experiments using synthesized images and actual range images taken by a laser scanner and verify that the proposed techniques suppress noise while preserving jump and roof edges and repair deteriorated range images.

We introduce a novel person identification method for a surveillance system of much wider area than conventional systems using CCTV cameras. In the proposed system, we install cameras to rooftops of buildings or a low altitude airship, and identify people by gait features extracted from shadows, which are projected on the ground by the sun in the daytime or lights in the evening. Since conventional systems extract gait features from actual body area, the correct classification ratio is reduced due to the lack of information of body area, in case that images are captured by overhead cameras. On the other hand, the proposed system enables to identify people by gait features which are extracted from shadows projected on the ground, even if images are captured by overhead cameras. In the proposed system, shadow areas projected on the ground are extracted automatically from captured images, and then analyze dynamics of shadow areas by the spherical harmonics. Experiments of person identification using actual outside images revealed that the proposed method showed the best performance than conventional methods, and the results indicate the feasibility of person identification based on shadow analysis.

People identification has numerous applications, ranging from surveillance/security to robotics. Face and body movement/ gait biometrics are the most important tools for this task. Traditional biometrics use direct observation of the body, yet in some situations a projection may offer more information than the direct signal, for example the shadow of a person observed from overhead, e.g. from an unmanned aerial vehicle, may contain more detail than the top view of the head/body. We introduced the idea of shadow biometrics, exploiting biometrics information in human shadow silhouettes as derived from video imagery; this enables “overhead biometrics”, for recognition of human identity and behavior from high altitude airborne platforms using overhead video sequences. In this paper, we provide a demonstration of person identification based on gait recognition from shadow analysis. We describe compensation steps to address shadow variation with conditions of observation (sun position, etc). We define measures of shape variation, such as horizontal stripes on the silhouette, their length change in time determines frequency components (here spherical harmonics) for each gait cycle, which are used for classification by a k-nearest neighbor classifier. A correct classification rate (CCR) of 95 % was obtained. A degradation of CCR from 95 % to 75 % was observed when reduced spatial and temporal resolution from 1cm to 2cm, and from 30fps to 15fps.

For a coexisting and collaborative society that incorporates humans and robots, the detection, tracking, and recognition of human motion are indispensable techniques for a robot to safely and securely interact with humans. The present paper proposes a motion tracking system using distributed network cameras that are placed in a sizeable environment, such as a street or a town. Model-based motion tracking is adopted in this system, and an asynchronous process is invoked for updating motion estimation in each camera individually. A 2D distance map created by the Fast Marching Method is used to estimate human motion in real-time. Experiments demonstrate that human motion while walking among eight distributed cameras is tracked correctly by automatically selecting appropriate cameras.

This paper proposes a new person identification method using physiological and behavioral biometrics. Various person recognition systems have been proposed so far, and one of the recently introduced human characteristics for the person identification is gait. Although the shape of one's body has not been considered much as a characteristic, it is closely related to gait and it is difficult to disassociate them. So, the proposed technique introduces a new hybrid biometric, combining body shape (physiological) and gait (behavioral). The new biometric is the full spatio-temporal volume carved by a person who walks. In addition to this biometric, we extract unique biometrics in individuals by the following way: creating the average image from the spatio-temporal volume and forming the new spatio-temporal volume from differential images which are created by subtracting an average image from original images. Affine moment invariants are derived from these biometrics, and classified by a support vector machine. We used the leave-one-out cross validation technique to estimate the correct classification rate of 94 %.

In order to construct three-dimensional shape models of large-scale architectural structures using a laser range finder, a number of range images are taken from various viewpoints. These images are aligned using post-processing procedures such as the ICP algorithm. However, in general, before applying the ICP algorithm, these range images must be aligned roughly by a human operator in order to converge to precise positions. The present paper proposes a new modeling system using a group of multiple robots and an on-board laser range finder. Each measurement position is identified by a highly precise positioning technique called Cooperative Positioning System (CPS), which utilizes the characteristics of the multiple-robot system. Thus, the proposed system can construct 3D shapes of large-scale architectural structures without any post-processing procedure or manual registration. ICP is applied optionally for a subsequent refinement of the model. Measurement experiments in unknown and large indoor/outdoor environments are carried out successfully using the newly developed measurement system consisting of three mobile robots named CPS-V. Generating a model of Dazaifu Tenmangu, a famous cultural heritage, for its digital archive completes the paper.

This paper presents a parallel algorithm of the Level Set Method named the Parallel Fast Level Set Method, and its application for real-time 3D reconstruction of human shape and motion. The Fast Level Set Method is an efficient implementation algorithm of the Level Set Method and has been applied to several applications such as object tracking in video images and 3D shape reconstruction using multiple stereo cameras. In this paper, we implement the Fast Level Set Method on a PC cluster and develop a real-time motion capture system for arbitrary viewpoint image synthesis. To obtain high performance on a PC cluster, efficient load-balancing and resource allocation algorithms are crucial problems. We develop a novel optimization technique of load distribution based on the estimation of moving direction of object boundaries. In this technique, the boundary motion is estimated in the framework of the Fast Level Set Method, and the optimum load distribution is predicted and performed according to the estimated boundary motion and the current load balance. Experiments of human shape reconstruction and arbitrary viewpoint image synthesis using the proposed system are successfully carried out.

This paper presents a sensor network system consisting of distributed cameras and laser range finders for multiple objects tracking. Sensory information from cameras is processed by the level set method in real time and integrated with range data obtained by laser range finders in a probabilistic manner using novel SIR/MCMC combined particle filters. Though the conventional SIR particle filter is a popular technique for object tracking, it has been pointed out that the conventional particle filter has some disadvantages in practical applications such as its low tracking performance for multiple targets due to the degeneracy problem. In this paper, the new combined particle filters consisting of a low-resolution MCMC particle filter and a high-resolution SIR particle filter is proposed. Simultaneous tracking experiments for multiple moving targets are successfully carried out and it is verified that the combined particle filters has higher performance than the conventional particle filters in terms of the number of particles, the processing speed, and the tracking performance for multiple targets.

In medical diagnostic imaging, an X-ray CT scanner or a MRI system have been widely used to examine 3D shapes or internal structures of living organisms or bones. However, these apparatuses are generally very expensive and of large size. A prior arrangement is also required before an examination, and thus, it is not suitable for an urgent fracture diagnosis in emergency treatment. This paper proposes a method to estimate a patient-specific 3D shape of a femur from only two fluoroscopic images using a parametric femoral model. Firstly, we develop a parametric femoral model by statistical analysis of a number of 3D femoral shapes created from CT images of 51 patients. Then, the pose and shape parameters of the parametric model are estimated from two 2D fluoroscopic images using a distance map constructed by the level set method. Experiments using synthesized images and fluoroscopic images of a phantom femur are successfully carried out and the usefulness of the proposed method is verified.

This paper introduces a new motion capture system for recovering 3D models of multiple persons separately and robustly against occlusion in real-time. Various markerless motion capture systems using video cameras have been proposed so far. However, in case that there are multiple persons in the scene at the same time, it is quite difficult to reconstruct a precise 3D model of each person separately due to the occlusion between them. To deal with this problem, the fast level set method is utilized in the proposed system for integrating stereo range data which is captured by multiple stereo cameras located around the target people. To reconstruct precise 3D models in real-time, the proposed system is implemented on a PC cluster with seven PCs and four stereo cameras. Tracking experiment of multiple persons and real-time reconstruction of 3D human models using the proposed system are successfully carried out