Semantic segmentation, a fundamental task in image processing, is applied to various significant applications such as visual navigation of a robot, where accurate results and high speed processing are required. GPUs play an important role in high speed implementation but other options should be shown because various heavy procedures run in the process of the visual navigation. In this study, we propose a novel method to improve the calculation speed of PSPNet on the latest AMD Ryzen 9950X CPU. First, in order to improve the calculation speed of the Convolution layer, which accounts for the majority of PSPNet calculations, we utilize SIMD instructions (AVX-512), multi-core parallelization, and L1, L2, and L3 caches to accelerate matrix multiplication calculations using im2col. Next, we improve the calculation speed of the entire model by fusing the Convolution layer and the Batch Normalization layer. Compared with the PyTorch implementation, the proposed method achieved more than $79 \%$ of the peak performance in matrix multiplication while maintaining accuracy and succeeded in speeding up inference time by $47.712 \%$ on the Cityscapes dataset and $39.096 \%$ on the Visual Navigation dataset.

The authors attempt to actualize 3D reconstruction from a single view for previewing a room in virtual space using results of semantic segmentation. The segmentation accuracy has been drastically improved by state-of-the-art method, which enables pixel-wise classification of walls, floors, ceilings, and objects in the target room with sufficient accuracy. Assuming a room can be represented as a cuboid, its parameters can be computed analytically when semantic edges of the room structure are accurately obtained. In the actual process of the estimation, lines constructing the cuboid are estimated from detected edges. These edges are derived by spatial filtering to a semantic map corresponding to an input image. To enhance the effectiveness of edge detection on a semantic map, we adopted a simple differential filter that incorporates only two active values as filter coefficients, utilizing the smallest filter size. Experimental results using synthetic datasets showed no significant difference in the accuracy of line parameter estimation when comparing our method with a typical filter, despite using half the samples during the optimization process though sample numbers for parameter estimation became smaller obviously.

The present work enhanced the classification accuracy of motor imagery (MI) based on EEG signals using a novel deep neural network (DNN) architecture that integrated InternImage and ST-pooling to capture spatiotemporal features. While this architecture outperformed existing methods in classification accuracy, the validation loss indicates overfitting during training. We introduce a bias-variance guided layer selection (BVG-LS) strategy into the fine-tuning process to address this issue. This approach adaptively adjusts the number and selection of layers updated during fine-tuning, replacing the conventional single-layer update with a more effective multi-layer configuration guided by BVG-LS. Experimental evaluation on the cross-individual validation task using the PhysioNet EEG Motor Movement/Imagery dataset showed that the accuracy of two-, three-, and four-class classification were improved to 89.06%, 81.13%, and 70.90%, respectively, surpassing the performance of existing fine-tuning techniques, including full fine-tuning.

Visual navigation rely heavily on semantic segmentation outcomes, which is invaluable for practical applications. However, the efficacy of this navigation method is compromised when the accuracy of semantic segmentation falls short. Crucially, the availability of an appropriate dataset containing pixel-wise class labels is imperative for constructing a robust classifier. To alleviate the burden of manual annotation, the authors have endeavored attempt to implement a semi-automatic process for generating a training dataset from 3D scanned data. To enhance the versatility of the approach, the present study introduces augmentation techniques that consider the semantic attributes of images within the target scenario: DMIT and ToD are employed to address color variations caused by seasonal changes lawn growth and fluctuations on the sun’s height, respectively. Experimental results based on images captured during the Tsukuba Challenge, a competition featuring autonomous moving robots in Japan, showed that the proposed methodology substantially enhances classification accuracy, particularly for images taken under conditions different from those during the creation of the 3D model.

Visual place recognition (VPR) seems effective for the localization process in visual navigation of a robot because a robot can recognize locations and poses using only visual information if the accuracy of VPR is sufficient. However, to apply VPR for actual scenarios where multiple locations to be distinguished are in a captured image, the classification accuracy should be improved. To solve this problem, we propose a novel method for likelihood computation in location matching composed of two kinds of components corresponding to local and global features. Experimental results using a novel dataset composed of actual images taken around the course of the Tsukuba Challenge showed that the proposed method improved accuracy by 1–6 % compared to conventional reranking methods using only local features.

Acquiring purpose-specific data is crucial for applying deep learning to applications. However, creating such data can be labor-intensive, necessitating the development of datasets at minimal cost. Consequently, there has been a recent trend towards utilizing computer-generated imagery (CGI) to generate purpose-specific datasets. Nonetheless, models trained on CGI data often exhibit suboptimal performance on real images due to domain gap issues. Thus, bridging this gap between synthetic and real data domains is crucial. We are developing a real parts classification application using 3D CAD models as training data. This study discusses our strategies for addressing the domain gap challenge in this application. To mitigate this gap, we explore three primary approaches: selecting architectures with a strong shape bias, leveraging large-scale pre-training to enhance generalization performance, and enhancing the shape bias of classifiers by manipulating color characteristics in the training data. We present evidence demonstrating the effectiveness of these meth-ods in bridging the domain gap between CG I training data and real-world images in our part classification application: the top-l accuracy became 98.34 % at the best case.

A visual navigation method based on results of semantic segmentation showed interesting results in previous researches. The most significant problem of the method is that the moving performance is affected by the segmentation accuracy, which strongly depends on the training data even though SOTA methods are adopted. To create high-quality dataset for this application without huge human efforts, the authors tries to generate datasets for semantic segmentation from a 3D scanned data composed of colored point clouds. In this study, we investigate what kinds of variations are effective to construct a classifier: variation of augmentation considering shadows, shooting angles, and shooting locations. Experimental results using actual images for evaluation and generated dataset for training captured at the course of Tsukuba Challenge, which is the famous competition for autonomous moving robots in Japan, showed that adding shadows in training datasets improved the mIoU but random changes to the shooting angle and location did not always work well. By the result, it is shown that augmentation considering the characteristic of the target environment becomes important for practical use.

Semantic segmentation can provide significant information for a robot to actualize autonomous moving. To increase the classification accuracy of segmentation, appropriate datasets should be prepared, which requires huge human labors. The authors attempt to realize a semi-automatic method that generates two dimensional images having pixel-wise class labels from 3D point clouds obtained by a 3D scanner. In this study, dense point clouds are generated to improve the quality of training samples. Experimental results using data obtained around the course of the Tsukuba Challenge, which is the famous competition for autonomous moving robots in Japan, showed that the dense point clouds effective but appropriate variety of textures should be included in the training samples.

Visual odometry is a key technology for an autonomous robot to accurately determine its locations on a map accurately if a camera is the main external sensor. If IMU is available, the scale information can also be estimated by combining visual and IMU information, which is called Visual Inertial Odometry (VIO). This research attempts to modify VINS-Mono, a widely used VIO method, to improve the estimation accuracy in symbiotic environments with people in outdoor scenes, where estimation accuracy becomes worsens according to the dynamic obstacles in images. The proposed method replaces a method for feature point extraction in VINS-Mono and adds a process to limit the area for the feature point extraction using semantic information obtained from segmentation results. Experimental results using datasets created from actual environments demonstrate that SuperPoint showed the best accuracy for most scenes and that area limitation improved estimation accuracy when many dynamic obstacles were included in the input images.

An autonomous moving scheme with semantic information extracted from images captured by the monocular camera was proposed, providing accurate results of semantic segmentation. A training dataset must accommodate the moving environment to train a classifier for autonomous moving. However, preparing a large-scale dataset composed of images having pixel-wise manually-annotated class labels is impractical. We generated datasets automatically from 3D point clouds, for reducing manpower. The dataset had significant problems: domain gap and shadows. Therefore, style transfer is incorporated in the proposed scheme to bridge the gap to resolve this problem. Moreover, pseudo shadows were added to improve the classification accuracy of testing images with shadows. Experimental results using 3D point clouds and testing images taken around Tsukuba City showed that classification accuracy was improved by the proposed scheme. The classification accuracy of the sidewalk, the most significant class for autonomous moving at the Tsukuba Challenge, was 95.7%.

Multispectral sensors are used to ensure visibility in various applications. However, when multiple sensors are used for capturing images, a misalignment may occur between the images taken by each sensor unless special care is taken. To correct such misalignments, image registration based on feature matching is conducted. However, the features captured by each sensor differ, thereby complicating the registration process. In this study, we develop an approach to overcome these challenges and to improve the registration accuracy between short-wave infrared and mid-wave infrared (SWIR and MWIR, respectively) images. First, we compare and validate SiLK, a detector-based feature matching method, and LoFTR, a detector-free feature matching method. The results clearly demonstrate the superior accuracy of LoFTR. Moreover, SWIR and MWIR images exhibit a characteristic color inversion according to Kirchhoff’ s law. Therefore, by inverting the color of a SWIR image and aligning the color tone between image pairs, we can improve the matching accuracy. Furthermore, by diversifying the color tones of the training data through augmentation, we can handle the domain gap between SWIR and MWIR images, thereby further enhancing the matching accuracy.

In this research, we propose a method for evaluating images to be inspected using only good images and assuming defective images are unavailable in anticipation of quality inspection applications in food processing facilities and other factories. We propose a discriminator based on a CNN that can evaluate the degree of deviation from good images by treating only good images as training data and assuming a fitness probability distribution. The results show that the proposed discriminator can detect defective products even without prior training data on defective products. The detection accuracy depends on the inspected object and the threshold that defines the deviation, which is comparable to previous studies that require defective images. With adjustable detection thresholds and automatic categorization of defective products, the proposed method is expected to be flexible enough to incorporate the knowledge of shop-floor workers on the production line.

Visual odometry is a key technology for the lo-calization of autonomous robots in real-world environments. In particular, when a robot adopts a camera as an external sensor, visual odometry can provide useful information for fine-grained localization. Many approaches have been developed for this task; however, this study proposes a novel scheme that uses traditional feature extraction and tracking to realize computationally efficient but sufficiently accurate for practical use when used with a visual navigation scheme based on semantic segmentation. The key features of the proposed scheme can be summarized as follows: the elimination of feature points on moving obstacles, feature point extraction on object boundaries, feature tracking, and outlier elimination, considering semantic information. The proposed scheme reduced the estimation error of visual odometry to 12% of the existing scheme in the best case, according to experimental results, using datasets created with the CARLA simulator.

Visual navigation based on the results of semantic segmentation requires high classification accuracy. Previous research has proven that a classifier of semantic segmentation trained upon a dataset generated from a 3D model performs well when the input images are also generated from a 3D model. However, when the input images are real 2D images captured at the same location by a camera mounted on a robot, the average classification accuracy deteriorates. To overcome this issue, a novel scheme is proposed to improve the classification accuracy of semantic segmentation when the training data is generated from a 3D point cloud. The key features of the proposed scheme are filling in the missing data by inpainting and domain adaptation by histogram matching. To evaluate the proposed scheme, datasets composed of real images captured during a variety of seasons, weathers, and times were created. Experimental results showed that ICNet trained upon our dataset could provide accurate results for visual navigation.

Several proposed schemes have shown remarkable results in autonomous navigation in actual scenes. However, most schemes are dependent on expensive three-dimensional sensing devices such as 3D LiDAR. To solve this problem, in this study, a novel scheme supporting autonomous movement is proposed, particularly for road direction estimation using Virtual LiDAR signals generated through results of semantic segmentation and geometrical information of the camera. Experimental results using the CARLA simulator and actual images taken at outdoor scenes showed that the proposed scheme achieves accurate results, even when moving obstacles are present on the road.

To collect vital data from a group of outdoor exercisers in real-time and reliably, we propose a wireless multihop network system based on a flooding/time division multiple access (TDMA) protocol with an extended superframe structure. For wireless vital sensor nodes (VSNs) put to the bodies of exercisers, the system first randomly and temporarily divides a whole group of VSNs into several subgroups and assigns a distinct superframe to each subgroup. Then, when VSNs move around in a ground, the system adaptively changes subgroup members so as to make them uniformly and widely spread in the ground by using the location information on all the VSNs.In this paper, we evaluate the performance of the system by an experiment. We conduct the experiment in an outdoor ground of 60m×90m for 45 min, involving 50 subjects, where we put VSNs to 18 subjects out of them. We show in the performance evaluation that the system can collect vital data from the 18 subjects even when all the subjects perform a variety of exercises, once in 2 sec in average, for 45 min regularly, and in 94.9 % data collection rate reliably.

The use of detailed metric maps for autonomous movement of robots has been popularized in recent times. Three-dimensional sensing devices, such as 3D LiDAR and RADAR, which are expensive yet indispensable, are utilized to generate these metric maps, and ultimately, perform localization. To reduce the cost of sensing devices, we try to realize autonomous movement of a robot using only cheap image sensors, such as webcams. For robot navigation, image processing tends to be applied to collision avoidance by finding obstacles. In contrast, our approach does not use visual object detection but adopts path planning based on movable area extraction from input images using semantic segmentation. To obtain accurate results for visual navigation, this paper proposes the use of novel datasets for semantic segmentation. Experimental results showed that ICNet could extract the movable area with more than 99% accuracy if it was trained with appropriate datasets, and a robot can run automatically based on the extracted movable area.

To improve the accuracy of fashion style estimation, this paper proposes a novel large-scale dataset named WEARStyle and two types of novel schemes that remove unnecessary pixels: SSD-based human detection and PSPNet-based pixel selection. The classification accuracy of the Hipster Wars dataset is improved to 78.8% by an SVM-based classifier when the WEARStyle dataset is used to train a ResNet50-based feature extractor. The accuracy is improved to 80.0% and 80.9%, when the SSD-based human detection and PSPNet-based pixel selection are applied, respectively. The achieved accuracy outperforms those of other existing schemes.

Imbalanced samples composed of limited number of positive samples corresponding to objects and huge number of negative samples extracted from background regions reduces the accuracy of visual object detection. To solve this problem this paper proposes a novel convolutional neural network named “CasNet”. CasNet introduces cascade structure that is used for rapid and accurate object detector in order to reduce the number of negative samples inputted to a main network for object detection. The CasNet becomes a cascade stage when it is attached to a layer of existing convolutional neural networks to construct cascaded classifier. Each stage composed of a CasNet performs two-class classification to reject easy negatives corresponding to background regions. By this early rejection of easy negatives, a main network can be trained to classify more complex samples. Experimental results using a dataset created from the PASCAL VOC2012 dataset showed that higher accuracy was obtained at less training iterations if CasNets were attached to VGG16 appropriately.

Reliable and real-time vital signs monitoring is essential to promote health and prevent disease/injury for a group of exercisers spread in a sports ground such as schoolchildren and professional athletes. For that purpose, we developed a wireless vital signs collection system and confirmed it worked effectively, but unfortunately, it required an infrastructure of data forwarding nodes placed around the area of interest. We really wanted to realize it infrastructurelessly.Now, we are developing an infrastructureless multi-hop wireless network system which has the same vital sensing and wireless communication functions, and we have so far successfully conducted one indoor experiment and two outdoor experiments for footballer subjects. In this paper, we will outline the design concept and show the experimental results.

Localization of body parts in image sequences is beginnings to become practical owing to the improvement in accuracy of image recognition by deep learning. Sports scene analysis is one application of localization of body parts, but existing schemes have difficulty obtaining sufficient accuracy because a sports scene includes variegated patterns of poses and occlusions that decrease the robustness of localization. To robustly localize body parts in sports scenes, this paper proposes a novel scheme that applies failure correction using interframe information to extend an existing scheme proposed by Cao et al. Experimental results using a novel dataset composed of image sequences of a tennis player showed that the proposed scheme reduced the occurrence rate of failure frames to 0% from 36.7% for the existing scheme.

The authors try to construct a novel sensor networking scheme that obtains the location of sensor nodes using image processing in order to enable dynamic routing when the speed and the density of the sensor nodes become high. In this scheme, visual object detection is applied for the localization of sensor nodes; however, certain failures occur during the region merging process, which is required for schemes based on sliding windows. For the widely used pairwise max suppression(PMS), the most significant problem is the fixed threshold for region merging. To solve this problem, this paper proposes a novel scheme for region merging that adopts adaptive thresholding for the existing PMS. The threshold is appropriately determined by taking into account the total number of detection targets. The experimental results, conducted using a dataset composed of top-view images generated from a CG-based virtual space, showed that the miss rate can be reduced to approximately 67.4% of the existing PMS.

It is required to construct a practical scheme that can obtain personal properties accurately based on gait analysis because it can work under severe conditions. The authors tackle this task and have proposed a feature extraction scheme based on joint motions. However, it is not clear that our scheme is feasible because the scheme uses 25 joints all of that cannot be estimated accurately in severe conditions based on only visible images for feature extraction. To show that our scheme has a potential for practical applications, the classification accuracy of personal identification is evaluated under practical conditions where the number of joints available for feature extraction is small. Experimental results show that the classification accuracy is 81.58% if the number of joints is 10 and the accuracy keeps 79.82% even if the number of joints is reduced to 6.

In the above paper (ibid., vol. 22, no. 12, pp. 4752-4761, Dec. 2013), several errors were introduced. These errors are corrected here.

In pedestrian detection, as sophisticated feature descriptors are used for improving detection accuracy, its processing speed becomes a critical issue. In this paper, we propose a novel speed-up scheme based on multiple-instance pruning (MIP), one of the soft cascade methods, to enhance the processing speed of support vector machine (SVM) classifiers. Our scheme mainly consists of three steps. First, we regularly split an SVM classifier into multiple parts and build a cascade structure using them. Next, we rearrange the cascade structure for enhancing the rejection rate, and then train the rejection threshold of each stage composing the cascade structure using the MIP. To verify the validity of our scheme, we apply it to a pedestrian classifier using co-occurrence histograms of oriented gradients trained by an SVM, and experimental results show that the processing time for classification of the proposed scheme is as low as one-hundredth of the original classifier without sacrificing detection accuracy.

Recently, many researchers tackle accurate object recognition algorithms and many algorithms are proposed. However, these algorithms have some problems caused by variety of real environments such as a direction change of the object or its shading change. The new tracking algorithm, cascade particle filter, is proposed to fill such demands in real environments by constructing the object model while tracking the objects. We have been investigating to implement accurate object recognition on embedded systems in real-time. In order to apply the cascade particle filter to embedded applications such as surveillance, automotives, and robotics, a hardware accelerator is indispensable because of limitations in power consumption. In this paper we propose a hardware architecture of the cascade particle filter. With our implementation, both the accurate object recognition and real-time processing are possible at an embedded system.

This paper proposes a hardware architecture for object detection based on an AdaBoost learning algorithm with Haar-like features as weak classifiers. We analyze and discuss the parallelism in this detection algorithm and propose a partially parallel execution model suitable for hardware implementation. This parallel execution model exploits the cascade structure of classifiers, in which classifiers located near the beginning of the cascade are used more frequently than subsequent classifiers. We assign more resources to these earlier classifiers to execute in parallel than to subsequent classifiers. This dramatically improves the total processing speed without a great increase in circuit area. Moreover, the partially parallel execution model achieves flexible processing performance by adjusting the balance of parallel processing. In addition, we implement the proposed architecture on a Virtex-5 FPGA to show that it achieves real-time object detection at 30 fps on VGA video without candidate extraction.

Robust and rapid object detection is one of the great challenges in the field of computer vision. This paper proposes a hardware architecture suitable for object detection by Viola and Jones based on an AdaBoost learning algorithm with Haar-like features as weak classifiers. Our architecture realizes rapid and robust detection with two major features: hybrid parallel execution and an image scaling method. The first exploits the cascade structure of classifiers, in which classifiers located near the beginning of the cascade are used more frequently than subsequent classifiers. We assign more resources to the former classifiers to execute in parallel than subsequent classifiers. This dramatically improves the total processing speed without a great increase in circuit area. The second feature is a method of scaling input images instead of scaling classifiers. This increases the efficiency of hardware implementation while retaining a high detection rate. In addition we implement the proposed architecture on a Virtex-5 FPGA to show that it achieves real-time object detection at 30 frames per second on VGA video.

A novel pedestrian tracking scheme based on a particle filter is proposed, which adopts a skeleton model of a pedestrian as a state space model and uses distance transformed images for likelihood estimation. The six-stick skeleton model used in the proposed approach is very distinctive in representing a pedestrian simply but effectively, with which the efficient state space for the pedestrian tracking can be derived. Experimental results by using PETS sample sequences demonstrate that the proposed approach achieves highly accurate pedestrian tracking without any of prior learning.