In this study, we propose a method for estimating ocean wave height using data acquired from an inertial measurement unit (IMU) mounted on a free-drifting buoy deployed at sea. The IMU captures three-axis acceleration and three-axis angular velocity, which are used as input for a time-series regression model based on the Transformer architecture. During training, ground truth wave height is derived by converting absolute altitude variations—measured via the Centimeter-Level Augmentation Service (CLAS), a high-precision positioning service of the Quasi-Zenith Satellite System, part of the Global Navigation Satellite System—into relative wave heights. The model is trained using both the converted wave height data and the corresponding IMU measurements. After training, the model estimates wave heights from new IMU input data, and its performance is evaluated by comparing the estimated values with the CLAS-derived ground truth. Results indicate that, when training and test data are obtained on the same measurement day, the maximum absolute error in significant wave height is 0.07 m. When training and test data are from different days, the maximum absolute error remains within 0.06 m. These results demonstrate that the proposed Transformer-Based time-series regression model effectively estimates wave height from IMU data with high accuracy.

Abstract Is it possible to use convolutional neural networks pre-trained without any natural images to assist natural image understanding? The paper proposes a novel concept, Formula-driven Supervised Learning (FDSL). We automatically generate image patterns and their category labels by assigning fractals, which are based on a natural law. Theoretically, the use of automatically generated images instead of natural images in the pre-training phase allows us to generate an infinitely large dataset of labeled images. The proposed framework is similar yet different from Self-Supervised Learning because the FDSL framework enables the creation of image patterns based on any mathematical formulas in addition to self-generated labels. Further, unlike pre-training with a synthetic image dataset, a dataset under the framework of FDSL is not required to define object categories, surface texture, lighting conditions, and camera viewpoint. In the experimental section, we find a better dataset configuration through an exploratory study, e.g., increase of #category/#instance, patch rendering, image coloring, and training epoch. Although models pre-trained with the proposed Fractal DataBase (FractalDB), a database without natural images, do not necessarily outperform models pre-trained with human annotated datasets in all settings, we are able to partially surpass the accuracy of ImageNet/Places pre-trained models. The FractalDB pre-trained CNN also outperforms other pre-trained models on auto-generated datasets based on FDSL such as Bezier curves and Perlin noise. This is reasonable since natural objects and scenes existing around us are constructed according to fractal geometry. Image representation with the proposed FractalDB captures a unique feature in the visualization of convolutional layers and attentions.

The paper proposes a method for automatic multi-view dataset construction based on formula-driven supervised learning (FDSL). Although data collection and human annotation of 3D objects are labor-intensive, we automatically generate their training data and labels in the proposed multi-view dataset. To create a large-scale multi-view dataset, we employ fractal geometry, which is considered the background information of many objects in the real world. We project in a circle from the rendered 3D fractal models to construct the Multi-view Fractal DataBase (MV-FractalDB), which is then used to make a pre-trained CNN model. According to the experimental results, the MV-FractalDB pre-trained model surpasses the accuracies with self-supervised methods (e.g., SimCLR and MoCo) and is close to supervised methods (e.g., ImageNet) in terms of performance rates on multi-view image datasets. We demonstrate the potential of FDSL for multi-view image recognition.

Change detection is the study of detecting changes between two different images of a scene taken at different times. The change detection methodology can provide us information in which area images changed time by time. However, for application use, especially on disaster investigation, it is highly required to understand not only where but also what changes are occurred in high precision and resolution. The paper proposes the concept of semantic change detection, which involves intuitively inserting semantic meaning into detected change areas. We mainly focus on the novel semantic segmentation in addition to a conventional change detection approach. In order to solve this problem and obtain a high-level of performance, we propose an improvement to the hypercolumns representation, hereafter known as hypermaps, which effectively uses convolutional maps obtained from convolutional neural networks (CNNs). We also employ multi-scale feature representation captured by different image patches. We applied our method to the TSUNAMI panoramic change detection dataset (TSUNAMI dataset), and re-annotated the changed areas of the dataset via semantic classes. The results show that our multi-scale hypermaps provided outstanding performance on the re-annotated TSUNAMI dataset.

We are developing an appearance-based proficiency evaluation system for individual fundamental football skills. As a basic stage of the research, we propose a method to discriminate between beginners and experts based on features of the inside-kick motion, which is a fundamental skill in football. In addition, we propose a method for scoring the inside-kick motion. To provide ground truth, a football coach (former professional football player) scored a series of inside-kick trials based on five criteria. Each criterion was scored out of 20 points, resulting in an overall score of up to 100 points for each trial. We utilized dense trajectories to extract features from the inside-kick motion and bag of features for feature vectorization. A random forest was then adopted to remove the effects of individual techniques from the feature vectors. Finally, a support vector machine was utilized to discriminate between beginners and experts. Each trial was evaluated based on five criteria to score the inside-kick motion using support vector regression. Total score estimates were calculated as the sum of the scores for each criterion. The results reveal that the accuracy of beginner/expert discrimination reached a maximum value of 91%. The average difference between the scores estimated by the proposed method and those rated by the football coach was approximately 0.17 points. Overall, these experimental results confirm the effectiveness of the appearance-based proficiency evaluation system.

We investigated effective features for human detection. The histogram of oriented gradients (HOG), which was proposed by N. Dalal, is an important representation that accumulates the edge-magnitude into a quantized histogram. Effective features similar to the HOG have been proposed. We question what the most effective feature is. We thus evaluate several features on three datasets of pedestrians, faces, and vehicles. We select the scale-invariant feature transform, local binary pattern, higher-order local auto correlation (HLAC), co-occurrence HOG, and extended CoHOG in addition to the HOG as features. These features have been adopted as effective features in related works. The features are applied to human detection on each dataset employing the real AdaBoost classifier. A comparison of classification results reveals that the combination of the HLAC and CoHOG is an effective feature for human detection.

We propose a wearable guide system for supporting the visually impaired. A wearable device equipped with a small laser range sensor and a gyroscope as sensors is developed. The sensors are attached to the user's chest. The range sensor is utilized to obtain distance information to the wall in the horizontal cross-sectional plane in front of the user. The gyroscope is adopted to estimate user's direction utilized in the fork pattern classification. The system classifies passage forks of the indoor environment into 7 patterns; left turn, straight, right turn, T-junction (left), T-junction (dead end), T-junction (right), crossroads. Based on the fork classification and prepared environmental topological map, the system instructs the user appropriate direction by voice at the fork. Experimental trials show the basic validity of the proposed system. Among 10 eye-masked subjects who join the experiment, the only two persons can reach the destination without the voice announcement. On the other hand, 8 of 10 subjects who wear the proposed wearable device can reach the destination.

We have proposed a system that recognizes text characters from the region pointed at with a fingertip for the visually handicapped. For example, in a supermarket, the blind user can pick up and touch the goods. However, the user cannot read the characters printed on the surface of the object. He/she cannot exactly know what the object is. It is very inconvenient for him/her. The proposed system will help the user to recognize the characters on the object. The system detects the ring which the user wears on the index finger and calculates the position of the user's fingertip of the index finger. Some acrylic infrared-reflective beads are arranged on the ring and it is easy to detect them from an infrared image. After fingertip detection, a region in which characters to be recognized is generated according to the position of the fingertip. The characters are extracted within the region using image processing. Thus, the redundant character recognition is avoided. The extracted characters are combined as a text and the text is read aloud by OCR software and free reading software for the user. Some experiments show the basic validity of the proposed method and system.

This paper proposes a position-free interface system that enables a user to input and obtain information everywhere in the environment with a projector-camera system. Hand-waving detected in the image is used as a cue to start the system, to distinguish gestures based on the user's intension of operation from other daily-life motions, and to specify the position of the user's hand. Hand-waving detection is implemented based on combination of background subtraction method and skin color extraction. In addition, we focus on changes of brightness value in each pixel. The user's hand is tracked using a particle filter algorithm. The user designates the projection area freely on a plane in the environment with his/her hand. The projector mounted on a pan-tilt unit projects information according to the user's designation. Then, the user's fingertip is detected using simple template matching and he/she interacts with the projected information to input commands without any devices. As one of applications, we have developed an interface system to operate electrical appliances (TV) intuitively in a daily life scene.

An effective human-robot interaction is essential for wide penetration of service robots into the market. Such robots need vision systems to recognize objects. It is, however, difficult to realize vision systems that can work in various conditions. More robust techniques of object recognition and image segmentation are essential. Thus, we have proposed to use the human user's assistance for object recognition through speech. The robot asks a question to which the user can easily answer and whose answer can efficiently reduce the number of candidate objects even if there are occluded objects and/or objects composed of multicolor parts in the scene. It considers the characteristics of features used for object recognition such as the easiness for humans to specify them by word, thus generating a user-friendly and efficient sequence of questions. Experimental results show that the robot can detect target objects by asking the questions generated by the method.

This paper deals with a basic dance training system with multimodal information presentation. The system targets on beginners and enables them to learn basics of dances easily. Watching a video showing the performance of dance masters is one of the most effective ways of learning dances. However, it is difficult for dance beginners to obtain useful information well only by watching video. They can hardly know timing information and the amount of translation. Although we can tell how to move our arms or legs from video, it is difficult to know when to start moving them. In addition, they cannot obtain translational motion, especially that in the depth direction (how far the dancers move forward or backward). We solve the first issue by developing active devices. The active devices consist of some vibro-motors and are designed to direct action-starting cues with vibration. Concerning the second issue, the image display on a mobile robot is introduced into the system. We can learn the amount of translation just by following the robot. Experimental results show the effectiveness of our proposed method.

Eye contact is an effective means of controlling human communication, such as in starting communication. It seems that we can make eye contact if we simply look at each other. However, this alone does not establish eye contact. Both parties also need to be aware of being watched by the other. We propose a method of two-way eye contact for human-robot communication. When a human wants to start communication with a robot, he/she watches the robot. If it finds a human looking at it, the robot turns to him/her, changing its facial expressions to let him/her know its awareness of his/her gaze. When the robot wants to initiate communication with a particular person, it moves its body and face toward him/her and changes its facial expressions to make the person notice its gaze. We show several experimental results to prove the effectiveness of this method. Moreover, we present a robot that can recognize hand gestures after making eye contact with the human to show the usefulness of eye contact as a means of controlling communication.

This paper presents a basic dance training system with a mobile robot. The system targets on beginners and enables them to learn basics of traditional dances easily. Learners can observe not only a fixed screen in front of them but also the display mounted on a mobile robot that moves according to the dance steps. These displays show actions of a dance master and real-time captured movement of the learners. The robot always takes its appropriate position so that the learners can easily know bad points of their actions. In other words, the robot plays a role as a "mobile mirror." We verified the effectiveness of the proposed method and system through experiments.

This paper presents a vision-speech system for service robots that can learn the user's customs and objects fixed in the environment while helping the user, and can perform their tasks more efficiently with less user's burden. We are working on a service robot that fetches the objects ordered by the user through speech. The robots need vision to recognize the objects. It asks the user for help by speech if its vision fails. In early stages, it asks the user for help many times and requires time to detect the objects by vision, because it does not know much about the environment. We assume that humans usually put a particular object on a small number of places. The robot obtains such information and the environmental knowledge while accomplishing the user's demands. In later stages, the user does not need to say details because the robot knows where the objects usually are. Moreover, the vision processing time is greatly reduced because it knows what operations can work in such cases. Experiments using a robot system show the usefulness of the proposed system.

This paper presents a display system that enables us to find necessary information quickly. Multiple windows containing information appear from the bottom of the display, moving upward swiftly. We move our eyes to obtain the information in the windows. The speed of the eye movements controls the upward moving speed of the windows so that we can obtain information from the system as fast as possible. If we find anything interesting in a window and stop the eye movements to look at it more carefully, the system stops the window motion, enlarging the window to show details.

This paper presents a user interface for a service robot that can bring the objects asked by the user. Speech-based interface is appropriate for this application, but it alone is not sufficient. The system needs a vision-based interface to recognize gestures as well. Moreover, it needs vision capabilities to obtain the real world information about the objects mentioned in the user's speech. For example, the robot needs to find the target object ordered by speech to carry out the task. This can be considered as a vision assisted speech. However, vision sometimes fails to detect the objects. Moreover, there are objects for which vision cannot be expected to work well. In these cases, the robot tells the current status to the user so that he/she can give advice by speech to the robot. This can be considered as a speech assisted vision through the user. This paper presents how the mutual assistance between speech and vision works and demonstrates promising results through experiments.

In order that robots may achieve tasks in unknown environments, research focusing on the following three points must be conducted: (1) design of commands from the operator to the robots; (2) environmental recognition to show the operator information about the remote work area where robots are working; and (3) interface between a system and the operator and information display of the work area for the operator. For (1), the focus is on the operation of the multiple robots by a single operator. For (2), the problem is divided into two parts; object recognition and self-positioning. As for (3), a graphical user interface is implemented. The operator mainly uses a mouse to input commands, and the operator watches a virtual environment to monitor remote working areas. The virtual environment is constructed with the environment information acquired by the method described in (2) above. After the problems mentioned above are solved, our system will provide a flexible command and monitoring structure and an operator would be able to command the robots appropriately. These are the main advantages of the method we are proposing. We verified the effectiveness of the proposed methods in experiments.

An idea of considering an autonomous mobile robot as a virtual local network system is proposed in developing a remote fault diagnosis system of a mobile robot. Within the developed diagnosis system, a mobile robot is taken as one of the management objects of the network management system for autonomous mobile robots, and the simple network management protocol, which is widely used in network management system, is applied and adapted to the developed system as the communication protocol for exchanging diagnosis information. Moreover, by taking advantage of active moving and sensing ability of autonomous mobile robots, an effective fault inference method is also discussed.

We propose an operation method to control mobile robot groups by a single human user, and build a system in which the human user and robots achieve tasks cooperatively. As for the commanding aspect, we divide orders from the user into four levels and consider the relationship between orders and autonomy of robots required. The main features of our method include: 1) the human user commands not just an individual robot, but groups of robots; and 2) our system has autonomy in proportion to user command levels and complements lack of command for the group (sometimes commands for the group is not sufficient for individual robot control). Regarding the monitoring aspect, we introduce a CCD camera system into our system and display information on a CRT display which shows the image of the work area. The user can monitor a remote environment and simultaneously obtain the information needed during operation. We verify the efficiency of our method and system by experiments.

The paper describes an integrated teleoperation system for maintenance tasks with planning functions based on manipulation skills. We embed planning functions into a telerobotics system to make the system more flexible and robust. A motion teaching system based on contact state transition, a geometric modeling system using teaching trees, and a task execution system based on manipulation skills are integrated. The design concept of the system and essential technologies are described. An experimental task is explained to demonstrate the efficiency of the telerobotics system.