Accurate indoor drone four degrees-of-freedom (4DoF) estimation, including both 3D position and heading, is essential for safe flight in GNSS-denied environments, opening up opportunities for fully autonomous IoT applications such as infrastructure inspection and surveillance. Approaches using a built-in camera are attractive due to their low cost and ease of deployment, however, existing methods face key limitations. Visual-Inertial Odometry suffers from drift and often fails in textureless environments, while Visible Light Positioning requires a direct line of sight to ceiling LEDs, which is challenging for drones with a limited field of view. We propose ReFDrone, a system that simultaneously estimates 3D position and heading from a single image using a downward-facing rolling-shutter (RS) landing camera and floor-reflected light from modulated ceiling LEDs. Our core insight is that RS distortion produces spectral lobes in the frequency domain whose orientation directly encodes the angle of arrival (AoA) of the LED. Leveraging this property, ReFDrone extracts AoA from a single RS image, enabling estimation of 2D position and heading from just two LEDs, and derives altitude using a virtual-view image with a distance model. We evaluated ReFDrone under 1296 conditions in 2.5 m x 2.5 m testbed, achieving a 90th percentile 3D position error of 0.335 m and a heading error of 8.101 degrees, demonstrating its accuracy and practicality.

Beamforming with speaker arrays allows for directional audio communication, selectively delivering information to users in specific locations. However, this approach typically requires a specialized array hardware. We propose an adaptive acoustic spot communication method targeting moving receivers that utilizes only two commercial off-the-shelf (COTS) speakers. A key feature of our method is its ability to achieve high-performance by predicting the receiver’s position. We conducted real-world experiments with a moving microphone to evaluate the effectiveness of the proposed method. The results showed a demodulation success rate of 94.7% for the adaptive spot communication, excluding the region directly in front of the speakers, demonstrating the feasibility of adaptive spot communication for a moving target.

Smartphone positioning utilizing a single indoor speaker and its reflections offers the advantage of low deployment cost. However, unlike the direct wave, reflected waves are not always stably detectable, even under Line-of-Sight (LOS) conditions, leading to inconsistent positioning performance. Additionally, associating each detected reflection with its corresponding virtual speaker poses a challenge. This study proposes a method to improve positioning performance by tracking the smartphone based on hypotheses about the identity of each detected reflection. Evaluations were conducted in a real-world environment along both rectangular and circular paths, using only the smartphone’s bottom built-in microphone. The results demonstrated 90th percentile errors of 0.62 m and 0.77 m for the rectangular and circular paths, respectively. This accuracy, approximately equivalent to a step length, suggests potential for navigation applications. These results represent error reductions of 55% and 46%, respectively, compared to conventional approaches.

Earphone-type wearable devices, also known as “hearables,” will have many functions in the future. Some of those functions will require authentication of the wearer for access to the user's privacy information or settlement of payments. In this study, we propose a new personal authentication system for hearables called EarLock. EarLock authenticates the wearer by acquiring and analyzing ear canal and auricle shape information using sound leakage from the device. The system can be implemented using a speaker and external microphone that are highly compatible with hearables. We implemented three prototype devices and investigated EarLock's authentication performance under various practical scenarios, including walking conditions, noisy environments, and situations with object interference. Experimental results showed that the in-ear, open-ear, and bone-conduction devices achieved balanced accuracy (BAC) scores of 87.2–93.7%, 83.4–94.7%, and 85.9–90.0%.

We present a visible light positioning (VLP) method that allows indoor positioning using a smartphone camera with a narrow field of view (FoV). Our method employs omni-directional LEDs, each modulated at a unique frequency, mounted on the ceiling. The system calculates the 2-D position and azimuth of the smartphone by analyzing the brightness of ceiling reflection captured by the user’s smartphone front camera. This approach does not necessitate a direct line-of-sight to the LEDs, effectively overcomes the FoV limitations, and enables positioning over a wider space. We model the ceiling reflection as a transmission path between the LEDs and the ceiling, improving coverage and accuracy of estimation for the user’s position and azimuth compared to previous work. In experiments with two LEDs placed 2.6 m apart, our method covered a space over ten times larger than traditional methods, achieving coverage of $50 \mathbf{m}^{2}$ with a mean absolute error (MAE) in the positioning of 0.46 m and an MAE in azimuth estimation of 13.0°.

Low-cost and accurate indoor location information can add spatiotemporal context to information systems, enabling new location-aware applications. Time-of-Flight (ToF)-based acoustic localization using speakers and microphones allows for localization accuracy within a few tens of centimeters, outperforming RF-based techniques. However, ToF-based localization requires synchronization between the speaker and microphone, i.e., the time offset between them must be known. Previous time offset estimation methods required custom hardware for speakers, limiting their practical use. Estimating the time offset using a single, unmodified speaker is essential for leveraging widely deployed speakers and enhancing coverage. This paper presents the first method for time offset estimation using a single speaker and a microphone, enabled by two key factors: (i) a time offset computation method that utilizes higher-order floor-ceiling reflections as multiple geometrically-constrained virtual speakers, and (ii) a signal processing pipeline that isolates these critical reflections from numerous others by leveraging the speaker's frequency-dependent radiation pattern. Experiments show that the proposed technique can achieve time offset estimation with a 90th percentile error of 259 μs at a 5 m distance. Furthermore, we implemented a ToF localization system based on SyncEcho, demonstrating a 11.0 cm localization accuracy with a 90th percentile error.

This study investigates an indoor 3D positioning method for a microphone using a single speaker. The proposed method estimates the distance, azimuth, and elevation of the microphone from the speaker and calculates the 3D position of the microphone. Multiple short signals with different frequency bands were transmitted sequentially at sufficient intervals to avoid the influence of indoor reflected waves. We extracted these signals at the receiver side, and their amplitude spectra were computed. The azimuth and elevation of the microphone were estimated by matching the obtained amplitude spectrum with the reference data previously measured at each azimuth and elevation. The distance was estimated from amplitude attenuation. The 3D positioning performance was evaluated at ten points in a practical environment. The results show that the 90-th percentile value of the error was 0.583 m.

This paper proposes a tracking system that integrates visible-light positioning and pedestrian dead reckoning using a smartphone’s built-in ambient light sensor and inertial sensors. By correcting the accumulated errors in the position estimation via ambient light sensing, the proposed system can achieve impressive tracking performance. The challenges of this approach and techniques to alleviate the various problems are described. Experiments demonstrate that the proposed system can achieve an average tracking error of 0.89 m. Issues to be investigated further are discussed, which will enable the system to be deployed in places such as museums and shopping malls.

In this paper, we propose an unauthorized photography detection technique by using a frequency spectrum obtained from a photo under light-emitting diode illuminations emitting multi-carrier modulated signals. We utilize characteristics of a rolling-shutter camera as a frequency filter whose function depends on its shutter opening ratio. By using one test photo image, the proposed method can judge whether it was taken with an authorized or unauthorized camera. When the value of an authorized shutter opening ratio is updated periodically or randomly on an authorized camera only, it is possible to use a frequency spectrum of a photo taken by the camera as its own one-time fingerprint. Technical details of the proposed method are described and its characteristics are established through controlled and laboratory-setting experiments. The experimental results confirm that the proposed method can achieve an f1-score of 99.1% and an area under the receiver operating characteristic curve of 0.9997. Causes of correct and wrong detection results are investigated and the robustness of the proposed method is discussed using compressed and resized photo images. Issues related to limitations and deployment for real-world applications are described.

Positioning with a small number of anchors is an important issue in the study of acoustic indoor positioning systems (AIPS). We present PT-Sync, a novel approach to time-synchronize between a commercial off-the-shelf (COTS) speaker and a mobile device by leveraging acoustic sensing and reflected signals from the floor. PT-Sync enables ranging with one speaker and 2-D positioning with two speakers without any additional hardware. Our proposed time synchronization method requires height information from the floor in the calculation process, which we estimate by active acoustic sensing. Using a unique averaging technique and IMU enables robust height estimation. PT-Sync can be used in a variety of indoor environments and can be time-synchronized with already installed speakers. The results of the evaluation experiment confirmed that a synchronization error of 0.16 ms was achieved even at a distance of 6 m from the speaker, and that synchronization on the order of microseconds could be achieved. Furthermore, pedestrian tracking experiments confirmed that positioning of less than 38.9 cm can be achieved at the 90th percentile.

In this paper, we propose a method for the 2D positioning of a smartphone by receiving sinusoidally modulated light signals emitted by ceiling-mounted LEDs. An ambient light sensor (ALS) mounted on the smartphone measures the reflected illumination from the floor. The advantage of this combination is that it consumes very little power, does not require the implementation of additional devices, and removes constraints on the placement of the LEDs caused by the field of view. To the best of our knowledge, this method of using the ALS on a smartphone with reflected light from the floor has not been proposed before. We conducted several experiments to investigate the effectiveness and limitations of our method for visible-light positioning. One experiment with different LED frequencies and floor materials showed that, for multiple LEDs emitting modulated light at around 60 Hz and highly reflective floor materials, positioning is possible with a 90th-percentile error of 35.5 cm. Other experiments involved different types of ambient light and different LED arrangements. Based on the results of these experiments, we can show how to improve the positioning performance. We also discuss future developments in real environments.

In this paper, we describe NL-Beep, which is a novel ranging method between multiple smartphones using acoustic sensing in non-line-of-sight (NLOS) environments. We need to know the distance between people to maintain social distancing because of the COVID-19 pandemic. Acoustic-based ranging methods that use a smartphone built-in speaker and microphone represent one approach to high-accuracy ranging methods. These methods can be used in line-of-sight (LOS) environments where there are no obstacles between the smartphones. However, in daily life, there are usually many obstacles, including pedestrians, and NLOS environments are the norm rather than the exception. Therefore, we propose the NL-Beep system for NLOS environments. To measure the distance between two smartphones without using a direct signal, NL-Beep efficiently uses a reflected signal from the ceiling in an indoor environment. It can also detect LOS and NLOS environments based on the profile of the received signal, and adapt the method used to estimate the distance between the smartphones. In our experiments, we used two smartphones in a room and estimated the distance between them for several placement configurations. In our results, we obtained 90th-percentile errors of less than 11.97 cm for the distances between the smartphones. In this paper, we also mention the effectiveness and limitations of NL-Beep.

In this paper, we propose a 2D localization method using a single speaker and a microphone. Our proposed method estimates the direction of the microphone based on the knowledge that the spectral amplitude of a speaker in each direction is different. Additionally, the distance between the speaker and microphone is estimated based on the received amplitude. The 2D location of the microphone is calculated from the estimated direction and distance values. We conducted real-environment experiments for two different types of speakers and confirmed that the 90-percentile localization error was 0.265 m. We also discuss the systematic errors in the distance estimation that occurred in the experiment, revealing that these errors were caused by the characteristics of the speakers.

This paper proposes a spot-communication method that can direct the spot to the user location adaptively. In the proposed method, the first transmitted signals are used to estimate user location, which is then reflected in the second transmitted signals; therefore, the spot directed to the user can be generated instantaneously. As the signals consist of mutually orthogonal sinusoidal waves, multiple spots can be generated simultaneously, and the shape of the spots can be controlled by overlapping them. Through real environmental experiments using four speakers, we confirmed that our proposed method can generate adaptive spots. Moreover, in the case that appropriate spots were not generated, comparisons were carried out with respect to the theoretical values to determine the cause.

In this study, we propose a new input method for mobile and wearable computing using facial expressions. Facial muscle movements induce physical deformation in the ear canal. Our system utilizes such characteristics and estimates facial expressions using the ear canal transfer function (ECTF). Herein, a user puts on earphones with an equipped microphone that can record an internal sound of the ear canal. The system transmits ultrasonic band-limited swept sine signals and acquires the ECTF by analyzing the response. An important novelty feature of our method is that it is easy to incorporate into a product because the speaker and the microphone are equipped with many hearables, which is technically advanced electronic in-ear-device designed for multiple purposes. We investigated the performance of our proposed method for 21 facial expressions with 11 participants. Moreover, we proposed a signal correction method that reduces positional errors caused by attaching/detaching the device. The evaluation results confirmed that the f-score was 40.2% for the uncorrected signal method and 62.5% for the corrected signal method. We also investigated the practical performance of six facial expressions and confirmed that the f-score was 74.4% for the uncorrected signal method and 90.0% for the corrected signal method. We found the ECTF can be used for recognizing facial expressions with high accuracy equivalent to other related work.

The following topics are dealt with: indoor radio; indoor navigation; wireless LAN; smart phones; learning (artificial intelligence); pedestrians; Global Positioning System; radionavigation; mobile computing; Kalman filters.

We propose a method to recognize road surface conditions using footsteps and inertial data. In areas where the road surface conditions change significantly with the seasons and weather, bad road conditions cause dangerous such as falls. If the road surface condition can be determined in advance, danger can be averted by selecting safe routes and suitable shoes. In this study, we focus on the footsteps and inertial data that change depending on road surface conditions, such as dry pavement, puddle, soil, and mud. We implemented the prototype device and evaluated the proposed method on six road surface conditions with eight participants. The evaluation results confirmed that the recognition accuracy was 83.0% in a low-noise environment. When there was noise, we compared the standard approach, which combines footsteps and inertial data, and the revised method, which changes the confidence of the result of footstep recognition by the signal-noise ratio (SNR). The evaluation results also confirmed that the recognition rate increased by a maximum of 16.4% using the revised method (when the SNR was 1 dB, the average accuracy was improved from 37.5% to 53.9%).

This paper describes a high-speed optical camera communication (OCC) technique using an LED and a rolling-shutter camera. In the proposed technique, the symbols being transmitted are encoded as time delays of optimally modulated signals derived theoretically. A receiver decodes the symbols by using intensities obtained from four consecutive line sensors of a camera. Experiments using a camera having performance similar to that of a general-purpose camera show that the proposed technique can achieve 0.833 ~ 1.17 bits per line sensing and that symbol transmission is possible with a longer exposure time setting; this is difficult to achieve using existing on-off keying OCC techniques.

In this paper, we describe a novel position-recognition method that uses passive acoustic signals from two previously installed speakers (passive acoustic sensing) and active acoustic signals from a smartphone's loudspeakers (active acoustic sensing). In passive acoustic sensing, a locus of positions for the smartphone can be calculated from the measured time difference of arrival from the two installed speakers. In active acoustic sensing, a chirp signal is transmitted from the speakers of the smartphone, and the distance to the side wall is measured from the propagation time of arrival at its microphone. We can obtain the smartphone position from our proposed model equations by combining these two results. In our experiments, we installed speakers at intervals of 10 m along a corridor and estimated the smartphone position at several places. From these results, we obtained 90th percentile errors of less than 0.224 m for 2-D positioning. We found that multipaths from the side wall were causing the positioning error in passive acoustic sensing, and the variance of the positioning error using the top microphone which was omnidirectional was smaller than the bottom one. When we introduced a weighting based on the result of the active acoustic sensing and the difference in the performance between microphones, the 90th percentile errors were reduced to less than 0.134 m.

This paper describes an indoor positioning technique using a video camera that captures LED light reflected by the floor. Indoor positioning for mobile devices can be very useful. In particular, localization techniques using LEDs and cameras, so-called visible light positioning, are known to be effective and have high accuracy. However, existing methods have the constraint that they must capture the light source directly. This requires a high-performance processor and a high-resolution image. However, light sources cannot always be detected directly (loss of signal: LOS). Our proposal aims to solve these problems by estimating the position of a camera that does not face the light directly but monitors light reflected by the floor. Specifically, individual LED ceiling lights emit sinusoidal waves modulated with different frequencies, and the camera captures the overlapped light from the LEDs reflected by the floor then demodulates the signal. The camera need not seek the ceiling lights directly from an image, unlike existing methods. The position can be estimated using any part of the image because usually the ceiling light is reflected by the whole floor. Experimental results show that the proposal requires less than 1/100 as many pixels for localization as existing methods and the position can be estimated within 0.4 m at the 90th percentile in a 2.5 m square room. We show that the cause of errors is mainly the difference between the LED diffusion model and the actual light diffusion, the occlusion and the noise and movement of the camera. Overcoming these problems remains as our future work.

In this paper, we propose a selective visible light communication method using a single light source and multiple video cameras by using their differences in reception sensitivity between carrier waves composed of OFDM signals. In our evaluation experiments, the transmitted signals were composed of 50th, 75th, 78th, and 80th order carrier waves encoding symbols modulated by 4, 8, and 16 PSK. The exposure time ratio for each camera was set to different values so that symbols encoded in one of the carrier waves could be decoded successfully, but erroneously in the other waves. The selectivity of individual carrier waves as theoretically derived was confirmed through the experimental results.

We describe a time-of-arrival-(ToA-) based localization system for smartphones. In this system, the transmitter emits modulated light-emitting diode (LED) light and sound waves, then the smartphone catches them. The smartphone measures the time of flight of sound waves and localizes its position using multilateration. The LED light is used for visible light communication, which also conveys the time reference of the sound emission. Using the time reference, we can synchronize between the transmitter and the receiver, then the ToA-based localization can be available. The precision of time synchronization is the key factor for localization based on ToA. Hence, we have proposed the SyncSync method using a modulated LED light and a smartphone video camera, which enables ToA localization by measuring the time of flight of sound waves. This method gives better results than those of time-difference-of-arrival localization. However, we had to use a dedicated light synchronization device for our method. Visible light communication (VLC) is becoming a popular application of smartphones. If VLC demodulation could be used for time synchronization in acoustic localization, VLC and indoor localization would be integrated into a single application. In this paper, we examined the feasibility of VLC time synchronization for localization. Then, ToA-based localization was performed using a smartphone application. The standard deviation of the 3D localization was around 100 mm in a dark room, which is sufficiently precise for practical applications.

The paper describes a rapid and accurate time-synchronization technique for smartphones using their built-in cameras and its preliminary evaluations for application development.

This paper describes a rapid and flicker-free visible light communication technique that uses an off-the-shelf video camera. Transfer rates at least 53% faster than existing methods were achieved in experiments. Features of the proposed method are discussed via theoretical analysis.

We have developed a full-body interaction game that allows children to cooperate and interact with other children in small groups. The aim of the game developed in this study is to encourage cooperation between children. The game requires two children to jump together at the same time. In a series of experiments, we observed children using several strategies to coordinate the timing of their jumps, such as shouting to one another and watching each other. The results of a questionnaire to evaluate the proposed interactive game indicate that children strategize to cooperate with each other while enjoying the game. Therefore, the game enables children to cooperate with others using bodily movements.

We are developing an art-appreciation system that will provide learning information about pictures while generating interest in the pictures themselves. This system can provide learning information on pictures by talking to an artist or people portrayed in the pictures using voice recognition. Furthermore, the system operates through movement and location of the user, and it generates sensation such that the user feels he/she is in the pictures, thus actively appreciating the pictures. As the first step toward this system, we have developed a system that provides learning information to the user by actively talking using ukiyo-e. Here, we quantitatively evaluate this system for primary schoolchildren and determine whether the user is interested in the pictures while talking to the pictures using voice recognition measured by electrodermal activity. In this paper, we summarize the current system and describe two evaluation results.

We have developed a full-body interaction game that allows children to cooperate and interact with other children in small groups. The intent of the full-body interaction game developed in this study is to encourage cooperation between children. The game requires two children to jump together with the same timing. We let children experience the game, and observed the children using several strategies to coordinate the timing of their jumps. These included shouting time, watching each other, and jumping in a constant rhythm as if they were skipping rope. In this manner, we observed the children playing the game while cooperating with each other. Therefore, the game enabled children to experience cooperation with others by using body movements.

We are developing an immersive learning support system for paleontological environments in museums. The system measures the physical movement of the learner using a Kinect sensor, and provides a sense of immersion in the paleontological environment by adapting the surroundings according to these movements. As the first step toward realizing an immersive learning support system for museums, Yoshida et al. developed and evaluated a prototype system. However, their system cannot learn about certain paleontological features such as the names of extinct animals and their characteristics. Therefore, we developed an improved version of this system that allows for an enriched knowledge of paleontological environments, focusing in particular on extinct animals and plants and the ecological environment. Here, we evaluate the system’s learning assistant and immersive features insofar as they are directed toward children in primary school. This paper summarizes the current system and describes the evaluation results.

This paper describes a spot-controllable data-transfer method. The proposed method generates a beam-shaped spot using two commercial off-the-shelf speakers. In our method, a symbol consists of a pair of sinusoidal waves having different angular frequencies. The width and direction of a beam-shaped spot are controlled by the angular-frequency difference between the sinusoidal waves and the transmission-time difference between the two speakers. Multiple spots can be generated by transmitting multiple pairs of sinusoidal waves based on the principle of orthogonal frequency-division multiplexing. By over-lapping multiple beam-shaped spots, the locations and sizes of the areas enabled to receive data are controllable. Experiments using four speakers and computer simulation show that the proposed method can generate controllable spots. An analysis of the errors in a real indoor environment indicate that they are caused by multipath signals, radiation damping of transmitted signals, and the incident/output angle characteristics of the microphone and speakers.

This paper proposes a method for separating the voice of a signal person from the voices of multiple speakers' in a mixed recording. When the sounds of multiple speakers are present when a voice recognition system is operating, the quality of speech recognition decreases. The proposed method separates the sound of multiple speakers using signal processing techniques and a microphone array. The position of the speaker is also required; however, the microphone array cannot be used for this purpose. Consequently, the position of the speaker was determined using the Kinect V2 depth image sensor. An experiment was performed to separate a signal voice from many voices. The results of the experiment show that, it is possible to separate two voices using the proposed technique.

This paper presents a support system to assist learning within museums. Museums represent important places of scientific learning for children. However, because the main learning method in museums is the study of displayed specimens and their corresponding explanations, the motivation of a child to learn is low. Hence, we are developing an experience-based learning support system such as entertainment that will enable learners to enhance their motivation to learn in any museum. We develop two systems to realize experience-based entertainment. One is "Live Biblia," which is a digital installed book system that uses real fossil. Another is the immersive learning system we called "Body Experience and Sense of Immersion in Digital paleontological Environment" or "BESIDE," which enables learners to explore a virtual paleontological environment. In this paper, we present a summary of the current system of Live Biblia and BESIDE.

In this paper, we propose and develop a full-body interaction system and simulation game called â??Human SUGOROKU,â?? which helps elementary school students learn about vegetation succession while having fun. We found that the students became more involved in the game because they were required to play it using their body movements. An experiment conducted with students verified that the participants became immersed in the virtual world of vegetation succession while playing Human SUGOROKU. This paper describes the structure of our game and the results of its evaluation.

In this paper we describe SyncSync, a novel time-of-arrival (ToA) localization method for smartphones. ToA measurements generally show better precision than time-difference-of-arrival measurements, but ToA systems require a synchronization mechanism between anchor and mobile nodes. For this synchronization, we employ modulated light with an acoustic signal for the time-of-flight distance measurement. These are detected by the smartphone's video camera and microphone. The time resolution in consumer video cameras is typically only a few tenths of a second, but by utilizing a CMOS image sensor's rolling shutter effect we obtain synchronization resolutions of a few microseconds, sufficient for precise acoustic ToA measurement. Experiments confirm operation of the system with localization errors within 10 mm.

This paper presents a rapid and accurate time-synchronization technique for acoustic localization. Two LED arrays for modulated illumination and an off-the-shelf camera are used. An equation that specifies the time difference between them accurately and precisely is proposed. Experiments in real environments show that the proposed technique using rectangular signals can achieve time synchronization to within 29.4 μs (5.94 μs standard deviation) and acoustic ranging to within 9.9 mm (2.01 mm standard deviation) for a measurement period of 0.1 s. Because of the smaller standard deviation in comparison with the error, it is indicated that there exist systematic errors to be removable. This technique will enable the implementation of an acoustic 3D localization system for mobile devices such as tablet PCs and smartphones based on time-of-arrival trilateration, which is more accurate than positioning taking a time-difference-of-arrival approach. Theoretical analyses for modulated illumination considering a camera exposure time clarify how an optimally modulated signal is designed for achieving rapid and accurate time synchronization.

This study proposes a system for supporting the Shotoku Taishi game, which is a language play game that uses the voice of children. The Shotoku Taishi game is a group game in which multiple people presenting a problem vocalize different words at the same time and the respondents are required to guess what the combination of the words is. The authors developed and implemented a system using a microphone array to extract the voice of a specific person presenting a problem in this game. The participants were 36 elementary school students whose native language was Japanese. The results showed that the participants were enjoying the Shotoku Taishi game and that this group activity was a creative activity that deepened their awareness of the Japanese language.

We describe a novel time-synchronization technique for mobile smart nodes. We capture modulated LED light with a video camera, which is usually built into a smart node. The CMOS image sensor of a video camera does not take a snapshot at a certain time. Instead, the sensor captures data on a line-byline basis and the sensor output consists of lines taken at slightly different times. Therefore, we can extract time information from the image. This can be used for time synchronization for time-of-arrival (ToA) measurements. In this paper, we describe the fundamentals and an experiment using light-synchronized acoustic ToA measurements with mobile smart nodes. The acknowledged precision of the obtained time information is equivalent to 5.8 mm with an airborne sound wave.

In this study, we developed a simulation game called "Human SUGOROKU" that consists of a full-body interaction system to enable elementary school students to enjoy and learn vegetation succession. The students' sense of immersion is improved by enabling them to play this game using their body movements. We conducted an experiment with the students and investigated the affects of the full-body interaction through interviews. The results showed that the full-body interaction promotes a sense of immersion in the game. This paper describes the structure of this system and the interview results.

We developed a full-body interaction digital game “Human SUGOROKU”. Th is game enables elementary school students to enjoy and learn vegetation succession by playing simulation game with their body movement. We conducted this game to elementary school students and effects of the system were investigated with questionnaires. The result showed that the full-body interaction promotes a sense of immersion in the game. This paper describes the structure of this game and the questionnaire results.

The purpose of this study was to investigate the ‘Pocket Plant Guide’ qualitatively through interviews. We allowed Japanese sixth-grade elementary school students to use the ‘Pocket Plant Guide’ in order to identify and observe plants. We interviewed the students to investigate their subjective impressions of using this guide. The results indicated that this guide was effective in supporting the identification and observation of indicator plants.

We describe an indoor localization technique for smart phones. Our new method, called the Frequency Division Multiplexing Phase Accordance Method (FDM-PAM), uses a beat called a sync pattern composed of a pair of sinusoidal waves with slightly different frequencies, which is similar to our original ultrasound ranging technique called the Phase Accordance Method (PAM). By generating multiple sync patterns with different central frequencies and transmitting them from different speakers, FDM-PAM conducts time-difference-of-arrival (TDOA) multilateration for localizing smart phones. In the current implementation of FDM-PAM, the 2D indoor position of a smart phone can be estimated. Three sync patterns are generated by using two out of six sinusoidal waves with frequencies ranging from 14.75 kHz to 17.25 kHz. The transmission of the sync pattern from the speakers lasts 4 ms. Through experiments, we have confirmed that FDM-PAM achieves accuracy of around 10 cm using only a short burst transmission, which indicates that the localization technique is sufficiently rapid and accurate.

We are developing a digital game, "Human SUGOROKU," in which enables game players as pieces to learn vegetation succession. In order to realize this game, it is necessary to robustly measure 3D position of people and identify them. In this paper, we proposed a method for 3D position measurement and identification of people by integrating ultrasonic sensors and Kinect sensors. The evaluation results show that proposed method is more robust than methods using ultrasonic sensor and Kinect sensor separately.

The authors developed a smartphone-based Pocket Plant Guide to support the outdoor learning of vegetation succession, and evaluated the utility of this device by administering a questionnaire to 35 elementary school children. The results indicated that the Pocket Plant Guide not only functions as a supportive learning tool, but also offers a user-friendly interface.

This paper presents a novel framework for developing automatic animation systems, which accept a 3D model that is created at runtime. Previous systems cannot deal with such a 3D model because it needs to be prepared by a manual process (rigging) that may not be suitable for nonprofessional users. The proposed framework solves this problem by employing an automatic rigging algorithm. Our algorithm can generate an animation skeleton for a given 3D model automatically, including the anatomical meaning of each joint. The relationship between motion data and this animation skeleton is created by identifying the corresponding joints in the motion data and the skeleton. The motion data for each joint is transferred automatically to the 3D model that has been rigged via our automatic rigging algorithm. Because all processes can be completed without any user intervention, an animation system for nonprofessional users is therefore available. We also discuss several motion editing techniques that can be used to generate new motion data without complex processing.

We propose a novel technique for 3D localization that integrates a single camera and ultrasound. We use the Extended Phase Accordance Method and the ultrasound to measure accurately the distance to a moving target and we use the camera to identify the target's 2D position on the image plane. A prototype system consists of a transmitter unit mounting one ultrasound transmitter and three infrared LEDs around it, and a receiver unit with one inexpensive camera and one ultrasound receiver. We implemented these units in a lightweight and compact way (receiver unit size: 55 mm × 44 mm), to make the system robust against the no-line-of-sight problems that frequently occur in trilateration or multicamera-based systems. Experimental results show that the RMSEs of the proposed system are 1.20 mm and 1.66 mm for static and mobile (velocity: 1.0 m/s) targets, respectively. These indicate that the performance of the system is comparable with that of high-end systems.

ABSTRACT Rigging is a process for creating skeletons used to animate articulated characters. In conventional computer‐animation software, this process must be performed manually. Although several automatic rigging algorithms have been proposed, these methods still require user intervention. This paper proposes an automatic algorithm that generates an inverse kinematic skeleton for a character by locating an appropriate template skeleton on the extracted curve skeleton of the input 3D character model. After the curve skeleton is extracted, it is analyzed and classified into an appropriate category. The classification conditions are developed from the characteristics of each kind of real animal. We also develop an algorithm to extract the anatomical meaning of each skeleton segment. On the basis of the classification result, a suitable template skeleton is retrieved from the database. Each bone of the template skeleton can then be located on the appropriate skeleton segment of the input skeleton graph by using the extracted anatomical meanings. In contrast to previous methods, the algorithm does not require the input 3D character models to have certain poses or orientations. Moreover, all processes can be completed without user intervention. Copyright © 2012 John Wiley & Sons, Ltd.

In this paper, we propose an innovative motion-capture system using ultrasonic communications. Compared with existing commercial motion-capture systems that use optical or magnetic sensing, the proposed system can provide a cost-effective solution for industrial and entertainment applications. To design and implement the system, a distance-estimation method called the Extended Phase Accordance Method (EPAM), which can measure the distance to a moving object with a standard deviation of less than 1 mm, was devised. To improve the capture rate of the proposed system, the EPAM algorithm was implemented in a field-programmable gate array (FPGA). The current version of the system conducts motion capture using five markers attached to a user. It can work at around 10 frames per second (fps), with an error of less than 55 mm and a standard deviation of 42 mm. This demonstrates a moderate level of accuracy, which will be useful for several applications.

The checkerboard pattern is widely used in computer vision techniques for camera calibration and simple geometry acquisition, both in practical use and research. However, most of the current techniques fail to recognize the checkerboard pattern under distorted, occluded or discontinuous conditions, especially when the checkerboard pattern is dense. This paper proposes a novel checkerboard recognition technique that is robust to noise, surface distortion or discontinuity, supporting checkerboard recognition in dynamic conditions for a wider range of applications. When the checkerboard pattern is used in a projector camera system for geometry reconstruction, by using epipolar geometry, this technique can recognize the corresponding positions of the crossing points, even if the checkerboard pattern is only partly detected.

We describe a fast and accurate indoor localization technique using the multiple signal classification (MUSIC) algorithm. This algorithm is known as a high-resolution method for estimating directions of arrival (DOAs) or propagation delays. A critical problem in using the MUSIC algorithm for localization is its computational complexity. Therefore, we devised a novel algorithm called Time Space additional Temporal MUSIC (TSaT-MUSIC), which can rapidly and simultaneously identify DOAs and delays of multicarrier ultrasonic waves from transmitters. Computer simulations have proved that the computation time of the proposed algorithm is almost constant in spite of increasing numbers of incoming waves, and is faster than that of existing methods based on the MUSIC algorithm. Experiments in real environments showed that the standard deviation of position estimations in 3D space is less than 10 mm, which is satisfactory for indoor localization.

An ultrasonic localization system is described in the paper. To the best of our knowledge, this is the first system that can simultaneously identify not only the 3D position, but also the velocity of a moving object. The proposed system uses an original and innovative method called Extended Phase Accordance Method (EPAM) that can precisely identify the distance between an ultrasonic microphone and a moving transmitter by rapidly estimating the frequency shift of the transmitted signal. One remarkable feature of the proposed system is the use of a single compact receiver unit, which will reduce deployment labor and costs. Experiments demonstrate that the proposed system shows the 3D position and velocity estimation with sufficient accuracy.

This paper presents a correspondence matching technique for a dense checkerboard pattern displayed by a projector-camera system for one-shot geometry acquisition purposes. It does not require color coding or complicated spatial encoding techniques to encode the corresponding positions of corners, and can find corresponding positions for an incomplete checkerboard pattern. We introduce a combination of epipolar geometry and topology constraints in the checkerboard pattern to solve correspondence ambiguities. To verify the feasibility of the technique, we have created a prototype scanning system that can construct the 3D geometry of a scenario for each image frame. The results of our experiments show that the technique can identify correspondences for a checkerboard pattern displayed on discontinuous surfaces and can reconstruct 3D geometry structures in real time.

The vector-field-based method is one of the 3D curve-skeleton extraction algorithms. Typically, critical points in the vector field inside 3D objects are connected to form the curve-skeleton. However, critical points usually do not distribute to all important parts of the 3D object. Therefore, other features are used to produce a reliable result. Although this strategy can deliver a curve-skeleton that captures all of the important parts, the curve-skeleton usually comes with unnecessary segments. This paper proposes the skeleton-growing algorithm that automatically produces the curve-skeleton with small amounts of such segments. It searches for a set of high-curvature boundary voxels as starting points to find a set of suitable seed points that will be used to grow the curve-skeleton. We propose an unnecessary segment removal algorithm that can reduce the skeleton-noise density. A direction-selection algorithm is developed to avoid searching in irrelevant directions. The proposed method can produce a single reliable result curve-skeleton that could be applied in many different applications, including matching, animation, and visualization.

This paper describes a system called GENTORO that uses a robot and a handheld projector for supporting children's storytelling activities. GENTORO differs from many existing systems in that children can make a robot play their own story in a physical space augmented by mixed-reality technologies. A user study conducted in collaboration with elementary school children and teachers indicates that GENTORO's features can enhance children's embodied participation in, and their level of engagement with, their storytelling activities, and can support children in designing and expressing creative and original stories

This paper presents a new structured light pattern generation technique, which allows acquisition of scene geometry from a single image, under various projection conditions with relatively high acquisition density. We construct the pattern by using the geometrical constraints of the projector-camera system, and solve the correspondence problem based on the local connectivity of the feature points and edges of the projection pattern. This method is robust from image processing perspective, and allows rapid geometry scanning using a mobile projector-camera system.

In this paper, a system called GENTORO that supports children's storytelling activities is proposed. By using GENTORO, children can make a robot play their story in the real world augmented with mobile projected graphical images. Preliminary user studies have been conducted in collaboration with elementary school children and university students, in order to investigate the acceptance level of GENTORO and clarify its design requirements, respectively. The development of GENTORO is in progress. The future plans of the GENTORO project are discussed.

We propose a novel dynamic display approach using a handheld projector embedded with an ability to be aware of obstructing objects, called clutters. One camera is fixed on a projector to retrieve surface information. By integrating multiple target tracking knowledge using particle filters and gabor filters, the appearance and disappearance of unknown clutters are monitored. As a result, the unknown number of clutters can be tracked efficiently, while spurious objects are filtered out. No computation effort for tracking will be expended unless those clutters are persistently detected. At every time step, the projection target area adapts itself to suit with current situations of clutters and a projector. The biggest undistorted target area is placed on the clutter-free area adaptively with respect to the previous locations. The simulation results revealed that the system is effective for creating clutter-aware dynamic projection when a projector and clutters are moving in an unpredictable manner.

This paper presents a framework for a computationally adaptive projection metaphor using a handheld projector inside a dynamic cluttered environment. In addition to conventional self-correcting projection features, the framework uses multiple clutter tracking and adaptive target generation to define the clutter-aware target area for projection in a reliable manner. Using a paired projector-camera system, the framework first builds high spatial frequency feature maps using a Laplacian pyramid approach. The feature maps are then passed to a rejection step to eliminate spurious features caused by contents of the projected image. After the resulting features representing clutters are processed by the appropriated design tracker, the target area for projection is generated. Finally, the desired information for projection is rendered and sent back to the projector. The framework can be used effectively for a clutter-aware handheld projector-based system without the need for a complex hardware setup or with any prior need to clean up the environment.

In this paper, a semi-automatic realtime calibration technique for a handheld projector is described. The proposed technique always keeps a shape of a projected screen rectangular on a specified projection surface, while a user continuously changes his standing position or the pose of the projector. The technique is especially useful for a future mobile phone with a built-in projector that allows a user to project its screen onto any surface in any location and share the screen with multiple people. Informal evaluations using the technique have been conducted to identify the level of its acceptability by users and to find problems to be improved. An example entertainment application to explore possibilities of the proposed technique has been developed.

We developed a new system called CarettaKids that supports face-to-face collaborative learning by children. CarettaKids uses a sensing board based on the Radio Frequency Identification (RFID) technology to support collaboration in a shared space, and the Personal Digital Assistant (PDA) device to support activity in personal spaces. We also introduced this system into an actual classroom environment to evaluate its performance in support for children's collaborative learning, by analyzing the childrens' interaction. As a result, we have confirmed that CarettaKids's feature of transition between two spaces, makes it possible for children to reflect on problems detected in the shared space so as to find solutions in their respective personal space, and to engage in an active exchange of opinions in the shared space, based on ideas generated from personal-space learning.

This paper describes a new manipulation technique for small-screen mobile devices. The proposed technique, called HybridTouch, uses a touchpad attached to the rear surface of a PDA. A user can manipulate the PDA by simultaneously touching the front surface with a stylus pen held by the dominant hand and the rear surface with a finger of the nondominant hand. User studies were conducted via applications augmented by HybridTouch, and proved that users could perform manipulation tasks intuitively.

This correspondence describes a system called ePro for supporting face-to-face group activities. ePro connects a sensor-embedded board that utilizes radio frequency identification (RFID) technology for object recognition and a computer simulation, and is currently used to discuss urban planning and environmental problems. Users collaboratively construct a town by placing pieces such as houses on the board. The computer simulation program automatically recognizes the arrangement of pieces on the board. It then shows environmental changes of the town through visualized simulation results. The goal of ePro is to initiate group discussions: The physical board supports the participation of each user by allowing them to manipulate physical objects, and the computer simulation gives users feedback on their manipulation, which triggers their further actions. A comparative user study of ePro (with and without the sensor-embedded board) was conducted. Through the study, ePro has proved its high usability and effectiveness in activating discussions in face-to-face group activities

In this paper, a system called Caretta that integrates personal and shared spaces to support face-to-face collaboration is described. We use PDAs and a multiple-input sensing board for personal and shared spaces, respectively. Users of Caretta can discuss and negotiate with each other in the shared space by manipulating physical objects, while they individually examine their ideas in their own personal spaces. Caretta allows users to participate in group activities interchangeably and seamlessly using both these spaces. Caretta is applicable to various collaborative tasks. In this paper, it supports users in urban planning tasks. User studies of Caretta demonstrated that it allowed users to collaborate in a flexible fashion: users could work individually in their personal spaces at their own pace, cooperatively work together in the shared space, and smoothly transition between both of the spaces.

The authors have so far developed a system that supports collaborative learning and evaluated it in school education. The system integrates physical and virtual worlds to teach urban planning and environmental problems by game playing by the pupils themselves. Intensive experiments with the system have proven that it can support the externalization of learners' thinking, active participation, interactions among them, and discussions in a learning situation. The paper addresses the effects of collaborative learning, and discusses how pupils discover the way of thinking of other pupils during game play.

Many systems that support group activities utilize computer and network technologies so that people in different places or at different times can communicate with each other. In this paper, we propose a system called ePro for supporting face-to-face collaboration by integrating physical and virtual worlds. In order to enhance interactions among group members and raise their engagement, ePro connects a sensor-embedded board and a computer simulation. In our current implementation, ePro is used to discuss urban planning and environmental problems. Group members collaboratively construct a town by placing a piece, such as a house, on a sensor-embedded board (physical world). The computer simulation program automatically recognizes the arrangement of pieces on the board. It then visualizes environmental changes to the town on the board through simulations (virtual world). The visualization shown to the group members amplifies the interaction among them and gives them feedback for re-designing the town on the board. In order to evaluate the effects of supporting group activities by combining physical and virtual worlds, preliminary experiments with ePro were carried out.

The authors have built and evaluated several systems to assist creative concept formation by professional engineers and scientists. Through experiments, it has become clear that a system that can reveal different viewpoints automatically is strongly needed by many users to support their creative activities. We are all surrounded by an almost infinite amount of information. If we can elicit different viewpoints from large information sources, we can arrive at new understandings that could not have been possible through discussions with other persons alone. This paper presents a system that automatically elicits and visualizes different viewpoints of authors concerning certain topics from a text database of journal and conference papers. Users can review different viewpoints of a topic of interest by looking at a space configured by the system. Through interaction with the system, users are expected to build their own concepts creatively. The effect of promoting creative concept formation and other benefits, such as enhanced information retrieval and knowledge sharing, have been validated by several experiments.

A document retrieval system for researchers in the field of science and technology is described. The main feature of the system is the visualization of the semantic relations between documents and interactive operation to it. The system has a test database composed of a large number of journal and conference papers. It elicits keywords from each paper by an automatic indexing algorithm and then visualizes the relations between the papers and their keywords in a metric space by a statistical method. We carried out several experiments. Through them we have confirmed that our system is effective because it can assist the users in their query formation and modification while they use it. Our system can also assist the users in their creative research because they can find the relations between the papers and their keywords which they haven't noticed so far. In the experiment on effectiveness of retrieval we have confirmed that our system can realize high recall and precision ratio.