In industrial production, the orientation of facility components can indicate whether the facility is on a regular operating track. For example, when a component gets loose, the orientation variation of the component would exceed the normal range. A common approach for orientation measurement is to attach an inertial measurement unit (IMU) to the target device. However, the IMU requires additional power maintenance. This article presents LightGyro, a cheap and efficient batteryless scheme to measure the orientation, in which we attach a reflective film to the target device and use a camera to capture the light spot on the reflective film. The basic idea of LightGyro is to extract the light spots in the captured frame and use their pixel coordinates to infer the orientation. It is difficult to recognize a single light spot, because the spot lacks distinctive features. To solve the problem, we switch light sources on and off to regulate the appearance of light spots and utilize frame subtraction to extract light spots. The depth of field of light spot is lost in the process of camera projection, which is necessary for the orientation measurement. To address the issue, we propose a light array-based reflection model to extract the depth of field from the relative positions of multiple light spots. To the best of our knowledge, this is the first work to utilize reflection to measure orientation. Experiment results show that the orientation error of LightGyro decreases with the increasing length of the reflection route and the orientation error can achieve less than 1 ˆ .

With the proliferation of voice assistants, microphone-based speech recognition technology usually cannot achieve good performance in the situation of multiple sound sources and ambient noises. In this paper, we propose a novel mmWave-based solution to perform speech recognition to tackle the issues of multiple sound sources and ambient noises, by precisely extracting the multi-modal features from lip motion and vocal-cords vibration from the single channel of mmWave. We propose a difference-based method for feature extraction of lip motion to suppress the dynamic interference from body motion and head motion. We propose a speech detection method based on cross-validation of lip motion and vocal-cords vibration so as to avoid wasting computing resources on nonspeaking activities. We propose a multi-modal fusion framework for speech recognition by fusing the signal features from lip motion and vocal-cords vibration with the attention mechanism. We implemented a prototype system and evaluated the performance in real test-beds. Experiment results show that the average speech recognition accuracy is 92.8% in realistic environments.

Voice assistant has been widely used for human-computer interaction and automatic meeting minutes. However, for multiple sound sources, the performance of speech recognition in voice assistant decreases dramatically. Therefore, it is crucial to separate multiple voices efficiently for an effective voice assistant application in multi-user scenarios. In this paper, we present a novel voice separation system using a 2D microphone array in multiple sound source scenarios. Specifically, we propose a spatial filtering-based method to iteratively estimate the Angle of Arrival (AoA) of each sound source and separate the voice signals with adaptive beamforming. We use BeamForming-based cross-Correlation (BF-Correlation) to accurately assess the performance of beamforming and automatically optimize the voice separation in the iterative framework. Different from cross-correlation, BF-Correlation further performs cross-correlation among the after-beamforming voice signals processed with each linear microphone array. In this way, the mutual interference from voice signals out of the specified direction can be effectively suppressed or mitigated via the spatial filtering technique. We implement a prototype system and evaluate its performance in real environments. Experimental results show that the average AoA error is 1.4 degree and the average ratio of automatic speech recognition accuracy is 90.2% in the presence of three sound sources.

In industrial production, the orientation of facility is a powerful indicator to verify whether the facility is in a normal operating track. In this paper, we present LightGyro, a cheap and efficient batteryless scheme to measure the facility orientation, it leverages the orientation amplification effect of reflection to improve the measuring accuracy to one degree. LightGyro system is composed of low-cost camera, batteryless reflective film and LEDs. In the working process of LightGyro, we attach a reflective film to the target and let it reflect the light from LEDs to the camera. Then the LightGyro would extract the LED-related spots in the captured frame and restore the reflection route to measure the orientation. To extract the LED-related spots from complicated background automatically, we propose to leverage the affine transformation to search for the topology of multiple spots which is related to the deployed LED array. To address the dimension missing issue caused by camera projection and restore the reflection route, we propose a light array-based reflection model to extract the missing dimension from relative positions of multiple spots. To the best of our knowledge, this is the first work to utilize light reflection to measure orientation. Our experiments show that the average accuracy of LightGyro achieves less than 2◦. When the reflective film is far from the camera, the mean error is less than 1◦.

Among all the road accidents, speeding is the most deadly factor. To reduce speeding, it is essential to devise efficient schemes for ubiquitous speed monitoring. Traditional approaches either suffers from using special equipment(e.g., radar speed gun) or special deployment(e.g., position-fixed cameras). In this article, we propose SpeedTalker, a mobile phone-based approach to perform speed detection on automobiles. By leveraging the built-in microphones and camera from the mobile phone, SpeedTalker estimates the automobile speed by passively sensing the acoustic and image signals. We propose an integrated solution to effectively estimate the automobile’s speed based on COTS devices, and provide a platform for every pedestrian to help report the speeding event of automobiles. Specifically, we use the time difference of arrivals (TDOA) model based on acoustic signals to figure out the candidate trajectories of automobile, and use the pin-hole model based on image frames to figure out the vertical distance between the user’s position and the automobile’s trajectory, thus to estimate the unique trajectory. Combined with the time stamp of the trajectory, the automobile speed can be estimated. Besides, we propose a method to effectively mitigate the influence of the movement jitters of mobile phone. We implemented a system prototype for SpeedTalker and estimated the automobile speed with high accuracy. Experiment results show that in the scenario of single automobile, SpeedTalker can achieve an average estimation error of 6.1 percent compared to radar speed guns. In the scenario of multiple automobiles, SpeedTalker can achieve an average estimation error of 9.8 percent, which is acceptable for usage.

Gait rehabilitation is a common method of postoperative recovery after the user sustains an injury or disability. However, traditional gait rehabilitations are usually performed under the supervision of rehabilitation specialists, which implies that the patients cannot receive adequate gait assessment anytime and anywhere. In this article, we propose GaitTracker, a novel system to remotely and continuously perform gait monitoring and analysis by three-dimensional (3D) skeletal tracking in a wearable approach. Specifically, this system consists of four Inertial Measurement Units (IMU), which are attached on the shanks and thighs of the human body. According to the measurements from these IMUs, we can obtain the motion signals of lower limbs during gait rehabilitation. By adaptively synchronizing coordinate systems of different IMUs and building the geometric model of lower limbs, the exact gait movements can be reconstructed, and gait parameters can be extracted without any prior knowledge. GaitTracker offers three key features: (1) a unified 3D skeletal model to depict the precise gait movement and parameters in 3D space, (2) a coordinate system synchronization scheme to perform space synchronization over all the IMU sensors, and (3) an automatic estimation method for the user-specific geometric parameters. In this way, GaitTracker is able to accurately perform 3D skeletal tracking of lower limbs for gait analysis, such as evaluating the gait symmetry and the gait parameters including the swing/stance time. We implemented GaitTracker and evaluated its performance in real applications. The experimental results show that, the average error for skeleton angle estimation, joint displacement estimation, and gait parameter estimation are 3∘, 2.3%, and 3%, respectively, outperforming the state of the art.

In typical scenarios for indoor localization and tracking, it is essential to accurately track the pedestrians when they are crossing the connections of different spaces. In this paper, we propose a magnet barcode scanning-based solution for indoor pedestrian tracking. We assemble multiple magnet bars into magnet arrays as a unique magnet barcode, and deploy different magnet barcodes at different connections to label them. We embed an inertial measurement unit (IMU) into the pedestrian`s shoes. When the pedestrian crosses these connections, the magnetometer from the IMU scans the magnet barcode and recognize its corresponding ID. In this way, indoor pedestrian tracking can be regarded as a process of continuously scanning different magnet barcodes. By performing correlation analysis on these barcodes, the trace of pedestrian can be effectively depicted in the indoor map. To build a unique magnet barcode based on the magnet bar arrays, we provide an optimized structure for building the magnet barcode. To tackle the diversities of the pedestrian's gait traces in identifying the magnet barcode, we provide a generalized model based on the space axis for magnet barcode identification. As far as we know, this is the first work to use the magnet bar array to construct the magnet barcode for indoor pedestrian tracking. The real experiment results show that our system can achieve an average accuracy of 88.9% in identifying the magnet barcodes and an average accuracy of 93.1 % for indoor pedestrian tracking.

We propose a new approach for 3D reconstruction of dynamic indoor and outdoor scenes in everyday environments, leveraging only cameras worn by a user. This approach allows 3D reconstruction of experiences at any location and virtual tours from anywhere. The key innovation of the proposed ego-centric reconstruction system is to capture the wearer's body pose and facial expression from near-body views, e.g. cameras on the user's glasses, and to capture the surrounding environment using outward-facing views. The main challenge of the ego-centric reconstruction, however, is the poor coverage of the near-body views - that is, the user's body and face are observed from vantage points that are convenient for wear but inconvenient for capture. To overcome these challenges, we propose a parametric-model-based approach to user motion estimation. This approach utilizes convolutional neural networks (CNNs) for near-view body pose estimation, and we introduce a CNN-based approach for facial expression estimation that combines audio and video. For each time-point during capture, the intermediate model-based reconstructions from these systems are used to re-target a high-fidelity pre-scanned model of the user. We demonstrate that the proposed self-sufficient, head-worn capture system is capable of reconstructing the wearer's movements and their surrounding environment in both indoor and outdoor situations without any additional views. As a proof of concept, we show how the resulting 3D-plus-time reconstruction can be immersively experienced within a virtual reality system (e.g., the HTC Vive). We expect that the size of the proposed egocentric capture-and-reconstruction system will eventually be reduced to fit within future AR glasses, and will be widely useful for immersive 3D telepresence, virtual tours, and general use-anywhere 3D content creation.