With the rapid development of mobile devices and the fast increase of sensitive data, secure and convenient mobile authentication technologies are desired. Except for traditional passwords, many mobile devices have biometric-based authentication methods (e.g., fingerprint, voiceprint, and face recognition), but they are vulnerable to spoofing attacks. To solve this problem, we study new biometric features which are based on the dental occlusion and find that the bone-conducted sound of dental occlusion collected in binaural canals contains unique features of individual bones and teeth. Motivated by this, we propose a novel authentication system, TeethPass$^+$, which uses earbuds to collect occlusal sounds in binaural canals to achieve authentication. Firstly, we design an event detection method based on spectrum variance to detect bone-conducted sounds. Then, we analyze the time-frequency domain of the sounds to filter out motion noises and extract unique features of users from four aspects: teeth structure, bone structure, occlusal location, and occlusal sound. Finally, we train a Triplet network to construct the user template, which is used to complete authentication. Through extensive experiments including 53 volunteers, the performance of TeethPass$^+$in different environments is verified. TeethPass$^+$achieves an accuracy of 98.6% and resists 99.7% of spoofing attacks.

Bowel sound (BS) is an important physiological signal of the human body, which is also an objective reflection of gastrointestinal motility. However, BS has characteristics of weak signal, strong noise, and randomicity, which bring great challenges to the daily detection of BS. In this paper, we propose BSMonitor, the first BS monitoring system with strong noise-resistant capability via earphones. BSMonitor uses one earphone attached to the abdomen to collect BS signals and the other earphone worn in the ear to collect external noises and internal noises. After eliminating the noises through the Kalman filter and band-pass filter, the signal containing BS is separated via the empirical mode decomposition. Then BSMonitor extracts MFCC features of BS signals and applies a carefully-designed LSTM network to perform highly-accurate BS detection. Finally, an alert mechanism calculates the frequency and duration of detected BS and compares with the normal values to alert users. Furthermore, to increase the amount and diversity of training data, we introduce a data augmentation method, which can further improve the accuracy and generalization of BSMonitor. Through extensive experiments with 18 volunteers, we find that BSMonitor not only achieves high accuracy of BS detection but also has strong generalization across different users and environments. Particularly, BSMonitor achieves accuracy up to 98.73% and 94.56% in thebenchmark experimentsand thecross experiments, respectively.

Drones have witnessed extensive popularity among diverse smart applications, and visual Simultaneous Localization and Mapping (SLAM) technology is commonly used to estimate the six-degrees-of-freedom pose for drone flight control systems. However, traditional image-based SLAM cannot ensure the flight safety of drones, especially in challenging environments such as high-speed flight and high dynamic range scenarios. The event camera, a new vision sensor, holds the potential to enable drones to overcome these challenging scenarios if fused with the image-based SLAM. Unfortunately, the computational demands of event-image fusion SLAM have grown manifold compared with image-based SLAM. Existing research on visual SLAM acceleration cannot achieve real-time operation of event-image fusion SLAM on on-board computing platforms for drones. To fill this gap, we present TrinitySLAM , a high-accuracy, real-time, low-energy consumption event-image fusion SLAM acceleration framework utilizing Xilinx Zynq, an on-board heterogeneous computing platform. The key innovations of TrinitySLAM include a fine-grained computation allocation strategy, several novel hardware–software co-acceleration designs, and an efficient data exchange mechanism. We fully implement TrinitySLAM on the latest Zynq UltraScale+ platform and evaluate its performance on one custom-made drone dataset and four official datasets covering various scenarios. Comprehensive experiments show that TrinitySLAM improves the pose estimation accuracy by 28% with half end-to-end latency and 1.2× energy consumption reduction compared with the most comparable state-of-the-art heterogeneous computing platform acceleration baseline.

Location awareness in environments is one of the key parts for drones’ applications and have been explored through various visual sensors. However, standard cameras easily suffer from motion blur under high moving speeds and low-quality image under poor illumination, which brings challenges for drones to perform motion tracking. Recently, a kind of bio-inspired sensors called event cameras emerge, offering advantages like high temporal resolution, high dynamic range and low latency, which motivate us to explore their potential to perform motion tracking in limited scenarios. In this paper, we propose FlyTracker, aiming at developing visual sensing ability for drones of both individual and circumambient location-relevant contextual, by using a monocular event camera. In FlyTracker, background-subtraction-based method is proposed to distinguish moving objects from background and fusion-based photometric features are carefully designed to obtain motion information. Through multilevel fusion of events and images, which are heterogeneous visual data, FlyTracker can effectively and reliably track the 6-DoF pose of the drone as well as monitor relative positions of moving obstacles. We evaluate performance of FlyTracker in different environments and the results show that FlyTracker is more accurate than the state-of-the-art baselines.

Fitness can help to strengthen muscles, increase resistance to diseases, and improve body shape. Nowadays, a great number of people choose to exercise at home/office rather than at the gym due to lack of time. However, it is difficult for them to get good fitness effects without professional guidance. Motivated by this, we propose the first personalized fitness monitoring system, HearFit$^+$, using smart speakers at home/office. We explore the feasibility of using acoustic sensing to monitor fitness. We design a fitness detection method based on Doppler shift and adopt the short time energy to segment fitness actions. Based on deep learning, HearFit$^+$can perform fitness classification and user identification at the same time. Combined with incremental learning, users can easily add new actions. We design 4 evaluation metrics (i.e., duration, intensity, continuity, and smoothness) to help users to improve fitness effects. Through extensive experiments including over 9,000 actions of 10 types of fitness from 12 volunteers, HearFit$^+$can achieve an average accuracy of 96.13% on fitness classification and 91% accuracy for user identification. All volunteers confirm that HearFit$^+$can help improve the fitness effect in various environments.

Sound-related respiratory symptoms are commonly observed in our daily lives. They are closely related to illnesses, infections, or allergies but ignored by the majority. Existing detection methods either depend on specific devices, which are inconvenient to wear, or are sensitive to noises and only work for indoor environment. Considering the lack of monitoring method for in-car environment, where there is high risk of spreading infectious diseases, we propose a smartphone-based system, named SymListener, to detect respiratory symptoms in driving environment. By continuously recording acoustic data through a built-in microphone, SymListener can detect the sounds of cough, sneeze, and sniffle. We design a modified ABSE-based method to remove the strong and changeable driving noises while saving energy of the smartphone. An LSTM network is adopted to classify the three types of symptoms according to the carefully designed acoustic features. We implement SymListener on different Android devices and evaluate its performance in real driving environment. The evaluation results show that SymListener can reliably detect target respiratory symptoms with an average accuracy of 92.19% and an average precision of 90.91%.

Accurate and real-time instance segmentation on mobile devices enables a wide spectrum of applications such as augmented reality, context-aware inspection and environ-mental cognition. However, the computation resource demanded by instance segmentation impedes its deployment on resource-constrained commercial mobile devices. Prior studies enable smartphones to conduct computational-intensive tasks in real-time with the assistance of an edge server. However, simply applying an edge-assisted framework hardly achieves delightful segmentation performance due to the movements of devices and targets, pixel-level precision requirements, and huge computational overhead even for edge nodes. This work proposes edgeIS, an edge-assisted system that enables real-time and accurate instance segmentation on mobile devices. edgeIS embraces the mobile device sensing ability of surroundings and its own motion, and redesigns an innovative mobile-edge collaboration paradigm suitable for segmentation tasks. We implement edgeIS on a lightweight edge node and different mobile devices. Extensive experiments are conducted under four datasets. The results show that edgeIS can run on mobile devices in real-time and achieve a 0.92 segmentation IoU, outperforming existing state-of-the-art solutions. We further embed edgeIS in an AR-based inspection system deployed in an oil field and the performance of edgeIS meets the demand of the industrial scenario.

With the rapid development of mobile devices and the fast increase of sensitive data, secure and convenient mobile authentication technologies are desired. Except for traditional passwords, many mobile devices have biometric-based authentication methods (e.g., fingerprint, voiceprint, and face recognition), but they are vulnerable to spoofing attacks. To solve this problem, we study new biometric features which are based on the dental occlusion and find that the bone-conducted sound of dental occlusion collected in binaural canals contains unique features of individual bones and teeth. Motivated by this, we propose a novel authentication system, TeethPass, which uses earbuds to collect occlusal sounds in binaural canals to achieve authentication. We design an event detection method based on spectrum variance and double thresholds to detect bone-conducted sounds. Then, we analyze the time-frequency domain of the sounds to filter out motion noises and extract unique features of users from three aspects: bone structure, occlusal location, and occlusal sound. Finally, we design an incremental learning-based Siamese network to construct the classifier. Through extensive experiments including 22 participants, the performance of TeethPass in different environments is verified. TeethPass achieves an accuracy of 96.8% and resists nearly 99% of spoofing attacks.

Speeding is one of the biggest threatens to road safety. However, facilities like radar detector and speed camera are not deployed everywhere, as roads in some areas like campus and residential areas often lack these facilities. Several solutions either depend on pre-deployed infrastructures, or require additional devices, which motivate us to explore the practicability of using smartphones’ acoustic sensors to detect vehicle speed. In this paper, we propose a Hybrid Detection system for vehicle Speed (HDSpeed). We first investigate the relationship between acoustic pattern and vehicle speed. According to our findings on typical patterns of both electric vehicles (EVs) and gasoline vehicles (GVs), we separately extract different features from the acoustic signals of EVs and GVs. A CNN and an LSTMN are designed for training EV and GV models, respectively. Considering that applying neural networks obtains coarse-grained information like a speed section, we propose a detection method based on active acoustic sensing, in which method HDSpeed calculates the fine-grained speed by detecting the distance change between the smartphone and the passing vehicle. In addition, the previously detected speed section can eliminate interferences of surrounding moving objects. Through extensive experiments in real driving environments, HDSpeed achieves an average error of$2.17km/h$.

Driving safety has drawn much public attention in recent years due to the fast-growing number of cars. Smoking is one of the threats to driving safety but is often ignored by drivers. Existing works on smoking detection either work in contact manner or need additional devices. This motivates us to explore the practicability of using smartphones to detect smoking events in driving environment. In this paper, we propose a cigarette smoking detection system, named HearSmoking, which only uses acoustic sensors on smartphones to improve driving safety. After investigating typical smoking habits of drivers, including hand movement and chest fluctuation, we design an acoustic signal to be emitted by the speaker and received by the microphone. We calculate Relative Correlation Coefficient of received signals to obtain movement patterns of hands and chest. The processed data is sent into a trained Convolutional Neural Network for classification of hand movement. We also design a method to detect respiration at the same time. To improve system performance, we further analyse the periodicity of the composite smoking motion. Through extensive experiments in real driving environments, HearSmoking detects smoking events with an average total accuracy of 93.44 percent in real-time.

Fitness can help to strengthen muscles, increase resistance to diseases and improve body shape. Nowadays, more and more people tend to exercise at home/office, since they lack time to go to the dedicated gym. However, it is difficult for most of them to get good fitness effect due to the lack of professional guidance. Motivated by this, we propose HearFit, the first non-invasive fitness monitoring system based on commercial smart speakers for home/office environments. To achieve this, we turn smart speakers into active sonars. We design a fitness detection method based on Doppler shift and adopt the short time energy to segment fitness actions. We design a high-accuracy LSTM network to determine the type of fitness. Combined with incremental learning, users can easily add new actions. Finally, we evaluate the local (i.e., intensity and duration) and global (i.e., continuity and smoothness) fitness quality of users to help to improve fitness effect and prevent injury. Through extensive experiments including over 7,000 actions of 10 types of fitness with and without dumbbells from 12 participants, HearFit can detect fitness actions with an average accuracy of 96.13%, and give accurate statistics in various environments.

The safety of the elderly has attracted much attention nowadays. Among various daily activities, fall is one of the most dangerous events for the elderly, especially those who live alone. Most existing works on fall detection are based on wearable devices, which are inconvenient in using. Several solutions only use coarse-grained Wi-Fi signal information that contains many biases, and lack considerations on environmental changes. These situations motivate us to design a fine-grained and robust fall detection approach. In this article, we propose a fall detection system, called FallViewer, based on analyzing the channel state information (CSI) of Wi-Fi signals. To get fine-grained information, we propose phase and amplitude calibration methods for deviation correction. Then, an adjustment approach for antenna power is designed to eliminate the multipath interference. Furthermore, we apply a double sliding window to get a flexible threshold, which improves the robustness of FallViewer to various environments. Finally, FallViewer extracts features of the processed Wi-Fi signal and sends the features to a LibSVM for classification. Through experiments in different environments, FallViewer can detect fall events with an average accuracy of 95.8%, which indicates that FallViewer can work reliably and effectively.

Drowsy driving is one of the biggest threats to driving safety, which has drawn much public attention in recent years. Thus, a simple but robust system that can remind drivers of drowsiness levels with off-the-shelf devices (e.g., smartphones) is very necessary. With this motivation, we explore the feasibility of using acoustic sensors on smartphones to detect drowsy driving. Through analyzing real driving data to study characteristics of drowsy driving, we find some unique patterns of Doppler shift caused by three typical drowsy behaviours (i.e., nodding, yawning and operating steering wheel), among which operating steering wheels is also related to drowsiness levels. Then, a real-time Drowsy Driving Detection system named D3-Guard is proposed based on the acoustic sensing abilities of smartphones. We adopt several effective feature extraction methods, and carefully design a high-accuracy detector based on LSTM networks for the early detection of drowsy driving. Besides, measures to distinguish drowsiness levels are also introduced in the system by analyzing the data of operating steering wheel. Through extensive experiments with five drivers in real driving environments, D3-Guard detects drowsy driving actions with an average accuracy of 93.31%, as well as classifies drowsiness levels with an average accuracy of 86%.

Mobile Crowd Sensing (MCS) enables the platform to offer data-based service by incentivizing mobile users to perform sensing task and collecting sensing data from them. Most of the existing works on MCS only consider designing incentive mechanisms for a single MCS platform. In this paper, we study the incentive mechanism in MCS with multiple platforms under two scenarios: competitive platform and cooperative platform. We correspondingly propose new competitive and cooperative mechanisms for each scenario. In the competitive platform scenario, platforms decide their prices on rewards to attract more participants, while the users choose which platform to work for. We model such a competitive platform scenario as a two-stage Stackelberg game. In the cooperative platform scenario, platforms cooperate to share sensing data with each other. We model it as many-to-many bargaining. Moreover, we first prove the NP-hardness of exact bargaining and then propose heuristic bargaining. Finally, numerical results show that (1) platforms in the competitive platform scenario can guarantee their payoff by optimally pricing on rewards and participants can select the best platform to contribute; (2) platforms in the cooperative platform scenario can further improve their payoff by bargaining with other platforms for cooperatively sharing collected sensing data.

In this article, we propose points-of-interest (POI)-tagging App-assisted incentive mechanism (PTASIM), an incentive mechanism that explores the cooperation with POI-tagging App for mobile edge crowdsensing (MEC). PTASIM requests App to tag some edges to be POI, which further guides App users to perform tasks at that location. We further model the interactions of users, platform, and App by a three-stage decision process. App first determines the POI-tagging price to maximize its payoff. Platform and users subsequently decide how to determine tasks reward and select edges to be tagged, and how to select the best task to perform, respectively. We analyze the optimal solution in those stages. Specifically, we prove that greedy algorithm could provide the optimal solution for platform's payoff maximization in polynomial time. The numerical results show that: 1) the cooperation with App brings long-term and sufficient participation; and 2) the optimal strategies reduce platform's tasks cost as well as improve App's revenues.

Since the number of cars has grown rapidly in recent years, driving safety draws more and more public attention. Drowsy driving is one of the biggest threatens to driving safety. Therefore, a simple but robust system that can detect drowsy driving with commercial off-the-shelf devices (such as smart-phones) is very necessary. With this motivation, we explore the feasibility of purely using acoustic sensors embedded in smart-phones to detect drowsy driving. We first study characteristics of drowsy driving, and find some unique patterns of Doppler shift caused by three typical drowsy behaviors, i.e., nodding, yawning and operating steering wheel. We then validate our important findings through empirical analysis of the driving data collected from real driving environments. We further propose a real-time Drowsy Driving Detection system (D3-Guard) based on audio devices embedded in smartphones. In order to improve the performance of our system, we adopt an effective feature extraction method based on undersampling technique and FFT, and carefully design a high-accuracy detector based on LSTM networks for the early detection of drowsy driving. Through extensive experiments with 5 volunteer drivers in real driving environments, our system can distinguish drowsy driving actions with an average total accuracy of 93.31% in real-time. Over 80% drowsy driving actions can be detected within first 70% of action duration.

Smart city is a key component in Internet of Things, so it has attracted much attention. The emergence of mobile crowd sensing (MCS) systems enables many smart city applications. In an MCS system, sensing tasks are allocated to a number of mobile users. As a result, the sensing related context of each mobile user plays a significant role on service quality. However, some important sensing context is ignored in the literature. This motivates us to propose a context-aware multiarmed bandit (C-MAB) incentive mechanism to facilitate quality-based worker selection in an MCS system. We evaluate a worker's service quality by its context (i.e., extrinsic ability and intrinsic ability) and cost. Based on our proposed C-MAB incentive mechanism and quality evaluation design, we develop a modified Thompson sampling worker selection (MTS-WS) algorithm to select workers in a reinforcement learning manner. MTS-WS is able to choose effective workers because it can maintain accurate worker quality information by updating evaluation parameters according to the status of task accomplishment. We theoretically prove that our C-MAB incentive mechanism is selection efficient, computationally efficient, individually rational, and truthful. Finally, we evaluate our MTS-WS algorithm on simulated and real-world datasets in comparison with some other classic algorithms. Our evaluation results demonstrate that MTS-WS achieves the highest cumulative utility of the requester and social welfare.

With the rapid increasing of the number of mobile devices and their embedded sensing technologies, mobile crowd sensing (MCS) has become an emerging modern sensing paradigm for performing large-scale urban sensing. One of the key challenges of large-scale mobile crowd sensing systems is how to effectively select the minimum set of appropriate participants from the huge user pool to perform the sensing tasks. The capability of a particular user for certain task depends on many factors, such as her moving pattern/behavior, device capability, sensor quality, or even uploading bandwidth. Many of these information of participants are unknown by the selection mechanism. Therefore, self-learning based approaches have been proposed to learn the users' capability for certain tasks via multiple trials and their online performances. In this paper, we first model the cumulative participant selection problem as a combinational multi-armed bandit problem and present an online selection algorithm which leverages the historical performing records of participants to learn the different capabilities (both sensing probability and time delay) of participants. Further, to consider the cost of switching participant for particular tasks, we then introduce the cumulative participant selection problem with switch costs and propose a corresponding online learning method. For both proposed learning algorithms, we provide regret analysis. In addition, extensive simulations with real-world mobile datasets are conducted for the evaluations of the proposed methods. Our simulation results confirm the effeteness of them.