Touchscreen interfaces have become ubiquitous in modern household appliances, yet they remain largely inaccessible to the blind due to their lack of tactile feedback. Existing accessibility solutions, such as tactile overlays, remote control applications, and proximity-based interfaces, suffer from failing to capture real-time appliance states, limited generalizability, indirect interactions, or strong environmental dependencies. In this work, we introduce SingSprite a novel appliance interaction system that allows blind users to seamlessly identify and control appliances by recognizing their unique acoustic signatures. The key insight is that many appliances emit distinct, device-specific humming sounds during operation, primarily generated by internal power supply components. SingSprite captures these acoustic fingerprints using commodity smartphone microphones, applying a tailored background noise cancellation scheme to enhance signal clarity. To address the issue of low mobile sampling rates, we integrate a Variational Mode Decomposition (VMD) approach with an autoencoder framework for effective feature extraction. The HumNet classifier, which concurrently classifies appliance types and operational states, demonstrates robust performance with an F1 score of 0.95 across a diverse set of 100 appliances. Additionally, user studies conducted with blind participants confirm the system’s usability and its practical effectiveness in real-world scenarios. Additionally, user studies conducted with blind participants confirm the system’s usability and its practical effectiveness in real-world scenarios.SingSprite offers a hardware-free and scalable accessibility solution, providing proximity-aware interaction in smart homes without the need for additional infrastructure, paving the way for more inclusive and intuitive environments for blind users. Using natural device acoustics, SingSprite pioneers a universal, proximity-sensitive interaction paradigm that significantly advances accessibility in smart home environments, without requiring any modifications to existing infrastructure.

Protecting privacy-sensitive hardware usage on mobile devices is crucial. Although mobile operating systems (OSs) and smartphone manufacturers have set the permission settings, attackers can evade these defenses using covert methods, enabling malicious camera recording, microphone eavesdropping, and screen capture. Electronic devices emit unique yet weak electromagnetic interference (EMI) signals when accessing privacy-sensitive hardware. But, these signals are easily affected by foreground application activities and geomagnetic fluctuations caused by device movement. Our prior work showed that supervised learning can extract EMI features correlated with privacy hardware states from complex magnetometer readings, but it requires substantial labeled data, limiting practical deployment to new device models or OS versions. To eliminate this reliance on labeled data, this paper proposes a multimodal contrastive learning framework that leverages the device's built-in magnetometer and synchronized system logs as dual-modal inputs. Through self-supervised training, the framework can learn the intrinsic associations between EMI features and the operating states of privacy-sensitive hardware. Building on this, we design an EMI-based eavesdropping classifier that can analyze a user device's magnetometer readings offline to detect covert eavesdropping activities. Experimental results show that the proposed method can effectively identify eavesdropping behavior related to access to camera, microphone, and screen recording data. Testing across ten diverse mobile devices achieved an average classification accuracy of 89.1% on Android devices and 88.5% on iOS devices for identifying the specific hardware being eavesdropped upon.

Understanding the nature of user-device interactions (e.g., who is using the device and what he/she is doing with it) is critical for many applications including time management, user profiles, and privacy protection. However, in scenarios where mobile devices are shared among family members or multiple employees in a company, conventional account-based statistics are not meaningful. This poses an even bigger problem when dealing with sensitive data. Moreover, fingerprint readers and front-facing cameras were not designed to continuously identify users. In this study, we developedMagPrint++, a novel approach to fingerprint users based on unique patterns in the electromagnetic (EM) signals associated with the specific use patterns of users. Initial experiments showed that time-varying EM patterns are unique to individual users. They are also temporally and spatially consistent, which makes them suitable for fingerprinting.MagPrint++has a number of advantages over existing schemes: i) Non-intrusive fingerprinting, ii) implementation both on COTS mobile phones and a small and easy-to-deploy device, and iii) high accuracy thanks to the proposed classification algorithm. In experiments involving 30 users,MagPrint++achieves$94.3\%$accuracy in classifying users from these traces, which represents a$10.9\%$improvement over the state-of-the-art classification method.

Electroencephalography (EEG) analysis plays a crit-ical role in applications such as brain-computer interfaces, epilepsy detection, and cognitive state recognition. However, EEG data are often limited in volume due to high acquisition costs and exhibit complex spatio-temporal coupling across multiple channels. Convolutional Neural Networks (CNNs) have become the predominant approach for EEG signal analysis, owing to their effectiveness in local feature extraction and compatibility with grid-like sensor topologies. Nevertheless, the locality as-sumption inherent in conventional CNN s restricts their ability to capture functional connectivity and dynamic dependencies between spatially distant channels. To address this limitation, we propose NLCTCN, a novel non-local Temporal Convolutional Network that leverages a hierarchical greedy strategy to identify and exploit long-range correlations in multi-channel time series. We further introduce a new fusion scheme, integrated into an end-to-end lightweight CNN architecture to effectively combine these non-local interactions and optimize their configurations for improved predictive performance. Experimental results are presented on 10 real-world EEG datasets. These datasets cover human physiology, cognitive tasks, and clinical applications. The results show that NLCTCN significantly outperforms state-of-the-art methods. On average, NLCTCN achieves an accuracy improvement of 7.5 %. These results validate the effectiveness and superiority of the proposed approach in modeling non-local spatio-temporal dynamics under data-scarce and multi-channel conditions.

Speech-based interaction systems are widely used in mobile devices like smartphones. With advances in deep neural networks, tasks such as speech emotion recognition (SER) enhance these systems’ user-friendliness. However, deploying SER models on mobile devices is challenging due to their complexity and computational demands. While pruning can reduce complexity, it often compromises accuracy, and hardware accelerators like FPGAs are difficult to integrate into mobile devices. This paper proposes AMSER, a real-time speech emotion recognition framework using signal compression and task offloading. AMSER utilizes logarithmic Mel-filter bank coefficients (Fbank) and singular value decomposition (SVD) for feature extraction and compression. The compressed signal is only 6.25% of the original size, achieving 2.24x faster transfer rates and 55.35% energy savings compared to raw audio transmission. Despite the compression, the features preserve key audio information for text and emotion recognition, performed server-side. Experiments show a WER of 4.68% (Librispeech), 10.69% (CommonVoice), and 69.83% emotion recognition accuracy (IEMOCAP).

Speech-based interaction systems are widely used in mobile devices like smartphones. With advances in deep neural networks, tasks such as speech emotion recognition (SER) enhance these systems user-friendliness. However, deploying SER models on mobile devices is challenging due to their complexity and computational demands. While pruning can reduce complexity, it often compromises accuracy, and hardware accelerators like FPGAs are difficult to integrate into mobile devices. This paper proposes Amser+, a real-time speech emotion recognition framework using signal compression and task offloading. Amser+utilizes logarithmic Mel-filter bank coefficients (Fbank) and singular value decomposition (SVD) for feature extraction and compression. The compressed signal is only 6.25% of the original size, achieving 2.24× faster transfer rates and 55.35% energy savings compared to raw audio transmission. Despite the compression, the features preserve key audio information for text and emotion recognition, performed server-side. Experiments show a WER of 4.68% (Librispeech), 10.69% (CommonVoice), and 72.85% emotion recognition accuracy (IEMOCAP).

Various characteristics of mobile applications (apps) and associated in-app services can reveal potentially-sensitive user information; however, privacy concerns have prompted third-party apps to restrict access to data related to mobile app usage. This paper outlines a novel approach to extracting detailed app usage information by analyzing electromagnetic (EM) signals emitted from mobile devices during app-related tasks. The proposed system, MagSpy, recovers user privacy information from magnetometer readings that do not require access permissions. This EM leakage becomes complex when multiple apps are used simultaneously and is subject to interference from geomagnetic signals generated by device movement. To address these challenges, MagSpy employs multiple techniques to extract and identify signals related to app usage. Specifically, the geomagnetic offset signal is canceled using accelerometer and gyroscope sensor data, and a Cascade-LSTM algorithm is used to classify apps and in-app services. MagSpy also uses CWT-based peak detection and a Random Forest classifier to detect PIN inputs. A prototype system was evaluated on over 50 popular mobile apps with 30 devices. Extensive evaluation results demonstrate the efficacy of MagSpy in identifying in-app services (96% accuracy), apps (93.5% accuracy), and extracting PIN input information (96% top-3 accuracy).

Over the past years, smart ecology has attracted much attention, especially for smart home applications. As a key component, monitoring appliances performs significant impact. However, appliances under monitoring usually contain smart modules such as WiFi or Bluetooth, which are limited to traditional appliances. Existing approaches such as distributed sensing, energy disaggregation, and infrastructure-mediated sensing, require the installation of external hardware or have a limited sensing range. In this study, we developed AUDIOSENSE to leverage the acoustic signal generated by the power supply to monitor electrical appliances throughout the house remotely from a single point. In realizing AUDIOSENSE, we proposed an optimized Variation Mode Decomposition scheme to extract the frequency components, as well as a data augmentation scheme to improve generalizability and enable multi-label classification. In experiments, AUDIOSENSE achieved mAP values of 99.3% in multi-label classification.

The efficiency of human-computer interaction is greatly hindered by the small size of the touch screens on mobile devices, such as smart phones and watches. This has prompted widespread interest in handwriting recognition systems, which can be divided into active and passive systems. Active systems require additional hardware devices to perceive movements of handwriting or the tracking accuracy is not adequate for handwriting recognition. Passive methods use the acoustic signal of pen rubbing and are susceptible to environmental noise (above 60$dB$). This paper presents a novel handwriting recognition system based on vibration signals detected by the built-in accelerometer of smartphones. The proposed scheme is implemented in three stages: signal segmentation, signal recognition, and word suggestion.VibWriteris highly resistant to interferences since the normal environmental noise (below 70$dB$) will not cause the vibration of the accelerometer. Extensive experiments demonstrated the efficacy of the system in terms of accuracy in letter recognition (75.3%), word recognition (86.4%) and number recognition (79%) in a variety of writing positions under a variety of environmental conditions.

Quick response (QR) codes have been widely used in mobile applications, especially mobile payments, such as Alipay, WeChat, PayPal, etc due to their convenience and the pervasive built-in cameras on smartphones. Recently, however, attacks against QR codes have been reported and attackers can capture a QR code of the victim and replay it to achieve a fraudulent transaction or intercept private information, just before the original QR code is scanned. In this study, we enhance the security of a QR code by identifying its authenticity. We propose ScreenID, which embeds a QR code with information of the screen which displays it, thereby the QR code can reveal whether it is reproduced by an adversary or not. In ScreenID, PWM frequency of screens is exploited as the unique screen fingerprint. To improve the estimation accuracy of PWM frequency, ScreenID incorporates a model for the interaction between the camera and screen in the temporal and spatial domains. Extensive experiments demonstrate that ScreenID can differentiate screens of different models, types, and manufacturers and thus improve the security of QR codes.

This study reveals that on-board hardware modules leak electromagnetic (EM) emissions whenever audio or camera data is accessed, and proposes Magdefender scheme that explores the possibility of using the magnetometer built into mobile devices to detect eavesdropping instances by malicious apps and even the unscrupulous phone vendors. However, the target EM signals generated by accessing multimedia data is weak and tends to be buried beneath other noisy EM signals from apps running in the foreground. It is also subject to the external interference from geomagnetic signals generated by the device movement. To cope with the challenges, we adopt a generative adversarial networks (GAN) based model to facilitate the extraction of target EM signals indicating the occurrence of eavesdropping from the overall magnetometer readings. We also develop a neural network-based classifier with triplet loss embedding to identify the EM signals from the camera and/or microphones. Empirical results demonstrate the efficacy of MagDefenderin recognizing instances of eavesdropping on cameras/microphones data, with average accuracy of 97.3% when applied to the trained devices, and average 91.5% on unseen mobile devices.

Quick response (QR) codes have been widely used in mobile applications due to its convenience and the pervasive built-in cameras on smartphones. Recently, however, attacks against QR codes have been reported that attackers can capture a QR code of the victim and replay it to achieve a fraudulent transaction or intercept private information, just before the original QR code is scanned. In this study, we enhance the security of a QR code by identifying its authenticity. We propose SCREENID, which embeds a QR code with information of the screen which displays it, thereby the QR code can reveal whether it is reproduced by an adversary or not. In SCREENID, PWM frequency of screens is exploited as the unique screen fingerprint. To improve the estimation accuracy of PWM frequency, SCREENID incorporates a model for the interaction between the camera and screen in the temporal and spatial domains. Extensive experiments demonstrate that SCREENID can differentiate screens of different models, types, and manufacturers, thus improve the security of QR codes.

The efficiency of human-computer interaction is greatly hindered by the small size of the touchscreens on mobile devices, such as smart phones and watches. This has prompted widespread interest in handwriting recognition systems, which can be divided into active and passive systems. Active systems require additional hardware devices to perceive movements of handwriting or the tracking accuracy is not adequate for hand-writing recognition. Passive methods use the acoustic signal of pen rubbing and are susceptible to environmental noise (above 60dB). This paper presents a novel handwriting recognition system based on vibration signals detected by the built-in accelerometer of smart phones. VibWriter is highly resistant to interference since the normal environmental noise will not cause the vibration of the accelerometer. Extensive experiments demonstrated the efficacy of the system in terms of accuracy in letter recognition (76.15%) and word recognition (88.14%) when dealing with words of various lengths written by various users in a variety of writing positions under a variety of environmental conditions.

Various characteristics of mobile applications (apps) and associated in-app services have been used reveal potentially-sensitive user information; however, privacy concerns have prompted third-party apps to rigorously restrict access to data related to mobile app usage. This paper outlines a novel approach to the extraction of detailed app usage information based on analysis of the electromagnetic (EM) signals emitted from mobile devices when executing app-related tasks. Note that this type of EM leakage becomes high-complex when multiple apps are used simultaneously and is subject to interference from geomagnetic signals generated by device movement. This paper proposes a deep learning-based multi-label classification system to identify apps and in-app services based on magnetometer readings. The proposed MAGTHIEF system uses accelerometer and gyroscope data to cancel out the offset in geomagnetic signals followed by an elaborate deep region convolution neural network (DRCNN) to differentiate among multiple apps and the corresponding inapp services. Experiments on 50 apps demonstrated the efficacy of MAGTHIEF in identifying multiple apps and in-app services, achieving high average macro F1 scores of 0.87 and 0.95, respectively. MAGTHIEF also achieved time duration accuracy of 89.5% in recognizing app trajectory in the real-world scene.

Understanding the nature of user-device interactions (e.g., who is using the device and what he/she is doing with it) is critical to many applications including time management, user profiles, and privacy protection. However, in scenarios where mobile devices are shared among family members or multiple employees in a company, conventional account-based statistics are not meaningful. This poses an even bigger problem when dealing with sensitive data. Moreover, fingerprint readers and front-facing cameras were not designed to continuously identify users. In this study, we developed MagPrint, a novel approach to fingerprint users based on unique patterns in the electromagnetic (EM) signals associated with the specific use patterns of users. Initial experiments showed that time-varying EM patterns are unique to individual users. They are also temporally and spatially consistent, which makes them suitable for fingerprinting. MagPrint has a number of advantages over existing schemes: i) Non-intrusive fingerprinting, ii) implementation using a small and easy-to-deploy device, and iii) high accuracy thanks to the proposed classification algorithm. In experiments involving 30 users, MagPrint achieves 94.3% accuracy in classifying users from these traces, which represents an 10.9% improvement over the state-of-the-art classification method.

Quick response (QR) codes have been widely used in mobile applications, due to its convenience and the pervasive built-in cameras on smartphones. Recently, however, QR codes have been reported suffering attacks for being sniffed just before the QR code is scanned, which lead to financial loss. In this study, we propose ScreenID, for enhancing the QR code security by identifying its authenticity, which embeds a QR code with information of unique screen fingerprint - PWM frequency. PWM frequencies are adjusted to different values by screen manufacturers, therefore can successfully differentiate screens. To improve the estimation accuracy of PWM frequency, ScreenID incorporates a model for the interaction between the camera and screen in the temporal and spatial domains. Extensive experiments demonstrate that ScreenID can differentiate screens of different models, types and manufacturers and thus improve the security of QR codes.

Quick response (QR) codes are becoming pervasive due to their rapid readability and the popularity of smartphones with built-in cameras. QR codes are also gaining importance in the retail sector as a convenient mobile payment method. However, researchers have concerns regarding the security of QR codes, which leave users susceptible to financial loss or private information leakage. In this study, we addressed this issue by developing a novel QR code (called mQRCode), which exploits patterns presenting a specific spatial frequency as a form of camouflage. When the targeted receiver holds a camera in a designated position (e.g., directly in front at a distance of 30 cm from the camouflaged QR code), the original QR code is revealed in form of a Moire pattern. From any other position, only the camouflaged QR code can be seen. In experiments, the decryption rate of mQRCode was > 98.6% within 10.2 frames via a multi-frame decryption method. The decryption rate for cameras positioned 20° off axis or > 10cm away from the designated location dropped to 0%, indicating that mQRCode is robust against attacks.

Quick response (QR) codes are becoming pervasive due to their rapid readability and the popularity of smartphones with built-in cameras. QR codes are also gaining importance in the retail sector as a convenient mobile payment method. However, researchers have concerns regarding the security of QR codes, which leave users susceptible to financial loss or private information leakage. In this study, we address this issue by developing a novel QR code (called mQR code), which exploits patterns presenting a specific spatial frequency as a form of camouflage. When the targeted receiver holds a camera in a designated position (e.g., directly in front at a distance of 30 cm from the camouflaged QR code), the original QR code is revealed in form of a Moiré pattern. From any other position, only the camouflaged QR code can be seen. In experiments, the decryption rate of mQR codes is $> 98%$. The decryption rate for cameras positioned $20\degree$ off axis or $> 10cm$ from the designated location drops to $0%$, indicating that any attackers will be unable to steal a usable image.