Wi-Fi sensing utilizing Channel State Information (CSI) has emerged as a promising non-invasive technique for environmental perception, but current approaches are hindered by task-specific architectures, limited generalization, and data inefficiency, impeding its versatility across diverse applications. We introduce WiSenseNet, a novel foundation model for multitask Wi-Fi sensing that addresses these challenges. Inspired by large language models, WiSenseNet adapts architectures like Mamba to process CSI data, employing a hybrid design of self-attention and state space layers to capture complex spatiotemporal dependencies in Wi-Fi signals. Our model demonstrates remarkable versatility across diverse sensing tasks, including gesture recognition, gait analysis, human activity recognition, and occupancy detection. Rigorous experiments on multiple benchmark datasets reveal WiSenseNet’s superior performance: in gesture recognition, it achieves 97.4% accuracy with only 797.70K Multiply-Accumulate Operations (MACs), surpassing both CNN and Transformer baselines by 3.6% and 1.8% respectively, while reducing computational complexity by 66%. This efficiency extends across tasks, with WiSenseNet achieving 95.4% accuracy in human activity recognition and 94% in gait analysis. For human presence sensing, WiSenseNet attains 89.7% accuracy with 462.34K MACs, demonstrating its ability to extract meaningful features from subtle CSI variations. Our comprehensive ablation study reveals task-specific optimal configurations, elucidating the relationship between model complexity and sensing performance. WiSenseNet represents a significant advancement in Wi-Fi sensing, offering a unified, scalable solution that outperforms tasks-pecific models while maintaining computational efficiency.

Accurate device-to-device distance estimation is crucial for Internet-of-things (IoT) applications. Traditional methods, such as RSSI-based ranging and Time-of-Flight narrowband systems, exhibit limitations. Bluetooth Low Energy (BLE)-based phase ranging, aka Channel Sounding is a preferred technology, but existing approaches can be improved further. This paper proposes a novel approach using a parametric neural network to improve BLE channel sounding performance for device-to-device localization. Our neural network optimizes range estimation by learning from raw channel sounding data, outperforming traditional super-resolution algorithms. We demonstrate significant improvements in accuracy and precision through experimental results, with a reduction in root mean square error of up to 0.4m (indoor) and 0.04m (outdoor) and corresponding standard deviation reductions of up to 0.19m (indoor) and 0.02m (outdoor). This approach enhances device-to-device localization in IoT applications using BLE, enabling a wide range of IoT applications.

Wi-Fi Sensing is emerging as a transformative paradigm in the realm of smart environments, enabling the ubiquitous detection of human presence and the identification of activities within indoor spaces. This paper presents WiFiAct, which leverages a 20 MHz 1 transmit 1 receive (1T1R) Wi-Fi monitor to achieve state-of-the-art generalization results. Our methodology capitalizes on a custom preprocessing pipeline and harnesses the power of self-supervised learning frame-work utilizing a Bayesian Convolutional Neural network (BCNN) and novel contrastive augmentation techniques. Our custom pre-processing pipeline is capable of extracting environment invariant features from Wi-Fi signals, enhancing the expressive power of the subsequent classifier. The proposed self-supervised methodology utilizes a combination of unlabeled and labeled data to effectively learn representations that enable accurate geofenced activity recognition amid uncertainties. Through extensive experimentation, we showcase the proposed solution’s generalization capabilities, paving the way for innovative applications in presence detection and activity identification within smart environments.

This article proposes a novel preprocessing technique for radar-based short-range gesture sensing using a frequency modulated continuous wave (FMCW) radar. The preprocessing is lightweight and works without Fourier transformation. The signal after preprocessing represents the backscattering central dynamics of the hand as a complex-valued time signal of a point target. It is shown that the proposed processing provides competitive classification results compared to conventional frequency domain-based solutions, while being less computationally intensive and having better generalization performance. The preprocessed time domain signal preserves a high-temporal resolution of the hand movement. Due to this fact, it is possible to integrate a periodic control gesture into the system. In doing so, the system not only detects that a gesture is performed continuously and periodically, but also estimates its speed. This is an essential property for controlling scalable parameters, such as brightness or volume, at different speeds. The real-time capability was proven on a Raspberry Pi 3B with an ARM Cortex-A53 CPU. The proposed processing causes a CPU utilization of only 6%. The neural network (NN) inference is done within 75 ms with a classification accuracy of 96.7%.

Data selection is essential for any data-based optimization technique, such as Reinforcement Learning. State-of-the-art sampling strategies for the experience replay buffer improve the performance of the Reinforcement Learning agent. However, they do not incorporate uncertainty in the Q-Value estimation. Consequently, they cannot adapt the sampling strategies, including exploration and exploitation of transitions, to the complexity of the task. To address this, this paper proposes a new sampling strategy that leverages the exploration-exploitation trade-off. This is enabled by the uncertainty estimation of the Q-Value function, which guides the sampling to explore more significant transitions and, thus, learn a more efficient policy. Experiments on classical control environments demonstrate stable results across various environments. They show that the proposed method outperforms state-of-the-art sampling strategies for dense rewards w.r.t. convergence and peak performance by 26% on average.

Abstract One of the key design requirements for any portable/mobile device is low power. To enable such a low powered device, we propose an embedded gesture detection system that uses spiking neural networks (SNNs) applied directly to raw ADC data of a 60GHz frequency modulated continuous wave radar. SNNs can facilitate low power systems because they are sparse in time and space and are event-driven. The proposed system, as opposed to earlier state-of-the-art methods, relies solely on the target’s raw ADC data, thus avoiding the overhead of performing slow-time and fast-time Fourier transforms (FFTs) processing. The proposed architecture mimics the discrete Fourier transformation within the SNN itself avoiding the need for FFT accelerators and makes the FFT processing tailored to the specific application, in this case gesture sensing. The experimental results demonstrate that the proposed system is capable of classifying 8 different gestures with an accuracy of 98.7%. This result is comparable to the conventional approaches, yet it offers lower complexity, lower power consumption and faster computations comparable to the conventional approaches.

In this paper, we introduce the Label-Aware Ranked loss, a novel metric loss function. Compared to the state-of-the-art Deep Metric Learning losses, this function takes advantage of the ranked ordering of the labels in regression problems. To this end, we first show that the loss minimises when datapoints of different labels are ranked and laid at uniform angles between each other in the embedding space. Then, to measure its performance, we apply the proposed loss on a regression task of people counting with a short-range radar in a challenging scenario, namely a vehicle cabin. The introduced approach improves the accuracy as well as the neighboring labels accuracy up to 83.0% and 99.9%: An increase of 6.7% and 2.1% on state-of-the-art methods, respectively.

Contactless detection of hand gestures with radar has gained a lot of attention as an intuitive form of human-computer interface. In this paper, we propose an air-writing system, writing of linguistic characters or words in free space by hand gesture movements using a network of milli-meter wave radars. Most of the works reported in the literature are based on deep learning approaches, which in some cases can involve prohibitively large computational/energy costs making them undesirable for edge IoT devices, where energy efficiency is the prime concern. We propose a highly energy-efficient air-writing system using spiking neural networks, where the trajectory of the character created by fine range estimates together with trilateration from a network of radars are recognized and classified by a spiking neural network (SNN). The proposed system achieves a similar level of classification accuracy (98.6%) compared to the state-of-the-art deep learning methods for 15 characters containing 10 alphabets (A to J) and 5 numerals (1 to 5). Additionally, the proposed SNN model is of 3.7 MB in size making it memory efficient in terms of storage. We demonstrated the proposed method in real-time using a network of 60-GHz frequency-modulated continuous wave radar chipset.

Human activity recognition (HAR) using IMU sensors, namely accelerometer and gyroscope, has several applications in smart homes, healthcare and human-machine interface systems. In practice, the IMU-based HAR system is expected to encounter variations in measurement due to sensor degradation, alien environment or sensor noise and will be subjected to unknown activities. In view of practical deployment of the solution, analysis of statistical confidence over the activity class score are important metrics. In this paper, we therefore propose XAI-BayesHAR, an integrated Bayesian framework, that improves the overall activity classification accuracy of IMU-based HAR solutions by recursively tracking the feature embedding vector and its associated uncertainty via Kalman filter. Additionally, XAI-BayesHAR acts as an out of data distribution (OOD) detector using the predictive uncertainty which help to evaluate and detect alien input data distribution. Furthermore, Shapley value-based performance of the proposed framework is also evaluated to understand the importance of the feature embedding vector and accordingly used for model compression.

Nowadays, Deep Learning (DL) methods often overcome the limitations of traditional signal processing approaches. Nevertheless, DL methods are barely applied in real-life applications. This is mainly due to limited robustness and distributional shift between training and test data. To this end, recent work has proposed uncertainty mechanisms to increase their reliability. Besides, meta-learning aims at improving the generalization capability of DL models. By taking advantage of that, this paper proposes an uncertainty-based Meta-Reinforcement Learning (Meta-RL) approach with Out-of-Distribution (OOD) detection. The presented method performs a given task in unseen environments and provides information about its complexity. This is done by determining first and second-order statistics on the estimated reward. Using information about its complexity, the proposed algorithm is able to point out when tracking is reliable. To evaluate the proposed method, we benchmark it on a radar-tracking dataset. There, we show that our method outperforms related Meta-RL approaches on unseen tracking scenarios in peak performance by 16% and the baseline by 35% while detecting OOD data with an F1-Score of 72%. This shows that our method is robust to environmental changes and reliably detects OOD scenarios.

Nowadays, deep neural networks are widely used in a variety of fields that have a direct impact on society. Although those models typically show outstanding performance, they have been used for a long time as black boxes. To address this, Explainable Artificial Intelligence (XAI) has been developing as a field that aims to improve the transparency of the model and increase their trustworthiness. We propose a retraining pipeline that consistently improves the model predictions starting from XAI and utilizing state-of-the-art techniques. To do that, we use the XAI results, namely SHapley Additive exPlanations (SHAP) values, to give specific training weights to the data samples. This leads to an improved training of the model and, consequently, better performance. In order to benchmark our method, we evaluate it on both real-life and public datasets. First, we perform the method on a radar-based people counting scenario. Afterward, we test it on the CIFAR-10, a public Computer Vision dataset. Experiments using the SHAP-based retraining approach achieve a 4% more accuracy w.r.t. the standard equal weight retraining for people counting tasks. Moreover, on the CIFAR-10, our SHAP-based weighting strategy ends up with a 3% accuracy rate than the training procedure with equal weighted samples.

Keyword spotting poses several challenges due to acoustic disturbances such as noise, reverberation, and speaker-to-speaker variations. Typical keyword spotting features such as Log-Mel spectrograms or Mel Frequency Cepstral Coefficients are highly sensitive to frequency perturbations. Denoising such features using an autoencoder improves performance in noise but often results in poor generalization for unseen speakers. This paper proposes an architecture that addresses the above challenges for keyword spotting systems. The audio features are combined with a latent representation extracted by a denoising autoencoder. In addition, this paper proposes a novel method for creating highly separable optimal latent representations from speech using a discriminative denoising autoencoder trained with a quadruplet loss metric learning approach. The proposed approach creates a discriminative latent representation, which when combined with the original input results in an architecture that is ideal for keyword spotting. The proposed architecture outperforms all approaches when tested in both clean and noisy environments with reverberation at various distances on unseen speakers.

Hand gesture recognition has become an increasingly important functionality in intelligent human-computer interfaces, finding applications in automotive, gaming and consumer industries. In this paper, we present an embedded gesture recognition system using frequency modulated continuous wave radar operating at 60 GHz. To facilitate low-power and low latency operation of the proposed system, spiking neural networks are used which are sparse in time and space, and event-driven. The experimental results demonstrate that the proposed neuromorphic implementation is capable of achieving a high recognition rate of 97.5% which is comparable to its deep learning counterparts in identifying radar gestures, namely up-down, down-up, swipe and finger rub.

Automatic radar based classification of automotive targets, such as pedestrians and cyclist, poses several challenges due to low inter-class variations among different classes and large intra-class variations. Further, different targets required to track in typical automotive scenario can have completely varying dynamics which gets challenging for tracker using conventional state vectors. Compared to state-of-the-art using independent classification and tracking, in this paper, we propose an integrated tracker and classifier leading to a novel Bayesian framework. The tracker’s state vector in the proposed framework not only includes the localization parameters of the targets but is also augmented with the targets’s feature embedding vector. In consequence, the tracker’s performance is optimized due to a better separability of the targets. Furthermore, the classifier’s performance is enhanced due to Bayesian formulation utilizing the temporal smoothing of classifier’s embedding vector.

The paper proposes an improved representation learning framework for human activity classification using IMU sensors, namely accelerometer and gyroscope. In practical deployment of the IMU-based activity classification the system is expected to encounter variations in data due to sensor degradation, alien environment or sensor noise and will be subjected to unknown activities. To address these issues pertaining to open world classification, in this paper we propose a novel Bayesian inference framework that uses variational embedding model to predict the activity class, followed by tracking through Kalman filter to smoothen these embedding vector, which is then fed into linear classifier for predicting the activity class. We evaluate the performance of our novel Bayesian inference framework on IMU activity classification and demonstrate that the classification accuracy, clustering scores, and the unknown class rejection performance improves substantially compared to its counter-part embedding model.

Air-writing systems offer users a virtual board to write characters or words in free space using fingers or hand movements. Several works have been proposed in literature that aim to use different sensors to enable such a system as an alternative to the keyboard and click form of human-machine interfaces. The advancement of miniature radar sensors and deep learning has enabled precise estimation and tracking of finger or marker movement followed by character recognition to offer an effective air-writing solution. However, deviating from earlier works in literature that make use of a network of radars to effectively track and recognize characters, in this paper, we propose to use only one or two radars to sense the local hand trajectory. We propose to use 1D temporal convolutional network (TCN) for simultaneous feature extraction and temporal modeling to recognize the drawn character from the local target trajectory. A dataset with 3750 character instances has been recorded using a 60-GHz millimeter-wave frequency-modulated continuous wave radar (FMCW) radar. We demonstrate the proposed end to end solution achieves a mean accuracy of 99.11% and 91.33% for two radar and one radar-based solution respectively outperforming other deep architectures.

Hand gesture and motion sensing offer an intuitive and natural form of human-machine interface. Air-writing systems allow users to draw alpha-numerical or linguistic characters in the virtual board in air through hand gestures. Traditionally, radar-based air-writing systems have been based on a network of radars, at least three, to localize the hand target through trilateration algorithm followed by tracking to extract the drawn trajectory, which is then followed by recognition of the drawn character by either Long-Short Term Memory (LSTM) utilizing the sensed trajectory or Deep Convolutional Neural Network (DCNN) utilizing a reconstructed 2D image from the trajectory. However, the practical deployments of such systems are limited since the detection of the finger or hand target by all three radars cannot be guaranteed leading to failure of the trilateration algorithm. Further placement of three or more radars for the air-writing solution is neither always physically plausible nor cost-effective. Furthermore, these solutions do not exploit the full potentials of deep neural networks, which are generally capable of learning features implicitly. In this paper, we propose an air-writing system based on a network of sparse radars, i.e. strictly less than three, using 1D DCNN-LSTM-1D transposed DCNN architecture to reconstruct and classify the drawn character utilizing only the range information from each radar. The paper employs real data using one and two 60 GHz milli-meter wave radar sensors to demonstrate the success of the proposed air-writing solution.

This paper presents a parametric variational autoencoder-based human target detection and localization framework working directly with the raw analog-to-digital converter data from the frequency modulated continuous wave radar. We propose a parametrically constrained variational autoencoder, with residual and skip connections, capable of generating the clustered and localized target detections on the range-angle image. Furthermore, to circumvent the problem of training the proposed neural network on all possible scenarios using real radar data, we propose domain adaptation strategies whereby we first train the neural network using ray tracing based model data and then adapt the network to work on real sensor data. This strategy ensures better generalization and scalability of the proposed neural network even though it is trained with limited radar data. We demonstrate the superior detection and localization performance of our proposed solution compared to the conventional signal processing pipeline and earlier state-of-art deep U-Net architecture with range-doppler images as inputs.

We propose a novel space-time waveform coding (STWC) scheme for joint radar and wireless communications (RadCom). In particular, we consider a frequency-modulated continuous-wave (FMCW) multiple-input multiple-output (MIMO) radar for near-range to medium-range radar applications. In order to establish the required orthogonal transmit signals, up- and down-chirp FMCW waveforms are combined with Alamouti space-time coding across two transmit antennas. We demonstrate that, from a radar perspective, the cross-ambiguity function is improved significantly by the orthogonal Alamouti code, while the FMCW waveforms ensure robustness in the presence of Doppler shifts. Regarding the communications part, the generic orthogonality of the Alamouti code allows us to embed random information symbols within the radar signal, enabling wireless communications in the 100 kb/s regime. To this end, we propose a suitable communications receiver algorithm for multiple antennas with corresponding maximum-ratio combining, which is able to extract the embedded information symbols while acquiring spatial diversity gains.

Gesture recognition provides an easy, convenient and intuitive way of remotely controlling several consumer electronics devices such as audio devices, television sets, projector or gaming consoles. In recent years, radar sensors have been shown to be effective sensing modality to sense and recognize fine-grained dynamic finger-gestures in watch or smartphone and thus offers an user-friendly human-computer interface in ultrashort range applications. However, hand-gesture recognition from a farther distance such as to control consumer devices like TV or projector pose challenge particularly arising due to interferences from multiple humans in the field of view. In this paper, we present a novel unguided spatio-Doppler attention mechanism to enable hand-gesture recognition in presence of multiple humans using a low power, compact 60-GHz FMCW radar operated in 500MHz ISM frequency band. The spatio-Doppler mechanism in 2D deep convolutional neural network with long short term memory (2D CNN-LSTM) makes use of the range-Doppler images and range-angle images. We experimentally present the classification accuracy of 94.75% of our proposed system on test dataset using eight gestures, namely wave, push forward, pull, left swipe, right swipe, clockwise rotate, anti-clockwise rotate, cross, in presence of interfering people, such as walking or arbitrary movements.

Radar is an attractive sensing technology for remote and non-intrusive human health monitoring and elderly fall detection due to its ability to work in low lighting conditions, its invariance to the environment, and its ability to operate through obstacles. Radar reflections from humans produce unique micro-Doppler signatures that can be used for classifying human activities and fall motion. However, radar-based elderly fall detection need to handle the indistinctive inter-class differences and large intra-class variations of human fall-motion in a real-world situation. Further, the radar placement in the room and varying aspect angle of the falling subject could result in differing radar micro-Doppler signature of human fall-motion. In this paper, we use a compact short-range 60-GHz frequency modulated continuous wave radar for detecting human fall motion using a novel deformable deep convolutional neural network with novel 1-class contrastive loss function in conjunction to focus loss to recognize elderly fall and address several of these signal processing system challenges. We demonstrate the performance of our proposed system in laboratory conditions under staged fall motion.

We present a radar-based detection processing framework for accurate detection and counting of human targets in an indoor environment. This can be used to control lighting, heating, ventilation and air conditioning (HVAC) in smart homes and other presence related loads in commercial, office, and public spaces. Such smart home applications can facilitate monitoring, controlling, and saving energy. Conventionally, the radar range-Doppler processing pipeline includes moving target indicators (MTI) to remove static targets, maximal ratio combining (MRC) to integrate data across antennas, constant false alarm rate (CFAR) based detectors and then clustering algorithms to generate the target range-Doppler detections. However, the conventional pipeline suffers from ghost targets and multi-path reflections from static objects such as walls, furniture, etc. Further, conventional parametric clustering algorithms lead to single target splits and adjacent target merges in the target range-Doppler detections. To overcome such issues, we propose a deep residual U-net architecture that generates human target detections directly from static target removed range-Doppler images (RDI). To train this network, we record RDIs from a variety of indoor scenes with different configurations and multiple humans targets. We devise a custom loss function and apply augmentation strategies to generalize this model during real-time inference of the model. We demonstrate that the proposed network can efficiently learn to detect and correctly count human targets under different indoor environments while the conventional signal processing pipeline fails.

Matched illumination (MI) radar signals provide improved target signal to interference noise ratios (SINR) and better spread ambiguity function performance compared to conventional radar in the presence of range-spread targets. Performance improvements reported in literature are based on the assumption of perfect knowledge of responses of the extended targets as well as interference spectra. In this paper, we analyze the SINR performance of MI systems from the learning phase to illumination phase using complex geometric theory of diffraction (GTD)-based target models. We numerically evaluate the worst-case performance degradation arising from aspect change of the target models from learning to illumination phase. The results set the stage for inclusion of a scheduler, which would facilitate selective MI based on system parameters and the targets being tracked.