Deep neural network (DNN) models are increasingly popular in edge video analytic applications. However, the computeintensive nature of DNN models pose challenges for energyefficient inference on resource-constrained edge devices. Most existing solutions focus on optimizing DNN inference latency and accuracy, often overlooking energy efficiency. They also fail to account for the varying complexity of video frames, leading to sub-optimal performance in edge video analytics. In this paper, we propose an EnergyEfficient Early-Exit (E4) framework that enhances DNN inference efficiency for edge video analytics by integrating a novel early-exit mechanism with dynamic voltage and frequency scaling (DVFS) governors. It employs an attentionbased cascade module to analyze video frame diversity and automatically determine optimal DNN exit points. Additionally, E4 features a just-in-time (JIT) profiler that uses coordinate descent search to co-optimize CPU and GPU clock frequencies for each layer before the DNN exit points. Extensive evaluations demonstrate that E4 outperforms current state-of-the-art methods, achieving up to 2.8× speedup and 26% average energy saving while maintaining high accuracy.

Data-driven radio frequency (RF) tomography has shown great potential to detect underground targets due to the capability of RF signals to penetrate soil. However, it is still challenging to achieve robust detection performance in dynamic environments. In this work, we design a cross-modal sensing system with RF and visual sensors and propose to train an RF sensing deep neural network (DNN) model following the cross-modal learning approach. We also propose to apply continual learning to automatically update the DNN model in a dynamic environment. Specifically, we design an environmental change detector and a one-shot fine-tuning module and integrate them into the DNN model to reconstruct the cross-section images of underground tubers even with significant changes in RF signals. Experimental results show that our approach achieves an average equivalent diameter error of 2.29 cm, 23.2% improvement upon the state-of-the-art approach. Our code and dataset are both made publicly available.

Non-invasive monitoring of underground biomass like root tubers is vital for smart agriculture. We present a wireless sensing system using Wi-Fi Channel State Information (CSI) from a low-cost ESP32 mesh network for high-resolution underground tuber imaging. Our approach uses synchronized CSI data collection and deep neural network (DNN) models including UNet, FCN and DeepLabV3+ for image reconstruction. We describe the testbed, data processing, experiments and DNN models in this paper. Comparative results show DNN models using the CSI data significantly outperforms those using traditional RSSI data. The DeepLabV3+ model using CSI data achieves the best imaging accuracy with an IoU of 0.6971, demonstrating the potential of WiFi CSI for fine-grained underground tuber sensing.

We demonstrate a non-invasive Wi-Fi sensing system that uses channel state information (CSI) data and deep neural network (DNN) models to reconstruct the cross-section images of potato tubers underground. We design a Wi-Fi mesh network that can leverage both the space and frequency diversities of the wireless network. We apply a multi-branch convolutional neural network (CNN) model to perform data-driven image reconstruction. We have performed extensive experiments to build a Wi-Fi potato sensing dataset, and our demo and experimental evaluations show that the Wi-Fi system outperforms the state-of-the-art root tuber wireless sensing system in terms of image quality and estimation accuracy.

Radio frequency sensors can penetrate non-metal objects and provide complementary information to vision sensors for person identification (PID) purposes. However, there is a lack of research on millimeter wave (mmWave) radar for PID under occlusions, particularly in addressing the open-set recognition problem. Thus, we propose an open-set occluded PID (OSO-PID) framework that can deal with various obstacle and occlusion scenarios with open-set recognition capability. We first introduce a new dataset, mmWave-ocPID, comprising mmWave radar measurements and RGB-depth images, collected from 23 human subjects. We next design a novel neural network, mm-PIDNet, for occluded person identification using mmWave radar measurements. mm-PIDNet incorporates a transformer encoder, a bidirectional long short-term memory module, and a novel supervised contrastive learning module to improve PID performance. For open-set recognition, we enhance the mmWave radar-based PID method by integrating supervised contrastive learning with the Weibull models, which can identify out-of-distribution samples. We perform extensive indoor experiments with a variety of obstacles and occlusion scenarios. Our experimental results show that mm-PIDNet achieves an F1-score of 0.93 on average, outperforming state-of-the-art methods by up to 13.41% for occluded cases. For open-set PID, the OSO-PID framework achieves an F1-score above 0.8 when the openness is less than 14.36%.

Fine-grained human motion detection has become increasingly important with the growing popularity of human computer interaction (HCI). However, traditional gesture-based HCI systems often require the design of new operation modes rather than conforming to user habits, thus increasing system learning costs. In this paper, we present TapWristband, a novel wearable sensor-based vibration sensing system that detects finger tapping by measuring wrist vibrations. We first perform real-world experiments to collect measurements for modeling the effects of the tapping motion on wearable wristband sensors including piezoelectric transducer (PZT) and inertial measurement unit (IMU). We find that a damped vibration model can be used to represent the relaxing phase of a vibration response due to tapping motion. Thus, we propose a mutual cross-correlation-based event segmentation algorithm to extract the vibration signal during the relaxing phase. After that, we develop feature extraction and classification algorithms to recognize the tapping patterns of five fingers across twelve key locations of a keypad system. Finally, we performed extensive experiments with thirteen participants to evaluate our system. Experimental results show that our low-cost vibration sensing system can achieve an average accuracy of over 93% with a tapping speed of over 100 taps per minute in real-world tapping scenarios.

Due to limited resources on edge and different characteristics of deep neural network (DNN) models, it is a big challenge to optimize DNN inference performance in terms of energy consumption and end-to-end latency. In addition to dynamic voltage frequency scaling (DVFS) technique, edge-cloud architecture provides a collaborative approach for efficient DNN inference. However, current edge-cloud collaborative inference methods have not optimized various compute resources on edge devices. Thus, we propose DVFO, a novel DVFS-enabled edge-cloud collaborative inference framework, which co-optimizes DVFS and offloading parameters via deep reinforcement learning (DRL). Specifically, DVFO automatically co-optimizes 1) the CPU, GPU and memory frequencies of edge devices, and 2) the offloaded feature map. In addition, it leverages athinking-while-movingconcurrent mechanism to accelerate the DRL learning process, and aspatial-channel attentionmechanism to identify the less important DNN feature map for efficient offloading. This approach improves inference performance for different DNN models under various edge-cloud network conditions. Extensive evaluations using two datasets and six widely-deployed DNN models on five heterogeneous edge devices show that DVFO significantly reduces the energy consumption by 33% on average, compared to state-of-the-art schemes. Moreover, DVFO achieves up to 28.6%∼59.1% end-to-end latency reduction, while maintaining accuracy within 1% loss on average.

As deep neural networks (DNNs) are increasingly used in a broad spectrum of edge intelligent applications, it is often necessary to provide multi-DNN model inference services, and it is nontrivial for edge inference platforms to simultaneously deliver high-throughput and low-latency. Such edge devices with multi-DNN model pose new challenges for scheduler designs. First, edge devices should be capable of efficiently scheduling multiple heterogeneous DNN models in order to optimize system utilization. Second, each inference request may have different service level objectives (SLOs) to improve quality of service (QoS). To address these challenges, this paper proposes BCEdge, a novel learning-based scheduling framework that incorporates adaptive batching and concurrent execution of DNN inference services on edge devices. We first propose a shared memory policy to reduce the memory contention among multiple DNN models. Afterwards, a utility function is defined to evaluate the trade-off between throughput and latency. The scheduler in BCEdge leverages branch-based deep reinforcement learning (DRL) to maximize utility by 1) optimizing batch size, 2) automatically identifying the number of concurrent instances for multiple DNN models, and 3) determining the shared memory configuration among multiple DNN models. Besides, the lightweight DNN-based prediction model in BCEdge can achieve SLO awareness by reducing the performance interference among multiple DNN models. Our prototype implemented on various edge devices illustrates that BCEdge enhances utility by up to 37.6% and reduces memory usage by up to 38% on average, compared to state-of-the-art schemes, while maintaining the SLO violation rate within 5%.

Edge intelligent applications, such as autonomous driving usually deploy multiple inference models on resource-constrained edge devices to execute a diverse range of concurrent tasks, given large amounts of input data. One challenge is that these tasks need to produce reliable inference results simultaneously with millisecond-level latency to achieve real-time performance and high quality of service (QoS). However, most of the existing deep learning frameworks only focus on optimizing a single inference model on an edge device. To accelerate multi-model inference on a resource-constrained edge device, in this paper we propose POS, a novel operator-level scheduling framework that combines four operator scheduling strategies. The key to POS is a maximum entropy reinforcement learning-based operator scheduling algorithm MEOS, which generates an optimal schedule automatically. Extensive experiments show that POS outperforms five state-of-the-art inference frameworks: TensorFlow, PyTorch, TensorRT, TVM, and IOS, by up to 1.2 × ∼ 3.9 × inference speedup consistently, with 40% improvement on GPU utilization. Meanwhile, MEOS reduces the scheduling overhead by 37% on average, compared to five baseline methods including sequential execution, dynamic programming, greedy scheduling, actor-critic, and coordinate descent search algorithms.

Due to resource constraints, it is challenging to optimize the inference performance in terms of energy consumption and latency on edge devices. In this paper, we leverage both the dynamic voltage frequency scaling (DVFS) technique and edge-cloud collaborative inference to minimize the overall energy consumption. We propose a deep reinforcement learning (DRL)-based method called DVFO to jointly optimize 1) CPU, GPU and memory frequencies, and 2) the ratio of offloaded feature maps in edge-cloud collaboration. Preliminary experimental results show that DVFO reduces the average energy consumption by 33% compared to the baselines. Moreover, it reduces the inference latency by more than 54%.

Unmanned aerial vehicles (UAV) with onboard sensors become a cost-effective way of crop remote sensing in largescale farms. However, current vision-based crop aerial sensing methods suffer from occlusion issue and require large amount of annotation data. In this paper, we target one particular crop species, corn, and propose to use 3D modeling and simulation to help resolve the issues. We first develop a corn-field 3D model and a crop aerial sensing (CAS) simulation framework. Then we use the CAS framework to generate synthetic data to train various deep learning models for corn leaf segmentation. In addition, we change the 3D model parameters in CAS, e.g., distances between individual corn plants, to derive leaf area index (LAI) correction coefficients for various corn plant and row spacings. Our experimental results from real-world UAV images show that our leaf segmentation model using synthetic data from the CAS framework outperforms state-of-the-art segmentation models by 1.4-3.3%. Our simulation results show that the plant and row spacings of a corn-field have significant effects on correcting the UAV image-based LAI, which can be underestimated by a factor of 2.6, due to the overlap and occlusion issues.

We propose mmWave-ocPID, a person identification (PID) method with millimeter-wave radar to identify individuals even when they are heavily occluded by obstacles. We collect a multi-modal dataset comprising mmWave radar point clouds and RGB images obtained from 9 human subjects, with over 180,000 frames for each modality. The mmWave-ocPID prototype employs a novel Neural Network integrated with two augmentation strategies for learning. Our initial experimental results show that mmWave-ocPID can achieve high identification accuracy, even when most of the human body of an individual is occluded in a controlled environment.

Deep neural networks (DNNs) are becoming extremely popular in video analytics applications at the edge. However, compute-intensive DNNs pose new challenges to achieve energy-efficient DNN inference on resource-constrained edge devices. In this paper, we propose E4, an energy-efficient DNN inference framework for edge video analytics. First, E4 analyzes video frame complexity by employing an attention-based cascade module that automatically determines DNN exit points. Second, E4's just-in-time (JIT) profiler leverages coordinate descent search to co-optimize the CPU and GPU clock frequencies for each layer before the DNN exit point. Preliminary experimental results show that E4 outperforms exiting methods in terms of power consumption and inference latency.

We present a containerized mobile sensing simulation (CMOS) framework developed for smart agriculture applications. This framework includes 1) 3D environment and object modeling, 2) mobile platform motion planning and control, and 3) optical sensing simulation, all implemented and connected within containers. Specifically, we build a user-friendly interface for 3D modeling, e.g., cornfield modeling using Blender. We use an unmanned aerial vehicle (UAV) as our mobile sensing platform and integrate UAV 3D model, flight path planning and control with robot operating system (ROS) packages and the Gazebo simulator. We also implemented optical sensing, e.g., collecting RGB image data from cameras in our simulation framework. This framework can be used not only in leaf area index correction and other analytical support for agriculture operations, but also as a synthetic data annotation tool for leaf segmentation and other smart agriculture applications. We demonstrate the major components of the CMOS framework, and how to use it to automatically annotate image data for the leaf segmentation application.

Human activity recognition (HAR) is an important task in many internet of things (IoT) applications. In recent years, significant efforts have been made towards achieving the highest possible recognition performance (accuracy and robustness) by using advanced machine learning techniques, including deep learning. However, to the best of our knowledge, the adversarial vulnerability of the Doppler sensor-based HAR systems has not been studied. In other domains such as computer vision, the vulnerability of deep learning algorithms to adversarial samples has attracted tremendous research interests in the past few years. In this work, we investigate the adversarial vulnerability of the Doppler-based human activity recognition system. Using a case study we demonstrate that the adversarial examples can significantly degrade the performance of the human activity recognition. Specifically, the basic iterative method (BIM) attack can reduce classification accuracy by as much as 85%. We also discuss different types of attacks, e.g., data poisoning attacks and potential strategies of protecting the Doppler-based HAR systems against adversarial attacks.

We present a multimodal sensor system for wound assessment and pressure ulcer care. Multiple imaging modalities including RGB, three- dimensional (3-D) depth, thermal, multispectral, and chemical sensing are integrated into a portable hand-held probe for real-time wound assessment. Analytic and quantitative algorithms for various assessments including tissue composition, wound measurement in 3-D, temperature profiling, spectral, and chemical vapor analysis are developed. After each assessment scan, 3-D models of the wound are generated on the fly for geometric measurement, while multimodal observations are analyzed to estimate healing progress. Collaboration between developers and clinical practitioners was conducted at the Charlie Norwood VA Medical Center for in-field data collection and experimental evaluation. A total of 133 assessment sessions from 23 enrolled subjects were collected, on which the multimodal data were analyzed and validated with respect to clinical notes associated with each subject. The system can be operated by nontechnical caregivers on a regular basis to aid wound assessment and care. A web portal front-end was developed for clinical decision and telehealth support, where all historical patient data including wound measurements and analysis can be organized online.

This paper presents a non-invasive human activity monitoring system with a low-cost Doppler sensor and a pair of radio frequency (RF) sensors. This radio-based system combines the strengths of two sensing modalities: fine granularity from a Doppler sensor and large sensing coverage from an RF link. The system is capable of detecting subtle human motion such as breathing, as well as walking in a large area. We deploy the system and perform experiments in a 5.5 m by 7.5 m room to classify four activities. Experimental results show that the average classification rate is 90%, 31% more accurate than a single Doppler system.

Device-free localization systems, such as variance-based radio tomographic imaging (VRTI), use received signal strength (RSS) variations caused by human motion in a static wireless network to locate and track people in the area of the network, even through walls. However, intrinsic motion, such as branches moving in the wind or rotating or vibrating machinery, also causes RSS variations which degrade the performance of a localization system. In this paper, we propose a new estimator, least squares variance-based radio tomography (LSVRT), which reduces the impact of the variations caused by intrinsic motion. We compare the novel method to subspace variance-based radio tomography (SubVRT) and VRTI. SubVRT also reduces intrinsic noise compared to VRTI, but LSVRT achieves better localization accuracy and does not require manually tuning additional parameters compared to VRTI. We also propose and test an online calibration method so that LSVRT and SubVRT do not require “empty-area” calibration and thus can be used in emergency situations. Experimental results from five data sets collected during three experimental deployments show that both estimators, using online calibration, can reduce localization root mean squared error by more than 40 percent compared to VRTI. In addition, the Kalman filter tracking results from both estimators have 97th percentile error of 1.3 m, a 60 percent reduction compared to VRTI.

Network radio frequency (RF) environment sensing (NRES) systems pinpoint and track people in buildings using changes in the signal strength measurements made by a wireless sensor network. It has been shown that such systems can locate people who do not participate in the system by wearing any radio device, even through walls, because of the changes that moving people cause to the static wireless sensor network. However, many such systems cannot locate stationary people. We present and evaluate a system which can locate stationary or moving people, without calibration, by using kernel distance to quantify the difference between two histograms of signal strength measurements. From five experiments, we show that our kernel distance-based radio tomographic localization system performs better than the state-of-the-art NRES systems in different non line-of-sight environments.

We present an interactive demonstration of histogram distance-based radio tomographic imaging (HD-RTI), a device-free localization (DFL) system that uses measurements of received signal strength (RSS) on static links in a wireless network to estimate the locations of people who do not participate in the system by wearing any radio device in the deployment area. Compared to prior methods of RSS-based DFL, using a histogram difference metric is a very accurate method to quantify the change in RSS on the link compared to historical metrics. The new method is remarkably accurate, and works with lower node densities than prior methods.

Tracking of people via active badges is important for location-aware computing and for security applications. However, the human body has a major effect on the antenna gain pattern of the device that the person is wearing. In this paper, the gain pattern due to the effect of the human body is experimentally measured and represented by a first-order directional gain pattern model. A method is presented to estimate the model parameters from multiple received signal strength (RSS) measurements. An alternating gain and position estimation (AGAPE) algorithm is proposed to jointly estimate the orientation and the position of the badge using RSS measurements at known-position anchor nodes. Lower bounds on mean squared error (MSE) and experimental results are presented that both show that the accuracy of position estimates can be greatly improved by including orientation estimates in the localization system. Next, we propose a new tracking filter that accepts orientation estimates as input, which we call the orientation-enhanced extended Kalman filter (OE-EKF), which improves tracking accuracy in active RFID tracking systems.

We propose to demonstrate a new radio tomographic localization algorithm - subspace variance-based radio tomography (SubVRT), which is more robust to RSS variations caused by objects that are intrinsic parts of the environment. We first introduce the subspace decomposition method, then we derive the formulations of SubVRT, and finally we describe the demonstration setup, requirements and procedures.

Human motion in the vicinity of a wireless link causes variations in the link received signal strength (RSS). Device-free localization (DFL) systems, such as variance-based radio tomographic imaging (VRTI) use these RSS variations in a wireless network to detect, locate and track people in the area of the network, even through walls. However, intrinsic motion, such as branches moving in the wind, rotating or vibrating machinery, also causes RSS variations which degrade the performance of a DFL system. In this paper, we propose and evaluate a subspace decomposition method subspace variance-based radio tomography (SubVRT) to reduce the impact of the variations caused by intrinsic motion. Experimental results show that the SubVRT algorithm reduces localization root mean squared error (RMSE) by 41%. In addition, the Kalman filter tracking results from SubVRT have 97% of errors less than 1.4 m, a 65% improvement compared to tracking results from VRTI.

Ptolemy II is a component-based design and modeling environment. It has a polymorphic type system that supports both base types and structured types, such as arrays, records, and unions. This paper presents the extensions to the base type system that support structured types. In the base type system, all the types are organized into a type lattice, and type constraints in the form of inequalities can be solved efficiently over the lattice. We take a hierarchical and granular approach to add structured types to the lattice and extend the format of inequality constraints to allow arbitrary nesting of structured types. We also analyze the convergence of the constraint-solving algorithm on an infinite lattice after structured types are added. To show the application of structured types, we present two Ptolemy II models that have direct real-world background. The first one describes the workflow of a charity organization, and the second one implements part of the IEEE 802.11 specification. These models make extensive use of record and union types to represent structured information. © 2009 Wiley Periodicals, Inc.

Discrete-event (DE) models are formal system specifications that have analysable deterministic behaviors. Using a global, consistent notion of time, DE components communicate via time-stamped events. DE models have primarily been used in performance modeling and simulation, where time stamps are a modeling property bearing no relationship to real time during execution of the model. In this paper, we extend DE models with the capability of relating certain events to physical time. We propose a programming model, called PTIDES (programming temporally integrated distributed embedded systems), which has DE semantics, but with carefully chosen relations between model time and real time. Key to making this model effective is to ensure that constraints that guarantee determinacy in the semantics are preserved at runtime. To accomplish this, we give a distributed execution strategy that obeys DE semantics without the penalty of totally ordered executions based on time stamps. Our technique relies on having a distributed common notion of time, known to some precision. Based on causality analysis of DE models, we define relevant dependency and relevant orders to enable out-of-order execution without compromising determinism and without requiring backtracking

Abstract Many scientific disciplines are now data and information driven, and new scientific knowledge is often gained by scientists putting together data analysis and knowledge discovery ‘pipelines’. A related trend is that more and more scientific communities realize the benefits of sharing their data and computational services, and are thus contributing to a distributed data and computational community infrastructure (a.k.a. ‘the Grid’). However, this infrastructure is only a means to an end and ideally scientists should not be too concerned with its existence. The goal is for scientists to focus on development and use of what we call scientific workflows . These are networks of analytical steps that may involve, e.g., database access and querying steps, data analysis and mining steps, and many other steps including computationally intensive jobs on high‐performance cluster computers. In this paper we describe characteristics of and requirements for scientific workflows as identified in a number of our application projects. We then elaborate on Kepler, a particular scientific workflow system, currently under development across a number of scientific data management projects. We describe some key features of Kepler and its underlying Ptolemy II system, planned extensions, and areas of future research. Kepler is a community‐driven, open source project, and we always welcome related projects and new contributors to join. Copyright © 2005 John Wiley & Sons, Ltd.

We are announcing the first release of Viptos (Visual Ptolemy and TinyOS), an integrated graphical development and simulation environment for TinyOS-based wireless sensor networks. Viptos allows developers to create block and arrow diagrams to construct TinyOS programs from any standard library of nesC/TinyOS components. The tool automatically transforms the diagram into a nesC program that can be compiled and downloaded from within the graphical environment onto any TinyOS-supported target hardware. In particular, Viptos includes the full capabilities of VisualSense [1], which can model communication channels, networks, and non-TinyOS nodes. This release of Viptos is compatible with nesC 1.2 and includes tools to harvest existing TinyOS components and applications and convert them into a format that can be displayed as block (and arrow) diagrams and simulated.Viptos is based on TOSSIM and Ptolemy II. TOSSIM is an interrupt-level simulator for TinyOS programs. It runs actual TinyOS code but provides software replacements for the simulated hardware and models network interaction at the bit or packet level. Ptolemy II is a graphical software system for modeling, simulation, and design of concurrent, real-time, embedded systems. Ptolemy II focuses on assembly of concurrent components with well-defined models of computation that govern the interaction between components. VisualSense is a Ptolemy II environment for modeling and simulation of wireless sensor networks at the network level.Viptos provides a bridge between VisualSense and TOSSIM by providing interrupt-level simulation of actual TinyOS programs, with packet-level simulation of the network, while allowing the developer to use other models of computation available in Ptolemy II for modeling various parts of the system. While TOSSIM only allows simulation of homogeneous networks where each node runs the same program, Viptos supports simulation of heterogeneous networks where each node may run a different program. Viptos simulations may also include non-TinyOS-based wireless nodes. The developer can easily switch to different channel models and change other parts of the simulated environment, such as creating models to generate simulated traffic on the wireless network.Viptos inherits the actor-oriented modeling environment of Ptolemy II, which allows the developer to use different models of computation at each level of simulation. At the lowest level, Viptos uses the discrete-event scheduler of TOSSIM to model the interaction between the CPU and TinyOS code that runs on it. At the next highest level, Viptos uses the discrete-event scheduler of Ptolemy II to model interaction with mote hardware, such as the radio and sensors. This level is then embedded within VisualSense to allow modeling of the wireless channels to simulate packet loss, corruption, delay, etc. The user can also model and simulate other aspects of the physical environment including those detected by the sensors (e.g., light, temperature, etc.), terrain, etc.At IPSN in April 2005, we demonstrated a pre-release developmental version of Viptos with two simple applications. The first was a single node sensing application that displayed the value of the light sensor on the LEDs. The second was a two node send and receive application that transmitted the value of the light sensor on the first node to the second node. This release version of Viptos supports more sophisticated applications, such as multi-node routing, and demonstrates some of the more advanced features described in this abstract.

This paper describes a modeling and simulation framework called VisualSense for wireless sensor networks that builds on and leverages Ptolemy II. This framework supports actor-oriented definition of sensor nodes, wireless communication channels, physical media such as acoustic channels, and wired subsystems. The software architecture consists of a set of base classes for defining channels and sensor nodes, a library of subclasses that provide certain specific channel models and node models, and an extensible visualization framework. Custom nodes can be defined by subclassing the base classes and defining the behavior in Java or by creating composite models using any of several Ptolemy II modeling environments. Custom channels can be defined by subclassing the WirelessChannel base class and by attaching functionality defined in Ptolemy II models.