Subtle periodic signals such as blood volume pulse and respiration can be extracted from RGB video, enabling non-contact health monitoring at low cost. Advancements in remote pulse estimation - or remote photoplethysmography (rPPG) - are currently driven by deep learning solutions. However, modern approaches are trained and evaluated on benchmark datasets with ground truth from contact-P PG sensors. We present the first non-contrastive unsuper-vised learning framework for signal regression to mitigate the need for labelled video data. With minimal assumptions of periodicity and finite bandwidth, our approach discovers the blood volume pulse directly from unlabelled videos. We find that encouraging sparse power spectra within normal physiological bandlimits and variance over batches of power spectra is sufficient for learning visual features of periodic signals. We perform the first experiments utilizing unlabelled video data not specifically created for rPPG to train robust pulse rate estimators. Given the limited inductive biases and impressive empirical results, the approach is theoretically capable of discovering other periodic signals from video, enabling multiple physiological measurements without the need for ground truth signals.

Remote photoplethysmography (rPPG) is a technique for estimating blood volume changes from reflected light without the need for a contact sensor. We present the first examples of presentation attacks in the digital and physical domains on rPPG from face video. Digital attacks are easily performed by adding imperceptible periodic noise to the input videos. Physical attacks are performed with illumination from visible spectrum LEDs placed in close proximity to the face, while still being difficult to perceive with the human eye. We also show that our attacks extend beyond medical applications, since the method can effectively generate a strong periodic pulse on 3D-printed face masks, which presents difficulties for pulse-based face presentation attack detection (PAD). The paper concludes with ideas for using this work to improve robustness of rPPG methods and pulse-based face PAD.

We present the Deception Detection and Physiological Monitoring (DDPM) dataset and initial baseline results on this dataset. Our application context is an interview scenario in which the interviewee attempts to deceive the interviewer on selected responses. The interviewee is recorded in RGB, near-infrared, and long-wave infrared, along with cardiac pulse, blood oxygenation, and audio. After collection, data were annotated for interviewer/interviewee, curated, ground-truthed, and organized into train / test parts for a set of canonical deception detection experiments. Baseline experiments found random accuracy for micro-expressions as an indicator of deception, but that saccades can give a statistically significant response. We also estimated subject heart rates from face videos (remotely) with a mean absolute error as low as 3.16 bpm. The database contains almost 13 hours of recordings of 70 subjects, and over 8 million visible-light, near-infrared, and thermal video frames, along with appropriate meta, audio and pulse oximeter data. To our knowledge, this is the only collection offering recordings of five modalities in an interview scenario that can be used in both deception detection and remote photoplethysmography research.

This paper overviews the state of the art, research challenges, and future opportunities in an emerging research direction: Social Sensing based Edge Computing (SSEC). Social sensing has emerged as a new sensing application paradigm where measurements about the physical world are collected from humans or from devices on their behalf. The advent of edge computing pushes the frontier of computation, service, and data along the cloud-to-things continuum. The merging of these two technical trends generates a set of new research challenges that need to be addressed. In this paper, we first define the new SSEC paradigm that is motivated by a few underlying technology trends. We then present a few representative real-world case studies of SSEC applications and several key research challenges that exist in those applications. Finally, we envision a few exciting research directions in future SSEC. We hope this paper will stimulate discussions of this emerging research direction in the community.

Modern Internet of Things (IoT) systems are increasingly leveraging deep neural networks (DNNs) with the goal of enabling intelligence at the edge of the network. While applying DNNs can greatly improve the accuracy of autonomous decisions and inferences, a significant challenge is that DNNs are traditionally designed and developed for advanced hardware (e.g., GPU clusters) and can not easily meet the real time requirements when deployed in a resource-constrained edge computing environment. While many systems have been proposed to facilitate deep learning at the edge, a key limitation lies in the under-utilization of the parallelizable GPU resources of edge nodes (e.g., IoT devices). In this paper, we propose EdgeBatch, a collaborative intelligent edge computing framework that minimizes the delay and energy consumption of executing DNN tasks at the edge by sharing idle GPU resources among privately owned IoT devices. EdgeBatch develops 1) a stochastic task batching mechanism that identifies the optimal batching strategy for the GPUs on IoT devices given uncertain task arrival times, and 2) a dynamic task offloading scheme that coordinates the collaboration among edge nodes to optimize the utilization of idle GPU resources in the system. We implemented EdgeBatch on a real-world edge computing testbed that consists of heterogeneous IoT devices (Jetson TX2, TX1, TK1, and Raspberry Pi3s). The results show that EdgeBatch achieved significant performance gains in terms of both the end-to-end delay and energy savings compared to the state-of-the-art baselines.

Social sensing based Edge Computing (SSEC) is an emerging application paradigm for rich context awareness in sensing applications in which people collaborate in both data acquisition and processing at the edge of the network. While keeping people in the loop can be an immense benefit, the unreliability and churn introduced can pose a dangerous stability threat for long-running SSEC applications. In the past this problem was addressed by offloading these responsibilities to more reliable server hardware in the cloud or a fog layer, however, this does not take full advantage of the power at the edge. In this paper we address the issue of reliable edge computing on dynamic high-churn edge systems. We develop a deviceless pipeline based approach (DPA) to establish workflows in which stages of the analysis pipeline are completed on edge devices, and any devices that leave the system can be replaced without data loss. We evaluate the performance of our system on a real-world edge computing system performing an object detection application and demonstrate that it can provide significant performance gains over traditional computation offloading schemes in terms of throughput and error recovery.

Identifying trustworthy information in the presence of noisy data contributed by numerous unvetted sources from online social media (e.g., Twitter, Facebook, and Instagram) has been a crucial task in the era of big data. This task, referred to as truth discovery, targets at identifying the reliability of the sources and the truthfulness of claims they make without knowing either a priori. In this work, we identified three important challenges that have not been well addressed in the current truth discovery literature. The first one is “misinformation spread” where a significant number of sources are contributing to false claims, making the identification of truthful claims difficult. For example, on Twitter, rumors, scams, and influence bots are common examples of sources colluding, either intentionally or unintentionally, to spread misinformation and obscure the truth. The second challenge is “data sparsity” or the “long-tail phenomenon” where a majority of sources only contribute a small number of claims, providing insufficient evidence to determine those sources' trustworthiness. For example, in the Twitter datasets that we collected during real-world events, more than 90 percent of sources only contributed to a single claim. Third, many current solutions are not scalable to large-scale social sensing events because of the centralized nature of their truth discovery algorithms. In this paper, we develop a Scalable and Robust Truth Discovery (SRTD) scheme to address the above three challenges. In particular, the SRTD scheme jointly quantifies both the reliability of sources and the credibility of claims using a principled approach. We further develop a distributed framework to implement the proposed truth discovery scheme using Work Queue in an HTCondor system. The evaluation results on three real-world datasets show that the SRTD scheme significantly outperforms the state-of-the-art truth discovery methods in terms of both effectiveness and efficiency.

Social sensing has emerged as a new application paradigm in networked sensing where data is collected from humans or devices on their behalf. This paper focuses on the opinion characterization problem in social sensing where the goal is to accurately characterize opinion attributes of the participants (e.g., analyze the sentiments, understand the opinion bias) from their sensor measurements. Several important challenges exist in solving the opinion characterization problem. First, human sensors often generate unstructured data (e.g., text, image, video) in which the opinion attributes are deeply embedded. Second, human sensors naturally generate measurements in different data modalities which encode the opinion attributes differently. Third, the possible imbalance between different data modalities may lead to potential bias in the opinion characterization results. To address the above challenges, this paper develops a Multi-View Opinion Characterization (MVOC) scheme to accurately characterize opinion attributes using a multi-view subspace learning approach. We evaluate the MVOC scheme through the real-world social sensing task of classifying the sentiments of reports from Twitter users. The evaluation results show that our scheme significantly outperforms the state-of-the-art baselines in solving the opinion characterization problem.

Social sensing has emerged as a new sensing paradigm where humans (or devices on their behalf) collectively report measurements about the physical world. This paper focuses on an optimized task allocation problem in multi- attribute social sensing applications where the goal is to effectively allocate the tasks of collecting multiple attributes of the measured variables to human sensors while respecting the application's budget constraints. While recent progress has been made to tackle the optimized task allocation problem, two important challenges have not been well addressed. The first challenge is "online task allocation": the task allocation schemes need to respond quickly to the potentially large dynamics of the measured variables (e.g., temperature, noise, traffic) in social sensing. Delayed task allocation may lead to inaccurate sensing results and/or unnecessarily high sensing costs. The second challenge is the "multi-attribute constrained optimization": minimizing the overall sensing error given the dependencies and constraints of multiple attributes of the measured variables is a non-trivial problem to solve. To address the above challenges, this paper develops an Online Optimized Multi-attribute Task Allocation (OO-MTA) scheme inspired by techniques from machine learning and information theory. We evaluate the OO-MTA scheme using an urban sensing dataset collected from a real-world social sensing application. The evaluation results show that OO- MTA scheme significantly outperforms the state-of-the-art baselines in terms of the sensing accuracy.

Privacy on the Internet has become a major concern in recent years and is an ethical dilemma for Internet based applications. Social Sensing based Edge Computing (SSEC)is a category of distributed applications in which privately owned devices at the “edge” of the network participate in data collection and processing. Traditionally in SSEC, it has been difficult to manage privacy without making a tradeoff with performance or accuracy. In this paper, we present Privacy-aware Edge Computing (PEC), a framework that utilizes ring signatures for implementing social sensing applications with full privacy and without sacrificing data quality. Under PEC, privacy is managed by the edge devices without the need of a third party trusted authority. We evaluate the performance of PEC on a realworld edge computing platform and through extensive simulation studies. The results demonstrate that PEC achieves a high level of anonymity with a constrained cryptographic overhead.

Social Sensing has emerged as a new distributed application paradigm in networked sensing where streaming data is constantly collected from humans or devices on their behalf. A key challenge in social sensing is that the applications are often both data-intensive and delay-sensitive, which often require extensive resources in the cloud to support the data processing and analytics tasks. To address this problem, this paper focuses on a quality-aware online adaptive sampling problem where the goal is to judiciously sample a small set of representative sensing measurements (i.e., representative set) that effectively represent the key information of the social sensing data stream. However, two technical challenges exist in identifying such a representative set: (i) “quality-aware quantification”: data collected in the social sensing applications is often unstructured (e.g., text, image, video) where the key information is so deeply embedded in the raw data that can not be easily quantified to ensure the desirable quality of the selected representative set; (ii) “online adaptive sampling”: the sampling decision has to be made in real-time and be adaptive to the dynamics of the streaming data in social sensing. To address these challenges, this paper develops a Light-weight and Quality-aware Online Adaptive Sampling (LQOAS) scheme to dynamically identify the representative set of measurements from the streaming social sensing data using a submodular maximization approach. We evaluate the LQOAS scheme using a real world social sensing dataset from Twitter. The evaluation results show that the LQOAS scheme significantly outperforms the state-of-the-art sampling schemes.

Collaborative sensing has emerged as a new sensing paradigm where sensors collaboratively report measurements about the phenomena or events in the physical world. This paper focuses on a robust state prediction problem in collaborative sensing where the goal is to provide an accurate prediction on the dynamic status of the physical environment (e.g., air quality index, temperature, traffic) based on the incomplete and noisy data contributed by collaborative sensors. While significant progress has been made to study the state prediction problem, we identify two fundamental challenges that have not been well addressed by the current literature. The first challenge is "latent spatial dynamics": the spatial correlations among measured variables are highly dynamic and the features that affect such dynamics may not be directly observable from the sensing data. The second challenge is "incomplete and noisy data": a significant amount of measurements are missing in collaborative sensing due to the prohibitively high cost of deploying sensors for full spatial-temporal coverage. The collaborative sensors are also known to be unreliable in many applications and can easily contribute to noisy measurements. To address these challenges, this paper develops a Context-Aware Collaborative Sensing Prediction (CACSP) scheme inspired by techniques from latent semantic analysis and statistics. We evaluate our scheme through two real-world collaborative sensing applications and the results show that CACSP significantly outperforms the state-of-the-art baselines.

In this demo, we present a Social Sensing Edge Computing (SSEC) system that supports real-time task allocation on non-cooperative and heterogeneous edge devices. We evaluate the system on two real-time social sensing applications and demonstrate the energy savings as well as the Quality of Service (QoS) of the SSEC system. The demonstrated framework complements the traditional centralized social sensing solutions by providing a more scalable and responsive computation paradigm.

Research in the field of Social Sensing Edge Computing (SSEC) faces a challenge in that most systems are developed on a small set of hardware and are never tested on large-scale networks. To address this issue, we present a Social Sensing Edge Computing Simulation Platform (SSEC SP) that is used to evaluate how SSEC systems scale to millions of devices. We demonstrate the platform by simulating several SSEC configurations to optimize Quality of Service (QoS) for large numbers of devices, while asserting that simulated results are consistent with the hardware implementations.