Fair decision-making in Machine Learning (ML) remains a critical challenge, particularly when access to sensitive information is restricted due to legal, ethical, or organizational constraints. These limitations affect both accuracy and fairness, creating tradeoffs central to the deployment of ML systems in the real world. While prior work has studied fairness-accuracy tradeoffs, most approaches focus on model outputs rather than directly examining how restricted data access impacts fairness. This leaves an important gap: understanding how fairness constraints affect model performance under real-world data restrictions . To address this gap, we propose a framework that explicitly models fairness-accuracy tradeoffs in data-restricted environments. Unlike prior work, our approach analyzes the behavior of the optimal Bayesian classifier using a discrete approximation of the data distribution, allowing us to systematically isolate the effects of fairness constraints. We evaluate our framework on three benchmark datasets—Adult, Law, and Dutch Census—revealing key insights: (1) enforcing equal accuracy on imbalanced datasets can substantially degrade performance under additional fairness constraints, (2) individual and group fairness often impose conflicting constraints, and (3) decorrelating sensitive attributes from features does not usually reduce accuracy. These findings demonstrate that our framework provides an effective, structured approach for practitioners to assess fairness constraints in decision-making pipelines.

The growing use of machine learning (ML) has raised concerns that an ML model may reveal private information about an individual who has contributed to the training dataset. To prevent leakage of sensitive data, we consider using differentially- private (DP), synthetic training data instead of real training data to train an ML model. A key desirable property of synthetic data is its ability to preserve the low-order marginals of the original distribution. Our main contribution comprises novel upper and lower bounds on the excess empirical risk of linear models trained on such synthetic data, for continuous and Lipschitz loss functions. We perform extensive experimentation alongside our theoretical results.

Voice interfaces have become one of the most ubiquitous human–computer interaction methods in recent years. Voice activity detection (VAD) is typically the first building block of a complex voice interface, often relying on audio signals. Acoustics-based VAD systems do not perform well in noisy and interference-prone environments. Smart assistants mitigate this problem by using a dictionary-based detection system. However, this approach is limited in its applicability. For instance, users may still need to manually mute and unmute their microphones during online meetings to prevent detection of interfering users, and speech leakage. In order to automate voice detection in challenging environments without these limitations, we propose RadioVAD, a noise and interference-resilient VAD system that uses radio modality, which is already available in various smartphones and home assistants. RadioVAD works by detecting possible human presence in the Field of View of the device, extracting the vocal fold’s vibration signal from the target speaker, and utilizing a time-domain neural network on raw radio signals to detect voice activity. Extensive experiments reveal that RadioVAD can detect voice activity in challenging environments with high accuracy and outperforms audio-based VAD when the audio signal has signal-to-noise ratio below 5 dB. Furthermore, RadioVAD reduces false alarm rate in interference-prone environments by 52%–72%, bringing significant improvements to VAD task. RadioVAD lays the foundation for future voice interfaces utilizing radio modality.

Increases in the deployment of machine learning algorithms for applications that deal with sensitive data have brought attention to the issue of fairness in machine learning. Many works have been devoted to applications that require different demographic groups to be treated fairly. However, algorithms that aim to satisfy inter-group fairness (also called group fairness) may inadvertently treat individuals within the same demographic group unfairly. To address this issue, this article introduces a formal definition of within-group fairness that maintains fairness among individuals from within the same group. A pre-processing framework is proposed to meet both inter- and within-group fairness criteria with little compromise in performance. The framework maps the feature vectors of members from different groups to an inter-group fair canonical domain before feeding them into a scoring function. The mapping is constructed to preserve the relative relationship between the scores obtained from the unprocessed feature vectors of individuals from the same demographic group, guaranteeing within-group fairness. This framework has been applied to the Adult, COMPAS risk assessment, and Law School datasets, and its performance is demonstrated and compared with two regularization-based methods in achieving inter-group and within-group fairness.

The inverse problem of inferring clinical gold-standard electrocardiogram (ECG) from photoplethysmogram (PPG) that can be measured by affordable wearable Internet of Healthcare Things (IoHT) devices is a research direction receiving growing attention. It combines the easy measurability of PPG and the rich clinical knowledge of ECG for long-term continuous cardiac monitoring. The prior art for reconstruction using a universal basis, such as discrete cosine transform (DCT), has limited fidelity for uncommon ECG shapes due to the lack of representative power. To better utilize the data and improve data representation, we design two dictionary learning frameworks, the cross-domain joint dictionary learning (XDJDL), and the label-consistent XDJDL (LC-XDJDL), to further improve the ECG inference quality and enrich the PPG-based diagnosis knowledge. Building on the K-SVD technique, the proposed joint dictionary learning frameworks extend the expressive power by optimizing simultaneously a pair of signal dictionaries for PPG and ECG with the transforms to relate their sparse codes and disease information. The proposed models are evaluated with a variety of PPG and ECG morphologies from two benchmark datasets that cover various age groups and disease types. The results show the proposed frameworks achieve better inference performance than previous methods with average Pearson coefficients being 0.88 using XDJDL and 0.92 using LC-XDJDL, suggesting an encouraging potential for ECG screening using PPG based on the proactively learned PPG-ECG relationship. By enabling the dynamic monitoring and analysis of the health status of an individual, the proposed frameworks contribute to the emerging digital twins paradigm for personalized healthcare.

Speech enhancement and separation have been a long-standing problem, especially with the recent advances using a single microphone. Although microphones perform well in constrained settings, their performance for speech separation decreases in noisy conditions. In this work, we proposeRadioSES, an audioradio speech enhancement and separation system that overcomes inherent problems in audio-only systems. By fusing a complementary radio modality,RadioSEScan estimate the number of speakers, solve the source association problem, separate and enhance noisy mixture speeches, and improve both intelligibility and perceptual quality. We perform millimeter-wave sensing to detect and localize speakers and introduce an audioradio deep learning framework to fuse the separate radio features with the mixed audio features. Extensive experiments using commercial off-the-shelf devices show thatRadioSESoutperforms a variety of state-of-the-art baselines, with consistent performance gains in different environmental settings. Similar to the audiovisual methods,RadioSESprovides significant performance improvements (e.g. 3 dB gains in SiSDR, when compared with the corresponding audio-only method), along with the benefits of lower computational complexity and better privacy preservation.

Electric network frequency (ENF) analysis is a promising forensic technique for authenticating multimedia recordings and detecting tampering. The validity of the ENF analysis heavily relies on the capability of extracting high-quality ENF signals from multimedia recordings. This paper analyzes and compares two representative methods for extracting ENF signals from visual signals acquired by cameras using the rolling-shutter mechanism. The first method proposed in prior work,direct concatenation, ignores the idle period of each frame. The second method proposed in this paper,periodic zeroing-out, inserts zeros to missing sample points instead of ignoring the idle period. Our theoretical analyses of using multirate signal processing reveal and experiments confirm that while the first method can extract ENF signals without knowing the exact value of camera read-out time, there exists some mild distortion to extracted ENF signals. In contrast, the second method taking the read-out time as the additional input is capable of extracting distortion-free ENF signals, and its frequency component of the highest strength is always located at the nominal frequency. Additionally, we examine aliased DC and negative ENF components caused by the two methods and show that their impact on the accuracy of frequency estimation is minimum. This paper facilitates the fundamental understanding of extracting ENF signals from videos. The research findings imply that the periodic zeroing-out method offers more accurate frequency estimates, but the performance improvement is not significant.

Peripheral blood oxygen saturation (SpO$_{2}$) is an essential indicator of respiratory functionality and received increasing attention during the COVID-19 pandemic. Clinical findings show that COVID-19 patients can have significantly low SpO$_{2}$before any obvious symptoms. Measuring an individual's SpO$_{2}$without having to come into contact with the person can lower the risk of cross contamination and blood circulation problems. The prevalence of smartphones has motivated researchers to investigate methods for monitoring SpO$_{2}$using smartphone cameras. Most prior schemes involving smartphones are contact-based: They require using a fingertip to cover the phone's camera and the nearby light source to capture reemitted light from the illuminated tissue. In this paper, we propose the first convolutional neural network based noncontact SpO$_{2}$estimation scheme using smartphone cameras. The scheme analyzes the videos of an individual's hand for physiological sensing, which is convenient and comfortable for users and can protect their privacy and allow for keeping face masks on. We design explainable neural network architectures inspired by the optophysiological models for SpO$_{2}$measurement and demonstrate the explainability by visualizing the weights for channel combination. Our proposed models outperform the state-of-the-art model that is designed for contact-based SpO$_{2}$measurement, showing the potential of the proposed method to contribute to public health. We also analyze the impact of skin type and the side of a hand on SpO$_{2}$estimation performance.

Osteoporosis is a common chronic metabolic bone disease often under-diagnosed and under-treated due to the limited access to bone mineral density (BMD) examinations, e.g., via Dual-energy X-ray Absorptiometry (DXA). This paper proposes a method to predict BMD from Chest X-ray (CXR), one of the most commonly accessible and low-cost medical imaging examinations. The proposed method first automatically detects Regions of Interest (ROIs) of local CXR bone structures. Then a multi-ROI deep model with transformer encoder is developed to exploit both local and global information in the chest X-ray image for accurate BMD estimation. The proposed method is evaluated on 13719 CXR patient cases with ground truth BMD measured by the gold standard DXA. The model predicted BMD has a strong correlation with the ground truth (Pearson correlation coefficient 0.894 on lumbar 1). When applied in osteoporosis screening, it achieves a high classification performance (average AUC of 0.968). As the first effort of using CXR scans to predict the BMD, the proposed algorithm holds strong potential to promote early osteoporosis screening and public health.

Recent studies have shown that physiological signals such as heart beat and breathing can be remotely captured from human faces using a regular color camera under ambient light. This technology, referred to as remote photoplethysmography (rPPG), can be used to collect the physiological status of users who are in front of a camera, which may raise privacy concerns. To avoid the privacy abuse of the rPPG technology, this paper develops PulseEdit, a novel and efficient algorithm that can edit the physiological signals in facial videos without affecting visual appearance and thus protect the user’s physiological signal from disclosure. PulseEdit can either remove the trace of the physiological signal in a video or transform the video to contain a target physiological signal chosen by a user. Experimental results show that PulseEdit can effectively edit physiological signals in facial videos and prevent heart rate measurement based on rPPG. It is possible to utilize PulseEdit in adversarial scenarios against rPPG-based visual security algorithms. We present analyses on the performance of PulseEdit against rPPG-based liveness detection and rPPG-based deepfake detection, and demonstrate its ability to circumvent these visual security algorithms and its important role in supporting the design of attack-resilient systems.

Geotagging images of interest are increasingly important to law enforcement, national security, and journalism. Today, many images do not carry location tags that are trustworthy and resilient to tampering; and landmark-based visual clues may not be readily present in every image, especially in those taken indoors. In this paper, we exploit an environmental signature from the power grid, the electric network frequency (ENF) signal, which can be inherently captured in a sensing stream at the time of recording and carries useful time–location information. Compared to the recent art of extracting ENF traces from audio and video recordings, it is very challenging to extract an ENF trace from a single image. We address this challenge by first mathematically examining the impact of the ENF embedding steps such as electricity to light conversion, scene geometry dilution of radiation, and image sensing. We then incorporate the verified parametric models of the physical embedding process into our proposed entropy minimization method. The optimized results of the entropy minimization are used for creating a two-level ENF presence–classification test for region-of-capturing localization. It identifies whether a single image has an ENF trace; if yes, whether it is at 50 or 60 Hz. We quantitatively study the relationship between the ENF strength and its detectability from a single image. This paper is the first comprehensive work to bring out a unique forensic capability of environmental traces that shed light on an image’s capturing location.

Accurate vertebra localization and identification are required in many clinical applications of spine disorder diagnosis and surgery planning. However, significant challenges are posed in this task by highly varying pathologies (such as vertebral compression fracture, scoliosis, and vertebral fixation) and imaging conditions (such as limited field of view and metal streak artifacts). This paper proposes a robust and accurate method that effectively exploits the anatomical knowledge of the spine to facilitate vertebra localization and identification. A key point localization model is trained to produce activation maps of vertebra centers. They are then re-sampled along the spine centerline to produce spine-rectified activation maps, which are further aggregated into 1-D activation signals. Following this, an anatomically-constrained optimization module is introduced to jointly search for the optimal vertebra centers under a soft constraint that regulates the distance between vertebrae and a hard constraint on the consecutive vertebra indices. When being evaluated on a major public benchmark of 302 highly pathological CT images, the proposed method reports the state of the art identification (id.) rate of 97.4%, and outperforms the best competing method of 94.7% id. rate by reducing the relative id. error rate by half.

In the era of pervasively connected and sensed Internet of Things, many of our interactions with machines have been shifted from conventional computer keyboards and mouses to hand gestures and writing in the air. While gesture recognition and handwriting recognition have been well studied, many new methods are being investigated to enable pervasive handwriting tracking. Most of the existing handwriting tracking systems either require cameras and handheld sensors or involve dedicated hardware restricting user convenience and the scale of usage. In this article, we present mmWrite, the first high-precision passive handwriting tracking system using a single commodity millimeter-wave (mmWave) radio. Leveraging the short wavelength and large bandwidth of 60-GHz signals and the radar-like capabilities enabled by the large phased array, mmWrite transforms any flat region into an interactive writing surface that supports handwriting tracking at millimeter accuracy. MmWrite employs an end-to-end pipeline of signal processing to enhance the range and spatial resolution limited by the hardware, boost the coverage, and suppress interference from backgrounds and irrelevant objects. We implement and evaluate mmWrite on a commodity 60-GHz device. The experimental results show that mmWrite can track a finger/pen with a median error of 2.8 mm and thus can reproduce handwritten characters as small as 1 cm × 1 cm, with a coverage of up to 8 m2supported. With minimal infrastructure needed, mmWrite promises ubiquitous handwriting tracking for new applications in the field of human-computer interactions.

This article studies the relation between electrocardiogram (ECG) and photoplethysmogram (PPG) and investigates the inference of the ECG waveforms from the PPG signals that can be obtained from affordable wearable Internet-of-Things (IoT) devices for mobile health. In order to address this inverse problem, a transform is proposed to map the discrete cosine transform (DCT) coefficients of each PPG cycle to those of the corresponding ECG cycle based on the proposed cardiovascular signal model. The proposed method is evaluated with different morphologies of the PPG and ECG signals on three benchmark data sets with a variety of combinations of age, weight, and health conditions under several training setups. The experimental results show that the proposed method can achieve a high prediction accuracy greater than 0.92 in averaged correlation for each data set when the model is trained subjectwise. With a signal processing and learning system that is designed synergistically, we are able to reconstruct ECG signals by exploiting the relation of these two types of cardiovascular measurement. The reconstruction capability of the proposed method can enable low-cost ECG screening from affordable wearable IoT devices for continuous and long-term monitoring. This work opens up a new research direction to transfer the clinical ECG knowledge base to build a knowledge base for PPG and sensing data from wearable devices.

In the field of information forensics, many emerging problems involve a critical step that estimates and tracks weak frequency components in noisy signals. It is often challenging for the prior art of frequency tracking to i) achieve a high accuracy under noisy conditions, ii) detect and track multiple frequency components efficiently, or iii) strike a good trade-off of the processing delay versus the resilience and the accuracy of tracking. To address these issues, we propose Adaptive Multi-Trace Carving (AMTC), a unified approach for detecting and tracking one or more subtle frequency components under very low signal-to-noise ratio (SNR) conditions and in near real time. AMTC takes as input a time-frequency representation of the system's preprocessing results (such as the spectrogram), and identifies frequency components through iterative dynamic programming and adaptive trace compensation. The proposed algorithm considers relatively high energy traces sustaining over a certain duration as an indicator of the presence of frequency/oscillation components of interest and track their time-varying trend. Extensive experiments using both synthetic data and real-world forensic data of power signatures and physiological monitoring reveal that the proposed method outperforms representative prior art under low SNR conditions, and can be implemented in near real-time settings. The proposed AMTC algorithm can empower the development of new information forensic technologies that harness very small signals.

This paper introduces an amplitude and frequency modulation (AM-FM) model to characterize the photoplethysmography (PPG) signal. The model indicates that the PPG signal spectrum contains one dominant frequency component - the heart rate (HR), which is guarded by two weaker frequency components on both sides; the distance from the dominant component to the guard components represents the respiratory rate (RR). Based on this model, an efficient algorithm is proposed to estimate both HR and RR by searching for the dominant frequency component and two guard components. The proposed method is performed in the frequency domain to estimate RR, which is more robust to additive noise than the prior art based on temporal features. Experiments were conducted on two types of PPG signals collected with a contact sensor (an oximeter) and a contactless visible imaging sensor (a color camera), respectively. The PPG signal from the contactless sensor is much noisier than the signal from the contact sensor. The experimental results demonstrate the effectiveness of the proposed algorithm, including under relatively noisy scenarios.

We present the model, design, and implementation of SMARS, the first practical Sleep Monitoring system that exploits Ambient Radio Signals to recognize sleep stages and assess sleep quality. This will enable a future smart home that monitors daily sleep in a ubiquitous, non-invasive and contactless manner, without instrumenting the subject's body or the bed. The key enabler underlying SMARS is a statistical model that accounts for all reflecting and scattering multipaths, allowing highly accurate and instantaneous breathing estimation with best-ever performance achieved on commodity devices. On this basis, SMARS then recognizes different sleep stages, including wake, rapid eye movement (REM), and non-REM (NREM), which was previously only possible with dedicated hardware. We implement a real-time system on commercial WiFi chipsets and deploy it in 6 homes, resulting in 32 nights of data in total. Our results demonstrate that SMARS yields a median absolute error of 0.47 breaths per minute (BPM) and a 95 percent-tile error of only 2.92 BPM for breathing estimation, and detects breathing robustly even when a person is 10 meters away from the link, or behind a wall. SMARS achieves a sleep staging accuracy of 88 percent, outperforming the prevalent unobtrusive commodity solutions using bed sensor or UWB radar. The performance is also validated upon a public sleep dataset of 20 patients. By achieving promising results with merely a single commodity RF link, we believe that SMARS will set the stage for a practical in-home sleep monitoring solution.

Work in Progress: This Innovative Practice Work-in-Progress Paper presents how a portable laboratory kit named Pocket Instruments has facilitated electronic laboratory learning in the undergraduate engineering programs. Traditionally, owing to the lack of necessary instruments at hand, students could only think abstractly and design on paper during pre-lab and post-lab time but could not physically work on their circuits outside the laboratories. Through the corresponding virtual instrument application software deployed on a laptop, the Pocket Instruments can act as a portable signal generator, an oscilloscope, a digital analyzer, and a spectrum analyzer, which makes it possible for students to experiment with electronic circuits freely outside the allocated laboratory time. The teaching practices in the engineering program of Tsinghua University for seven years have shown that such an always-available lab kit can achieve a measurable increase in students' engagement and achievement in experimental exploration. With the growing needs in online learning, the Pocket Instruments can also provide an effective means for distance learning of laboratory skill training.

Gesture recognition using wireless sensing opened a plethora of applications in the field of human-computer interaction. However, most existing works are not robust without requiring wearables or tedious training/calibration. In this work, we propose WiGRep, a time reversal based gesture recognition approach using Wi-Fi, which can recognize different gestures by counting the number of repeating gesture segments. Built upon the time reversal phenomenon in RF transmission, the Time Reversal Resonating Strength (TRRS) is used to detect repeating patterns in a gesture. A robust low-complexity algorithm is proposed to accommodate possible variations of gestures and indoor environments. The main advantages of WiGRep are that it is calibration-free and location and environment independent. Experiments performed in both line of sight and non-line-of-sight scenarios demonstrate a detection rate of 99.6% and 99.4%, respectively, for a fixed false alarm rate of 5%.

An emerging research direction considers the inverse problem of inferring electrocardiogram (ECG) from photoplethysmogram (PPG) to bring about the synergy between the easy measurability of PPG and the rich clinical knowledge of ECG to facilitate preventive healthcare. Previous reconstruction using a universal basis has limited accuracy due to the lack of rich representative power. This paper proposes a cross-domain joint dictionary learning (XDJDL) framework to maximize the expressive power for the two cross-domain signals. Building on K-SVD technique, XDJDL optimizes simultaneously the PPG and ECG signal representations and the transform between them, enabling the joint learning of a pair of signal dictionaries with a transform to characterize the relation between their sparse codes. The proposed model is evaluated with 34,000+ ECG/PPG cycle pairs containing a variety of ECG morphologies and cardiovascular diseases. Experimental results validate the accuracy and the generality of the proposed algorithm, suggesting an encouraging potential for disease screening using PPG measurement based on the proactive learned PPG-ECG relationship.

A variety of nearly invisible "micro-signals" have played important roles in media security and forensics. These noise-like micro-signals are ubiquitous and typically an order of magnitude lower in strength or scale than the dominant ones. They are traditionally removed or ignored as nuances outside the forensic domain. This keynote talk discusses the recent research harnessing micro-signals to infer a person's physiological conditions. One type of such signals is the subtle changes in facial skin color in accordance with the heartbeat. Video analysis of this repeating change provides a contact-free way to capture photo-plethysmogram (PPG). While heart rate can be tracked from videos of resting cases, it is challenging to do so for cases involving substantial motion, such as when a person is walking around, running on a treadmill, or driving on a bumpy road. It will be shown in this talk how the expertise with micro-signals from media forensics has enabled the exploration of new opportunities in physiological forensics and a broad range of applications.

Effective machine learning models can automatically learn useful information from a large quantity of data and provide decisions in a high accuracy. These models may, however, lead to unfair predictions in certain sense among the population groups of interest, where the grouping is based on such sensitive attributes as race and gender. Various fairness definitions, such as demographic parity and equalized odds, were proposed in prior art to ensure that decisions guided by the machine learning models are equitable. Unfortunately, the "fair" model trained with these fairness definitions is threshold sensitive, i.e., the condition of fairness may no longer hold true when tuning the decision threshold. This paper introduces the notion of threshold invariant fairness, which enforces equitable performances across different groups independent of the decision threshold. To achieve this goal, this paper proposes to equalize the risk distributions among the groups via two approximation methods. Experimental results demonstrate that the proposed methodology is effective to alleviate the threshold sensitivity in machine learning models designed to achieve fairness.

In the present evolving world, automobiles have become an intelligent electronic machine and are no longer a mere transport medium. In this article, we attempt to make them smarter by introducing the idea of in-car driver authentication using wireless sensing and develop a system that can recognize drivers automatically. The proposed system can recognize human identity by identifying the unique radio biometric information recorded in the channel state information (CSI) through multipath propagation. However, since the environmental information is also captured in the CSI, the performance of radio biometric recognition may be degraded by the changing environment. In this article, we first address the problem of “in-car changing environments” where the existing wireless sensing-based human identification system fails. We build a long-term driver radio biometric database consisting of radio biometrics of seven people collected over a period of two months. We leverage this database to create machine learning models that make the proposed system adaptive to new in-car environments. Second, we study the performance of the in-car driver authentication system with increasing effective bandwidth. We realize an effective bandwidth of 960 MHz by exploiting the multiantenna and frequency diversities in commercial WiFi devices. The performance of the proposed system is shown to improve with increasing effective bandwidth and the long-term experiments demonstrate the feasibility and accuracy of the proposed system. The accuracy achieved in the two-driver scenario is up to 99.13% for the best case.

Automobiles have become an essential part of everyday lives. In this work, we attempt to make them smarter by introducing the idea of in-car driver authentication using wireless sensing. Our aim is to develop a model which can recognize drivers automatically. Firstly, we address the problem of "changing in-car environments", where the existing wireless sensing based human identification system fails. To this end, we build the first in-car driver radio biometric dataset to understand the effect of changing environments on human radio biometrics. This dataset consists of radio biometrics of five people collected over a period of two months. We leverage this dataset-to create machine learning (ML) models that make the proposed system adaptive to new in-car environments. We obtained a maximum accuracy of 99.3% in classifying two drivers and 90.66% accuracy in validating a single driver.

By sensing wirelessly the radio propagation environment and analyzing the channel state information (CSI), one can extend human senses beyond our traditional reach and enrich the insight into the surrounding environment and activities, with or without line-of-sight. On one hand, different indoor activities bring distinctive perturbations to wireless radio propagations. On the other hand, thanks to the nature of multipaths, indoor environmental information is contained and embedded in the wireless CSI. Since the occurrence of an indoor event lasts for a certain period of duration and repeats a similar transition pattern among different realizations, information is embedded not only in each instantaneous CSI sample, but also in how CSI changes along time, e.g., the CSI time series. Inspired by that, this paper proposes an indoor monitoring system that monitors the occurrence of different indoor events in real time with commercial WiFi devices, by exploiting the temporal information embedded in the CSI time series. Through extensive experiments, this paper studies the robustness of the proposed system to variabilities in event instances and human motion interference, and its long-term performance in a one-month test.

The electric network frequency (ENF) signal is an environmental signature that can be captured in audiovisual recordings made in locations where there is electrical activity. This signal is influenced by the power grid in which the recording is made, and recent work has shown that it can be useful toward a number of forensics and security applications. An under-studied area of ENF research is the factors that can affect the capture of ENF traces in media recordings. Not all recordings made in the areas of electrical activity will carry prominent ENF traces, and the strengths by which the ENF traces are captured can vary from one recording to another. A thorough understanding of the factors that can affect the capture of ENF traces in recordings is essential to understanding the applicability of ENF-based approaches and can help inform related studies in the future. This paper carried out a study on such factors, with a focus on audio signals. The impact of the characteristics of an audio recorder and the environment and manner of recording on the intrinsically captured ENF are shown and analyzed.

Geo-tagging images of interest is increasingly important to law enforcement, national security, and journalism. Many images today do not carry location tags that are trustworthy and resilient to tampering; and the landmark-based visual clues may not be readily present in every image, especially in those taken indoors. In this paper, we exploit an invisible signature from the power grid, the Electric Network Frequency (ENF) signal, which can be inherently recorded in a sensing stream at the time of capturing and carries useful location information. It is, however, very challenging to extract an ENF signal from a single image, as compared to the recent art in extracting ENF traces from audio and video. This paper presents novel investigations toward this challenge, by synergistically exploring the rolling shutter effect of CMOS imaging sensors and entropy differences of composite signals. We study quantitatively the relationship between the ENF strength and its detectability from a single image, and bring out a unique forensics capability of invisible traces that shine a light on an image's capturing location.

Environmental information is recorded in the multipath propagation, and can be accessed in the form of channel state information (CSI) through commodity WiFi devices. A single CSI reading for event detection is adopted in most existing CSI based indoor monitoring system. However, due to the impact of noise, event inconsistency and environmental dynamics, this type of approaches is not very robust. In this work, we design efficient algorithms to fully exploit the information embedded in the CSI time series to combat interference introduced by CSI perturbations, and propose an indoor event monitoring system that achieves accurate real-time monitoring. The accuracy and robustness of the proposed system is evaluated through experiments, which illustrate its potential in future smart home applications.

The digital image forensics academic community is facing a growing challenge. The volume of images presented to digital image forensic practitioners increases every day, and with it, more variety of possible outcomes in image analysis. When an academic forensic tool is applied to a realistic case, the effects of imaging factors, including the noise level, should be seriously considered. Although there have been conjectures that shot noise would affect the empirical accuracy of a forensic analyzer, it has not yet received enough experimental support. In this paper, instead of estimating the noise, we inspect two measurable factors of exposure settings, ISO speed and exposure time, and present a set of experiments using mobile phone data to demonstrate the effect of exposure settings on steganalysis and PRNU-based camera device identification. Our results show that more investigations into the characteristics of image data with respect to exposure settings is required to fully understand identification or classification that is concerned with low-magnitude noise measurements, such as PRNU or stegoembedded messages.

Recent studies showed that subtle changes in human's face color due to the heartbeat can be captured by digital video recorders. Most existing work focused on still/rest cases or those with relatively small motions. In this work, we propose a heart-rate monitoring method for fitness exercise videos. We focus on designing a highly precise motion compensation scheme with the help of the optical flow, and use motion information as a cue to adaptively remove ambiguous frequency components for improving the heart rates estimates. Experimental results show that our proposed method can achieve highly precise estimation with an average error of 1.1 beats per minute (BPM) or 0.58% in relative error.

It is paramount to ensure secure and trustworthy operations in Cyber-Physical Systems (CPSs), guaranteeing the integrity of sensing data, enabling access control, and safeguarding system-level operations. In this paper, we address trustworthy operations of next generation CPSs. Our idea is inspired by a trustworthy computing framework known as Proof-Carrying Code, in which foreign executables carry a model to prove that they have not been tampered with and they function as expected. In our context, we leverage the physical world--a channel that encapsulates properties impossible to tamper with remotely, such as proximity and causality--to create a challenge-response function. We call it Proof-Carrying Sensing and use it to help authenticate devices, collected data, and locations. A unique advantage of this approach, vis-à-vis traditional multi-factor or out-of-band authentication mechanisms, is that authentication proofs are embedded in sensor data and can be continuously validated over time and space without resorting to complicated cryptographic algorithms. This, in turn, makes it fit particularly well to CPSs where mobility and resource constraints are common.

Camcorder piracy refers to the process of using a camcorder to record a screen that displays copyrighted content. In contrast to the previous works that aimed at detecting the occurrence of camcorder piracy, this paper conducts an in-depth study of the luminance flicker that is naturally present in camcorded videos due to the interplay between a liquid-crystal-display (LCD) screen and a camcorder. We first model the flicker signal and show that its parameters are tied to such internal characteristics of the pirate devices as the back-light frequency of the LCD screen and the read-out time of the camcorder. We then present new estimation techniques to recover these hidden parameters directly from camcorded videos and demonstrate that such forensic cues could provide intelligence on the pirate devices. We also discuss how to recover the shape of the low-power flicker signal itself and show that it could be used to infer which back-light technology employed in the pirate LCD screen.

This paper studies the authentication problem of specific pieces of paper using mobile imaging devices. Prior work showing high matching accuracy has used the normal vector field, which serves as a unique, microscopic, physically unclonable feature of paper surfaces, estimated by consumer grade scanners. Industrial cameras were also used to capture the appearance of the surface rendered after the normal vector field based on the laws of optics under a semi-controlled lighting condition. In comparison, past explorations based on mobile cameras were very limited and have not had substantial success in obtaining consistent appearance images due to the uncontrolled nature of the ambient light. We show in this paper that images captured by mobile cameras can be used for authentication when the camera flashlight is exploited for creating a semi-controlled lighting condition. We have proposed new algorithms to demonstrate that the normal vector field of paper surface can be estimated by using multiple camera-captured images of different viewpoints. Perturbation analysis shows that the proposed method is robust to inaccurate estimates of camera locations, and a matching accuracy of 10-4in equal error rate can be achieved using 6 to 8 images under a lab-controlled ambient light environment. Our findings can relax the restricted imaging setups and enable paper authentication under a more casual, ubiquitous setting with a mobile imaging device, which may facilitate duplicate detection of paper documents and counterfeit mitigation of merchandise packaging.

The electric network frequency (ENF) is a signature of power distribution networks that can be captured by multimedia signals recorded near electrical activity. This has led to the emergence of multiple forensic applications based on the use of ENF signals. Examples of such applications include validating the time of recording of an ENF-containing multimedia signal or inferring the grid in which it was recorded. In this letter, we explore a novel ENF-based application that seeks to characterize the camera that has produced a given video. Inspired by recent work on exploiting flicker for pirate device identification, we investigate the use of ENF captured in a video to characterize the camera that has produced the video through a nonintrusive procedure that estimates the camera's read-out time. The proposed technique has achieved a high accuracy in estimating this discriminating parameter with a relative estimation error within 1.5%.

Digitally acquired high dynamic range (HDR) video baseband signal can take 10-12 bits per color channel. It is economically important to be able to reuse the legacy 8 or 10-bit video codecs to efficiently compress the HDR video. Linear or nonlinear mapping on the intensity can be applied to the baseband signal to reduce the dynamic range before the signal is sent to the codec, and we refer to this range reduction step as a baseband quantization. We show analytically and verify using test sequences that the use of the baseband quantizer lowers the coding efficiency. Experiments show that as the baseband quantizer is strengthened by 1.6 bits, the drop of PSNR at a high bitrate is up to 1.60 dB. Our result suggests that in order to achieve high coding efficiency, information reduction of videos in terms of quantization error should be introduced in the video codec instead of on the baseband signal.

Low-cost physically unclonable functions (PUFs) can be deployed with consumer products to deter counterfeiting. An intrinsic physical property - unique textures of paper or other surface - has received strong interest. Extrinsically introduced features, such as randomly positioned bubbles and fiber segments, have also been deployed in the industry to facilitate authentication. This paper carries out a study to gain a better understanding in the factors affecting the authentication performance, with a consideration of the friendliness under mobile imaging. Comparisons are made for paper-based PUFs of different characteristics. It is found that the density of foreground objects have a dominant impact on the authentication performance.

The electric network frequency (ENF) is a signature of power distribution networks that can be captured by multimedia signals recorded near electrical activities. This has led to the emergence of multiple forensic applications based on the use of ENF signals. Examples of such applications include validating the time-of-recording of an ENF-containing multimedia signal or estimating its recording location based on concurrent reference signals from power grids. In this paper, we examine a novel ENF-based application that infers the power grid in which the ENF-containing multimedia signal was recorded without relying on the availability of concurrent power references. We investigate features based on the statistical differences in ENF variations between different power grids to serve as signatures for the region-of-recording of the media signal. We use these features in a multiclass machine learning implementation that is able to identify the grid-of-recording of a signal with high accuracy. In addition, we explore techniques for building multiconditional learning systems that can adapt to changes in the noise environment between the training and testing data.

The electric network frequency (ENF) signal can be captured in multimedia recordings due to electromagnetic influences from the power grid at the time of recording. Recent work has exploited the ENF signals for forensic applications, such as authenticating and detecting forgery of ENF-containing multimedia signals, and inferring their time and location of creation. In this paper, we explore a new potential of ENF signals for automatic synchronization of audio and video. The ENF signal as a time-varying random process can be used as a timing fingerprint of multimedia signals. Synchronization of audio and video recordings can be achieved by aligning their embedded ENF signals. We demonstrate the proposed scheme with two applications: multi-view video synchronization and synchronization of historical audio recordings. The experimental results show the ENF based synchronization approach is effective, and has the potential to solve problems that are intractable by other existing methods.

The electric network frequency (ENF) signal can be embedded in multimedia recordings created in areas of electrical activities. Recent work has used the ENF signal for such applications as time stamp authentication and forgery detection. It is more challenging to extract ENF signals from video recordings than from audio recordings because of the low temporal sampling rate or frame rate of video cameras. The rolling shutter of CMOS image sensor can be exploited as it exposes a frame line by line, and the effective ENF sampling rate by treating each line as a signal sample can be increased. This scheme was shown to work well with static videos. This paper conducts a further study on the exploitation of the rolling shutter for extracting ENF traces from videos. The rolling shutter mechanism is modeled and analyzed using multirate signal processing theory. Challenging cases of videos with motions are examined, and solutions to extracting ENF from them are explored.

Electric Network Frequency (ENF) fluctuations based forensic analysis is an emerging way for such multimedia authentication tasks as time-of-recording estimation, timestamp verification, and clip insertion/deletion forgery detection. ENF fluctuates due to dynamic changes in load demand and power supply, and these fluctuations travel over the power lines with a finite speed. In this paper, experiments are conducted on ENF data collected across different locations in the eastern grid of the United States to understand the relationship between the signals recorded at the same time at these locations. Based on these experiments, a signal processing mechanism is developed to demonstrate that ENF fluctuations across different locations exhibit a measure of similarity with each other, which is proportional to the distance between the locations. Such observations motivated a location estimation protocol based on the similarity of ENF signals with respect to anchor nodes. Under certain conditions, the proposed protocol is shown to provide an estimation accuracy of 90%. Challenges in the application of ENF signal analysis for location of recording estimation of multimedia signal are also discussed.

Electric Network Frequency (ENF) based forensic analysis is a promising tool for timestamp authentication and forgery detection in such multimedia recordings as audios and videos. ENF signal is embedded in an audio recording due to electromagnetic interference from the power lines. The time of creation of a multimedia recording can be determined by comparing the ENF signal embedded in the recording with a reference ENF database collected from the power grid. In this paper, we conduct a study of the effect of recapturing of audio recordings on the ENF embedding. We demonstrate that recaptured audio recordings pick up two ENF signals: the content ENF signal which is inherited from the original audio recording; and the recapturing ENF signal which is embedded from the recapturing process. Conventional ENF signal extraction techniques on such recordings may fail when the two ENF signals are at the same nominal value. A decorrelation algorithm is proposed to extract the content ENF signal and the recapturing ENF signal. The experimental results show the effectiveness of the proposed method in the estimation of both the ENF signals.

The Electric Network Frequency (ENF) is the supply frequency of power distribution networks, and is often captured by audio or video measurements recorded near power supplies. The time varying nature of the ENF allows it to be used for such forensic applications as estimating the time and location of media recordings, and discerning their integrity. An initial step in such applications is to extract the ENF signal-instantaneous ENF values over time-as accurately as possible. Existing techniques rely on estimating the ENF around the nominal frequency of 50/60 Hz, or around one of its harmonics at a time. In this letter, a novel spectrum combining approach is proposed, which exploits the presence of the ENF around different harmonics of the nominal frequency. The ENF signal is estimated by combining the ENF at multiple harmonics, based on the local signal-to-noise ratio at each harmonic. A hypothesis testing performance of an ENF-based timestamp verification application is examined to validate that the proposed approach achieves a more robust and accurate performance than conventional ENF estimation techniques.

The electrical network frequency (ENF) signal is a time stamp that has been used by an emerging class of approaches for determining the creation time of digital audio and video recordings. However, in adversarial environments, anti-forensic operations may be conducted to manipulate ENF-based time stamps, and it is crucial to understand the resilience of ENF analysis against anti-forensics. This paper explores possible anti-forensic operations that can remove and alter the ENF signal while trying to preserve the host signal, and devises detection methods targeting these operations. Concealment techniques that can circumvent detection are also discussed and their corresponding trade-offs are examined. Based on the understanding of individual anti-forensic operations and countermeasures, this paper further characterizes the dynamic interplay between forensic analysts and adversaries by providing an evolutionary perspective and a game-theoretical perspective as well as studying representative scenarios and the optimal forensic/anti-forensic strategies.

Electric Network Frequency (ENF) fluctuates slightly over time from its nominal value of 50 Hz/60 Hz. The fluctuations in ENF remain consistent across the entire power grid including when measured at physically distant geographical locations. The light intensity from such indoor lighting as fluorescent lamps and incandescent bulbs, which are connected to the power mains, varies in accordance with the ENF, and the fluctuations can be recorded using visual sensors. In this paper, mechanisms using optical sensors and video cameras to record and validate the presence of ENF fluctuations in indoor lighting are presented. Spectrogram and subspace-based signal processing techniques are applied to such recordings to extract the ENF signal by estimating its instantaneous frequencies as a function of time. A high correlation is observed between the ENF fluctuations obtained from indoor lighting and that of the ENF signal captured directly from the power mains supply. A similar mechanism is then used to demonstrate the presence of ENF signals in video recordings taken in different geographical areas. Experimental results show that ENF signals are present in visual recordings made in different geographical areas and can be used as a natural timestamp for optical sensor recordings and video surveillance recordings conducted in indoor lighting environments. Robustness of ENF fluctuation traces under strong compression and CMOS rolling shutter cameras is discussed. Applications of the ENF signal analysis to tampering detection of surveillance video recordings and forensic binding of the audio and visual track of a video are also demonstrated. An analytical model based on an autoregressive process is also developed for ENF signals, and the effectiveness of using innovation sequences from the model for timestamp verification is demonstrated.

A time stamp based on the power network signature called the Electrical Network Frequency (ENF) has been used by an emerging class of approaches for authenticating digital audio and video recordings in computer-to-computer communications. However, the presence of adversaries may render the time stamp insecure, and it is crucial to understand the robustness of ENF analysis against anti-forensic operations. This paper investigates possible anti-forensic operations that can remove and alter the ENF signal while trying to preserve the host signal, and develops detection methods targeting these operations. Improvements over anti-forensic operations that can circumvent the detection are also examined, for which various trade-offs are discussed. To develop an understanding of the dynamics between a forensic analyst and an adversary, an evolutionary perspective and a game-theoretical perspective are proposed, which allow for a comprehensive characterization of plausible anti-forensic strategies and countermeasures. Such an understanding has the potential to lead to more secure and reliable time stamp schemes based on ENF analysis.

Localization of constituent nodes is of fundamental importance in many wireless sensor networks (WSNs) related applications. Existing research has mainly investigated the problem of localization in static WSNs, where the localization is performed mainly at the time of the node deployment. In contrast, it is important to keep track of the current locations of the nodes by invoking the localization algorithm periodically in mobile nodes. The high computation cost associated with most existing localization algorithms makes them less practical to use in resource constrained mobile sensor networks (MSNs). Additionally, these existing techniques often fail in hostile environments where some of the nodes may be compromised by adversaries, and used to transmit misleading information aimed at preventing accurate localization of the remaining sensors. In this paper, we build on our earlier work to propose an iterative gradient descent based technique with low computational complexity to securely localize nodes in MSNs. The proposed algorithm combines iterative gradient descent with selective pruning of inconsistent measurements to achieve a high localization accuracy. Simulation results demonstrate that the proposed algorithm can find a map of relative locations of the MSN even when some nodes are compromised and transmit false information.

Recent years have witnessed tremendous growth in the generation and consumption of digital images. Monitoring and evaluating image quality is an important issue for online and mobile media applications. Conventional quality assessment work mostly focus on intensity level distortion caused by operations that do not change image size. In this work, we study the problem of quality assessment for images having undergone content-adaptive resizing, also known as retargeting operations. We design a reduced-reference algorithm to analyze the structural distortion caused by retargeting and propose a quality score that achieves positive correlation with human observations. The quality score and distortion analysis from the proposed algorithm provide rich information for both objective and subjective quality assessment tasks.

Scale-Invariant Feature Transform (SIFT) is one of the most popular local image features that are widely used in computer vision, image processing and image retrieval. In this paper we study the relation between the SIFT descriptor and its matching accuracy. We propose a method to quantitatively assess the quality of a SIFT feature descriptor in terms of robustness and discriminability. This would enable us to gain a better understanding of the strength and limitations of SIFT in emerging applications of SIFT-based image hash, and also to improve matching accuracy and efficiency in applications such as object search. The experimental results demonstrate the effectiveness of the proposed method.

Many applications of wireless sensor networks require precise knowledge of the locations of constituent nodes. In these applications, it is desirable for the nodes to be able to autonomously determine their locations before they start sensing and transmitting data. Most localization algorithms use anchor nodes with known locations to determine the positions of the remaining nodes. However, these existing techniques often fail in hostile environments where some of the nodes may be compromised by adversaries and used to transmit misleading information aimed at preventing accurate localization of the remaining sensors. In this paper, a computationally efficient secure localization algorithm that withstands such attacks is described. The proposed algorithm combines iterative gradient descent with selective pruning of inconsistent measurements to achieve high localization accuracy. Results show that the proposed algorithm utilizes fewer computational resources and achieves an accuracy better than or comparable to that of existing schemes. The proposed secure localization algorithm can also be used in mobile sensor networks, where all nodes are moving, to estimate the relative locations of the nodes without relying on anchor nodes. Simulations demonstrate that the proposed algorithm can find the relative location map of the entire mobile sensor network even when some nodes are compromised and transmit false information.

Secure signal processing is an emerging technology to enable signal processing tasks in a secure and privacy-preserving fashion. It has attracted a great amount of research attention due to the increasing demand to enable rich functionalities for private data stored online. Desirable functionalities may include search, analysis, clustering, etc. In this paper, we discuss the research issues and challenges in secure video processing with focus on the application of secure online video management. Video is different from text due to its large data volume and rich content diversity. To be practical, secure video processing requires efficient solutions that may involve a trade-off between security and complexity. We look at three representative video processing tasks and review existing techniques that can be applied. Many of the tasks do not have efficient solutions yet, and we discuss the challenges and research questions that need to be addressed.

Previous analysis of content fingerprints has mainly focused on the case of independent and identically distributed finger prints. Practical fingerprints, however, exhibit correlations between components computed from successive frames. In this paper, a Markov chain based model is used to capture the temporal correlations, and the suitability of this model is evaluated through experiments on a video database. The results indicate that the Markov chain model is a good fit only in a certain regime. A hybrid model is then developed to account for this behavior and a corresponding adaptive detector is derived. The adaptive detector achieves better identification accuracy at a small computational expense.

This paper presents a study of the video compression effect on source camera identification based on the Photo-Response Non-Uniformity (PRNU). Specifically, the reliability of different types of frames in a compressed video is first investigated, which shows quantitatively that I-frames are more reliable than P-frames for PRNU estimation. Motivated by this observation, a new mechanism for estimating the reference PRNU and two mechanisms for estimating the test-video PRNU are proposed to achieve higher accuracy with fewer frames used. Experiments are performed to validate the effectiveness of the proposed mechanisms.

Electric Network Frequency (ENF) fluctuates slightly over time from its nominal value of 50 Hz/60 Hz. The fluctuations in the ENF remain consistent across the entire power grid even when measured at physically distant locations. The near-invisible flickering of fluorescent lights connected to the power mains reflect these fluctuations present in the ENF. In this paper, mechanisms using optical sensors and video cameras to record and validate the presence of the ENF fluctuations in fluorescent lighting are presented. Signal processing techniques are applied to demonstrate a high correlation between the fluctuations in the ENF signal captured from fluorescent lighting and the ENF signal captured directly from power mains supply. The proposed technique is then used to demonstrate the presence of the ENF signal in video recordings taken in various geographical areas. Experimental results show that the ENF signal can be used as a natural timestamp for optical sensor recordings and video surveillance recordings from indoor environments under fluorescent lighting. Application of the ENF signal analysis to tampering detection of surveillance video recordings is also demonstrated.

Multimedia identification via content fingerprints is used in many applications, such as content filtering on user-generated content websites, and automatic multimedia identification and tagging. A compact “fingerprint” is computed for each multimedia signal that captures robust and unique properties of the perceptual content, which is later used for identifying the multimedia. Several different multimedia fingerprinting schemes have been proposed in the literature and have been evaluated through experiments. To complement these experimental evaluations and provide guidelines for choosing system parameters and designing better schemes, this paper develops models for content fingerprinting and provides an analysis of the identification performance under these models. As a first step, bounds on the identification accuracy and the required fingerprint length for the simplest case when the fingerprint bits are modeled as i.i.d. are summarized. Markov Random Fields are then used to address more realistic settings of fingerprints with correlated components. The optimal likelihood ratio detector is derived and a statistical physics inspired approach for computing the probability of detection and probability of false alarm is described. The analysis shows that the commonly used Hamming distance detection criterion is susceptible to correlations among fingerprint bits, whereas the optimal log-likelihood ratio decision rule yields 5-20% improvement in the accuracy over a range of correlations. Simulation results demonstrate the validity of the theoretical predictions.

Many sensor network related applications require precise knowledge of the location of constituent nodes. In these applications, it is desirable for the wireless nodes to be able to autonomously determine their locations before they start sensing and transmitting data. Most localization algorithms rely on anchor nodes whose locations are known to determine the positions of the remaining nodes. In an adversarial scenario, some of these anchor nodes could be compromised and used to transmit misleading information aimed at preventing the accurate localization of the remaining sensors. In this paper, a computationally efficient algorithm to determine the location of sensors that can resist such attacks is described. The proposed algorithm combines gradient descent with a selective pruning of inconsistent measurements to achieve good localization accuracy. Simulation results show that the proposed algorithm has performance comparable to existing schemes while requiring less computational resources.

Content fingerprinting provides a compact representation of multimedia objects for copy identification. This paper analyzes the impact of the ordinal-ranking based feature encoding on the performance of content fingerprinting. Expressions are derived for the identification performance of a fingerprinting system with and without ordinal ranking. The analysis indicates that when the number of features is moderately large, ordinal ranking can improve the robustness of the fingerprinting system to large distortions of the features and significantly increase the probability of detection. These results enhance understandings of ordinal ranking and provide design guidelines for choosing different system parameters to achieve a desired identification accuracy.

With the increasing popularity of digital multimedia such as images and videos and the advent of the cloud computing paradigm, a fast growing amount of private and sensitive multimedia data are being stored and managed over the network cloud. To provide enhanced security and privacy protection beyond traditional access control techniques, privacy preserving multimedia retrieval techniques have been proposed recently to allow content-based multimedia retrieval directly over encrypted databases and achieve accurate retrieval comparable to conventional retrieval schemes. In this paper, we introduce a security definition for the privacy preserving retrieval scenario and show that the recently proposed schemes are secure under the proposed security definition.

In recent years, digital images and videos have become increasingly popular over the internet and bring great social impact to a wide audience. In the meanwhile, technology advancement allows people to easily alter the content of digital multimedia and brings serious concern on the trustworthiness of online multimedia information. Forensic hash is a short signature attached to an image before transmission and acts as side information for analyzing the processing history and trustworthiness of the received image. In this paper, we propose a new construction of forensic hash based on visual words representation. We encode SIFT features into a compact visual words representation for robust estimation of geometric transformations and propose a hybrid construction using both SIFT and block-based features to detect and localize image tampering. The proposed hash construction achieves more robust and accurate forensic analysis than prior work.

With the widespread popularity of digital images and the presence of easy-to-use image editing software, content integrity can no longer be taken for granted, and there is a strong need for techniques that not only detect the presence of tampering but also identify its type. This paper focusses on tampering-type identification and introduces a new approach based on the empirical frequency response (EFR) to address this problem. We show that several types of tampering operations, both linear shift invariant (LSI) and non-LSI, can be characterized consistently and distinctly by their EFRs. We then extend the approach to estimate the EFR for scenarios where only the final image is available. Theoretical reasoning supported by experimental results verify the effectiveness of this method for identifying the type of a tampering operation.

This paper addresses the problem of image retrieval from an encrypted database, where data confidentiality is preserved both in the storage and retrieval process. The paper focuses on image feature protection techniques which enable similarity comparison among protected features. By utilizing both signal processing and cryptographic techniques, three schemes are investigated and compared, including bit-plane randomization, random projection, and randomized unary encoding. Experimental results show that secure image retrieval can achieve comparable retrieval performance to conventional image retrieval techniques without revealing information about image content. This work enriches the area of secure information retrieval and can find applications in secure online services for images and videos.

Content fingerprinting provides a compact content-based representation of a multimedia document. An important application of fingerprinting is the identification of modified copies of the original media content. These modifications may be incidental changes that occur during the usage of multimedia, or intentional modifications made by an adversary to avoid detection. Currently, the effectiveness of content identification techniques is often assessed through benchmark databases. To complement these experimental performance evaluations, this paper develops a theoretical framework for analyzing content identification techniques. Beneficial aspects from decision theory and game theory are exploited to gain insights toward optimal system design and parameter selection.

A large portion of digital images available today are acquired using digital cameras or scanners. While cameras provide digital reproduction of natural scenes, scanners are often used to capture hard-copy art in a more controlled environment. In this paper, new techniques for nonintrusive scanner forensics that utilize intrinsic sensor noise features are proposed to verify the source and integrity of digital scanned images. Scanning noise is analyzed from several aspects using only scanned image samples, including through image denoising, wavelet analysis, and neighborhood prediction, and then obtain statistical features from each characterization. Based on the proposed statistical features of scanning noise, a robust scanner identifier is constructed to determine the model/brand of the scanner used to capture a scanned image. Utilizing these noise features, we extend the scope of acquisition forensics to differentiating scanned images from camera-taken photographs and computer-generated graphics. The proposed noise features also enable tampering forensics to detect postprocessing operations on scanned images. Experimental results are presented to demonstrate the effectiveness of employing the proposed noise features for performing various forensic analysis on scanners and scanned images.

Digital fingerprinting is a technique to deter unauthorized redistribution of multimedia content by embedding a unique identifying signal in each legally distributed copy. The embedded fingerprint can later be extracted and used to trace the originator of an unauthorized copy. A group of users may collude and attempt to create a version of the content that cannot be traced back to any of them. As multimedia data is commonly stored in compressed form, this paper addresses the problem of fingerprinting compressed signals. Analysis is carried out to show that due to the quantized nature of the host signal and the embedded fingerprint, directly extending traditional fingerprinting techniques for uncompressed signals to the compressed case leads to low collusion resistance. To overcome this problem and improve the collusion resistance, a new technique for fingerprinting compressed signals called Anti-Collusion Dither (ACD) is proposed, whereby a random dither signal is added to the compressed host before embedding so as to make the effective host signal appear more continuous. The proposed technique is shown to reduce the accuracy with which attackers can estimate the host signal, and from an information theoretic perspective, the proposed ACD technique increases the maximum number of users that can be supported by the fingerprinting system under a given attack. Both analytical and experimental studies confirm that the proposed technique increases the probability of identifying a guilty user and can approximately quadruple the collusion resistance compared to conventional Gaussian fingerprinting.

High fidelity is a demanding requirement for data hiding, especially for images with artistic or medical value. This correspondence proposes a high-fidelity image watermarking for annotation with robustness to moderate distortion. To achieve the high fidelity of the embedded image, we introduce a visual perception model that aims at quantifying the local tolerance to noise for arbitrary imagery. Based on this model, we embed two kinds of watermarks: a pilot watermark that indicates the existence of the watermark and an information watermark that conveys a payload of several dozen bits. The objective is to embed 32 bits of metadata into a single image in such a way that it is robust to JPEG compression and cropping. We demonstrate the effectiveness of the visual model and the application of the proposed annotation technology using a database of challenging photographic and medical images that contain a large amount of smooth regions.

Content identification has many applications, ranging from preventing illegal sharing of copyrighted content on video sharing websites, to automatic identification and tagging of content. Several content identification techniques based on watermarking or robust hashes have been proposed in the literature, but they have mostly been evaluated through experiments. This paper analyzes binary hash-based content identification schemes under a decision theoretic framework and presents a lower bound on the length of the hash required to correctly identify multimedia content that may have undergone modifications. A practical scheme for content identification is evaluated under the proposed framework. The results obtained through experiments agree very well with the performance suggested by the theoretical analysis.

With widespread availability of digital images and easy-to-use image editing softwares, the origin and integrity of digital images has become a serious concern. This paper introduces the problem of image acquisition forensics and proposes a fusion of a set of signal processing features to identify the source of digital images. Our results show that the devices' color interpolation coefficients and noise statistics can jointly serve as good forensic features to help accurately trace the origin of the input image to its production process and to differentiate between images produced by cameras, cell phone cameras, scanners, and computer graphics. Further, the proposed features can also be extended to determining the brand and model of the device. Thus, the techniques introduced in this work provide a unified framework for image acquisition forensics.

Component forensics is an emerging methodology for forensic analysis that aims at estimating the algorithms and parameters in each component of a digital device. This paper proposes a theoretical foundation to examine the performance limits of component forensics. Using ideas from pattern classification theory, we define formal notions of identifiability of components in the information processing chain. We show that the parameters of certain device components can be accurately identified only in controlled settings through semi non-intrusive forensics, while the parameters of some others can be computed directly from the available sample data via complete non-intrusive analysis. We then extend the proposed theoretical framework to quantify and improve the accuracies and confidence in component parameter identification for several forensic applications.

Digital imaging has experienced tremendous growth in recent decades, and digital camera images have been used in a growing number of applications. With such increasing popularity and the availability of low-cost image editing software, the integrity of digital image content can no longer be taken for granted. This paper introduces a new methodology for the forensic analysis of digital camera images. The proposed method is based on the observation that many processing operations, both inside and outside acquisition devices, leave distinct intrinsic traces on digital images, and these intrinsic fingerprints can be identified and employed to verify the integrity of digital data. The intrinsic fingerprints of the various in-camera processing operations can be estimated through a detailed imaging model and its component analysis. Further processing applied to the camera captured image is modelled as a manipulation filter, for which a blind deconvolution technique is applied to obtain a linear time-invariant approximation and to estimate the intrinsic fingerprints associated with these postcamera operations. The absence of camera-imposed fingerprints from a test image indicates that the test image is not a camera output and is possibly generated by other image production processes. Any change or inconsistencies among the estimated camera-imposed fingerprints, or the presence of new types of fingerprints suggest that the image has undergone some kind of processing after the initial capture, such as tampering or steganographic embedding. Through analysis and extensive experimental studies, this paper demonstrates the effectiveness of the proposed framework for nonintrusive digital image forensics.

This paper proposes a distributed multi-point video conferencing system over packet erasure channels, where the aggregation of multiple video streams and resource allocation are performed in a distributed manner. Video stream combiners, which are located in different geographical areas and serve as portals for conferees, aggregate incoming streams supplied by local users with other streams aggregated from nearby video stream combiners. A packet-division multiple-access (PDMA)-based error protection scheme is proposed to be performed at each video stream combiner to minimize the maximal expected video distortion among aggregated streams. The proposed error protection scheme for multi-stream aggregation also supports user preference. In order to deliver video streams to end users with different preferred quality, a consensus algorithm is proposed to adaptively perform resource allocation based on user preference. Simulation results show that the proposed multi-stream aggregation and error protection scheme has significant gains over traditional multi-stream error protection schemes for a multi-point video conferencing system.

This paper considers protecting multimedia content from unauthorized redistribution in subscription based services, where adversaries work together to pirate multiple multimedia programs during a subscription period. Collusion-resistant fingerprinting is an emerging tool for traitor tracing. However, most of the existing fingerprinting works did not consider multiple rounds of interaction between collusion and detection. In this paper, we exploit the temporal dimension and propose a dynamic fingerprinting scheme that adjusts the fingerprint design based on the detection result of previously pirated signal. We also examine colluders' strategies to combat the tracing by dynamic fingerprinting. Both analytical and simulation results show that the proposed dynamic fingerprinting provides better collusion resistance than conventional static fingerprinting.

Digital fingerprinting is an emerging tool to protect multimedia content from unauthorized distribution by embedding a unique fingerprint into each user's copy. Although several fingerprinting schemes have been proposed in related work, disproportional effort has been targeted towards identifying effective collusion attacks on fingerprinting schemes. Recent introduction of the gradient attack has refined the definition of an optimal attack and demonstrated strong effect on direct-sequence, uniformly distributed, and Gaussian spread spectrum fingerprints when applied to synthetic signals. In this paper, we apply the gradient attack on an existing well-engineered video fingerprinting scheme, refine the attack procedure, and demonstrate that the gradient attack is effective on Laplace fingerprints. Finally, we explore an improvement on fingerprint design to thwart the gradient attack. Results suggest that Laplace fingerprint should be avoided. However, we show that a signal mixed of Laplace and Gaussian fingerprints may serve as a design strategy to disable the gradient attack and force pirates into averaging as a form of adversary collusion.

This paper considers the problem of semi non-intrusive component forensics and proposes a methodology to identify the algorithms and parameters employed by various processing modules inside a digital camera. The proposed analysis techniques assume the availability of the camera; and introduce a forensic methodology to estimate the parameters of the color interpolation and white balancing algorithms employed in cameras. We devise testing conditions, and design good input patterns to improve the overall accuracy in parameter estimation. As demonstrated by the results in the paper, the proposed techniques provide a much lower estimation bias and variance compared to non-intrusive analysis. The features obtained from component forensic analysis provide useful evidence for such applications as analyzing technology evolution trend, detecting technology infringement/licensing, protecting intellectual property rights, and determining camera source.

Most existing collusion-resistant fingerprinting techniques are for fingerprinting uncompressed signals. In this paper, we first study the performance of the traditional Gaussian based spread spectrum sequences for fingerprinting compressed signals and show that the system can be easily defeated by averaging or taking the median of a few copies. To overcome the collusion problem for compressed multimedia host signals, we propose a technique called anti-collusion dithering to mimic an uncompressed signal. Results show higher probability of catching a colluder using the proposed scheme compared to using Gaussian based fingerprints.

With increasing availability of low-cost image editing softwares, the authenticity of digital images can no longer be taken for granted. Digital images have also been used as cover data for transmitting secret information in the field of steganography. In this paper, we introduce a new set of features for multimedia forensics to determine if a digital image is an authentic camera output or if it has been tampered or embedded with hidden data. We perform such image forensic analysis employing three sets of statistical noise features, including those from denoising operations, wavelet analysis, and neighborhood prediction. Our experimental results demonstrate that the proposed method can effectively distinguish digital images from their tampered or stego versions.

Hiding data in binary images can facilitate authentication of important documents in the digital domain, which generally requires a high embedding payload. Recently, a steganography framework known as the wet paper coding has been employed in binary image watermarking to achieve high embedding payload. In this paper, we introduce a new concept of super-pixels, and study how to incorporate them in the framework of wet paper coding to further improve the embedding payload in binary images. Using binary text documents as an example, we demonstrate the effectiveness of the proposed super-pixel technique.

In this paper, we analyze the effect of various collusion attacks on fingerprinting systems for compressed multimedia. We evaluate the effectiveness of the collusion attacks in terms of the probability of detection and accuracy in estimating the host signal. Our analysis shows that applying averaging collusion on copies of moderately compressed content gives a highly accurate estimation of the host, and can effectively remove the embedded fingerprints. Averaging is thus the best choice for an attacker as the probability of detection and the distortion introduced are the lowest.

With a rapid growth of imaging technologies and an increasingly widespread usage of digital images and videos for a large number of high security and forensic applications, there is a strong need for techniques to verify the source and integrity of digital data. Component forensics is new approach for forensic analysis that aims to estimate the algorithms and parameters in each component of the digital device. In this paper, we develop a novel theoretical foundation to understand the fundamental performance limits of component forensics. We define formal notions of identifiability of components in the information processing chain, and present methods to quantify the accuracies at which the component parameters can be estimated. Building upon the proposed theoretical framework, we devise methods to improve the accuracies of component parameter estimation for a wide range of forensic applications.

Grouping strategy has been proposed recently to leverage the prior knowledge on collusion pattern to improve collusion resistance in multimedia fingerprinting. However, the improvement is not consistent, as reduced performance of the existing group-based fingerprinting schemes than the non-grouped ones is observed when the grouping does not match the true collusion pattern well. In this letter, we propose a new adaptive detection method, where the threshold for the group detection can be adjusted automatically according to the detection statistics that reflect the underlying collusion pattern. Experimental results show that the proposed adaptive detection outperforms nonadaptive detection and provides consistent performance improvement over non-grouped orthogonal fingerprinting schemes under various collusion scenarios.

Digital fingerprinting protects multimedia content from illegal redistribution by uniquely marking copies of the content distributed to users. Most existing multimedia fingerprinting schemes consider a user set on the scale of thousands. However, in such real-world applications as video-on-demand distribution, the number of potential users can be as many as 10-100 million. This large user size demands not only strong collusion resistance but also high efficiency in fingerprint construction, and detection, which makes most existing schemes incapable of being applied to these applications. A recently proposed joint coding and embedding fingerprinting framework provides a promising balance between collusion resistance, efficient construction, and detection, but several issues remain unsolved for applications involving a large group of users. In this paper, we explore how to employ the joint coding and embedding framework and develop practical algorithms to fingerprint video in such challenging settings as to accommodate more than ten million users and resist hundreds of users' collusion. We investigate the proper code structure for large-scale fingerprinting and propose a trimming detection technique that can reduce the decoding computational complexity by more than three orders of magnitude at the cost of less than 0.5% loss in detection probability under moderate to high watermark-to-noise ratios. Both analytic and experimental results show a high potential of joint coding and embedding to meet the needs of real-world large-scale fingerprinting applications.

A cooperative communication system explores a new dimension of diversity in wireless communications to combat the unfriendly wireless environment. While this emerging technology is promising in improving communication quality, some security problems inherent to cooperative relay also arise. This paper investigates the security issues in cooperative communications under the context of multiple relay nodes using decode-and-forward strategy, where one of the relay nodes is adversarial and tries to corrupt the communications by sending garbled signals. We show that the conventional physical-layer signal detector will lead to a high error rate in signal detection in such a scenario, and the application-layer cryptography alone will not be able to distinguish the adversarial relay from legitimate ones. To trace and identify the adversarial relay, we propose a cross-layer tracing scheme that uses adaptive signal detection at the physical layer, coupled with pseudorandom tracing symbols at the application layer. Analytical results for tracing statistics as well as experimental simulations are presented to demonstrate the effectiveness of the proposed tracing scheme

An image hash is a randomized compact representation of image content and finds applications in image authentication, image and video watermarking, and image similarity comparison. Usually, an image-hashing scheme is required to be robust and secure, and the security issue is particularly important in applications, such as multimedia authentication, watermarking, and fingerprinting. In this paper, we investigate the security of image hashing from the perspective of unicity distance, a concept pioneered by Shannon in one of his seminal papers. Using two recently proposed image-hashing schemes as representatives, we show that the concept of unicity distance can be adapted to evaluate the security of image hashing. Our analysis shows that the secret hashing key, or its equivalent form, can be estimated with high accuracy when the key is reused several dozen times. The estimated unicity distance determines the maximum number of key reuses in the investigated hashing schemes. A countermeasure of randomized key initialization is discussed to avoid key reuse and strengthen the security of robust image hashing.

Rapid technology development and the widespread use of visual sensors have led to a number of new problems related to protecting intellectual property rights, handling patent infringements, authenticating acquisition sources, and identifying content manipulations. This paper introduces nonintrusive component forensics as a new methodology for the forensic analysis of visual sensing information, aiming to identify the algorithms and parameters employed inside various processing modules of a digital device by only using the device output data without breaking the device apart. We propose techniques to estimate the algorithms and parameters employed by important camera components, such as color filter array and color interpolation modules. The estimated interpolation coefficients provide useful features to construct an efficient camera identifier to determine the brand and model from which an image was captured. The results obtained from such component analysis are also useful to examine the similarities between the technologies employed by different camera models to identify potential infringement/licensing and to facilitate studies on technology evolution

Cooperative communication system explores a new dimension of diversity in wireless communications to combat unfriendly wireless environment through strategic relays. While this emerging technology is promising in improving communication quality, some security problems inherent to cooperative relay also arise. In this paper we investigate the security issues in cooperative communications that consist of multiple relay nodes using decode-and-forward strategy. In particular, we consider the situation where one of the relay nodes is adversarial and tries to corrupt the communications by sending garbled signals. We show that the conventional physical-layer signal detection will not be effective in such a scenario, and the application-layer cryptography alone is not sufficient to identify the adversarial relay. To combat adversarial relay, we propose a cross-layer scheme that uses pseudo-random tracing symbols, with an adaptive signal detection rule at the physical layer, and direct sequence spread spectrum symbol construction at the application layer for tracing and identifying adversarial relay. Our experimental simulations show that the proposed tracing scheme is effective and efficient.

This paper considers the problem of non-intrusive forensic analysis of the individual components in visual sensors and its implementation. As a new addition to the emerging area of forensic engineering, we present a framework for analyzing technologies employed inside digital cameras based on output images, and develop a set of forensic signal processing algorithms for visual sensors based on color array sensor and interpolation methods. We show through simulations that the proposed method is robust against compression and noise, and can help identify various processing components inside the camera. Such a non-intrusive forensic framework would provide useful evidence for analyzing technology infringement and evolution for visual sensors.

Digital fingerprinting protects multimedia content from illegal redistribution by uniquely marking copies of the content distributed to each user. Most existing multimedia fingerprinting schemes consider userset on the scale of thousands for evaluation. However, in real applications such as cable TV and DVD distribution, the potential users can be as many as 10~100 million. This large scale user set demands not only high collusion resistance but also high efficiency in the fingerprint construction and detection, which makes most existing schemes incapable of practical application. A recently proposed joint coding and embedding fingerprinting framework provides a promising balance between collusion resistance, efficient construction and detection. In this paper, we explore how to employ such a framework and develop practical algorithms to fingerprint video in such challenging practical settings as to accommodate more than ten million users and resisting hundreds of users' collusion. Both analysis and experimental results show the great potential of joint coding and embedding fingerprinting for real video fingerprinting applications.

In this paper, we consider aggregating multiple video streams and transmitting the merged stream over multiple error-prone paths. We propose a novel multi-stream error protection scheme based on packet division multiplexing access (PDMA) and cross-path forward error coding. Compared with the traditional time division multiplexing access (TDMA)-based error protection scheme, the proposed scheme outperforms 1.43~1.88 dB for the averaged PSNR of all received streams.

Digital images have been used in growing number of applications from law enforcement and surveillance, to medical diagnosis and consumer photography. With such widespread popularity and the presence of low-cost image editing softwares, the integrity of image content can no longer be taken for granted. In this paper, we propose a novel technique based on blind deconvolution to verify image authenticity. We consider the direct output images of a camera as authentic, and introduce algorithms to detect further processing such as tampering applied to the image. Our proposed method is based on the observation that many tampering operations can be approximated as a combination of linear and non-linear components. We model the linear part of the tampering process as a filter, and obtain its coefficients using blind deconvolution. These estimated coefficients are then used to identify possible manipulations. We demonstrate the effectiveness of the proposed image authentication technique and compare our results with existing works.

In this paper, we propose a novel multi-point video conferencing system through error-prone channels, where the aggregation of multiple video streams and resource allocation are performed in a distributed manner. Video stream combiners, who are located in different geographical areas and serve as portals for conferees, aggregate incoming streams supplied by local users with other streams aggregated from nearby video stream combiners. A distributed multi-stream error protection scheme is performed in each video stream combiner to minimize the maximal expected video distortion among all aggregated streams. The simulation results demonstrate that our proposed scheme outperforms the traditional multicasting scheme by 1 dB~1.4 dB in terms of average PSNR

In this paper, we propose a framework to transmit multiple scalable video programs over downlink multiuser orthogonal frequency division multiplex (OFDM) networks in real time. The framework explores the scalability of the video codec and multidimensional diversity of multiuser OFDM systems to achieve the optimal service objectives subject to constraints on delay and limited system resources. We consider two essential service objectives, namely, the fairness and efficiency. Fairness concerns the video quality deviation among users who subscribe the same quality of service, and efficiency relates to how to attain the highest overall video quality using the available system resources. We formulate the fairness problem as minimizing the maximal end-to-end distortion received among all users and the efficiency problem as minimizing total end-to-end distortion of all users. Fast suboptimal algorithms are proposed to solve the above two optimization problems. The simulation results demonstrated that the proposed fairness algorithm outperforms a time division multiple (TDM) algorithm by 0.5 ~ 3 dB in terms of the worst received video quality among all users. In addition, the proposed framework can achieve a desired tradeoff between fairness and efficiency. For achieving the same average video quality among all users, the proposed framework can provide fairer video quality with 1 ~ 1.8 dB lower PSNR deviation than a TDM algorithm

Digital fingerprinting protects multimedia content from illegal redistribution by uniquely marking every copy of the content distributed to each user. The collusion attack is a powerful attack where several different fingerprinted copies of the same content are combined together to attenuate or even remove the fingerprints. One major category of collusion-resistant fingerprinting employs an explicit step of coding. Most existing works on coded fingerprinting mainly focus on the code-level issues and treat the embedding issues through abstract assumptions without examining the overall performance. In this paper, we jointly consider the coding and embedding issues for coded fingerprinting systems and examine their performance in terms of collusion resistance, detection computational complexity, and distribution efficiency. Our studies show that coded fingerprinting has efficient detection but rather low collusion resistance. Taking advantage of joint coding and embedding, we propose a permuted subsegment embedding technique and a group-based joint coding and embedding technique to improve the collusion resistance of coded fingerprinting while maintaining its efficient detection. Experimental results show that the number of colluders that the proposed methods can resist is more than three times as many as that of the conventional coded fingerprinting approaches.

Image hash functions find extensive applications in content authentication, database search, and watermarking. This paper develops a novel algorithm for generating an image hash based on Fourier transform features and controlled randomization. We formulate the robustness of image hashing as a hypothesis testing problem and evaluate the performance under various image processing operations. We show that the proposed hash function is resilient to content-preserving modifications, such as moderate geometric and filtering distortions. We introduce a general framework to study and evaluate the security of image hashing systems. Under this new framework, we model the hash values as random variables and quantify its uncertainty in terms of differential entropy. Using this security framework, we analyze the security of the proposed schemes and several existing representative methods for image hashing. We then examine the security versus robustness tradeoff and show that the proposed hashing methods can provide excellent security and robustness.

In recent years, there has been an increasing trend for multimedia applications to use delegate service providers for content distribution, archiving, search, and retrieval. These delegate services have brought new challenges to the protection of multimedia content confidentiality. This paper discusses the importance and feasibility of applying a joint signal processing and cryptographic approach to multimedia encryption, in order to address the access control issues unique to multimedia applications. We propose two atomic encryption operations that can preserve standard compliance and are friendly to delegate processing. Quantitative analysis for these operations is presented to demonstrate that a good tradeoff can be made between security and bitrate overhead. In assisting the design and evaluation of media security systems, we also propose a set of multimedia-oriented security scores to quantify the security against approximation attacks and to complement the existing notion of generic data security. Using video as an example, we present a systematic study on how to strategically integrate different atomic operations to build a video encryption system. The resulting system can provide superior performance over both generic encryption and its simple adaptation to video in terms of a joint consideration of security, bitrate overhead, and friendliness to delegate processing.

Transmitting multiple real-time encoded videos to multiple users over wireless cellular networks is a key driving force for developing broadband technology. We propose a new framework to transmit multiple users' video programs encoded by MPEG-4 FGS codec over downlink multicode CDMA networks in real time. The proposed framework jointly manages the rate adaptation of source and channel coding, CDMA code allocation, and power control. Subject to the limited system resources, such as the number of pseudo-random codes and the maximal power for CDMA transmission, we develop an adaptive scheme of distortion management to ensure baseline video quality for each user and further reduce the overall distortion received by all users. To efficiently utilize system resources, the proposed scheme maintains a balanced ratio between the power and code usages. We also investigate three special scenarios where demand, power, or code is limited, respectively. Compared with existing methods in the literature, the proposed algorithm can reduce the overall system's distortion by 14% to 26%. In the demand-limited case and the code-limited but power-unlimited case, the proposed scheme achieves the optimal solutions. In the power-limited but code-unlimited case, the proposed scheme has a performance very close to a performance upper bound

Digital fingerprinting protects multimedia content from illegal redistribution by uniquely marking copies of the content distributed to each user. Collusion is a powerful attack whereby several differently fingerprinted copies of the same content are combined together to attenuate or remove the fingerprints. Focusing on the error correction code (ECC) based fingerprinting, we explore in this paper new avenues that can substantially improve its collusion resistance, and in the mean time retain its advantages in detection complexity and fast distribution. Our analysis suggests a great need of jointly considering the coding, embedding, and detection issues, and inspires the proposed technique of permuted subsegment embedding that is able to substantially improve the collusion resistance of ECC based fingerprinting.

Hiding data in curves can be achieved by parameterizing a curve using the B-spline model and adding spread spectrum sequences in B-spline control points. In this paper, we propose an iterative alignment-minimization algorithm to perform curve registration and deal with the non-uniqueness of B-spline control points. We demonstrate through experiments the robustness of our method against various attacks such as collusion, geometric transformation, and printing-and-scanning. We also show the feasibility of our method for fingerprinting topographic maps and detecting fingerprints from printed copies.

A real-time video conferencing framework is proposed for multiple conferencing pairs by jointly considering the uplink and downlink conditions within IEEE 802.11 networks. We formulate this system so as to minimize the maximal end-to-end expected distortion received by all users by selecting the PHY modes and transmission time. Compared with the strategy of individually optimizing uplink and downlink, the proposed framework outperforms by 3.67-8.65 dB for the average received PSNR among all users.

Security and robustness are two important requirements for image hash functions. We introduce "differential entropy" as a metric to quantify the amount of randomness in image hash functions and to study their security. We present a mathematical framework and derive expressions for the proposed security metric for various common image hashing schemes. Using the proposed security metric, we discuss the trade-offs between security and robustness in image hashing.

This paper studies efficient bandwidth resource allocation for streaming multiple MPEG-4 fine granularity scalability (FGS) video programs to multiple users. We begin with a simple single-user scenario and propose a rate-control algorithm that has low delay and achieves an excellent tradeoff between the average visual distortion and the quality fluctuation. The proposed algorithm employs two weight factors for adjusting the tradeoff, and the optimal choice of these factors is derived. We then extend to the multiuser case and propose a dynamic resource allocation algorithm with low delay and low computational complexity. By exploring the variations in the scene complexity of video programs as well as dynamically and jointly distributing the available system resources among users, our proposed algorithm provides low fluctuation of quality for each user, and can support consistent or differentiated quality among all users to meet applications' needs. Experimental results show that compared to traditional look-ahead sliding-window approaches, our algorithm can achieve comparable visual quality and channel utilization at a much lower cost of delay, computation, and storage.

Digital fingerprinting is a method for protecting digital data in which fingerprints that are embedded in multimedia are capable of identifying unauthorized use of digital content. A powerful attack that can be employed to reduce this tracing capability is collusion, where several users combine their copies of the same content to attenuate/remove the original fingerprints. In this paper, we study the collusion resistance of a fingerprinting system employing Gaussian distributed fingerprints and orthogonal modulation. We introduce the maximum detector and the thresholding detector for colluder identification. We then analyze the collusion resistance of a system to the averaging collusion attack for the performance criteria represented by the probability of a false negative and the probability of a false positive. Lower and upper bounds for the maximum number of colluders K(max) are derived. We then show that the detectors are robust to different collusion attacks. We further study different sets of performance criteria, and our results indicate that attacks based on a few dozen independent copies can confound such a fingerprinting system. We also propose a likelihood-based approach to estimate the number of colluders. Finally, we demonstrate the performance for detecting colluders through experiments using real images.

Digital fingerprinting is a technology for tracing the distribution of multimedia content and protecting them from unauthorized redistribution. Unique identification information is embedded into each distributed copy of multimedia signal and serves as a digital fingerprint. Collusion attack is a cost-effective attack against digital fingerprinting, where colluders combine several copies with the same content but different fingerprints to remove or attenuate the original fingerprints. In this paper, we investigate the average collusion attack and several basic nonlinear collusions on independent Gaussian fingerprints, and study their effectiveness and the impact on the perceptual quality. With unbounded Gaussian fingerprints, perceivable distortion may exist in the fingerprinted copies as well as the copies after the collusion attacks. In order to remove this perceptual distortion, we introduce bounded Gaussian-like fingerprints and study their performance under collusion attacks. We also study several commonly used detection statistics and analyze their performance under collusion attacks. We further propose a preprocessing technique of the extracted fingerprints specifically for collusion scenarios to improve the detection performance.

In this paper, we propose a system to transmit multiple 3D embedded wavelet video programs over downlink multiuser OFDM. We consider the fairness among users and formulate the problem as minimizing the users' maximal distortion subject to power, rate, and subcarrier constraints. By exploring frequency, time, and multiuser diversity in OFDM and flexibility of the 3D embedded wavelet video codec, the proposed algorithm can achieve fair video qualities among all users. Compared to a scheme similar to the current multiuser OFDM standard (IEEE 802.11a), the proposed scheme outperforms it by 1-5 dB on the worst received PSNR among all users and has much smaller PSNR deviation.

In this paper, a protocol is designed to manage source rate/channel coding rate adaptation, code allocation, and power control to transmit real-time MPEG-4 FGS video over downlink multicode CDMA networks. We develop a fast adaptive scheme of distortion management to reduce the overall distortion received by all users subject, to the limited number of codes and maximal transmitted power. Compared with a modified greedy method in literature, our proposed algorithm can reduce the overall system's distortion by at least 45%.

An efficient bandwidth resource allocation algorithm with low delay and low fluctuation of quality to transmit multiple MPEG-4 fine granularity scalability (FGS) video programs to multiple users is proposed in this paper. By exploring the variation in the scene complexity of each video program and jointly redistributing available system resources among users, our proposed algorithm provides low fluctuation of quality for each user and consistent quality among all users. Experimental results show that compared to a traditional look-ahead sliding-window approach, our scheme can achieve comparable perceptual quality and channel utilization at a much lower cost of delay, computation, and storage.

This paper presents a new data hiding method for curves. The proposed algorithm parameterizes a curve using the B-spline model and adds a spread spectrum sequence in the coordinates of the B-spline control points. We demonstrate through experiments the robustness of the proposed data hiding algorithm against printing-and-scanning and collusions, and show its feasibility for collusion-resistant fingerprinting of topographic maps as well as writings/drawings from pen-based input devices.

This paper addresses the access control issues unique to multimedia, by using a joint signal processing and cryptographic approach to multimedia encryption. Based on three atomic encryption primitives, we present a systematic study on how to strategically integrate different atomic operations to build a video encryption system. We also propose a set of multimedia-specific security metrics to quantify the security against approximation attacks and to complement the existing notion of generic data security. The resulting system can provide superior performance to both generic encryption and its simple adaptation to video in terms of a joint consideration of security, bitrate overhead, and communication friendliness.

We propose a time-efficient contributory key agreement framework for secure communications in dynamic groups. The proposed scheme employs a special join-tree/exit-tree topology in the logical key tree and effectively exploits the efficiency of amortized operations. We derive the optimal parameters and design an activation algorithm for the join and exit trees. We also show that the asymptotic average time cost per user join and leave event is /spl theta/(log (log n)), where n is the group size. Our experiment results on both simulated user activities and the real MBone data have shown that the proposed scheme outperforms the existing tree-based schemes.

Image hash functions provide compact representations of images, which is useful for search and authentication applications. In this work, we have identified a general three step framework and proposed a new image hashing scheme that achieves a better overall performance than the existing approaches under various kinds of image processing distortions. By exploiting the properties of discrete polar Fourier transform and incorporating cryptographic keys, the proposed image hash is resilient to geometric and filtering operations, and is secure against guessing and forgery attacks.

This paper proposes a new method to embed data in binary images, including scanned text, figures, and signatures. The method manipulates "flippable" pixels to enforce specific block-based relationship in order to embed a significant amount of data without causing noticeable artifacts. Shuffling is applied before embedding to equalize the uneven embedding capacity from region to region. The hidden data can be extracted without using the original image, and can also be accurately extracted after high quality printing and scanning with the help of a few registration marks. The proposed data embedding method can be used to detect unauthorized use of a digitized signature, and annotate or authenticate binary documents. The paper also presents analysis and discussions on robustness and security issues.

Digital fingerprinting is a means to offer protection to digital data by which fingerprints embedded in the multimedia are capable of identifying unauthorized use of digital content. A powerful attack that can be employed to reduce this tracing capability is collusion. We study the collusion resistance of a fingerprinting system employing Gaussian distributed fingerprints and orthogonal modulation. We propose a likelihood-based approach to estimate the number of colluders, and introduce the thresholding detector for colluder identification. We first analyze the collusion resistance of a system to the average attack by considering the probability of a false negative and the probability of a false positive when identifying colluders. Lower and upper bounds for the maximum number of colluders are derived. We then show that the detectors are robust to different attacks. We further study different sets of performance criteria.

Digital fingerprinting is a technology for tracing the distribution of multimedia content and protecting them from unauthorized redistribution. Collusion attack is a cost effective attack against digital fingerprinting where several copies with the same content but different fingerprints are combined to remove the original fingerprints. In this paper, we investigate average and nonlinear collusion attacks of independent Gaussian fingerprints and study both their effectiveness and the perceptual quality. We also propose the bounded Gaussian fingerprints to improve the perceptual quality of the fingerprinted copies. We further discuss the tradeoff between the robustness against collusion attacks and the perceptual quality of a fingerprinting system.

This paper presents a new, classification-based spatial error concealment algorithm for images. The proposed scheme takes advantage of two state-of-the-art concealment schemes and adaptively selects a better suitable concealment scheme for each corrupted block. Using a support vector machine (SVM) classifier, our proposed approach outperforms the prior art in terms of the concealment quality and has moderate computational complexity.

This paper studies joint security and robustness enhancement of quantization based data embedding for multimedia authentication applications. We present analysis showing that through a lookup table (LUT) of nontrivial run that maps quantized multimedia features randomly to binary data, the detection error probability can be considerably smaller than the traditional quantization embedding. We quantify the security strength of LUT embedding and enhance its robustness through distortion compensation. Introducing a joint security and capacity measure, we show that the proposed distortion compensated LUT embedding provides joint enhancement of security and robustness over the traditional quantization embedding.

Digital fingerprinting of multimedia data involves embedding information in the content, and offers protection to the digital rights of the content by allowing illegitimate usage of the content to be identified by authorized parties. One potential threat to fingerprints is collusion, whereby a group of adversaries combine their individual copies in an attempt to remove the underlying fingerprints. Former studies indicate that collusion attacks based on a few dozen independent copies can confound a fingerprinting system that employs orthogonal modulation. However, since an adversary is more likely to collude with some users than other users, we propose a group-based fingerprinting scheme where users likely to collude with each other are assigned correlated fingerprints. We evaluate the performance of our group-based fingerprints by studying the collusion resistance of a fingerprinting system employing Gaussian distributed fingerprints. We compare the results to those of fingerprinting systems employing orthogonal modulation.

Digital fingerprinting is a means to offer protection to digital data by which fingerprints embedded in the multimedia are capable of identifying unauthorized use of digital content. A powerful attack that can be employed to reduce this tracing capability is collusion. In this paper, we study the collusion resistance of a fingerprinting system employing Gaussian distributed fingerprints and orthogonal modulation. We propose a likelihood-based approach to estimate the number of colluders, and introduce the thresholding detector for colluder identification. We first analyze the collusion resistance of a system to the average attack by considering the probability of a false negative and the probability of a false positive when identifying colluders. Lower and upper bounds for the maximum number of colluders K/sub max/ are derived. We then show that the detectors are robust to different attacks. We further study different sets of performance criteria.

Digital fingerprinting is a technology for tracing the distribution of multimedia content and protecting them from unauthorized redistribution. Collusion attack is a cost effective attack against digital fingerprinting where several copies with the same content but different fingerprints are combined to remove the original fingerprints. In this paper, we consider average attack and several nonlinear collusion attacks on independent Gaussian based fingerprints, and study the detection performance of several commonly used detection statistics in the literature under collusion attacks. Observing that these detection statistics are not specifically designed for collusion scenarios and do not take into account the characteristics of the newly generated fingerprints under collusion attacks, we propose pre-processing techniques to improve the detection performance of the detection statistics under collusion attacks.

Digital fingerprinting is a technology for tracing the distribution of multimedia content and protecting them from unauthorized redistribution. Collusion attack is a cost effective attack against digital fingerprinting where several copies with the same content but different fingerprints are combined to remove the original fingerprints. In this paper, we investigate average and nonlinear collusion attacks of independent Gaussian fingerprints and study both their effectiveness and the perceptual quality. We also propose the bounded Gaussian fingerprints to improve the perceptual quality of the fingerprinted copies. We further discuss the tradeoff between the robustness against collusion attacks and the perceptual quality of a fingerprinting system.

The paper studies joint security and robustness enhancement of quantization-based data embedding for multimedia authentication applications. We present an analysis showing that through a nontrivial run lookup table (LUT) that maps quantized multimedia features randomly to binary data, the probability of detection error can be considerably smaller than the traditional quantization embedding. We quantify the security strength of LUT embedding and enhance its robustness through distortion compensation. Introducing a joint security and capacity measure, we show that the proposed distortion-compensated LUT embedding provides joint enhancement of security and robustness over the traditional quantization embedding.

In this Part I of a two-part paper, we address a number of fundamental issues of data hiding in image and video and propose general solutions to them. We begin with a review of two major types of embedding, based on which we propose a new multilevel embedding framework to allow the amount of extractable data to be adaptive according to the actual noise condition. We then study the issues of hiding multiple bits through a comparison of various modulation and multiplexing techniques. Finally, the nonstationary nature of visual signals leads to highly uneven distribution of embedding capacity and causes difficulty in data hiding. We propose an adaptive solution switching between using constant embedding rate with shuffling and using variable embedding rate with embedded control bits. We verify the effectiveness of our proposed solutions through analysis and simulation. And Part II will apply these solutions to specific design problems for embedding data in grayscale and color images and video.

This paper applies the solutions to the fundamental issues addressed in Part I to specific design problems of embedding data in image and video. We apply multilevel embedding to allow the amount of embedded information that can be reliably extracted to be adaptive with respect to the actual noise conditions. When extending the multilevel embedding to video, we propose strategies for handling uneven embedding capacity from region to region within a frame as well as from frame to frame. We also embed control information to facilitate the accurate extraction of the user data payload and to combat such distortions as frame jitter. The proposed algorithm can be used for a variety of applications such as copy control, access control, robust annotation, and content-based authentication.

Digital fingerprinting is an effective method to identify users who might try to redistribute multimedia content, such as images and video. These fingerprints are typically embedded into the content using watermarking techniques that are designed to be robust to a variety of attacks. A cheap and effective attack against such digital fingerprints is collusion, where several differently marked copies of the same content are averaged or combined to disrupt the underlying fingerprint. In this paper, we study the problem of designing fingerprints that can withstand collusion, yet trace colluders. Since, in antipodal CDMA-type watermarking, the correlation contributions only decrease where watermarks differ, by constructing binary code vectors where any subset of k or fewer of these vectors have unique overlap, we may identify groups of k or less colluders. Our construction of such anti-collusion codes (ACC) uses the theory of combinatorial designs, and for n users requires O(√n) bits. Further, we explore a block matrix structure for the ACC that reduces the computational complexity for identifying colluders and improves the detection capability when colluders belong to the same subgroup.

Digital fingerprinting is an effective method to identify users who might try to redistribute multimedia content, such as images and video. These fingerprints are typically embedded into the content using watermarking techniques that are designed to be robust to a variety of attacks. A cheap and effective attack against such digital fingerprints is collusion, where several differently marked copies of the same content are averaged or combined to disrupt the underlying fingerprint. We present a construction of collusion-resistant fingerprints based upon anti-collusion codes (ACC) and binary code modulation. ACC have the property that the composition of any subset of K or fewer codevectors is unique. Using this property, we build fingerprints that allow for the identification of groups of K or less colluders. We present a construction of binary-valued ACC under the logical AND operation using the theory of combinatorial designs. Our code construction requires only /spl Oscr/(/spl radic/n) orthogonal signals to accommodate n users. We demonstrate the performance of our ACC for fingerprinting multimedia by identifying colluders through experiments using real images.

The paper proposes an algorithm for recognizing Chinese characters in many diverse fonts including Song, Fang, Kai, Hei, Yuan, Lishu, Weibei and Xingkai. The algorithm is based on features derived from peripheral direction contributions and utilizes a set of dictionaries. A 3-level matching is first performed with respect to each dictionary. The distance measures associated with these matches are then fed into a central discriminator to output the final recognition result. We propose a new multi-dictionary matching algorithm for use in the central discriminator that utilizes estimated information of neighborhood fonts. Experiments have been performed on a practical OCR software system whose recognition kernel is based on the proposed algorithm. Fast and accurate recognition has been accomplished both in title recognition, involving all of the 8 fonts, and in main-body recognition, that usually involves only the first 4 most commonly used fonts.

This paper explains and analyzes the successful attacks submitted by the authors on four audio watermark proposals during a 3-week SDMI public challenge. Our analysis points out some weaknesses in the watermark techniques currently under SDMI consideration and suggests directions for further improvement. The paper also discusses the framework and strategies for analyzing the robustness and security of watermarking systems as well as the difficulty, uniqueness, and unrealistic expectations of the attack setup.

Many electronic watermarks for still images and video content are sensitive to geometric distortions. For example, simple rotation, scaling, and/or translation (RST) of an image can prevent blind detection of a public watermark. In this paper, we propose a watermarking algorithm that is robust to RST distortions. The watermark is embedded into a one-dimensional (1-D) signal obtained by taking the Fourier transform of the image, resampling the Fourier magnitudes into log-polar coordinates, and then summing a function of those magnitudes along the log-radius axis. Rotation of the image results in a cyclical shift of the extracted signal. Scaling of the image results in amplification of the extracted signal, and translation of the image has no effect on the extracted signal. We can therefore compensate for rotation with a simple search, and compensate for scaling by using the correlation coefficient as the detection measure. False positive results on a database of 10,000 images are reported. Robustness results on a database of 2000 images are described. It is shown that the watermark is robust to rotation, scale, and translation. In addition, we describe tests examining the watermarks resistance to cropping and JPEG compression.

The reliable and efficient transmission of high-quality variable bit rate (VBR) video through the Internet generally requires network resources be allocated in a dynamic fashion. This includes the determination of when to renegotiate for network resources, as well as how much to request at a given time. The accuracy of any resource request method depends critically on its prediction of future traffic patterns. Such a prediction can be performed using the content and traffic information of short video segments. This paper presents a systematic approach to select the best features for prediction, indicating that while content is important in predicting the bandwidth of a video hit stream, the use of both content and available short-term bandwidth statistics can yield significant improvements. A new framework for traffic prediction is proposed in this paper; experimental results show a smaller mean-square resource prediction error and higher overall link utilization.

Dynamic resource allocation is critical in the transmission of VBR video. Our study shows that content is one of the major factors that controls the bandwidth of the video bit-stream, yet content alone may not be sufficient in predicting future traffic and in determining how much resource to request. A new framework of traffic prediction is proposed, taking into account both content features and available short-term bandwidth statistics.

In network delivery of digital video, if the bandwidth required for a video is not available, the video has to be recoded at a reduced bit rate. It is highly desirable that the transcoding is carried out in real time while maintaining reasonable image quality. In this paper, we propose a fast approach to derive from an MPEG stream a new MPEG stream with half the spatial resolution. For the downsized video, we first generate from the original compressed video an improved estimate of the motion vectors. We then propose a compressed domain approach with data hiding to produce DCT residues by an open-loop method. The computational complexity is significantly lower than a number of previous approaches. Simulation suggests that our approach produces reasonable image quality.

Summary form only given. Watermarking algorithms that are robust to the common geometric transformations of rotation, scale and translation (RST) have been reported for cases in which the original unwatermarked content is available at the detector so as to allow the transformations to be inverted. However, for public watermarks the problem is significantly more difficult since there is no original content to register with. Two classes of solution have been proposed. The first embeds a registration pattern into the content while the second seeks to apply detection methods that are invariant to these geometric transformations. This paper describes a public watermarking method which is invariant (or bares a simple relation) to the common geometric transforms of rotation, scale, and translation. It is based on the Fourier-Mellin transform which has previously been suggested. We extend this work, using a variation based on the Radon transform. The watermark is inserted into a projection of the image. The properties of this projection are such that RST transforms produce simple or no effects on the projection waveform. When a watermark is inserted into a projection, the signal must eventually be back projected to the original image dimensions. This is a one to many mapping that allows for considerable flexibility in the watermark insertion process. We highlight some theoretical and practical issues that affect the implementation of an RST invariant watermark. Finally, we describe preliminary experimental results.

This paper applies shuffling to digital watermarking and data hiding. The data embedding capacity in the multimedia source generally varies significantly from one part of the source to another. Sequential embedding is very sensitive to noise which may cause synchronization problem; the common but conservative solution via partitioning an image into large segments and embedding only one bit per segment is wasteful of the data embedding capacity. This paper shows how random shuffling can be used to equalize the uneven distribution of embedding capacity. The effectiveness of random shuffling is demonstrated by analysis and experiments.

A data embedding method is proposed for image authentication based on table look-up in frequency domain. A visually meaningful watermark and a set of simple features are embedded invisibly in the marked image, which can be stored in the compressed form. The scheme can detect and localize alterations of the original image, such as the tampering of images exported from a digital camera.

The detection of transitions between shots in video programs is an important first step in analyzing video content. The wipe is a frequently used transitional form between shots. Wipe detection is more involved than the detection of abrupt and other gradual transitions because a wipe may take various patterns and because of the difficulty in discriminating a wipe from object and camera motion. In this paper, we propose an algorithm for detecting wipes using both structural and statistical information. The algorithm can effectively detect most wipes used in current TV programs. It uses the DC sequence which can be easily extracted from the MPEG stream without full decompression.