Lipstick virtual try-on (VTO) experiences have become widespread across the e-commerce sector and assist users in eliminating the guesswork of shopping online. However, such experiences still lack in both realism and accuracy. In this work, we propose LipAT, a neural framework that blends the strengths of Physics-Based Rendering (PBR) and Neural Style Transfer (NST) approaches to directly apply lipstick onto face images given lipstick attributes (e.g., colour, finish type). LipAT consists of a physics aware neural lipstick application module (LAM) to apply lipstick on face images given its attributes and Lipstick Refiner Module (LRM) to improve the realism by refining the imperfections. Unlike the NST approaches, LipAT allows precise and controllable lipstick attribute preservation, without requiring crude approximations and inference of various intertwined environment factors (e.g., scene lighting, face structure etc) involved in image generation that is required for accurate PBR. We propose an experimental framework with quantitative metrics to evaluate different desirable aspects of the lipstick attribute driven try-on alongside user studies to further validate our findings. Our results show that LipAT considerably outperforms fully-automated PBR approaches in preserving realism and the NST approaches in preserving various lipstick attributes such as finish types.

Semi-supervised semantic segmentation methods use a small amount of clean pixel-level annotations to guide the interpretation of a larger quantity of unlabelled image data. The challenges of providing pixel-accurate annotations at scale mean that the labels are typically noisy, and this contaminates the final results. In this work, we propose an approach that is robust to label noise in the annotated data. The method uses two diverse learning groups with different network architectures to effectively handle both label noise and unlabelled images. Each learning group consists of a teacher network, a student network and a novel filter module. The filter module of each learning group utilizes pixel-level features from the teacher network to detect incorrectly labelled pixels. To reduce confirmation bias, we employ the labels cleaned by the filter module from one learning group to train the other learning group. Experimental results on two different benchmarks and settings demonstrate the superiority of our method over state-of-the-art approaches.

We introduce Retrieval Augmented Classification (RAC), a generic approach to augmenting standard image classification pipelines with an explicit retrieval module. RAC consists of a standard base image encoder fused with a parallel retrieval branch that queries a non-parametric external memory of pre-encoded images and associated text snippets. We apply RAC to the problem of long-tail classification and demonstrate a significant improvement over previous state-of-the-art on Places365-LT and iNaturalist-2018 (14.5% and 6.7% respectively), despite using only the training datasets themselves as the external information source. We demonstrate that RAC's retrieval module, without prompting, learns a high level of accuracy on tail classes. This, in turn, frees the base encoder to focus on common classes, and improve its performance thereon. RAC represents an alternative approach to utilizing large, pretrained models without requiring fine-tuning, as well as a first step towards more effectively making use of external memory within common computer vision architectures.

The advent of generative adversarial networks (GAN) has enabled new capabilities in synthesis, interpolation, and data augmentation heretofore considered very challenging. However, one of the common assumptions in most GAN architectures is the assumption of simple parametric latent-space distributions. While easy to implement, a simple latent-space distribution can be problematic for uses such as interpolation. This is due to distributional mismatches when samples are interpolated in the latent space. We present a straightforward formalization of this problem; using basic results from probability theory and off-the-shelf-optimization tools, we develop ways to arrive at appropriate non-parametric priors. The obtained prior exhibits unusual qualitative properties in terms of its shape, and quantitative benefits in terms of lower divergence with its mid-point distribution. We demonstrate that our designed prior helps improve image generation along any Euclidean straight line during interpolation, both qualitatively and quantitatively, without any additional training or architectural modifications. The proposed formulation is quite flexible, paving the way to impose newer constraints on the latent-space statistics.

In this work we present a self-supervised learning framework to simultaneously train two Convolutional Neural Networks (CNNs) to predict depth and surface normals from a single image. In contrast to most existing frameworks which represent outdoor scenes as fronto-parallel planes at piece-wise smooth depth, we propose to predict depth with surface orientation while assuming that natural scenes have piece-wise smooth normals. We show that a simple depth-normal consistency as a soft-constraint on the predictions is sufficient and effective for training both these networks simultaneously. The trained normal network provides state-of-the-art predictions while the depth network, relying on much realistic smooth normal assumption, outperforms the traditional self-supervised depth prediction network by a large margin on the KITTI benchmark.

Despite learning based methods showing promising results in single view depth estimation and visual odometry, most existing approaches treat the tasks in a supervised manner. Recent approaches to single view depth estimation explore the possibility of learning without full supervision via minimizing photometric error. In this paper, we explore the use of stereo sequences for learning depth and visual odometry. The use of stereo sequences enables the use of both spatial (between left-right pairs) and temporal (forward backward) photometric warp error, and constrains the scene depth and camera motion to be in a common, real-world scale. At test time our framework is able to estimate single view depth and two-view odometry from a monocular sequence. We also show how we can improve on a standard photometric warp loss by considering a warp of deep features. We show through extensive experiments that: (i) jointly training for single view depth and visual odometry improves depth prediction because of the additional constraint imposed on depths and achieves competitive results for visual odometry; (ii) deep feature-based warping loss improves upon simple photometric warp loss for both single view depth estimation and visual odometry. Our method outperforms existing learning based methods on the KITTI driving dataset in both tasks. The source code is available at https://github.com/Huangying-Zhan/Depth-VO-Feat.

Place recognition is an essential component of Simultaneous Localization And Mapping (SLAM). Under severe appearance change, reliable place recognition is a difficult perception task since the same place is perceptually very different in the morning, at night, or over different seasons. This work addresses place recognition as a domain translation task. Using a pair of coupled Generative Adversarial Networks (GANs), we show that it is possible to generate the appearance of one domain (such as summer) from another (such as winter) without requiring image-to-image correspondences across the domains. Mapping between domains is learned from sets of images in each domain without knowing the instance-to-instance correspondence by enforcing a cyclic consistency constraint. In the process, meaningful feature spaces are learned for each domain, the distances in which can be used for the task of place recognition. Experiments show that learned features correspond to visual similarity and can be effectively used for place recognition across seasons.

We present “just-in-time reconstruction” as realtime image-guided inpainting of a map with arbitrary scale and sparsity to generate a fully dense depth map for the image. In particular, our goal is to inpaint a sparse map - obtained from either a monocular visual SLAM system or a sparse sensor - using a single-view depth prediction network as a virtual depth sensor. We adopt a fairly standard approach to data fusion, to produce a fused depth map by performing inference over a novel fully-connected Conditional Random Field (CRF) which is parameterized by the input depth maps and their pixel-wise confidence weights. Crucially, we obtain the confidence weights that parameterize the CRF model in a data-dependent manner via Convolutional Neural Networks (CNNs) which are trained to model the conditional depth error distributions given each source of input depth map and the associated RGB image. Our CRF model penalises absolute depth error in its nodes and pairwise scale-invariant depth error in its edges, and the confidence-based fusion minimizes the impact of outlier input depth values on the fused result. We demonstrate the flexibility of our method by real-time inpainting of ORB-SLAM, Kinect, and LIDAR depth maps acquired both indoors and outdoors at arbitrary scale and varied amount of irregular sparsity.

There exist a number of problem classes for which obtaining the exact solution becomes exponentially expensive with increasing problem size. The quadratic assignment problem (QAP) or the travelling salesman problem (TSP) are just two examples of such NP-hard problems. In practice, approximate algorithms are employed to obtain a suboptimal solution, where one must face a trade-off between computational complexity and solution quality. In this paper, we propose to learn to solve these problem from approximate examples, using recurrent neural networks (RNNs). Surprisingly, such architectures are capable of producing highly accurate solutions at minimal computational cost. Moreover, we introduce a simple, yet effective technique for improving the initial (weak) training set by incorporating the objective cost into the training procedure. We demonstrate the functionality of our approach on three exemplar applications: marginal distributions of a joint matching space, feature point matching and the travelling salesman problem. We show encouraging results on synthetic and real data in all three cases.

This paper presents an efficient framework for dense 3D scene reconstruction using input from a moving monocular camera. Visual SLAM (Simultaneous Localisation and Mapping) approaches based solely on geometric methods have proven to be quite capable of accurately tracking the pose of a moving camera and simultaneously building a map of the environment in real-time. However, most of them suffer from the 3D map being too sparse for practical use. The missing points in the generated map correspond mainly to areas lacking texture in the input images, and dense mapping systems often rely on hand-crafted priors like piecewise-planarity or piecewise-smooth depth. These priors do not always provide the required level of scene understanding to accurately fill the map. On the other hand, Convolutional Neural Networks (CNNs) have had great success in extracting high-level information from images and regressing pixel-wise surface normals, semantics, and even depth. In this work we leverage this high-level scene context learned by a deep CNN in the form of a surface normal prior. We show, in particular, that using the surface normal prior leads to better reconstructions than the weaker smoothness prior.

The electric network frequency (ENF) is a signature of power distribution networks that can be captured by multimedia signals recorded near electrical activities. This has led to the emergence of multiple forensic applications based on the use of ENF signals. Examples of such applications include validating the time-of-recording of an ENF-containing multimedia signal or estimating its recording location based on concurrent reference signals from power grids. In this paper, we examine a novel ENF-based application that infers the power grid in which the ENF-containing multimedia signal was recorded without relying on the availability of concurrent power references. We investigate features based on the statistical differences in ENF variations between different power grids to serve as signatures for the region-of-recording of the media signal. We use these features in a multiclass machine learning implementation that is able to identify the grid-of-recording of a signal with high accuracy. In addition, we explore techniques for building multiconditional learning systems that can adapt to changes in the noise environment between the training and testing data.

The electric network frequency (ENF) signal can be embedded in multimedia recordings created in areas of electrical activities. Recent work has used the ENF signal for such applications as time stamp authentication and forgery detection. It is more challenging to extract ENF signals from video recordings than from audio recordings because of the low temporal sampling rate or frame rate of video cameras. The rolling shutter of CMOS image sensor can be exploited as it exposes a frame line by line, and the effective ENF sampling rate by treating each line as a signal sample can be increased. This scheme was shown to work well with static videos. This paper conducts a further study on the exploitation of the rolling shutter for extracting ENF traces from videos. The rolling shutter mechanism is modeled and analyzed using multirate signal processing theory. Challenging cases of videos with motions are examined, and solutions to extracting ENF from them are explored.

This paper offers the first variational approach to the problem of dense 3D reconstruction of non-rigid surfaces from a monocular video sequence. We formulate non-rigid structure from motion (nrsfm) as a global variational energy minimization problem to estimate dense low-rank smooth 3D shapes for every frame along with the camera motion matrices, given dense 2D correspondences. Unlike traditional factorization based approaches to nrsfm, which model the low-rank non-rigid shape using a fixed number of basis shapes and corresponding coefficients, we minimize the rank of the matrix of time-varying shapes directly via trace norm minimization. In conjunction with this low-rank constraint, we use an edge preserving total-variation regularization term to obtain spatially smooth shapes for every frame. Thanks to proximal splitting techniques the optimization problem can be decomposed into many point-wise sub-problems and simple linear systems which can be easily solved on GPU hardware. We show results on real sequences of different objects (face, torso, beating heart) where, despite challenges in tracking, illumination changes and occlusions, our method reconstructs highly deforming smooth surfaces densely and accurately directly from video, without the need for any prior models or shape templates.

Electric Network Frequency (ENF) fluctuations based forensic analysis is an emerging way for such multimedia authentication tasks as time-of-recording estimation, timestamp verification, and clip insertion/deletion forgery detection. ENF fluctuates due to dynamic changes in load demand and power supply, and these fluctuations travel over the power lines with a finite speed. In this paper, experiments are conducted on ENF data collected across different locations in the eastern grid of the United States to understand the relationship between the signals recorded at the same time at these locations. Based on these experiments, a signal processing mechanism is developed to demonstrate that ENF fluctuations across different locations exhibit a measure of similarity with each other, which is proportional to the distance between the locations. Such observations motivated a location estimation protocol based on the similarity of ENF signals with respect to anchor nodes. Under certain conditions, the proposed protocol is shown to provide an estimation accuracy of 90%. Challenges in the application of ENF signal analysis for location of recording estimation of multimedia signal are also discussed.

Secure fingerprint authentication via encrypted-domain processing imposes constraints on the underlying feature extraction method: Firstly, it requires fixed-length feature vectors to be amenable to computing distances or correlations. Secondly, extra information must be stored in the clear so that the fingerprints can be aligned prior to feature extraction and secure comparison. These constraints potentially restrict the flexibility, increase computational complexity, and even reduce the security of the scheme. We desire feature vectors suitable for encrypted-domain matching while being free of the above constraints. To this end, a local neighborhood is defined around certain detected minutiae points, and features are extracted based on relative locations of close minutia points, local ridge texture and local ridge orientation. The locality of the features provides robustness to rotation and translation. Feature vectors are compared using operations that can be performed using secure primitives. The process of computing the matching scores - genuine or impostor - implicitly yields the best alignment without needing to store unencrypted side information at the access control device. The scheme achieves an Equal Error Rate of 1.46% on a proprietary database and 7.86% on the FVC2002 public database.

Electric Network Frequency (ENF) based forensic analysis is a promising tool for timestamp authentication and forgery detection in such multimedia recordings as audios and videos. ENF signal is embedded in an audio recording due to electromagnetic interference from the power lines. The time of creation of a multimedia recording can be determined by comparing the ENF signal embedded in the recording with a reference ENF database collected from the power grid. In this paper, we conduct a study of the effect of recapturing of audio recordings on the ENF embedding. We demonstrate that recaptured audio recordings pick up two ENF signals: the content ENF signal which is inherited from the original audio recording; and the recapturing ENF signal which is embedded from the recapturing process. Conventional ENF signal extraction techniques on such recordings may fail when the two ENF signals are at the same nominal value. A decorrelation algorithm is proposed to extract the content ENF signal and the recapturing ENF signal. The experimental results show the effectiveness of the proposed method in the estimation of both the ENF signals.

This paper addresses the problem of non-rigid video registration, or the computation of optical flow from a reference frame to each of the subsequent images in a sequence, when the camera views deformable objects. We exploit the high correlation between 2D trajectories of different points on the same non-rigid surface by assuming that the displacement of any point throughout the sequence can be expressed in a compact way as a linear combination of a low-rank motion basis. This subspace constraint effectively acts as a trajectory regularization term leading to temporally consistent optical flow. We formulate it as a robust soft constraint within a variational framework by penalizing flow fields that lie outside the low-rank manifold. The resulting energy functional can be decoupled into the optimization of the brightness constancy and spatial regularization terms, leading to an efficient optimization scheme. Additionally, we propose a novel optimization scheme for the case of vector valued images, based on the dualization of the data term. This allows us to extend our approach to deal with colour images which results in significant improvements on the registration results. Finally, we provide a new benchmark dataset, based on motion capture data of a flag waving in the wind, with dense ground truth optical flow for evaluation of multi-frame optical flow algorithms for non-rigid surfaces. Our experiments show that our proposed approach outperforms state of the art optical flow and dense non-rigid registration algorithms.

The Electric Network Frequency (ENF) is the supply frequency of power distribution networks, and is often captured by audio or video measurements recorded near power supplies. The time varying nature of the ENF allows it to be used for such forensic applications as estimating the time and location of media recordings, and discerning their integrity. An initial step in such applications is to extract the ENF signal-instantaneous ENF values over time-as accurately as possible. Existing techniques rely on estimating the ENF around the nominal frequency of 50/60 Hz, or around one of its harmonics at a time. In this letter, a novel spectrum combining approach is proposed, which exploits the presence of the ENF around different harmonics of the nominal frequency. The ENF signal is estimated by combining the ENF at multiple harmonics, based on the local signal-to-noise ratio at each harmonic. A hypothesis testing performance of an ENF-based timestamp verification application is examined to validate that the proposed approach achieves a more robust and accurate performance than conventional ENF estimation techniques.

The electrical network frequency (ENF) signal is a time stamp that has been used by an emerging class of approaches for determining the creation time of digital audio and video recordings. However, in adversarial environments, anti-forensic operations may be conducted to manipulate ENF-based time stamps, and it is crucial to understand the resilience of ENF analysis against anti-forensics. This paper explores possible anti-forensic operations that can remove and alter the ENF signal while trying to preserve the host signal, and devises detection methods targeting these operations. Concealment techniques that can circumvent detection are also discussed and their corresponding trade-offs are examined. Based on the understanding of individual anti-forensic operations and countermeasures, this paper further characterizes the dynamic interplay between forensic analysts and adversaries by providing an evolutionary perspective and a game-theoretical perspective as well as studying representative scenarios and the optimal forensic/anti-forensic strategies.

Electric Network Frequency (ENF) fluctuates slightly over time from its nominal value of 50 Hz/60 Hz. The fluctuations in ENF remain consistent across the entire power grid including when measured at physically distant geographical locations. The light intensity from such indoor lighting as fluorescent lamps and incandescent bulbs, which are connected to the power mains, varies in accordance with the ENF, and the fluctuations can be recorded using visual sensors. In this paper, mechanisms using optical sensors and video cameras to record and validate the presence of ENF fluctuations in indoor lighting are presented. Spectrogram and subspace-based signal processing techniques are applied to such recordings to extract the ENF signal by estimating its instantaneous frequencies as a function of time. A high correlation is observed between the ENF fluctuations obtained from indoor lighting and that of the ENF signal captured directly from the power mains supply. A similar mechanism is then used to demonstrate the presence of ENF signals in video recordings taken in different geographical areas. Experimental results show that ENF signals are present in visual recordings made in different geographical areas and can be used as a natural timestamp for optical sensor recordings and video surveillance recordings conducted in indoor lighting environments. Robustness of ENF fluctuation traces under strong compression and CMOS rolling shutter cameras is discussed. Applications of the ENF signal analysis to tampering detection of surveillance video recordings and forensic binding of the audio and visual track of a video are also demonstrated. An analytical model based on an autoregressive process is also developed for ENF signals, and the effectiveness of using innovation sequences from the model for timestamp verification is demonstrated.

A time stamp based on the power network signature called the Electrical Network Frequency (ENF) has been used by an emerging class of approaches for authenticating digital audio and video recordings in computer-to-computer communications. However, the presence of adversaries may render the time stamp insecure, and it is crucial to understand the robustness of ENF analysis against anti-forensic operations. This paper investigates possible anti-forensic operations that can remove and alter the ENF signal while trying to preserve the host signal, and develops detection methods targeting these operations. Improvements over anti-forensic operations that can circumvent the detection are also examined, for which various trade-offs are discussed. To develop an understanding of the dynamics between a forensic analyst and an adversary, an evolutionary perspective and a game-theoretical perspective are proposed, which allow for a comprehensive characterization of plausible anti-forensic strategies and countermeasures. Such an understanding has the potential to lead to more secure and reliable time stamp schemes based on ENF analysis.

Localization of constituent nodes is of fundamental importance in many wireless sensor networks (WSNs) related applications. Existing research has mainly investigated the problem of localization in static WSNs, where the localization is performed mainly at the time of the node deployment. In contrast, it is important to keep track of the current locations of the nodes by invoking the localization algorithm periodically in mobile nodes. The high computation cost associated with most existing localization algorithms makes them less practical to use in resource constrained mobile sensor networks (MSNs). Additionally, these existing techniques often fail in hostile environments where some of the nodes may be compromised by adversaries, and used to transmit misleading information aimed at preventing accurate localization of the remaining sensors. In this paper, we build on our earlier work to propose an iterative gradient descent based technique with low computational complexity to securely localize nodes in MSNs. The proposed algorithm combines iterative gradient descent with selective pruning of inconsistent measurements to achieve a high localization accuracy. Simulation results demonstrate that the proposed algorithm can find a map of relative locations of the MSN even when some nodes are compromised and transmit false information.

Existing approaches to camera tracking and reconstruction from a single handheld camera for Augmented Reality (AR) focus on the reconstruction of static scenes. However, most real world scenarios are dynamic and contain multiple independently moving rigid objects. This paper addresses the problem of simultaneous segmentation, motion estimation and dense 3D reconstruction of dynamic scenes. We propose a dense solution to all three elements of this problem: depth estimation, motion label assignment and rigid transformation estimation directly from the raw video by optimizing a single cost function using a hill-climbing approach. We do not require prior knowledge of the number of objects present in the scene - the number of independent motion models and their parameters are automatically estimated. The resulting inference method combines the best techniques in discrete and continuous optimization: a state of the art variational approach is used to estimate the dense depth maps while the motion segmentation is achieved using discrete graph-cut based optimization. For the rigid motion estimation of the independently moving objects we propose a novel tracking approach designed to cope with the small fields of view they induce and agile motion. Our experimental results on real sequences show how accurate segmentations and dense depth maps can be obtained in a completely automated way and used in marker-free AR applications.

Many applications of wireless sensor networks require precise knowledge of the locations of constituent nodes. In these applications, it is desirable for the nodes to be able to autonomously determine their locations before they start sensing and transmitting data. Most localization algorithms use anchor nodes with known locations to determine the positions of the remaining nodes. However, these existing techniques often fail in hostile environments where some of the nodes may be compromised by adversaries and used to transmit misleading information aimed at preventing accurate localization of the remaining sensors. In this paper, a computationally efficient secure localization algorithm that withstands such attacks is described. The proposed algorithm combines iterative gradient descent with selective pruning of inconsistent measurements to achieve high localization accuracy. Results show that the proposed algorithm utilizes fewer computational resources and achieves an accuracy better than or comparable to that of existing schemes. The proposed secure localization algorithm can also be used in mobile sensor networks, where all nodes are moving, to estimate the relative locations of the nodes without relying on anchor nodes. Simulations demonstrate that the proposed algorithm can find the relative location map of the entire mobile sensor network even when some nodes are compromised and transmit false information.

Electric Network Frequency (ENF) fluctuates slightly over time from its nominal value of 50 Hz/60 Hz. The fluctuations in the ENF remain consistent across the entire power grid even when measured at physically distant locations. The near-invisible flickering of fluorescent lights connected to the power mains reflect these fluctuations present in the ENF. In this paper, mechanisms using optical sensors and video cameras to record and validate the presence of the ENF fluctuations in fluorescent lighting are presented. Signal processing techniques are applied to demonstrate a high correlation between the fluctuations in the ENF signal captured from fluorescent lighting and the ENF signal captured directly from power mains supply. The proposed technique is then used to demonstrate the presence of the ENF signal in video recordings taken in various geographical areas. Experimental results show that the ENF signal can be used as a natural timestamp for optical sensor recordings and video surveillance recordings from indoor environments under fluorescent lighting. Application of the ENF signal analysis to tampering detection of surveillance video recordings is also demonstrated.

Many sensor network related applications require precise knowledge of the location of constituent nodes. In these applications, it is desirable for the wireless nodes to be able to autonomously determine their locations before they start sensing and transmitting data. Most localization algorithms rely on anchor nodes whose locations are known to determine the positions of the remaining nodes. In an adversarial scenario, some of these anchor nodes could be compromised and used to transmit misleading information aimed at preventing the accurate localization of the remaining sensors. In this paper, a computationally efficient algorithm to determine the location of sensors that can resist such attacks is described. The proposed algorithm combines gradient descent with a selective pruning of inconsistent measurements to achieve good localization accuracy. Simulation results show that the proposed algorithm has performance comparable to existing schemes while requiring less computational resources.