Abstract We present a kernel‐predicting neural denoising method for path‐traced deep‐Z images that facilitates their usage in animation and visual effects production. Deep‐Z images provide enhanced flexibility during compositing as they contain color, opacity, and other rendered data at multiple depth‐resolved bins within each pixel. However, they are subject to noise, and rendering until convergence is prohibitively expensive. The current state of the art in deep‐Z denoising yields objectionable artifacts, and current neural denoising methods are incapable of handling the variable number of depth bins in deep‐Z images. Our method extends kernel‐predicting convolutional neural networks to address the challenges stemming from denoising deep‐Z images. We propose a hybrid reconstruction architecture that combines the depth‐resolved reconstruction at each bin with the flattened reconstruction at the pixel level. Moreover, we propose depth‐aware neighbor indexing of the depth‐resolved inputs to the convolution and denoising kernel application operators, which reduces artifacts caused by depth misalignment present in deep‐Z images. We evaluate our method on a production‐quality deep‐Z dataset, demonstrating significant improvements in denoising quality and performance compared to the current state‐of‐the‐art deep‐Z denoiser. By addressing the significant challenge of the cost associated with rendering path‐traced deep‐Z images, we believe that our approach will pave the way for broader adoption of deep‐Z workflows in future productions.

Although Monte Carlo path tracing is a simple and effective algorithm to synthesize photo-realistic images, it is often very slow to converge to noise-free results when involving complex global illumination. One of the most successful variance-reduction techniques is path guiding, which can learn better distributions for importance sampling to reduce pixel noise. However, previous methods require a large number of path samples to achieve reliable path guiding. We present a novel neural path guiding approach that can reconstruct high-quality sampling distributions for path guiding from a sparse set of samples, using an offline trained neural network. We leverage photons traced from light sources as the primary input for sampling density reconstruction, which is effective for challenging scenes with strong global illumination. To fully make use of our deep neural network, we partition the scene space into an adaptive hierarchical grid, in which we apply our network to reconstruct high-quality sampling distributions for any local region in the scene. This allows for effective path guiding for arbitrary path bounce at any location in path tracing. We demonstrate that our photon-driven neural path guiding approach can generalize to diverse testing scenes, often achieving better rendering results than previous path guiding approaches and opening up interesting future directions.

Vehicle detection with visual sensors like lidar and camera is one of the critical functions enabling autonomous driving. While they generate fine-grained point clouds or high-resolution images with rich information in good weather conditions, they fail in adverse weather (e.g., fog) where opaque particles distort lights and significantly reduce visibility. Thus, existing methods relying on lidar or camera experience significant performance degradation in rare but critical adverse weather conditions. To remedy this, we resort to exploiting complementary radar, which is less impacted by adverse weather and becomes prevalent on vehicles. In this paper, we present Multimodal Vehicle Detection Network (MVDNet), a two-stage deep fusion detector, which first generates proposals from two sensors and then fuses region-wise features between multimodal sensor streams to improve final detection results. To evaluate MVDNet, we create a procedurally generated training dataset based on the collected raw lidar and radar signals from the open-source Oxford Radar Robotcar. We show that the proposed MVDNet surpasses other state-of-the-art methods, notably in terms of Average Precision (AP), especially in adverse weather conditions. The code and data are available at https://github.com/qiank10/MVDNet.

Path guiding is a promising technique to reduce the variance of path tracing. Although existing online path guiding algorithms can eventually learn good sampling distributions given a large amount of time and samples, the speed of learning becomes a major bottleneck. In this paper, we accelerate the learning of sampling distributions by training a light-weight neural network offline to reconstruct from sparse samples. Uniquely, we design our neural network to directly operate convolutions on a sparse quadtree, which regresses a high-quality hierarchical sampling distribution. Our approach can reconstruct reasonably accurate sampling distributions faster, allowing for efficient path guiding and rendering. In contrast to the recent offline neural path guiding techniques that reconstruct low-resolution 2D images for sampling, our novel hierarchical framework enables more fine-grained directional sampling with less memory usage, effectively advancing the practicality and efficiency of neural path guiding. In addition, we take advantage of hybrid bidirectional samples including both path samples and photons, as we have found this more robust to different light transport scenarios compared to using only one type of sample as in previous work. Experiments on diverse testing scenes demonstrate that our approach often improves rendering results with better visual quality and lower errors. Our framework can also provide the proper balance of speed, memory cost, and robustness.

Gait recognition, i.e., recognizing persons from their walking postures, has found versatile applications in security check, health monitoring, and novel human-computer interaction. The millimeter-wave (mmWave) based gait recognition represents the most recent advance. Compared with traditional camera-based solutions, mmWave based gait recognition bears unique advantages of being still effective under non-line-of-sight scenarios, such as in black, weak light, or blockage conditions. Moreover, they are able to accomplish person identification while preserving privacy. Currently, there are only few works in mmWave gait recognition, since no public data set is available. In this paper, we build a first-of-its-kind mmWave gait data set, in which we collect gait of 95 volunteers 'seen' from two mmWave radars in two different scenarios, which together lasts about 30 hours. Using the data set, we propose a novel deep-learning driven mmWave gait recognition method called mmGaitNet, and compare it with five state-of-the-art algorithms. We find that mmGaitNet is able to achieve 90% accuracy for single-person scenarios, 88% accuracy for five co-existing persons, while the existing methods achieve less than 66% accuracy for both scenarios.

We present Uncertainty-aware Cascaded Stereo Network (UCS-Net) for 3D reconstruction from multiple RGB images. Multi-view stereo (MVS) aims to reconstruct fine-grained scene geometry from multi-view images. Previous learning-based MVS methods estimate per-view depth using plane sweep volumes (PSVs) with a fixed depth hypothesis at each plane; this requires densely sampled planes for high accuracy, which is impractical for high-resolution depth because of limited memory. In contrast, we propose adaptive thin volumes (ATVs); in an ATV, the depth hypothesis of each plane is spatially varying, which adapts to the uncertainties of previous per-pixel depth predictions. Our UCS-Net has three stages: the first stage processes a small PSV to predict low-resolution depth; two ATVs are then used in the following stages to refine the depth with higher resolution and higher accuracy. Our ATV consists of only a small number of planes with low memory and computation costs; yet, it efficiently partitions local depth ranges within learned small uncertainty intervals. We propose to use variance-based uncertainty estimates to adaptively construct ATVs; this differentiable process leads to reasonable and fine-grained spatial partitioning. Our multi-stage framework progressively sub-divides the vast scene space with increasing depth resolution and precision, which enables reconstruction with high completeness and accuracy in a coarse-to-fine fashion. We demonstrate that our method achieves superior performance compared with other learning-based MVS methods on various challenging datasets.

Abstract Recently, deep learning‐based denoising approaches have led to dramatic improvements in low sample‐count Monte Carlo rendering. These approaches are aimed at path tracing, which is not ideal for simulating challenging light transport effects like caustics, where photon mapping is the method of choice. However, photon mapping requires very large numbers of traced photons to achieve high‐quality reconstructions. In this paper, we develop the first deep learning‐based method for particle‐based rendering, and specifically focus on photon density estimation, the core of all particle‐based methods. We train a novel deep neural network to predict a kernel function to aggregate photon contributions at shading points. Our network encodes individual photons into per‐photon features, aggregates them in the neighborhood of a shading point to construct a photon local context vector, and infers a kernel function from the per‐photon and photon local context features. This network is easy to incorporate in many previous photon mapping methods (by simply swapping the kernel density estimator) and can produce high‐quality reconstructions of complex global illumination effects like caustics with an order of magnitude fewer photons compared to previous photon mapping methods. Our approach largely reduces the required number of photons, significantly advancing the computational efficiency in photon mapping.

We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information. Our dataset consists of 573,585 part instances over 26,671 3D models covering 24 object categories. This dataset enables and serves as a catalyst for many tasks such as shape analysis, dynamic 3D scene modeling and simulation, affordance analysis, and others. Using our dataset, we establish three benchmarking tasks for evaluating 3D part recognition: fine-grained semantic segmentation, hierarchical semantic segmentation, and instance segmentation. We benchmark four state-of-the-art 3D deep learning algorithms for fine-grained semantic segmentation and three baseline methods for hierarchical semantic segmentation. We also propose a baseline method for part instance segmentation and demonstrate its superior performance over existing methods.

Binary neural networks (BNN) have been studied extensively since they run dramatically faster at lower memory and power consumption than floating-point networks, thanks to the efficiency of bit operations. However, contemporary BNNs whose weights and activations are both single bits suffer from severe accuracy degradation. To understand why, we investigate the representation ability, speed and bias/variance of BNNs through extensive experiments. We conclude that the error of BNNs is predominantly caused by intrinsic instability (training time) and non-robustness (train & test time). Inspired by this investigation, we propose the Binary Ensemble Neural Network (BENN) which leverages ensemble methods to improve the performance of BNNs with limited efficiency cost. While ensemble techniques have been broadly believed to be only marginally helpful for strong classifiers such as deep neural networks, our analysis and experiments show that they are naturally a perfect fit to boost BNNs. We find that our BENN, which is faster and more robust than state-of-the-art binary networks, can even surpass the accuracy of the full-precision floating number network with the same architecture.

Binarized Neural Networks (BNNs) eliminate bitwidth redundancy in Convolutional Neural Networks (CNNs) by using a single bit (-1/+1) for network parameters and intermediate representations. This greatly reduces off-chip data transfer and storage overhead. However, considerable computation redundancy remains in BNN inference. To tackle this problem, we investigate the similarity property in input data and kernel weights. We identify an average of 79% input similarity and 61% kernel similarity measured by our proposed metric across common network architectures. Motivated by this observation, we propose SimBNN, a fast and energy-efficient acceleration framework for BNN inference that leverages similarity properties. SimBNN consists of a set of similarity-aware accelerators, a weight reuse optimization algorithm, and a similarity selection mechanism. SimBNN incorporates two types of BNN accelerators, which exploit the input similarity and kernel similarity, respectively. More specifically, the result from the previous stage is reused if similarity is identified, thus significantly reducing BNN computation overhead. Furthermore, we propose a weight reuse optimization algorithm, which increases the weight similarity by off-line re-ordering weight kernels. Finally, our framework provides a systematic method to determine the optimal strategy between input data and kernel weights reuse, based on the similarity characteristics of input data and pre-trained BNNs.

Binarized Neural Network (BNN) removes bitwidth redundancy in classical CNN by using a single bit (-1/+1) for network parameters and intermediate representations, which has greatly reduced the off-chip data transfer and storage overhead. However, a large amount of computation redundancy still exists in BNN inference. By analyzing local properties of images and the learned BNN kernel weights, we observe an average of ~78% input similarity and ~59% weight similarity among weight kernels, measured by our proposed metric in common network architectures. Thus there does exist redundancy that can be exploited to further reduce the amount of on-chip computations.

Fusing the predictions of multiple received signal strength (RSS)-based classifiers is an efficient strategy to mitigate the impact of the fluctuation of the RSS. However, most of the existing fusion methods exhibit two remarkable shortcomings: 1) they need to train and store offline weights by the supervised learning and 2) they directly fuse the so-called candidate location set (CLS), which is collected from the most likely location estimate of each classifier (location with the largest probability of being the true location predicted by the classifier), and thus do not fully leverage the knowledge of classifiers. In general, the fluctuation of RSS does not guarantee the location predicted by each classifier with the highest probability to be the true location, thus leading to severe performance degeneration of the existing fusion methods. To overcome the above shortcomings, we propose an accurate WiFi localization framework by unsupervised fusion of an extended CLS (ECLS). First, we train multiple classifiers by only using RSS fingerprints in the offline phase. In the online phase, instead of collecting the CLS from the trained classifiers, we construct an ECLS by augmenting CLS with other location estimates (locations with predication probability greater than a certain threshold) from each classifier. As compared with the CLS, ECLS provides a bigger fusion space that likely includes the true location of the user. Furthermore, an unsupervised fusion localization algorithm based on the ECLS is derived from the joint optimization of weights and the location of the user. Furthermore, a point of inflection searching algorithm is also proposed to intelligently construct the ECLS. Real experimental results show that our proposed algorithm is more robust to changing environments and model errors, and can significantly improve the localization accuracy without any fingerprint and hardware calibrations.

The ubiquity of mobile camera devices has been triggering an outcry of privacy concerns, whereas privacy protection still relies on the cooperation of the photographer or camera hardware, which can hardly be guaranteed in practice. In this paper, we introduce LiShield, which automatically protects a physical scene against photographing, by illuminating it with smart LEDs flickering in specialized waveforms. We use a model-driven approach to optimize the waveform, so as to ensure protection against the (uncontrollable) cameras and potential image-processing based attacks. We have also designed mechanisms to unblock authorized cameras and enable graceful degradation under strong ambient light interference. Our prototype implementation and experiments show that LiShield can effectively destroy unauthorized capturing while maintaining robustness against potential attacks.

The ubiquity of mobile camera devices has been triggering an outcry of privacy concerns, whereas privacy protection still relies on the compliance of the photographer or camera hardware, which can hardly be guaranteed in practice. In this demo, we introduce LiShield, which automatically protects a physical scene against photographing, by illuminating it with smart LEDs flickering in a specialized waveform. We use a model-driven approach to optimize the waveform, so as to ensure protection against the (uncontrollable) cameras and potential image-processing based attacks. We have also designed mechanisms to unblock authorized cameras and enable graceful degradation under strong ambient light interference. This demo will show our prototype implementation, with simple on-site experiments that demonstrate how LiShield effectively destroys unauthorized photo capturing.

Despite the 4G LTE's 10X capacity improvement over 3G, mobile Web loading latency remains a major issue that hampers user experience. The root cause lies in the inefficient transport-layer that underutilizes LTE capacity, due to high channel dynamics, wireless link losses, and insufficient application traffic to propel the bandwidth probing. In this paper, we propose Cellular Link-Aware Web loading (CLAW), which boosts mobile Web loading using a physical-layer informed transport protocol. CLAW harnesses the limited PHY-layer statistics available on LTE phones to quantitatively model the LTE channel resource utilization, which is then translated into a transport window that best fits the bandwidth. Consequently, CLAW can estimate and fully utilize the available bandwidth almost within one RTT. In addition, CLAW can precisely differentiate LTE wireless loss from congestion loss, and identify the rare cases when the wireline backhaul becomes the bottleneck. We have prototyped CLAW on commodity LTE phones. Across a wide range of experimental settings, CLAW consistently reduces Web loading latency by more than 30%, compared to classical TCP variants and state-of-the-art congestion controls for cellular networks.

For over one decade, research in visible light positioning has focused on using modulated LEDs as location landmarks. But the need for specialized LED fixtures, and the associated retrofitting cost, has been hindering the adoption of VLP. In this paper, we forgo this approach and design iLAMP to enable reliable, high-precision VLP using conventional LEDs and fluorescent lamps inside today's buildings. Our key observation is that these lamps intrinsically possess hidden visual features, which are imperceptible to human eyes, but can be extracted by capturing and processing the lamps' images using a computational imaging framework. Simply using commodity smartphones' front cameras, our approach can identify lamps within a building with close to 100% accuracy. Furthermore, we develop a geometrical model which combines the camera image with gyroscope/accelerometer output, to estimate a smartphone's 3D location and heading direction relative to each lamp landmark. Our field tests demonstrate a mean localization (heading) precision of 3 cm (2.6 degree) and 90-percentile 3.5 cm (2.8 degree), even if a single lamp falls in the camera's field of view.

BACKGROUND: Precision cancer medicine (PCM) will require ready access to genomic data within the clinical workflow and tools to assist clinical interpretation and enable decisions. Since most electronic health record (EHR) systems do not yet provide such functionality, we developed an EHR-agnostic, clinico-genomic mobile app to demonstrate several features that will be needed for point-of-care conversations. METHODS: Our prototype, called Substitutable Medical Applications and Reusable Technology (SMART)® PCM, visualizes genomic information in real time, comparing a patient's diagnosis-specific somatic gene mutations detected by PCR-based hotspot testing to a population-level set of comparable data. The initial prototype works for patient specimens with 0 or 1 detected mutation. Genomics extensions were created for the Health Level Seven® Fast Healthcare Interoperability Resources (FHIR)® standard; otherwise, the prototype is a normal SMART on FHIR app. RESULTS: The PCM prototype can rapidly present a visualization that compares a patient's somatic genomic alterations against a distribution built from more than 3000 patients, along with context-specific links to external knowledge bases. Initial evaluation by oncologists provided important feedback about the prototype's strengths and weaknesses. We added several requested enhancements and successfully demonstrated the app at the inaugural American Society of Clinical Oncology Interoperability Demonstration; we have also begun to expand visualization capabilities to include cancer specimens with multiple mutations. DISCUSSION: PCM is open-source software for clinicians to present the individual patient within the population-level spectrum of cancer somatic mutations. The app can be implemented on any SMART on FHIR-enabled EHRs, and future versions of PCM should be able to evolve in parallel with external knowledge bases.

As multi-domain elastic optical networks (EONs) can enhance network scalability, extend service reach, and accommodate the inter-operability issues, it is very relevant to consider them in practical network operations. In this work, we study the problem of how to achieve energy-aware service provisioning in a multi-domain EON, where the optoelectronic regenerators only exist in border nodes. We consider dynamic lightpath requests and propose two algorithms, i.e., the greedy regenerator allocation (GRA) and the set-cover based regenerator allocation (STC), to realize the joint optimization of routing, modulation and spectrum assignment (RMSA) and regenerator allocation. The algorithms are evaluated with extensive simulations that use multi-domain EONs built with different regenerator placement strategies. The results verify that GRA and STC outperform the existing algorithm for energy-aware multi-domain service provisioning, and STC achieves the best performance in terms of both blocking probability and power efficiency.