Hand tracking in XR serves as a fundamental interaction mechanism, as it allows users to directly interact with virtual content. Accurate 3D hand pose estimation is essential in scenarios involving dynamic hand motions, such as gaming, sports, and virtual musical instruments. These dynamic hand motions often result in motion blur when the hand moves faster than the frame rate of the cameras, making pose estimation challenging. The state of the art methods for 3D hand pose estimation uses deep learning that requires large amounts of data with 3D hand pose ground truth. However, most of the existing publicly available hand pose datasets are captured from static or slowly moving hands that do not contain any explicit motion blur. While techniques such as using short exposure times with higher frame rates have been employed to reduce motion blur, they still pose limitations for developing accurate hand pose estimation algorithms in the presence of fast motion. To address these challenges, firstly, we introduce a new dataset, EgoBlur, consisting of egocentric hand videos with real blur captured from a prototype Head-mounted headset. Our dataset contains$\sim 100 \mathrm{k}$images along with accurate and temporally consistent 3D hand pose ground truth. Secondly, we propose EgoBlurNet, a deep learning model capable of estimating 3D hand keypoints from blurry egocentric images by employing a teacher-student paradigm. Experimental results demonstrate that our method provides reliable and accurate 3D hand pose for blurred hand images compared to existing methods, especially in realistic dynamic XR scenarios.

Recent advances in deep learning have enabled considerable strides in hand pose estimation in well-lit conditions. However, to the best of our knowledge, there is no existing method for hand pose estimation from RGB images captured in low-light conditions. This task is highly challenging due to the overwhelming amount of noise which plague image capture in low-light conditions (<1 lux). In this paper, we propose XPose, the first method for extreme low light hand pose estimation from RGB images. We also introduce the first dataset for low light hand pose estimation consisting of ~ 120k images along with accurate hand pose labels. Our dataset consists of images captured in low light and well-lit conditions from multiple viewpoints. We propose an innovative deep learning based methodology for monocular low-light hand pose estimation using guidance from well-lit and multi-view images available in our dataset, during training time. We show that our method, using the proposed LLPose dataset, significantly outperforms existing methods for hand pose estimation both qualitatively and quantitatively in low light conditions.

Single image reflection removal is a severely ill-posed problem and it is very hard to separate the desirable transmission and undesirable reflection layers. Most of the existing single image reflection removal methods try to recover the transmission layer by exploiting cues that are extracted only from the given input image. However, there is abundant unutilized information in the form of millions of reflection free images available publicly. Even though this information is easily available, utilizing the same for effectively removing reflections is non-trivial. In this paper, we propose a novel method, termed R^2SFD, for improving single image reflection removal using a Semantic Feature Dictionary (SFD) constructed from a database of reflection-free images. The SFD is constructed using a novel Reflection Aware Feature Extractor (RAFENet) that extracts features invariant to the presence of reflections. The SFD and the input image are then passed to another novel network termed SFDNet. This network first extracts RAFENet features from the reflection-corrupted input image, searches for similar features in the SFD, and transfers the semantic content to generate the final output. To further improve reflection removal, we also introduce a Large Scale Reflection Removal (LSRR) dataset consisting of 2650 image pairs comprising of a variety of real world reflection scenarios. The proposed method achieves superior results both qualitatively and quantitatively compared to the state of the art single image reflection removal methods on real public datasets as well as our LSRR dataset. We will release the dataset at https://github.com/ee19d005/r2sfd.

Reflections caused due to surfaces such as glass affect the aesthetics of an image, and are hence undesirable. Most of the recent works on reflection removal use supervised learning based approaches using deep neural networks. However, most of these methods require large amount of paired data for training, which is difficult to obtain. Moreover, it is difficult to deploy existing deep learning based algorithms on multiple devices with different computational power, since it is very hard to control the trade-off between the strength of reflection removal and computational complexity during inference. To address these challenges, we propose a novel deep learning based approach for reflection removal, that is both unsupervised and controllable. We use Denoising Diffusion Probability Models to learn a distribution of reflection-free images. The learnt model is then used to generate reflection-free images using an input conditioned forward diffusion process during inference. We also perform qualitative and quantitative comparison and show that our method is at par or better than existing methods for deep supervised reflection removal, while outperforming unsupervised method by ~ 6.5 dB.

Single image reflection removal has attracted lot of interest in the recent past with data driven approaches demonstrating significant improvements. However deep learning based approaches for multi-image reflection removal remains relatively less explored. The existing multi-image methods require input images to be captured at sufficiently different view points with wide baselines. This makes it cumbersome for the user who is required to capture the scene by moving the camera in multiple directions. A more convenient way is to capture a burst of images in a short time duration without providing any specific instructions to the user. A burst of images captured on a hand-held device provide crucial cues that rely on the subtle handshakes created during the capture process to separate the reflection and the transmission layers. In this paper, we propose a multi-stage deep learning based approach for burst reflection removal. In the first stage, we perform reflection suppression on the individual images. In the second stage, a novel reflection motion aggregation (RMA) cue is extracted that emphasizes the transmission layer more than the reflection layer to aid better layer separation. In our final stage we use this RMA cue as a guide to remove reflections from the input. We provide the first real world burst images dataset along with ground truth for reflection removal that can enable future benchmarking. We evaluate both qualitatively and quantitatively to demonstrate the superiority of the proposed approach. Our method achieves ~ 2dB improvement in PSNR over single image based methods and ~ 1dB over multi-image based methods.

Shadow Removal is an important and widely researched topic in computer vision. Recent advances in deep learning have resulted in addressing this problem by using convolutional neural networks (CNNs) similar to other vision tasks. But these existing works are limited to low-resolution images. Furthermore, the existing methods rely on heavy network architectures which cannot be deployed on resource-constrained platforms like smartphones. In this paper, we propose SHARDS, a shadow removal method for high-resolution images. The proposed method solves shadow removal for high-resolution images in two stages using two lightweight networks: a Low-resolution Shadow Removal Network (LSRNet) followed by a Detail Refinement Network (DRNet). LSRNet operates at low-resolution and computes a low-resolution, shadow-free output. It achieves state-of-the-art results on standard datasets with 65x lesser network parameters than existing methods. This is followed by DRNet, which is tasked to refine the low-resolution output to a high-resolution output using the high-resolution input shadow image as guidance. We construct high-resolution shadow removal datasets and through our experiments, prove the effectiveness of our proposed method on them. It is then demonstrated that this method can be deployed on modern day smartphones and is the first of its kind solution that can efficiently (2.4secs) perform shadow removal for high-resolution images (12MP) in these devices. Like many existing approaches, our shadow removal network relies on a shadow region mask as input to the network. To complement the lightweight shadow removal network, we also propose a lightweight shadow detector in this paper.

Hand-held photography in low-light conditions presents a number of challenges to capture high quality images. Capturing using a high ISO results in noisy images, while capturing using longer exposure results in blurry images. This necessitates post-processing techniques to restore the latent image. Most existing methods try to estimate the latent image either by denoising or by deblurring a single image. Both these approaches are ill-posed and often result in unsatisfactory results. A few methods try to alleviate this ill-posedness using a pair of noisy-blurry images as inputs. However, most of the methods using this approach are computationally very expensive. In this paper, we propose a fast method to estimate a latent image given a pair of noisy-blurry images. To accomplish this, we propose a deep-learning based approach that uses scale space representation of the images. To improve computational efficiency, we process higher scale spaces using shallower networks and the lowest scale using a deeper network. Also, unlike existing scale-space methods that use bi-cubic interpolation, we propose a content preserving scale space transformation for decimation and interpolation. The proposed method generates state-of-the-art results at reduced computational complexity compared to state-of-the-art method. Finally, we also show that computational efficiency can be improved by 90% compared to baseline with only a marginal drop in PSNR.

Reflections in images are typically caused due to presence of glass like reflective objects or surfaces that affect the overall visual appeal and hence undesirable. There has been extensive interest in the past to use data driven approaches for both single image as well as multi image reflection removal. However, recently there has been only minor incremental improvements in single image reflection removal given the challenging ill-posed nature of the problem. Reference based methods has yielded state of the art performance in areas such as super resolution, however has been unexplored for reflection removal. In this paper, we propose a novel multi-stage deep learning based method for reference based reflection removal. We also propose a novel visual attribute cue that represents the reflection free semantic content of the input image. This cue is generated using the reference image while maintaining the geometric structure of the input image. We formulate the reference based reflection removal problem as extraction of visual attribute cues followed by a guided image restoration. We perform qualitative and quantitative evaluation to demonstrate the superiority of the proposed approach over the existing state of the art single image reflection removal methods.

Real world images often gets corrupted due to unwanted reflections and their removal is highly desirable. A major share of such images originate from smart phone cameras capable of very high resolution captures. Most of the existing methods either focus on restoration quality by compromising on processing speed and memory requirements or, focus on removing reflections at very low resolutions, there by limiting their practical deploy-ability. We propose a light weight deep learning model for reflection removal using a novel scale space architecture. Our method processes the corrupted image in two stages, a Low Scale Sub-network (LSSNet) to process the lowest scale and a Progressive Inference (PI) stage to process all the higher scales. In order to reduce the computational complexity, the sub-networks in PI stage are designed to be much shallower than LSSNet. Moreover, we employ weight sharing between various scales within the PI stage to limit the model size. This also allows our method to generalize to very high resolutions without explicit retraining. Our method is superior both qualitatively and quantitatively compared to the state of the art methods and at the same time 20× faster with 50× less number of parameters compared to the most recent state-of-the-art algorithm RAGNet. We implemented our method on an android smart phone, where a high resolution 12 MP image is restored in under 5 seconds.

Multi-frame High Dynamic Range (HDR) imaging is a technique to improve details while capturing scenes with large variations in luminosity by blending together a stack of images captured using varying exposure values. Local motion between the stack of images often result in ghost artifacts or reduced dynamic range especially in over-exposed or underexposed regions. This paper proposes a method to mitigate these problems using a novel multi-stage algorithm. At each stage of the proposed method, intermediate HDR images are generated from consecutive exposure pairs using a computational unit termed as Basis HDR. The outputs produced by each Basis HDR unit is ensured to be free of ghosts using a novel algorithm to estimate local motion in the scene. Qualitative and quantitative comparison studies show that the proposed method generates images with improved dynamic range, minimal ghosts and a higher MEF-SSIM score as compared to existing state-of-the-art. The proposed method can process a 3-frame HDR on a smartphone in under a second thus making it a fast and reliable algorithm for smartphones.

High Dynamic Range (HDR) imaging provides a methodology to capture a wide luminance range in a single image which traditional imaging techniques fail to capture. State of the art deep learning methods in multi-frame HDR imaging follow an end-to-end learning approach and often fail to generate realistic details in large occluded regions. In this paper, we propose to split the HDR problem into multiple stages and tackle them using separate Convolutional Neural Networks (CNNs) rather than attempting an end-to-end learning. First, in the Exposure Alignment Stage, we propose to generate virtual exposure images which are similar in structure to a chosen reference input using deep CNNs. This is followed by an HDR Merge stage, where another CNN learns to generate the HDR output from the virtual exposure images. We perform extensive comparative studies to show that the proposed method generates artifact-free outputs with plausible details in occluded regions.