Recent advancements in learning-based methods have opened new avenues for exploring and interpreting art forms, such as shadow art, origami, and sketch art, through computational models. One notable visual art form is 3D Anamorphic Art in which an ensemble of arbitrarily shaped 3D objects creates a realistic and meaningful expression when observed from a particular viewpoint and loses its coherence over the other viewpoints. In this work, we build on insights from 3D Anamorphic Art to perform 3D object arrangement. We introduce RASP, a differentiable-rendering-based framework to arrange arbitrarily shaped 3D objects within a bounded volume via shadow (or silhouette)-guided optimization with an aim of minimal inter-object spacing and near-maximal occupancy. Furthermore, we propose a novel SDF-based formulation to handle inter-object intersection and container extrusion. We demonstrate that RASP can be extended to part assembly alongside object packing considering 3D objects to be "parts" of another 3D object. Finally, we present artistic illustrations of multi-view anamorphic art, achieving meaningful expressions from multiple viewpoints within a single ensemble.

Photometric stereo is a powerful technique for estimating per-pixel surface normals from images under varied il-lumination. Although several methods address photometric stereo with different image (or light) counts ranging from one to two to a hundred, very few focus on learning optimal lighting configuration. Finding an optimal configuration is challenging due to the large number of possible lighting di-rections. Moreover, exhaustive sampling of all possibilities is impractical due to time and resource constraints. Pho-tometric stereo methods have demonstrated promising per-formance on existing datasets, which feature limited light directions sparsely sampled from the light space. There-fore, can we optimally utilize these datasets for illumination planning? In this work, we introduce LIPIDS - Learning-based Illumination Planning In Discretized light Space to achieve minimal and optimal lighting configurations for photometric stereo under arbitrary light distribution. We propose a Light Sampling Network (LSNet) that optimizes the lighting direction for a fixed number of lights by min-imizing the normal loss through a normal regression net-work. The learned light configurations can directly estimate surface normals during inference, even using an off-the-shelf photometric stereo method. Extensive qualitative and quantitative analysis on synthetic and real-world datasets show that photometric stereo under learned lighting config-urations through LIPIDS either surpasses or is nearly com-parable to existing illumination planning methods across different photometric stereo backbones.

Abstract Estimating scattering parameters of heterogeneous media from images is a severely under‐constrained and challenging problem. Most of the existing approaches model BSSRDF either through an analysis‐by‐synthesis approach, approximating complex path integrals, or using differentiable volume rendering techniques to account for heterogeneity. However, only a few studies have applied learning‐based methods to estimate subsurface scattering parameters, but they assume homogeneous media. Interestingly, no specific distribution is known to us that can explicitly model the heterogeneous scattering parameters in the real world. Notably, procedural noise models such as Perlin and Fractal Perlin noise have been effective in representing intricate heterogeneities of natural, organic, and inorganic surfaces. Leveraging this, we first create HeteroSynth, a synthetic dataset comprising photorealistic images of heterogeneous media whose scattering parameters are modeled using Fractal Perlin noise. Furthermore, we propose Tensorial Inverse Scattering (TensoIS), a learning‐based feed‐forward framework to estimate these Perlin‐distributed heterogeneous scattering parameters from sparse multi‐view image observations. Instead of directly predicting the 3D scattering parameter volume, TensoIS uses learnable low‐rank tensor components to represent the scattering volume. We evaluate TensoIS on unseen heterogeneous variations over shapes from the HeteroSynth test set, smoke and cloud geometries obtained from open‐source realistic volumetric simulations, and some real‐world samples to establish its effectiveness for inverse scattering. Overall, this study is an attempt to explore Perlin noise distribution, given the lack of any such well‐defined distribution in literature, to potentially model real‐world heterogeneous scattering in a feed‐forward manner. Project Page: https://yashbachwana.github.io/TensoIS/

Graph convolutional networks (GCNs) have achieved impressive performance in learning from graph-structured data. Although GCN and its variants have shown promising results, they continue to remain shallow as their performance drops with an increasing number of layers - a problem popularly known as oversmoothing. This work introduces a simple yet effective idea of feature gating over graph convolution layers to facilitate deeper graph neural networks and address oversmoothing. The proposed feature gating is easy to implement without changing the underlying network architecture and is broadly applicable to GCN and almost any of its variants. Further, we demonstrate the use of feature gating in assigning importance to node features and the nodes for the node classification task. Quantitative analysis on real-world datasets shows that feature gating paves the way for constructing deeper GCNs.

Photometric stereo is a method to obtain surface normals of an object using its images captured under varying illumination directions. The existing deep learning-based methods require multiple images of an object captured using complex image acquisition systems. In this work, we propose a deep learning framework to perform three tasks jointly: (i) lighting estimation, (ii) image relighting, and (iii) surface normal estimation, all from a single input image of an object with non-Lambertian surface and general reflectance. The network explicitly segregates global geometric features and local lighting-specific features of the object from a single image. The local features resemble attached shadows, shadings, and specular highlights, providing valuable lighting estimation and relighting cues. The global features capture the lighting-independent geometric attributes that effectively guide the surface normal estimation. The joint training transfers valuable insights to achieve significant improvements across all three tasks. We show that the proposed single-image-based relighting framework outperforms several existing photometric stereo methods which require multiple images of a static object.

Handwritten document image binarization is challenging due to high variability in the written content and complex background attributes such as page style, paper quality, stains, shadow gradients, and non-uniform illumination. While the traditional thresholding methods do not effectively generalize on such challenging real-world scenarios, deep learning-based methods have performed relatively well when provided with sufficient training data. However, the existing datasets are limited in size and diversity. This work proposes LS-HDIB - a large-scale handwritten document image binarization dataset containing over a million document images that span numerous real-world scenarios. Additionally, we introduce a novel technique that uses a combination of adaptive thresholding and seamless cloning methods to create the dataset with accurate ground truths. Through an extensive quantitative and qualitative evaluation over eight different deep learning based models, we demonstrate the enhancement in the performance of these models when trained on the LS-HDIB dataset and tested on unseen images.

The fidelity of a pose transfer system depends on its ability to generate realistic images of a person under novel poses while preserving the desired human attributes (like face, hairstyle, and clothes). However, the visual fidelity is often compromised as the existing methods fail to extract rich appearance and pose features since they propagate the pose and the appearance information through the same pathway. Also, the repeated downsampling in these pathways leads to the loss of finer details, thus producing blurry results. Further, these methods use vanilla convolution that treats all the pixels as important and fail to focus primarily on significant regions needed for the desired transformation. This work proposes an appearance-consistent human pose transfer framework that progressively transforms the person in the source image to the desired target pose using the information from three pathways: an image pathway, a pose pathway, and an appearance pathway. We propose the use of gated convolution to dynamically extract features relevant for generating the transformed image. The appearance pathway generates an appearance code to produce an image consistent in appearance with that of the source image. We establish the efficacy of the proposed framework through an extensive set of experiments on DeepFashion, Market-1501, and the Action Class dataset. We also generate coherent action sequences through a given set of desired poses from the action class dataset that contains humans in three actions: golf, yoga/workouts, and tennis.

While recent learning-based methods have been observed to be superior for several vision-related applications, their potential in generating artistic effects has not been explored much. One such exciting application is Shadow Art - a unique form of sculptural art that produces artistic effects through 2D shadows cast by a 3D sculpture. In this work, we revisit shadow art using differentiable rendering-based optimization frameworks to obtain the 3D sculpture from a set of shadow (binary) images and their corresponding projection information. Specifically, we discuss shape optimization through voxel as well as mesh-based differentiable renderers. Our choice of using differentiable rendering for generating shadow art sculptures can be attributed to its ability to learn the underlying 3D geometry solely from image data, thus reducing the dependence on 3D ground truth. The qualitative and quantitative results demonstrate the potential of the proposed framework in generating complex 3D sculptures that transcend the ones seen in contemporary art pieces using just a set of shadow images as input. Further, we demonstrate the generation of 3D sculptures to cast shadows of faces, animated movie characters, and the applicability of the proposed framework to sketch-based 3D reconstruction of the underlying shapes.