In recent years, the community has seen the emergence of neural-based super-resolution and frame generation techniques. These methods have effectively sped up high-resolution rendering by exploiting the spatial and temporal coherence between sequential frames, but none of them are designed specifically for improving the rendering performance in VR applications, where stereo rendering doubles the rendering cost.

Advances in mobile cameras have made it easier to capture ultra-high resolution (UHR) portraits. However, existing face reconstruction methods lack specific adaptations for UHR input (e.g., 4096 × 4096), leading to under-use of high-frequency details that are crucial for achieving photorealistic rendering. Our method supports 4096 × 4096 UHR input and utilizes a divide-and-conquer approach for end-to-end 4K albedo, micronormal, and specular texture reconstruction at the original resolution. We employ a two-stage strategy to capture both global distributions and local high-frequency details, effectively mitigating mosaic and seam artifacts common in patch-based prediction. Additionally, we innovatively apply hash encoding to facial U-V coordinates to boost the model’s ability to learn regional high-frequency feature distributions. Our method can be easily incorporated in state-of-the-art facial geometry reconstruction pipelines, significantly improving the texture reconstruction quality, facilitating artistic creation workflows.

The generation of global illumination in real time has been a long-standing challenge in the graphics community, particularly in dynamic scenes with complex illumination. Recent neural rendering techniques have shown great promise by utilizing neural networks to represent the illumination of scenes and then decoding the final radiance. However, incorporating object parameters into the representation may limit their effectiveness in handling fully dynamic scenes. This work presents a neural rendering approach, dubbed LightFormer , that can generate realistic global illumination for fully dynamic scenes, including dynamic lighting, materials, cameras, and animated objects, in real time. Inspired by classic many-lights methods, the proposed approach focuses on the neural representation of light sources in the scene rather than the entire scene, leading to the overall better generalizability. The neural prediction is achieved by leveraging the virtual point lights and shading clues for each light. Specifically, two stages are explored. In the light encoding stage, each light generates a set of virtual point lights in the scene, which are then encoded into an implicit neural light representation, along with screen-space shading clues like visibility. In the light gathering stage, a pixel-light attention mechanism composites all light representations for each shading point. Given the geometry and material representation, in tandem with the composed light representations of all lights, a lightweight neural network predicts the final radiance. Experimental results demonstrate that the proposed LightFormer can yield reasonable and realistic global illumination in fully dynamic scenes with real-time performance.