In recent years, the community has seen the emergence of neural-based super-resolution and frame generation techniques. These methods have effectively sped up high-resolution rendering by exploiting the spatial and temporal coherence between sequential frames, but none of them are designed specifically for improving the rendering performance in VR applications, where stereo rendering doubles the rendering cost.

Rendering realistic images in real-time on high-frame-rate display devices poses considerable challenges, even with advanced graphics cards. This stimulates a demand for frame prediction technologies to boost frame rates. The key to these algorithms is to exploit spatiotemporal coherence by warping rendered pixels with motion representations. However, existing motion estimation methods can suffer from low precision, high overhead, and incomplete support for visual effects. In this article, we present a rendered frame prediction framework with a novel motion representation, dubbed motion-guided flow (MoFlow) , aiming at overcoming the intrinsic limitations of optical flow and motion vectors and precisely capture the dynamics of intricate geometries, lighting, and translucent objects. Notably, we construct MoFlows using a recurrent feature streaming network, which specializes in learning latent motion features from multiple frames. The results of extensive experiments demonstrate that, compared to state-of-the-art methods, our method achieves superior visual quality and temporal stability with lower latency. The recurrent mechanism allows our method to predict single or multiple consecutive frames, increasing the frame rate by over 2×. The proposed approach represents a flexible pipeline to meet the demands of various graphics applications, devices, and scenarios.

Automatic segmentation of Organs-At-Risk (OARs) and Clinical Target Volume(CTV) is crucial for the radiotherapy treatment planning of cervical cancer. This task is challenging due to the variation in sizes, shapes, and positions as well as the similar textures among the OARs and CTV. In this paper, we propose a manifold learning-based method based on U-Net. Firstly, the weight matrix of each convolutional layer is constrained to the Stiefel manifold. This constraint enhances the model’s ability to preserve the consistency of the learned feature from CT images. Secondly, we transform the optimization in Euclidean space into Riemannian optimization. This enables the model to optimize the segmentation performance on the manifold space, allowing the model to adapt to the irregular shapes of CTV and OARs. Our experimental results demonstrate that the proposed manifold learning-based method achieves superior performance in segmenting OARs and CTV for cervical cancer as compared to other SOTA methods. Overall, our proposed method demonstrates the potential of manifold learning techniques to improve the segmentation performance of medical images.

The utilization of dedicated ray tracing graphics cards has revolutionized the production of stunning visual effects in real-time rendering. However, the demand for high frame rates and high resolutions remains a challenge. The pixel warping approach is a crucial technique for increasing frame rate and resolution by exploiting the spatio-temporal coherence. To this end, existing super-resolution and frame prediction methods rely heavily on motion vectors from rendering engine pipelines to track object movements. This work builds upon state-of-the-art heuristic approaches by exploring a novel adaptive recurrent frame prediction framework that integrates learnable motion vectors. Our framework supports the prediction of transparency, particles, and texture animations, with improved motion vectors that capture shading, reflections, and occlusions, in addition to geometry movements. In addition, we introduce a feature streaming neural network, dubbed FSNet, that allows for the adaptive prediction of one or multiple sequential frames. Extensive experiments against state-of-the-art methods demonstrate that FSNet can operate at lower latency with significant visual enhancements and can upscale frame rates by at least two times. This approach offers a flexible pipeline to improve the rendering frame rates of various graphics applications and devices.

High dynamic range (HDR) illumination estimation from a single low dynamic range (LDR) image is a significant task in computer vision, graphics, and augmented reality. We present a two-stage deep learning-based method to predict an HDR environment map from a single narrow field-of-view LDR image. We first learn a hybrid parametric representation that sufficiently covers high- and low-frequency illumination components in the environment. Taking the estimated illuminations as guidance, we build a generative adversarial network to synthesize an HDR environment map that enables realistic rendering effects. We specifically consider the rendering effect by supervising the networks using rendering losses in both stages, on the predicted environment map as well as the hybrid illumination representation. Quantitative and qualitative experiments demonstrate that our approach achieves lower relighting errors for virtual object insertion and is preferred by users compared to state-of-the-art methods.

Channel estimation in millimeter-wave (mmWave) massive MIMO systems is challenging, owing to the large number of antennas and the sparsity of mmWave channels. In this paper, we apply a distributed computing technique and design a channel estimation algorithm inspired by accelerated projection-based consensus (APC). The algorithm can thus run in parallel fashion. Simulation results show that the distributed algorithm has high reliability and high convergence rate under different noise levels.