Image super-resolution (SR) is essential for bridging the gap between modern hardware and real-time computer graphics (CG) applications. It reduces CG workload by allowing low-resolution rendering, with original quality restored later via mathematical operations or machine learning. However, recent learning-based SR methods often rely on complex models, demanding high computational resources and undermining the benefits of reduced rendering workload. Our qualitative and quantitative analysis of the SR process and rendering reveals that readily accessible rendering information can significantly enhance neural network design by serving as additional features. To capitalize on this, we propose CGSR, an optimization framework designed for lightweight real-time super-resolution. CGSR utilizes rendering information to boost both network extensibility and efficiency. It utilizes progressively available rendering information from the pipeline, which arrives earlier than the rendered frame, enabling pre-processing and masking of latency. These features are then integrated into a selected SR network backbone to form a CG-enhanced network. This network is further optimized and refined into a CG-optimized version using neural architecture search (NAS). To improve runtime performance, CGSR also employs rendering-aware hybrid pruning, which dynamically prunes the network based on temporal rendering data. Evaluation results show that CGSR significantly reduces parameter size, multi-add operations, and inference time while maintaining high SR quality across various backbone SR networks.

Online semantic 3D modeling from streaming RGB-D data fundamentally requires consistent fusion of 2D segmentation. Popular approaches address segmentation inconsistencies through histogram-based label aggregation, where each 3D element (point/voxel) maintains the frequency of candidate labels, which introduces prohibitive memory and computational overhead for resource-constrained devices. In response to this challenge, we propose MEFusion, a memory-efficient probabilistic fusion framework to avoid element-wise histogram aggregation. Specifically, we propose an element-wise probability update algorithm based on Bayesian Estimation, where each voxel stores only one instance label and updates it based on a posterior probability to maintain segmentation consistency. Following 3D segmentation, we establish a segment-wise voting framework to aggregate the semantic labels from historical data, where co-segment voxels share the semantic voting histogram, for semantic consistency. Our experiments demonstrate that our method achieves a memory reduction of 77%(85%) and a speed improvement of 58%(6.12x) on the desktop (embedded) platform while maintaining comparable reconstruction accuracy to the state-of-the-art point-cloud-based method.

In 3D reconstruction, mirrors introduce significant challenges by creating distorted and fragmented spaces, resulting in inaccurate and unreliable 3D models. As 3D reconstruction typically relies on multi-view images to capture different perspectives of a scene, detecting and labeling mirrors in multi-view images before reconstruction can effectively address this issue. However, existing methods focus solely on single-image detection, overlooking the rich information provided by multi-view setups. To overcome this limitation, we propose MVMD, a novel Multi-View Mirror Detection method, along with the first database specifically designed for mirror detection in multi-view scenes. The design of MVMD is grounded in the inherent associations between objects seen from different views and those reflected inside and outside of mirrors. These relationships are learned through cross-and self-attention mechanisms. MVMD consists of three key blocks: the Inter-Views Block tracks the shifts of objects within mirrors caused by changes in viewpoint; the Intra-View Block detects object reflections inside mirrors; and the Refinement Block sharpens mirror boundaries and enhances detected details. Experimental results show that our method improves accuracy by up to 2.6% and IoU by up to 11.1%, compared to single-image mirror detection techniques. This substantial improvement makes MVMD particularly effective for computer vision tasks, especially in enhancing the accuracy of 3D reconstruction in mirror-dense environments. Code and data are available at: https://github.com/mvmdwacv25.

Deep learning and computer vision have experienced significant advancements, particularly in critical applications such as autonomous driving and real-time surveillance, where object detection (OD) plays a pivotal role. Ensuring the accuracy and speed of these systems is paramount to prevent accidents or failures. Recently, latency-based attacks have emerged as a new threat, driven by the essential need for real-time performance in various applications. These attacks target model responsiveness to disrupt system performance without necessarily compromising accuracy. Our preliminary experiments show that introducing just a few bit flips to key parameters in OD models can significantly increase latency, degrading performance. Meanwhile, recent advancements in memory-based attacks, such as Row Hammer [18], demonstrate the ability to conveniently introduce bit flips at desired locations without physical hardware interaction. Based on the observations, we propose a novel attack on OD models that leverages row-hammer to introduce bit-flips via side channels, targeting the non-maximum suppression (NMS) filter and significantly increasing latency. Unlike previous methods that modify input data, our technique ensures efficiency by minimizing bit-flips through critical path exploitation and achieves practical applicability with only a subset of validation data. Experiments across various datasets and models validate our approach, demonstrating latency increases up to 71.6 ms (20.4×) with just 31 bit-flips.

Multi-user Augmented Reality (MuAR) allows multiple users to interact with shared virtual objects, facilitated by exchanging environment information. Current MuAR systems rely on 3D point clouds for real-world analysis, view synchronization, object rendering, and movement tracking. However, the complexity of 3D point clouds leads to significant processing delays, with approximately 80% of overhead in commercial frameworks. This hampers usability and degrades user experience. Our analysis reveals that maintaining the facing side of the real-world scene in a stable environment provides sufficient information for virtual object placement and rendering. To address this, we introduce a lightweight quadtree structure, representing 2D scenes through semantic segmentation and geometry, as an alternative to 3D point clouds. Additionally, we propose a novel correction method to handle potential shifts in virtual object placement during view synchronization among users. Combining all designs, we implement a fast and lightweight MuAR framework namedAR-Lightand test our framework on commercial AR devices. The evaluation results on real-world applications demonstrate that AR-Light can achieve high performance in various real-world scenes while maintaining a comparable virtual object placement accuracy.

We introduce the Safe Offline-to-Online Multi-Agent Decision Transformer (SO2-MADT), an innovative framework that revolutionizes safety considerations in Multi-agent Reinforcement Learning (MARL) through a novel sequence modeling approach. Leveraging the dynamic capabilities inherent in Decision Transformers, our methodology seamlessly incorporates safety protocols as a cornerstone element, ensuring secure operations throughout both the offline pre-training phase and the adaptive online fine-tuning phase. At the core of our framework lie two pivotal innovations: the Safety-To-Go (STG) token, embedding safety at a macro level, and the Agent Prioritization Module (APM), facilitating explicit credit assignment at a micro level. Through extensive testing against the challenging environments of the StarCraft Multi-Agent Challenge (SMAC) and Multi-agent MuJoCo, our SO2-MADT not only excels in offline pre-training but also demonstrates superior performance during online fine-tuning, without any degradation in performance. The implications of our work provide a pathway for deployment in critical real-world applications where safety is paramount and non-negotiable. The code is available at https://github.com/shahaamirbader/SO2-MADT.

To provide users with a fully immersive environment, VR post-processing, which adds numerous realistic effects on the frame after rendering, plays a key role in modern VR systems. Current post-processing is processed separately from normal rendering by the graphics processing unit (GPU). As a result, the GPU needs to first render a high-resolution frame and then add the post-processing effects within a very short time frame. Our in-depth experimental results on commercial VR products demonstrate that the post-processing in VR applications extends the VR frame time by approximately 2X on average. Furthermore, the ever-increasing resolution requirements of modern VR significantly increase the workloads for post-processing in the execution pipeline. This long delay causes VR real-time execution to frequently miss the critical frame-time deadline, thus hurting users’ quality of experience.Based on the analysis of VR post-processing workflow and its common realistic effects, we observe that post-processing shares the same hardware pipeline with normal rendering, and even reuses the intermediate data produced by normal rendering. To fully utilize this hardware-level similarity and capture the data locality, we propose a novel universal realistic rendering architecture for VR, named Post0-VR, which eliminates post-processing by directly merging the common realistic effects into the normal rendering process. Based on our newly proposed VR architecture design, we further propose a dynamic accuracy adjustment method to simplify the normal rendering without hurting users’ perception. The evaluation results on real-world applications demonstrate that Post0-VR can support different types of realistic effects while significantly improving the overall VR rendering performance.