This paper presents an approach for extending 360 VR to support viewpoint translation and collaboration, without increasing the acquisition cost. The approach models the ground under the 360° image acquisition viewpoint with a planar patch, on which the user stands and walks. The planar patch is connected to the background with visual continuity. The approach anchors the user's avatar to the ground patch, and supports collaboration, with multiple avatars standing on the patch. The approach only requires defining two trivial parameters: the approximate acquisition height, and the size of the patch. Therefore, the approach improves the immersive exploration of a real-world scene captured with a 360° image, without increasing the cost of the acquisition of the virtual environment. The approach was evaluated in a controlled user study (N = 30) with five user t asks. The results show that the approach has an advantage over conventional 360 VR in terms of anchoring the user to the ground, of user viewpoint translation, of collaboration, and of accurate integration of stationary and dynamic virtual objects into the 3D scene.

This paper presents an approach to alleviating the server-client communication bottleneck in collocated multi-user virtual reality (VR). Instead of transmitting unique packets to each client, the system multicasts the same packets to all users, combining packetization, visibility-aware prioritization, and lightweight acknowledgments to ensure efficiency and fairness. To enable rapid scene completion, the environment is partitioned into independent fixed-size packets that can be decoded immediately upon arrival, and transmission order is guided by precomputed visibility footprints. Scalability is achieved through repeated multicast until compact bitmap acknowledgments confirm reception, keeping communication cost mostly independent of the number of clients. A controlled study with 23 participants in groups of 7 and 16 showed that this method reconstructs environments several times faster and with fewer missing parts than a conventional unicast TCP-based approach. These findings demonstrate that multicast transmission over commodity Wi-Fi can support large collocated VR groups.

Immersive virtual reality (VR) experiences require transmission and rendering of large-scale 3D content, often represented as point clouds or polygon meshes. Unfortunately, existing networked VR systems often fail to fully exploit the flexibility of VR data representations. To address this problem, we propose a cross-layer design that elevates a network data unit to a usable rendering unit for VR applications. Our aim is to bridge the gap between networks and applications in order to enhance visual quality, especially over constrained and variable networks. Our approach, Rendering Unit that is Network-aware (RUN), with two variants, RUN-Packet and RUN-Hybrid, includes mechanisms to effectively utilize network data units when encoding, transmitting, decoding, and rendering. Specifically, we develop additive detail refinement mechanisms and address streaming challenges such as head-of-line (HoL) blocking. We prototype our system in Unity 3D and evaluate it using synthetic network environments and real network traces. Our results with both static and dynamic point clouds demonstrate that RUN significantly reduces stalls and delivers smoother frame updates, enhancing visual quality.

This paper describes XRXL, an extended-reality system for increasing student engagement in large lectures. Students wear XR headsets to see 3D visualizations controlled by the instructor. The instructor can virtually retract the roof and walls of the classroom to allow for large-scale visualizations that extend beyond the physical boundaries of the classroom, or to turn the classroom into a 360° theater. The instructor can also partition the classroom into small groups of students and to assist individual groups as needed. XRXL was tested in an IRB-approved user study with 82 students in the context of a mock-lecture on neural networks. To the best of our knowledge, the study is the largest deployment of a co-located collaborative XR application to date. The study shows that students had a favorable opinion of XRXL, that XRXL had a low task load, an acceptable usability level, and that it did not cause cybersickness.

All-in-one VR headsets have limited rendering power which limits the complexity of the virtual environments (VEs) that can be used in VR applications. This paper describes a novel visibility algorithm for making complex VEs tractable on all-in-one VR headsets. Given a view segment, the algorithm finds the set of triangles visible as a camera translates on the view segment. When run on the perimeter of a user view region, the algorithm provides a quality approximation of the visible set from inside the view region. The visibility algorithm supports static and dynamic VEs, and it solves visibility with either triangle, particle, or object granularity. The visible sets yield output frames that are virtually indistinguishable from ground truth frames rendered from the original VEs.

This article describes a mixed reality visual instruction approach for remote collaboration between a trainee and an expert. The expert authors the visual instructions through a virtual reality interface. The instructions are shown to the trainee overlaid onto the workspace using an augmented reality interface. The approach achieves effectiveness and efficiency by addressing three challenges. First, the expert-authored visual instructions are shown to the trainee by taking into account occlusions with the 3D workspace; Second, in addition to abstract visual instructions implemented by arrows, the expert can also author highly suggestive instructions by depicting the target state of the workspace realistically by selecting, copying, pasting, and repositioning workspace objects; Third, multiple instructions can be concatenated in sequences that the trainee executes on their own, without any additional guidance from the expert; The approach has been evaluated in a controlled user study with three experiments. The experiment verification confirms that compared to the conventional instruction, this approach achieves significantly lower error rates, shorter task completion times, and lower rotation angular errors. Moreover, the approach allows the trainee to execute the entire sequence robustly, without real-time instruction from the expert.

In this special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the top papers from the 32nd IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2025), held March 8–12, 2025, in Saint-Malo, France.

Redirection in virtual reality (VR) enhances haptic feedback versatility by relaxing the need for precise alignment between virtual and physical objects. In mixed reality (MR), where users see the real world and their own hands, haptic redirection enables a physical interaction with virtual objects but poses greater challenges due to altering real-world perception. This paper investigates the effect of the realism of the user's surroundings and of the user's hand on haptic redirection. The user's familiarity with their actual physical surroundings and their actual hand could make the redirection manipulations easier-or harder-to detect. In a user study (N = 30) participants saw either a virtual environment or their actual physical surroundings, and saw their hand rendered either with a generic 3D model or with a live 2D video sprite of their actual hand. The study used a two-alternative forced choice (2AFC) design asking participants to detect hand redirections that bridged physical to virtual offsets of varying magnitudes. The results show that participants were not more sensitive to 2D video sprite hand redirection than to VR hand redirection, which supports the use of haptic redirection in MR.

Abstract An encountered-type haptic device (ETHD) moves a physical object to align it with the virtual object with which the user of a virtual reality application makes contact, providing haptic feedback. One limitation of ETHDs is their limited reachability due to mechanical constraints. One approach to extending the reachability of an ETHD involves redirection, achieved by altering the pose of the user’s body, hand, or handheld prop in the virtual world. While previous studies have quantified the detection thresholds of redirection in the context of stationary objects, ETHD’s present the opportunity and the need to study redirection in the context of dynamic objects. This paper presents a user study $(\mathrm{N}=25)$ with two experiments aimed at investigating whether dynamic object properties, such as direction and speed, significantly affect redirection detection thresholds. The first experiment finds that the mere presence of dynamic objects does not decrease the detection threshold. Consequently, previously measured detection thresholds remain applicable for ETHD reachability extension. However, the second experiment unveils a crucial relationship between the speed of dynamic objects and the original reachability of the ETHD. Although the virtual object can move $15 \mathrm{~cm} / \mathrm{s}$ faster than the ETHD, this increase in speed is insufficient to compensate for the extended reachability enabled by the detection threshold.

This paper investigates HyperXRC, a hybrid classroom design that accommodates both local and remote students. The instructor wears an extended reality (XR) headset that shows the local classroom and the local students, as well as remote students modeled with video sprites. The remote students are displayed either on virtual banners hanging off the classroom ceiling, or on virtual billboards placed in empty classroom seats. Thereby, the remote students are integrated into the field of view of the instructor, who remains aware of the remote students while teaching. A controlled user study with two experiments evaluated the HyperXRC design from the instructor and from the local students perspective. In the first experiment (N = 15) participants served as instructors to a hybrid classroom of 14 local and 15 remote students. Participants were more likely to detect hand-raising and head-on-desk remote student actions in the HyperXRC conditions (59%) than in a conventional videoconferencing condition (36%). This advantage did not come at the cost of decreasing the detection rate of local student actions. Furthermore, instructor participants preferred the HyperXRC to the videoconferencing approach. In the second experiment (N = 16) participants served as local students. The participants preferred the lecture when the instructor used videoconferencing to the one when the instructor used HyperXRC, wearing the XR headset.

We are pleased to present the proceedings of the 31st IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2024), held March 16–21, 2024, in Orlando, Florida, US, in on-site format. These proceedings contain 101 of the 180 IEEE VR 2024 papers, with the other 79 papers being published in the IEEE VR 2024 special issue of the Transactions on Visualization and Computer Graphics. IEEE VR 2024 had 627 submissions for an acceptance rate of 101/627=16.1% as conference papers, or 180/627=28.7% when including the journal papers.

In this special issue ofIEEE Transactions on Visualization and Computer Graphics(TVCG), we are pleased to present the top papers from the 31th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2024), held March 16-21, 2024, in Orlando, Florida, US.

The paper introduces PreVR, a method for allowing the user of a VR application to preview a virtual environment (VE) around any number of corners. This way the user can gain line of sight to any part of the VE, no matter how distant or how heavily occluded it is. PreVR relies on a multiperspective visualization that implements a higher-order disocclusion effect with piecewise linear rays that bend multiple times as needed to reach the visualization target. PreVR was evaluated in a user study ($\mathrm{N}=88$) that investigates four points on the VR interface design continuum defined by the maximum disocclusion order $\delta$. In a first control condition (CC0), $\delta=0$, corresponds to conventional VR exploration with no preview capability. In a second control condition (CC1), $\delta=1$, corresponds to the prior art approach of giving the user a preview around the first corner. In a first experimental condition (EC3), $\delta=3$, so PreVR provided up to third-order disocclusion. In a second experimental condition (ECN), $\delta$ was not capped, so PreVR could provide a disocclusion effect of any order, as needed to reach any location in the VE. Participants searched for a stationary target, for a dynamic target moving on a random continuous trajectory, and for a transient dynamic target that appeared at random locations in the maze and disappeared 5s later. The study quantified VE exploration efficiency with four metrics: viewpoint translation, view direction rotation, number of teleportations, and task completion time. Results show that the previews afforded by PreVR bring a significant VE exploration efficiency advantage. ECN outperforms EC3, CC1, and CC0 for all metrics and all tasks, and EC3 frequently outperforms CC1 and CC0.

This article presents two from-point visibility algorithms: one aggressive and one exact. The aggressive algorithm efficiently computes a nearly complete visible set, with the guarantee of finding all triangles of a front surface, no matter how small their image footprint. The exact algorithm starts from the aggressive visible set and finds the remaining visible triangles efficiently and robustly. The algorithms are based on the idea of generalizing the set of sampling locations defined by the pixels of an image. Starting from a conventional image with one sampling location at each pixel center, the aggressive algorithm adds sampling locations to make sure that a triangle is sampled at all the pixels it touches. Thereby, the aggressive algorithm finds all triangles that are completely visible at a pixel regardless of geometric level of detail, distance from viewpoint, or view direction. The exact algorithm builds an initial visibility subdivision from the aggressive visible set, which it then uses to find most of the hidden triangles. The triangles whose visibility status is yet to be determined are processed iteratively, with the help of additional sampling locations. Since the initial visible set is almost complete, and since each additional sampling location finds a new visible triangle, the algorithm converges in a few iterations.

VR headsets have limited rendering capability, which limits the size and detail of the virtual environment (VE) that can be used in VR applications. One solution is cloud VR, where the "thin" VR clients are assisted by a server. This paper describes Cio VR, a cloud VR system that provides fast loading times, as needed to let users see and interact with the VE quickly at session startup or after teleportation. The server reduces the original VE to a compact representation through near-far partitioning. The server renders the far region to an environment map which it sends to the client together with the near region geometry, from which the client renders quality frames locally, with low latency. The near region starts out small and grows progressively, with strict visual continuity, minimizing startup time. The low-latency and fast-startup advantages of CloVR have been validated in a user study where groups of 8 participants wearing all-in-one VR headsets (Quest 2's) were supported by a laptop server to run a collaborative VR application with a 25 million triangle VE.

We are pleased to present the proceedings of the 30th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2023), held March 25–29, 2023, in Shanghai, China, in hybrid format. These proceedings contain 69 of the 130 papers presented at IEEE VR 2023, with the other 61 papers being published in the IEEE VR 2023 special issue of the Transactions on Visualization and Computer Graphics. IEEE VR 2023 had 612 submissions for an acceptance rate of 21%.

An important application of augmented reality (AR) is the design of interfaces that reveal parts of the real world to which the user does not have line of sight. The design space for such interfaces is vast, with many options for integrating the visualization of the occluded parts of the scene into the user's main view. This paper compares four AR interfaces for disocclusion: X-ray, Cutaway, Picture-in-picture, and Multiperspective. The interfaces are compared in a within-subjects study (N = 33) over four tasks: counting dynamic spheres, pointing to the direction of an occluded person, finding the closest object to a given object, and finding pairs of matching numbers. The results show that Cutaway leads to poor performance in tasks where the user needs to see both the occluder and the occludee; that Picture-in-picture and Multiperspective have a visualization comprehensiveness advantage over Cutaway and X-ray, but a disadvantage in terms of directional guidance; that X-ray has a task completion time disadvantage due to the visualization complexity; and that participants gave Cutaway and Picture-in-picture high, and Multiperspective and X-ray low usability scores.

In this special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the top papers from the 30th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2023), held March 25–29, 2023, in Shanghai, China, in hybrid format.

This paper proposes a general handheld stick haptic redirection method that allows the user to experience complex shapes with haptic feedback through both tapping and extended contact, such as in contour tracing. As the user extends the stick to make contact with a virtual object, the contact point with the virtual object and the targeted contact point with the physical object are continually updated, and the virtual stick is redirected to synchronize the virtual and real contacts. Redirection is applied either just to the virtual stick, or to both the virtual stick and hand. A user study (N = 26) confirms the effectiveness of the proposed redirection method. A first experiment following a two-interval forced-choice design reveals that the offset detection thresholds are [-15cm, +15cm]. A second experiment asks participants to guess the shape of an invisible virtual object by tapping it and by tracing its contour with the handheld stick, using a real world disk as a source of passive haptic feedback. The experiment reveals that using our haptic redirection method participants can identify the invisible object with 78% accuracy.

This paper describes a method for reducing rendering load such that complex virtual environments (VEs) can be deployed on “thin” VR systems with limited rendering power. The method partitions the VE into four regions: a near region, an intermediate region, a stationary far region, and a dynamic far region. The stationary far region is replaced with an environment map, which brings a substantial rendering load reduction. The other three regions are rendered from geometry: the near region is rendered from the user viewpoint, the dynamic far geometry is rendered from the center of the environment map, and the intermediate region is rendered with a morph that switches viewpoint gradually from the user viewpoint to the center of the environment map. The intermediate region connects the near and far regions seamlessly. Furthermore, the environment map is enhanced with per pixel range which allows depth compositing the dynamic and stationary far geometry. An IRB approved user study (N = 22) found significant advantages for our method over conventional near-far partitioning.

This paper investigates providing grounded passive haptic feedback to a user of a VR application through a handheld stick with which the user taps virtual objects. Such an investigation benefits VR applications beyond those where the stick interaction is actually an integral part of the narrative. Providing passive haptic feedback through a handheld stick as opposed to directly through the user’s hand has the potential for more believable and more frequent feedback opportunities. The stick is likely to dull the user’s haptics perception and proprioception, potentially avoiding a haptics perception uncanny valley and increasing the redirection detection thresholds. Two haptics redirection methods are proposed: the DriftingHand method, which alters the position of the user’s virtual hand, and the Vari-Stick method, which alters the length of the virtual stick. Detection thresholds were measured in a user study (N = 60) by testing the two methods for a range of offsets between the virtual and the real object, for multiple stick lengths, and multiple distances from the user to the real object. Overall, the study reveals that VariStick and DriftingHand provide an undetectable range of offsets of [-20cm, +13cm] and [-11cm, +11cm], respectively.

This article presents an occlusion management approach that handles fine-grain occlusions, and that quantifies and localizes occlusions as a user explores a virtual environment (VE). Fine-grain occlusions are handled by finding the VE region where they occur, and by constructing a multiperspective visualization that lets the user explore the region from the current location, with intuitive head motions, without first having to walk to the region. VE geometry close to the user is rendered conventionally, from the user's viewpoint, to anchor the user, avoiding disorientation and simulator sickness. Given a viewpoint, residual occlusions are quantified and localized as VE voxels that cannot be seen from the given viewpoint but that can be seen from nearby viewpoints. This residual occlusion quantification and localization helps the user ascertain that a VE region has been explored exhaustively. The occlusion management approach was tested in three controlled studies, which confirmed the exploration efficiency benefit of the approach, and in perceptual experiments, which confirmed that exploration efficiency does not come at the cost of reducing spatial awareness and sense of presence, or of increasing simulator sickness.

We propose CoRE, a 360° video streaming approach that reduces bandwidth requirements compared to transferring the entire 360° video. CoRE uses non-linear sampling in both the spatial and temporal domains to achieve robustness to view direction prediction error and to transient wireless network bandwidth fluctuation. Each CoRE frame samples the environment in all directions, with full resolution over the predicted field of view and gradually decreasing resolution at the periphery, so that missing pixels are avoided, irrespective of the view prediction error magnitude. A CoRE video chunk has a main part at full frame rate, and an extension part at a gradually decreasing frame rate, which avoids stalls while waiting for a delayed transfer. We evaluate a prototype implementation of CoRE through trace-based experiments and a user study, and find that, compared to tiling with low-resolution padding, CoRE reduces data transfer amounts, stalls, and H.264 decoding overhead, increases frame rates, and eliminates missing pixels.

VTubers are live streamers who embody computer animation virtual avatars. VTubing is a rapidly rising form of online entertainment in East Asia, most notably in Japan and China, and it has been more recently introduced in the West. However, animating an expressive VTuber avatar remains a challenge due to budget and usability limitations of current solutions, i.e., high-fidelity motion capture is expensive, while keyboard-based VTubing interfaces impose a cognitive burden on the streamer. This paper proposes a novel approach for VTubing animation based on the key principle of loosening the coupling between the VTuber and their avatar, and it describes a first implementation of the approach in the AlterEcho VTubing animation system. AlterEcho generates expressive VTuber avatar animation automatically, without the streamer’s explicit intervention; it breaks the strict tethering of the avatar from the streamer, allowing the avatar’s nonverbal behavior to deviate from that of the streamer. Without the complete independence of a true alter ego, but also without the constraint of mirroring the streamer with the fidelity of an echo, AlterEcho produces avatar animations that have been rated significantly higher by VTubers and viewers (N = 315) compared to animations created using simple motion capture, or using VMagicMirror, a state-of-the-art keyboard-based VTubing system. Our work also opens the door to personalizing the avatar persona for individual viewers.

We introduce the disocclusion headlight, a method for VR selection assistance based on alleviating occlusions at the center of the user's field of view. The user's visualization of the VE is modified to reduce overlap between objects. This way, selection candidate objects have larger image footprints, which facilitates selection. The modification is confined to the center of the frame, with continuity to the periphery of the frame which is rendered conventionally. The selection assistance is provided automatically, without any interaction from the user. Furthermore, our method disoccludes without destroying the local spatial relationships between selection candidates, which allows solving complex selection queries based on the relative position of objects. We have tested our method on three selection tasks, where we compared it to two state-of-the-art VR selection techniques, i.e., the alpha cursor and the flower cone. Our method showed significant advantages in terms of shorter task completion times, and of fewer selection errors.

A common approach for Augmented Reality labeling is to display the label text on a flag planted into the real world element at a 3D anchor point. When there are more than just a few labels, the efficiency of the interface decreases as the user has to search for a given label sequentially. The search can be accelerated by sorting the labels alphabetically, but sorting all labels results in long and intersecting leader lines from the anchor points to the labels. This paper proposes a partially-sorted concentric label layout that leverages the search efficiency of sorting while avoiding the label display problems of long or intersecting leader lines. The labels are partitioned into a small number of sorted sequences displayed on circles of increasing radii. Since the labels on a circle are sorted, the user can quickly search each circle. A tight upper bound derived from circular permutation theory limits the number of circles and thereby the complexity of the label layout. For example, 12 labels require at most three circles. When the application allows it, the labels are presorted to further reduce the number of circles in the layout. The layout was tested in a user study where it significantly reduced the label searching time compared to a conventional single-circle layout.

Aligning hand-held objects into mid-air positions and orientations is important for many applications. Task performance depends on speed and accuracy, and also on minimizing the user's physical exertion. Augmented reality head-mounted displays (AR HMDs) can guide users during mid-air alignments by tracking an object's pose and delivering visual instruction directly into the user's field of view (FoV). However, it is unclear which AR HMD interfaces are most effective for mid-air alignment guidance, and how the form factor of current AR HMD hardware (such as heaviness and low FoV) affects how users put themselves into tiring body poses during mid-air alignment. We defined a set of design requirements for mid-air alignment interfaces that target reduction of high-exertion body poses during alignment. We then designed, implemented, and tested several interfaces in a user study in which novice participants performed a sequence of mid-air alignments using each interface.Results show that interfaces that rely on visual guidance located near the hand-held object reduce acquisition times and translation errors, while interfaces that involve aiming at a faraway virtual object reduce rotation errors. Users tend to avoid focus shifts and to position the head and arms to maximize how much AR visualization is contained within a single FoV without moving the head. We found that changing the size of visual elements affected how far out the user extends the arm, which affects torque forces. We also found that dynamically adjusting where visual guidance is placed relative to the mid-air pose can help keep the head level during alignment, which is important for distributing the weight of the AR HMD.

In a co-located multi-user collaborative virtual reality (VR) application a collaborator should be able to indicate a workspace location to the user, such that they can refer to it simultaneously as they work together. Due to their different viewpoints, the collaborator sees some parts of the virtual environment (VE) that the user does not, and communication breaks down when the collaborator's reference is not visible to the user. The conventional solutions of asking the user to move around to gain line of sight to the collaborator's reference, or of asking the user to toggle back and forth between their view and that of the collaborator can be inefficient and ineffective. This paper proposes a method for improving collaboration in VR by alleviating the disparity between the user and the collaborator views of the VE. The user is shown a multiperspective visualization of the VE that transitions smoothly from the user to the collaborator's perspective. The multiperpsective visualization is based on the switch camera, a novel camera model with curved rays that splice together the user and collaborator views. The multiperspective visualization is computed first by warping the VE geometry, through projection with the switch camera followed by unprojection with a conventional camera, and then by rendering the warped VE conventionally, for each user eye. A controlled user study with three tasks shows that VR collaboration using the switch camera multiperpsective visualization is faster, more reliable, and less taxing on the user than the conventional approaches of viewpoint translation or view toggling.

Path redirection for virtual reality (VR) navigation allows the user to explore a large virtual environment (VE) while the VR application is hosted in a limited physical space. Static mapping redirection methods deform the virtual scene to fit the physical space. The challenge is to deform the virtual scene in a reasonable way, making the distortions friendly to the user’s visual perception. In this paper we propose a feature-guided path redirection method that finds and takes into account the visual features of 3D virtual scenes. In a first offline step, a collection of view-independent and view-dependent visual features of the VE are extracted and stored in a visual feature map. Then, in a second offline step, the navigation path is deformed to fit in the confines of the available physical space through a mass-spring system optimization, according to distortion sensitive factors derived from the visual feature map. Finally, a novel detail preserving rendering algorithm is employed to preserve the original visual detail as the user navigates the VE on the redirected path. We tested our method on several scenes, where our method showed a reduced VE 3D mesh distortion, when compared to the path redirection methods without feature guidance.

Augmented Reality (AR) benefits telementoring by enhancing the communication between the mentee and the remote mentor with mentor authored graphical annotations that are directly integrated into the mentee’s view of the workspace. An important problem is conveying the workspace to the mentor effectively, such that they can provide adequate guidance. AR headsets now incorporate a frontfacing video camera, which can be used to acquire the workspace. However, simply providing to the mentor this video acquired from the mentee’s first-person view is inadequate. As the mentee moves their head, the mentor’s visualization of the workspace changes frequently, unexpectedly, and substantially. This paper presents a method for robust high-level stabilization of a mentee first-person video to provide effective workspace visualization to a remote mentor. The visualization is stable, complete, up to date, continuous, distortion free, and rendered from the mentee’s typical viewpoint, as needed to best inform the mentor of the current state of the workspace. In one study, the stabilized visualization had significant advantages over unstabilized visualization, in the context of three number matching tasks. In a second study, stabilization showed good results, in the context of surgical telementoring, specifically for cricothyroidotomy training in austere settings.

CoRE is an approach for streaming 360° videos based on a non-linear sampling of the equirectangular video cube. CoRE is robust to view prediction errors.

Occlusions disrupt the visualization of an object of interest, or target, in a real world scene. Video inpainting removes occlusions from a video stream by cutting out occluders and filling in with a plausible visualization of the object, but the approach is too slow for real-time performance. In this paper, we present a method for realtime occlusion removal in the visualization of a real world scene that is captured with an RGBD stream. Our pipeline segments the current RGBD frame to find the target and the occluders, searches for the best matching disoccluded view of the target in an earlier frame, computes a mapping between the target in the current frame and the target in the best matching frame, inpaints the missing pixels of the target in the current frame by resampling from the earlier frame, and visualizes the disoccluded target in the current frame. We demonstrate our method in the case of a walking human occluded by stationary or walking humans. Our method does not rely on a known 2D or 3D model of the target or of the occluders, and therefore it generalizes to other shapes. Our method runs at an interactive frame rate of 30fps.

This poster presents the design, implementation, and evaluation of a method for robust high-level stabilization of mentees first-person video in augmented reality (AR) telementoring. This video is captured by the front-facing built-in camera of an AR headset and stabilized by rendering from a stationary view a planar proxy of the workspace projectively texture mapped with the video feed. The result is stable, complete, up to date, continuous, distortion free, and rendered from the mentee's default viewpoint. The stabilization method was evaluated in two user studies, in the context of number matching and for cricothyroidotomy training, respectively. Both showed a significant advantage of our method compared with unstabilized visualization.

VR applications rely on the user's ability to explore the virtual scene efficiently. In complex scenes, occlusions limit what the user can see from a given location, and the user has to navigate the viewpoint around occluders to gain line of sight to the hidden parts of the scene. When the disoccluded regions prove to be of no interest, the user has to retrace their path, making scene exploration inefficient. Furthermore, the user might not be able to assume a viewpoint that would reveal the occluded regions due to physical limitations, such as obstacles in the real world hosting the VR application, viewpoints beyond the tracked area, or viewpoints above the user's head that cannot be reached by walking. Several occlusion management methods have been proposed in visualization research, such as top view, X-ray, and multiperspective visualization, which help the user see more from the current position, having the potential to improve the exploration efficiency of complex scenes. This paper reports on a study that investigates the potential of these three occlusion management methods in the context of VR applications, compared to conventional navigation. Participants were required to explore two virtual scenes to purchase five items in a virtual Supermarket, and to find three people in a virtual parking garage. The task performance metrics were task completion time, total distance traveled, and total head rotation. The study also measured user spatial awareness, depth perception, and simulator sickness. The results indicate that users benefit from top view visualization which helps them learn the scene layout and helps them understand their position within the scene, but the top view does not let the user find targets easily due to occlusions in the vertical direction, and due to the small image footprint of the targets. The X-ray visualization method worked better in the garage scene, a scene with a few big occluders and a low occlusion depth complexity' and less well in the Supermarket scene, a scene with many small occluders that create high occlusion depth complexity. The multi-perspective visualization method achieves better performance than the top view method and the X-ray method, in both scenes. There are no significant differences between the three methods and the conventional method in terms of spatial awareness, depth perception, and simulator sickness.

Photogrammetry is a popular method of 3D reconstruction that uses conventional photos as input. This method can achieve high quality reconstructions so long as the scene is densely acquired from multiple views with sufficient overlap between nearby images. However, it is challenging for a human operator to know during acquisition if sufficient coverage has been achieved. Insufficient coverage of the scene can result in holes, missing regions, or even a complete failure of reconstruction. These errors require manually repairing the model or returning to the scene to acquire additional views, which is time-consuming and often infeasible. We present a novel approach to photogrammetric acquisition that uses an AR HMD to predict a set of covering views and to interactively guide an operator to capture imagery from each view. The operator wears an AR HMD and uses a handheld camera rig that is tracked relative to the AR HMD with a fiducial marker. The AR HMD tracks its pose relative to the environment and automatically generates a coarse geometric model of the scene, which our approach analyzes at runtime to generate a set of human-reachable acquisition views covering the scene with consistent camera-to-scene distance and image overlap. The generated view locations are rendered to the operator on the AR HMD. Interactive visual feedback informs the operator how to align the camera to assume each suggested pose. When the camera is in range, an image is automatically captured. In this way, a set of images suitable for 3D reconstruction can be captured in a matter of minutes. In a user study, participants who were novices at photogrammetry were tasked with acquiring a challenging and complex scene either without guidance or with our AR HMD based guidance. Participants using our guidance achieved improved reconstructions without cases of reconstruction failure as in the control condition. Our AR HMD based approach is self-contained, portable, and provides specific acquisition guidance tailored to the geometry of the scene being captured.

We present a method for the fast computation of the intersection between a ray and the geometry of a scene. The scene geometry is simplified with a 2D array of voxelizations computed from different directions, sampling the space of all possible directions. The 2D array of voxelizations is compressed using a vector quantization approach. The ray-scene intersection is approximated using the voxelization whose rows are most closely aligned with the ray. The voxelization row that contains the ray is looked up, the row is truncated to the extent of the ray using bit operations, and a truncated row with non-zero bits indicates that the ray intersects the scene. We support dynamic scenes with rigidly moving objects by building a separate 2D array of voxelizations for each type of object, and by using the same 2D array of voxelizations for all instances of an object type. We support complex dynamic scenes and scenes with deforming geometry by computing and rotating a single voxelization on the fly. We demonstrate the benefits of our method in the context of interactive rendering of scenes with thousands of moving lights, where we compare our method to ray tracing, to conventional shadow mapping, and to imperfect shadow maps.

Virtual Reality (VR) applications allow a user to explore a scene intuitively through a tracked head-mounted display (HMD). However, in complex scenes, occlusions make scene exploration inefficient, as the user has to navigate around occluders to gain line of sight to potential regions of interest. When a scene region proves to be of no interest, the user has to retrace their path, and such a sequential scene exploration implies significant amounts of wasted navigation. Furthermore, as the virtual world is typically much larger than the tracked physical space hosting the VR application, the intuitive one-to-one mapping between the virtual and real space has to be temporarily suspended for the user to teleport or redirect in order to conform to the physical space constraints. In this paper we introduce a method for improving VR exploration efficiency by automatically constructing a multiperspective visualization that removes occlusions. For each frame, the scene is first rendered conventionally, the z-buffer is analyzed to detect horizontal and vertical depth discontinuities, the discontinuities are used to define disocclusion portals which are 3D scene rectangles for routing rays around occluders, and the disocclusion portals are used to render a multiperpsective image that alleviates occlusions. The user controls the multiperspective disocclusion effect, deploying and retracting it with small head translations. We have quantified the VR exploration efficiency brought by our occlusion removal method in a study where participants searched for a stationary target, and chased a dynamic target. Our method showed an advantage over conventional VR exploration in terms of reducing the navigation distance, the view direction rotation, the number of redirections, and the task completion time. These advantages did not come at the cost of a reduction in depth perception or situational awareness, or of an increase in simulator sickness.

We present a system that enables a novice user to acquire a large indoor scene in minutes as a collection of images that are sufficient for five degrees-of-freedom virtual navigation by image morphing. The user walks through the scene wearing an augmented reality head-mounted display (AR HMD) enhanced with a panoramic video camera. The AR HMD visualizes a 2D grid partitioning of a dynamically generated floor plan, which guides the user to acquire a panorama from each grid cell. The panoramas are registered offline using both AR HMD tracking data and structure-from - motion tools. Feature correspondences are established between neighboring panoramas. The resulting panoramas and correspondences support interactive rendering via image morphing with any view direction and from any viewpoint on the acquisition plane.

Immersive navigation in virtual reality (VR) and augmented reality (AR) leverages physical locomotion through pose tracking of the head-mounted display. While this navigation modality is intuitive, regions of interest in the scene may suffer from occlusion and require significant viewpoint translation. Moreover, limited physical space and user mobility need to be taken into consideration. Some regions of interest may require viewpoints that are physically unreachable without less intuitive methods such as walking in-place or redirected walking. We propose a novel approach for increasing navigation efficiency in VR and AR using multiperspective visualization. Our approach samples occluded regions of interest from additional perspectives, which are integrated seamlessly into the user's perspective. This approach improves navigation efficiency by bringing simultaneously into view multiple regions of interest, allowing the user to explore more while moving less. We have conducted a user study that shows that our method brings significant performance improvement in VR and AR environments, on tasks that include tracking, matching, searching, and ambushing objects of interest.

Occlusions are a severe bottleneck for the visualization of large and complex datasets. Conventional images only show dataset elements to which there is a direct line of sight, which significantly limits the information bandwidth of the visualization. Multiperspective visualization is a powerful approach for alleviating occlusions to show more than what is visible from a single viewpoint. However, constructing and rendering multiperspective visualizations is challenging. We present a framework for designing multiperspective focus+context visualizations with great flexibility by manipulating the underlying camera model. The focus region viewpoint is adapted to alleviate occlusions. The framework supports multiperspective visualization in three scenarios. In a first scenario, the viewpoint is altered independently for individual image regions to avoid occlusions. In a second scenario, conventional input images are connected into a multiperspective image. In a third scenario, one or several data subsets of interest (i.e., targets) are visualized where they would be seen in the absence of occluders, as the user navigates or the targets move. The multiperspective images are rendered at interactive rates, leveraging the camera model's fast projection operation. We demonstrate the framework on terrain, urban, and molecular biology geometric datasets, as well as on volume rendered density datasets.

Abstract We present a soft shadow rendering algorithm that is general, efficient and accurate. The algorithm supports fully dynamic scenes, with moving and deforming blockers and receivers, and with changing area light source parameters. For each output image pixel, the algorithm computes a tight but conservative approximation of the set of triangles that block the light source as seen from the pixel sample. The set of potentially blocking triangles allows estimating visibility between light points and pixel samples accurately and efficiently. As the light source size decreases to a point, our algorithm converges to rendering pixel accurate hard shadows.

Remote visualization has become both a necessity, as data set sizes have grown faster than computer network performance, and an opportunity, as laptop, tablet, and smartphone mobile computing platforms have become ubiquitous. However, the conventional remote visualization (CRV) approach of sending a new image from the server to the client for every view parameter change suffers from reduced interactivity. One problem is high latency, as the network has to be traversed twice, once to communicate the view parameters to the server and once to transmit the new image to the client. A second problem is reduced image quality due to aggressive compression or low resolution. We address these problems by constructing and transmitting enhanced images that are sufficient for quality output frame reconstruction at the client for a range of view parameter values. The client reconstructs thousands of frames locally, without any additional data from the server, which avoids latency and aggressive compression. We introduce animated depth images, which not only store a color and depth sample at every pixel, but also store the trajectory of the samples for a given time interval. Sample trajectories are stored compactly by partitioning the image into semi-rigid sample clusters and by storing one sequence of rigid body transformations per cluster. Animated depth images leverage sample trajectory coherence to achieve a good compression of animation data, with a small and user-controllable approximation error. We demonstrate animated depth images in the context of finite element analysis and SPH data sets.

The feed-forward pipeline based on projection followed by rasterization handles the rays that leave the eye efficiently: these first-order rays are modeled with a simple camera that projects geometry to screen. Second-order rays however, as, for example, those resulting from specular reflections, are challenging for the feed-forward approach. We propose an extension of the feed-forward pipeline to handle second-order rays resulting from specular and glossy reflections. The coherence of second-order rays is leveraged through clustering, the geometry reflected by a cluster is approximated with a depth image, and the color samples captured by the second-order rays of a cluster are computed by intersection with the depth image. We achieve quality specular and glossy reflections at interactive rates in fully dynamic scenes.

Abstract Texture bombing is a texture synthesis approach that saves memory by stopping short of assembling the output texture from the arrangement of input texture patches; instead, the arrangement is used directly at run time to texture surfaces. However, several problems remain in need of better solutions. One problem is improving texture diversification. A second problem is that mipmapping cannot be used because texel data is not stored explicitly. The lack of an appropriate level‐of‐detail (LoD) scheme results in severe minification artefacts. We present a just‐in‐time texturing method that addresses these two problems. Texture diversification is achieved by modelling a texture patch as an umbrella, a versatile hybrid 3‐D geometry and texture structure with parameterized appearance. The LoD is adapted continuously with a hierarchical algorithm that acts directly on the arrangement map. Results show that our method can model and render the diversity present in nature with only small texture memory requirements.

We propose to aid the interactive visualization of time-varying spatial data sets by simplifying node position data over the entire simulation as opposed to over individual states. Our approach is based on two observations. The first observation is that the trajectory of some nodes can be approximated well without recording the position of the node for every state. The second observation is that there are groups of nodes whose motion from one state to the next can be approximated well with a single transformation. We present data set simplification techniques that take advantage of this node data redundancy. Our techniques are general, supporting many types of simulations, they achieve good compression factors, and they allow rigorous control of the maximum node position approximation error. We demonstrate our approach in the context of finite element analysis data, of liquid flow simulation data, and of fusion simulation data.

Most images used in visualization are computed with the planar pinhole camera. This classic camera model has important advantages such as simplicity, which enables efficient software and hardware implementations, and similarity to the human eye, which yields images familiar to the user. However, the planar pinhole camera has only a single viewpoint, which limits images to parts of the scene to which there is direct line of sight. In this paper we introduce the curved ray camera to address the single viewpoint limitation. Rays are C1-continuous curves that bend to circumvent occluders. Our camera is designed to provide a fast 3-D point projection operation, which enables interactive visualization. The camera supports both 3-D surface and volume datasets. The camera is a powerful tool that enables seamless integration of multiple perspectives for overcoming occlusions in visualization while minimizing distortions.

We introduce the general pinhole camera (GPC), defined by a center of projection (i.e., the pinhole), an image plane, and a set of sampling locations in the image plane. We demonstrate the advantages of the GPC in the contexts of remote visualization, focus-plus-context visualization, and extreme antialiasing, which benefit from the GPC sampling flexibility. For remote visualization, we describe a GPC that allows zooming-in at the client without the need for transferring additional data from the server. For focus-plus-context visualization, we describe a GPC with multiple regions of interest with sampling rate continuity to the surrounding areas. For extreme antialiasing, we describe a GPC variant that allows supersampling locally with a very high number of color samples per output pixel (e.g., 1,024{\times}), supersampling levels that are out of reach for conventional approaches that supersample the entire image. The GPC supports many types of data, including surface geometry, volumetric, and image data, as well as many rendering modes, including highly view-dependent effects such as volume rendering. Finally, GPC visualization is efficient-GPC images are rendered and resampled with the help of graphics hardware at interactive rates.

A conventional pinhole camera captures only a small fraction of a 3-D scene due to occlusions. We introduce the graph camera, a non-pinhole with rays that circumvent occluders to create a single layer image that shows simultaneously several regions of interest in a 3-D scene. The graph camera image exhibits good continuity and little redundancy. The graph camera model is literally a graph of tens of planar pinhole cameras. A fast projection operation allows rendering in feed-forward fashion, at interactive rates, which provides support for dynamic scenes. The graph camera is an infrastructure level tool with many applications. We explore the graph camera benefits in the contexts of virtual 3-D scene exploration and summarization, and in the context of real-world 3-D scene visualization. The graph camera allows integrating multiple video feeds seamlessly, which enables monitoring complex real-world spaces with a single image.

A depth image constructed with a pinhole camera suffers from disocclusion errors: even a minimal viewpoint translation exposes samples not visible from the original viewpoint. The conventional solution to employ additional depth images is inefficient. A recent approach is to render the depth image with an occlusion camera, a non-pinhole that also gathers samples not seen from the reference viewpoint but needed for nearby viewpoints.

In this application paper, we describe the efforts of a multidisciplinary team towards producing a visualization of the September 11 Attack on the North Tower of New York's World Trade Center. The visualization was designed to meet two requirements. First, the visualization had to depict the impact with high fidelity, by closely following the laws of physics. Second, the visualization had to be eloquent to a nonexpert user. This was achieved by first designing and computing a finite-element analysis (FEA) simulation of the impact between the aircraft and the top 20 stories of the building, and then by visualizing the FEA results with a state-of-the-art commercial animation system. The visualization was enabled by an automatic translator that converts the simulation data into an animation system 3D scene. We built upon a previously developed translator. The translator was substantially extended to enable and control visualization of fire and of disintegrating elements, to better scale with the number of nodes and number of states, to handle beam elements with complex profiles, and to handle smoothed particle hydrodynamics liquid representation. The resulting translator is a powerful automatic and scalable tool for high-quality visualization of FEA results.

A fundamental challenge for existing shadow map based algorithms is dealing with partially illuminated surfaces. A conventional shadow map built with a pinhole camera only determines a binary light visibility at each point, and this all-or-nothing approach to visibility does not capture penumbral regions. We present an interactive soft shadow algorithm based on a variant of the depth discontinuity occlusion camera, a non-pinhole camera with rays that reach around blockers to sample normally hidden surfaces. Our soft shadow occlusion camera (SSOC) classifies a fragment on a continuum from fully visible to fully hidden, as seen from the light. The SSOC is used directly in fragment illumination computation without building an explicit "soft shadow map." This method renders plausible soft shadows at interactive speeds under fully dynamic conditions.

In this paper we report on our experience of teaching a multi-expertise application-driven course in which upper-level undergraduate and entry-level graduate students from computer science, computer graphics technology, and educational science worked together to design, develop and evaluate a distance learning system. We describe the activities pursued in the class, the interactions between the various groups of students, as well as the challenges and advantages ensuing from the great variety of student backgrounds. The success of the course is measured in two ways. First, the class achieved the goal of building a distance learning system that surpasses the state of the art by improving the integration of the remote students into regular on-campus learning. Second, a survey conducted at the end of the course indicates that the application-driven, collaborative, and multi-expertise structure of the class provided a uniquely effective learning experience.

Simulations that model in detail complex interactions in complex environments are now possible, and visualization is a uniquely powerful analysis tool. However, visualization features of simulation codes are typically intended for scientists designing simulations, providing little support for presenting simulation results in a form suitable for non-experts. Conversely, graphics software and hardware progress has been fueled by applications where precisely abiding by the laws of physics is of secondary importance compared to visual realism.This half-day tutorial presents an approach for state-of-the-art visualization of simulation data based on connecting the worlds of computer simulation and computer animation. Concretely, the attendees will learn how to simplify, import, and integrate the simulation data into the surrounding scene, with examples from our simulation of the September 11 Attack on the Pentagon. The resulting realistic visualization enables effective dissemination of simulation results, and helps simulations reach their full potential for high societal impact.

Rendering a scene using a single depth image suffers from disocclusion errors as the view translates away from the reference view. We present the depth discontinuity occlusion camera (DDOC), a non-pinhole camera that samples surfaces which are hidden in the reference view, but are likely to become visible in nearby views. The DDOC reference image alleviates disocclusion errors; since it has a single layer, it maintains the advantages of regular depth images such as bounded number of samples, efficient incremental processing, and implicit connectivity. The DDOC is defined by the reference view and the geometry it encompasses. The reference planar pinhole camera is 3D distorted at depth discontinuities. The distortion is fine-grain controlled with a distortion map, which allows handling complex scenes. The DDOC provides fast projection so the reference image is constructed efficiently with the feed-forward graphics pipeline.

Abstract We describe a system for the rapid acquisition of building interiors. In 40 hours, a two member team with a single acquisition device captured a model of the corridors and 20 individual rooms spanning 6 floors of a large building. Our custom acquisition device operates at interactive rates. The system provides immediate feedback to the operator. The operator guides the acquisition device in realtime and trivially avoids over sampling the planar parts of the scene such as floors, ceilings, walls, or doors. Most of the acquisition time budget is spent on the parts of the scene with complex geometry. A corridor section is modeled by acquiring a depth enhanced panorama (DEP) a teach one of its two ends and by fitting proxy geometry to the two DEPs. A room is acquired with a single DEP and proxy geometry is fitted to the planar parts. A room or a corridor section is acquired in less than 15 minutes. The acquisition device acquires high quality color intrinsically registered with the depth data. The resulting model is a texture mapped triangle mesh that supports photo realistic interactive rendering an dissuitable for applications such as virtual training and simulation. Categories and Subject Descriptors (ACM CCS): I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling.

Abstract We present a technique for rendering reflections on complex reflectors at interactive rates based on approximating the geometry of the reflected scene with impostors. The reflections correctly convey the distance to the reflector surface and provide motion parallax. Two types of impostors are adapted to the reflections framework: billboards and depth maps. Billboards remove most of the problems of environment mapped reflections at only a small additional cost. Second order reflections are supported by introducing reflective billboards. Higher quality reflections that provide motion parallax within a reflected object are obtained by approximating the reflective geometry with depth maps. The computation of the intersection between a reflected ray and a depth map is accelerated by leveraging epipolar constraints. Like environment mapping, our technique does not pose any restriction on the geometry of the reflector, supports dynamic scenes, and runs at interactive rates with the help of graphics hardware. Categories and Subject Descriptors (ACM CCS): I.3.3. [Computer Graphics]—Three‐Dimensional Graphics and Realism.

We describe forward rasterization, a class of rendering algorithms designed for small polygonal primitives. The primitive is efficiently rasterized by interpolation between its vertices. The interpolation factors are chosen to guarantee that each pixel covered by the primitive receives at least one sample which avoids holes. The location of the samples is recorded with subpixel accuracy using a pair of offsets which are then used to reconstruct/resample the output image. Offset reconstruction has good static and temporal antialiasing properties. We present two forward rasterization algorithms, one that renders quadrilaterals and is suitable for scenes modeled with depth images like in image-based rendering by 3D warping, and one that renders triangles and is suitable for scenes modeled conventionally. When compared to conventional rasterization, forward rasterization is more efficient for small primitives and has better temporal antialiasing properties.

This paper presents sample-based cameras for rendering high quality reflections on convex reflectors at interactive rates. The method supports change of view, moving objects and reflectors, higher order reflections, view-dependent lighting of reflected objects, and reflector surface properties. In order to render reflections with the feed forward graphics pipeline, one has to project reflected vertices. A sample-based camera is a collection of BSP trees of pinhole cameras that jointly approximate the projection function. It is constructed from the reflected rays defined by the desired view and the scene reflectors. A scene point is projected by invoking only the cameras that contain it in their frustums. Reflections are rendered by projecting the scene geometry and then rasterizing in hardware.

We introduce the occlusion camera: a non-pinhole camera with 3D distorted rays. Some of the rays sample surfaces that are occluded in the reference view, while the rest sample visible surfaces. The extra samples alleviate disocclusion errors. The silhouette curves are pushed back, so nearly visible samples become visible. A single occlusion camera covers the entire silhouette of an object, whereas many depth images are required to achieve the same effect. Like regular depth images, occlusion-camera images have a single layer thus the number of samples they contain is bounded by the image resolution, and connectivity is defined implicitly. We construct and use occlusion-camera images in hardware. An occlusion-camera image does not guarantee that all disocclusion errors are avoided. Objects with complex geometry are rendered using the union of the samples stored by a planar pinhole camera and an occlusion camera depth image. Categories and Subject Descriptors (according to ACM CCS): I.3.3. [Computer Graphics]—Three-Dimensional Graphics and Realism.

We describe a method for interactive modeling and visualization of room-size indoor scenes that is fast, easy, and inexpensive. We have designed a novel data acquisition device, called a ModelCamera, that consists of a video camera with an attached laser system that generates a 7x7 pattern of depth samples. The operator pans and tilts the ModelCamera around the camera's center of projection and it acquires a sequence of color and depth frames. The frames are registered in world coordinates and are merged into an evolving scene model, called a depth enhanced panorama. The model is visualized continually; the immediate feedback allows the operator to monitor the model quality and to identify missing scene surfaces. Our approach extends color panoramas to support viewpoint translation, while retaining their speed, convenience, and low cost (Figure 1). It is a practical alternative to modeling systems that provide complete models at a high cost.

This paper describes the work of a team of researchers in computer graphics, geometric computing, and civil engineering to produce a visualization of the September 2001 attack on the Pentagon. The immediate motivation for the project was to understand the behavior of the building under the impact. The longer term motivation was to establish a path for producing high-quality visualizations of large scale simulations. The first challenge was managing the enormous complexity of the scene to fit within the limits of state-of-the art simulation software systems and supercomputing resources. The second challenge was to integrate the simulation results into a high-quality visualization. To meet this challenge, we implemented a custom importer that simplifies and loads the massive simulation data in a commercial animation system. The surrounding scene is modeled using image-based techniques and is also imported in the animation system where the visualization is produced. A specific issue for us was to federate the simulation and the animation systems, both commercial systems not under our control and following internally different conceptualizations of geometry and animation. This had to be done such that scalability was achieved. The reusable link created between the two systems allows communicating the results to non-specialists and the public at large, as well as facilitating communication in teams with members having diverse technical backgrounds.

Article The vacuum buffer Share on Authors: Voicu Popescu Univ. of North Carolina, Chapel Hill Univ. of North Carolina, Chapel HillView Profile , Anselmo Lastra Univ. of North Carolina, Chapel Hill Univ. of North Carolina, Chapel HillView Profile Authors Info & Claims I3D '01: Proceedings of the 2001 symposium on Interactive 3D graphicsMarch 2001 Pages 73–76https://doi.org/10.1145/364338.364367Online:01 March 2001Publication History 5citation285DownloadsMetricsTotal Citations5Total Downloads285Last 12 Months1Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

We present the WarpEngine, an architecture designed for real-time imaged-based rendering of natural scenes from arbitrary viewpoints. The modeling primitives are real-world images with per-pixel depth. Currently they are acquired and stored off-line; in the near future real-time depth-image acquisition will be possible, the WarpEngine is designed to render in immediate mode from such data sources.The depth-image resolution is locally adapted by interpolation to match the resolution of the output image. 3D warping can occur either before or after the interpolation; the resulting warped/interpolated samples are forward-mapped into a warp buffer, with the precise locations recorded using an offset. Warping processors are integrated on-chip with the warp buffer, allowing efficient, scalable implementation of very high performance systems. Each chip will be able to process 100 million samples per second and provide 4.8GigaBytes per second of bandwidth to the warp buffer.The WarpEngine is significantly less complex than our previous efforts, incorporating only a single ASIC design. Small configurations can be packaged as a PC add-in card, while larger deskside configurations will provide HDTV resolutions at 50 Hz, enabling radical new applications such as 3D television.WarpEngine will be highly programmable, facilitating use as a test-bed for experimental IBR algorithms.

This paper presents efficient image-based rendering techniques used in the context of an architectural walkthrough system. Portals (doors and windows) are rendered by warping layered depth images (LDIs). In a preprocessing phase, for every portal, a number of pre-rendered images are combined into an LDI. The resulting LDI stores, exactly once, all surfaces visible in at least one of the images used in the construction, so most of the exposure errors are efficiently eliminated. The LDI can be warped in the McMillan occlusion compatible ordering. A substantial increase in performance is obtained by warping in parallel. Our parallelization scheme achieves good load balancing, scales with the number of processors, and preserves the occlusion compatible ordering. A fast, conservative reference-image-space clipping algorithm also reduces the warping effort.