Large Language Models (LLMs) show promising learning and reasoning abilities. Compared to other NLP tasks, multilingual and multi-label emotion evaluation tasks are under-explored in LLMs. In this paper, we present EthioEmo, a multi-label emotion classification dataset for four Ethiopian languages, namely, Amharic (amh), Afan Oromo (orm), Somali (som), and Tigrinya (tir). We perform extensive experiments with an additional English multi-label emotion dataset from SemEval 2018 Task 1. Our evaluation includes encoder-only, encoder-decoder, and decoder-only language models. We compare zero and few-shot approaches of LLMs to fine-tuning smaller language models. The results show that accurate multi-label emotion classification is still insufficient even for high-resource languages such as English, and there is a large gap between the performance of high-resource and low-resource languages. The results also show varying performance levels depending on the language and model type. EthioEmo is available publicly to further improve the understanding of emotions in language models and how people convey emotions through various languages.

Image processing workflows typically consist of a series of different functions, each working with “image” inputs and outputs in an abstract sense. However, the specific in-memory representation of images differs between and sometimes within libraries. Conversion is therefore necessary when integrating functions from several sources into a single program. The conversion process forces users to consider low-level implementation details, including data types, color channels, channel order, minibatch layout, memory locations, and pixel intensity ranges. Specifically in the case of visual programming languages (VPLs), this distracts from high- level operations. We introduce im2im, a Python library that automates the conversion of in-memory image representations. The central concept of this library is a knowledge graph that describes image representations and how to convert between them. The system queries this knowledge graph to generate the conversion code and execute it, converting an image to the desired representation. The effectiveness of the approach is evaluated through two case studies in VPLs. In each case, we compared a workflow created in a basic blockbased VPL with the same workflow enhanced using im2im. These evaluations show that im2im automates type conversions and eliminates the need for manual intervention. Additionally, we compared the overhead of using explicit intermediate representations versus im2im, both of which avoid manual type conversions in VPLs. The results indicate that im2im generates only the necessary conversions, avoiding the runtime overhead associated with converting to and from intermediate formats. A performance comparison between the step-by-step approach used by im2im and a single-function approach demonstrates that the overhead introduced by im2im does not impact practical usability. While focused on block-based VPLs, im2im can be generalized to other VPLs and textual programming environments. Its principles are also applicable to domains other than images. The source code and analyses are available via GitHub.

Multiple importance sampling (MIS) is a vital component of most rendering algorithms. MIS computes a weighted sum of samples from many different techniques to achieve generalization, that is, to handle a wide range of scene types and lighting effects. A key factor to the performance of MIS is the choice of weighting function. The go-to default - the balance heuristic - performs well in many cases, but prior work has shown that it can yield unsatisfactory results. A number of challenges cause this suboptimal performance, including low-variance techniques, sample correlation, and unknown sampling densities. Prior work has suggested improvements for some of these problems, but a general optimal solution has yet to be found. We propose a general and practical weight correction scheme: We optimize, on-the-fly, a set of correction factors that are multiplied into any baseline MIS heuristic (e.g., balance or power). We demonstrate that this approach yields consistently better equal-time performance on two rendering applications: bidirectional algorithms and resampled importance sampling for direct illumination.

Inverse rendering has emerged as a standard tool to reconstruct the parameters of appearance models from images (e.g., textured BSDFs). In this work, we present several novel contributions motivated by the practical challenges of recovering high-resolution surface appearance textures, including spatially-varying subsurface scattering parameters. First, we propose Laplacian mipmapping , which combines differentiable mipmapping and a Laplacian pyramid representation into an effective preconditioner. This seemingly simple technique significantly improves the quality of recovered surface textures on a set of challenging inverse rendering problems. Our method automatically adapts to the render and texture resolutions, only incurs moderate computational cost and achieves better quality than prior work while using fewer hyperparameters. Second, we introduce a specialized gradient computation algorithm for textured, path-traced subsurface scattering, which facilitates faithful reconstruction of translucent materials. By using path tracing, we enable the recovery of complex appearance while avoiding the approximations of the previously used diffusion dipole methods. Third, we demonstrate the application of both these techniques to reconstructing the textured appearance of human faces from sparse captures. Our method recovers high-quality relightable appearance parameters that are compatible with current production renderers.

Large language models (LLMs) have gained popularity recently due to their outstanding performance in various downstream Natural Language Processing (NLP) tasks. However, low-resource languages are still lagging behind current state-of-the-art (SOTA) developments in the field of NLP due to insufficient resources to train LLMs. Ethiopian languages exhibit remarkable linguistic diversity, encompassing a wide array of scripts, and are imbued with profound religious and cultural significance. This paper introduces EthioLLM – multilingual large language models for five Ethiopian languages (Amharic, Ge’ez, Afan Oromo, Somali, and Tigrinya) and English, and Ethiobenchmark – a new benchmark dataset for various downstream NLP tasks. We evaluate the performance of these models across five downstream NLP tasks. We open-source our multilingual language models, new benchmark datasets for various downstream tasks, and task-specific fine-tuned language models and discuss the performance of the models. Our dataset and models are available at the https://huggingface.co/EthioNLP repository.

Multiple importance sampling (MIS) is an indispensable tool in rendering that constructs robust sampling strategies by combining the respective strengths of individual distributions. Its efficiency can be greatly improved by carefully selecting the number of samples drawn from each distribution, but automating this process remains a challenging problem. Existing works are mostly limited to mixture sampling, in which only a single sample is drawn in total, and the works that do investigate multi-sample MIS only optimize the sample counts at a per-pixel level, which cannot account for variations beyond the first bounce. Recent work on Russian roulette and splitting has demonstrated how fixed-point schemes can be used to spatially vary sample counts to optimize image efficiency but is limited to choosing the same number of samples across all sampling strategies. Our work proposes a highly flexible sample allocation strategy that bridges the gap between these areas of work. We show how to iteratively optimize the sample counts to maximize the efficiency of the rendered image using a lightweight data structure, which allows us to make local and individual decisions per technique. We demonstrate the benefits of our approach in two applications, path guiding and bidirectional path tracing, in both of which we achieve consistent and substantial speedups over the respective previous state-of-the-art.

Distributed ray tracing algorithms are widely used when rendering massive scenes, where data utilization and load balancing are the keys to improving performance. One essential observation is that rays are temporally coherent, which indicates that temporal information can be used to improve computational efficiency. In this paper, we use temporal coherence to optimize the performance of distributed ray tracing. First, we propose a temporal coherence-based scheduling algorithm to guide the task/data assignment and scheduling. Then, we propose a virtual portal structure to predict the radiance of rays based on the previous frame, and send the rays with low radiance to a precomputed simplified model for further tracing, which can dramatically reduce the traversal complexity and the overhead of network data transmission. The approach was validated on scenes of sizes up to 355 GB. Our algorithm can achieve a speedup of up to 81% compared to previous algorithms, with a very small mean squared error.

Minimizing the attack surface of Federated Learning (FL) systems is a field of active research. FL turns out to be highly vulnerable to various threats coming from the edge of the network. Current approaches rely on robust aggregation, anomaly detection and generative models for defending against poisoning attacks. Yet, they either have limited defensive capabilities due to their underlying design or are impractical to use as they rely on constraining building blocks.We introduce FedGuard, a novel FL framework that utilizes the generative capabilities of Conditional Variational AutoEncoders (CVAE) to effectively defend against poisoning attacks with tuneable overhead in communication and computation. Whilst the idea of hardening a FL system using generative models is not entirely new, FedGuard’s original contribution is in its selective parameter aggregation operator with parameter selection being driven by synthetic validation data sampled from the CVAEs trained locally by each participating party.Experimental evaluations in a 100-client setup demonstrates FedGuard to be more effective than previous approaches against several types of attacks (label and sign flipping, additive noise, same value attacks). FedGuard successfully defends in scenarios with up to 50% malicious peers where other strategies fail. In addition, FedGuard does not require auxiliary datasets or centralized (pre-) training. It provides resilience against poisoning attacks from the very first round of federated training.

Concurrent queue algorithms have been subject to extensive research. However, the target hardware and evaluation methodology on which the published results for any two given concurrent queue algorithms are based often share only minimal overlap. A meaningful comparison is, thus, exceedingly difficult. With the continuing trend towards more and more heterogeneous systems, it is becoming more and more important to not only evaluate and compare novel and existing queue algorithms across a wider range of target architectures, but to also be able to continuously re-evaluate queue algorithms in light of novel architectures and capabilities.To address this need, we present AnyQ, an evaluation framework for concurrent queue algorithms. We design a set of programming abstractions that enable the mapping of concurrent queue algorithms and benchmarks to a wide variety of target architectures. We demonstrate the effectiveness of these abstractions by showing that a queue algorithm expressed in a portable, high-level manner can achieve performance comparable to hand-crafted implementations. We design a system for testing and benchmarking queue algorithms. Using the developed framework, we investigate concurrent queue algorithm performance across a range of both CPU as well as GPU architectures. In hopes that it may serve the community as a starting point for building a common repository of concurrent queue algorithms as well as a base for future research, all code and data is made available as open source software at https://anydsl.github.io/anyq.

Virtual character animation and movement synthesis have advanced rapidly during recent years, especially through a combination of extensive motion capture datasets and machine learning. A remaining challenge is interactively simulating characters that fatigue when performing extended motions, which is indispensable for the realism of generated animations. However, capturing such movements is problematic, as performing movements like backflips with fatigued variations up to exhaustion raises capture cost and risk of injury. Surprisingly, little research has been done on faithful fatigue modeling. To address this, we propose a deep reinforcement learning-based approach, which—for the first time in literature—generates control policies for full-body physically simulated agents aware of cumulative fatigue. For this, we first leverage Generative Adversarial Imitation Learning (GAIL) to learn an expert policy for the skill; Second, we learn a fatigue policy by limiting the generated constant torque bounds based on endurance time to non-linear, state- and time-dependent limits in the joint-actuation space using a Three-Compartment Controller (3CC) model. Our results demonstrate that agents can adapt to different fatigue and rest rates interactively, and discover realistic recovery strategies without the need for any captured data of fatigued movement.

Independently estimating pixel values in Monte Carlo rendering results in a perceptually sub-optimal white-noise distribution of error in image space. Recent works have shown that perceptual fidelity can be improved significantly by distributing pixel error as blue noise instead. Most such works have focused on static images, ignoring the temporal perceptual effects of animation display. We extend prior formulations to simultaneously consider the spatial and temporal domains, and perform an analysis to motivate a perceptually better spatio-temporal error distribution. We then propose a practical error optimization algorithm for spatio-temporal rendering and demonstrate its effectiveness in various configurations.

Abstract Can we make virtual characters in a scene interact with their surrounding objects through simple instructions? Is it possible to synthesize such motion plausibly with a diverse set of objects and instructions? Inspired by these questions, we present the first framework to synthesize the full‐body motion of virtual human characters performing specified actions with 3D objects placed within their reach. Our system takes textual instructions specifying the objects and the associated ‘intentions’ of the virtual characters as input and outputs diverse sequences of full‐body motions. This contrasts existing works, where full‐body action synthesis methods generally do not consider object interactions, and human‐object interaction methods focus mainly on synthesizing hand or finger movements for grasping objects. We accomplish our objective by designing an intent‐driven full‐body motion generator, which uses a pair of decoupled conditional variational auto‐regressors to learn the motion of the body parts in an autoregressive manner. We also optimize the 6‐DoF pose of the objects such that they plausibly fit within the hands of the synthesized characters. We compare our proposed method with the existing methods of motion synthesis and establish a new and stronger state‐of‐the‐art for the task of intent‐driven motion synthesis.

Memory efficiency is crucial in training deep learning networks on resource-restricted devices. During backpropagation, forward tensors are used to calculate gradients. Despite the option of keeping those dependencies in memory until they are reused in backpropagation, some forward tensors can be discarded and recomputed later from saved tensors, so-called checkpoints . This allows, in particular, for resource-constrained heterogeneous environments to make use of all available compute devices. Unfortunately, the definition of these checkpoints is a non-trivial problem and poses a challenge to the programmer—improper or excessive recomputations negate the benefit of checkpointing. In this article, we present XEngine, an approach that schedules network operators to heterogeneous devices in low memory environments by determining checkpoints and recomputations of tensors. Our approach selects suitable resources per timestep and operator and optimizes the end-to-end time for neural networks taking the memory limitation of each device into account. For this, we formulate a mixed-integer quadratic program (MIQP) to schedule operators of deep learning networks on heterogeneous systems. We compare our MIQP solver XEngine against Checkmate [ 12 ], a mixed-integer linear programming (MILP) approach that solves recomputation on a single device. Our solver finds solutions that are up to 22.5% faster than the fastest Checkmate schedule in which the network is computed exclusively on a single device. We also find valid schedules for networks making use of both central processing units and graphics processing units if memory limitations do not allow scheduling exclusively to the graphics processing unit.

Monte Carlo rendering makes heavy use of mixture sampling and multiple importance sampling (MIS). Previous work has shown that control variates can be used to make such mixtures more efficient and more robust. However, the existing approaches failed to yield practical applications, chiefly because their underlying theory is based on the unrealistic assumption that a single mixture is optimized for a single integral. This is in stark contrast with rendering reality, where millions of integrals are computed---one per pixel---and each is infinitely recursive. We adapt and extend the theory introduced by previous work to tackle the challenges of real-world rendering applications. We achieve robust mixture sampling and (approximately) optimal MIS weighting for common applications such as light selection, BSDF sampling, and path guiding.

We introduce a practical general-purpose neural appearance filtering pipeline for physically-based rendering. We tackle the previously difficult challenge of aggregating visibility across many levels of detail from local information only, without relying on learning visibility for the entire scene. The high adaptivity of neural representations allows us to retain geometric correlations along rays and thus avoid light leaks. Common approaches to prefiltering decompose the appearance of a scene into volumetric representations with physically-motivated parameters, where the inflexibility of the fitted models limits rendering accuracy. We avoid assumptions on particular types of geometry or materials, bypassing any special-case decompositions. Instead, we directly learn a compressed representation of the intra-voxel light transport. For such high-dimensional functions, neural networks have proven to be useful representations. To satisfy the opposing constraints of prefiltered appearance and correlation-preserving point-to-point visibility, we use two small independent networks on a sparse multi-level voxel grid. Each network requires 10--20 minutes of training to learn the appearance of an asset across levels of detail. Our method achieves 70--95% compression ratios and around 25% of quality improvements over previous work. We reach interactive to real-time framerates, depending on the level of detail.

Jupyter Notebook [1] is an open source, interactive computing platform widely used in the scientific computing and artificial intelligence community [2], [3], [4], [5]. The popularity of the platform is a consequence of the generated single notebook document combining source code, markdown, and visualizations (Fig. 1). This makes the platform ideal for tasks such as data analysis and scientific image processing, where repeatability and transparency of analysis tasks are just as important as functionality and performance. However, the obligatory use of code is an obstacle to acceptance of the platform in scientific communities where programming is not generally taught in the curriculum. Consequently, many experimental communities rely on manual image processing using graphical user interfaces [6], [7], [8]. The obvious disadvantages are the lack of repeatability, transparency, and precision in image processing and data analysis tasks. To solve these issues, we propose to extend Jupyter Notebook with visual programming cells. In each visual programming cell, users can create the program by assembling graphical nodes that represent computational instructions, and the textual program is automatically generated and executed by the environment. Cells will support version control aware serialization and deserialization. The core innovation of our proposed work lies in a change of workflow and the adaption of a jupyter-based workflow in experimental communities that have no culture of working with source code. The system can be adapted to multiple applications and domains by integrating new node types. We hereby present an early version of the system and provide one use case from microscopy image processing to demonstrate the integration of existing non-Python software.

Multiple importance sampling (MIS) is an indispensable tool in light-transport simulation. It enables robust Monte Carlo integration by combining samples from several techniques. However, it is well understood that such a combination is not always more efficient than using a single sampling technique. Thus a major criticism of complex combined estimators, such as bidirectional path tracing, is that they can be significantly less efficient on common scenes than simpler algorithms like forward path tracing. We propose a general method to improve MIS efficiency: By cheaply estimating the efficiencies of various technique and sample-count combinations, we can pick the best one. The key ingredient is a numerically robust and efficient scheme that uses the samples of one MIS combination to compute the efficiency of multiple other combinations. For example, we can run forward path tracing and use its samples to decide which subset of VCM to enable, and at what sampling rates. The sample count for each technique can be controlled per-pixel or globally. Applied to VCM, our approach enables robust rendering of complex scenes with caustics, without compromising efficiency on simpler scenes.

Russian roulette and splitting are widely used techniques to increase the efficiency of Monte Carlo estimators. But, despite their popularity, there is little work on how to best apply them. Most existing approaches rely on simple heuristics based on, e.g., surface albedo and roughness. Their efficiency often hinges on user-controlled parameters. We instead iteratively learn optimal Russian roulette and splitting factors during rendering, using a simple and lightweight data structure. Given perfect estimates of variance and cost, our fixed-point iteration provably converges to the optimal Russian roulette and splitting factors that maximize the rendering efficiency. In our application to unidirectional path tracing, we achieve consistent and significant speed-ups over the state of the art.

FPGAs have found their way into data centers as accelerator cards, making reconfigurable computing more accessible for high-performance applications. At the same time, new high-level synthesis compilers like Xilinx Vitis and runtime libraries such as XRT attract software programmers into the reconfigurable domain. While software programmers are familiar with task-level and data-parallel programming, FPGAs often require different types of parallelism. For example, data-driven parallelism is mandatory to obtain satisfactory hardware designs for pipelined dataflow architectures. However, software programmers are often not acquainted with dataflow architectures— resulting in poor hardware designs. In this work we present FLOWER, a comprehensive compiler infrastructure that provides automatic canonical transformations for high-level synthesis from a domain-specific library. This allows programmers to focus on algorithm implementations rather than low-level optimizations for dataflow architectures. We show that FLOWER allows to synthesize efficient implementations for high-performance streaming applications targeting System-on-Chip and FPGA accelerator cards, in the context of image processing and computer vision.

"How can we animate 3D-characters from a movie script or move robots by simply telling them what we would like them to do?" "How unstructured and complex can we make a sentence and still generate plausible movements from it?" These are questions that need to be answered in the long-run, as the field is still in its infancy. Inspired by these problems, we present a new technique for generating compositional actions, which handles complex input sentences. Our output is a 3D pose sequence depicting the actions in the input sentence. We propose a hierarchical two-stream sequential model to explore a finer joint-level mapping between natural language sentences and 3D pose sequences corresponding to the given motion. We learn two manifold representations of the motion, one each for the upper body and the lower body movements. Our model can generate plausible pose sequences for short sentences describing single actions as well as long complex sentences describing multiple sequential and compositional actions. We evaluate our proposed model on the publicly available KIT Motion-Language Dataset containing 3D pose data with human-annotated sentences. Experimental results show that our model advances the state-of-the-art on text-based motion synthesis in objective evaluations by a margin of 50%. Qualitative evaluations based on a user study indicate that our synthesized motions are perceived to be the closest to the ground-truth motion captures for both short and compositional sentences.

Abstract Combining diverse sampling techniques via multiple importance sampling (MIS) is key to achieving robustness in modern Monte Carlo light transport simulation. Many such methods additionally employ correlated path sampling to boost efficiency. Photon mapping, bidirectional path tracing, and path‐reuse algorithms construct sets of paths that share a common prefix. This correlation is ignored by classical MIS heuristics, which can result in poor technique combination and noisy images. We propose a practical and robust solution to that problem. Our idea is to incorporate correlation knowledge into the balance heuristic, based on known path densities that are already required for MIS. This correlation‐aware heuristic can achieve considerably lower error than the balance heuristic, while avoiding computational and memory overhead.

A common problem of mid-air interaction is excessive arm fatigue, known as the "Gorilla arm" effect. To predict and prevent such problems at a low cost, we investigate user testing of mid-air interaction without real users, utilizing biomechanically simulated AI agents trained using deep Reinforcement Learning (RL). We implement this in a pointing task and four experimental conditions, demonstrating that the simulated fatigue data matches human fatigue data. We also compare two effort models: 1) instantaneous joint torques commonly used in computer animation and robotics, and 2) the recent Three Compartment Controller (3CC-) model from biomechanical literature. 3CC- yields movements that are both more efficient and relaxed, whereas with instantaneous joint torques, the RL agent can easily generate movements that are quickly tiring or only reach the targets slowly and inaccurately. Our work demonstrates that deep RL combined with the 3CC- provides a viable tool for predicting both interaction movements and user experiencein silico, without users.

Multilingualism is a cultural cornerstone of Europe and firmly anchored in the European treaties including full language equality. However, language barriers impacting business, cross-lingual and cross-cultural communication are still omnipresent. Language Technologies (LTs) are a powerful means to break down these barriers. While the last decade has seen various initiatives that created a multitude of approaches and technologies tailored to Europe’s specific needs, there is still an immense level of fragmentation. At the same time, AI has become an increasingly important concept in the European Information and Communication Technology area. For a few years now, AI – including many opportunities, synergies but also misconceptions – has been overshadowing every other topic. We present an overview of the European LT landscape, describing funding programmes, activities, actions and challenges in the different countries with regard to LT, including the current state of play in industry and the LT market. We present a brief overview of the main LT-related activities on the EU level in the last ten years and develop strategic guidance with regard to four key dimensions.

Field programmable gate arrays (FPGAs) excel in low power and high throughput computations, but they are challenging to program. Traditionally, developers rely on hardware description languages, such as Verilog or VHDL to specify the hardware behavior at the register-transfer level. High-level synthesis (HLS) raises the level of abstraction but still requires FPGA design knowledge. Programmers usually write pragma-annotated C/C++ programs to define the hardware architecture of an application. However, each hardware vendor extends its own C dialect using its own vendor-specific set of pragmas. This prevents portability across different vendors. Furthermore, pragmas are not first-class citizens in the language. This makes it hard to use them in a modular way or design proper abstractions. In this article, we present AnyHLS, an approach to synthesize FPGA designs in a modular and abstract way. AnyHLS is able to raise the abstraction level of the existing HLS tools by resorting to programming language features such as types and higher order functions as follows. It relies on partial evaluation to specialize and to optimize the user application based on a library of abstractions. Then, vendor-specific HLS code is generated for Intel and Xilinx FPGAs. Portability is obtained by avoiding any vendor-specific pragmas at the source code. In order to validate achievable gains in productivity, a library for the domain of image processing is introduced as a case study, and its synthesis results are compared with several state-of-the-art domain-specific language (DSL) approaches for this domain.

Path guiding is a promising tool to improve the performance of path tracing algorithms. However, not much research has investigated what target densities a guiding method should strive to learn for optimal performance. Instead, most previous work pursues the zero-variance goal: The local decisions are guided under the assumption that all other decisions along the random walk will be sampled perfectly. In practice, however, many decisions are poorly guided, or not guided at all. Furthermore, learned distributions are often marginalized, e.g., by neglecting the BSDF. We present a generic procedure to derive theoretically optimal target densities for local path guiding. These densities account for variance in nested estimators, and marginalize provably well over, e.g., the BSDF. We apply our theory in two state-of-the-art rendering applications: a path guiding solution for unidirectional path tracing [Müller et al. 2017] and a guiding method for light source selection for the many lights problem [Vévoda et al. 2018]. In both cases, we observe significant improvements, especially on glossy surfaces. The implementations for both applications consist of trivial modifications to the original code base, without introducing any additional overhead.

The pedestrian intention is not only signalized by their past trajectory and head gaze direction, but by their whole body movement. In order to improve existing algorithms, analysis of pedestrian motions before entering a shared environment, like a street crossing, is necessary. In addition, more accurate human models are required for a digital reality in order to enable the scalable evaluation of autonomous systems using synthetic data, most of all in the area of autonomous driving. In this work, we present a first approach to capturing pedestrian locomotion in a field experiment using an existing motion capture solution. The motion of 20 participants in 80 crossing trials was recorded and numerically evaluated. The evaluation suggests, that the hip and shoulder rotation with respect to the street is more pronounced before crossing the street compared to other, non-critical behavior.

Due to the current developments towards autonomous driving and vehicle active safety, there is an increasing necessity for algorithms that are able to perform complex criticality predictions in real-time. Being able to process multi-object traffic scenarios aids the implementation of a variety of automotive applications such as driver assistance systems for collision prevention and mitigation as well as fall-back systems for autonomous vehicles. We present a fully model-based algorithm with a parallelizable architecture. The proposed algorithm can evaluate the criticality of complex, multi-modal (vehicles and pedestrians) traffic scenarios by simulating millions of trajectory combinations and detecting collisions between objects. The algorithm is able to estimate upcoming criticality at very early stages, demonstrating its potential for vehicle safety-systems and autonomous driving applications. An implementation on an embedded system in a test vehicle proves in a prototypical manner the compatibility of the algorithm with the hardware possibilities of modern cars. For a complex traffic scenario with 11 dynamic objects, more than 86 million pose combinations are evaluated in 21 ms on the GPU of a Drive PX 2.

We propose a novel approach to create generative models for distinctive styles of locomotion for humanoid characters. Our approach only requires a single or a few style examples and a neutral motion database. We are inspired by the observation that human styles can be easily distinguished from a few examples. However, learning a generative model for natural human motions which can display huge amounts of variations and randomness would require a lot of training data. Furthermore, it would require considerable efforts to create such a large motion database for each style. One solution for that is motion style transfer, which provides the possibility of converting the content of the motion from one style to the other. Typically style transfer focuses on transferring the content motion to target style explicitly. We propose a variational generative model to combine the large variation in neutral motion database and style information from a limited number of examples. We formulate the style motion modeling as a conditional distribution learning problem and style transfer is implicitly applied during the model learning process. A conditional variational autoencoder (CVAE) is applied to learn the distribution and stylistic examples are used as constraints. We demonstrate that our approach can generate any number of natural-looking, various human motions with a similar style to the target.

The simulation of humanoid avatars is relevant for a multitude of applications, such as movies, games, simulations for autonomous vehicles, virtual avatars and many more. In order to achieve the simulation of realistic and believable characters, it is important to simulate motion with the natural motion style matching the character’s characteristic. A female avatar, for example, should move in a female style and different characters should vary in their expressiveness of this style. However, the manual definition, as well as the acting of a natural female or male style, is non-trivial. Previous work on style transfer is insufficient, as the style examples are not necessarily a natural depiction of female or male locomotion. We propose a novel data-driven method to infer the style information based on individual samples of male and female motion capture data. For this purpose, the data of 12 female and 12 male participants was captured in an experimental setting. A neural network based motion model is trained for each participant and the style dimension is learned in the latent representation of these models. Thus a linear style model is inferred on top of the motion models. It can be utilized to synthesize network models of different style expressiveness on a continuous scale while retaining the performance and content of the original network model. A user study supports the validity of our approach while highlighting issues with simpler approaches to infer the style.

To enable the physically correct simulation of the interaction of a 3D character with its environment the internal joint forces of a physical model of the character need to be estimated. Recently, derivative-free sampling-based optimization methods, which treat the objective function as a black box, have shown great results for finding control signals for articulated figures in physics simulations. We present a novel sampling-based approach for the reconstruction of control signals for a rigid body model based on motion capture data that combines ideas of previous approaches. The algorithm optimizes control trajectories along a sliding window using the Covariance Matrix Adaption Evolution Strategy. The sampling distribution is represented as a mixture model with a dynamically selected number of clusters based on the variation detected in the samples. During the optimization we keep track of multiple states which enables the exploration of multiple paths. We evaluate the algorithm for the task of motion capture following using figures that were automatically generated from 3D character models.

Many existing Monte Carlo methods rely on multiple importance sampling (MIS) to achieve robustness and versatility. Typically, the balance or power heuristics are used, mostly thanks to the seemingly strong guarantees on their variance. We show that these MIS heuristics are oblivious to the effect of certain variance reduction techniques like stratification. This shortcoming is particularly pronounced when unstratified and stratified techniques are combined (e.g., in a bidirectional path tracer). We propose to enhance the balance heuristic by injecting variance estimates of individual techniques, to reduce the variance of the combined estimator in such cases. Our method is simple to implement and introduces little overhead.

Multiple Importance Sampling (MIS) is a key technique for achieving robustness of Monte Carlo estimators in computer graphics and other fields. We derive optimal weighting functions for MIS that provably minimize the variance of an MIS estimator, given a set of sampling techniques. We show that the resulting variance reduction over the balance heuristic can be higher than predicted by the variance bounds derived by Veach and Guibas, who assumed only non-negative weights in their proof. We theoretically analyze the variance of the optimal MIS weights and show the relation to the variance of the balance heuristic. Furthermore, we establish a connection between the new weighting functions and control variates as previously applied to mixture sampling. We apply the new optimal weights to integration problems in light transport and show that they allow for new design considerations when choosing the appropriate sampling techniques for a given integration problem.

Monte-Carlo Renderers must generate many color samples to produce a noise-free image, and for each of those, they must evaluate complex mathematical models representing the appearance of the objects in the scene. These models are usually in the form of shaders: Small programs that are executed during rendering in order to compute a value for the current sample. Renderers often compile and optimize shaders just before rendering, taking advantage of the knowledge of the scene. In principle, the entire renderer could benefit from a-priori code generation. For instance, scheduling can take advantage of the knowledge of the scene in order to maximize hardware usage. However, writing such a configurable renderer eventually means writing a compiler that translates a scene description into machine code. In this paper, we present a framework that allows generating entire renderers for CPUs and GPUs without having to write a dedicated compiler: First, we provide a rendering library in a functional/imperative language that elegantly abstracts the individual rendering concepts using higher-order functions. Second, we use partial evaluation to combine and specialize the individual components of a renderer according to a particular scene. Our results show that the renderers we generate outperform equivalent high-performance implementations written with state-of-the-art ray tracing libraries on the CPU and GPU.

No abstract available.

Head-mounted displays (HMDs) with integrated eye trackers have opened up a new realm for gaze-contingent rendering. The accurate estimation of gaze depth is essential when modeling the optical capabilities of the eye. Most recently multifocal displays are gaining importance, requiring focus estimates to control displays or lenses. Deriving the gaze depth solely by sampling the scene's depth at the point-of-regard fails for complex or thin objects as eye tracking is suffering from inaccuracies. Gaze depth measures using the eye's vergence only provide an accurate depth estimate for the first meter. In this work, we combine vergence measures and multiple depth measures into feature sets. This data is used to train a regression model to deliver improved estimates. We present a study showing that using multiple features allows for an accurate estimation of the focused depth (MSE<0.1m) over a wide range (first 6m).

Abstract Robust and efficient rendering of complex lighting effects, such as caustics, remains a challenging task. While algorithms like vertex connection and merging can render such effects robustly, their significant overhead over a simple path tracer is not always justified and – as we show in this paper ‐ also not necessary. In current rendering solutions, caustics often require the user to enable a specialized algorithm, usually a photon mapper, and hand‐tune its parameters. But even with carefully chosen parameters, photon mapping may still trace many photons that the path tracer could sample well enough, or, even worse, that are not visible at all. Our goal is robust, yet lightweight, caustics rendering. To that end, we propose a technique to identify and focus computation on the photon paths that offer significant variance reduction over samples from a path tracer. We apply this technique in a rendering solution combining path tracing and photon mapping. The photon emission is automatically guided towards regions where the photons are useful, i.e., provide substantial variance reduction for the currently rendered image. Our method achieves better photon densities with fewer light paths (and thus photons) than emission guiding approaches based on visual importance. In addition, we automatically determine an appropriate number of photons for a given scene, and the algorithm gracefully degenerates to pure path tracing for scenes that do not benefit from photon mapping.

This paper advocates programming high-performance code using partial evaluation. We present a clean-slate programming system with a simple, annotation-based, online partial evaluator that operates on a CPS-style intermediate representation. Our system exposes code generation for accelerators (vectorization/parallelization for CPUs and GPUs) via compiler-known higher-order functions that can be subjected to partial evaluation. This way, generic implementations can be instantiated with target-specific code at compile time. In our experimental evaluation we present three extensive case studies from image processing, ray tracing, and genome sequence alignment. We demonstrate that using partial evaluation, we obtain high-performance implementations for CPUs and GPUs from one language and one code base in a generic way. The performance of our codes is mostly within 10%, often closer to the performance of multi man-year, industry-grade, manually-optimized expert codes that are considered to be among the top contenders in their fields.

In recent years, a variety of methods have been introduced to exploit the decrease in visual acuity of peripheral vision, known as foveated rendering. As more and more computationally involved shading is requested and display resolutions increase, maintaining low latencies is challenging when rendering in a virtual reality context. Here, foveated rendering is a promising approach for reducing the number of shaded samples. However, besides the reduction of the visual acuity, the eye is an optical system, filtering radiance through lenses. The lenses create depth-of-field (DoF) effects when accommodated to objects at varying distances. The central idea of this article is to exploit these effects as a filtering method to conceal rendering artifacts. To showcase the potential of such filters, we present a foveated rendering system, tightly integrated with a gaze-contingent DoF filter. Besides presenting benchmarks of the DoF and rendering pipeline, we carried out a perceptual study, showing that rendering quality is rated almost on par with full rendering when using DoF in our foveated mode, while shaded samples are reduced by more than 69%.

Virtual Environments have become a compelling tool for various applications beyond gaming and Virtual Worlds, for example for education, collaborative engineering, or simulation and visualization. In the emerging field of smart environments, like Smart Cities and Smart Factories for industry or agriculture, a digital counterpart of a real world site, driven by hundreds of Internet of Things (IoT) sensors that emit their data in realtime, is a central part of the application. Maintaining such a large-scale virtual environment and keeping it up-to-date with changing requirements set by connected sensors and services is a challenge. For this, we present FiVES, a server framework that is based on an aspect-oriented architecture to create highly maintainable large-scale virtual environments by avoiding crosscutting concerns like tangling or scattering of code between modules in resulting applications. We present a fully functional implementation of our system that builds on an extendable and versatile data model, a flexible plugin mechanism, and a transparent yet efficient synchronization layer that is independent of the modules from which applications are assembled.

In order to achieve the highest possible performance, the ray traversal and intersection routines at the core of every high-performance ray tracer are usually hand-coded, heavily optimized, and implemented separately for each hardware platform—even though they share most of their algorithmic core. The results are implementations that heavily mix algorithmic aspects with hardware and implementation details, making the code non-portable and difficult to change and maintain.

We present a spatial index structure to accelerate ray tracing on GPUs. It is a flat, non-hierarchical spatial subdivision of the scene into axis aligned cells of varying size. In order to construct it, we first nest an octree into each cell of a uniform grid. We then apply two optimization passes to increase ray traversal performance: First, we reduce the expected cost for ray traversal by merging cells together. This adapts the structure to complex primitive distributions, solving the “teapot in a stadium” problem. Second, we decouple the cell boundaries used during traversal for rays entering and exiting a given cell. This allows us to extend the exiting boundaries over adjacent cells that are either empty or do not contain additional primitives. Now, exiting rays can skip empty space and avoid repeating intersection tests. Finally, we demonstrate that in addition to the fast ray traversal performance, the structure can be rebuilt efficiently in parallel, allowing for ray tracing dynamic scenes.

Advances in computer graphics enable us to create digital images of astonishing complexity and realism. However, processing resources are still a limiting factor. Hence, many costly but desirable aspects of realism are often not accounted for, including global illumination, accurate depth of field and motion blur, spectral effects, etc. especially in real-time rendering. At the same time, there is a strong trend towards more pixels per display due to larger displays, higher pixel densities or larger fields of view. Further observable trends in current display technology include more bits per pixel (high dynamic range, wider color gamut/fidelity), increasing refresh rates (better motion depiction), and an increasing number of displayed views per pixel (stereo, multi-view, all the way to holographic or lightfield displays). These developments cause significant unsolved technical challenges due to aspects such as limited compute power and bandwidth. Fortunately, the human visual system has certain limitations, which mean that providing the highest possible visual quality is not always necessary. In this report, we present the key research and models that exploit the limitations of perception to tackle visual quality and workload alike. Moreover, we present the open problems and promising future research targeting the question of how we can minimize the effort to compute and display only the necessary pixels while still offering a user full visual experience.

Abstract This paper poses the problem of fabricating physical construction sets from example geometry: A construction set provides a small number of different types of building blocks from which the example model as well as many similar variants can be reassembled. This process is formalized by tiling grammars. Our core contribution is an approach for simplifying tiling grammars such that we obtain physically manufacturable building blocks of controllable granularity while retaining variability, i.e., the ability to construct many different, related shapes. Simplification is performed by sequences of two types of elementary Operations: non‐local joint edge collapses in the tile graphs reduce the granularity of the decomposition and approximate replacement Operations reduce redundancy. We evaluate our method on abstract graph grammars in addition to computing several physical construction sets, which are manufactured using a commodity 3D printer.

Abstract Head‐mounted displays with dense pixel arrays used for virtual reality applications require high frame rates and low latency rendering. This forms a challenging use case for any rendering approach. In addition to its ability of generating realistic images, ray tracing offers a number of distinct advantages, but has been held back mainly by its performance. In this paper, we present an approach that significantly improves image generation performance of ray tracing. This is done by combining foveated rendering based on eye tracking with reprojection rendering using previous frames in order to drastically reduce the number of new image samples per frame. To reproject samples a coarse geometry is reconstructed from a G‐Buffer. Possible errors introduced by this reprojection as well as parts that are critical to the perception are scheduled for resampling. Additionally, a coarse color buffer is used to provide an initial image, refined smoothly by more samples were needed. Evaluations and user tests show that our method achieves real‐time frame rates, while visual differences compared to fully rendered images are hardly perceivable. As a result, we can ray trace non‐trivial static scenes for the Oculus DK2 HMD at 1182 × 1464 per eye within the the VSync limits without perceived visual differences.

This paper investigates shallow embedding of DSLs by means of online partial evaluation. To this end, we present a novel online partial evaluator for continuation-passing style languages. We argue that it has, in contrast to prior work, a predictable termination policy that works well in practice. We present our approach formally using a continuation-passing variant of PCF and prove its termination properties. We evaluate our technique experimentally in the field of visual and high-performance computing and show that our evaluator produces highly specialized and efficient code for CPUs as well as GPUs that matches the performance of hand-tuned expert code.

Partial evaluation allows for specialization of program fragments. This can be realized by staging, where one fragment is executed earlier than its surrounding code. However, taking advantage of these capabilities is often a cumbersome endeavor. In this paper, we present a new metaprogramming concept using staging parameters that are first-class citizen entities and define the order of execution of the program. Staging parameters can be used to define MetaML-like quotations, but can also allow stages to be created and resolved dynamically. The programmer can write generic, polyvariant code which can be reused in the context of different stages. We demonstrate how our approach can be used to define and apply domain-specific optimizations. Our implementation of the proposed metaprogramming concept generates code which is on a par with templated C++ code in terms of execution time.

Abstract In computer graphics a material is a visual concept that is parameterizable and should work for arbitrary 3D assets and rendering systems. Since provided parameters and attributes as well as the capabilities of rendering systems vary considerably, a material needs to adapt to its execution environment. In current approaches, the adaptation logic is ‘baked’ into the rendering application based on string manipulation, compiler directives, or metaprogramming facilities. However, in order to achieve application‐independent and self‐contained material descriptions, the adaptation logic needs to be part of the material description itself. In this paper we present shade.js, a novel material description using a dynamic language to achieve the necessary adaptivity. A shader can inspect its execution environment and adapt to the available parameters and renderer capabilities at run time. Additionally, shade.js exploits the polymorphism that comes with non‐explicit declaration of types. These two novel features allow for writing adaptable and thus more general material descriptions. Based on the concrete execution environment at run time, the accompanied compiler generates specialized shader code that is specifically typed and optimized for the target rendering system and algorithm. We evaluate shade.js with examples targeting four different rendering approaches (forward and deferred rasterization, ray‐tracing, and global illumination). We show that we can improve convenience and flexibility for specifying materials without sacrificing performance.

Graphics Processing Units (GPUs) recently became general enough to enable implementation of a variety of light transport algorithms. However, the efficiency of these GPU implementations has received relatively little attention in the research literature and no systematic study on the topic exists to date. The goal of our work is to fill this gap. Our main contribution is a comprehensive and in-depth investigation of the efficiency of the GPU implementation of a number of classic as well as more recent progressive light transport simulation algorithms. We present several improvements over the state-of-the-art. In particular, our light vertex cache, a new approach to mapping connections of subpath vertices in bidirectional path tracing on the GPU, outperforms the existing implementations by 30--60%. We also describe a first GPU implementation of the recently introduced vertex connection and merging algorithm [Georgiev et al. 2012], showing that even relatively complex light transport algorithms can be efficiently mapped on the GPU. With the implementation of many of the state-of-the-art algorithms within a single system at our disposal, we present a unique direct comparison and analysis of their relative performance.

The convergence of information technologies (IT) has enabled the Digital Enterprise in which engineering, production planning, manufacturing and sales processes are supported by IT-based collaboration, simulation and enactment. As a result, borders between reality and its virtual epresentations be

The area of virtual environments has received an increasingly growing attention from the research community, leading to the design of various innovative albeit incompatible protocols. In this paper, we present an open modular middleware that allows the clients and servers of multiple virtual worlds to be dynamically interfaced via a granular design. Our technology significantly alleviates the workload traditionally imposed to developers, and we see it as an important step forward towards the design of universal interfaces for interoperable virtual environments.

Abstract Ray tracing has become a viable alternative to rasterization for interactive applications and also forms the basis of most global illumination methods. However, even today's fastest ray‐tracers offer only a tight budget of rays per pixel per frame. Rendering performance can be improved by increasing this budget, or by developing methods that use it more efficiently. In this paper we propose a global visibility caching algorithm that reduces the number of shadow rays required for shading to a fraction of less than 2% in some cases. We quantize the visibility function's domain while ensuring a minimal degradation of the final image quality. To control the introduced error, we adapt the quantization locally, accounting for variations in geometry, sampling densities on both endpoints of the visibility queries, and the light signal itself. Compared to previous approaches for approximating visibility, e.g. shadow mapping, our method has several advantages: (1) it allows caching of arbitrary visibility queries between surface points and is thus applicable to all ray tracing based methods; (2) the approximation error is uniform over the entire image and can be bounded by a user‐specified parameter; (3) the cache is created on‐the‐fly and does not waste any resources on queries that will never be used. We demonstrate the benefits of our method on Whitted‐style ray tracing combined with instant radiosity, as well as an integration with bidirectional path tracing.

Abstract Realistic rendering requires computing the global illumination in the scene, and Monte Carlo integration is the best‐known method for doing that. The key to good performance is to carefully select the costly integration samples, which is usually achieved via importance sampling. Unfortunately, visibility is difficult to factor into the importance distribution, which can greatly increase variance in highly occluded scenes with complex illumination. In this paper, we present importance caching – a novel approach that selects those samples with a distribution that includes visibility, while maintaining efficiency by exploiting illumination smoothness. At a sparse set of locations in the scene, we construct and cache several types of probability distributions with respect to a set of virtual point lights (VPLs), which notably include visibility. Each distribution type is optimized for a specific lighting condition. For every shading point, we then borrow the distributions from nearby cached locations and use them for VPL sampling, avoiding additional bias. A novel multiple importance sampling framework finally combines the many estimators. In highly occluded scenes, where visibility is a major source of variance in the incident radiance, our approach can reduce variance by more than an order of magnitude. Even in such complex scenes we can obtain accurate and low noise previews with full global illumination in a couple of seconds on a single mid‐range CPU.

Abstract In this paper, we develop a theoretical framework for characterizing shapes by building blocks. We address two questions: First, how do shape correspondences induce building blocks? For this, we introduce a new representation for structuring partial symmetries (partial self‐correspondences), which we call “microtiles”. Starting from input correspondences that form point‐wise equivalence relations, microtiles are obtained by grouping connected components of points that share the same set of symmetry transformations. The decomposition is unique, requires no parameters beyond the input correspondences, and encodes the partial symmetries of all subsets of the input. The second question is: What is the class of shapes that can be assembled from these building blocks? Here, we specifically consider r‐similarity as correspondence model, i.e., matching of local r‐neighborhoods. Our main result is that the microtiles of the partial r‐symmetries of an object S can build all objects that are (r+ε)‐similar to S for any ε >0. Again, the construction is unique. Furthermore, we give necessary conditions for a set of assembly rules for the pairwise connection of tiles. We describe a practical algorithm for computing microtile decompositions under rigid motions, a corresponding prototype implementation, and conduct a number of experiments to visualize the structural properties in practice.

Developing robust light transport simulation algorithms that are capable of dealing with arbitrary input scenes remains an elusive challenge. Although efficient global illumination algorithms exist, an acceptable approximation error in a reasonable amount of time is usually only achieved for specific types of input scenes. To address this problem, we present a reformulation of photon mapping as a bidirectional path sampling technique for Monte Carlo light transport simulation. The benefit of our new formulation is twofold. First, it makes it possible, for the first time, to explain in a formal manner the relative efficiency of photon mapping and bidirectional path tracing, which have so far been considered conceptually incompatible solutions to the light transport problem. Second, it allows for a seamless integration of the two methods into a more robust combined rendering algorithm via multiple importance sampling. A progressive version of this algorithm is consistent and efficiently handles a wide variety of lighting conditions, ranging from direct illumination, diffuse and glossy inter-reflections, to specular-diffuse-specular light transport. Our analysis shows that this algorithm inherits the high asymptotic performance from bidirectional path tracing for most light path types, while benefiting from the efficiency of photon mapping for specular-diffuse-specular lighting effects.

The Web and virtual worlds are currently crossing their ways, and although there are some efforts made to integrate them into each other, those typically rely on technologies that are rather esoteric to most web-developers. In this paper, we present a new open architecture that combines several emerging and established technologies to provide convenient tools for developing virtual worlds in the Web. These technologies are easy to learn and understand by the web community and allow for quick prototyping. Overall the modular architecture allows virtual worlds to be developed more quickly and more widely deployed.

We present vertex merging -- a bidirectional path sampling technique for Monte Carlo light transport integration. Vertex merging is simple and more computationally efficient for specular-diffuse-specular effects than the currently available techniques in bidirectional path tracing. It brings the advantages of photon mapping to the path integral framework, while avoiding the concept of density estimation altogether. This makes it possible for the first time to quantitatively reason about the efficiency of two rendering approaches that have been historically considered conceptually different. The practical result is a combined bidirectional rendering algorithm that efficiently handles a wide variety of lighting conditions, ranging from direct illumination and diffuse inter-reflections to the notoriously problematic reflected caustics. This algorithm also has a higher order of convergence than progressive photon mapping.

Abstract We investigate the use of two‐level nested grids as acceleration structure for ray tracing of dynamic scenes. We propose a massively parallel, sort‐based construction algorithm and show that the two‐level grid is one of the structures that is fastest to construct on modern graphics processors. The structure handles non‐uniform primitive distributions more robustly than the uniform grid and its traversal performance is comparable to those of other high quality acceleration structures used for dynamic scenes. We propose a cost model to determine the grid resolution and improve SIMD utilization during ray‐triangle intersection by employing a hybrid packetization strategy. The build times and ray traversal acceleration provide overall rendering performance superior to previous approaches for real time rendering of animated scenes on GPUs.

Abstract Available GPUs provide increasingly more processing power especially for multimedia and digital signal processing. Despite the tremendous progress in hardware and thus processing power, there are and always will be applications that require using multiple GPUs either running inside the same machine or distributed in the network due to computationally intensive processing algorithms. Existing solutions for developing applications for GPUs still require a lot of hand‐optimization when using multiple GPUs inside the same machine and provide in general no or only limited support for using remote GPUs distributed in the network. In this paper we address this problem and show that an open distributed multimedia middleware, like the Network‐Integrated Multimedia Middleware (NMM), is able (1) to seamlessly integrate processing components using GPUs, while completely hiding GPU‐specific issues from the application developer, (2) to transparently combine processing components using GPUs or CPUs, and (3) to transparently use local and remote GPUs for distributed processing. Furthermore, we present a generic distribution framework to simplify the development of complex application scenarios. Copyright © 2010 John Wiley & Sons, Ltd.

Molecular visualization is one of the cornerstones in structural bioinformatics and related fields. Today, rasterization is typically used for the interactive display of molecular scenes, while ray tracing aims at generating high-quality images, taking typically minutes to hours to generate and requiring the usage of an external off-line program. Recently, real-time ray tracing evolved to combine the interactivity of rasterization-based approaches with the superb image quality of ray tracing techniques. We demonstrate how real-time ray tracing integrated into a molecular modelling and visualization tool allows for better understanding of the structural arrangement of biomolecules and natural creation of publication-quality images in real-time. However, unlike most approaches, our technique naturally integrates into the full-featured molecular modelling and visualization tool BALL View, seamlessly extending a standard workflow with interactive high-quality rendering.

Accurately rendering glossy materials in design applications, where previewing and interactivity are important, remains a major challenge. While many fast global illumination solutions have been proposed, all of them work under limiting assumptions on the materials and lighting in the scene. In the presence of many glossy (directionally scattering) materials, fast solutions either fail or degenerate to inefficient, brute-force simulations of the underlying light transport. In particular, many-light algorithms are able to provide fast approximations by clamping elements of the light transport matrix, but they eliminate the part of the transport that contributes to accurate glossy appearance. In this paper we introduce a solution that separately solves for the global (low-rank, dense) and local (highrank, sparse) illumination components. For the low-rank component we introduce visibility clustering and approximation, while for the high-rank component we introduce a local light technique to correct for the missing illumination. Compared to competing techniques we achieve superior gloss rendering in minutes, making our technique suitable for applications such as industrial design and architecture, where material appearance is critical.

Today's multimedia infrastructures adopt a centralized approach, where all multimedia processing takes place within a single system. The network is, at best, used for streaming data transmission. Since there is a strong trend towards networked systems, these traditional approaches are becoming obsolete. In contrast, the Network-Integrated Multimedia Middleware (NMM) offers a multimedia architecture, which considers the network as an integral part and enables the intelligent use of devices distributed across the network.

This course instructs students in the software and hardware strategies needed for real-time visualization and interaction with massive models. Seven international researchers and practitioners are the instructors. The general form of the course will be lecture with live demos.

Abstract Recent research on high‐performance ray tracing has achieved real‐time performance even for highly complex surface models already on a single PC. In this report, we provide an overview of techniques for extending real‐time ray tracing also to interactive volume rendering. We review fast rendering techniques for different volume representations and rendering modes in a variety of computing environments. The physically‐based rendering approach of ray tracing enables high image quality and allows for easily mixing surface, volume and other primitives in a scene, while fully accounting for all of their optical interactions. We present optimized implementations and discuss the use of upcoming high‐performance processors for volume ray tracing.

Abstract Significant advances have been achieved for realtime ray tracing recently, but realtime performance for complex scenes still requires large computational resources not yet available from the CPUs in standard PCs. Incidentally, most of these PCs also contain modern GPUs that do offer much larger raw compute power. However, limitations in the programming and memory model have so far kept the performance of GPU ray tracers well below that of their CPU counterparts. In this paper we present a novel packet ray traversal implementation that completely eliminates the need for maintaining a stack during kd‐tree traversal and that reduces the number of traversal steps per ray. While CPUs benefit moderately from the stackless approach, it improves GPU performance significantly. We achieve a peak performance of over 16 million rays per second for reasonably complex scenes, including complex shading and secondary rays. Several examples show that with this new technique GPUs can actually outperform equivalent CPU based ray tracers.

The importance of high-performance rendering of unstructured or curvilinear data sets has increased significantly, mainly due to its use in scientific simulations such as computational fluid dynamics and finite element computations. However, the unstructured nature of these data sets lead to rather slow implementations for ray tracing. The approaches discussed in this paper are fast and scalable towards realtime ray tracing applications. We evaluate new algorithms for rendering tetrahedral and hexahedral meshes. In each algorithm, the first cell along a ray is found using common realtime ray tracing techniques. For traversing subsequent cells within the volume, Plücker coordinates as well as ray-bilinear patch intersection tests are used. Since the volume is rendered directly, all algorithms are applicable for isosurface rendering, maximum-intensity projection, and emissionabsorption models.

Abstract Though ray tracing has recently become interactive, its high precomputation time for building spatial indices usually limits its applications to walkthroughs of static scenes. This is a major limitation, as most applications demand support for dynamically animated models. In this paper, we present a new approach to ray trace a special but important class of dynamic scenes, namely models whose connectivity does not change over time and for which all possible poses are known in advance. We support these kinds of models by introducing two new concepts: motion decomposition, and fuzzy kd‐trees. We analyze the animation and break the model down into submeshes with similar motion. For each of these submeshes and for every time step, we calculate a best affine transformation through a least square approach. Any residual motion is then captured in a single "fuzzy kd‐tree" for the entire animation. Together, these techniques allow for ray tracing animations without rebuilding the spatial index structures for the submeshes, resulting in interactive frame rates of 5 to 15 fps even on a single CPU. Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Ray tracing I.3.6 [Methodology and Techniques]: Graphics data structures and data types

Recursive ray tracing is a simple yet powerful and general approach for accurately computing global light transport and rendering high quality images. While recent algorithmic improvements and optimized parallel software implementations have increased ray tracing performance to realtime levels, no compact and programmable hardware solution has been available yet.This paper describes the architecture and a prototype implementation of a single chip, fully programmable Ray Processing Unit (RPU). It combines the flexibility of general purpose CPUs with the efficiency of current GPUs for data parallel computations. This design allows for realtime ray tracing of dynamic scenes with programmable material, geometry, and illumination shaders.Although, running at only 66 MHz the prototype FPGA implementation already renders images at up to 20 frames per second, which in many cases beats the performance of highly optimized software running on multi-GHz desktop CPUs. The performance and efficiency of the proposed architecture is analyzed using a variety of benchmark scenes.

The visualization of high-quality isosurfaces at interactive rates is an important tool in many simulation and visualization applications. Today, isosurfaces are most often visualized by extracting a polygonal approximation that is then rendered via graphics hardware or by using a special variant of preintegrated volume rendering. However, these approaches have a number of limitations in terms of the quality of the isosurface, lack of performance for complex data sets, or supported shading models. An alternative isosurface rendering method that does not suffer from these limitations is to directly ray trace the isosurface. However, this approach has been much too slow for interactive applications unless massively parallel shared-memory supercomputers have been used. In this paper, we implement interactive isosurface ray tracing on commodity desktop PCs by building on recent advances in real-time ray tracing of polygonal scenes and using those to improve isosurface ray tracing performance as well. The high performance and scalability of our approach will be demonstrated with several practical examples, including the visualization of highly complex isosurface data sets, the interactive rendering of hybrid polygonal/isosurface scenes, including high-quality ray traced shading effects, and even interactive global illumination on isosurfaces.

Abstract Photon mapping is one of the most important algorithms for computing global illumination. Especially for efficiently producing convincing caustics, there are no real alternatives to photon mapping. On the other hand, photon mapping is also quite costly: Each radiance lookup requires to find the k nearest neighbors in a kd‐tree, which can be more costly than shooting several rays. Therefore, the nearest‐neighbor queries often dominate the rendering time of a photon map based renderer. In this paper, we present a method that reorganizes — i.e. un balances — the kd‐tree for storing the photons in a way that allows for finding the k‐nearest neighbors much more efficiently, thereby accelerating the radiance estimates by a factor of 1.2–3.4. Most importantly, our method still finds exactly the same k‐nearest‐neighbors as the original method, without introducing any approximations or loss of accuracy. The impact of our method is demonstrated with several practical examples. Categories and Subject Descriptors (according to ACM CCS): I.3.3 [Computer Graphics]: Global Illumination I.3.7 [Computer Graphics]: Raytracing

Abstract The addition of global illumination can dramatically increase the realism achievable when rendering virtual environments.In particular with interactive applications we expect the environment to reflect changes in the scenedue to global lighting effects instead of it being just a static backdrop. However, a sufficiently fast and accuratecomputation of global illumination at interactive rates has been difficult even with recent approaches based onrealtime ray tracing. In this paper we present a highly scalable approach to interactive global illumination. It fully recomputes a high‐qualitysolution for each frame and thus offers immediate feedback even for dynamic scenes, achieving more than20 fps for simple scenes. Compared to previous systems we increased the raw performance by a factor of up toeight and removed the bottlenecks that were limiting scalability. The system now scales linearly in quality andavailable computing resources, tested with up to 48 CPUs in a commodity PC‐cluster. Due to its logarithmicscaling property with respect to scene complexity it even supports lighting simulation in complex scenes with morethan 50 million triangles. This scalability allows applications to perform flexible performance trade‐offs. We alsoargue that the realism achievable through interactive global illumination will make it a standard feature of future3D graphics systems once the required computing resources are readily available.

For almost two decades researchers have argued that ray tracing will eventually become faster than the rasterization technique that completely dominates todays graphics hardware. However, this has not happened yet. Ray tracing is still exclusively being used for off-line rendering of photorealistic images and it is commonly believed that ray tracing is simply too costly to ever challenge rasterization-based algorithms for interactive use. However, there is hardly any scientific analysis that supports either point of view. In particular there is no evidence of where the crossover point might be, at which ray tracing would eventually become faster, or if such a point does exist at all. This paper provides several contributions to this discussion: We first present a highly optimized implementation of a ray tracer that improves performance by more than an order of magnitude compared to currently available ray tracers. The new algorithm make better use of computational resources such as caches and SIMD instructions and better exploits image and object space coherence. Secondly, we show that this software implementation can challenge and even outperform high-end graphics hardware in interactive rendering performance for complex environments. We also provide an brief overview of the benefits of ray tracing over rasterization algorithms and point out the potential of interactive ray tracing both in hardware and software.

The paper introduces a method to calibrate a wide area system of unsynchronized cameras with respect to a single global coordinate system. The method is simple and does not require the physical construction of a large calibration object. The user need only wave an identifiable point in front of all cameras. The method generates a rough estimate of camera pose by first performing pair-wise structure-from-motion on observed points, and then combining the pair-wise registrations into a single coordinate frame. Using the initial camera pose, the moving point can be tracked in world space. The path of the point defines a "virtual calibration object" which can be used to improve the initial estimates of camera pose. Iterating the above process yields a more precise estimate of both camera pose and the point path. Experimental results show that it performs as well as calibration from a physical target, in cases where all cameras share some common working volume. We then demonstrate its effectiveness in wide area settings by calibrating a system of cameras in a configuration where traditional methods cannot be applied directly.

Clustering is a very efficient technique to apply finite element methods to the computation of radiosity solutions of complex scenes. Both computation time and memory consumption can be reduced dramatically by grouping the primitives of the input scene into a hierarchy of clusters and allowing for light exchange between all levels of this hierarchy. However, problems can arise due to clustering, when gross approximations about a cluster's content result in unsatisfactory solutions or unnecessary computations. In the clustering approach for diffuse global information described in the paper, light exchange between two objects-patches or clusters-is bounded by using geometrical and shading information provided by every object through a uniform interface. With this uniform view of various kinds of objects, comparable and reliable error bounds on the light exchange can be computed, which then guide a standard hierarchical radiosity algorithm.

Progressive addition lenses are a relatively new approach to compensate for defects of the human visual system. While traditional spectacles use rotationally symmetric lenses, progressive lenses require the specification of free‐form surfaces. This poses difficult problems for the optimal design and its visual evaluation. This paper presents two new techniques for the visualization of optical systems and the optimization of progressive lenses. Both are based on the same wavefront tracing approach to accurately evaluate the refraction properties of complex optical systems. We use the results of wavefront tracing for continuously re‐focusing the eye during rendering. Together with distribution ray tracing, this yields high‐quality images that accurately simulate the visual quality of an optical system. The design of progressive lenses is difficult due to the trade‐off between the desired properties of the lens and unavoidable optical errors, such as astigmatism and distortions. We use wavefront tracing to derive an accurate error functional describing the desired properties and the optical error across a lens. Minimizing this error yields optimal free‐form lens surfaces. While the basic approach is much more general, in this paper, we describe its application to the particular problem of designing and evaluating progressive lenses and demonstrate the benefits of the new approach with several example images.

Hierarchical radiosity with clustering has positioned itself as one of the most efficient algorithms for computing global illumination in non‐trivial environments. However, using hierarchical radiosity for complex scenes is still problematic due to the necessity of storing a large number of transport coefficients between surfaces in the form of links. In this paper, we eliminate the need for storage of links through the use of a modified shooting method for solving the radiosity equation. By distributing only unshot radiosity in each step of the iteration, the number of links decreases exponentially. Recomputing these links instead of storing them increases computation time, but reduces memory consumption dramatically. Caching may be used to reduce the time overhead. We analyze the error behavior of the new algorithm in comparison with the normal gathering approach for hierarchical radiosity. In particular, we consider the relation between the global error of a hierarchical radiosity solution and the local error threshold for each link.

Procedural shaders have become popular tools for describing surface reflectance functions and other material properties. In comparison to fixed resolution textures, they have the advantage of being resolution-independent and storage-efficient. While procedural shaders provide an interface for evaluating the shader at a single point, it is not easily possible to obtain an average value of the shader together with accurate error bounds over a finite area. Yet the ability to compute such error bounds is crucial for several interesting applications, most notably heirarchical area sampling for global illumination, using the finite element approach, and for generation of textures used in interactive computer graphics. Using affine arithmetic for evaluating the shader over a finite area yields a tight, conservative error interval for the shader function. Compilers can automatically generate code for utilizing affine arithmetic from within shaders implemented in a dedicated language such as the RenderMann shading language.

Since the beginning of computer graphics, one of the primary goals has been to create convincingly realistic images of three-dimensional environments that would be impossible to distinguish from photographs of the real scene. The goal to create photo-realistic images has lead to the development of completely new software techniques for dealing with the inherent geometric and optical complexity of real world scenes. This paper gives an overview of advanced algorithms for photo-realistic rendering and in particular discusses hierarchical techniques for global illumination computations.

Traditionally, Radiosity algorithms have been restricted to scenes made from planar patches. Most algorithms for computing form factors and the subdivision criterion for hierarchical methods implicitly assume planar patches. In this paper, we present a new radiosity algorithm that is solely based on simple geometric information about surface elements, namely their bounding boxes and cone of normals. Using this information allows to compute efficient error bounds that can be used for the subdivision oracle and for computing the energy transfer. Due to the simple interface to geometric objects, our algorithm not only allows for computing illumination on general curved surfaces, but it can also be directly applied to a hieararchy of clusters. Several examples demonstrate the advantages of the new approach.

Abstract Global illumination techniques like radiosity or Monte‐Carlo ray‐tracing are becoming standard features of rendering systems. However, there is currently no accepted interface format which supports an appropriate physically‐based scene description. In this paper we present extensions to the well‐known RenderMan interface, which allow for a physically based scene description and support advanced global illumination techniques. Special emphasis has been laid on the support for procedural descriptions of reflection and emission by RenderMan surface shaders. So far, they could not be used with most global illumination algorithms. The extensions have been implemented in a physically‐based rendering system and are illustrated with examples.

So far, the problem of global illumination calculation has almost exclusively been approached from an algorithmic point of view. We propose an architectural approach to global illumination. The proposed rendering architecture Vision is derived from a model of the physical rendering process, which is subsequently mapped onto an object-oriented hierarchy of classes. This design is powerful and flexible enough to support and exploit a large body of existing illumination algorithms for the simulation of various aspects of the underlying physical model. Additionally, the Vision architecture offers a platform for developing new algorithms and for combining them to create new rendering solutions. We discuss both abstract design as well as implementation issues. In particular, we give a detailed description of the global lighting subsystem and show how algorithms for path tracing, bidirectional estimators, irradiance caching, hierarchical radiosity, wavelet radiosity, and wavelet radiance have been implemented within Vision.>

Abstract The RenderMan interface has been proposed as a general interface to rendering systems, yet only a few implementations of the interface exist. In this paper we describe the implementation of the RenderMan interface on a general rendering architecture that supports various rendering algorithms. Specifically we discuss the implementation of the RenderMan Shading Language and its integration into our rendering architecture. Special attention is focused on the problems that we have encountered and how they can be solved. Additionally, we suggest extensions and enhancements to the current interface definition, which would make RenderMan easier to implement and more flexible to use.

TRIMO has been designed as a workstation-based interactive system for the generation, manipulation, and display of surfaces over arbitrary toplogical meshes. In addition to rational tensor product Bezier and B-spline surfaces, TRIMO also supports piecewise rational triangular Bezier and B-patch surfaces. TRIMO has been implemented in C++ under the X Window System. Special emphasis has been given to a hierarchical data structure and to a menu-and-mouse-driven hierarchical user interface.