Abstract In this work, we focus on outdoor lighting estimation by aggregating individual noisy estimates from images, exploiting the rich image information from wide-angle cameras and/or temporal image sequences. Photographs inherently encode information about the lighting of the scene in the form of shading and shadows. Recovering the lighting is an inverse rendering problem and as that ill-posed. Recent research based on deep neural networks has shown promising results for estimating light from a single image, but with shortcomings in robustness. We tackle this problem by combining lighting estimates from several image views sampled in the angular and temporal domains of an image sequence. For this task, we introduce a transformer architecture that is trained in an end-2-end fashion without any statistical post-processing as required by previous work. Thereby, we propose a positional encoding that takes into account camera alignment and ego-motion estimation to globally register the individual estimates when computing attention between visual words. We show that our method leads to improved lighting estimation while requiring fewer hyperparameters compared to the state of the art.

This erratum aims to correct errors in the sections 1, 3, and 5 of Lee et al. (2022).Some of the texts in these sections were reproduced in non-final form.It resulted in omissions of several major extensions that are made during the revision process.Figures and Tables are not affected.

Abstract In this paper, we propose a new method for structuring multi‐modal representations of shapes according to semantic relations. We learn a metric that links semantically similar objects represented in different modalities. First, 3D‐shapes are associated with textual labels by learning how textual attributes are related to the observed geometry. Correlations between similar labels are captured by simultaneously embedding labels and shape descriptors into a common latent space in which an inner product corresponds to similarity. The mapping is learned robustly by optimizing a rank‐based loss function under a sparseness prior for the spectrum of the matrix of all classifiers. Second, we extend this framework towards relating multi‐modal representations of the geometric objects. The key idea is that weak cues from shared human labels are sufficient to obtain a consistent embedding of related objects even though their representations are not directly comparable. We evaluate our method against common base‐line approaches, investigate the influence of different geometric descriptors, and demonstrate a prototypical multi‐modal browser that relates 3D‐objects with text, photographs, and 2D line sketches.

Abstract Conventional depth video compression uses video codecs designed for color images. Given the performance of current encoding standards, this solution seems efficient. However, such an approach suffers from many issues stemming from discrepancies between depth and light perception. To exploit the inherent limitations of human depth perception, we propose a novel depth compression method that employs a disparity perception model. In contrast to previous methods, we account for disparity masking, and model a distinct relation between depth perception and contrast in luminance. Our solution is a natural extension to the H.264 codec and can easily be integrated into existing decoders. It significantly improves both the compression efficiency without sacrificing visual quality of depth of rendered content, and the output of depth‐reconstruction algorithms or depth cameras.

Abstract In this work, we present an analysis of feature descriptors for objective image quality assessment. We explore a large space of possible features including components of existing image quality metrics as well as many traditional computer vision and statistical features. Additionally, we propose new features motivated by human perception and we analyze visual saliency maps acquired using an eye tracker in our user experiments. The discriminative power of the features is assessed by means of a machine learning framework revealing the importance of each feature for image quality assessment task. Furthermore, we propose a new data‐driven full‐reference image quality metric which outperforms current state‐of‐theart metrics. The metric was trained on subjective ground truth data combining two publicly available datasets. For the sake of completeness we create a new testing synthetic dataset including experimentally measured subjective distortion maps. Finally, using the same machine‐learning framework we optimize the parameters of popular existing metrics.

Abstract Synthetically generating images and video frames of complex 3D scenes using some photo‐realistic rendering software is often prone to artifacts and requires expert knowledge to tune the parameters. The manual work required for detecting and preventing artifacts can be automated through objective quality evaluation of synthetic images. Most practical objective quality assessment methods of natural images rely on a ground‐truth reference, which is often not available in rendering applications. While general purpose no‐reference image quality assessment is a difficult problem, we show in a subjective study that the performance of a dedicated no‐reference metric as presented in this paper can match the state‐of‐the‐art metrics that do require a reference. This level of predictive power is achieved exploiting information about the underlying synthetic scene (e.g., 3D surfaces, textures) instead of merely considering color, and training our learning framework with typical rendering artifacts. We show that our method successfully detects various non‐trivial types of artifacts such as noise and clamping bias due to insufficient virtual point light sources, and shadow map discretization artifacts. We also briefly discuss an inpainting method for automatic correction of detected artifacts.

Reliable detection of global illumination and rendering artifacts in the form of localized distortion maps is important for many graphics applications. Although many quality metrics have been developed for this task, they are often tuned for compression/transmission artifacts and have not been evaluated in the context of synthetic CG-images. In this work, we run two experiments where observers use a brush-painting interface to directly mark image regions with noticeable/objectionable distortions in the presence/absence of a high-quality reference image, respectively. The collected data shows a relatively high correlation between the with-reference and no-reference observer markings. Also, our demanding per-pixel image-quality datasets reveal weaknesses of both simple (PSNR, MSE, sCIE-Lab) and advanced (SSIM, MS-SSIM, HDR-VDP-2) quality metrics. The most problematic are excessive sensitivity to brightness and contrast changes, the calibration for near visibility-threshold distortions, lack of discrimination between plausible/implausible illumination, and poor spatial localization of distortions for multi-scale metrics. We believe that our datasets have further potential in improving existing quality metrics, but also in analyzing the saliency of rendering distortions, and investigating visual equivalence given our with- and no-reference data.

Abstract In this paper, we focus on efficient compression and streaming of frames rendered from a dynamic 3D model. Remote rendering and on‐the‐fly streaming become increasingly attractive for interactive applications. Data is kept confidential and only images are sent to the client. Even if the client's hardware resources are modest, the user can interact with state‐of‐the‐art rendering applications executed on the server. Our solution focuses on augmented video information, e.g., by depth, which is key to increase robustness with respect to data loss, image reconstruction, and is an important feature for stereo vision and other client‐side applications. Two major challenges arise in such a setup. First, the server workload has to be controlled to support many clients, second the data transfer needs to be efficient. Consequently, our contributions are twofold. First, we reduce the server‐based computations by making use of sparse sampling and temporal consistency to avoid expensive pixel evaluations. Second, our data‐transfer solution takes limited bandwidths into account, is robust to information loss, and compression and decompression are efficient enough to support real‐time interaction. Our key insight is to tailor our method explicitly for rendered 3D content and shift some computations on client GPUs, to better balance the server/client workload. Our framework is progressive, scalable, and allows us to stream augmented high‐resolution (e.g., HD‐ready) frames with small bandwidth on standard hardware.

Pixel processing is becoming increasingly expensive for real-time applications due to the complexity of today's shaders and high-resolution framebuffers. However, most shading results are spatially or temporally coherent, which allows for sparse sampling and reuse of neighboring pixel values. This paper proposes a simple framework for spatio-temporal upsampling on modern GPUs. In contrast to previous work, which focuses either on temporal or spatial processing on the GPU, we exploit coherence in both. Our algorithm combines adaptive motion-compensated filtering over time and geometry-aware upsampling in image space. It is robust with respect to high-frequency temporal changes, and achieves substantial performance improvements by limiting the number of recomputed samples per frame. At the same time, we increase the quality of spatial upsampling by recovering missing information from previous frames. This temporal strategy also allows us to ensure that the image converges to a higher quality result.

Abstract Computing global illumination in complex scenes is even with todays computational power a demanding task. In this work we propose a novel irradiance caching scheme that combines the advantages of two state‐of‐the‐art algorithms for high‐quality global illumination rendering:lightcuts, an adaptive and hierarchical instant‐radiosity based algorithm and the widely used (ir)radiance caching algorithm for sparse sampling and interpolation of (ir)radiance in object space. Our adaptive radiance caching algorithm is based on anisotropic cache splatting, which adapts the cache footprints not only to the magnitude of the illumination gradient computed with light‐cuts but also to its orientation allowing larger interpolation errors along the direction of coherent illumination while reducing the error along the illumination gradient. Since lightcuts computes the direct and indirect lighting seamlessly, we use a two‐layer radiance cache, to store and control the interpolation of direct and indirect lighting individually with different error criteria. In multiple iterations our method detects cache interpolation errors above the visibility threshold of a pixel and reduces the anisotropic cache footprints accordingly. We achieve significantly better image quality while also speeding up the computation costs by one to two orders of magnitude with respect to the well‐known photon mapping with (ir)radiance caching procedure.

Abstract Currently 3D animation rendering and video compression are completely independent processes even if rendered frames are streamed on‐the‐fly within a client‐server platform. In such scenario, which may involve time‐varying transmission bandwidths and different display characteristics at the client side, dynamic adjustment of the rendering quality to such requirements can lead to a better use of server resources. In this work, we present a framework where the renderer and MPEG codec are coupled through a straightforward interface that provides precise motion vectors from the rendering side to the codec and perceptual error thresholds for each pixel in the opposite direction. The perceptual error thresholds take into account bandwidth‐dependent quantization errors resulting from the lossy com‐pression as well as image content‐dependent luminance and spatial contrast masking. The availability of the discrete cosine transform (DCT) coefficients at the codec side enables to use advanced models of the human visual system (HVS) in the perceptual error threshold derivation without incurring any significant cost. Those error thresholds are then used to control the rendering quality and make it well aligned with the compressed stream quality. In our prototype system we use the lightcuts technique developed by Walter et al., which we enhance to handle dynamic image sequences, and an MPEG‐2 implementation. Our results clearly demonstrate many advantages of coupling the rendering with video compression in terms of faster rendering. Furthermore, temporally coherent rendering leads to a reduction of temporal artifacts.

We present the design and implementation of a semantics-enabled service discovery framework in the data Grids for process and product development using numerical simulation and knowledge discovery (SIMDAT) Pharma Grid, an industry-oriented Grid environment for integrating thousands of Grid-enabled biological data services and analysis services. The framework consists of three major components: the Web ontology language (OWL)-description logic (DL)-based biological domain ontology, OWL Web service ontology (OWL-S)-based service annotation, and semantic matchmaker based on the ontology reasoning. Built upon the framework, workflow technologies are extensively exploited in the SIMDAT to assist biologists in (semi)automatically performing in silico experiments. We present a typical usage scenario through the case study of a biological workflow: IXodus.

Standard density estimation approaches suffer from visible bias due to low-pass filtering of the lighting function. Therefore, most photon density estimation methods have been used primarily with inefficient Monte Carlo final gathering to achieve high-quality results for the indirect illumination. We present a density estimation technique for efficiently computing all-frequency global illumination in diffuse and moderately glossy scenes. In particular, we compute the direct, indirect, and caustics illumination during photon tracing from the light sources. Since the high frequencies in the illumination often arise from visibility changes and surface normal variations, we consider a kernel that takes these factors into account. To efficiently detect visibility changes, we introduce a hierarchical voxel data structure of the scene geometry, which is generated on GPU. Further, we preserve the surface orientation by computing the density estimation in ray space.

Abstract We present a novel framework for efficiently computing the indirect illumination in diffuse and moderately glossy scenes using density estimation techniques. Many existing global illumination approaches either quickly compute an overly approximate solution or perform an orders of magnitude slower computation to obtain high‐quality results for the indirect illumination. The proposed method improves photon density estimation and leads to significantly better visual quality in particular for complex geometry, while only slightly increasing the computation time. We perform direct splatting of photon rays, which allows us to use simpler search data structures. Since our density estimation is carried out in ray space rather than on surfaces, as in the commonly used photon mapping algorithm, the results are more robust against geometrically incurred sources of bias. This holds also in combination with final gathering where photon mapping often overestimates the illumination near concave geometric features. In addition, we show that our photon splatting technique can be extended to handle moderately glossy surfaces and can be combined with traditional irradiance caching for sparse sampling and filtering in image space.

BACKGROUND: DNA methylation plays an important role in development and tumorigenesis by epigenetic modification and silencing of critical genes. The development of PCR-based methylation assays on bisulphite modified DNA heralded a breakthrough in speed and sensitivity for gene methylation analysis. Despite this technological advancement, these approaches require a cumbersome gene by gene primer design and experimental validation. Bisulphite DNA modification results in sequence alterations (all unmethylated cytosines are converted into uracils) and a general sequence complexity reduction as cytosines become underrepresented. Consequently, standard BLAST sequence homology searches cannot be applied to search for specific methylation primers. RESULTS: To address this problem we developed methBLAST, a sequence similarity search program, based on the original BLAST algorithm but querying in silico bisulphite modified genome sequences to evaluate oligonucleotide sequence similarities. Apart from the primer specificity analysis tool, we have also developed a public database termed methPrimerDB for the storage and retrieval of validated PCR based methylation assays. The web interface allows free public access to perform methBLAST searches or database queries and to submit user based information. Database records can be searched by gene symbol, nucleotide sequence, analytical method used, Entrez Gene or methPrimerDB identifier, and submitter's name. Each record contains a link to Entrez Gene and PubMed to retrieve additional information on the gene, its genomic context and the article in which the methylation assay was described. To assure and maintain data integrity and accuracy, the database is linked to other reference databases. Currently, the database contains primer records for the most popular PCR-based methylation analysis methods to study human, mouse and rat epigenetic modifications. methPrimerDB and methBLAST are available at http://medgen.ugent.be/methprimerdb and http://medgen.ugent.be/methblast. CONCLUSION: We have developed two integrated and freely available web-tools for PCR based methylation analysis. methBLAST allows in silico assessment of primer specificity in PCR based methylation assays that can be stored in the methPrimerDB database, which provides a search portal for validated methylation assays.

We present a new algorithm for computing indirect illumination based on density estimation similarly to photon mapping. We accelerate the search for final gathering by reorganizing the computation in the reverse order. We use two trees that organize spatially not only the position of photons but also the position of final gather rays. The achieved speedup is algorithmic, the performance improvement takes advantage of logarithmic complexity of searching in trees. The algorithm requires almost no user settings unlike many known acceleration techniques for photon mapping. The image quality is the same as for traditional photon mapping with final gathering, since the algorithm does not approximate or interpolate. Optionally, the algorithm can be combined with other techniques such as density control and importance sampling. The algorithm creates a coherent access pattern to the main memory. This further improves on performance and also allows us to use efficient external data structures to alleviate the increased memory requirements.