We present a novel approach for compression of depth maps based on Compressed Sensing and Sparse Coding. Our proposed scheme compresses and stores the depth map, and then-during the decompression step-makes use of the readily available additional RGB information to guide the reconstruction. We introduce additional constraints to the underlying optimization problem enforcing the correctness of the RGB image in the decompression step. A comparison with established compression schemes shows that our proposed method leads to a lower error rate at high compression ratios.

An algorithmic framework to compute sparse or minimal-TV solutions of linear systems is proposed. The framework includes both the Kaczmarz method and the linearized Bregman method as special cases and also several new methods such as a sparse Kaczmarz solver. The algorithmic framework has a variety of applications and is especially useful for problems in which the linear measurements are slow and expensive to obtain. We present examples for online compressed sensing, TV tomographic reconstruction and radio interferometry.

The linearized Bregman method is a method to calculate sparse solutions to systems of linear equations. We formulate this problem as a split feasibility problem, propose an algorithmic framework based on Bregman projections, and prove a general convergence result for this framework. Convergence of the linearized Bregman method will be obtained as a special case. Our approach also allows for several generalizations such as other objective functions, incremental iterations, incorporation of non-Gaussian noise models, and box constraints.

Abstract Astronomical nebulae are among the most complex and visually appealing phenomena known outside the bounds of the Solar System. However, our fixed vantage point on Earth limits us to a single known view of these objects, and their intricate volumetric structure cannot be recovered directly. Recent approaches to reconstructing a volumetric 3D model use the approximate symmetry inherent to many nebulae, but require several hours of computation time on large multi‐GPU clusters. We present a novel reconstruction algorithm based on group sparsity that reaches or even exceeds the quality of prior results while taking only a fraction of the time on a conventional desktop PC, thereby enabling end users in planetariums or educational facilities to produce high‐quality content without expensive hardware or manual modeling. In principle, our approach can be generalized to other transparent phenomena with arbitrary types of user‐specified symmetries.

Display resolution is frequently exceeded by available image resolution. Recently, apparent display resolution enhancement (ADRE) techniques show how characteristics of the human visual system can be exploited to provide super-resolution on high refresh rate displays. In this paper, we address the problem of generalizing the ADRE technique to conventional videos of arbitrary content. We propose an optimization-based approach to continuously translate the video frames in such a way that the added motion enables apparent resolution enhancement for the salient image region. The optimization considers the optimal velocity, smoothness, and similarity to compute an appropriate trajectory. In addition, we provide an intuitive user interface that allows to guide the algorithm interactively and preserves important compositions within the video. We present a user study evaluating apparent rendering quality and show versatility of our method on a variety of general test scenes.

We propose a method that extracts sparse and spatially localized deformation modes from an animated mesh sequence. To this end, we propose a new way to extend the theory of sparse matrix decompositions to 3D mesh sequence processing, and further contribute with an automatic way to ensure spatial locality of the decomposition in a new optimization framework. The extracted dimensions often have an intuitive and clear interpretable meaning. Our method optionally accepts user-constraints to guide the process of discovering the underlying latent deformation space. The capabilities of our efficient, versatile, and easy-to-implement method are extensively demonstrated on a variety of data sets and application contexts. We demonstrate its power for user friendly intuitive editing of captured mesh animations, such as faces, full body motion, cloth animations, and muscle deformations. We further show its benefit for statistical geometry processing and biomechanically meaningful animation editing. It is further shown qualitatively and quantitatively that our method outperforms other unsupervised decomposition methods and other animation parameterization approaches in the above use cases.

This paper proposes changing of picture orientation on a picture-by-picture basis, without requiring the encoding of an Instantaneous Decoder Refresh (IDR) picture. It can be implemented by small modifications to H.264/MPEG-4 AVC and the design of the upcoming High Efficiency Video Coding (HEVC) standard, or most other video coding technologies employing inter picture prediction. To enable picture orientation changes, a high layer syntax element is needed to indicate the rotation to be applied to the decoded pictures. Modifications to the decoding process are also required to avoid the use of IDR pictures. For some applications with handheld cameras that may be held at different orientations, the proposed modifications allow shifting the rotation function to the decoding end, reducing overall system complexity. Additionally, selection by the encoder of the coded picture orientation may result in better coding efficiency. In this respect, an algorithm for selecting the coded picture orientation limited to intra coding is also described, and experimental results show coding efficiency improvements up to 3.4% BD-rate gain.

We present an audio retargeting technique to create custom soundtracks for movies and games from existing audio material (typically music) by automatically rearranging audio segments. Constraints can be specified to make the length of the audio fit the length of a movie scene, or to align parts of a piece of music with particular events. Existing approaches typically create soundtracks with many unnecessary and often disruptive transitions. We extend a recent analysis and resynthesis method with a novel genetic algorithm for finding an optimal succession of audio segments that minimizes the number of audible transitions and repetitions as well as the deviation from user-specified constraints. Compared to prior work, our experiments with audio examples from different musical genres show a significant improvement with respect to the optimization criteria, and the resulting soundtracks contain few, if any, noticeable transitions.

This paper describes the integration of High Efficiency Video Coding (HEVC) into end-to-end multimedia systems, formats, and protocols such as Real-time transport Protocol, the transport stream of the MPEG-2 standard suite, and dynamic adaptive streaming over the Hypertext Transport Protocol. This paper gives a brief overview of the high-level syntax of HEVC and the relation to the Advanced Video Coding standard (H.264/AVC). A section on HEVC error resilience concludes the HEVC overview. Furthermore, this paper describes applications of video transport and delivery such as broadcast, television over the Internet Protocol, Internet streaming, video conversation, and storage as provided by the different system layers.

The increasing proportion of video traffic in telecommunication networks puts an emphasis on efficient video compression technology. High Efficiency Video Coding (HEVC) is the forthcoming video coding standard that provides substantial bit rate reductions compared to its predecessors. In the HEVC standardization process, technologies such as picture partitioning, reference picture management, and parameter sets are categorized as “high-level syntax.” The design of the high-level syntax impacts the interface to systems and error resilience, and provides new functionalities. This paper presents an overview of the HEVC high-level syntax, including network abstraction layer unit headers, parameter sets, picture partitioning schemes, reference picture management, and supplemental enhancement information messages.

The 3D visualization of astronomical nebulae is a challenging problem since only a single 2D projection is observable from our fixed vantage point on Earth. We attempt to generate plausible and realistic looking volumetric visualizations via a tomographic approach that exploits the spherical or axial symmetry prevalent in some relevant types of nebulae. Different types of symmetry can be implemented by using different randomized distributions of virtual cameras. Our approach is based on an iterative compressed sensing reconstruction algorithm that we extend with support for position-dependent volumetric regularization and linear equality constraints. We present a distributed multi-GPU implementation that is capable of reconstructing high-resolution datasets from arbitrary projections. Its robustness and scalability are demonstrated for astronomical imagery from the Hubble Space Telescope. The resulting volumetric data is visualized using direct volume rendering. Compared to previous approaches, our method preserves a much higher amount of detail and visual variety in the 3D visualization, especially for objects with only approximate symmetry.

Music and sound play an important role for the emotional involvement of the spectator in visual media such as movies, video games or artistic shows. However, the generation of suitable background music and atmospheric sound matching the visual context is a tedious and typically completely manual task. For content creators, it would be desirable to be able to specify a sound example and a few key points at which certain parts are to be played, and to then automatically generate a soundtrack subject to these constraints. For interactive media such as video games, on the other hand, example-based background sound could automatically adapt to the game events. We propose a new example-based audio synthesis method for both applications which works for musical and texture-like sound examples alike. Our algorithm is evaluated on a variety of musical styles, from classical music to punk rock, and on several texture-like atmospheric sounds.

We present a flexible interactive 3D morpho-kinematical modeling application for astrophysics. Compared to other systems, our application reduces the restrictions on the physical assumptions, data type, and amount that is required for a reconstruction of an object's morphology. It is one of the first publicly available tools to apply interactive graphics to astrophysical modeling. The tool allows astrophysicists to provide a priori knowledge about the object by interactively defining 3D structural elements. By direct comparison of model prediction with observational data, model parameters can then be automatically optimized to fit the observation. The tool has already been successfully used in a number of astrophysical research projects.

The theory of compressed sensing has a natural application in interferometric aperture synthesis. As in many real-world applications, however, the assumption of random sampling, which is elementary to many propositions of this theory, is not met. Instead, the induced sampling patterns exhibit a large degree of regularity. In this paper, we statistically quantify the effects of this kind of regularity for the problem of radio interferometry where astronomical images are sparsely sampled in the frequency domain. Based on the favorable results of our statistical evaluation, we present a practical method for interferometric image reconstruction that is evaluated on observational data from the Very Large Array (VLA) telescope.

Scalable video coding (SVC) and transmission has been a research topic for many years. Among other objectives, it aims to support different receiving devices, perhaps connected through a heterogeneous network structure, using a single bit stream. Earlier attempts of standardized scalable video coding, for example in MPEG-2, H.263, or MPEG-4 Visual, have not been commercially successful. Nevertheless, the Joint Video Team has recently focused on the development of the scalable video extensions of H.264/AVC, known as SVC. Some of the key problems of older scalable compression techniques have been solved in SVC and, at the same time, new and compelling use cases for SVC have been identified. While it is certainly important to develop coding tools targeted at high coding efficiency, the design of the features of the interface between the core coding technologies and the system and transport are also of vital importance for the success of SVC. Only through this interface, and novel mechanisms defined therein, applications can take advantage of the scalability features of the coded video signal. This paper provides an overview of the system interface features defined in the SVC specification. We discuss, amongst other features, bit stream structure, extended network abstraction layer (NAL) unit header, and supplemental enhancement information (SEI) messages related to scalability information.

The transport of scalable media, and in particular of scalable video conforming to the forthcoming Scalable Video Coding (SVC) technology, presents challenges not only in the video compression technology, but also in transport and signaling. This paper discusses the current status of standardization of the support for scalable media, and SVC in particular, over IP based networks. Both the transport of SVC over the Real-time Transport Protocol (RTP), and the signaling support-namely the additional mechanisms in the Session Description Protocol (SDP)-are covered. As it turns out, the support of SVC over RTP is not quite as straightforward as that of nonscalable video bit streams. Specifically, the signaling architecture requires an almost complete overhaul, and new protocol mechanisms need to be introduced into the packetization.

The scalable extension of H.264/AVC, known as scalable video coding or SVC, has recently been the main focus of the Joint Video Team. The higher level syntax of SVC follows the design principles of H.264/AVC. This allows the work towards an optimized file format and RTF payload format to be conducted in parallel with the core SVC specification. This paper provides an overview of the system and transport interface design of SVC, including the SVC high-level syntax and functionalities, SVC file format, and SVC RTF payload format.

In this paper, we propose a real-time, low-complexity video rate control algorithm designed to obey buffer constraints. The algorithm is optimized for streaming and local recording applications in mobile devices. Today, most mobile phones include a digital camera that can be used to capture video. The on-phone processor technology has become powerful enough to encode video in real-time. The resulting file can, for example, be archived in the phone's memory, or (progressively or as one block) downloaded, through the 3G mobile network, Bluetooth, or WLAN, to Internet-connected computer systems. From here, all forms of multimedia transmission, such as streaming, file sharing, or multimedia mail become possible. In local recording and streaming applications on a mobile phone, we assume that no memory for storage of uncompressed video is available. Therefore, look-ahead and multi-path rate control is not possible. Furthermore, considering the processing power and, more importantly, battery life constraints in mobile devices, the proposed algorithm needs to be as simple as possible. The described algorithm implements a variable bitrate (VBR) by controlling the quantization scale (QS) on a per picture basis. The QS is calculated based on two other QSs, which correspond to constant rate and constant quality rate controls. The algorithm utilizes the variable bitrate benefits as much as possible so as to minimize the variation of the QS scale, and to provide encoded video with high visual quality. Although it strictly obeys buffering constraints as discussed later, the experimental results show that it allows encoded video at average quality levels significantly higher than reported in earlier works

H.264 is the ITU-T's new, nonbackward compatible video compression Recommendation that significantly outperforms all previous video compression standards. It consists of a video coding layer (VCL) which performs all the classic signal processing tasks and generates bit strings containing coded macroblocks, and a network adaptation layer (NAL) which adapts those bit strings in a network friendly way. The paper describes the use of H.264 coded video over best-effort IP networks, using RTP as the real-time transport protocol. After a description of the environment, the error-resilience tools of H.264 and the draft specification of the RTP payload format are introduced. Next the performance of several possible VCL- and NAL-based error-resilience tools of H.264 are verified in simulations.

The JVT/H.26L video coding scheme conceptually consists of a video coding layer (VCL) responsible mainly for coding efficiency and a network abstraction layer (NAL) that supports video specific transport features for a variety of networks. This paper describes the H.26L/JVT NAL in general, including the parameter set concept and the transport over packet-based and bit-stream oriented networks. Encapsulation of coded video data to RTP/UDP/IP transport is discussed, and applications of H.26L in fixed and wireless IP-based transmission environments are presented.

Transmission of hybrid coded video including motion compensation and spatial prediction over error-prone channels results in the well-known problem of spatio-temporal error propagation at the decoder. A widely accepted standard-compliant technique to enhance the quality of the decoded video significantly is the more frequent introduction of intra-coded macroblocks. However, intra-coded information generally requires more bit rate. Therefore, a careful selection of intra-updates in terms of rate and distortion is necessary. A flexible and robust rate-distortion optimization technique is presented to select coding mode and reference frame for each macroblock. The channel statistics are included in the optimization process. We derive a method to obtain an estimate of the decoder pixel distortion at the encoder. The presented techniques are verified within the new H.26L/JVT video coding standard based on common test conditions.

Many of the newer video coding/compression standards, including MPEG-4 and H.263+ support some form of temporal scalability as part of their layered codec concept. This is generally realized by using bi-directionally predicted pictures (B-pictures), which use both an earlier and a subsequent P-picture as reference (anchor) pictures. This paper introduces a new form of temporal scalability employing only P-pictures. This mechanism results in improved real-time behavior (particularly lower latency) and a more flexible layering structure, at the cost of less efficient coding. The P-picture based temporal scalability mechanism is particularly useful for interactive multimedia communication on networks that offer several independent transport streams (possibly with different quality of service), but have sub-optimal real-time characteristics. Typical applications include both the Internet and some forms of mobile communication. The 1998 version of H.263 (known as H.263+ in both academia and industry) offers a mechanism to support P-picture scalability within the bit-stream through the reference picture selection mode. Other video coding standards, including those of the MPEG family, require slight modifications to the decoder in order to support the proposed mechanism.

Version 2 of ITU Recommendation H.263, better known as H.263+, includes a number of new mechanisms to improve coding efficiency and support various types of networks more efficiently. This paper provides an overview of the error resilience optional modes of H.263+ and describes the use of such modes in various multimedia network scenarios.