Depth map sequences are commonly compressed using standard video codecs. However, the bitdepth of acquired depth data often exceeds the maximum bitdepth supported by deployed codecs, resulting in significant quantization loss. To overcome this, depth data are frequently mapped on multiple streams before encoding. In this paper, we propose a depth range decomposition of a high-bitdepth source depth map sequence into two lower-bitdepth sequences: a depth band mask sequence and a residual map sequence. The former indicates the depth band for each pixel, while the latter carries the residual within that band. The presented transformation is lossless and exploits the piecewise smoothness of depth data. Both sequences can be compressed independently with any standard video codec, allowing seamless integration into existing video processing pipelines.

Modern information exchange and telecommunication systems make immersive media increasingly more relevant. Immersive applications introduce challenges in ensuring optimal performance and scalability of media decoding, composition, and synchronization operations. Nowadays, a solution to accommodate media decoding in this context is to use multiple parallel video decoder instances instead of a single video decoder. This would require a robust and predictable decoder management system, coupled with dynamic buffer organization. In practice, however, this is often implemented without any special curation to manage the simultaneous video decoder instances, with resulting behaviour and implications not being fully understood yet. In this paper, we extend our previous work by utilizing the VidBench tool and investigate further the performance of modern chipsets integrated into Android-operated mobile devices in handling multiple video decoder instances running in parallel. Our results indicate the need for suitable synchronization solutions to counter uncertainty and variability across different video decoder instances and devices.

Acquiring depth data has become easily achievable with advancements in depth sensing and depth estimation technologies. As a result, obtaining a depth stream to describe the topology of a corresponding video stream has been considerably simplified. Presence of a depth stream offers numerous benefits, including the integration of advanced visual enhancements to the corresponding video stream in a flexible and efficient manner. This can enrich video-centric applications and facilitate their transition to Augmented Reality (AR) environments, where processing capabilities and battery power are limited. In this paper, we introduce VidDepth, an application developed for mobile devices to demonstrate examples of visual enhancements in video playback scenarios across both traditional and AR settings.

Designing XR applications creates challenges regarding the performance and the scaling of media decoding operations, composition and synchronization of the various assets. Going beyond the single decoder paradigm of conventional video applications, XR applications tend to compose more and more visual streams such as 2D video assets but also textures and 2D/3D graphics encoded in video streams. All this demands a robust and predictable decoder management and a dynamic buffer organization. However, the behaviour of multiple decoder instances running in parallel is yet to be well understood on mobile platforms. To this end, we present in this paper VidBench - a parallel video decoding performance measurement tool for mobile Android devices. With VidBench, we quantify the challenges for applications using parallel video decoding pipelines with objective measurements and subjectively, we illustrate the current state of decoding multiple media streams and the possible visual artefacts resulting from unmanaged parallel video pipelines. Test results provide hints on the feasibility and the potential performance gain of using technologies like the MPEG-I Part 13 - Video Decoding Interface for immersive media (VDI) to alleviate those problems. We briefly present the main goals of VDI, standardised by the SC29 WG3 Moving Picture Experts Group (MPEG) Systems, which introduces functions and related constraints for optimizing such decoding instances as well as relevant video decoding APIs on which VDI is building upon such as the Khronos Vulkan Video extension.

Video call systems rely on being able to capture and transmit a self view, while at the same time rendering the view of the other party. Due to the lack of inwards facing cameras in XR devices (AR Glasses, HMD etc.) this is not a straightforward process. As a solution, recent XR teleconferencing platforms are trying to create a more "immersive" experience by replacing the self view with avatars, placing 3D models in space, creating shared spaces and other engaging features; approaches that are quite demanding and even then do not create a "traditional" teleconferencing experience. In this work, we are using an XR device (AR Glasses, or smartphone) to create a seamless and natural video calling experience. By using the AR Glasses to record an existing self-view from a reflective surface, like a mirror, the user is able to easily conduct a video call with a party, even if they are using a different setup. To demonstrate this concept, we present the MiroAR application. We conclude this paper by discussing the roadmap, shortcomings and possible extensions of our work.

Emerging video applications, such as cloud gaming and cloud rendering, rely on time critical decoding operations so that user interactions are reflected on the user's screen as soon as possible. In these applications content is encoded, transmitted and decoded before finally being displayed on the user's screen with each of these steps adding latency. Furthermore, often between the output of a decoded frame and its display additional operations may be required, such as up/downscaling, UI overlays or color space conversions. Current decoders only release the decoded samples once a full frame has been decoded, although under such low delay requirements, it would be beneficial if applications could start to process the decoded samples prior to the complete decoding of a coded picture. This would allow for parallel processing of a frame in the decoder and throughout the display pipeline, which could ultimately allow for lower latencies and eventually pave the way for more enhancement operations without inducing latency. We modified a decoder for the newly published Versatile Video Coding (VVC) codec to demonstrate that it is possible to output partial frames. The changes in the decoder to enable such partial output are straightforward and allow for the new partitioning schemes to be leveraged for partial output. The demonstrator will show how the progressive output can then be used by elements further on in the decoding/display pipeline.

Effective haptic feedback in virtual reality (VR) is an essential element for creating convincing immersive experiences. To design such feedback, state-of-the-art VR setups provide APIs for programmatically generating controller vibration patterns. While tools for designing vibrotactile feedback keep evolving, they often require expert knowledge and rarely support direct manipulation methods for mapping feedback to user interactions within the VR environment. To address these challenges, we contribute a novel concept called Weirding Haptics, that supports fast-prototyping by leveraging the user’s voice to design such feedback while manipulating virtual objects in-situ. Through a pilot study (N = 9) focusing on how tactile experiences are vocalized during object manipulation, we identify spatio-temporal mappings and supporting features needed to produce intended vocalizations. To study our concept, we built a VR design tool informed by the results of the pilot study. This tool enables users to design tactile experiences using their voice while manipulating objects, provides a set of modifiers for fine-tuning the created experiences in VR, and allows to rapidly compare various experiences by feeling them. Results from a validation study (N = 8) show that novice hapticians can vocalize experiences and refine their designs with the fine-tuning modifiers to match their intentions. We conclude our work by discussing uncovered design implications for direct manipulation and vocalization of vibrotactile feedback in immersive virtual environments.

With omnidirectional videos, the viewer is able to direct her Field-of-View (FoV) to any part of the scene while watching the content. This is achieved by rendering the 360 video content on the inside of a (conceptual) sphere in which the viewer is typically placed at the center. This is in contrast with traditional video that is rendered on a 2D plane and the viewer is watching always through a viewport directed by the content creator. These two approaches create a conflict between user experience and creativity, since omnidirectional video provides the user with viewing freedom, while traditional video allows for greater artistic expression by controlling the viewport. In order to combine these two approaches we propose an immersive setup in which the content changes between free-form viewing of omnidirectional (360 video mode) and directed viewing of traditional videos (director's mode). In this demo paper we present the benefits and reasoning behind this proposal and the means to implement it using the OMAF (MPEG-I - Part 2) standard.

This paper presents the design and evaluation of a localized crowd-sourced multimedia production and distribution system to enable today’s mobile producers of multimedia content to serve their content in real-time to other nearby users, while at the same time drastically reducing the bandwidth as compared to traditional crowd-sourced multimedia services. To achieve this, we created a modular system for many-to-many live production and distribution and deployed it within the media delivery platform developed in the FLAME project. The FLAME platform provides distributed edge computing as well as a programmable network infrastructure.

Currently on the market there is a plethora of affordable dedicated cameras or smartphones, able to record video and timed geospa-tial data (device location and orientation). This timed metadata can be used to identify relevant (in time and space) recordings. However, there has not been a platform that allows to exploit this information in order to utilize the relevant recordings in an interactive consumption scenario. In this paper we present SWAPUGC, a browser-based platform for building applications that use the accompanying geospatial data to dynamically select the streams for watching an event (or any spatiotemporal reference point). The view selection can be performed either manually, or automatically by a predefined algorithm that switches to the most suitable stream according to the recording characteristics. SWAPUGC is a research tool to test such adaptation algorithms and it is provided as an open-source project, accompanied by an example demo application and references to a compatible dataset and recorder. In this paper, we explain and then demonstrate the capabilities of the platform by an example implementation and examine future prospects and extensions.

In multi-hop IEEE 802.15.4 networks with nodes operating in the beacon-enabled mode, the time and energy spent in the cluster-tree construction may be long and highly variable especially if there are no a priori restrictions on duty-cycle durations or on the channels used in the network. In this paper, we propose a Multi-Channel Beacon Train (MCBT) protocol in which coordinator nodes send trains of beacons on all channels at random instants during the inactive part of a superframe, so that nodes entering the network can quickly acquire the network parameters or routing metrics and join the network. The scheme drastically shortens the delay for topology construction and lowers the consumed energy.