Extended Reality (XR) applications demand substantial bandwidth, ultra-low latency and high refresh rates, challenging network performance. While 3GPP Release 18 introduced the Protocol Data Unit (PDU) Set based Quality of Service (QoS) framework to enable application-aware packet handling for XR traffic, Release 19 advances this by addressing dynamic traffic characteristics with enhanced burst signaling, expedited transfer mechanisms for latency-critical content, and refined guidelines for PDU Set Importance (PSI). These updates improve power management, scheduling, and resource allocation, strengthening the foundation for intelligent network handling and seamless immersive XR experiences using 3GPP networks.

This paper introduces the Auxiliary I-Frame Splicing method to reduce bandwidth waste in adaptive streaming. This method involves fetching a high-quality I-frame and splicing it into the already downloaded low-quality segment, resulting in a higher-quality rendering at a lower overhead than replacing the entire low-quality segment. In our experiments with three videos and four quantization parameters, the results show that the bandwidth can be saved up to 87% while still increasing the peak signal-to-noise ratio score by 20% and the video multi-method assessment fusion score by 73%. In the demo, we demonstrate the visual differences between the original and spliced videos.

Omnidirectional MediA Format (OMAF) is the first worldwide virtual reality (VR) standard to store and distribute immersive media, completed in 2019. Later, in 2021, the second edition of this standard (OMAF v2) was published. The second edition kept all the features defined in the first OMAF edition while introducing some new ones, such as overlays and multi-viewpoints. OMAF v2’s Tile Index Segments that contain metadata to track fragment data per segment and quality levels create a bandwidth overhead. During the OMAF v2 standardization, multiple methods for the track fragment run representation were studied to deal with this overhead. This paper presents the implementation of one of these methods, the compressed box method using the DEFLATE algorithm (OMAF v2*). It also provides comprehensive test results of OMAF v1, OMAF v2 and OMAF v2* with various combinations of three tile grids (6x4, 8x6 and 12x8), three segment durations (300 ms, 900 ms and 3 s), two videos (RollerCoaster and Timelapse), two bitrate groups (each group with four different bitrates) and two HTTP versions (HTTP/1.1 and H2).

Viewport-dependent delivery (VDD) is a technique to save network resources during the transmission of immersive videos. However, it results in a non-zero motion-to-high-quality delay (MTHQD), which is the delta time from the moment where the current viewport has at least one low-quality tile to when all the tiles in the new viewport are rendered in high quality. MTHQD is an important metric in the evaluation of the VDD systems. This paper improves an earlier concept called viewport margins by introducing head-motion awareness. The primary benefit of this improvement is the reduction (up to 64%) in the average MTHQD.

Immersive 360-degree video delivery is more and more widespread. New use cases are constantly emerging and make it a promising video technology for Extended Reality applications. Viewport Dependent Delivery (VDD) is an established technique used for saving network bit rate when transmitting omnidirectional video. One of the hardest challenges in VDD of 360-degree video is how to ensure that the video quality in the user's viewport is always the highest possible, independent of the user's head motion speed and span of motion. This paper introduces the concept of viewport margins. These can be understood as an extra high-quality spatial safety area around the user's viewport. Viewport margins provide a better user experience for the receiver by reducing the Motion to High Quality Delay and the percentage of low-quality viewport seen by the user. We provide simulation results that show the advantage of using viewport margins for real-time low-delay VDD of 360-degree video. In particular, for a head motion of 90 degrees, using a 10-30% margins can reduce the percentage of viewport at low quality by 5-10% and using 30% margins reduces the motion to high quality delay to zero for head speeds up to 360 degrees per second, when the viewport feedback is sent every 33ms.

The second edition of the Omnidirectional MediA Format (OMAF) standard developed by the Moving Picture Experts Group (MPEG) defines two major features: overlays and multi-viewpoints. Overlays help in enhancing the immersive experience by providing additional information about the omnidirectional background video content. The multi-viewpoint feature enables the content to be captured/experienced from multiple spatial locations. These two powerful features along with interactivity, dispense the content provider with new possibilities of storytelling (for example, non-linear) using immersive media.

Omnidirectional videos are becoming more and more popular as display devices, such as Head Mounted Displays (HMD), become more user friendly. One of the biggest challenges with such video content is the great amount of bandwidth required for delivery at a good quality. The key difference between omnidirectional and the traditional content streaming is that with the former, users watch only part of the streamed content. Therefore, the non-watched areas of a streamed omnidirectional video are delivered unnecessarily. To reduce the network bandwidth requirements, the delivery mechanism could make use of Viewport-Dependent Streaming (VDS), according to which high-quality (HQ) video is maintained only in the current viewport, while lower quality video is streamed for the remaining part. In this paper, we examine the streaming data wastage, i.e., the parts of HQ video not watched by a user, in VDS when users linearly move from a generic viewport to another one within the 360 degrees space. We used both simulated trajectories as well real trajectories from 20 users. We show how the user's HMD exploration speed and the DASH transport segment lengths impact the streaming data wastage, leading up to 78% of wasted HQ content and up to 40% of total stream bit rate wasted due to users' motion. We also show how the users' trajectory prediction can decrease the streaming data wastage making the system up to 25% more efficient. This paper gives very useful hints that could lead to VDS optimizations, and it introduces quantitative estimates on how the data wastage characteristics change with respect to the streaming parameters.

Modern smartphones have become the most popular means for recording videos. In fact, thanks to their portability, smartphones allow for recording anything and at any moment of our everyday life. One common occasion is represented by sport happenings, where people often record their favourite team or players. Automatic analysis of such videos is important for enabling applications such as automatic organization, browsing and summarization of the content. This paper proposes novel algorithms for the detection of salient events in videos recorded at basketball games. The novel approach consists of jointly analyzing visual data and magnetometer data. The magnetometer data provides information about the horizontal orientation of the camera. The proposed joint analysis allows for a reduced number of false positives and for a reduced computational complexity. The algorithms are tested on data captured during real basketball games. The experimental results clearly show the advantages of the proposed approach.

This paper describes a user evaluation study of automated creation of mobile video remixes in three different event contexts. The evaluation contributes to the design process of the Automatic Video Remixing System, deepening knowledge to wider usage context. The study was completed with 30 users in three different contexts: a sports event, a music concert and a doctoral dissertation. It was discovered that users are motivated to provide their material to the service when knowing they get an automatically created remix containing many capturers' content in return. Automatic video remixing was stated to ease the task of editing videos and to improve the quality of amateur videos. The study reveals requirements for pleasurable remix creation in different event contexts and details the user experience factors related to the content capturing, sharing, and viewing of captured content and the remixes. The results provide insights into media creation in small event-based groups.

In this work we propose methods that exploit context sensor data modalities for the task of detecting interesting events and extracting high-level contextual information about the recording activity in user generated videos. Indeed, most camera-enabled electronic devices contain various auxiliary sensors such as accelerometers, compasses, GPS receivers, etc. Data captured by these sensors during the media acquisition have already been used to limit camera degradations such as shake and also to provide some basic tagging information such as the location. However, exploiting the sensor-recordings modality for subsequent higher-level information extraction such as interesting events has been a subject of rather limited research, further constrained to specialized acquisition setups. In this work, we show how these sensor modalities allow inferring information (camera movements, content degradations) about each individual video recording. In addition, we consider a multi-camera scenario, where multiple user generated recordings of a common scene (e.g., music concerts) are available. For this kind of scenarios we jointly analyze these multiple video recordings and their associated sensor modalities in order to extract higher-level semantics of the recorded media: based on the orientation of cameras we identify the region of interest of the recorded scene, by exploiting correlation in the motion of different cameras we detect generic interesting events and estimate their relative position. Furthermore, by analyzing also the audio content captured by multiple users we detect more specific interesting events. We show that the proposed multimodal analysis methods perform well on various recordings obtained in real live music performances.

The recent proliferation of mobile video content has emphasized the need for applications such as automatic organization and automatic editing of videos. These applications could greatly benefit from domain knowledge about the content. However, extracting semantic information from mobile videos is a challenging task, due to their unconstrained nature. We extract domain knowledge about sport events recorded by multiple users, by classifying the sport type into soccer, American football, basketball, tennis, ice-hockey, or volleyball. We adopt a multi-user and multimodal approach, where each user simultaneously captures audio-visual content and auxiliary sensor data (from magnetometers and accelerometers). Firstly, each modality is separately analyzed; then, analysis results are fused for obtaining the sport type. The auxiliary sensor data is used for extracting more discriminative spatio-temporal visual features and efficient camera motion features. The contribution of each modality to the fusion process is adapted according to the quality of the input data. We performed extensive experiments on data collected at public sport events, showing the merits of using different combinations of modalities and fusion methods. The results indicate that analyzing multimodal and multi-user data, coupled with adaptive fusion, improves classification accuracies in most tested cases, up to 95.45%.

We present a robust multimodal approach for classifying the sport genre in videos recorded by mobile phone users at a sport event. In addition to traditional audio-visual content analysis tools, we propose to analyze auxiliary sensor data (electronic compass data and accelerometer data) captured simultaneously with the video recording. By means of machine learning techniques, we build models of visual appearance, camera motion (from auxiliary sensor data) and audio scene, which are used for classifying the data from each modality. The sport genre is obtained by fusing the information provided by the models. We propose to use the quality of each modality as an indication of its reliability. Extensive experiments were performed on real test data collected at public sport events. We provide comparisons on the use of different modality sets and fusion methods. Finally, we show how the proposed methods achieve robust classification even in the considered unconstrained scenarios.

Digital memorabilia, such as video remixes, can increase the value of attending music events. Remixes can be made using video clips recorded by attendees during the event; however, producing them is a laborious task. In this paper we study the prospects of an automatic video remixing and present the results of a study on users' perceptions and attitudes towards collaborative automatic mobile video production. The three findings are as follows: People assess automatic video remix memorabilia as fairly equal to amateur-made manual ones, even if the manually-created video remixes are better in overall quality; as a remixing actor, a computer can be perceived to be more trustworthy than a human remixer; and, the quality of the video remix and the publication forum of the remix outcome plays a significant role when people are deciding whether or not they need public acknowledgement for their contribution. We conclude by discussing the design implications for collaborative automatic mobile video production.

This paper presents a multimedia streaming service in a mobile (3G) environment that, in addition to in-band congestion signals such as packet losses and delay variations, receives congestion cues from a Network Coverage Map Service (NCMS) to make rate-control decisions. The streaming client routinely queries the NCMS to assess the network conditions at future locations along its expected path. The streaming client may ask the streaming server for short-term transmission bursts to increase pre-buffering when it is approaching areas with bad network performance to maintain media quality. If needed, the client may also switch to a different encoding rate (rate-switching) depending on the severity of expected congestion. These notifications are scheduled as late as possible, so that any changes in network conditions and/or changes in user's movements can be taken into account (late scheduling). Using this type of geo-predictive media streaming service we show that the streaming client can provide pause-less playback and better quality of experience to the user.

Heterogeneous networks are made up of wireless and wired links. The wireless link quality may vary due to interference and fading phenomena and, peered with radio and link layer reconfigurations lead to varying error rates, latencies, and, most importantly, changes in the available bit rate. In both fixed and wireless networks, varying amounts of cross traffic from other nodes may lead to fluctuations in queue size (reflected again in a path latency) and to congestion (reflected in packet drops from router queues). We propose a rate-adaptation algorithm for conversational video communication in a heterogeneous environment and use the standardized RTCP extensions as a starting point. We use metrics such as PSNR, loss rate, bandwidth utilization and fairness to evaluate the proposed algorithm.

Recording and publishing mobile video clips from music concerts is popular. There is a high potential to increase the concert's perceived value when producing video remixes from individual video clips and using them socially. A digital production of a video remix is an interactive process between human and computer. However, it is not clear what the collaboration implications between human and computer are.

Nowadays most camera-enabled electronic devices contain various auxiliary sensors such as accelerometers, gyroscopes, compasses, GPS receivers, etc. These sensors are often used during the media acquisition to limit camera degradations such as shake and also to provide some basic tagging information such as the location used in geo-tagging. Surprisingly, exploiting the sensor-recordings modality for high-level event detection has been a subject of rather limited research, further constrained to highly specialized acquisition setups. In this work, we show how these sensor modalities, alone or in combination with content-based analysis, allow inferring information about the video content. In addition, we consider a multi-camera scenario, where multiple user generated recordings of a common scene (e.g., music concerts, public events) are available. In order to understand some higher-level semantics of the recorded media, we jointly analyze the individual video recordings and sensor measurements of the multiple users. The detected semantics include generic interesting events and some more specific events. The detection exploits correlations in the camera motion and in the audio content of multiple users. We show that the proposed multimodal analysis methods perform well on various recordings obtained in real live music performances.

The Mobile and Interactive Social Television (MIST) demo introduces the concept of watching TV/Video content with people of interest via the mobile in a virtual shared space. In this shared space, the participants can interact and watch content together with others. The participants can talk to each other, see each other and also send text messages while watching the content. The coupling of content consumption and interaction allows users to have a common shared context of the viewing experience which creates a social experience that is both participative and interactive. The user feedback reveals that there is a clear and positive benefit and added value to the viewing experience.

Smart phones are becoming more and more powerful. Services that were traditionally designed for a static environment can now be implemented into mobile devices. One such service is Interactive Social TV, which allows geographically dispersed people to meet in a virtual shared space and watch TV while being able to interact with each other. This paper presents two novel architectures of a Mobile and Interactive Social TV system. In both of the architectures, the interaction is represented by a rich audio-visual media, allowing users to hear and see each other. In the first architecture, the mixing of the TV content with the interaction media is performed at the server side. In the second architecture, the mixing is performed in each client device. The issues of decoding and rendering simultaneous content and interaction media streams on a mobile device are discussed, and the related implementation is presented.

This paper studies the consumer experience for Mobile and Interactive Social TV, which allows users to create an experience of watching together. A focus group study approach was used. The study found that the feeling of social presence of people of interest when watching content with them was considered to add value to the viewing experience. Users found value in availability of all (in this case audio, video, text) interaction modalities. The key requirement is the ability for selective enabling/disabling of individual interaction features as per the user preferences and context. The context was considered to be influenced by both the relation with other participants and the content being consumed. Sports, news, user generated content, short clips were considered to be most suited for social consumption in the mobile context. Battery life, cost and network quality were the major concerns for adopting such a system in a real-world environment.

Communication in 3G networks may experience packet losses due to transmission errors on the wireless link(s) which may severely impact the quality of video services, with conversational video being most challenging to repair due to tighter delay constraints. Many error resilience mechanisms have been developed that can be applied at the source (codec) level and transport/application layer to address these challenges. Their respective performance varies depending on the network conditions. This paper analyzes and compares the performance of four error resilience mechanisms under different realistic wireless link conditions: selective retransmissions, slice size adaptation, reference picture selection, and unequal error protection using packet-based forward error correction. We derive suggestions for the applicability of the individual mechanisms.

During the past few years a new trend, user generated content, has gained more and more popularity. Current website-based sharing services provide already means to share content to a group of other users, but it is not possible to share content with other mobile users at a competitive cost. The increasing use of mobile devices in our daily life will create demand for the new types of services in the usergenerated content field. This paper presents a personal mobile broadcasting system as an extension to the currently existing 3GPP MBMS system. A new user-level interface between the BM-SC and the personal mobile broadcaster is proposed. Also some new functionality, to be able to restrict the delivery only to a subset of user population, are presented in the paper.

In literature, many studies about human perception of lip synchronization refer to experiences based on TV sets. In these cases, researchers give some hints on how easily humans percept lip synchronization problems. The in-sync region is typically known to be in the range -80ms to +80ms. Within this range most of the test candidates does not detect any lip synchronization skew. The out-of-synch region typically falls outside the range between -160ms and +160ms, where nearly everybody detects the lip synchronization error. Between those boundaries there is a transient area, where the detection of any lip synchronization problem depends on speaker size (head, shoulder or body view). These earlier assumptions based on TV environment do not hold for mobile environment, where the viewing characteristics and of the device itself are different. Therefore, further research is needed. The purpose of this paper is to find new results for human perception of lip synchronization in mobile environment. Results will show that new lip synchronization thresholds apply when using mobile phones and mobile media concepts.

IP-based Multimedia creation and consumption is becoming available on an increasing spectrum of devices ranging from low-powered portable devices like cell phones and PDA's to high powered static devices like desktops PCs and IPTVs. High-speed wireless and wire-line network access is becoming widespread. Multimedia services like IPTV, Video-on-demand and Video-conferencing are becoming main stream. These developments have paved the way for creating innovative concepts like session mobility for multimedia applications. Session mobility enables seamless transfer of an on-going multimedia session between different devices based on user preferences. We examine the motivations and use cases for session mobility. In this paper we evaluate architectural design constraints and propose architectural solutions for session mobility. We evaluate current session mobility mechanisms and propose requirements for new mechanisms.

Streaming is one of the key 3G mobile multimedia services provided by network operators. Traditional streaming applications need to be redesigned in order to work well over wireless. 3GPP has standardized streaming services for Release 4, 5 and 6 specifications. The last release overcomes the technical challenges posed by wireless and best effort networks. This paper describes the solution standardized in Release 6 for adaptive streaming.

Third generation (3G) mobile phones are leading the telecommunications world into a new era. 3G networks offer increased speed for mobile users and enable new services. Mobile streaming applications benefit from these new capabilities, although streaming doesn't strictly requires a 3G network to work. The purpose of this paper is to show how 3G can improve the performance of streaming applications. In particular, we reference the 3GPP packet-switched streaming service (PSS).

As the adoption of 2.5G and 3G systems grows, multimedia streaming is one of the services that operators will increasingly seek to provide. The multimedia streaming solutions that have successfully been deployed commercially on the Internet in the past are all based on proprietary technologies. However, existing streaming applications need to be considerably rethought to work well over wireless. The 3GPP standardization body has standardized streaming services for Release 4, 5 and 6 specifications. The Release 6 advanced streaming service overcomes the technical challenges posed by wireless and best effort networks. A standard solution for advanced multimedia streaming will ultimately drive adoption by operators and users. The paper describes the solution standardized in 3GPP(2) for adaptive streaming.

Mobile streaming is thought to be mainly a 3G service offered with guaranteed bit rates, error rates and delay bounds. In this paper we analyze the feasibility of streaming services in a 2.5G environment. GPRS Rel. '97 allows enough bandwidth for streaming, but it cannot guarantee bit rates or delays. We study different QoS parameter configurations to find optimal values for streaming over GPRS Rel. '97. Its error protection is good enough, if the correct QoS profile is chosen. GPRS cell reselections cause additional problems for streaming, since several seconds of data can be lost Results show that despite all the limitations, streaming is possible in a GPRS environment and this service can be started even before the deployment of 3G networks.

In this paper, call setup time in Session Initiation Protocol (SIP)-based videotelephony is analyzed. We used a 3G network emulator to measure post-dialing delay, answer-signal delay and call-release delay. The results are compared to local, national, international and overseas intranet LAN calls. Furthermore, we have also studied the effect SIP calls over lossy channels with restricted bandwidth, typical of mobile network signaling bearers.

This paper is a contribution to studying the feasibility of multimedia applications over GSM HSCSD networks. We report simulation results of a mobile videophone application at bit rates ranging from 14.4 to 57.6 kbit/s. We focus on QoS of video data in both mobile-to-land and mobile-to-mobile H.324 connections. The results obtained in terms of visual quality, received PSNR, delay, and frame rate are promising, especially at bit rates of 28.8 kbit/s or higher.