The past few years has witnessed a promising surge in immersive media technologies, however, a widespread adoption is still some years away. Recently, virtual reality (VR) head-mounted displays with inside-out tracking and hand-tracking are becoming popular in various fields. Representing users' hands and fingers in VR is crucial for many tasks. This is because hand-tracking enables a more natural, direct, interaction with a virtual environment (VE) which, compared to traditional handheld controllers, improves the VR immersive experience. This paper investigates user-perceived quality of experience for a reach-pick-place task inside VR. It presents a performance assessment methodology based on in-game analytics to evaluate user experience for different VR interaction types. We ran a controlled experiment (N=33) comparing two commodities: controller vs. hand-tracking. Results indicate a significant difference between the two data sets. The performance analytics show lower play-durations and trigger frequencies for the handheld controller compared to hand-tracking. The perceived mental workload for the tasks was also evaluated using Rated Scale Mental Effort (RSME). The reported scores were higher when using hand-tracking. The results help our understanding of the two interaction modalities in terms of their viability for naturalistic experiences akin to real-world scenarios.

3D visualization has witnessed exponential growth driven by advances in computer-generated imagery that now include immersive technologies enabling first-person experiences with depth perception and spatial awareness. We investigated two popular uses of VR in 3D architectural visualization: a “passive walkthrough” vs. an “interactive walkthrough”. We designed a within-subject experiment to measure the user-perceived quality and conduct a behavior analysis of users in both experiences. All participants (N=34) were exposed to both conditions and afterwards responded to a post-experience questionnaire. We recorded the physical activity of all participants while they were immersed within the virtual environments and each session was logged in a time diary. So far QoE measurements have relied on subjective and objective evaluations. In this paper, we discuss the behavioral analysis of the effects of immersion and interaction on the simple active behaviors (movements + gestures) of the users. We apply quantitative behavioral observation to cross-examine user behavior against their self-reported responses to a “presence” questionnaire. We conclude that there is significant potential for applying cross-disciplinary behavior analysis tools to overall Quality of Experience within virtual environments.

This study introduces a Quality of Experience (QoE) model of loudspeaker-based spatial speech reproduction, which specifies quality elements and quality features relevant to overall listening experience and Quality of Service (QoS), respectively. Assumptions about the relations between selected quality elements and quality features were validated in a listening-only test. Participants had the task to behaviorally identify the voices of two talkers and provided subjective judgments on category rating scales. Sentences uttered by the two talkers were alternately presented through only a central loudspeaker (non-spatial mode) versus through either the central or one talker-specific lateral loudspeaker (spatial mode). The transmitted speech signals were clean, superimposed with background noise or bandpass-filtered. Results showed that transmission quality, but not reproduction mode significantly affected evaluative (speech quality, speech intelligibility), immersion (voice naturalness, spatial presence, social presence) and task-related (voice similarity, talker identification effort) aspects of overall listening experience. Contrary to previous findings, the spatial mode did not reduce the effort of talker identification. This suggested that noticeable advantages from spatial reproduction may not be evident in relatively simple listening situations, but could manifest if complexity is increased. Besides their task-related influence, noise and spectral distortion should be mitigated when targeting a higher level of immersion.

The P3 component of the event-related brain potential (ERP) is promoted as an indicator of perceived change in speech quality. However, previously identified effects of quality change on P3 might actually be traced back to modulations in earlier portions of the ERP signal, evoked by physical variability between test stimuli. The present study introduces a rigorous control procedure to isolate late ERP signal portions by eliminating such physical stimulus confounds. Due to successful removal of auditory evoked activity, observed quality-related ERP signal modulations can be attributed to P3 with higher validity. The implemented control procedure promises to form an essential part of future experimental designs in psychophysiological assessment of multimedia quality.

In this paper, the immersiveness of three variations of spatial content was tested and compared. Content A is a high quality architectural visualization, which is characterized as purely spatial immersion. Content B is the best goal moments of real football games, which is mainly spatial immersion, with slight tactical immersion focus, compiled with mixed qualities. And content C is a high quality recorded virtual game animation, which is mainly spatial immersion, with slight emotional immersion focus. Each of the three spatial contents was cut into different lengths of 3 min, 7 min and 11 min. The participants report the ratings of their immersive experience on a 34-item questionnaire, after watching a combination of three media clips fully randomized in content types and durations, on a 10-inch tablet. Results show that overall, 7 min duration allows the users to feel significantly greater immersive experience than 3 min and 11 min durations for content A and content C. And for content B, 3 min duration stands out as the most immersive. Our study suggests that it is not the longer the more immersive, but there is an optimal duration for spatial immersion (around 7 min). After that, if there is not enough dramaturgical structure to sustain the audience interest, the immersiveness of the spatial content would significantly diminish (i.e. immersion turns into boredom). Our study also shows that realism factors also play a crucial role in inducing spatial immersion.

Enabling high Quality of Experience (QoE) with video-conferencing solutions based on Web Real-Time Communication (WebRTC) protocols anywhere, anytime, is challenging and triggers the exploration of new ways to gain QoE insights. In this paper, we share initial observations from a within-subjects experiment (N = 22) in which 2-party WebRTC-based audiovisual conversations took place under varying technical conditions. We collected self-report data, peripheral physiological data and application-level performance statistics with the overall aim of exploring their usefulness and compatibility for assessing QoE in the context of WebRTC. Our preliminary findings indicate that the varying quality is well-reflected in the self-reported overall quality and annoyance. However, this might not apply when considering the emotional valence ratings and ECG data, implying that other factors may play a role here.

In this paper, we defined and measured two types of immersion, namely, spatial immersion and emotional immersion, on flat 2D screen displays and using a 33-item questionnaire. Our results show that emotional immersion is significantly more immersive than spatial immersion in terms of sense of “being there”, time perception, realism, sense of engagement, emotional aspects, sensory cues, etc. Spatial immersion is almost as immersive as emotional emotion in terms of attention and image motion, and spatial immersion is more immersive than emotional immersion in terms of spatial dis-orientation. Our results also show that there are individual differences in the perception of immersion. Finally, we linked parameters of measurement with QoE influencing factors, in an aim to bridge the gap in the QoE assessment and modelling of immersive experience in storytelling.

Many modern applications exhibit dynamic behavior, which can be exploited for reduced energy consumption. We employ a two-phase combined design-time/run-time methodology that identifies different run-time situations and clusters similar behaviors into system scenarios. This methodology is integrated with our framework for system scenario based designs, which dynamically tune the hardware to match the application behavior. We achieve significant energy reductions for an extracted control structure of a video codec widely used in hand-held devices today. We encode a video stream consisting of different frame sizes based on measured available wireless bandwidth. Energy consumption is measured with a modified microcontroller board from Atmel with two alternative voltage and frequency settings. While maintaining the perceptual video quality and frame rate, our method results in up to 44% energy reduction for our encoded streams, even after including an average worst-case tuning overhead of 8.5%. In reality this overhead is typically negligible since the worst-case assumes very frequent tuning, while in realistic situations it will be performed at a much lower rate. This makes the expected gain over 50%.

In this tutorial we are going to elaborate on how technology and art can contribute in stimulating creative placemaking in public-use spaces. This presentation will highlight the authors approach and several other past and potential approaches that embrace on technology and connectivity to refocus on a creative function in public spaces that aims to move away from traditional planning and textbook concepts. Our approach, which is demonstrated by our design and implementation of a multi-use public space called "Adressaparken" is aimed to create a space towards greater flexibility, changeability, and appreciation for the public`s contribution. The presenters will also identify key lessons learned in the design and implementation of a technology re-usable public space. Potential variables needed to address the challenges in the integration and evaluation of digital public art installations will also be elaborated. By surveying past and present exhibited contributions and the presenters' own contributions in a public realm, this tutorial will offer guidance to artists, designers, researchers and practitioners on how they can successfully blend in to this interdisciplinary area of research.

Automatic license plate recognition (ALPR) from vehicle images plays an important role in intelligent traffic management. A critical problem is that the cameras can be dirtied such that the captured images are unrecognizable. This paper defines a new concept of Quality of Dirty (QoD) of captured images indicating when the dirtied cameras need to be cleaned. An accurate image QoD metric is proposed based on relevant image features and support vector regression (SVR), and it can handle a tricky issue of differentiating captured imaged containing dirtied vehicles from images captured by a dirtied camera. A subjective assessment to collect ground-truth QoD scores of real captured images has also been conducted. Experiments have demonstrated that the proposed image QoD metric achieves high accuracy when predicting the dirtied degree of cameras in the tunnel environment.

Humans are known to be good in creating pleasure technologies. In fact, some evolutionary psychologists believe that aesthetic experiences are biologically hardwired. This article reviews how the pleasure stimulus concept can be explored to enhance human experience in digital media. It is argued that the individual's consumption motivation is more than the by-products of the biological pleasure circuits. For instance, in daily life, one experiences various information processing, some of which does not emerge into one's explicit consciousness but relevantly contributes to one's experience. Although craving or desiring is mostly an explicit process, it can also manifest as an unconscious aspect of experience leading to an irrational or intrusive thoughts that can in turn alter or contribute to the aesthetic character of an experience. Using Quality of Experience assessment methodologies and extensive literature from a multidisciplinary standpoint, this article shows that intrusive mental concepts on food craving saliently affect the user's aesthetic experience and perception of quality in digital media.

Contrast sensitivity of the human visual system to visual stimuli can be significantly affected by several mechanisms, e.g., vision foveation and attention. Existing studies on foveation based video quality assessment only take into account static foveation mechanism. This paper first proposes an advanced foveal imaging model to generate the perceived representation of video by integrating visual attention into the foveation mechanism. For accurately simulating the dynamic foveation mechanism, a novel approach to predict video fixations is proposed by mimicking the essential functionality of eye movement. Consequently, an advanced contrast sensitivity function, derived from the attention driven foveation mechanism, is modeled and then integrated into a wavelet-based distortion visibility measure to build a full reference attention driven foveated video quality (AFViQ) metric. AFViQ exploits adequately perceptual visual mechanisms in video quality assessment. Extensive evaluation results with respect to several publicly available eye-tracking and video quality databases demonstrate promising performance of the proposed video attention model, fixation prediction approach, and quality metric.

This paper proposes a full-reference video quality metric based on foveated vision mechanism. Due to a non-uniform distribution of photo-receptors on the retina, the human visual system (HVS) has the highest resolution around the fixation point of eyes and dramatically decreases away from this point. Two key factors in the foveated vision are fixation point and retinal eccentricities of different visual objects. Based on an advanced video attention model in quality assessment scenarios, eye fixations are predicted from the attention map using a winner-takes-all (WTA) neural network. Four quality features describing distortions on luminance, spatial and temporal activities, as well as chrominance are derived between foveated representations of reference and distorted video frames. These quality features are then combined together by an appropriate spatiotemporal pooling scheme to build a video quality metric. Experimental results with respect to publicly available video quality databases demonstrate that the proposed quality model outperforms a previously proposed foveated video quality metric as well as state-of-the-art video quality models.

Cross talk is one of the most annoying distortions in the visualization stage of stereoscopic systems. Specifically, both pattern and amount of cross talk in multi-view auto-stereoscopic displays are more complex because of viewing angle dependability, when compared to cross talk in 2-view stereoscopic displays. Regarding system cross talk there are objective measures to assess it in auto-stereoscopic displays. However, in addition to system cross talk, cross talk perceived by users is also impacted by scene content. Moreover, some cross talk is arguably beneficial in auto-stereoscopic displays. Therefore, in this paper, we further assess how cross talk is perceived by users with various scene contents and different viewing positions using auto-stereoscopic displays. In particular, the proposed subjective cross talk assessment methodology is realistic without restriction of the users viewing behavior and is not limited to the specific technique used in auto-stereoscopic displays. The test was performed on a slanted parallax barrier based auto-stereoscopic display. The subjective cross talk assessment results show their consistence to the system cross talk meanwhile more scene content and viewing position related cross talk perception information is provided. This knowledge can be used to design new cross talk perception metrics.

Contrast sensitivity is an important characteristic of the human visual system (HVS), which is widely used in image and video signal processing. For visual quality assessment, static spatio-temporal frequency based contrast sensitivity function (CSF) is often used, while the contrast sensitivity can also be affected by smooth pursuit of eyes when tracking attentive regions in the field of view. This paper proposes to tune CSF based on an attention map derived from a visual attention model. The tuned CSF formulated by spatial frequency, temporal velocity, and visual attention map is used to filter video signals in order to construct a quality metric according to the difference of the filtered signals between a reference video and its distorted version. Experimental results demonstrate that the proposed attention tuned spatio-velocity CSF outperforms the traditional spatio-temporal CSF in evaluating the perceived video quality.

Stereoscopic three-dimensional (3-D) services do not always prevail when compared with their two-dimensional (2-D) counterparts, though the former can provide more immersive experience with the help of binocular depth. Various specific 3-D artefacts might cause discomfort and severely degrade the Quality of Experience (QoE). In this paper, we analyze one of the most annoying artefacts in the visualization stage of stereoscopic imaging, namely, crosstalk, by conducting extensive subjective quality tests. A statistical analysis of the subjective scores reveals that both scene content and camera baseline have significant impacts on crosstalk perception, in addition to the crosstalk level itself. Based on the observed visual variations during changes in significant factors, three perceptual attributes of crosstalk are summarized as the sensorial results of the human visual system (HVS). These are shadow degree, separation distance, and spatial position of crosstalk. They are classified into two categories: 2-D and 3-D perceptual attributes, which can be described by a Structural SIMilarity (SSIM) map and a filtered depth map, respectively. An objective quality metric for predicting crosstalk perception is then proposed by combining the two maps. The experimental results demonstrate that the proposed metric has a high correlation (over 88%) when compared with subjective quality scores in a wide variety of situations.

Stereoscopic Quality of Experience (QoE) is the result of a complex combination of different influencing factors. Previously we had investigated the effect of factors such as scene content, camera baseline, screen size and viewing position on stereoscopic QoE using subjective tests. In this paper, we propose two objective metrics for predicting stereoscopic QoE using bottom-up and top-down approaches, respectively. Specifically, the bottom-up metric is based on characterizing the significant factors of QoE directly, which are scene content, camera baseline, screen size and crosstalk level. While the top-down metric interprets QoE from its perceptual attributes, including crosstalk perception and perceived depth. These perceptual attributes are modeled by their individual relationship with the significant factors and then combined linearly to build the top-down metric. Both proposed metrics have been validated against our own database and a publicly available database, showing a high correlation (over 86%) with the subjective scores of stereoscopic QoE.

Contrast sensitivity is an important characteristic of the human visual system (HVS), which is widely used in image and video signal processing. For visual quality assessment, static spatio-temporal frequency based contrast sensitivity function (CSF) is often used, while the contrast sensitivity can also be affected by smooth pursuit of eyes when tracking attentive regions in the field of view. This paper proposes to tune CSF based on an attention map derived from a visual attention model. The tuned CSF formulated by spatial frequency, temporal velocity, and visual attention map is used to filter video signals in order to construct a quality metric according to the difference of the filtered signals between a reference video and its distorted version. Experimental results demonstrate that the proposed attention tuned spatio-velocity CSF outperforms the traditional spatio-temporal CSF in evaluating the perceived video quality.

For research on quality of experience (QoE), mean opinion scores (MOS) are widely chosen as the results of subjective tests and the ground-truth reference for further research on objective quality modeling. Furthermore, the results of objective quality modeling are used for QoE management subsequently. Therefore, the performance of QoE management process actually depends heavily on MOS. However, the rationality of MOS for QoE management is not yet technically proven in the literature. In this paper, we first prove that subject homogeneity is implicitly assumed for obtaining MOS by modeling the arithmetic averaging process from a systematic viewpoint. However, we point out that actually subjects exhibit variability in terms of quality assessment. Then we elaborate that this mismatch may results in failures if we conduct QoE management based on MOS. Finally we propose a utility-based averaging method (uMOS) which improves the performance of QoE management.

Since the Dadaist refusal of the conventional standards in art, followed by Fluxus` rejection of art as a commodity, and recently, the popularity of Internet and technology in art, artworks have become difficult to recognize as artworks in themselves. Modern works of art are no longer readily only seen today, more often fully experienced. The processing of an aesthetic experience needs a new understanding in terms of the changing context of art and the experiential perspective of art recipients. In the multimedia arena, the valid assumption is that, evaluations of aesthetic experiences are mostly based on the accessible information on the surface of the medium. Several research groups in psychology question the singularity of exterior-level assumptions demonstrating that there are modulating factors that affect aesthetic experiences and one of these is implicit experience. In this paper, we review the significance of empirical aesthetics in psychology and from the artistic point of view combined with a technical and experiential perspective. We also discuss our approach of considering the implications of implicit experiences to the modelling of Quality of Experience (QoE) where this is used as a measurement of perception and aesthetic judgment of contemporary and modern works of art.

Contrast sensitivity of Human Visual System (HVS) plays an important role in perceiving visual stimuli, and consequently, it has a significant impact on the perceived video quality. This paper proposes a visual perception model based on foveated vision and motion perception. The reference and the distorted video sequences are processed by the visual perception model to generate the perceived stimuli in HVS. The perceived difference of the processed sequences is measured in spatial and temporal domains considering the visual sensitivity. An advanced pooling scheme is proposed based on the visual attention mechanism, eye movement type, and influence of temporal quality variation, in order to estimate the perceived video quality. Experimental results demonstrate that the proposed metric significantly outperforms state-of-the-art quality models with respect to a combined eye-tracking and subjective video quality assessment data set.

The visual attention mechanism plays a key role in the human perception system and it has a significant impact on our assessment of perceived video quality. In spite of receiving less attention from the viewers, unattended stimuli can still contribute to the understanding of the visual content. This paper proposes a quality model based on the late attention selection theory, assuming that the video quality is perceived via two mechanisms: global and local quality assessment. First we model several visual features influencing the visual attention in quality assessment scenarios to derive an attention map using appropriate fusion techniques. The global quality assessment as based on the assumption that viewers allocate their attention equally to the entire visual scene, is modeled by four carefully designed quality features. By employing these same quality features, the local quality model tuned by the attention map considers the degradations on the significantly attended stimuli. To generate the overall video quality score, global and local quality features are combined by a content adaptive linear fusion method and pooled over time, taking the temporal quality variation into consideration. The experimental results have been compared to results from appropriate eye tracking and video quality assessment experiments, demonstrating promising performance.

Video streaming through bandwidth-limited channels often suffer from packet losses. Therefore, perceptual quality assessment on video sequences with packet losses is a critical issue in digital video communications. This paper analyzes several image quality metrics and evaluates their applications using spatial and temporal pooling schemes in perceptual video quality assessment for video streams with packet losses. Several approaches using Minkowski summation and averages over different distorted spatial regions and temporal frames to pool the spatial and temporal qualities are evaluated. The experimental results with respect to the subjective video quality measurements demonstrate that the subjects are more sensitive to the most annoying spatial regions and temporal segments when assessing the video quality of the lossy streams.

In this paper we propose a new convex optimization framework for precise spatial noise shaping. The effectiveness of this new technique is demonstrated in the application of perceptual coding of images. A modified JPEG 2000 codec is implemented using the proposed new framework and compared with existing perceptual coding algorithms. Results of subjective tests show that the new framework can provide a significant improvement in bitrate savings compared to the best performing wavelet domain technique. The algorithm allows much more precise control of distortion than existing spatial domain techniques and is fully compliant with part 1 of the JPEG 2000 standard.

Compared to metrics proposed to assess the quality of two-dimensional (2D) images, there are very few metrics devoted to quality assessment of stereoscopic presentations. Crosstalk is one of the most annoying distortions in the visualization stage of stereoscopic imaging technology. This paper proposes a perceptual quality metric which takes characteristics of stereoscopic images into account for predicting quality levels of crosstalk perception in stereoscopic images, based on an understanding of three main factors, crosstalk level, camera baseline and scene content. The experimental results demonstrate that the proposed metric has Pearson correlation of 87.7% when compared to the ground truth results from the subjective experiments on the crosstalk perception, which is much better than the traditional 2D metrics without integrating 3D depth information.

This paper proposes to evaluate video quality by balancing two quality components: global quality and local quality. The global quality is a result from subjects allocating their attention equally to all regions in a frame and all frames in a video. It is evaluated by image quality metrics (IQM) with averaged spatiotemporal pooling. The local quality is derived from visual attention modeling and quality variations over frames. Saliency, motion, and contrast information are taken into account in modeling visual attention, which is then integrated into IQMs to calculate the local quality of a video frame. The local quality of a video sequence is calculated by pooling local quality values over all frames with a temporal pooling scheme derived from the known relationship between perceived video quality and the frequency of temporal quality variations. The overall quality of a distorted video is a weighted average between the global quality and the local quality. Experimental results demonstrate that the combination of the global quality and local quality outperforms both sole global quality and local quality, as well as other quality models, in video quality assessment. In addition, the proposed video quality modeling algorithm can improve the performance of image quality metrics on video quality assessment compared to the normal averaged spatiotemporal pooling scheme.

We have developed an installation called Chroma Space to serve as a platform for experimenting the novel usage of affective colors in an interactive synthetic scenario. Chroma Space demonstrates the effective impacts of using a stylistic approach to address emotional sensations, by making colors move in space. We conducted a study to assess the impact of exposure to achromatic colors to different interactive 3D scenarios. The results suggest that presentation of color has an emotional impact to viewers. We argue that designers of synthetic 3D environments should consider the application of stylistic or thematic use of color on screen to increase the emotional attractiveness of an application.

Most quality models for stereoscopic presentations are dedicated to measuring quality degradation caused by compression artefacts. However, non-compression distortions induced during acquisition and presentation usually have significant influence on 3D viewing experience. In this paper, we propose an objective metric for viewing experience assessment by taking camera baseline and binocular distortion crosstalk into consideration. In particular, the proposed metric is based on our previous work on both subjective evaluation and objective assessment of crosstalk perception. Results on a publicly available stereoscopic quality database demonstrate that the proposed metric can achieve more than 87% correlation with subjective assessment of viewing experience.

Based on whether the competing active nodes are forwarding packets of the same flow, the channel access contention in multi-hop wireless networks can be classified into intra-flow contention and inter-flow contention. In this paper, we analyze the effect of intra-flow contention on multi-hop network performance. First, we demonstrate that intra-flow contention makes up a large proportion of contention in the network and may severely degrade the network performance. Then, to understand the performance improvement by eliminating intra-flow contention, we consider an ideal MAC based on 802.11 DCF where intra-flow contention is eliminated. We analyze the performance of the ideal MAC by proposing a three-dimensional Markov model. Furthermore, we present an approximate implementation of the ideal MAC based on simple modifications to 802.11 DCF. Finally, extensive simulations are performed and the results validate the proposed model and reveal the performance improvement by eliminating intra-flow contention.

While the causes and nature of crosstalk, as well as crosstalk reduction techniques have been extensively studied, it is still difficult to eliminate. Perceptually, crosstalk is one of the most annoying distortions in the visualization stage of stereoscopic imaging. Therefore, to understand how users perceive crosstalk is of fundamental importance to improve the quality of 3D presentations. In this paper, we aim at analyzing the impact of crosstalk level, camera baseline and scene content on users' perception of crosstalk. Extensive subjective tests are conducted and the opinion scores are statistically analyzed and discussed. The results indicate that crosstalk level, camera baseline, as well as scene content all have major impacts on the perception of crosstalk. We also show that these three factors correlate with each other in terms of impact on the crosstalk perception. Furthermore, we propose a content descriptor for crosstalk perception (CDCP) and show its effectiveness.

Even though several unequal erasure protection (UEP) schemes have been proposed for video streaming, it is still a challenge to define the optimal relative protection level for different data units. In this paper, we study loss-distortion modeling in realistic video streaming scenarios. The main observation is that when video streams with complex hierarchical structures and error resilience features are considered, the relative significance of different units cannot be reliably estimated from mutual dependencies between units as suggested in related research. Therefore, data units can be classified reliably into different perceptual priority classes via analysis by synthesis only. For estimating the co-impact of multiple losses, we propose a simple loss-distortion model. The model can be used for optimizing UEP in practical streaming systems.

Most existing quality metrics do not take the human attention analysis into account. Attention to particular objects or regions is an important attribute of human vision and perception system in measuring perceived image and video qualities. This paper presents an approach for extracting visual attention regions based on a combination of a bottom-up saliency model and semantic image analysis. The use of PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural SIMilarity) in extracted attention regions is analyzed for image/video quality assessment, and a novel quality metric is proposed which can exploit the attributes of visual attention information adequately. The experimental results with respect to the subjective measurement demonstrate that the proposed metric outperforms the current methods.

Several different schemes for unequal error/erasure protection (UEP) in video streaming have been proposed during the past years. However, it is not a trivial task to define the optimal relative protection levels for different media units (ie. the basic units of decoding handled individually by media decoder) in practical streaming applications. In this paper, we use a theoretical loss-distortion model to evaluate the performance of different UEP schemes in realistic video streaming scenarios with fixed redundancy overhead budget. Our results confirm the observation that equal error/erasure protection (EEP) is desired when the packet loss rate is low compared to the redundancy overhead. The higher the packet loss rate, the more segregating UEP should be used to achieve optimal performance. The results also suggest that comparable performances can be obtained by using two protection levels only (protected part and unprotected part) instead of more complex UEP schemes.

In our recent work, an analytic power-channe/error-rate-distortion (PERD) model was developed to estimate the end-to-end distortion of power constrained portable devices in real-time video communications over bandwidth limited and lossy channels. A random mode selection scheme is implemented after obtaining optimal percentages of intra, inter and skip mode through a constrained optimization process. In this paper, advanced solution to the problem of mode selection for each macroblock (MB) is proposed. A concealment aware PERD model is first derived, which takes error concealment at the decoder into consideration to estimate the end-to-end distortion. By using this model, optimal percentages of intra, inter and skip modes that minimize decoder distortion are obtained under any power and rate constraints. An intelligent mode selection scheme is then generalized by ranking the priority of MB according to how well it gets reconstructed if lost. Experimental results show that, the new framework consistently achieves better power-rate-distortion (P-R-D) performance with little extra computation load, which is critical in real-time communications of portable devices.

Variable bit rate encoding is considered to outper- form constant bit rate encoding with regard to compression gain and constancy of the video quality. The drawback is, however, that variable bit rate video streams usually exhibit a significant degree of burstiness which may result in poor multiplexing properties, buffer overflows, and large network delays. In this work we show how the explicit selection of slice mode types can effectively be used to generally avoid the use of large intracoded frames while still providing robustness against packet losses in the network. The employed encoding scheme provides a considerable smoothing of variable bit rate video traffic, resulting in less delay, less delay jitter, and less loss due to buffer overflows and late arrivals. It preserves the efficiency of variable bit rate video encoding, has little overhead, and is in accordance with current video coding standards.

The potential problem of oversubscribing receivers in receiver-driven multicast is addressed. We present a framework based on harmonizing the erasure-resilience properties of video with existing congestion control algorithms. The result offers subscription alternatives for receivers in which a penalty in terms of visual quality will be experienced by oversubscribing receivers. Thus, we provide an incentive for performing proper congestion control. The presented framework is independent of specific congestion control algorithms. Simulation results show the intended performance.

An encoder based rate smoothing scheme which uses explicit slice-based mode selection has been proposed. The algorithm provides significantly smoother bitstream from a variable bit rate (VBR) encoder and reduces the network queuing delay while maintaining quality for the end-user. In this paper we investigate the error robustness of the proposed scheme based on H.264/AVC codec. The results show that, the proposed scheme maintains error resilience properties without extra delay. Compared to the standard frame based scheme, the algorithm provides almost the same average distortion while reducing the distortion variance which helps improving the subjective video quality. We investigate the performance in random and bursty packet loss environments, and it shows that the proposed scheme is applicable for error prone networks, for example, wireless networks.

Rate smoothing is essential for achieving lower delay when transmitting real-time video over the network. Recently, "explicit slice-based mode selection" (ESM) is proposed as a new way of achieving this goal together with its inherent quality smoothness and error resilience features. However previous studies focus on the practical aspects and do not address an optimized solution. In this paper, we propose a rate-distortion (RD) optimized solution for finding the best location and size of the intra-coded slices. The experimental results show that for a target bit rate the optimized scheme is able to offer performance close to that of mode selection on a macroblock level, over wireless channels with different packet loss rates. Moreover, the optimized ESM algorithm provides significant advantages of granular bit stream prioritization for network transmission. However, the RD based optimization is in general computationally expensive. We therefore propose a heuristic approach which incorporates both channel statistics and sequence characteristics. Results show that it yields close to optimal performance at lower complexity.

In this paper, we present a novel no-reference (NR) metric to assess the quality of JPEG-coded images. The features for predicting the perceived image quality are extracted by considering the key human visual sensitivity factors such as, edge amplitude, edge length, background activity and background luminance. The extracted features with the subjective test results are used to train a multi-layer perceptron (MLP) neural network. Experimental results show that the prediction of the trained neural network is very close to the mean opinion score (MOS). The subjective test results of the proposed metric are compared with the Wang-Bovik's NR blockiness metric. Further, this metric can be extended to assess the quality of the MPLG/H.26x compressed videos.

We present a streaming media test bed for IP networks. Besides a streaming server and a streaming media client, it consists of an IP network emulator, a high-performance packet capture device and a packet flow regenerator enabling repeatable performance measurements of streaming media applications and network QoS mechanisms in a controlled environment. Some initial results from the work involved in verifying the performance of the packet regenerator are also presented.

This paper presents extensions to previous algorithms for unequal error protection (UEP) of multimedia encoded using embedded source coders. Whereas earlier work on this topic has considered the single-channel case only, we investigate the case where multiple parallel channels with possibly different loss characteristics are used. Such scenarios are becoming more relevant with the gradual deployment of priority-enabled networks (e.g. DiffServ). We present results indicating that significant performance gains can be achieved by using UEP instead of the simpler EEP (equal error protection) scheme also in the multichannel case. Furthermore, it is seen that the performance gap between UEP and EEP increases with an increasing number of channels

This paper presents a scheme for scalable, error resilient coding of audio signals. The scheme is implemented here as part of a subband-based audio codec and involves the union of two interconnected ideas: tree-structured vector quantization (TSVQ) and reallocation of bits (RoB). The former ensures that the encoding procedure results in fixed-length codes while the latter is a low-complexity scaling algorithm that computes optimal bitrates for each subband when the bitstream needs to be downscaled to a lower bitrate. The functionality enabled by these two systems assumes great importance when the encoded material is to be delivered over channels and networks that have a time-varying bitrate, such as IP-based wireless networks. Such channels are often error-prone and the potentially debilitating effects of introduced errors are mitigated by the inherent fixed-length property of all codewords. We present results that highlight the performance of this system under noisy conditions and show that the degradation in quality is graceful.

The MPEG-21 vision is to define a multimedia framework to enable transparent and augmented use of multimedia resources across a wide range of networks and devices used by the different communities. This paper describes the evolution of a test bed simulating an MPEG-21 framework considering a streaming media use case scenario. The test bed has been set up in order to realize, validate and perform experimentations on streaming media considering the integration of MPEG-4 content representation and MPEG-7 audiovisual content description interface within the MPEG-21 multimedia framework. The application scenario is within the concept of universal multimedia access (UMA) realizing the challenge of "create once play any time, any how, any where".

A multi-dimensional variable-length subband coder for prestack seismic data was presented. A 2-D and 3-D separable near perfect reconstruction filter bank was optimized to maximize the coding gain, assuming that the correlation properties of pre-stack seismic data can be modeled by directional dependent autoregressive processes. Identical quantization and entropy coder allocation strategies were utilized to isolate the compression efficiency of the different high-dimensional filter bank methods. An example, with compression ratios ranging from 160:1 to 320:1, showed that 3-D subband coding of common shot gathers performed 50% better in terms of bit rate at a given signal-to-noise ratio compared to 2-D subband coding of common shot gathers.

The performance of several decorrelating transforms is evaluated in the JPEG-2000 baseline system. The transforms considered are based on uniform parallel 2-channel and 8-channel filter banks and discrete cosine transform (DCT) building blocks. Through the analysis of lossy compression results, a system employing a combination of 8- channel and 2-channel filter banks is found to perform best overall. Also, the performance of several reversible transforms is evaluated in the same coding framework. Based on lossy and lossless compression results, a reversible version of the Cohen-Daubechies-Feauveau (2,2) wavelet transform was found to be most effective.

Line spectral pairs (LSP) are, because of their advantageous properties, commonly used to represent the spectral information in a linear predictive (LP) speech coder. However, the quantised spectrum is relatively sensitive to channel errors. This paper describes a robust quantiser for LSPs transmitted over noisy waveform channels, based on an optimisation of the source coder over the modulation scheme. Modulation organized vector quantisation, MOR-VQ, jointly optimises a codebook and its index assignment, governed by the defined modulation symbol constellation. The system closely follows the optimum performance theoretically available (OPTA). Moreover, the MOR-VQ scheme allows for efficient codebook search strategies.>