A high effort in Quality of Experience (QoE) research has been put into subjective assessment to determine the perceived quality of video. Most laboratory experiments follow guidelines from the ITU-T Recommendations, which suggest Absolute Category Rating (ACR) as a method to conduct such experiments. However, this method of video assessment radically limits confounding variables, is unnatural, and is far from how people cope with quality degradation and the cues they receive in these situations. This paper addresses this issue and proposes a more realistic subjective experiment based on the participant's behavior. Instead of passively rating the degraded quality of silent videos, we created the possibility to react to the annoying quality of chosen Netflix movies and reward participants by increasing the quality to the best possible. To cope with the data obtained, we adapted the method of fitting psychometric functions known in neuroscience, auditory science, animal science, and psychology. As a result, we obtained a more comprehensive image of the participants and their perceived quality, including their consistency, lapses, and differences. We estimated the parameters using Maximum Likelihood Estimation (MLE) and evaluated the goodness-of-fit of three S-shaped functions: Weibull, cumulative normal, and logistic. In the end, as the most common, we analyze the basic properties of the fitted functions, such as the Point of Subjective Equality (PSE), confidence intervals, and slope (β). We examine their role in describing individual differences among 34 subjects.

The term ecological validity is widely used across IEEE publications, but its definition and application remain inconsistent and ambiguous. This paper explores how conflicting definitions of ecological validity lead to confusion about experimental conditions and data interpretation. Because the term ecological validity conflates methods with outcomes and presumes that realism creates validity, we argue against its continued use as a separate term. Instead, we recommend using external validity and mundane realism. We believe this change would increase rigor in scientific communications, eliminate ambiguities, and open a transdisciplinary dialogue.

The datasets of most image quality assessment studies contain ratings on a categorical scale with five levels, from bad (1) to excellent (5). For each stimulus, the number of ratings from 1 to 5 is summarized and given in the form of the mean opinion score. In this study, we investigate families of multinomial probability distributions parameterized by mean and variance that are used to fit the empirical rating distributions. To this end, we consider quantized metric models based on continuous distributions that model perceived stimulus quality on a latent scale. The probabilities for the rating categories are determined by quantizing the corresponding random variables using threshold values. Furthermore, we introduce a novel discrete maximum entropy distribution for a given mean and variance. We compare the performance of these models and the state of the art given by the generalized score distribution for two large data sets, KonIQ-10k and VQEG HDTV. Given an input distribution of ratings, our fitted two-parameter models predict unseen ratings better than the empirical distribution. In contrast to empirical distributions of absolute category ratings and their discrete models, our continuous models can provide fine-grained estimates of quantiles of quality of experience that are relevant to service providers to satisfy a certain fraction of the user population.

The notion of experiment precision quantifies the variance of user ratings in a subjective experiment. Although there exist measures that assess subjective experiment precision, to the best of our knowledge, there is no systematic framework in the Multimedia Quality Assessment (MQA) field for comparing subjective experiments in terms of their precision. Therefore, the main idea of this paper is to propose a framework for comparing subjective experiments in the field of MQA based on appropriate experiment precision measures. We present three experiment precision measures and three related experiment precision comparison methods. We analyze the performance of the measures by using data from real-world Quality of Experience (QoE) subjective experiments. We believe our experiment precision assessment framework will help compare different subjective experiment methodologies. For example, it may help decide which methodology results in more precise user ratings. This may potentially inform future standardization activities.

This paper discusses the problem of ecological va-lidity of Quality of Experience experiments in a broader context of internal and external validity. We argue that the issue of trade-off between control in the experiment and generalization of the results requires diverse experimental protocols. To enable this process comparability of experiments has to be guaran-teed. To improve comparability and communication between researchers, we propose using theoretical models as tools for clearly communicating the assumptions underlying subjective experiments. These models describe the relationships between variables and enable comparability of research. We demonstrate how the theoretical model of video QoE can be used for the comparison of three independent QoE studies. In conclusion, we stress that ecological validity is a mean to an extended external validity. We suggest that new experimental protocols that increase the number of Influence Factors (IFs) measured have to provide comparability with other experiments. Theoretical models can facilitate this process and stimulate higher comparability between studies in QoE subdomains. A community effort is needed to adjust the theoretical model to new use cases, particularly with new immersive multimedia. This paper inspires further research on ecological validity in QoE and encourages the adoption of theoretical models for more comparable research designs.

In traditional subjective video quality experiments, the presented sequences are short and quality ratings are based on a single interaction with a service (i.e., one session). However, in real-life scenarios, users interact with a video service for a longer period of time. If decisions are made, such as to abandon a service, they are formulated based on longitudinal multi-episodic interaction. Therefore, it is important to better understand how quality is perceived in a longer interaction and how quality perception is linked to behavioral implications. My PhD work encompasses a longitudinal study of users’ interactions with a video service using a mobile device. In our study, which consists of six phases, we use different study designs to investigate how users perceive quality in a more ecologically valid setting. The study is carried out using a previously validated setup, which consists of compression software and a mobile application.

Most user studies in the QoE multimedia domain are done by asking users about quality. This approach has advantages: it obtains many answers and reduces variance by repeated measurements. However, the results obtained in this context may be different from those obtained in the real application, since quality is not asked about this often in everyday life. It is more natural is to focus on user behavior. The proposed PhD focuses on a method for performing experiments based on observations of a participant’s behavior. We address two main challenges that exist in any new experiment design: how to calculate the interval validity of the proposed method and how to analyze the obtained data. The data analysis we propose is based on psychometric functions. We propose two different experiments, one of which is already ongoing.

Subjective responses from Multimedia Quality Assessment (MQA) experiments are conventionally analyzed with methods not suitable for the data type these responses represent. Furthermore, obtaining subjective responses is resource intensive. Thus, a method that allows the reuse of existing responses would be beneficial. Applying improper data analysis methods leads to difficulty in interpreting results. This increases the probability of drawing erroneous conclusions. Building upon existing subjective responses is resource friendly and helps develop machine learning (ML) based visual quality predictors. In this work, we show that using a discrete model for analyzing responses from MQA subjective experiments is feasible. We indicate that our proposed Generalized Score Distribution (GSD) properly describes response distributions observed in typical MQA experiments. We also highlight interpretability of GSD parameters and indicate that the GSD outperforms the approach based on sample empirical distribution when it comes to bootstrapping. Furthermore, we provide evidence that the GSD outcompetes the state-of-the-art model both in terms of goodness-of-fit and bootstrapping capabilities. To accomplish the aforementioned objectives, we analyze more than one million subjective responses from over 30 subjective experiments.

Recently an impressive development in immersive technologies, such as Augmented Reality (AR), Virtual Reality (VR) and 360${^\circ }$video, has been witnessed. However, methods for quality assessment have not been keeping up. This paper studies quality assessment of 360${^\circ }$video from the cross-lab tests (involving ten laboratories and more than 300 participants) carried out by the Immersive Media Group (IMG) of the Video Quality Experts Group (VQEG). These tests were addressed to assess and validate subjective evaluation methodologies for 360${^\circ }$video. Audiovisual quality, simulator sickness symptoms, and exploration behavior were evaluated with short (from 10 seconds to 30 seconds) 360${^\circ }$sequences. The following factors’ influences were also analyzed: assessment methodology, sequence duration, Head-Mounted Display (HMD) device, uniform and non-uniform coding degradations, and simulator sickness assessment methods. The obtained results have demonstrated the validity of Absolute Category Rating (ACR) and Degradation Category Rating (DCR) for subjective tests with 360${^\circ }$videos, the possibility of using 10-second videos (with or without audio) when addressing quality evaluation of coding artifacts, as well as any commercial HMD (satisfying minimum requirements). Also, more efficient methods than the long Simulator Sickness Questionnaire (SSQ) have been proposed to evaluate related symptoms with 360${^\circ }$videos. These results have been instrumental for the development of the ITU-T Recommendation P.919. Finally, the annotated dataset from the tests is made publicly available for the research community.

Recent studies have shown that it is possible to characterize subject bias and variance in subjective assessment tests. Apparent differences among subjects can, for the most part, be explained by random factors. Building on that theory, we propose a subjective test design where four to six team members each rate the stimuli multiple times. The results are comparable to a high performing objective metric. This provides a quick and simple way to analyze new technologies and perform pre-tests for subjective assessment.

In this paper we reproduce experimental results presented in our earlier work titled "Describing Subjective Experiment Consistency by p-Value P-P Plot" that was presented in the course of the 28th ACM International Conference on Multimedia. The paper aims at verifying the soundness of our prior results and helping others understand our software framework. We present artifacts that help reproduce tables, figures and all the data derived from raw subjective responses that were included in our earlier work. Using the artifacts we show that our results are reproducible. We invite everyone to use our software framework for subjective responses analyses going beyond reproducibility efforts.

There are phenomena that cannot be measured without subjective testing. However, subjective testing is a complex issue with many influencing factors. These interplay to yield either precise or incorrect results. Researchers require a tool to classify results of subjective experiment as either consistent or inconsistent. This is necessary in order to decide whether to treat the gathered scores as quality ground truth data. Knowing if subjective scores can be trusted is key to drawing valid conclusions and building functional tools based on those scores (e.g., algorithms assessing the perceived quality of multimedia materials). We provide a tool to classify subjective experiment (and all its results) as either consistent or inconsistent. Additionally, the tool identifies stimuli having irregular score distribution. The approach is based on treating subjective scores as a random variable coming from the discrete Generalized Score Distribution (GSD). The GSD, in combination with a bootstrapped G-test of goodness-of-fit, allows to construct p-value P--P plot that visualizes experiment's consistency. The tool safeguards researchers from using inconsistent subjective data. In this way, it makes sure that conclusions they draw and tools they build are more precise and trustworthy. The proposed approach works in line with expectations drawn solely on experiment design descriptions of 21 real-life multimedia quality subjective experiments.

How accurately are people able to use the absolute category rating (ACR) 5-level scale? Put another way, how repeatable are an individual subject's scores? Several subjective experiments have asked subjects to rate the same sequences a couple of times. Analyses indicate that none of the subjects exactly repeated their prior scores for these sequences. We would like to better understand this imperfection. This paper uses ACR subjective video quality tests to explore the precision of subjective ratings. To make formal measurements possible, we propose a theoretical subject model that is the main contribution of this paper. The proposed subject model indicates three major factors that influence accuracy: subject bias, subject inaccuracy, and stimulus scoring difficulty. These appear to be separate random effects and their existence is a reason why none of the subjects were able to perfectly repeat scores. There are three key consequences. First, subject scoring behavior includes a random component that spans approximately half of the rating scale. Second, the sensitivity and accuracy of most subjective analyses can be improved if the subject scores are normalized by removing subject bias. Third, to some extent, multiple subjects can be replaced with a single subject who rates each sequence multiple times.

Video transmission and analysis is often utilized in applications outside of the entertainment sector, and generally speaking this class of video is used to perform specific tasks. Examples of these applications include security and public safety. The Quality of Experience (QoE) concept for video content used for entertainment differs significantly from the QoE of surveillance video used for recognition tasks. This is because, in the latter case, the subjective satisfaction of the user depends on achieving a given functionality. Recognizing the growing importance of video in delivering a range of public safety services, we focused on developing critical quality thresholds in license plate recognition tasks based on videos streamed in constrained networking conditions. Since the number of surveillance cameras is still growing it is obvious that automatic systems will be used to do the tasks. Therefore, the presented research includes also analysis of automatic recognition algorithms.

Several factors limit the advances on automatizing video quality measurement. Modelling the human visual system requires multi- and interdisciplinary efforts. A joint effort may bridge the large gap between the knowledge required in conducting a psychophysical experiment on isolated visual stimuli to engineering a universal model for video quality estimation under real-time constraints. The verification and validation requires input reaching from professional content production to innovative machine learning algorithms. Our paper aims at highlighting the complex interactions and the multitude of open questions as well as industrial requirements that led to the creation of the Joint Effort Group in the Video Quality Experts Group. The paper will zoom in on the first activity, the creation of a hybrid video quality model.

Video communications require the use of networks capable of optimally allocating resources and adapting dynamically themselves to the changing operating conditions. When a video stream has to be delivered to various receivers, the appropriate service to accomplish the delivery process is the multicast communication service. In the particular case of the widely used IEEE 802.11 Wireless LANs (WLANs), the multicast service is simply specified as an unreliable broadcasting service, i.e., it does not include any error recovery mechanism. The absence of feedback from the receivers does not only prevent the sender from taking proper action to recover the corrupted packets, but it also makes unfeasible the implementation of a channel rate adaptation mechanism as the one used by the IEEE 802.11 unicast service. The absence of such mechanism may render the multicast service useless if the channel conditions are so that it makes unreliable the delivery process of the multicast traffic. Even though the IEEE has been working on an amendment to the standard specifying the operation of various error recovery mechanisms for the multicast service, these services have not been designed to support any rate adaptation mechanism. In this paper, we present a novel auto rate selection multicast mechanism, whose design is based on the collision prevention to increase the reliability. We have developed QoS and QoE evaluations via simulation, demonstrating that our proposal outperforms the video quality perceived by the end-user.

The paper proposes a No-Reference (NR) metric to objectively assess the H.264/AVC video quality. The proposed model takes into account the typical artefacts introduced by hybrid block-based motion compensated predictive video codecs as the one related to the H.264/AVC standard. More specifically, these artefacts are the blockiness introduced at the boundaries of each coded block and the temporal flickering due to different coding modes used for the same macroblock along the video sequence. Furthermore, a flickering metric for intra coded frames is also derived. The quality prediction accuracy of the proposed NR quality metric is validated over subjective data collected during a video subjective evaluation experiments. Moreover, the quality prediction accuracy is also compared with the one provided by the well known state-of-the-art Structural SIMilarity (SSIM) metric which works in a full-reference mode. The proposed metric achieves a higher Pearson's correlation coefficient with subjective scores than the one achieved by the SSIM metric.

In this paper, we propose an accurate sampling scheme for defeating SYN flooding attacks as well as TCP portscan activity. The scheme examines TCP segments to find at least one of multiple ACK segments coming from the server. The method is simple and scalable, because it achieves good detection performance with false positive rate close to zero even for very low sampling rates. Our trace-based simulations show that the effectiveness of the proposed scheme only relies on the sampling rate regardless on the sampling method.

In this study we address the problem of an image quality trade-off that can be observed when dealing with content reconstruction schemes based on self-embedding. We derive two models for the estimation of optimal system parameters and the optimization of the overall image quality. This goal is achieved by balancing the distortions of a different nature that affect the resulting images. The performance of the derived models is verified with an accurate reference model and compared to traditional parameter selection strategies. The models are based on basic image features only and allow for rapid prediction of the best values for system parameters in a fully automatic manner.

In the field of still and moving pictures digital imaging one of the most important factors affecting the quality of the acquisition is a correct exposure. This factor is especially important for video surveillance. It is particularly applicable in the case of CCTV cameras often operating in low-light conditions or blinded by an external light source. It is quite common that the lighting conditions are well below the adaptive capacity of a camera. Therefore, it is necessary to continuously monitor the level of exposure to determine image quality. It should be noted that the level of exposure is not expressed using quantitative parameters of acquisition (being sometimes meaningless from the perceived quality assessment point of view) but is measured directly on the image. The purpose of the presented research was to develop a No Reference (NR) metric assessing video Quality of Experience (QoE) affected with the exposure distortion. It was presumed that both over- and under-exposure degrade QoE, so the research task was to derive a proper mapping function between QoE and exposure. The presented exposure metric is unique and no similar research was found in the literature.

Cross talk is one of the most important parameters of the 3D displays' quality. Different cross talk definitions exist, which makes cross talk measurement and comparison difficult. We take a step back and focus on a detailed 3D display luminance analysis. The conclusions we draw from the temporal luminance analysis can be used to propose an effective approach to cross talk measurements. In scope of the presented work we have measured four different 3D displays.

A host of methods and algorithms have been proposed to solve the issue of traffic classification in recent years. However, a comparison of results between different studies is very difficult due to the lack of structure and common understanding of notions in the domain, especially a precise definition of application classes. This paper aims to fill this gap and propose a first attempt to systematically classify traffic definitions. To attain this goal, we take advantage of the ontology paradigm.

The aim of this paper is to determine how to model router interface in order to accurately predict packet drops. There is an enormous amount of research on traffic models reported, however, a model of router interface has not gained proper consideration yet.