The term ecological validity is widely used across IEEE publications, but its definition and application remain inconsistent and ambiguous. This paper explores how conflicting definitions of ecological validity lead to confusion about experimental conditions and data interpretation. Because the term ecological validity conflates methods with outcomes and presumes that realism creates validity, we argue against its continued use as a separate term. Instead, we recommend using external validity and mundane realism. We believe this change would increase rigor in scientific communications, eliminate ambiguities, and open a transdisciplinary dialogue.

Recent studies have shown that it is possible to characterize subject bias and variance in subjective assessment tests. Apparent differences among subjects can, for the most part, be explained by random factors. Building on that theory, we propose a subjective test design where four to six team members each rate the stimuli multiple times. The results are comparable to a high performing objective metric. This provides a quick and simple way to analyze new technologies and perform pre-tests for subjective assessment.

The new video coding standard, MPEG-H Part 2 High Efficiency Video Coding (HEVC) or H.265, was developed to be roughly twice as efficient as H.264/AVC -- meaning H.265/HEVC could deliver the same quality as H.264/AVC using roughly half the bitrate. In this paper we describe a subjective experiment designed to test this claim. We present an experiment using 20 different 1080p 29.97 fps scenes and 12 impairment levels spanning MPEG-2, H.264/AVC and H.265/HEVC. Additionally we compare the results obtained from the subjective assessment to quality estimates from two objective metrics: VQM and PSNR. Our subjective results show that H.265/HEVC can deliver the same quality at half the bitrate compared to H.264/AVC and can perform better at one quarter the bitrate compared to MPEG-2 in many, but not all, situations. For all 20 scenes coded with H.265/HEVC at 4 Mbps mean opinion scores span 38% of the subjective scale, which indicates the importance of scene selection. Objective quality estimations of HEVC have a low correlation with subjective results (0.60 for VQM, 0.64 for PSNR).

How accurately are people able to use the absolute category rating (ACR) 5-level scale? Put another way, how repeatable are an individual subject's scores? Several subjective experiments have asked subjects to rate the same sequences a couple of times. Analyses indicate that none of the subjects exactly repeated their prior scores for these sequences. We would like to better understand this imperfection. This paper uses ACR subjective video quality tests to explore the precision of subjective ratings. To make formal measurements possible, we propose a theoretical subject model that is the main contribution of this paper. The proposed subject model indicates three major factors that influence accuracy: subject bias, subject inaccuracy, and stimulus scoring difficulty. These appear to be separate random effects and their existence is a reason why none of the subjects were able to perfectly repeat scores. There are three key consequences. First, subject scoring behavior includes a random component that spans approximately half of the rating scale. Second, the sensitivity and accuracy of most subjective analyses can be improved if the subject scores are normalized by removing subject bias. Third, to some extent, multiple subjects can be replaced with a single subject who rates each sequence multiple times.

This paper describes objective video quality validation efforts conducted in the past two decades. Validation efforts to be examined include a validation test performed by the T1A1 committee in the early 1990's; five rounds of validation testing performed by the Video Quality Experts Group; and validation tests performed by ITU-T Study Group 12. Useful products that resulted from those efforts will be identified, including standards, datasets, and model validation techniques.

Over the past few years, there have been a number of proposals aimed at introducing different levels of service in the Internet. One of the more recent proposals is the Differentiated Services (Diff-Serv) architecture, and in this paper we explore how the policing actions and associated rate guarantees provided by the Expedited Forwarding (EF) translate into perceived benefits for applications that are the presumed users of such enhancements. Specifically, we focus on video streaming applications that arguably have relatively strong service quality requirements, and which should, therefore, stand to benefit from the availability of some form of enhanced service. Our goal is to gain a better understanding of the relation that exists between application level quality measures and the selection of the network level parameters that govern the delivery of the guarantees that an EF based service would provide. Our investigation, which is experimental in nature, relies on a number of standard streaming video servers and clients that have been modified and instrumented to allow quantification of the perceived quality of the received video stream. Quality assessments are performed using a Video Quality Measurement tool based on the ANSI objective quality standard. Measurements were made over both a local Diff-Serv testbed and across the QBone, a QoS enabled segment of the Internet2 infrastructure. The paper reports and analyzes the results of those measurements.