In adaptive bitrate streaming, resolution cross-over refers to the point on the convex hull where the encoding resolution should switch to achieve better quality. Accurate cross-over prediction is crucial for streaming providers to optimize resolution at given bandwidths. Most existing works rely on objective Video Quality Metrics (VQM), particularly VMAF, to determine the resolution cross-over. However, these metrics have limitations in accurately predicting resolution cross-overs. Furthermore, widely used VQMs are often trained on subjective datasets collected using the Absolute Category Rating (ACR) methodologies, which we demonstrate introduces significant uncertainty and errors in resolution cross-over predictions. To address these problems, we first investigate different subjective methodologies and demonstrate that Pairwise Comparison (PC) achieves better cross-over accuracy than ACR. We then propose a novel metric, Resolution Cross-over Quality Loss (RCQL), to measure the quality loss caused by resolution cross-over errors. Furthermore, we collected a new subjective dataset (LSCO) focusing on live streaming scenarios and evaluated widely used VQMs, by benchmarking their resolution cross-over accuracy.

In an adaptive bitrate streaming application, the efficiency of video compression and the encoded video quality depend on both the video codec and the quality metric used to perform encoding optimization. The development of such a quality metric need large scale subjective datasets. In this work we merge several datasets into one to support the creation of a metric tailored for video compression and scaling. We proposed a set of HEVC lightweight features to boost performance of the metrics. Our metrics can be computed from tightly coupled encoding process with 4% compute overhead or from the decoding process in real-time. The proposed method can achieve better correlation than VMAF and P.1204.3. It can extrapolate to different dynamic ranges, and is suitable for real-time video quality metrics delivery in the bitstream. The performance is verified by in-distribution and cross-dataset tests. This work paves the way for adaptive client-side heuristics, real-time segment optimization, dynamic bitrate capping, and quality-dependent post-processing neural network switching, etc.

We conducted a large-scale study of human perceptual quality judgments of High Dynamic Range (HDR) and Standard Dynamic Range (SDR) videos subjected to scaling and compression levels and viewed on three different display devices. While conventional expectations are that HDR quality is better than SDR quality, we have found subject preference of HDR versus SDR depends heavily on the display device, as well as on resolution scaling and bitrate. To study this question, we collected more than 23,000 quality ratings from 67 volunteers who watched 356 videos on OLED, QLED, and LCD televisions, and among many other findings, observed that HDR videos were often rated as lower quality than SDR videos at lower bitrates, particularly when viewed on LCD and QLED displays. Since it is of interest to be able to measure the quality of videos under these scenarios, e.g. to inform decisions regarding scaling, compression, and SDR vs HDR, we tested several well-known full-reference and no-reference video quality models on the new database. Towards advancing progress on this problem, we also developed a novel no-reference model called HDRPatchMAX, that uses a contrast-based analysis of classical and bit-depth features to predict quality more accurately than existing metrics.

As compared to standard dynamic range (SDR) videos, high dynamic range (HDR) content is able to represent and display much wider and more accurate ranges of brightness and color, leading to more engaging and enjoyable visual experiences. HDR also implies increases in data volume, further challenging existing limits on bandwidth consumption and on the quality of delivered content. Perceptual quality models are used to monitor and control the compression of streamed SDR content. A similar strategy should be useful for HDR content, yet there has been limited work on building HDR video quality assessment (VQA) algorithms. One reason for this is a scarcity of high-quality HDR VQA databases representative of contemporary HDR standards. Towards filling this gap, we created the first publicly available HDR VQA database dedicated to HDR10 videos, called the Laboratory for Image and Video Engineering (LIVE) HDR Database. It comprises 310 videos from 31 distinct source sequences processed by ten different compression and resolution combinations, simulating bitrate ladders used by the streaming industry. We used this data to conduct a subjective quality study, gathering more than 20,000 human quality judgments under two different illumination conditions. To demonstrate the usefulness of this new psychometric data resource, we also designed a new framework for creating HDR quality sensitive features, using a nonlinear transform to emphasize distortions occurring in spatial portions of videos that are enhanced by HDR, e.g., having darker blacks and brighter whites. We apply this new method, which we call HDRMAX, to modify the widely-deployed Video Multimethod Assessment Fusion (VMAF) model. We show that VMAF+HDRMAX provides significantly elevated performance on both HDR and SDR videos, exceeding prior state-of-the-art model performance. The database is now accessible at: https://live.ece.utexas.edu/research/LIVEHDR/LIVEHDR_index.html. The model will be made available at a later date at: https://live.ece.utexas.edu//research/Quality/index_algorithms.htm.

Massive multiple-input multiple-output (MIMO) has gained widespread popularity in recent years due to its ability to increase data rates, improve signal quality, and provide better coverage in challenging environments. In this paper, we investigate the MIMO beam selection (MBS) problem, which is proven to be NP-hard and computationally intractable. To deal with this problem, quantum computing that can provide faster and more efficient solutions to large-scale combinatorial optimization is considered. MBS is formulated in a quadratic unbounded binary optimization form and solved with Coherent Ising Machine (CIM) physical machine. We compare the performance of our solution with two classic heuristics, simulated annealing and Tabu search. The results demonstrate an average performance improvement by a factor of 261.23 and 20.6, respectively, which shows that CIM-based solution performs significantly better in terms of selecting the optimal subset of beams. This work shows great promise for practical 5G operation and promotes the application of quantum computing in solving computationally hard problems in communication.

Videos often have to be transmitted and stored at low bitrates due to poor network connectivity during adaptive bitrate streaming. Designing optimal bitrate ladders that would select the perceptually-optimized resolution, frame-rate, and compression level for low-bitrate videos for adaptive streaming across the internet is therefore a task of great interest. Towards that end, we conducted the first large-scale study of medium and low-bitrate videos from live sports for two codecs (Elemental AVC and HEVC) and created the Amazon Prime Video Low-Bitrate Sports (APV LBS) dataset. The study involved 94 participants and 742 videos, with more than 23,000 human opinion scores collected in total. We analyzed the data obtained and we also conducted an extensive evaluation of objective Video Quality Assessment (VQA) algorithms and benchmarked their performance, and make recommendations on bitrate ladder design. We're making the metadata and VQA features available at https://github.com/JoshuaEbenezer/lbmfr-public.

High Dynamic Range (HDR) videos can represent a much greater range of brightness and color than Standard Dynamic Range (SDR) videos and are rapidly becoming an industry standard. HDR videos have more challenging capture, transmission, and display requirements than legacy SDR videos. With their greater bit depth, advanced electro-optical transfer functions, and wider color gamuts, comes the need for video quality algorithms that are specifically designed to predict the quality of HDR videos. Towards this end, we present the first publicly released large-scale subjective study of HDR videos. We study the effect of distortions such as compression and aliasing on the quality of HDR videos. We also study the effect of ambient illumination on perceptual quality of HDR videos by conducting the study in both a dark lab environment and a brighter living-room environment. A total of 66 subjects participated in the study and more than 20,000 opinion scores were collected, which makes this the largest in-lab study of HDR video quality ever. We anticipate that the dataset will be a valuable resource for researchers to develop better models of perceptual quality for HDR videos.

Video livestreaming is gaining prevalence among video streaming service s, especially for the delivery of live, high motion content such as sport ing events. The quality of the se livestreaming videos can be adversely affected by any of a wide variety of events, including capture artifacts, and distortions incurred during coding and transmission. High motion content can cause or exacerbate many kinds of distortion, such as motion blur and stutter. Because of this, the development of objective Video Quality Assessment (VQA) algorithms that can predict the perceptual quality of high motion, live streamed videos is greatly desired. Important resources for developing these algorithms are appropriate databases that exemplify the kinds of live streaming video distortions encountered in practice. Towards making progress in this direction, we built a video quality database specifically designed for live streaming VQA research. The new video database is called the Laboratory for Image and Video Engineering (LIVE) Livestream Database. The LIVE Livestream Database includes 315 videos of 45 source sequences from 33 original contents impaired by 6 types of distortions. We also performed a subjective quality study using the new database, whereby more than 12,000 human opinions were gathered from 40 subjects. We demonstrate the usefulness of the new resource by performing a holistic evaluation of the performance of current state-of-the-art (SOTA) VQA models. We envision that researchers will find the dataset to be useful for the development, testing, and comparison of future VQA models. The LIVE Livestream database is being made publicly available for these purposes at https://live.ece. utexas.edu/research/LIVE_APV_Study/apv_index.html.

We present a new method and a large-scale database to detect audio-video synchronization(A/V sync) errors in tennis videos. A deep network is trained to detect the visual signature of the tennis ball being hit by the racquet in the video stream. Another deep network is trained to detect the auditory signature of the same event in the audio stream. During evaluation, the audio stream is searched by the audio network for the audio event of the ball being hit. If the event is found in audio, the neighboring interval in video is searched for the corresponding visual signature. If the event is not found in the video stream but is found in the audio stream, A/V sync error is flagged. We developed a large-scaled database of 504,300 frames from 6 hours of videos of tennis events, simulated A/V sync errors, and found our method achieves high accuracy on the task.

Video live streaming is gaining prevalence among video streaming services, especially for the delivery of popular sporting events. Many objective Video Quality Assessment (VQA) models have been developed to predict the perceptual quality of videos. Appropriate databases that exemplify the distortions encountered in live streaming videos are important to designing and learning objective VQA models. Towards making progress in this direction, we built a video quality database specifically designed for live streaming VQA research. The new video database is called the Laboratory for Image and Video Engineering (LIVE) Live stream Database. The LIVE Livestream Database includes 315 videos of 45 contents impaired by 6 types of distortions. We also performed a subjective quality study using the new database, whereby more than 12,000 human opinions were gathered from 40 subjects. We demonstrate the usefulness of the new resource by performing a holistic evaluation of the performance of current state-of-the-art (SOTA) VQA models. The LIVE Livestream database is being made publicly available for these purposes at https://live.ece.utexas.edu/research/LIVE_APV_Study/apv_index.html.

We propose a new model for no-reference video quality assessment (VQA). Our approach uses a new idea of highly-localized space-time (ST) slices called Space-Time Chips (ST Chips). ST Chips are localized cuts of video data along directions that implicitly capture motion. We use perceptually-motivated bandpass and normalization models to first process the video data, and then select oriented ST Chips based on how closely they fit parametric models of natural video statistics. We show that the parameters that describe these statistics can be used to reliably predict the quality of videos, without the need for a reference video. The proposed method implicitly models ST video naturalness, and deviations from naturalness. We train and test our model on several large VQA databases, and show that our model achieves state-of-the-art performance at reduced cost, without requiring motion computation.

We propose a new prototype model for no-reference video quality assessment (VQA) based on the natural statistics of space-time chips of videos. Space-time chips (ST-chips) are a new, quality-aware feature space which we define as space-time localized cuts of video data in directions that are determined by the local motion flow. We use parametrized distribution fits to the bandpass histograms of space-time chips to characterize quality, and show that the parameters from these models are affected by distortion and can hence be used to objectively predict the quality of videos. Our prototype method, which we call ChipQA-0, is agnostic to the types of distortion affecting the video, and is based on identifying and quantifying deviations from the expected statistics of natural, undistorted ST-chips in order to predict video quality. We train and test our resulting model on several large VQA databases and show that our model achieves high correlation against human judgments of video quality and is competitive with state-of-the-art models.

With the massive growth of Internet video streaming, it is critical to accurately measure video quality subjectively and objectively, especially HD and UHD video which is bandwidth intensive. We summarize the creation of a database of 200 clips, with 20 unique sources tested across a variety of devices. By classifying the test videos into 2 distinct quality regions SD and HD, we show that the high correlation claimed by objective video quality metrics is led mostly by videos in the SD quality region. We perform detailed correlation analysis and statistical hypothesis testing of the HD subjective quality scores, and establish that the commonly used ACR methodology of subjective testing is unable to capture significant quality differences, leading to poor measurement accuracy for both subjective and objective metrics even on large-screen display devices.

Lossless image compression is particularly important in applications requiring high fidelity such as medical imaging, remote sensing and scientific imaging. These applications cannot tolerate the minute artifacts that are caused by lossy compression methods. We first describe a new predictor for lossless image compression based on plane fitting. Our main contribution is an adaptive model switching algorithm that locally selects the best predictor for each pixel based on context. Our experiments show that the new predictor substantially outperform common lossless methods such as CALIC, JPEG-LS, CCSDS SZIP and SFALIC for various medical images of different modalities (including CT and MR images) and bit depths. The simplicity and inherently parallel nature of the model switching algorithm makes a very fast implementation possible.

Carbon nanotube field-effect transistors (CNFETs) are promising candidates for building energy-efficient digital systems at highly scaled technology nodes. However, carbon nanotubes (CNTs) are inherently subject to variations that reduce circuit yield, increase susceptibility to noise, and severely degrade their anticipated energy and speed benefits. Joint exploration and optimization of CNT processing options and CNFET circuit design are required to overcome this outstanding challenge. Unfortunately, existing approaches for such exploration and optimization are computationally expensive, and mostly rely on trial-and-error-based ad hoc techniques. In this paper, we present a framework that quickly evaluates the impact of CNT variations on circuit delay and noise margin, and systematically explores the large space of CNT processing options to derive optimized CNT processing and CNFET circuit design guidelines. We demonstrate that our framework: 1) runs over 100× faster than existing approaches and 2) accurately identifies the most important CNT processing parameters, together with CNFET circuit design parameters (e.g., for CNFET sizing and standard cell layouts), to minimize the impact of CNT variations on CNFET circuit speed with ≤5% energy cost, while simultaneously meeting circuit-level noise margin and yield constraints.

Carbon nanotube field-effect transistors (CNFETs) are promising candidates for building energy-efficient digital systems at highly-scaled technology nodes. However, carbon nanotubes (CNTs) are inherently subject to variations that reduce circuit yield, increase susceptibility to noise, and severely degrade their anticipated energy and speed benefits. Joint exploration and optimization of CNT processing options and CNFET circuit design are required to overcome this outstanding challenge. Unfortunately, existing approaches for such exploration and optimization are computationally expensive, and mostly rely on trial-and-error-based ad-hoc techniques. In this paper, we present a systematic framework which quickly evaluates the impact of CNT variations on circuit delay and noise margin, and automatically explores the large space of CNT processing options to derive optimized CNT processing and CNFET circuit design guidelines. We demonstrate that: 1. Our new framework runs over 100X faster than existing approaches. 2. It accurately identifies the most important CNT processing parameters, together with CNFET circuit sizing, to minimize the impact of CNT variations while meeting circuit-level noise margin constraints.

Carbon Nanotube Field-Effect Transistors (CNFETs) are excellent candidates for building highly energy-efficient digital systems. However, imperfections inherent in carbon nanotubes (CNTs) pose significant hurdles to realizing practical CNFET circuits. In order to achieve CNFET VLSI systems in the presence of these inherent imperfections, careful orchestration of design and processing is required: from device processing and circuit integration, all the way to large-scale system design and optimization. In this paper, we summarize the key ideas that enabled the first experimental demonstration of CNFET arithmetic and storage elements. We also present an overview of a probabilistic framework to analyze the impact of various CNFET circuit design techniques and CNT processing options on system-level energy and delay metrics. We demonstrate how this framework can be used to improve the energy-delay-product (EDP) of CNFET-based digital systems.

This paper presents a novel content-adaptive image compression system. Utilizing a pattern-driven model, we explore the synergy between content-based analysis and compression. For a given image, disparate low-level visual patterns are automatically separated, modeled, and encoded using compact and “customized” features and parameters. The feasibility and efficiency of the proposed system were corroborated by quantitative experiments and comparisons. Since different patterns are separated and modeled explicitly during the compression, our method holds potentials for providing better support for compressed-domain analysis.

Carbon nanotube field-effect transistors (CNFETs) are excellent candidates for building highly energy-efficient electronic systems of the future. Fundamental limitations inherent to carbon nanotubes (CNTs) pose major obstacles to the realization of robust CNFET digital very large-scale integration (VLSI): 1) it is nearly impossible to guarantee perfect alignment and positioning of all CNTs despite near-perfect CNT alignment achieved in recent years; 2) CNTs can be metallic or semiconducting depending on chirality; and 3) CNFET circuits can suffer from large performance variations, reduced yield, and increased susceptibility to noise. Today's CNT process improvements alone are inadequate to overcome these challenges. This paper presents an overview of: 1) imperfections and variations inherent to CNTs; 2) design and processing techniques, together with a probabilistic analysis framework, for robust CNFET digital VLSI circuits immune to inherent CNT imperfections and variations; and 3) recent experimental demonstration of CNFET digital circuits that are immune to CNT imperfections. Significant advances in design tools can enable robust and scalable CNFET circuits that overcome the challenges of the CNFET technology while retaining its energy-efficiency benefits.

This paper presents a novel pattern-driven image compression technique for exploring the synergy between content-based analysis and compression. Within the pattern-driven paradigm, image data are considered as relational and classifiable entities, which are low level visual patterns including: (1) flat or homogeneous patterns (HP), (2) structural lines, curves and boundaries indicating the intensity/structural discontinuities (SD), and (3) complex/composite patterns (CP). Concisely, the pattern-based image model / can be loosely defined as fimage= fHPU fSDU fCP.

Carbon Nanotube Field-Effect Transistors (CNFETs) are excellent candidates for designing highly energy-efficient future digital systems. However, carbon nanotubes (CNTs) are inherently highly subject to imperfections that pose major obstacles to robust CNFET digital VLSI. This paper summarizes commonly raised questions and concerns about CNFET technology through a series of frequently asked questions. The specific questions addressed in this paper are motivated by recent advances in the field since the publication of our earlier paper on frequently asked questions in the Proceedings of the 2009 Design Automation Conference.

This paper presents a novel adaptive pattern-driven approach for compressing large-area high-resolution terrain data. Utilizing a pattern-driven model, the proposed approach achieves efficient terrain data reduction by modeling and encoding disparate visual patterns using a compact set of extracted features. The feasibility and efficiency of the proposed technique were corroborated by experiments using various terrain datasets and comparisons with the state-of-the-art compression techniques. Since different visual patterns are separated and modeled explicitly during the compression process, the proposed technique also holds a great potential for providing a good synergy between compression and compressed-domain analysis.

Carbon Nanotube Field-Effect Transistors (CNFETs) show big promise as extensions to silicon-CMOS because: 1) Ideal CNFETs can provide significant energy and performance benefits over silicon-CMOS, and 2) CNFET processing is compatible with existing silicon-CMOS processing. However, future gigascale systems cannot rely solely on existing chemical synthesis for guaranteed ideal devices. VLSI-scale logic circuits using CNFETs must overcome major challenges posed by: 1) Misaligned and mis-positioned Carbon Nanotubes (CNTs); 2) Metallic CNTs; and, 3) CNT density variations. This paper performs detailed analysis of the impact of these challenges on CNFET circuit performance. A combination of design and processing techniques, presented this paper, can enable VLSI-scale CNFET logic circuits that are immune to high rates of inherent imperfections. These techniques are inexpensive compared to traditional defect- and fault-tolerance, do not impose major changes in VLSI design flows, and are compatible with VLSI processing because they do not require special customization on chip-by-chip basis.

This paper presents our research and development effort towards a real-time 3-D situational awareness visualization for emergency response (3D-SAVER) in urban environments. For achieving the 3D-SAVER, key enabling technological components (including GIS image/data processing, 3-D building modeler, interactive visualization, and dynamic scene update) were developed and integrated. The resultant 3D-SAVER prototype is capable of not only automatically collecting and processing image & information (pertinent to the incident site) for generating complex building models, but also taking real-time positional measurements of people & resources within multi-story buildings and providing an interactive visualization of the integrated information to the incident commanders for 3-D situational awareness. The feasibility and efficiency of 3D-SAVER were demonstrated by a fire incident simulation using practical San Diego datasets. By enabling improved 3-D situational awareness, our 3D-SAVER holds the great potential to increase the speed and effectiveness of emergency response, thus saving lives and decreasing costs.

We introduce a novel feature set, which we call HDRMAX features, that when included into Video Quality Assessment (VQA) algorithms designed for Standard Dynamic Range (SDR) videos, sensitizes them to distortions of High Dynamic Range (HDR) videos that are inadequately accounted for by these algorithms. While these features are not specific to HDR, and also augment the equality prediction performances of VQA models on SDR content, they are especially effective on HDR. HDRMAX features modify powerful priors drawn from Natural Video Statistics (NVS) models by enhancing their measurability where they visually impact the brightest and darkest local portions of videos, thereby capturing distortions that are often poorly accounted for by existing VQA models. As a demonstration of the efficacy of our approach, we show that, while current state-of-the-art VQA models perform poorly on 10-bit HDR databases, their performances are greatly improved by the inclusion of HDRMAX features when tested on HDR and 10-bit distorted videos.