This paper presents a novel approach for unsupervised video summarization using reinforcement learning (RL), addressing limitations like unstable adversarial training and reliance on heuristic-based reward functions. The method operates on the principle that reconstruction fidelity serves as a proxy for informativeness, correlating summary quality with reconstruction ability. The summarizer model assigns importance scores to frames to generate the final summary. For training, RL is coupled with a unique reward generation pipeline that incentivizes improved reconstructions. This pipeline uses a generator model to reconstruct the full video from the selected summary frames; the similarity between the original and reconstructed video provides the reward signal. The generator itself is pre-trained self-supervisedly to reconstruct randomly masked frames. This two-stage training process enhances stability compared to adversarial architectures. Experimental results show strong alignment with human judgments and promising F-scores, validating the reconstruction objective. The code for this project will be available online1.

Autonomous driving sensors generate an enormous amount of data. In this paper, we explore learned multimodal compression for autonomous driving, specifically targeted at 3D object detection. We focus on camera and LiDAR modalities and explore several coding approaches. One approach involves joint coding of fused modalities, while others involve coding one modality first, followed by conditional coding of the other modality. We evaluate the performance of these coding schemes on the nuScenes dataset. Our experimental results indicate that joint coding of fused modalities yields better results compared to the alternatives.

Recently, soft video multicasting has gained a lot of attention, especially in broadcast and mobile scenarios where the bit rate supported by the channel may differ across receivers, and may vary quickly over time. Unlike the conventional designs that force the source to use a single bit rate according to the receiver with the worst channel quality, soft video delivery schemes transmit the video such that the video quality at each receiver is commensurate with its specific instantaneous channel quality. In this paper, we present a soft video multicasting system using an adaptive block-based compressed sensing (BCS) method. The proposed system consists of an encoder, a transmission system, and a decoder. At the encoder side, each block in each frame of the input video is adaptively sampled with a rate that depends on the texture complexity and visual saliency of the block. The obtained BCS samples are then placed into several packets, and the packets are transmitted via a channel-aware OFDM (orthogonal frequency division multiplexing) transmission system with a number of subchannels. At the decoder side, the received BCS samples are first used to build an initial approximation of the transmitted frame. To further improve the reconstruction quality, an iterative BCS reconstruction algorithm is then proposed that uses an adaptive transform and an adaptive soft-thresholding operator, which exploits the temporal similarity between adjacent frames to achieve better reconstruction quality. The extensive objective and subjective experimental results indicate the superiority of the proposed system over the state-of-the-art soft video multicasting systems.

Just noticeable difference (JND) models are widely used for perceptual redundancy estimation in images and videos. A common method for measuring the accuracy of a JND model is to inject random noise in an image based on the JND model, and check whether the JND-noise-contaminated image is perceptually distinguishable from the original image or not. Also, when comparing the accuracy of two different JND models, the model that produces the JND-noise-contaminated image with better quality at the same level of noise energy is the better model. But in both of these cases, a subjective test is necessary, which is very time consuming and costly. In this paper, we present a full-reference metric called PDP (perceptual distinguishability predictor), which can be used to determine whether a given JND-noise-contaminated image is perceptually distinguishable from the reference image. The proposed metric employs the concept of sparse coding, and extracts a feature vector out of a given image pair. The feature vector is then fed to a multilayer neural network for classification. To train the network, we built a public database of 999 natural images with distinguishbility thresholds for four different JND models obtained from an extensive subjective experiment. The results indicated that PDD achieves high classification accuracy of 97.1%. The proposed method can be used to objectively compare various JND models without performing any subjective test. It can also be used to obtain proper scaling factors to improve the JND thresholds estimated by an arbitrary JND model.

In this paper we present a novel method for full-reference image quality assessment (IQA) of tone-mapped images displayed on standard low dynamic range (LDR) displays. Due to the dynamic range compression caused by the tone-mapping process a mixture of several artifacts and distortions may be produced in the tone-mapped images. This makes the quality assessment of the tone-mapped images very challenging. Due to the diversity of such artifacts and distortions we propose a “bag of features” (BOF) approach to tackle this problem. Specifically in the proposed method a number of different perceptually relevant quality-related features are first extracted from a given tone-mapped image and its reference HDR image. These features are designed such that they capture different aspects and attributes of the tone-mapped image such as its structural fidelity naturalness and overall brightness. A support vector regressor is then trained based on the extracted features and it is used for measuring the visual quality of a tone-mapped image. Our experimental results indicate that the proposed method achieves high accuracy as compared to several existing methods.

The human visual system (HVS), like any other physical system, has limitations. For instance, it is known that the HVS can only sense the content changes that are larger than the so-called just noticeable distortion (JND) threshold. Also, to reduce the computational load on the brain, the visual attention mechanism is deployed such that regions with higher visual saliency are processed with higher priority than other less-salient regions. It is also known that visual saliency has a modulatory effect on JND thresholds. In this letter, we present a novel pixel-wise JND estimation method that considers the interplay between visual saliency and JND thresholds. In the proposed method, the largest JND thresholds of a given image are found such that the perceptual distance between the image and its JND noise-contaminated version is minimized in a perceptual space defined by the coefficients of the image in a normalized Laplacian pyramid. Experimental results indicate that the proposed method outperforms four of the latest JND models for static images.

In this paper, a novel method is presented for producing energy-efficient images, i.e., images that consume less electrical energy on energy-adaptive displays, yet have the same or very similar perceptual quality to their original images. The proposed method relies on the fact that the energy consumption of pixels in modern energy-adaptive displays like OLED displays is directly proportional to the luminance of the pixels. Hence, in this paper, to reduce the energy consumption of an image, while at the same time preserving its perceptual quality, it is proposed to reduce the luminance of the pixels in the image by one just-noticeable-difference (JND) threshold. To determine the JND thresholds, an adaptive saliency-modulated JND (SJND) model is developed. In the proposed model, the JND thresholds of each block in the given image are elevated by two non-linear saliency modulation functions using the visual saliency of the block. The parameters of the saliency modulation functions are estimated through an adaptive optimization framework, which utilizes a state-of-the-art saliency-based objective image quality assessment method. To evaluate the proposed methods, a set of subjective experiments were conducted, and the real energy consumption of the produced energy-efficient images were measured by an accurate power monitor equipment on an OLED display. The obtained experimental results demonstrated that, on average, the proposed method is able to reduce the energy consumption by about 14.1% while preserving the perceptual quality of the displayed images.

In this letter we present a novel no-reference image quality assessment (NR-IQA) method for the visual quality prediction of multiply-distorted color images. In the proposed method, to describe the image structure, a number of feature maps are first calculated based on the color Gaussian jet of the image. The popular local binary pattern operator is then applied on the computed feature maps to measure any potential structural degradations caused by multiple distortions. The performance of the proposed method was compared with 14 prominent full-reference IQA methods as well as 11 NR-IQA methods on two multidistortion IQA databases. The results indicate that the proposed method outperforms all the compared methods with a high accuracy at a moderate complexity.

In region-of-interest (ROI)-based video coding, ROI parts of the frame are encoded with higher quality than non-ROI parts. At low bit rates, such encoding may produce attention-grabbing coding artifacts, which may draw viewer's attention away from ROI, thereby degrading visual quality. In this paper, we present a saliency-aware video compression method for ROI-based video coding. The proposed method aims at reducing salient coding artifacts in non-ROI parts of the frame in order to keep user's attention on ROI. Further, the method allows saliency to increase in high quality parts of the frame, and allows saliency to reduce in non-ROI parts. Experimental results indicate that the proposed method is able to improve visual quality of encoded video relative to conventional rate distortion optimized video coding, as well as two state-of-the art perceptual video coding methods.

Error concealment in packet-loss-corrupted streaming video is inherently an under-determined problem, as there are insufficient number of well-defined criteria to recover the missing blocks perfectly. When a Region-of-Interest (ROI) based unequal error protection (UEP) scheme is deployed during video streaming-i.e., more visually salient regions are strongly protected-a lost block is likely to be of low saliency in the original frame. In this paper, we propose to add a low-saliency prior to the error concealment problem as a regularization term. It serves two purposes. First, in ROI-based UEP video streaming, low-saliency prior provides the correct side information for the client to identify the correct replacement blocks for concealment. Second, in the event that a perfectly matched block cannot be unambiguously identified, the low-saliency prior reduces viewer's visual attention on the loss-stricken region, resulting in higher overall subjective quality. We study the effectiveness of a low-saliency prior in the context of a previously proposed RECAP error concealment system. RECAP transmits a low-resolution (LR) version of an image alongside the original high-resolution (HR) version, so that if blocks in the HR version are lost, the correctly-received LR version can serve as a template for matching of suitable replacement blocks from a previously correctly-decoded HR frame. We add a low-saliency prior to the block identification process, so that only replacement candidate blocks with good match and low saliency can be selected. Further, we develop a low-complexity convex approximation to the well known Itti-Koch-Niebur saliency model, which enables the low-saliency error concealment problem to be solved efficiently. Experimental results show that: i) PSNR of the error-concealed frames can be increased dramatically (up to 3.6 dB over the original RECAP), showing the effectiveness of a low-saliency prior in the under-determined error concealment problem; and ii) subjective quality of the repaired video using our proposal, as confirmed by an extensive user study, is better than the original RECAP.

Error concealment in packet-loss-corrupted streaming video is inherently an under-determined problem, as there are insufficient number of well-defined criteria to recover the missing blocks perfectly. When a Region-of-Interest (ROI) based unequal error protection (UEP) scheme is deployed during video streaming -- i.e., more visually salient regions are strongly protected -- a %(e.g., using strong Forward Error Correction (FEC) codes) -- a lost block is likely to be of low saliency in the original frame. In this paper, we propose to add a low-saliency prior to the error concealment problem as a regularization term. It serves two purposes. First, in ROI-based UEP video streaming, low-saliency prior provides the right side information for the client to identify the correct replacement blocks for concealment. Second, in the event that a perfectly matched block cannot be unambiguously identified, the low-saliency prior reduces viewer's visual attention on the loss-stricken region, resulting in higher overall subjective quality. We study the effectiveness of a low-saliency prior in the context of a previously proposed RECAP[1] error concealment system. RECAP transmits a low-resolution (LR) version of an image alongside the original high-resolution (HR) version, so that if blocks in the HR version are lost, the correctly-received LR version can serve as a template for matching of suitable replacement blocks from a previously correctly-decoded HR frame. We add a low-saliency prior to the block identification process, so that only replacement candidate blocks with good match and low saliency can be selected. Further, we design and apply four saliency reduction operators iteratively in a loop, in order to reduce the saliency of candidate blocks. Experimental results show that: i) PSNR of the error-concealed frames can be increased dramatically (up to $3.2$dB over the original RECAP), showing the effectiveness of a low-saliency prior in the under-determined error concealment problem, and ii) subjective quality of the repaired video using our proposal, as confirmed by an extensive user study, is better than the original RECAP.

This correspondence describes a publicly available database of eye-tracking data, collected on a set of standard video sequences that are frequently used in video compression, processing, and transmission simulations. A unique feature of this database is that it contains eye-tracking data for both the first and second viewings of the sequence. We have made available the uncompressed video sequences and the raw eye-tracking data for each sequence, along with different visualizations of the data and a preliminary analysis based on two well-known visual attention models.

In this paper we present a novel perceptually-based algorithm for color quantization that produces images that consume less energy than conventionally quantized images when displayed on modern energy-adaptive displays. To evaluate the performance of the proposed algorithm, we performed a subjective study on a standard Kodak color image database. Experimental results indicate that the proposed algorithm is able to reduce the energy consumption by 4.25% on average, while achieving the same or better subjective image quality as conventional color quantization.

In region-of-interest (ROI) video coding, the part of the frame designated as ROI is encoded with higher quality relative to the rest of the frame. At low bit rates, coding artifacts in non-ROI parts of the frame may become salient and draw user's attention away from ROI, thereby degrading visual quality. In this paper we propose a saliency-preserving framework for ROI video coding. This approach aims at reducing attention-grabbing visual artifacts in non-ROI parts of the frame in order to keep user's attention on ROI. Experimental results indicate that the proposed method is able to improve the visual quality of ROI video at low bit rates.

Reference picture selection (RPS) is a powerful error-control technique for video streaming. Previously, two fast block-based motion vector concatenation (MVC) algorithms were proposed for video transcoding based on forward dominant vector selection (FDVS) and activity dominant vector selection (ADVS). In this paper, we cast these algorithms in the rate-distortion (RD) optimization framework. We also present two novel RD optimized pixel-based MVC schemes for RPS. Experimental results indicate that the proposed methods provide higher video quality compared to both FDVS and ADVS. In addition, we study the complexity of various RPS algorithms as a function of the loss rate and round trip time.

In video transmission over packet-based networks, packet losses often occur in bursts. In this paper, we present a novel packetization method for increasing the robustness of compressed video against bursty packet losses. The proposed method is based on creating a coding order of macroblocks (MBs) so that the blocks that are close to each other in the coding order end up being far from each other in the frame. We formulate this idea as a discrete optimization problem, prove its NP-hardness, and discuss several possible solution methods. Experimental results indicate that the proposed method improves the quality of reconstructed frames under burst loss by several decibels compared to conventional flexible MB ordering techniques, and about 0.7 dB compared to the state-of-the-art method called explicit chessboard wipe.

In this paper we present a high-level graphical simulator for video coding and transmission using the H.264/AVC standard. The main objective of the developed simulator is to build an overall video communication system model including source encoding, channel modeling and decoding in order to interactively investigate the performance of various H.264/AVC coding schemes in the face of bandwidth constraints and channel errors. The developed simulator can be employed as a useful research or educational tool for video communication systems based on the H.264/AVC video coding standard.

In video transmission over packet-based networks, packet losses usually occur in bursts. In this paper, we present a novel packetization method for increasing the robustness of compressed video against bursty packet losses. The proposed method is based on creating a coding order of macroblocks so that the blocks that are close to each other in the coding order end up being far from each other in the frame. Experimental results indicate that the proposed method improves the quality of reconstructed frames under burst loss by several dB compared to conventional FMO techniques, and about 0.7 dB compared to the state-of-the-art method called Explicit Chessboard-Wipe (ECW).

Reference Picture Selection (RPS) is a powerful error control technique for video streaming. Previously, two fast block-based motion vector concatenation (MVC) algorithms were proposed for video transcoding based on forward dominant vector selection (FDVS) and activity dominant vector selection (ADVS). In this paper, we examine the use of these algorithms in RPS, in the context of video transmission. We also present a novel pixel-based MVC scheme for RPS. Experimental results indicate that the proposed method provides higher video quality compared to both the FDVS and ADVS. In addition, we study the complexity of various RPS algorithms as a function of the loss rate and round trip time.