Point cloud (PC) compression is crucial to immersive visual applications such as autonomous vehicles to classify objects on the roads. The Motion Picture Experts Group (MPEG) standardization group has achieved a notable compression efficiency, called video-based PC compression (V-PCC), which consists of an encoder-decoder. The V-PCC encoder takes original 3D PC data and projects them onto multiple 2D planes to generate several 2D feature images. These images are then compressed using the well-established High-Efficiency Video Coding (HEVC) method. The V-PCC decoder uses compressed information and decoding techniques to reconstruct the 3D PC. However, the PCs produced by V-PCC are often sparse, non-uniform, and contain artifacts. In many practical applications, it is necessary to recover complete PCs from partial ones in real time. This article presents a method for enhancing decoded PCs as a post-processing step in the V-PCC with reduced computational time. Our approach involves a 2D upsampling for the V-PCC occupancy image, which increases the density of the PC, and a 2D high-resolution auxiliary information modification algorithm for the 2D-3D conversion of high-resolution 3D PCs, which improves the uniformity and reduces the noise in the PC. The 3D high-resolution PC has been further enhanced using the developed 3D outlier removal and point regeneration algorithm. Our proposed work can significantly simplify the state-of-the-art super resolution methods for PCs and reduce the time complexity of 61–75% while maintaining a high level of quality in PCs.

With the advent of immersive multimedia technologies, interest in dynamic point cloud compression standards, such as video-based point cloud compression (V-PCC), has been steadily growing. At the core of V-PCC, conventional video codecs such as High Efficiency Video Coding are employed for encoding the geometry video, attribute video and the occupancy map. Standard video codecs perform temporal prediction using motion compensation by simple pixel-copying from the reference frame. However, conventional pixel-domain prediction does not exploit spatial correlation within the block. This is highly sub-optimal especially due to large homogeneous regions in the geometry video. In this paper, spatial decorrelation is first achieved via the Discrete Cosine Transform. Prediction is then performed in the transform domain, where the observed variation in temporal correlation across frequencies is exploited for a more accurate prediction. Design of the correlation filters required for this procedure poses challenges. The inherent instability in closed-loop design is tackled using the Asymptotic Closed Loop paradigm which uses an iterative open-loop algorithm that asymptotically converges to closed-loop operation. Experimental results on geometry videos generated from common test conditions show significant bitrate savings of up to 5.8% (about 3% on average) and highlight the efficacy of our approach.

Video compression is an important procedure for digital applications. A commonly used format for inputs to video coders is YUV420, which is a subsampled result from YUV444, that is transformed and demosaicked from a color filter array (CFA) format. The process of chroma subsampling is a crucial step for the reconstructed image quality. A state-of-the-art method utilizes the solution of a prior method to compute the starting point, and devises a search method to improve the results. However, the search method does not consider the optimality condition of the problem, and the overall computational complexity is high because of the execution of the prior method for the starting point. In the proposed work, the cost function of the optimization problem is analyzed and decomposed into 4 subchannels based on the CFA format. The optimal line of each subterm is studied in 3-dimensional space. The problem is then mathematically reduced to a 3-subchannel problem. Based on the combinations of the optimal lines, a triangle search area is formed and proved to contain the optimal solution. And because the derived triangle search area is small on average, the search for the optimal value can be considerably fast. Experimental results demonstrate that the proposed approach can produce reconstructed images with qualities that are exactly the same as those generated using the state-of-the-art method. Relative to the state-of-the-art method, the proposed approach can reduce the time complexity by 84.08%–84.91%, and reduce the number of search points by 40.24%–85.25%, for the image datasets Kodak and IMAX with different demosaicking methods.

The classical chroma subsampling involves downsampling chrominance components (U and V) while maintaining the luminance component (Y). Recently, a study has attempted to change Y for the chroma-subsampling process, and improved results in the reconstructed image quality are obtained. However, computational complexity remains an issue because the study examines all the candidates in the determined range for minimal pixel distortion for the modified Y; the complexity can be high if the range is large. In this study, we reduced the candidate number to seven at most, which is mathematically optimized. Boundary points for in-the-range red, green, and blue (RGB) values are first decided, followed by the determination of the intervals that concatenate the entire curve (at most seven intervals). Each interval belongs to one of the seven sub-cases of different linear combinations of individual curves. The optimal solution (modified Y) for each sub-case is derived, which is in highly efficient form. Compared with the existing method, the proposed fast method ensures that the image quality is preserved, while the number of search candidates is reduced by 61.30%-69.19% on average, and the computational time of the process is reduced by 31.39%-58.97%, thereby demonstrating an efficient performance.

The next-generation video compression standard H.266/Future Video Coding (FVC) provides high compression efficiency in terms of the cost of computing the optimal intra mode from 67 modes. We propose an intra mode prediction method based on a convolutional neural network (CNN). An input image set of 20 × 20 blocks is used to train the CNN; the CNN is used to predict the best classes of intra mode direction. The CNN architecture comprises two convolutional layers and a fully connected layer. Compared with the default fast search method in FVC, the proposed method can achieve a 0.033% decrease in Bjøntegaard delta bit rate (BDBR) with only a slight increase in time.

The images obtained by the camera sensors are stored in the format of CFA, Color Filter Array, which only contains limited information about the R, G, and B signals of images. The images in CFA are upsampled into RGB 444 format by demosaicking algorithms, converted into YCbCr 444, then chroma-downsampled into YCrCb 420 format for image/video compression. In this paper, a novel chroma-downsampling method is developed to improve a state-of-the-art method. The optimization problem for the proposed method considers more factors and provides a better closed-form solution for the optimally sampled chroma values. Reversely in the decoder, a chroma-upsampling method is required for the image reconstruction. In the proposed chroma-upsampling method, the neighboring chroma pixels and their distances to the recovered pixel are considered; the procedure is to produce weightings to redistribute the total pixel values in the original estimates. Combining the proposed chroma-downsampling and the proposed chroma-upsampling works, the proposed method outperforms the state-of-the-art method by BDPSNR (Bjøntegaard Delta peak signal-to-noise ratio) 0.3161 for H.264/AVC, 0.3845 for HEVC (High Efficiency Video Compression) and 0.3739 for VVC (Versatile Video Coding), averaged over all tested videos and all 7 different considered CFA formats.

In this paper, we consider an integrated error concealment system for lost color frames and lost depth frames in multiview videos with depths. We first proposed a pixel-based color error-concealment method with the use of depth information. Instead of assuming that the same moving object in consecutive frames has minimal depth difference, as is done in a state-of-the-art method, a more realistic situation in which the same moving object in consecutive frames can be in different depths is considered. In the derived motion vector candidate set, we consider all the candidate motion vectors in the set, and weight the reference pixels by the depth differences to obtain the final recovered pixel. Compared with the two state-of-the-art methods, the proposed method has average peak signal-to-noise ratio gains of up to 8.73 and 3.98 dB, respectively. Second, we proposed an iterative depth frame error-concealment method. The initial recovered depth frame is obtained by depth-image-based rendering from another available view. The holes in the recovered depth frame are then filled in the proposed priority order. Preprocessing methods (depth difference compensation and inconsistent pixel removal) are performed to improve the performance. Compared with a method that uses the available motion vector in a color frame to recover the lost depth pixels, the hybrid motion vector extrapolation method, the inpainting method and the proposed method have gains of up to 4.31, 10.29, and 6.04 dB, respectively. Finally, for the situation in which the color and the depth frames are lost at the same time, our two methods jointly perform better with a gain of up to 7.79 dB.

We examine the visual effect of whole frame loss by different decoders. Whole frame losses are introduced in H.264/AVC compressed videos which are then decoded by two different decoders with different common concealment effects: frame copy and frame interpolation. The videos are seen by human observers who respond to each glitch they spot. We found that about 39% of whole frame losses of B frames are not observed by any of the subjects, and over 58% of the B frame losses are observed by 20% or fewer of the subjects. Using simple predictive features which can be calculated inside a network node with no access to the original video and no pixel level reconstruction of the frame, we developed models which can predict the visibility of whole B frame losses. The models are then used in a router to predict the visual impact of a frame loss and perform intelligent frame dropping to relieve network congestion. Dropping frames based on their visual scores proves superior to random dropping of B frames.

We propose a packet dropping algorithm for various packet loss rates. A network-based packet loss visibility model is used to evaluate the visual importance of each H.264 packet inside the network. During network congestion, based on the estimated loss visibility of each packet, we drop the least visible frames and/or the least visible packets until the required bit reduction rate is achieved. Based on a computable perceptually-based metric, our algorithm performs better than an existing approach (dropping B packets or frames).

Individual packet losses can have differing impact on video quality. Simple factors such as packet size, average motion, and DCT coefficient energy can be extracted from an individual compressed video packet inside the network without any inverse transforms or pixel-level decoding. Using only such factors that are self-contained within packets, we aim to predict the impact on quality as measured by VQM (video quality metric) that the loss of this packet would entail. In the context of both compression artifacts and packet loss artifacts, we develop generalized linear models to predict VQM scores and our final model gives a good performance on objective evaluation of packet importance.

We conduct subjective experiments on visual quality following packet loss, and then construct models to predict these visual importance scores. The models are fully self-contained at the packet level, meaning that they use only information within one packet to predict the importance of that packet, requiring no frame-level reconstruction nor any information on the reference frame. Models are created for SDTV and HDTV resolutions, and the differences in the important factors between them are discussed.

In an error-prone communication channel, more important video packets should be assigned stronger channel codes. With various packet sizes and distortions for each packet, we use the subgradient method to search in the dual domain for the optimal RCPC channel code rate allocation for each packet, to minimize the end-to-end video quality degradation for an AWGN channel. We exploit the advantage of not sending or not coding packets of lower importance.

In this paper, we propose a generalized linear model for video packet loss visibility that is applicable to different group-of-picture structures. We develop the model using three subjective experiment data sets that span various encoding standards (H.264 and MPEG-2), group-of-picture structures, and decoder error concealment choices. We consider factors not only within a packet, but also in its vicinity, to account for possible temporal and spatial masking effects. We discover that the factors of scene cuts, camera motion, and reference distance are highly significant to the packet loss visibility. We apply our visibility model to packet prioritization for a video stream; when the network gets congested at an intermediate router, the router is able to decide which packets to drop such that visual quality of the video is minimally impacted. To show the effectiveness of our visibility model and its corresponding packet prioritization method, experiments are done to compare our perceptual-quality-based packet prioritization approach with existing Drop-Tail and Hint-Track-inspired cumulative-MSE-based prioritization methods. The result shows that our prioritization method produces videos of higher perceptual quality for different network conditions and group-of-picture structures. Our model was developed using data from high encoding-rate videos, and designed for high-quality video transported over a mostly reliable network; however, the experiments show the model is applicable to different encoding rates.

Our work builds a general visibility model of video packets which is applicable to various types of GOP (group of pictures). The data used for analysis and building the model come from three subjective experiment sets with different encoding and decoding parameters on H.264 and MPEG-2 videos. We consider factors not only within a packet but also across its vicinity to account for possible temporal and spatial masking effects. This model can be useful for an intermediate router in a congested network to drop less visible packets to maintain overall video quality. Experiments are done to compare our perceptual-quality-based packet dropping approach with existing drop-tail and hint-track-inspired cumulative-MSE-based dropping methods. The result shows that our dropping method produces videos of higher perceptual quality for different network conditions and GOP structures.

When video packets are lost in congested networks, one loss pattern creates a different visual impact than another. We conduct a subjective experiment with H.264 videos and conclude that isolated losses are better than bursty losses in terms of perceptual video quality. A network-implementable video quality model is developed for a router to drop packets so as to achieve good visual quality.