In this paper, we propose a cross-space alignment-based attribute artifact removal method for V-PCC. Firstly, we construct a cross-space frame alignment module to align the adjacent projected 2D frames based on the temporal continuity between the 3D point clouds. Then, we design a multi-frame fusion network that is developed based on U-Net to fuse the current frame with the aligned frames to remove the artifact that occurred in the attribute. Experimental results demonstrate that our proposed method achieves superior performance for the enhancement of point clouds.

Video-based point cloud compression (V-PCC) converts the dynamic point cloud data into video sequences using traditional video codecs for efficient encoding. However, this lossy compression scheme introduces artifacts that degrade the color attributes of the data. This paper introduces a framework designed to enhance the color quality in the V-PCC compressed point clouds. We propose the lightweight de-compression Unet (LDC-Unet), a 2D neural network, to optimize the projection maps generated during V-PCC encoding. The optimized 2D maps will then be back-projected to the 3D space to enhance the corresponding point cloud attributes. Additionally, we introduce a transfer learning strategy and develop a customized natural image dataset for the initial training. The model was then fine-tuned using the projection maps of the compressed point clouds. The whole strategy effectively addresses the scarcity of point cloud training data. Our experiments, conducted on the public 8i voxelized full bodies long sequences (8iVSLF) dataset, demonstrate the effectiveness of our proposed method in improving the color quality.

Talking-head video is very popular in video conference and social media, where the camera captures the movement of user’s head and the change of facial expression. In this paper, we propose a hierarchical coding scheme for the compression of talking-head video. In our proposed method, three data layers, including one base layer, one enhancement layer and one feature layer, are formed as the input of encoder. More specifically, the base layer is generated by spatially sub-sampling the source video. The enhancement layer is composed by the specific key frames and the feature layer is produced based on the extracted facial landmarks. These layers are separately compressed but fused together to reconstruct the video signal in the decoder side. To achieve a high-quality reconstruction, we design the multi-feature fusion network in which the feature layer is used to guide the fusion of base layer and enhancement layer. The experiment results demonstrate the good performance of our proposed method for the coding of talking-head video.

An alternative-transforms-based scheme has recently been proposed to achieve perceptual encryption of video signals in which multiple transforms are designed by using different rotation angles at the final stage of the discrete cosine transforms (DCTs) butterfly flow-graph structure. More recently, it is found that a set of more efficient alternative transforms can be derived by introducing sign-flips at the same stage, which is equivalent to an extra rotation angle of π. In this paper, we generalize this sign-flipping technique by randomly embedding sign-flips into all stages of the DCTs butterfly structure so that the encryption space becomes much larger to yield a higher security. We pursue this study for H.264-compatible videos, assuming that the integer DCT of size 4 × 4 is used. First, we follow the separable implementation of the 4 × 4 2-D DCT in which different sign-flipping strategies will be employed along its horizontal and vertical dimensions. Second, we convert the 4 × 4 2-D DCT into a 16-point 1-D butterfly structure so that more sign-flips can be embedded at its various stages. Third, we choose different schemes to pair the node-variables in the 16-point 1-D butterfly structure, thus further enlarging the encryption space. Extensive experiments are conducted to show the performance of these improved encryption schemes and some security analyzes are also presented to confirm their persistence to various attacking strategies.

Perceptual (or partial) video encryption has recently been achieved via a random-key controlled selection among multiple transforms, where these transforms are designed by modifying the rotation angles at the last stage of the DCT's butterfly structure. Later on, it was found that more efficient transforms can be obtained by applying random sign-flips at the same stage, which is equivalent to an extra rotation angle of 180°. In this paper, we generalize this technique by embedding random sign-flips into other stages in the DCT's implementation structure. To this end, we first convert the separable implementation of the H.264 4×4 2-D DCT into a 1-D 16-point structure so that more sign-flips can be embedded at its various stages. Then, we introduce different pairing schemes (i.e., forming N/2 pairs for a group of N node-variables) to further increase the encryption space. Extensive experiments are carried out to show the performance of this improved video encryption scheme; whereas security analyses are conducted to show the newly designed transforms out-perform the previous ones in terms of search space and encryption efficiency.

It has been demonstrated in our earlier works [1, 2] that perceptual video encryption can be effectively achieved by using multiple transforms where the block size 4×4 has been considered. In this paper, we study the extension to the transforms of size 8×8. In this case, a more complex flow-graph structure is resulted, thus leading to a larger room for encryption. In addition, special technique on controlling the encrypted video quality is presented by carefully selecting the number of rotations in the flow-graph structure of an 8×8 transform. The proposed scheme is first evaluated using the high profile of H.264. It is then further tested for the MPEG-4 standard that completely relies on 8×8 transform. Both cases show that promising results can be achieved with our proposed scheme.

Multiple-description coding (MDC) provides a promising solution to support the error-prone transmission over multiple channels. One extremely important application is to design efficient multiple description video coding (MDVC) systems for the peer to peer (P2P) scenario. To this end, one needs to solve the mismatching problem between the reference frames used at the encoder and decoder sides (for motion compensation) in most non-scalable MDVC schemes or get rid of the inter-dependency within various enhancement layers in the scalable MDVC schemes. In the meantime, it is highly preferable to enforce that all descriptions transmitted over the network are well-balanced so as to have an equal payload to each peer. In this paper, we propose an MDVC scheme with a number of well-balanced descriptions. These descriptions are generated from some newly-developed unitary transforms and they can solve all of the problems mentioned above.

In our earlier work , we proposed for the first time that the perceptual video encryption be performed at the transformation stage by selecting one out of multiple unitary transforms according to the encryption key. In this letter, we aim to design some more efficient transforms to be used in this framework. Two criteria are followed for designing such transforms: 1) they are significantly different from discrete cosine transform (DCT) or discrete sine transform (DST), and 2) the resulted coding efficiency is exactly the same to what can be achieved by using DCT or just falls very slightly. As a result, we find that these transforms are actually derived by the sign-flipping on some node-variables in the flow-graph structure of DCT - a special case of the plane-based rotations (by π or 180°). Extensive simulations based on the H.264 codec are performed to demonstrate their effectiveness through both objective and subjective assessments. Finally, we present the security analysis to show the resistance of our algorithms to different types of attacks.

It has been demonstrated in our earlier work [1] that partial video encryption can be effectively achieved by using alternative transforms. In this paper, we modify those floating-point based transforms to integer-based counterparts so as to be compatible with the H.264 standard. To this end, we present the design details of these integer-based transforms and their efficient implementations. To follow the H.264 standard that combines the transform and quantization processes, we present some special handling of the necessary scaling factors and the associated quantization. Finally, we demonstrate that the derived integer transforms can achieve partial video encryption with nearly the same performance as what has been achieved with floating-point transforms.

It is well known that the discrete cosine transform (DCT) closely approximates the optimal Karhunen-Loève transform (KLT) under the first-order stationary Markov condition with a strong inter-pixel correlation. However, if the inter-pixel correlation is weak or becomes negative, transforms other than the DCT would possibly become better. In this letter, we present a design framework for finding new unitary transforms according to two principles: 1) they are indeed more efficient than the DCT in encoding of signals with a weak or negative inter-pixel correlation (e.g., the residual signals after motion-compensation or intra- prediction) and 2) each has a fixed transform matrix (i.e., signal-independent) and can be implemented as efficiently as the DCT. Some of the new transforms will be used to demonstrate an improved R-D performance in the video coding scenario.

Multiple-description coding (MDC) encodes a source image or video into several distinct descriptions. Each description can be decoded independently and more than one description can also be decoded jointly so as to deliver a higher quality. One typical design of an MDC encoder is based on the so-called multiple description scalar quantization (MDSQ). A simple MDSQ system is to choose different step-sizes used in all involved quantizers. Another implementation is to apply flexible dead-zones in all quantizers (whereas keeping the same step-size across them). Two issues turn to be particularly important when MDC is used in video streaming applications in the P2P scenario. First, we need to consider more than two descriptions to fit the P2P reality. Second, we have to eliminate any possible drifting errors that are resulted from the use of different reference frames (in different descriptions) at the motion-compensation stage. In this paper, we introduce a DCT-domain translation to solve the mismatching problem in different reference frames. Then, we present some comparative results of the two approaches mentioned above, with detailed pros and cons for each one as well as some practical considerations.

Pixel-domain sub-sampling based on the quincunx lattice is one of the traditional approaches to carry out multiple description coding (MDC). Unfortunately, the re-alignment (either horizontal or vertical) would deteriorate the sample correlation within each description, thus leading to a lower coding efficiency. In this paper, we propose to apply a diamond-shape DCT on the quincunx sub-sampling lattice to create an MDC representation. A theoretical analysis is first presented assuming the covariance matrix of the Toeplitz type. The results show that such DS-DCT can achieve a better coding gain as well as energy packing efficiency as compared to the normal DCT with re-alignment. Then, the proposed DS-DCT is tested on various images using the standard JPEG compression scheme and the results show that it out-performs the traditional re-alignment approach by a similar margin as obtained in our theoretical analysis.

In this letter, we propose a novel video encryption technique that is used to achievepartialencryption where an annoying video can still be reconstructed even without the security key. In contrast to the existing methods where the encryption usually takes place at the entropy-coding stage or the bit-stream level, our proposed scheme embeds the encryption at thetransformstage during the encoding process. To this end, we develop a number of new unitary transforms that are demonstrated to be equally efficient as the well-known DCT and thus used as alternates to DCT during the encoding process. Partial encryption is achieved through alternately applying these transforms to individual blocks according to a pre-designed secret key. Analysis on the security level of this partial encryption scheme is carried out against various common attacks and some experimental results based on H.264/AVC are presented.

Traditionally, any 2D transform (such as 2D DCT) is implemented through two separable 1D transforms along the vertical and horizontal dimensions. Such a framework is however not most suitable for a 2D directional source in which the dominant directional information is neither horizontal nor vertical. In this letter, we attempt to determine the R-D performance upper bound for block-based transform coding schemes applied on such 2D directional sources. It is not a surprise that the Karhunen-Loeve transform (KLT) plays a critical role here. Specifically, we show that a nonseparable KLT can be determined directly from the given 2D directional source model to yield the R-D performance upper bound. We also show that there exists a significant gap between this upper bound and the R-D performance that can be achieved by using the traditional 2D DCT.

Speaker adaptation has been widely used in speech recognition. With small amount of adaptation data, Reference Speaker Weighting (RSW) adaptation was previously proposed for fast HMM adaptation, and has been shown to outperform the more commonly used maximum likelihood linear regression (MLLR) adaptation. Extending our previous work [1,2] of applying the Polynomial Segment Models (PSMs) in large vocabulary continuous speech recognition (LVCSR) on the WSJ Nov 92 evaluation, we derive the PSM-based RSW fast adaptation technique in this paper. Different from the HMMs, in which the model means are constants within a state, the PSM means are curves represented by polynomials. Experimental results showed that the PSM-based RSW gave approximately the same relative improvement over the unadapted model as in the HMM case. Comparing the PSM-based RSW and MLLR, the PSM-based RSW is more powerful when the amount of adaptation data available is limited. However, it could quickly saturate with increase in adaptation data.

Robustness has been an important issue for applying speech technologies to real applications. While the Polynomial Segment Models (PSMs) have been shown to outperform HMM under the clean environment, the segmental likelihood evaluation may make the PSM distributions sharper and may adversely affect their performance in mis-matched conditions. In this paper, we explore the robustness properties of the PSM under noisy and channel mis-match conditions. In addition, unsupervised adaptation techniques have been shown to work well for environmental adaptation even with small amount of adaptation data. Thus, it is interesting to compare the PSMs' and the HMMs' performances after applying two types of unsupervised adaptation: the Maximum Likelihood Linear Regression (MLLR) and the Reference Speaker Weighting (RSW). Experiments were performed on the Aurora 4 corpus under both clean and multi-conditional training. Our results show that even under noisy and mis-match conditions, the PSMs performed well compared to the HMMs both before and after environmental adaptation. Using the best lattice, the RSW adapted PSM gave word error rates of 26.5% and 21.3% for clean and multi-conditional training respectively which were approximately 24% better than the unadapted HMM.

Speaker adaptation has long been applied to improve speech recognition performance of hidden Markov model (HMM)-based systems. Recently, the polynomial segment model (PSM) has been shown as a viable alternative that can significantly improve the performance of large vocabulary continuous speech recognition (LVCSR). In this letter, we extend the widely used HMM-based maximum likelihood linear regression (MLLR) speaker adaptation technique to PSMs. PSM properties, such as using segment as a modeling unit, and a polynomial curve as model mean, are taken into account in deriving the PSM-based MLLR. Experiments show that PSM-based MLLR adaptation performs equally well as the HMM-based MLLR adaptation with about 19% relative improvement from the SI model. In addition, another 5% relative improvement can be obtained by combining the adapted PSMs and HMMs

Polynomial segment models (PSMs), which are generalization of the hidden Markov models (HMMs), have opened an alternative research direction for speech recognition. However, they have been limited by their computational complexity. Traditionally, any change in PSM segment boundary requires likelihood recomputation of all the frames within the segment. This makes the PSM's segment likelihood evaluation an order of magnitude more expensive than the HMM's. Furthermore, because recognition using segment models needs to search over all possible segment boundaries, the PSM recognition is computationally unfeasible beyond N-best rescoring. By exploiting the properties of the time normalization in PSM, and by decomposing the PSM segment likelihood into a simple function of "sufficient statistics", in this paper, we show that segment likelihood can be evaluated efficiently in an order of computational complexity similar to HMM. In addition, by reformulating the PSM recognition as a search for the optimal path through a graph, this paper introduces a fast PSM search algorithm that intelligently prunes the number of hypothesized segment boundaries, such that PSM recognition can be performed in an order of complexity similar to HMM. We demonstrate the effectiveness of the proposed algorithms with experiments using a PSM-based recognition system on two different recognition tasks: TIDIGIT digit recognition and the Wall Street Journal dictation task. In both tasks, PSM recognition is feasible and out-performed traditional HMM by more than 14%.

The polynomial segment model (PSM) has opened up an alternative research direction for acoustic modeling. In our previous papers, we proposed efficient incremental likelihood evaluation and EM training algorithms for PSM, that significantly improve the speed of PSM training and recognition. In this paper, we shift our focus to use PSM on large vocabulary recognition. Recognition via N-best re-scoring shows that PSM models out-performed HMM on the 5 k closed vocabulary Wall Street Journal Nov 92 testset. Our best PSM model achieved 7.15% WER compare with 7.81% using 16 mixture HMM model. Specifically, we used sub-phonetic PSM that represents a phoneme as multiple independent segmental units that allows for more effective model sharing. Also, we derived and compared different top-down mixture growing approaches that are orders of magnitude more efficient than previously proposed bottom-up agglomerative clustering techniques. Experimental results show that the top-down clustering performs better than the bottom-up approaches.

With the advance in semiconductor memory and the availability of very large speech corpora (of hundreds to thousands of hours of speech), we would like to revisit the use of discrete hidden Markov model (DHMM) in automatic speech recognition. To estimate the discrete density in a DHMM state, the acoustic space is divided into bins and one simply count the relative amount of observations falling into each bin. With a very large speech corpus, we believe that the number of bins may be greatly increased to get a much higher density than before, and we will call the new models, the high-density discrete hidden Markov model (HDDHMM). Our HDDHMM is different from traditional DHMM in two aspects: firstly, the codebook will have a size in thousands or even tens of thousands; secondly, we propose a method based on scalar quantization indexing so that for a d-dimensional acoustic vector, the discrete codeword can be determined in O(d) time. During recognition, the state probability is reduced to an O(1) table look-up. The new HDDHMM was tested on WSJO with 5K vocabulary. Compared with a baseline 4-stream continuous density HMM system which has a WER of 9.71 %, a 4-stream HDDHMM system converted from the former achieves a WER of 11.60%, with no distance or Gaussian computation.

The introduction of Aurora 4 tasks provides a standard database and methodology for comparing the effectiveness of different robust algorithms on LVCSR. One important issue on Aurora 4 tasks is the computation time involved in evaluating different test conditions. In this paper we show that by employing HTK as the recognition frontend and backend on Aurora 4 tasks with the use of cepstral mean subtraction, 14% relative improvement is achieved on the baseline clean train tasks at a 82.5% time reduction in training time and 40% time reduction on decoding. Furthermore, we found that optimizing the model complexity can increase the recognition performance (in both computation time and accuracy). Accuracy can be further improved with the use of unsupervised MLLR adaptation on one or multiple sentences. The adaptation results show that most of the gain from adaptation comes from adapting to the environment instead of to the speaker. With the use of adaptation,the error rate is reduced from the baseline result of 69.6% to 40%.