The newest video coding standard, Versatile Video Coding (VVC), adopts a quad-tree (QT) plus multi-type tree (QTMT) block partition structure and improves the compression performance by about 30%∼50%, compared with the HEVC, at the cost of higher time complexity. To reduce VVC time complexity, we proposed to use a learned model to predict Coding Unit (CU) split modes and setup thresholds based on statistical image features to eliminate unnecessary Rate-Distortion Optimization (RDO) operations. Experiments showed that, compared with the default VVC intra-coding, the proposed method saves 46.73% of encoding time, with Bjøntegaard Delta Bit Rate (BDBR) increment of 1.16%. After retraining the learned model with a specified Quantization Parameter (QP), the time reduction rate can achieve 51.79%, and the BDBR slightly increased to 2.07%. The proposed speedup coding scheme effectively reduced the VVC time complexity to make it feasible for practical application.

Performing Content-Based Image Retrieval (CBIR) from Internet databases connected through Peer-to-Peer (P2P) network, abbreviated as P2P-CBIR, helps to effectively explore the large-scale image database distributed over connected peers. Decentralized unstructured P2P framework is adopted in our system to compromise with the structured one while still reserving flexible routing control when peers join/leave or network fails. The P2P-CBIR search engine is designed to provide multi-instance query with multi-feature types to effectively reduce network traffic and maintain high retrieval accuracy. The proposed P2P-CBIR system is also designed to provide scalable retrieval control, which can adaptively control the query scope and progressively refine the accuracy of retrieved results. We also proposed to provide the most updated local database characteristics for the P2P-CBIR users. By reconfiguring system at each regular interval time, it can effectively reduce trivial peer routing and retrieval operations due to imprecise configuration. Experiments demonstrated that the average recall rate of the proposed P2P-CBIR with reconfiguration is higher than the one without about 20%, and the latter outperforms previous methods, i.e., firework query model (FQM) and breadth-first search (BFS) about 20% and 120%, respectively, under the same range of TTL values.

Multiple description coding (MDC) decomposes one single media into several descriptions and transmits them over different channels for error resilience. Each description contributes to improving the reconstructed media quality when decoded. Distributed video coding (DVC) encodes multiple correlated images and utilizes error correction codes to shift the codec complexity to a joint decoder. Combining MDC with DVC (MDVC) yields a stable codec for mobile encoders. In this paper, to improve the MDVC codec performance, image correlations among the MDVC processing modules were exploited to improve reconstructed video quality and enhance transmission robustness. At the side encoder, a DVC-based adaptive differential pulse code modulation was designed to remove interframe redundancy to enhance rate-distortion performances. For the MDVC central decoding, intradescription and interdescription correlations were utilized to dynamically select the best reconstructed frames from two descriptions, instead of selecting just one description or all key-frames from two descriptions. Experiments showed that, as compared to previous methods, the proposed MDVC control method yielded 1-2 dB higher in image PSNRs for Wyner-Ziv reconstructed frames at the side decoder when encoding low-to-medium complexity videos. For high-complexity videos, it effectively prevents error correction of Wyner-Ziv frames from malfunctioning and yields about 3 dB higher in PSNR. The proposed MDVC central decoder control yields 1-4 dB higher PSNRs, as compared to side decoders. Under lossy transmission, it demonstrates 27-64% smaller PSNR variations, as compared to that of combining key-frames as the decoded video. The proposed MDVC system and control not only improve the DVC reconstructed video quality, but also reduce the quality fluctuation artifacts of MDC coded video for mobile coders.

Processing images for specific targets on a large scale has to handle various kinds of contents with regular processing steps. To segment objects in one image, we utilized dual multiScalE Graylevel mOrphological open and close recoNstructions (SEGON) to build a background (BG) gray-level variation mesh, which can help to identify BG and object regions. It was developed from a macroscopic perspective on image BG gray levels and implemented using standard procedures, thus robustly dealing with large-scale database images. The image segmentation capability of existing methods can be exploited by the BG mesh to improve object segmentation accuracy. To evaluate the segmentation accuracy, the probability of coherent segmentation labeling, i.e., the normalized probability random index (PRI), between a computer-segmented image and the hand-labeled one is computed for comparisons. Content-based image retrieval (CBIR) was carried out to evaluate the object segmentation capability in dealing with large-scale database images. Retrieval precision-recall (PR) and rank performances, with and without SEGON, were compared. For multi-instance retrieval with shape feature, AdaBoost was used to select salient common feature elements. For color features, the histogram intersection between two scalable HSV descriptors was calculated, and the mean feature vector was used for multi-instance retrieval. The distance measure for color feature can be adapted when both positive and negative queries are provided. The normalized correlation coefficient of features among query samples was computed to integrate the similarity ranks of different features in order to perform multi-instance with multifeature query. Experiments showed that the proposed object segmentation method outperforms others by 21% in the PRI. Performing SEGON-enabled CBIR on large-scale databases also improves on the PR performance reported elsewhere by up to 42% at a recall rate of 0.5. The proposed object segmentation method can be extended to extract other image features, and new feature types can be incorporated into the algorithm to further improve the image retrieval performance.

With the advance of multi-view video codec (MVC) technology, the multimedia platform can display videos of different views. The distributed video coder (DVC) that adopts the Wynzer-Ziv (WZ) codec can shift encoding complexity to decoder under the MVC framework, denoted as multi-view DVC (MDVC), for possible low-complexity encoder applications. For side information (SI) generation, the intra- and inter-view video correlations among images in MDVC were exploited to improve the SI confidence to yield better reconstructed WZ images. We proposed a new block matching and prediction (BMP) algorithm based on Scale Invariant Feature Transform, abbreviated as SIFT-BMP, to yield more accurate SI to improve the reconstructed video quality. Simulations verified that the proposed SIFT-BMP yielded more accurate SI and higher PSNR of reconstructed images, as compared to previous methods. The PSNR improvements are 1.6~2.5 dB and higher, as compared to MCTI and H.264/AVC, for medium to high complexity videos.

Performing Content-Based Image Retrieval (CBIR) on Internet connected databases through Peer-to-Peer (P2P) network (P2P-CBIR) effectively explores the large-scale image database distributed over connected peers. In additional to enlarge the retrieval scale from server-client to P2P networks, the required computation and network traffics for performing CBIR can also be distributed. Decentralized unstructured P2P framework is adopted to perform the P2P-CBIR in considering control flexibility, under which reconfiguration is feasible to improve the retrieval performance. Based on this framework, the proposed P2P-CBIR can progressively refine the accuracy of retrieved images when they are transmitted on the routes toward the query peer. With flexible P2P framework, reconfiguration can be carried out on the P2P-CBIR, which helps to improve retrieval efficiency. Experiments verified that the P2P-CBIR with reconfiguration outperforms the one without, and the latter outperforms server-client approaches in both retrieval accuracy and network traffics.

Multiple description coding (MDC) provides stable wireless multimedia communications with the help of multiple transmission paths, while distributed video coding (DVC) effectively shifts the encoder complexity to decoder for mobile media devices. We proposed to combine MDC with DVC, abbreviated as MDVC, to construct a reliable and efficient video codec system. To improve the codec performance, the DPCM prediction gain is exploited from the correlations among sub-video sequences inside the MDVC to increase the reconstructed video quality. In addition, a MDVC central decoder control method, operated dynamically at frame-level, is proposed to improve the adaptation capability, as compared to the current MDC central decoder that selects one best description for the output video. The intra-and inter-correlations among MDC descriptions were investigated to make this frame selection decision for the central decoder. Simulations showed that the proposed MDVC control method demonstrates higher average PSNR and lower PSNR variations of the output videos, with the help of DPCM prediction gain and the efficient central decoder control method that correctly selected high quality WZ frames for low quality key-frames.

A simple and regular foreground identification (FGID) method is proposed for image indexing. The gray-level Morphological open/close by reconstruction (MOR/MCR) is operated on one image in a dual and multi-scale approach to construct a background gray-level mesh to distinguish foregrounds (FGs). The highly regular MOR/MCR operations make it feasible to deal with FG segmentation of volume images. The FGID efficiency is verified by the image retrieval performance, i.e., recall, precision and rank.With precisely identified FGs, MPEG-7 shape descriptors, in additional to color ones, can be used to improve the image retrieval performance. For the retrieval unit, a greedy boosting retrieval method is used to perform shape-based multiinstance query in considering the feature element dependency. To perform multi-instance query with multiple features, the retrieval unit integrates the similarity ranks of different feature types according to the feature saliency among query samples to yield the final similarity rank. The normalized correlation coefficient of features among query samples is computed to provide weighting factors for integrating ranks. Experiments show that the FGID unit helps much in improving the retrieval performances, i.e., 7% improvement for the precision-recall (PR) and 20% improvement for the averaged normalized modified retrieval rank (ANMRR), as compared to non-FGID ones.

We proposed to utilize the scalable peer-to-peer network to perform the content-based image retrieval and mining, i.e, P2P-CBIRM. The decentralized unstructured P2P model with certain overheads, i.e., peer clustering and update procedures, is adopted to compromise with the structured one while still reserving flexible routing control when peers join/leave or network fails. The peer CBIRM engine is designed to utilize multi-instance query with multi-feature types to effectively reduce network traffic while maintaining high retrieval accuracy. It helps to enhance the knowledge discovery and image data mining capability. The proposed P2P-CBIRM system provides the scalable retrieval and mining function that the query scope and retrieval accuracy can be adaptively and progressively controlled. To improve the query efficiency (recall-rate/query-scope), it effectively utilizes both: 1) forwarding query message (forward phase) to reduce the query scope and 2) transmitting retrieval results (backward phase) such that activated peers keep filtering high similarity images on the link-path toward the query peer. Experiments show that the query efficiency of the scalable retrieval approach is better than previous methods, i.e., firework query model and breadth-first search. It provides a scalable knowledge discovery platform for efficient image data mining applications. We also proposed to optimally configure the P2P-CBIRM system such that, under a certain number of online users, it would yield the highest recall rate. Simulations demonstrate that, with the optimal configuration, recall rates can be improved to 2.5 to 3 times larger while the network traffic of each peer is reduced to 30% of the original, under the same number of on-line users.

Three-dimensional (3D) wavelet transforms have been used in video codecs (WVC) to generate quality scalable bitstreams for communications over heterogeneous networks. However, reconstructed pictures of 3D-WVC suffer temporal quality fluctuations due to unequal error distributions in the code/decode process. Theoretical WVC coding models had been proposed to smooth temporal picture quality fluctuations. However, in practical coding processes, signal properties do not comply with assumptions in the ideal WVC model. We propose to bridge this gap between ideal and practical signal properties to improve the temporal quality smoothness of WVC. The relation between distortions of subbands and qualities of reconstructed pictures of one WVC was exploited, from which the TS algorithm was developed. The most important features of the proposed TS-WVC method are: 1) low computation complexity; 2) largely reduce the temporal picture quality variations; and 3) make it feasible for real-time WVC applications

The wavelet-based video codec is capable of providing scalable bitstreams such that quality of service (QoS) among heterogeneous networks can be manipulated in a unified framework. The three-dimensional wavelet video coder 3D WVC had been developed to provide this fully scalable bit stream. However, the quality control or rate allocation was performed on the scale of one GOP such that the reconstructed pictures suffer temporal quality fluctuations. The theoretical coding model provided ratio parameters among subbands to smooth temporal picture quality. However, it needs further adaptation for dealing with practical WVC. We propose to investigate the rate-distortion relation, between the decomposed subbands (ª) and the reconstructed pictures in one GOP (P), under which the rate allocation for constant quality (RACQ) can be operated. Experiments show that the temporal quality fluctuations can be reduced to 10-30 times smaller as compared to 3D-SPIHT and 5-10 times smaller to the ratio parameter approach.

We proposed an image retrieval algorithm that boosts salient common features from heterogeneous query images and performs the similarity measurement. The retrieval process is carried out in intra and inter feature domains: (1) the intra retrieval utilizes a method of multi-instance query with one feature and; (2) the inter retrieval coordinates the individual intra retrieval results to boost salient common features among query samples. To represent feature saliency, the averaged normalized correlation coefficient is computed in the intra domain and is normalized in the inter domain. The most distinguished point of this algorithm is that it coordinates heterogeneous features for multi-instance retrieval. Up to five different image types could be integrated for retrieval. Simulations demonstrate satisfactory subjective retrieval performances.

A universal query mechanism (UQM) for shape-based similarity retrieval in image/video database is proposed. In UQM, we investigate statistically common and salient features among multi-instance query samples to reflect user's definition on shape similarity. For two-dimensional binary shape images, Zernike moments (ZMs) adopted by MPEG-7 are computed as the normal shape descriptors. To get un-biased description by the ZMs, we proposed to locate the minimum bounding circle (MBC) for shape content, which accommodates major shape information in images inside while excludes erroneous noises outside. A fast algorithm to locate the MBC for shapes in image has been developed. The best functionality of the proposed UQM is that, excluding simplicity, new feature sets could be plugged directly into system to improve, while free of degradation in the retrieval performance. Simulations demonstrate that, the UQM-retrieved candidates from thirty thousand images in database really match subjective definitions on shape similarity as the complicated boosting query can do.

Several ways of 2D shape (or contour) description in terms of Fourier and wavelet transform coefficients have been proposed. They provide data compression capability, and some of the descriptors are invariant under scaling, rotation, and choice of the starting point for contour tracing. Several methods for 3D shape description also exist. However, they lack either the simplicity, the generality, or the data compression ability comparable to the 2D methods. We propose a generalized sampling method for efficient description of free-form 3D shape surfaces. The key idea is to warp a 3D spherical coordinate system onto the 3D surface, so that the spatial coordinates of each point on the surface may be represented parametrically as {x(/spl alpha/,/spl beta/), y(/spl alpha/,/spl beta/), z(/spl alpha/,/spl beta/)}, where /spl les//spl alpha//spl les/2/spl pi/ and 0/spl les//spl beta//spl les//spl pi/ with /spl alpha/ and /spl beta/ giving normalized arc lengths on the 3D surface. Fourier, wavelet, or other 2D transforms can then be applied to the three coordinate functions for purposes of data compression, database storage, or indexing. Simulations show that wavelet-based method yields efficient 3D shape compression based on this generalized sampling approach.

We consider optimal coding of video sequences in the sense of employing an optimal selection of quantization scales under buffer and channel constraints. As a result of the delayed-coding framework, multiple rate constraints emerge, leading to the appearance of multiple Lagrange multipliers in a Lagrangian solution approach. We review the applicable Lagrangian optimization theory developed to-date, and present an efficient new solution algorithm.

We consider optimal encoding of video sequences for ATM networks. Two cases are investigated. In one, the video units are coded independently (e.g., motion JPEG), while in the other, the coding quality of a later picture may depend on that of an earlier picture (e.g., H.26x and MPEGx). The aggregate distortion-rate relationship for the latter case exhibits a tree structure, and its solution commands a higher degree of complexity than the former. For independent coding, we develop an algorithm which employs multiple Lagrange multipliers to find the constrained bit allocation. This algorithm is optimal up to a convex-hull approximation of the distortion-rate relations in the case of CBR (constant bit-rate) transmission. It is suboptimal in the case of VBR (variable bit-rate) transmission by the use of a suboptimal transmission rate control mechanism for simplicity. For dependent coding, the Lagrange-multiplier approach becomes rather unwieldy, and a constrained tree search method is used. The solution is optimal for both CBR and VBR transmission if the full constrained tree is searched. Simulation results are presented which confirm the superiority in coding quality of the encoding algorithms. We also compare the coded video quality and other characteristics of VBR and CBR transmission.

In the first part of this paper, we derived a source model describing the relationship between bits, distortion, and quantization step size for transform coders. Based on this source model, a variable frame rate coding algorithm is developed. The basic idea is to select a proper picture frame rate to ensure a minimum picture quality for every frame. Because our source model can predict approximately the number of coded bits when a certain quantization step size is used, we could predict the quality and bits of coded images without going through the entire real-coding process. Therefore, we could skip the right number of picture frames to accomplish the goal of constant image quality. Our proposed variable frame rate coding schemes are simple but quite effective as demonstrated by simulation results. The results of using another variable frame rate scheme, Test Model for H.263 (TMN-5), and the results of using a fixed frame rate coding scheme, Reference Model 8 for H.261 (RM8), are also provided for comparison.

A source model describing the relationship between bits, distortion, and quantization step sizes of a large class of block-transform video coders is proposed. This model is initially derived from the rate-distortion theory and then modified to match the practical coders and real image data. The realistic constraints such as quantizer dead-zone and threshold coefficient selection are included in our formulation. The most attractive feature of this model is its simplicity in its final form. It enables us to predict the bits needed to encode a picture at a given distortion or to predict the quantization step size at a given bit rate. There are two aspects of our contribution: one, we extend the existing results of rate-distortion theory to the practical video coders, and two, the nonideal factors in real signals and systems are identified, and their mathematical expressions are derived from empirical data. One application of this model, as shown in the second part of this paper, is the buffer/quantizer control on a CCITT P/spl times/64 k coder with the advantage that the picture quality is nearly constant over the entire picture sequence.

We consider optimal encoding of video sequences (for ATM networks) under buffer and channel constraints. For independent coding, we develop an algorithm which employs multiple Lagrange multipliers to find the constrained optimal bit allocation, Simulation results are presented. We also compare the coded video quality and other characteristics of variable bit-rate and constant bit-rate transmission.

We study video coding in the ATM environment under the constraints of finite encoder and decoder buffer sizes. ATM allows flexible transmission rates which can be tuned to better match picture property variation over time. We consider the leaky bucket policing mechanism and obtain the rate constraints on encoded video. We then address the optimization of video coding under the constraints. Two cases are investigated. In one the pictures are coded independently (e.g., motion JPEG), while in the other the coding quality of a later picture may depend on that of an earlier picture (e.g., MPEG). The aggregate distortion-rate relationship for the latter case exhibits a tree structure and its solution commands a higher degree of complexity than the former. We derive an efficient solution for distortion-rate relations which display certain monotonic and hierarchical characteristics. These characteristics are assumed to hold for most distortion-rate relations. Realistic nonconvex distortion-rate relations can be handled with a minor modification of the algorithm.

A source model describes the relationship between the bits, distortion, and quantization step sizes of a large class of block-transform video coder is proposed. This model is derived from rate-distortion theory, and verified by real images. It enables us to predict the bits needed to encode a picture for a given distortion or to adjust the quantization scales of a coder for a given bit rate. Based on this derived model, a variable frame rate coding algorithm is developed. It can be used to control the frame rate of a coder to ensure a minimum picture quality of every frame. Simulation results indicate that improved performance is obtained by using this variable frame rate coding scheme when compared to the simple approach of varying the quantization scale linearly proportional to the encoder buffer level.>

A video encoder has the task of producing lowest-distortion coded video subject to some constraints on delay, rate, and buffer conditions. We present a general optimization approach to this problem in a framework of delayed coding and we motivate a certain formulation of the optimization objective. Two forms of distortion measures are considered, namely, the maximum distortion and the total distortion, each defined over a segment of the video to be coded. These distortion measures are chosen for their mathematical tractability and practical importance. A solution (computational algorithm) for each case is described. Subject to some conditions, the solutions may be suboptimal. Simulation results show an improved performance with this approach compared to a simple typical approach which varies the quantization scale linearly with the encoder buffer level.