We propose a new transform design method that targets the generation of compression-optimized transforms for next-generation multimedia applications. The fundamental idea behind transform compression is to exploit regularity within signals such that redundancy is minimized subject to a fidelity cost. Multimedia signals, in particular images and video, are well known to contain a diverse set of localized structures, leading to many different types of regularity and to nonstationary signal statistics. The proposed method designs sparse orthonormal transforms (SOTs) that automatically exploit regularity over different signal structures and provides an adaptation method that determines the best representation over localized regions. Unlike earlier work that is motivated by linear approximation constructs and model-based designs that are limited to specific types of signal regularity, our work uses general nonlinear approximation ideas and a data-driven setup to significantly broaden its reach. We show that our SOT designs provide a safe and principled extension of the Karhunen-Loeve transform (KLT) by reducing to the KLT on Gaussian processes and by automatically exploiting non-Gaussian statistics to significantly improve over the KLT on more general processes. We provide an algebraic optimization framework that generates optimized designs for any desired transform structure (multiresolution, block, lapped, and so on) with significantly better n -term approximation performance. For each structure, we propose a new prototype codec and test over a database of images. Simulation results show consistent increase in compression and approximation performance compared with conventional methods.

In this paper, a new low-complexity true-motion estimation (TME) algorithm is proposed for video processing applications, such as motion-compensated temporal frame interpolation (MCTFI) or motion-compensated frame rate up-conversion (MCFRUC). Regular motion estimation, which is often used in video coding, aims to find the motion vectors (MVs) to reduce the temporal redundancy, whereas TME aims to track the projected object motion as closely as possible. TME is obtained by imposing implicit and/or explicit smoothness constraints on the block-matching algorithm. To produce better quality-interpolated frames, the dense motion field at interpolation time is obtained for both forward and backward MVs; then, bidirectional motion compensation using forward and backward MVs is applied by mixing both elegantly. Finally, the performance of the proposed algorithm for MCTFI is demonstrated against recently proposed methods and smoothness constraint optical flow employed by a professional video production suite. Experimental results show that the quality of the interpolated frames using the proposed method is better when compared with the MCFRUC techniques.

Single sensor digital cameras use color filter arrays to capture a subset of the color data at each pixel coordinate. Demosaicing or color filter array (CFA) interpolation is the process of estimating the missing color samples to reconstruct a full color image. In this paper, we propose a demosaicing method that uses multiscale color gradients to adaptively combine color difference estimates from different directions. The proposed solution does not require any thresholds since it does not make any hard decisions, and it is noniterative. Although most suitable for the Bayer CFA pattern, the method can be extended to other mosaic patterns. To demonstrate this, we describe its application to the Lukac CFA pattern. Experimental results show that it outperforms other available demosaicing methods by a clear margin in terms of CPSNR and S-CIELAB measures for both mosaic patterns.

This paper presents a block-overlap-based validity metric for use as a measure of motion vector (MV) validity and to improve the quality of the motion field. In contrast to other validity metrics in the literature, the proposed metric is not sensitive to image features and does not require the use of neighboring MVs or manual thresholds. Using a hybrid de-interlacer, it is shown that the proposed metric outperforms other block-based validity metrics in the literature. To help regularize the ill-posed nature of motion estimation, the proposed validity metric is also used as a regularizer in an energy minimization framework to determine the optimal MV. Experimental results show that the proposed energy minimization framework outperforms several existing motion estimation methods in the literature in terms of MV and interpolation quality. For interpolation quality, our algorithm outperforms all other block-based methods as well as several complex optical flow methods. In addition, it is one of the fastest implementations at the time of this writing.

In this paper, we study the use of peer-assisted server-based cooperative transmission strategies in IPTV networks for the delivery of on-demand services to end users. The proposed techniques aim to support the resource efficient delivery of on-demand content to end users in a timely manner. Within the proposed framework, a cooperative transmission strategy suggests that users who have access to the requested content cooperatively transmit to the targeted set of users. In doing so, we can minimize the servicing requirements at the server side and improve the scalability performance in the network. We conducted extensive simulations and showed that significant performance improvements can be achieved with the proposed delivery techniques to enable efficient access to an ever growing on-demand content. We also showed the robustness of the proposed techniques in regard to variations observed in network state.

In this paper, we propose a realistic analytical framework to evaluate the performance of concurrent channel change stream delivery techniques. Our analysis utilizes the results from the latest statistical research on user distributions and channel switching activities in IPTV networks to improve the accuracy of performance evaluations. We discuss in detail the proposed framework and examine the critical performance measures based on the derived analysis. Comparisons with the earlier studies suggests improved accuracy in the decision making process that can optimize the resource allocation in the network to effectively support the concurrent delivery techniques.

In IPTV networks, channel change latency represents a major obstacle in achieving broadcast-level quality video delivery. Since the content for channels, other than the one currently viewed, is not readily available at the client side, to receive the content corresponding to these channels, each request needs to go through the network, leading to significant, and oftentimes unacceptable, delays. To minimize this latency, various approaches have been proposed. These approaches typically require the use of dedicated servers near the access network to speed up the delivery of channel change packets. In this paper, we use such server-driven framework to develop a joint unicast- and multicast-based channel change protocol that relies on the use of (i) session-dependent multicast channel change streams to deliver the key-frame content to the zapping clients, and (ii) a dedicated unicast streaming server, which is referred to as the Channel Switch Coordinator (CSC), to deliver the channel change packets corresponding to the non-key-frames. In our framework, CSC server is also responsible for coordinating the received channel change requests and optimally allocating system resources for each received request.

In this paper, we investigate the use of cooperative transmission strategies to support timely and efficient delivery of on-demand content to end users. Within the proposed framework, a cooperative transmission strategy suggests that users who have access to the requested content cooperatively transmit to the targeted user(s) to minimize the servicing requirements at the server side and to improve the scalability performance in the network. Through extensive simulation-based studies, we show that significant performance improvements can be achieved with the proposed framework to enable efficient access to an ever growing on-demand content.

This paper presents a block-overlap-based validity metric in the context of hybrid de-interlacing. The proposed metric uses the degree of motion-compensated block overlap as a measure of motion vector (MV) validity. In contrast to other validity methods in the literature, the proposed method is not sensitive to image features and does not require the use of neighboring MVs or manual thresholds. To perform hybrid de-interlacing, the confidence value for each MV is used to weigh the contribution from two de-interlacing methods. Experimental results show that the de-interlaced images produced with the proposed method are of significant higher quality than those produced by existing validity methods.

This paper presents a misalignment correction method for reducing depth errors that result from camera shift. The proposed method uses a real-time motion estimation approach to correct for alignment errors between stereo cameras. Unlike existing methods in the literature, the natural disparity between stereo views is incorporated into a constrained motion estimation framework. The proposed method is shown to accurately estimate synthetic misalignments due to translation, rotation, scaling, and perspective transformation. In addition, real images taken from a stereo camera rig confirm that the proposed method is capable of significantly reducing misalignments due to camera yaw, pitch, and roll.

This paper presents a block-overlap-based method for handling the ill-posed nature of motion estimation. The proposed method uses the volume of motion-compensated block overlap as an additional term in minimizing the overall energy. By reducing the amount of block overlap, the proposed method results in a significant improvement in the quality of the motion field. Experimental results show that the proposed method outperforms several existing methods in the literature in terms of motion vector (MV) and interpolation quality. In terms of interpolation quality, our algorithm outperforms all other block-based methods as well as several complex optical flow methods. In addition, it is the fastest non-GPU implementation at the time of this writing.

For economic reasons, most digital cameras use color filter arrays instead of beam splitters to capture image data. As a result of this, only one of the required three color samples becomes available at each pixel location and the other two need to be interpolated. This process is called Color Filter Array (CFA) interpolation or demosaicing. Many demosaicing algorithms have been introduced over the years to improve subjective and objective interpolation quality. We propose an orientation-free edge strength filter and apply it to the demosaicing problem. Edge strength filter output is utilized both to improve the initial green channel interpolation and to apply the constant color difference rule adaptively. This simple edge directed method yields visually pleasing results with high CPSNR.

To achieve the desired service quality in IPTV networks, we need effective and resource-efficient error recovery techniques to overcome the problems encountered during the delivery of the IPTV content to the end-users. To enable quick recovery from packet losses, one idea is to utilize an error recovery server (ERServ) that is placed in-between the end-users and the content delivery server. Within this framework, error recovery process is initiated once an end-user sends a repair packet request to the ERServ, and the process is finalized when the repair packet transmitted by the ERServ in response is successfully delivered to the end-user. Even though the use of an ERServ presents an effective solution to improve the perceived quality at the end-users, as the number of users connected to the ERServ increases, we may observe scalability related problems (e.g., dropped requests or increased response times). Consequently, during such overloaded periods, we cannot guarantee the timely delivery of the repair packets, which may lead to significant performance degradations at the end-users. To prevent this from happening, we propose to utilize peer-based error recovery in addition to server-based recovery, for which the main objective is to prevent the ERServ from entering a non-responsive state. We achieve this objective by redirecting the error recovery load towards the end-users during times when the ERServ reaches a critical server-load level.

In this paper, we propose a resource-efficient cooperative multiple-input single-output (MISO) technique to reliably deliver multicast IPTV content in WiMAX networks. To support the strict quality of service requirements of the IPTV service within a cooperative framework, we utilize a two-phase based transmission strategy, which allows the session users to reliably receive the multicast data either from the base station during the multicast phase, or from other peers during the cooperation phase. Within this framework, our objective is to maximize the session throughput, while keeping delivery latency and energy usage within acceptable limits. To achieve this objective, an adaptive rate selection algorithm is proposed to maximize the session throughput for a given set of users. We simulated the proposed reliable content delivery framework and compared its throughput, latency, overhead, and energy usage performances, to that of MIMO-based and non-cooperative (or direct) transmission techniques. When compared to the considered MIMO and direct transmission techniques, we found that the proposed approach presents a desirable balance for the given performance metrics, especially between throughput-efficiency and energy-efficiency.

In this paper we introduce a distributed channel change framework that utilizes a dedicated server and session peers to support the fast channel change process. Support peers represent the prior choice to deliver the channel change packets to the zapping users. The dedicated server, which we refer to as the Channel Switch Coordinator (CSC), is essentially responsible for the coordination of the channel change requests. To minimize the average channel change delay for all the users connected to the same CSC server, CSC server is also responsible for the delivery of channel change and proactive repair packets, whenever peer-based delivery fails to achieve the desired performance results in terms of channel change latency. Because of the increased scalability offered by the use of support peers during the channel change process, the proposed approach allows the use of longer group-of-picture (GOP) sizes for the transmission of the IPTV content.

To successfully deliver the IPTV content to end-users, robust and resource-efficient error recovery techniques are required that can overcome the problems encountered during the end-to-end delivery process. To achieve this objective by also enabling quick error recovery, one preferred approach is to install a dedicated server, referred to as the error recovery server, between the head-end office and the end-users, to handle the repair packet requests coming from the end-users. However, as the number of users connected to the error recovery server increases, the probability of the server being overloaded by too many requests also increases, leading to scalability problems and increased response times. As a result, we cannot anymore guarantee the timely delivery of the repair packets, which would lead to unacceptable performance degradations at the end users. To prevent this from happening, we can minimize the recovery load at the error recovery server by using a peer-based error recovery approach in conjunction with a server-based error recovery approach. In this paper, our objective is to develop an effective joint server-based and peer-based error recovery solution that minimizes the probability of the server entering a non-responsive state by pushing the error recovery load towards the end-users, whenever needed and possible. The information on the perceived network characteristics is used to make resource-optimal decisions to find the recovery-peers. By taking into account the impact of latency on the peer-selection policy, we improve the fairness in the network and avoid over-utilizing the resources at the error recovery server. Through simulations, we show the significant performance improvements that can be achieved with the proposed joint error recovery framework when compared to a server-based error recovery framework.

In IPTV networks, quality of experience (QoE) is typically expressed in terms of perceived packet loss rate and latency performances. As the IPTV content is delivered over diverse transmission channels, to meet the QoE objectives of all the users connected to the same IPTV network, we need to utilize resource efficient error control techniques. Recent research on IPTV error control has mostly focused on the use of a dedicated error recovery server (ERServ) to support reactive and unicast-based error control services. It is known that a unicast-based strategy is typically preferred when the accumulative packet loss rate is small and/or the correlation among the users' loss processes is negligible. However, if the users start to observe correlated packet losses, then unicast-based strategies may lead to significant performance degradations. For such scenarios, to improve the effectiveness of the error recovery process, we need to also utilize multicast-based repair strategies. In this paper, our objectives are (i) to develop spatial correlation models that are applicable to IPTV networks, (ii) to design an error recovery framework that is capable of exploiting the spatial correlation characteristics of the network, and, finally, using the proposed packet loss models and recovery protocols (iii) to analyze the performance tradeoffs associated with the use of unicast- and multicast-based error recovery strategies for correlated packet loss scenarios in IPTV networks.

Most of the current digital cameras feature a single sensor de sign which limits the number of channels recorded at each pixel location to one. However, a color image is represented with three channels for each pixel. Color Filter Array (CFA) interpolation is the process of generating a full three channel color image from a single channel mosaicked input. We pro pose a simple edge strength filter to interpolate the missing color values adaptively. While the filter is readily applicable to the Bayer mosaic pattern, we argue that the same idea could be extended to other mosaic patterns and describe its application to the Lukac mosaic pattern. The proposed solution outperforms other available algorithms for the Lukac pattern in terms of both objective and subjective comparison.

Single sensor digital cameras capture one color value for every pixel location. The remaining two color channel values need to be estimated to obtain a complete color image. This process is called demosaicing or Color Filter Array (CFA) interpolation. We propose a directional approach to the CFA interpolation problem that makes use of multiscale color gradients. The relationship between color gradients on different scales is used to generate signals in vertical and horizontal directions. We determine how much each direction should contribute to the green channel interpolation based on these signals. The proposed method is easy to implement since it is noniterative and threshold free. Experiments on test images show that it offers superior objective and subjective interpolation quality.

In IPTV networks, channel change latency is a major concern for achieving the desired quality of experience at the subscribers. Since IPTV connections are established over limited capacity links, the number of channels available to each subscriber's use, at any given point in time, is strictly limited. As a direct consequence of this limitation, channel change requests need to go through the IP infrastructure, causing significant, and oftentimes unacceptable, channel switching delays. To minimize this latency, various approaches have been proposed. Among these approaches, server-based approaches are shown to achieve the best latency performance. However, since server-based approaches typically utilize unicast-based delivery techniques, they are susceptible to introducing significant overhead at the designated servers, leading to scalability related problems. In this paper, we address these problems by proposing a semi-distributed, i.e., user-assisted server-based, channel change framework. The proposed approach integrates the capabilities of a dedicated channel change server with that of IPTV subscribers to create a resource-efficient and scalable channel change framework. Through extensive simulations, we showed that the proposed framework can achieve sufficiently low channel change latency values, while significantly improving the servicing capacity of the network when compared to server-based-only channel change policies.

Prediction of the 3D structure greatly benefits from the information related to secondary structure, solvent accessibility, and nonlocal contacts that stabilize a protein's structure. We address the problem of \beta-sheet prediction defined as the prediction of \beta--strand pairings, interaction types (parallel or antiparallel), and \beta-residue interactions (or contact maps). We introduce a Bayesian approach for proteins with six or less \beta-strands in which we model the conformational features in a probabilistic framework by combining the amino acid pairing potentials with a priori knowledge of \beta-strand organizations. To select the optimum \beta-sheet architecture, we significantly reduce the search space by heuristics that enforce the amino acid pairs with strong interaction potentials. In addition, we find the optimum pairwise alignment between \beta-strands using dynamic programming in which we allow any number of gaps in an alignment to model \beta-bulges more effectively. For proteins with more than six \beta-strands, we first compute \beta-strand pairings using the BetaPro method. Then, we compute gapped alignments of the paired \beta-strands and choose the interaction types and \beta--residue pairings with maximum alignment scores. We performed a 10-fold cross-validation experiment on the BetaSheet916 set and obtained significant improvements in the prediction accuracy.

The prospect of delivering high-quality broadcast content over the Internet relies on the promise of presenting end-users with the minimum quality of viewing experience. Quality of Experience (QoE) represents the perceived performance at the end-users in terms of resource utilization, error control, and delivery latency. Among the considered performance measures, error control typically represents the biggest challenge to achieve the minimum required QoE at the end-users. Furthermore, as the end-users that are subscribed to the same IPTV multicast session observe varying, and oftentimes independent, packet loss characteristics, it becomes a major challenge to provide all the end-users with the desired QoE under limited network resources available for the IPTV service. Consequently, system scalability and network stability become the major issues that can significantly limit the accessibility of the IPTV service. In this paper, we address the error recovery problem for the IPTV networks by proposing a performance evaluation framework which can be used to develop resource efficient error control techniques.

Color Filter Array (CFA) interpolation is an integral part of image processing pipeline for single sensor digital cameras. Many CFA algorithms have been proposed over the years to improve resulting image quality. One such algorithm is the highly successful Directional Linear Minimum Mean-Square Error Estimation (DLMMSE) method. We make several observations on this algorithm and propose a new method to address those points. The proposed method yields visually pleasing results and outperforms all CFA interpolation algorithms that are included in a recent survey paper in terms of PSNR.

This paper describes the construction of a new multiresolutional decomposition with applications to image compression. The proposed method designs sparsity-distortion-optimized orthonormal transforms applied in wavelet domain to arrive at a multiresolutional representation which we term the Sparse Multiresolutional Transform (SMT). Our optimization operates over sub-bands of given orientation and exploits the inter-scale and intra-scale dependencies of wavelet co-efficients over image singularities. The resulting SMT is substantially sparser than the wavelet transform and leads to compaction that can be exploited by well-known coefficient coders. Our construction deviates from the literature, which mainly focuses on model-based methods, by offering a data-driven optimization of wavelet representations. Simulation experiments show that the proposed method consistently offers better performance compared to the original wavelet-representation and can reach up to 1dB improvements within state-of-the-art coefficient coders.

In this letter, a new fast motion estimation (FME) algorithm capable of producing sub-sample motion vectors at low computational-complexity is proposed. Unlike existing FME algorithms, the proposed algorithm considers the low complexity sub-sample accuracy in designing the search pattern for FME. The proposed FME algorithm is designed in such a way that the block distortion measure is modeled as a parametric surface in the vicinity of the integer-sample motion vector; this modeling enables low computational-complexity sub-sample motion estimation (ME) without pixel interpolation. Experimental results on video test sequences show that the proposed FME algorithm reduces computational complexity of integer and sub-sample ME considerably compared with traditional methods at the cost of negligible performance degradation.

Sparse orthonormal transforms (SOT) has recently been proposed as a data compression method that can achieve sparser representations in transform domain. Given initial conditions, the optimization method utilized to generate the dictionary of SOT also achieves the optimal orthonormal transform for hard thresholding. In the context of translation-invariant denoising, one can use this dictionary to represent the local neighborhood around each pixel and obtain denoised estimates for that neighborhood with hard thresholding. Building upon this approach, here we propose a method to fuse the overlapping denoised estimates via weighted linear averaging to compute final denoised signal.

Spatial resolution of digital images are limited due to optical/sensor blurring and sensor site density. In single-chip digital cameras, the resolution is further degraded because such devices use a color filter array to capture only one spectral component at a pixel location. The process of estimating the missing two color values at each pixel location is known as demosaicking. Demosaicking methods usually exploit the correlation among color channels. When there are multiple images, it is possible not only to have better estimates of the missing color values but also to improve the spatial resolution further (using super-resolution reconstruction). In this paper, we propose a multi-frame spatial resolution enhancement algorithm based on the projections onto convex sets technique.

A general framework based on histogram equalization for image contrast enhancement is presented. In this framework, contrast enhancement is posed as an optimization problem that minimizes a cost function. Histogram equalization is an effective technique for contrast enhancement. However, a conventional histogram equalization (HE) usually results in excessive contrast enhancement, which in turn gives the processed image an unnatural look and creates visual artifacts. By introducing specifically designed penalty terms, the level of contrast enhancement can be adjusted; noise robustness, white/black stretching and mean-brightness preservation may easily be incorporated into the optimization. Analytic solutions for some of the important criteria are presented. Finally, a low-complexity algorithm for contrast enhancement is presented, and its performance is demonstrated against a recently proposed method.

In [1] we proposed a hyperspectral imaging model that represents spectral observations at different wavelengths as weighted linear combinations of a small number of basis images obtained through principal component analysis (PCA). Based on this imaging model we formulated a multiple observation resolution enhancement method for hyperspectral imagery. In this work we focus on material specific resolution enhancement. We start with pointing out a shortcoming of the PCA based imaging model. We then introduce modifications to integrate linear spectral mixing into the imaging model. Based on the updated model we introduce and implement material specific multiple observation resolution enhancement for hyperspectral imagery. We test the proposed method on AVIRIS data, and present numeric and visual results.

Due to memory bandwidth limitations and computational complexity considerations in hardware implementations, block matching combined with lscr1error norm and translational motion model is preferred in motion-estimation algorithms. Performance of this scheme is degraded by noise, compression artifacts, rotation, repeating structures, motion boundaries, zooming, and brightness changes. In this work, we present a Bayesian approach to incorporate prior information into block matching. Hypothesis testing is utilized to choose the most applicable prior motion vector and to compute a prior motion-vector distribution and its precision. Prior distribution is then updated with motion-vector likelihood derived from pixel data to obtain the posterior distribution, which is maximized via a search on the feasible motion-vector space.

This paper presents a new normalcy model of a scene for change detection using images taken from multiple views and varying illumination conditions. Each coregistered pixel site is statistically modeled by a probability distribution conditioned on a set of pixels in a non-local neighborhood that are less likely to be affected by a change that happens at the pixel of interest. These ldquonon-compact neighborsrdquo are located using information theoretic approaches. The associated change detection algorithm is called non-compact Markovian Likelihood (NorMaL), which predicts normalcy of a scene based on non-compact neighborhoods using non-parametric conditional density estimation.

H.264 video coding standard supports several inter- prediction coding modes that use macroblock (MB) partitions with variable block sizes. Rate-distortion (R-D) optimal selection of both the motion vectors (MVs) and the coding mode of each MB is essential for an H.264 encoder to achieve superior coding efficiency. Unfortunately, searching for optimal MVs of each possible subblock incurs a heavy computational cost. In this paper, in order to reduce the computational burden of integer-pel motion estimation (ME) without sacrificing from the coding performance, we propose a R-D and complexity joint optimization framework. Within this framework, we develop a simple method that determines for each MB which partitions are likely to be optimal. MV search is carried out for only the selected partitions, thus reducing the complexity of the ME step. The mode selection criteria is based on a measure of spatiotemporal activity within the MB. The procedure minimizes the coding loss at a given level of computational complexity either for the full video sequence or for each single frame. For the latter case, the algorithm provides a tight upper bound on the worst case complexity/execution time of the ME module. Simulation results show that the algorithm speeds up integer-pel ME by a factor of up to 40 with less than 0.2 dB loss in coding efficiency.

In the above-named work (ibid., vol. 49, no. 10, pp. 2669-2681, Oct. 2003) the authors developed performance bounds for multiple antenna systems that use receive antenna selection and proved results regarding the diversity and coding gains in such systems. In reference to above correspondence, the probability density function given by eq. (30) should not include the scaling factor of 1/L. This is shown via a short derivation. As a result, the factor 1/L should be omitted from the pairwise error probability expressions, and (32), (34), (35), and (38)-(41) are corrected herein. The main results regarding the diversity gain of the system are not effected by the absence of this scaling factor, while the ones on the coding gain should be modified accordingly.

In this study, we propose bit-rate adaptation techniques for compressed high-definition video transmission over wireless home networks. Application-layer methods are developed to minimize quality impairments caused by wireless channel degradations. A transrater located at a home video server adjusts the transmission rate of the video stream based on bandwidth measurements. We consider delay constraints of multiple video frames across various time scales, and dynamically select an appropriate time scale for bit-rate adjustments. This novel feature prevents playout buffer underflows and avoids unnecessary rate reductions. Furthermore, distortion due to transrating is minimized by coding type-dependent rate allocation among the frames to be transmitted in the projected time scale. Using a realistic simulation setup, we demonstrated that the proposed technique significantly reduces video glitches. Up to 3.12 dB average PSNR improvement is achieved, compared to a non-delay constrained rate adaptation method, which also utilizes bandwidth measurements. Our approach maintains compliance with existing wireless local area (WLAN) devices and standards, as well as consumer video standards and applications.

Protein structure prediction aims to determine the three-dimensional structure of proteins form their amino acid sequences. When a protein does not have similarity (homology) to any known fold, threading or fold recognition methods are used to predict structure. Fold recognition methods frequently employ secondary structure, solvent accessibility, and evolutionary information to enhance the accuracy and the quality of the predictions. In this paper, we present a residue based alignment method as an alternative to the state-of-the-art SSEA method, originally introduced by Przytycka et al., and further modified by McGuffin et al. We introduce a residue-based score function, which can incorporate amino acid similarity matrices such as BLOSUM into secondary structure similarity scoring and compute joint alignments. We show that the power of the SSEA method comes from the length normalization instead of the element alignment technique and similar performance can be achieved using residue-based alignments of secondary structures by optimizing gap costs. In simulations with the two benchmark datasets, our method performs slightly better than the SSEA in terms of the fold recognition accuracy. When the secondary structure similarity matrix is combined with the amino acid based BLOSUM30 matrix, the accuracy of our method improves further (4% for the McGuffin set and 10% for the Ding and Dubchak set). The availability of aligning the amino acid and secondary structure sequences in a joint manner offers a better starting point for more elaborate techniques that employ profile-profile alignments and machine learning methods.

Developing error-control and error-resiliency methods for transmitting delay-sensitive media content over the best-effort networks poses several challenges. Due to the lack of QoS guarantees in the conventional Internet as well as in emerging wireless networks, these methods must continuously monitor the characteristics of the underlying network and try to infer the incipient network conditions so that they can take the necessary actions on time. This is utmost important for enhancing the end-user quality, particularly in low-delay multimedia applications. In this study, we tackle this problem from an error-control method perspective and develop an innovative framework that optimizes the retransmission decisions based on the urgency and importance of the media packets.

Spatial resolution of digital images are limited due to optical/sensor blurring and sensor site density. In single-chip digital cameras, the resolution is further degraded because such devices use a color filter array to capture only one spectral component at a pixel location. The process of estimating the missing two color values at each pixel location is known as demosaicking. Demosaicking methods usually exploit the correlation among color channels. When there are multiple images, it is possible not only to have better estimates of the missing color values but also to improve the spatial resolution further (using super-resolution reconstruction). Previously, we have proposed a demosaicking algorithm based on the projection onto convex sets (POCS) technique. In this paper, we improve the results of that algorithm adding a new constraint set based on the spatio-intensity neighborhood. We extend the algorithm to image sequences for multi-frame demosaicking and super resolution.

For many digital image/video processing applications increasing the spatial resolution through modifications in the imaging system is infeasible. Hence post-processing algorithms designed to enhance resolution of the acquired image data prove beneficial. In this paper, we analyze recent work on classification based resolution enhancement and discuss its applicability to low-complexity display systems. In the light of our observations we point out certain short-comings of resolution synthesis and propose a modified training scheme to improve the performance under certain conditions.

Edges that are visible in color images may not be detected in the corresponding grayscale image. This is due to the neighboring objects having different hues but the same intensities. Hence, a color edge preserving grayscale conversion algorithm is proposed that helps detect color edges using only the luminance component. The algorithm calculates an approximation to the first principal component to form a new set of luminance coefficients instead of using the conventional luminance coefficients. This method can be directly applied to all existing grayscale edge detectors for color edge detection. Processing only one channel instead of three channels results in lower computational complexity compared to other color edge detectors. Experimental results on test images show similar edge detection capabilities to typical color edge detectors at reduced complexity levels.

Time-constrained error recovery is an integral component of reliable low-delay video applications. Regardless of the error-control method adopted by the application, unacknowledged or missing packets must be quickly identified as lost or delayed, so that necessary actions can be taken by the server/client on time. Historically, this problem has been referred to as retransmission timeout (RTO) estimation. Earlier studies show that existing RTO estimators suffer from either long loss detection times or a large number of spurious timeouts. The goal of this study is to address these problems by developing an RTO estimation method specifically tailored for low-delay video applications. In the media-unaware mode, this method exploits the temporal dependence in packet delay to optimally manage the tradeoff between the amount of overwaiting and redundant retransmission rate. As opposed to existing methods, our approach is completely adaptive to the source video characteristics and time-varying network conditions, and does not use any preset parameters. In the media-aware mode, on the other hand, the timeout estimates are jointly optimized based on the importance and urgency of the video packets such that the rendering quality is maximized under the given rate constraints. With a comprehensive set of simulation and experimental results, we show that both the media-unaware and media-aware RTO estimators detect lost packets faster and more accurately than their rivals. Furthermore, our results also substantiate the fact that the media-aware RTO estimator outperforms all other RTO estimators in terms of video quality

Transmission of time-critical video traffic over the networks with large end-to-end delays poses two main difficulties: slow adaptation to changing network conditions and inability of timely recovery from lost packets. Previously, we proposed a proxy-based solution that enabled us to cope with these problems. Our Internet experiments with a single proxy system showed that the QoS delivered by interactive video applications could be greatly improved by the Intermediate-Proxy approach. In multi-proxy systems, however, the availability of a large number of proxies throughout the Internet will bring the proxy selection problem to the fore. This is an important problem since the benefits of the intermediate-proxy approach can be best exploited with a proper proxy selection. In this study, our goal is to model the dynamics involved in the networks with proxies, and investigate mathematical and practical proxy selection methods.

We present a locally adaptive non-linear (YENI) filter to obtain the unsharp mask of an image. The unsharp mask obtained by the YENI filter preserves the edges in the image while filtering out the local details, which correspond to mid-range frequencies in the spectrum. The enhanced image using this unsharp mask effectively prevents over/under (o/u) shooting artifacts often observed with other unsharp masking techniques. The enhanced frequency range also spans lower frequencies compared to the techniques that are based on Laplacian filter variants. This improves the visual quality of the image, as measured subjectively and objectively in the real-video experiments. Furthermore, since the YENI filter reduces to an IIR filter at each pixel location, it has a low computational complexity.

H.264 video coding standard offers several coding modes including inter-prediction modes that use macroblock partitions with variable block sizes, Choosing a rate-distortion optimal mode among these possibilities contributes significantly to the superior coding efficiency of the H.264 encoder. Unfortunately, searching for optimal motion vectors of each possible subblock incurs a heavy computational cost, In this paper, in order to reduce the complexity of integer-pel motion estimation, we propose a rate-distortion and complexity-joint optimization method that selects for each MB a subset of partitions to evaluate during motion estimation. This selection is based on simple, measures of spatio-temporal activity within the MB. The procedure is optimized to minimize mode estimation error at a certain level of computational complexity. Simulation results show that the algorithm speeds up the motion estimation module by a factor of up to 20 with little loss in coding efficiency.

The coding efficiency of the H.264/AVC standard enables the transmission of high quality video over bandwidth limited networks. Due to the use of multiple macroblock (MB) partitions, the motion estimation module has extremely high complexity that makes it unpractical for most real-time applications on resource-limited platforms such as hand held devices. In this paper we propose a novel algorithm that significantly reduces the encoding complexity while maintaining high rate distortion performance. The proposed method reduces the motion estimation (ME) computational complexity by accurately predicting the optimal MB partitions and restricting the number of candidate modes based on a-priori probabilities computed from spatio-temporal information. The experimental results show that the speed up of UmHexagonS (one of the most efficient ME algorithms) can be doubled while maintaining the coding efficiency of full search.

This paper studies linear prediction algorithms for packet-delay modeling. A detailed examination of the delay traces collected from video streams encoded at different bitrates, suggests that autoregressive (AR) models can exploit the correlation among the delay samples and produce the best estimates in terms of the mean-squared error criterion. Simulation results show that AR prediction can reduce the average prediction-error power significantly as compared to the exponentially-weighted moving average prediction as well as the recursive weighted median filtering. This is a promising result since many layers in the multimedia communication protocol stack, e.g., rate control, error control and network adaptation, can greatly benefit from accurate packet-delay prediction.

This paper considers the problem of multiple video streaming over wireless local area networks. In particular, we propose video streaming techniques to improve the quality of video streams over the link from an access point to multiple wireless clients. The proposed video streaming techniques decide on clients to be served, packets to be (re)transmitted, and forward error correction (FEC) rates by considering video, channel, and network properties in a rate-distortion optimized manner. Multi-client rate distortion optimized video streaming is the first technique operated by an access point to jointly optimize client, packet, and FEC rate selection for video streaming. Second, we propose multi-client sub-optimal video streaming technique that optimizes packet and FEC rate, but not client. The client is selected by using the results of optimization to reduce the computational complexity. The simulation results show that the proposed video streaming techniques achieve signal-to-noise ratio (SNR) improvement up-to 5 dB as compared with time-division multiple access (TDMA) and packet adaptive TDMA.

An unsharp masking type local contrast enhancement method that uses spatially adaptive 2-dimensional (2-D) recursive filters is presented. The proposed 2-D recursive filters are a family of exponential smoothing filters. Three different forms are proposed: separable extension of the 1-dimensional (1-D) case, non-separable filter that uses 4-neighbors, and non-separable filter that uses 8-neighbors. The isotropy of the transfer function improves when moved from the first to the last; hence, better visual quality is obtained in the real video experiments. The computational complexity of the proposed recursive filters, unlike non-recursive filters, is independent of the desired cutoff frequency. This independence enables different levels of smoothing simply by spatially adapting the recursion coefficients without any additional computation

Time-constrained error recovery is an integral component of reliable low-delay video applications. Regardless of the error-control method adopted by the application, unacknowledged or missing packets must be quickly identified as lost or delayed, so that necessary timely actions can be taken by the server/client. Historically, this problem has been referred to as the retransmission timeout (RTO) estimation. Earlier studies show that existing RTO estimators suffer from either long loss detection times or a large number of pre-mature timeouts. The goal of this study is to address these problems by developing an adaptive RTO estimator for high-bitrate low-delay video applications. By exploiting the temporal dependence between consecutive delay samples, we propose an adaptive linear delay predictor. This way, our RTO estimator configures itself based on the video characteristics and varying network conditions. Our approach also features a controller that optimally manages the trade-off between the amount of overwaiting and redundant retransmission rate. The skeleton implementation shows that the proposed RTO estimator discriminates lost packets from excessively-delayed packets faster and more accurately than its rivals, which consequently enables the applications to recover more packets under stringent delay requirements.

In a cooperative network, by combining multiple copies of the same packet that are received either simultaneously (i.e., same time-slot) or non-simultaneously (i.e., different time-slot), the aggregate power level of the cooperatively transmitting set, required to achieve reliable reception at the receiver node, can be considerably lowered. This approach is especially advantageous for networks, where the number of nodes in the network, the network density, or the potential for wireless broadcasting advantage is high. In such networks, required transmit energy can be considerably reduced by routing packets over cooperative paths. Since the success of cooperation strictly depends on the selection of the appropriate power levels for each cooperating node, cooperative routing becomes difficult in networks where the nodes are mobile. In such networks, we need to select paths that can maximize the energy savings averaged over the connection time rather than the ones that only maximize savings at the current time. For this reason, we need to design routing protocols that takes into account the mobility characteristics within the network. In this paper, we propose a cooperative routing protocol that can improve the reliability level in the network against mobility variations.

BACKGROUND: The accuracy of protein secondary structure prediction has been improving steadily towards the 88% estimated theoretical limit. There are two types of prediction algorithms: Single-sequence prediction algorithms imply that information about other (homologous) proteins is not available, while algorithms of the second type imply that information about homologous proteins is available, and use it intensively. The single-sequence algorithms could make an important contribution to studies of proteins with no detected homologs, however the accuracy of protein secondary structure prediction from a single-sequence is not as high as when the additional evolutionary information is present. RESULTS: In this paper, we further refine and extend the hidden semi-Markov model (HSMM) initially considered in the BSPSS algorithm. We introduce an improved residue dependency model by considering the patterns of statistically significant amino acid correlation at structural segment borders. We also derive models that specialize on different sections of the dependency structure and incorporate them into HSMM. In addition, we implement an iterative training method to refine estimates of HSMM parameters. The three-state-per-residue accuracy and other accuracy measures of the new method, IPSSP, are shown to be comparable or better than ones for BSPSS as well as for PSIPRED, tested under the single-sequence condition. CONCLUSIONS: We have shown that new dependency models and training methods bring further improvements to single-sequence protein secondary structure prediction. The results are obtained under cross-validation conditions using a dataset with no pair of sequences having significant sequence similarity. As new sequences are added to the database it is possible to augment the dependency structure and obtain even higher accuracy. Current and future advances should contribute to the improvement of function prediction for orphan proteins inscrutable to current similarity search methods.

We propose a joint source-channel coding scheme for wireless communication systems with multiple transmit and receive antennas. The source coder is realized by a multiple description encoder that generates multiple bit streams. Each description is then separately turbo coded and transmitted using multiple antennas. For the receiver, we describe a suitable iterative joint source-channel decoding technique that exploits the correlations between the descriptions. We present several examples that illustrate the performance of the proposed system, and compare it with other approaches.

We present a statistical method that uses prediction modeling to decrease the temporally redundant data transmitted back to the sink. The major novelties are fourfold: First, a prediction model is fit to the sensor data. Second, prediction error is utilized to adaptively update the model parameters using hypothesis testing. Third, a data transformation is proposed to bring the sensor sample series closer to weak stationarity. Finally, an efficient implementation is presented. We show that our proposed preDiction eRror bASed hypoThesis testInG (DRASTIG) method achieves low energy dissipation while keeping the prediction errors at user-defined tolerable magnitudes based on real data experiments.

In a previous paper by Bahceci et al., antenna selection ' for multiple-antenna transmission systems under the assumption that the subchannels between antenna pairs fade independently was studied. In this paper, the performance of such systems when the subchannels experience correlated fading is considered. It is assumed that the channel-state information (CSI) is available only at the receiver, the antenna selection is performed only at the receiver, and the selection is based on the instantaneous received signal power. The effects of channel correlations on the diversity and coding gain when the receiver system is a subset of the antennas are quantified. Theoretical results indicate that the correlations in the channel do not degrade the diversity order, provided that the channel is full rank. However, it does result in some performance loss in the coding gain.

Cooperative diversity techniques exploit the spatial characteristics of the network to create transmit-diversity, in which the same information can be forwarded through multiple paths towards a single destination or a set of destination nodes. In this paper, we study the integration of cooperative diversity into wireless routing protocols by developing distributed cooperative MAC (C-MAC) and routing protocols. The proposed protocols employ efficient relay selection-coordination and power allocation techniques to maximize the cooperation benefits in the network. Simulation results show that the energy-saving performance of the minimum-energy routing protocols can be significantly improved when they are implemented together with the proposed C-MAC protocol (%50). We also show that the performance of the C-MAC protocol can be further enhanced when the initial path is selected using the cooperation characteristics of the network (%11 more energy-savings compared to the previous case, i.e., C-MAC with minimum-energy routing)

Parallel distributed detection for wireless sensor networks is studied in this paper. The network consists of a set of local sensors and a fusion center. Each local sensor makes a binary (single-bit) or M-ary (multi-bit) decision and passes it to the fusion center where a final decision is made. The links between the local sensors and the fusion center are subject to fading and additive noise resulting in corruption of the transmitted decisions. We analyze the performance of the decision fusion based on likelihood ratio tests and derive false alarm and detection probabilities. Based on the theoretical probability expressions, we design optimal decision rules for the local sensors and the fusion center. Finally, we illustrate the performance of the parallel fusion by numerical examples

The adoption of multiple macroblock partitions with variable block sizes is one of the main reasons behind the superior coding efficiency of H.264 video coding standard. Unfortunately, in the motion estimation phase, repeating sum of absolute difference (SAD) calculations for every possible block size incurs a heavy computational cost for the encoder. In this paper, in order to reduce the encoder complexity, we propose a hierarchical block matching based motion estimation algorithm that uses a common set of SAD computations for motion estimation of different block sizes. Based on the hierarchical prediction and the median motion vector predictor of H.264, the algorithm defines a limited set of candidate vectors; and the optimal motion vectors for all partitions are chosen from this common set. Simulation results show that hierarchical estimation with SAD reuse reduces the total computations by a factor of 17.6 with slight loss in coding efficiency.

Secondary structure prediction is an important step in determining the structure and function of proteins. A fundamental assumption of current Bayesian secondary structure prediction methods is the conditional independence of residues which occur in distinct segments. This assumption enables the exact calculation of posterior probabilities by using pre-determined probabilistic models. However, this assumption is clearly violated in the case of protein sequences due to the existence of structural motifs which rely on sequentially distant segments interacting in three-dimensional space, including /spl beta/-sheets. It has been suggested that the inability to capture such nonlocal interactions may be the main reason for the low accuracy typically achieved in /spl beta/-strand prediction (Schmidler, S.C. et al., 2000; Frishman, D. and Argos, P., 1996). Furthermore, current Bayesian segmentations are based on maximum a posteriori or marginal posterior mode searches, which return a single segmentation that is optimal in some sense. We introduce a new secondary structure prediction method based on a modified version of the well-known stack decoder. The proposed method is an N-best search algorithm, which enables us to use the returned multiple segmentations to improve over a single segmentation. Also, due to the way the segmentations are constructed, it is possible to exploit the non-local interactions between /spl beta/-strands in a sub-optimal way with the ultimate goal of increasing the overall prediction accuracy.

Based on the observation that a Cauchy density is more accurate in estimating the distribution of the AC coefficients than the traditional Laplacian density for H.264 video coders, a rate model with improved accuracy is derived. Using the new rate model, a frame bit-allocation algorithm for the rate control of an H.264 video coder is developed. Extensive analysis with carefully selected anchor video sequences demonstrates a 0.43 dB average PSNR improvement over the JM 8.4 rate control algorithm, and a 0.27 dB average PSNR improvement over the TM5-based bit-allocation algorithm that has recently been proposed for H.264 by S. Ma et al. (see Proc. 2003 Int. Conf. Image Process., ICIP '03, vol.3, p.793-6, 2003). The analysis also demonstrates 20% and 60% reductions in PSNR variation among the encoded pictures when compared to the JM 8.4 rate control algorithm and the TM5-based bit-allocation algorithm, respectively.

In this paper, we investigate rate distortion optimized streaming of H.264 coded video sequences over time-varying wireless channels. Our main objective is to minimize average end-to-end distortion to satisfy a certain quality of service (QoS) by considering the media and channel characteristics such as packet importance, packet dependencies, decoding deadlines, channel state information and channel capacity. In particular, we propose a sender-driven optimized joint ARQ-FEC scheme that decides on packet transmissions, re-transmissions as well as the FEC rate used in each transmission. The optimized joint ARQ-FEC scheme along with packet scheduling aims to minimize a weighted sum of distortion and total transmission cost under capacity constraints. The other objective in this paper is to satisfy the QoS of the optimized joint ARQ-FEC scheme without minimizing end-to-end distortion. We improved two approximate algorithms that use the media and channel characteristics to transmit the best selected and FEC coded packet. The real-time simulation results with H.264 sequences demonstrate the efficacy of the all proposed algorithms over the classical error recovery scheme uses ARQ and FEC.

Wireless mesh systems offer several advantages for emerging high-bandwidth networks because of their cooperative routing capabilities. In this study, we consider the potential benefits of using wireless meshes within residential networks for applications that require high bandwidth and low latency. In particular, we consider high-bitrate video transmission inside a mesh-networked home. To quantify our findings, we present experimental results obtained from a high-resolution video streaming application. Our experiments involve single-hop, multi-hop, single-path and multi-path transmission methods, and two types of video coding techniques, namely single description coding and multiple description coding. In the light of our results, we discuss the pros and cons of each streaming method, and motivate a set of research problems.

In retransmission-based error-control methods, the most fundamental yet the paramount problem is to determine how long the sender (or the receiver) should wait before deciding that an unacknowledged (or a missing) packet is lost. This waiting time is generally referred to as retransmission timeout (RTO). An accurate RTO estimation has two main advantages: First, the lost packets can be identified earlier, and hence, can be recovered faster. Second, redundant retransmissions can be avoided, which subsequently not only saves the network resources, but also helps existing network congestion alleviate sooner. Although it is statistically possible to prevent any unnecessary retransmission at the expense of long error-recovery times, such an approach can only be justified for data applications; it is not well-suited for delay-sensitive applications, for which the agility in recovering the lost packets is as important. With this motivation, we recently introduced an RTO estimation algorithm for delay-sensitive applications (A. C. Begen et al., 2004). Provided that the packets are transmitted at equal intervals, this technique successfully estimates the arrival times based on the interarrival-time observations. In this study, we relax the requirement of equal transmission intervals and generalize our technique to handle bursty video applications

Serial distributed detection for wireless sensor networks is studied in this paper. Unlike the traditional serial distributed detection schemes where it is assumed that the sensor decision at one stage is known exactly to the subsequent sensor node, we assume that the links between the consecutive sensor nodes are subject to fading and additive noise resulting in corruption of the transmitted decisions. Incorporating the effect of fading in the detection process, a decision fusion rule based on likelihood ratio tests is derived. Optimality of likelihood ratio test is also investigated. By numerical examples, we illustrate the performance of the proposed serial detection scheme and compare it with that of the parallel distributed detection

This work presents a joint source and channel coding method for transmission of progressively compressed three-dimensional (3-D) models where the bit budget is allocated optimally. The proposed system uses the Compressed Progressive Mesh (CPM) algorithm to produce a hierarchical bitstream representing different levels of detail (LODs). In addition to optimizing the transmitted bitstream with respect to the channel characteristics, we also choose the optimum combination of quantization and tessellation to maximize the expected decoded model quality. That is, given a 3-D mesh, a total bit budget (B) and a channel packet-loss rate (P/sub LR/), we determine the optimum combination of: 1) the geometry coordinates quantizer step size (l-bit); 2) the number of transmitted batches (L); 3) the total number of channel coding bits (C); and 4) the distribution of these channel coding bits among the transmitted levels (C/sub L/). Experimental results show that, with our unequal error protection approach that uses the proposed bit-allocation algorithm, the decoded model quality degrades more gracefully (compared to either no error protection or equal error protection methods) as the packet-loss rate increases.

Based on the observation that a Cauchy density is more accurate in estimating the distribution of the ac coefficients than the traditional Laplacian density, rate and distortion models with improved accuracy are developed. The entropy and distortion models for quantized discrete cosine transform coefficients are justified in a frame bit-allocation application for H.264. Extensive analysis with carefully selected anchor video sequences demonstrates a 0.24-dB average peak signal-to-noise ratio (PSNR) improvement over the JM 8.4 rate control algorithm, and a 0.33-dB average PSNR improvement over the TM5-based bit-allocation algorithm that has recently been proposed for H.264 by Li et al. The analysis also demonstrates 20% and 60% reductions in PSNR variation among the encoded pictures when compared to the JM 8.4 rate control algorithm and the TM5-based bit-allocation algorithm, respectively.

Rate-distortion optimization can significantly improve encoder performance in MPEG-like video coding applications. Nevertheless, mode selection and motion estimation (ME) in a rate-distortion optimized setting can entail a prohibitive computational cost. In this paper, we propose a macroblock mode selection algorithm by employing computationally-efficient, yet highly-effective rate-distortion models. We also consider joint optimization of the ME with the mode selection. With our extensive analysis on various video sequences, we show that the proposed algorithm provides significant improvements with negligible increase in overall computational load. With the proposed method, we achieve 3.13, 1.78, and 1.64 dB peak signal-to-noise ratio gains over TM5 on the average, at the bit rates 2, 4, and 6 Mb/s, respectively, with CCIR-601 video. (More information about the DSP/VLSI implementation of the proposed algorithm may be found online at: http://www.ece.gatech.edu/research/labs/MCCL).

Historically, multiple description coding (MDC) and postprocessing error concealment (ECN) algorithms have evolved separately. In this paper, we propose a coordinated application of multiple description scalar quantizers (MDSQ) and ECN, where the smoothness of the video signal helps to compensate for the loss of descriptions. In particular, we perform a reconstruction that is consistent with the data received at the decoder. When only a single description is available, the video is reconstructed in such a way that: 1) if we were to regenerate two descriptions (from the reconstructed video), one of them would be equivalent to the received description and 2) the reconstructed video is spatiotemporally smooth. Experimental results with several video sequences demonstrated a peak signal-to-noise ratio (PSNR) improvement of 0.9-2.8 dB for intracoded frames. The PSNR improvements for intercoded frames were negligible. However, for both cases, the visual improvements were much more striking than what the PSNR improvement suggested.

Hyperspectral images are used for aerial and space imagery applications, including target detection, tracking, agricultural, and natural resource exploration. Unfortunately, atmospheric scattering, secondary illumination, changing viewing angles, and sensor noise degrade the quality of these images. Improving their resolution has a high payoff, but applying super-resolution techniques separately to every spectral band is problematic for two main reasons. First, the number of spectral bands can be in the hundreds, which increases the computational load excessively. Second, considering the bands separately does not make use of the information that is present across them. Furthermore, separate band super-resolution does not make use of the inherent low dimensionality of the spectral data, which can effectively be used to improve the robustness against noise. In this paper, we introduce a novel super-resolution method for hyperspectral images. An integral part of our work is to model the hyperspectral image acquisition process. We propose a model that enables us to represent the hyperspectral observations from different wavelengths as weighted linear combinations of a small number of basis image planes. Then, a method for applying super resolution to hyperspectral images using this model is presented. The method fuses information from multiple observations and spectral bands to improve spatial resolution and reconstruct the spectrum of the observed scene as a combination of a small number of spectral basis functions.

This paper presents a distributed video streaming framework using unbalanced multiple description coding (MDC) and unequal error protection. In the proposed video streaming framework, two senders simultaneously stream complementary descriptions to a single receiver over different paths. To minimize the overall distortion and exploit the benefits of multipath transport when the characteristics of each path are different, an unbalanced MDC method for wavelet-based coders combined with a TCP-friendly rate allocation algorithm is proposed. The proposed rate allocation algorithm adjusts the transmission rates and the channel coding rates for all senders in a coordinated fashion to minimize the overall distortion. Simulation results show that the proposed unbalanced MDC combined with our rate allocation algorithm achieves about 1-6 dB higher peal signal-to-noise ratio compared to conventional balanced MDC when the available bandwidths along the two paths are different under time-varying network conditions.

In this paper, we present a packet-loss resilient system for the transmission of progressively compressed three-dimensional (3D) models. It is based on a joint source and channel coding approach that trades off geometry precision for increased error resiliency to optimize the decoded model quality on the client side. We derive a theoretical framework for the overall system by which the channel packet loss behavior and the channel bandwidth can be directly related to the decoded model quality at the receiver. First, the 3D model is progressively compressed into a base mesh and a number of refinement layers. Then, we assign optimal forward error correction code rates to protect these layers according to their importance to the decoded model quality. Experimental results show that with the proposed unequal error protection approach, the decoded model quality degrades more gracefully (compared to either no error protection or equal error protection methods) as the packet-loss rate increases.

This paper addresses the problem of streaming progressively compressed three-dimensional (3-D) models over lossy networks. Out of all encoded packets that can be transmitted, we intelligently choose a subset of packets to be transmitted using transport control protocol in order to meet a distortion constraint, while transmitting the remaining packets using user datagram protocol to minimize the end-to-end delay. We call this new application-layer protocol 3-D models transport protocol. We show the effectiveness of this protocol both experimentally and theoretically. We compare the performance of the proposed protocol with systems that do not optimize transmission according to the content of the encoded bitstream. When the maximum distortion is 30, measured using the Hausdorff distance, we achieve savings in delay time ranging from 39% to 68% for packet-loss rates between 1% and 19%.

In this article, we propose an error-resilient transmission method for progressively compressed 3D models. The proposed method is scalable with respect to both channel bandwidth and channel packet-loss rate. We jointly design source and channel coders using a statistical measure that (i) calculates the number of both source and channel coding bits, and (ii) distributes the channel coding bits among the transmitted refinement levels in order to maximize the expected decoded model quality. In order to keep the total number of bits before and after applying error protection the same, we transmit fewer triangles in the latter case to accommodate the channel coding bits. When the proposed method is used to transmit a typical model over a channel with a 10% packet-loss rate, the distortion (measured using the Hausdorff distance between the original and the decoded models) is reduced by 50% compared to the case when no error protection is applied.

Hyperspectral images are used for aerial and space imagery applications including target detection, tracking, agricultural and natural resource exploration. Unfortunately, atmospheric scattering, secondary illumination, changing viewing angles and sensor noise degrade the quality of these images. We introduce a novel superresolution reconstruction method for hyperspectral images. An integral part of our work is to model the hyperspectral image acquisition process. We propose a model that enables us to represent the hyperspectral observations from different wavelengths as weighted linear combinations of a small number of basis image planes. Then a method for applying superresolution to hyperspectral images using this model is presented. The method fuses information from multiple observations and spectral bands to improve spatial resolution and reconstruct the spectrum of the observed scene.

By extensive analysis with several video sequences, we observed that the statistical distribution of the DCT coefficients in typical video coding applications is closer to a Cauchy distribution than to a Laplacian distribution. We developed the rate and the distortion expressions as a function of the video coder quantization parameter based on this observation. Experiments with an H.264 codec demonstrate that the Cauchy distribution based expressions provide better estimates for the actual rate and distortion than those that are based on the Laplacian distribution.

Secondary structure prediction has been an essential task in determining the structure and function of proteins. Prediction accuracy is improving every year towards the estimated 88% theoretical limit (Rost. B., http://cubic.bioc.columbia.edu/papers/2002 rev dekker/paper.html). There are two approaches for the secondary structure prediction. The first one, ab initio (single sequence) prediction, does not use any homology information. The evolutionary information, if available, is used by the second approach to improve the prediction accuracy by a few percentages (Schmidler, S.C. et al., J. Computational Biology, vol.7, no.1/2, p.233-48, 2000). We address the problem of single sequence prediction by developing a semi Markov HMM, similar to the one proposed by Schmidler et al. We introduce a better dependency model by considering the statistically significant amino acid correlation patterns at segment borders. Also, we propose an internal dependency model considering right to left dependencies without modifying the left to right HMM topology. In addition, we propose an iterative training method to estimate the HMM parameters better. Putting all these together, we obtained 1.5% improvement in three-state-per-residue accuracy.

Most real-time MPEG-like encoders are designed to perform in a constant bit rate (CBR) mode, in which large deviations from a target rate are avoided at any time instant because of video buffer rules set at the decoder. A rate-control method at the encoder ensures that by intelligent selection of encoding parameters, the video buffer constraints are met, while at the same time a best possible video quality is achieved. Almost all reported rate-control methods model the rate and distortion in terms of the quantization step size. A recently suggested alternative is to use /spl rho/, the fraction of zeros among the quantized DCT coefficients (He, Z. and Mitra, S.K., IEEE Trans. on Circuits and Systems for Video Technology, vol.12, no.11, 2002). We address the problems associated with the /spl rho/-domain rate-control approaches in the literature. In addition, we propose a robust MPEG-2 rate-control algorithm that aims to minimize picture quality variations. Our extensive analysis with several video sequences and using the MPEG-2 codec indicates that the proposed algorithm can achieve a more uniform visual quality, and provides a PSNR gain that ranges from 0.5 dB at 2 Mbps to 0.9 dB at 6 Mbps over TM5 on the average.

In I. Bahceci et al. (Oct 2003), antenna selection for multiple antenna transmission systems has been studied under the assumption that the subchannels between antenna pairs fade independently. In this paper, we consider the performance of such systems when the subchannels experience correlated fading. We assume that the channel state information is available only at the receiver, the antenna selection is performed only at the receiver, and the selection is based on the instantaneous received signal power. We quantify the effects of channel correlation on the diversity and coding gain when the receiver system uses all or a subset of the antennas. Theoretical results indicate that the correlations in the channel does not degrade the diversity order provided that the channel is full-rank. However, it does result in some performance loss in the coding gain. Furthermore, for non-full-rank channels, the diversity order of the system degrades significantly and is determined by the rank of the channel correlation matrix.

This paper presents a client-driven rate-distortion optimized packet scheduling algorithm for streaming multiple description (MD) encoded video, where the descriptions are distributed among different video-on-demand servers. Previously, multiple description coding (MDC) has been proposed to provide reliable video communication within a content delivery network, where the main goal was to imitate path diversity by using multiple servers concurrently. Although this approach can alleviate the severe effects of bursty packet losses, transmitting video packets in a time-sensitive and network-adaptive manner is fundamental to the success of MD streaming. Hence, in this paper we propose a packet scheduling algorithm that maximizes the quality of the video rendered at the client under the given rate constraints. This algorithm jointly considers the timeliness requirements of the streaming application, dependency structure of the streamed video, network conditions as well as the error-resiliency features of MDC.

This paper addresses the problem of streaming packetized media through a proxy server to a mobile client over multiple wireless channels. We follow a rate-distortion optimized streaming approach. A real-time sender-driven streaming framework is employed at the proxy server to maximize the playback quality at the client. The experimental results show that the proposed algorithm improves the video quality by 1.3-3dB as compared to the conventional system which simply transmits the video data in the order they are displayed.

Lagrangian minimization is one of the most powerful tools for rate-distortion optimal coding mode selection. However, in the latest video compression standard H.264/AVC, the total calculation for costs of all candidate modes may become huge since the new standard supports a large number of coding modes. In this paper, we propose an algorithm for fast coding mode selection in H.264/AVC encoders by reducing the number of candidate modes. We also present a rate estimation method for further reduction of Lagrangian cost calculation. Extensive experiments show that the proposed methods can reduce the execution time of mode selection in H.264/AVC reference software by 85% on the average while average PSNR loss is only 0.07 dB.

Rate-distortion optimization can significantly improve encoder performance in MPEG-like video coding applications especially when it is applied to coding mode selection and motion estimation. Nevertheless, it is not easy to implement rate-distortion optimization in real-time video encoders because of its prohibitive computational cost. In this paper, we propose a joint macroblock mode selection and motion estimation by employing a computationally-efficient, yet highly-effective rate-distortion model. Experimental results show that the proposed algorithm provides significant PSNR improvements with a small increase in overall computational load over the TM5-based encoder.

We present adaptive sensing, an energy-efficient topology configuration method for environment monitoring using densely deployed wireless sensor networks. Adaptive sensing puts redundant nodes into passive mode as auxiliary nodes to be used later on, thereby extending the system lifetime. Sensor data is collected at powerful macronodes periodically and a low-order model is fitted to compute a prediction area for each sensor. This prediction area is used to measure the redundancy level of the sensors covered in it. The redundant nodes are put into passive mode by macronodes while keeping the distortion in the output low. We analyze the energy-distortion tradeoff in monitoring applications, introduce the adaptive sensing algorithm based on this analysis. We also include a performance study demonstrating the advantages of our approach based on simulation results.

This paper studies the antenna selection for space-time coded orthogonal frequency division multiplexing (OFDM) systems that employ multiple transmit and receive antennas. We assume that the channel state information (CSI) is exactly known at the receiver, and hence the selection is available only at the receiver. The selection criterion is based on the instantaneous signal-to-noise ratio at each receive antenna averaged over all carrier frequencies. We also assume that space-time codes are used at the transmitter. We analyze the performance of such systems by deriving explicit upper bounds on the pairwise error probability (PEP). Closed form expressions for the PEP hounds are derived for special cases. We also present numerical examples and simulation results that validate our analysis. It turns out that it is difficult to make remarks about the diversity order since the expressions are not simple, however, for some special cases with M = 2, N = 2 antennas, we show that using antenna selection, one can achieve the same diversity gain as the one obtained by using all the receive antennas, provided that the underlying space-time code has full spatial diversity.

In this work, we propose an error-control technique for efficient streaming of packetized multimedia over time-varying wireless channels. Our objective is to obtain maximum media presentation quality for a given channel capacity. In the proposed technique, forward error correction (FEC) redundancy rate of media packets are optimized based on a distortion model. Such multimedia-source and channel characteristics as the packet importance, dependencies, decoding deadlines and channel state information are utilized in the optimization process. In addition, we also consider the effect of residual channel resources for future transmissions on the overall quality. Real-time streaming simulations with H.263+ video sequences show that the proposed technique achieves up to 1.8 dB quality improvement over nonoptimized error control methods.

In wireless video transmission, burst packet errors generally produce more catastrophic results than equal number of isolated errors. To miniimize the playback distortion it is crucial for the sender to know the packet errors at the receiver and then optimally schedule next transmissions. Unfortunately, in practice, feedback errors result in inaccurate observations of the receiving status. In this paper, we develop an optimal scheduling framework to minimize the expected distortion by first estimating the receiving status. Then, we jointly consider the source and channel characteristics and optimally choose the packets to transmit. The optimal transmission strategy is computed through a partially observable Markov decision process. The experimental results show that the proposed framework improves the average peak signal-to-noise ratio (PSNR) by 0.6-1.3 dB upon using a traditional system without packet scheduling. Moreover, we show that the proposed method smoothes out the bursty distortion periods and results in less fluctuating PSNR values.

Multiple description coding (MDC) is a source coding technique that exploits path diversity to increase the robustness of transmitting a compressed signal over error-prone channels. However, the application of MDC to video coding is still problematic because a prediction mismatch problem, called drift, may occur at the decoder when one description is lost. In the literature, it has been shown that drift can be effectively prevented by encoding this prediction mismatch and sending it as side information in a three-prediction-loop structure. However, this side information is totally redundant when both descriptions are received. We propose a very simple, but efficient two-stage MDC in the central prediction loop based on the three-prediction-loop structure of A. Reibman et al. (see Proc. IEEE Int. Conf. Image Processing - ICIP'99, p.837-41, 1999). We introduce an additional multiple description scalar quantizer into the central prediction loop. The encoding of the side information is modified in such a way that it can be used to improve the video quality when no drift occurs, but still maintain a drift prevention capability. The simulation results, with different video sequences at various packet loss rates, demonstrate that the proposed two-stage MDC structure indeed enhances the quality when the channel is error-free, yet effectively avoids drift when one description is completely lost.

Considerable attention has been directed to the problem of producing high-resolution video and still images from multiple low-resolution images. This multiframe reconstruction, also known as super-resolution reconstruction, is beginning to be applied to compressed video. Super-resolution techniques that have been designed for raw (i.e., uncompressed) video may not be effective when applied to compressed video because they do not incorporate the compression process into their models. The compression process introduces quantization error, which is the dominant source of error in some cases. In this paper, we propose a stochastic framework where quantization information as well as other statistical information about additive noise and image prior can be utilized effectively.

This paper presents a coordinated multiple-substream unequal error-protection and error-concealment algorithm for SPIHT-coded bitstreams transmitted over lossy channels. In the proposed scheme, we divide the video sequence corresponding to a group of pictures into two subsequences and independently encode each subsequence using a three-dimensional SPIHT algorithm. We use two different partitioning schemes to generate the substreams, each of which offers some advantages under the appropriate channel condition. Each substream is protected by an FEC-based unequal error-protection algorithm, which assigns unequal forward error correction codes to each bit plane. Any information that is lost during the transmission for any substream is estimated at the receiver by using the correlation between the substreams and the smoothness of the video signal. Simulation results show that the proposed multiple-substream UEP algorithm is simple, fast, and robust in hostile network conditions, and that the proposed error-concealment algorithm can achieve 2-3-dB PSNR gain over the case when error concealment is not used at high packet-loss rates.

We propose a joint source-channel coding scheme for wireless communication systems with multiple transmit and receive antennas. The source coder is realized by a multiple description encoder that generates multiple bit streams for the same source. Each description is then separately turbo coded and transmitted using multiple antennas. For the receiver, we describe a suitable iterative joint source-channel decoding technique that exploits the correlations between the descriptions. Finally, we present several examples that illustrate the performance of the proposed system and compare it with other approaches.

This paper presents a distributed video streaming framework using unbalanced multiple description coding (MDC) and forward error correction. In the proposed video streaming framework, two senders simultaneously stream complementary descriptions to the receiver over different paths. To minimize the overall distortion and exploit the benefits of multi-path transport when the characteristics of each path are different, an unbalanced MDC method for wavelet-based coders combined with a TCP-friendly rate allocation algorithm is proposed. The proposed rate allocation algorithm adjusts the transmission rates and the channel coding rates for all senders in a coordinated fashion to minimize the overall distortion. Simulation results show that the proposed unbalanced MDC combined with our rate allocation algorithm achieves about 1-3 dB higher PSNR compared to conventional balanced MDC when the available bandwidth along each path is different in time-varying network conditions.

Multiple description coding addresses the problem of encoding a source signal into two or more sub-streams. Each sub-stream is assumed to be transmitted over an independent path. However, multiple independent paths may not be available, and for most cases only a single path can be used. The scheduling of these descriptions over a single path is still problematic because of the bursty packet loss behavior. In this paper, we propose a novel packet scheduling method for multiple description coded video transmission over a single path. The proposed method adaptively schedules the descriptions according to channel conditions by jointly considering the bursty packet loss behavior and the transmission delay of the network. The simulation results show that the proposed method can improve the video quality by 1.5 dB on average as compared to the method of sending one packet of each description right after the other. It also outperforms a heuristic interleaving approach by roughly 0.3-0.45 dB, depending on the channel conditions.

This paper presents a new framework for multimedia streaming that integrates the application and network layer functionalities to meet such stringent application requirements as delay and loss. The coordination between these two layers provides more robust media transmission even under severe network conditions. In this framework, a multiple description source coder is used to produce multiple independently-decodable streams that are routed over partially link-disjoint (non-shared) path to combat bursty packet losses. We model multi-path streaming and propose a multi-path streaming and propose a multi-path selection method that chooses a set of paths maximizing the overall quality at the client. Overlay infrastructure is then used to achieve multi-path routing over these selected paths. The simulation results show that the average peak signal-to-noise ratio (PSNR) improves by up to 8.1 dB, if the same source video is routed over intelligently selected multiple paths instead of the shortest path or maximally link-disjoint paths. In addition to PSNR improvement in quality, the end-user experiences a more continual steaming quality.

Layered (LC) and multiple description coding (MDC) have been proposed as source coding techniques that are robust to channel errors for video transmission. In this paper, we investigate and study the error-resilience capabilities of these two encoding techniques through extensive experimentation, and explore the environments where they perform best. Although some of our conclusions agree with those in the literature, we believe that this paper provides the most comprehensive performance comparison yet between LC and MDC.

In this paper, we present PINCO, an in-network compression scheme for energy constrained, distributed, wireless sensor networks. PINCO reduces redundancy in the data collected from sensors, thereby decreasing the wireless communication among the sensor nodes and saving energy. Sensor data is buffered in the network and combined through a pipelined compression scheme into groups of data, while satisfying a user-specified end-to-end latency bound. We introduce a PINCO scheme for single-valued sensor readings. In this scheme, each group of data is a highly flexible structure so that compressed data can be recompressed without decompressing, in order to reduce newly available redundancy at a different stage of the network. We discuss how PINCO parameters affect its performance, and how to tweak them for different performance requirements. We also include a performance study demonstrating the advantages of our approach over other data collection schemes based on simulation and prototype deployment results.

This paper studies the streaming of packetized media from multiple servers to a client over a lossy network. In particular, we propose a client-driven rate-distortion optimal packet scheduling algorithm that decides which packet(s) to be requested from which server(s) at a given request opportunity. In doing so, the proposed scheduling algorithm not only attains the maximal presentation quality but also conforms to the rate constraints dictated by the flow, window and congestion control mechanisms. The simulation results clearly demonstrate the efficacy of the proposed algorithm over the single-server rate-distortion optimized streaming.

An imaging system produces a degraded measurement of a real-valued quantity that varies temporally, spectrally, and spatially. When there are multiple measurements of the same scene, it is possible to combine the nonredundant information in those measurements and produce an improved image that has more information than any of the measurements does alone. This type of reconstruction is known as image fusion. Depending on the diversity of information in the measurements, it is possible to achieve temporal, spectral, spatial, and gray-scale improvements with an image fusion algorithm. In this paper, we are proposing an image fusion algorithm that produces an image of higher spatial and gray-scale information from multiple measurements. The algorithm estimates the spatial and illumination correlation between multiple measurements, and employs a set-theoretic reconstruction technique.

This paper addresses the problem of streaming progressively compressed 3-D models over lossy networks. Out of all encoded packets that can be transmitted, we intelligently choose a subset of packets to be transmitted using TCP in order to meet a distortion constraint, while transmitting the remaining packets using UDP to minimize the end-to-end delay. We call this new application-layer protocol 3TP (3-D models transport protocol). Experimental results show savings of 42% and 68% in delay time at packet-loss rates of 6% and 19%, respectively, compared to systems that do not optimize transmission according to the encoded bitstream content.

In a previous work [A. C. Begen, et al., 2003], we proposed an optimal multi-path selection method for multiple description (MD) encoded video streaming. To do so, we first modelled multi-path streaming and then developed an expression, i.e., an objective (cost) function, that estimated average streaming distortion in terms of network statistics, media characteristics and application requirements. Naturally, the ultimate goal was to find the set of paths that minimized this cost function. However, finding such sets of paths turned out to be intractable in large topologies. Hence, in this paper, we provide a fast heuristics-based solution by exploiting the infrastructure features of the Internet. The simulations run over various random Internet topologies show that the proposed heuristic is able to find a good solution in a much shorter time than the brute-force approach. Particularly, this heuristic is best suited to such interactive multimedia applications as video-conferencing and VoIP, where multi-path computation is a time-critical process. In addition, it is also suitable for the clients whose processing power capabilities are limited.

The emerging ITU-T H.264 video coding standard has been developed to achieve significant improvements over the existing standards in the compression performance. Although the basic coding framework of the standard is similar to that of the existing standards, H.264 introduces many new features. The standard is close to completion, and preliminary tests were reported to give satisfactory results. However, thorough performance evaluations by independent parties have yet to come. In this paper, we provide an independent analysis of the coding performance improvement of H.264 over various other existing video coding standards. Our experiments demonstrated that the H.264 standard can achieve 50% coding gain over MPEG-2, 47% coding gain over H.263 baseline, and 24% coding gain over H.263 high profile encoders.

Constrained optimization techniques provide an effective solution to the rate-distortion optimized mode selection in hybrid motion compensation-DCT based video encoders. However, the minimization of the Lagrangian cost function often presents a computational burden in real-time applications. In this paper, we propose a new method for rate distortion optimized coding mode selection. Instead of using the Lagrangian method, we employ a computationally efficient, yet highly effective rate-distortion model to minimize the overall distortion of a macroblock. Extensive experimental results show that the proposed method provides PSNR improvements of 2.79, 1.64 and 1.54 dB over TM5-based MPEG-2 encoder, at the bit-rates of 2, 4 and 6 Mbps, respectively, with negligible increase in computational complexity.

This paper presents a TCP-friendly rate allocation algorithm for multiple description coding combined with path diversity. A 3-D SPIHT-based multiple description coding is used to generate two independent substreams and each substream is protected with an FEC-based unequal error protection algorithm. In the proposed video streaming framework, two senders simultaneously stream complementary substreams to the receiver over different paths. The proposed rate allocation algorithm coordinates distributed video streaming from multiple senders to a single receiver to minimize the overall distortion. Simulation results show that the proposed algorithm increases tolerance to packet loss due to network congestion and achieves higher visual quality compared to non-adaptive methods.

Multiple description coding (MDC) is a source coding technique that exploits path diversity to increase the robustness of transmitting a compressed signal over error-prone channels. However, the applicant of MDC to video coding is still problematic because a prediction mismatch problem, called drift, may occur at the decoder when one description is lost. In this paper, we propose a drift free motion-compensated predictive coding method for multiple description scalar quantizers. The proposed method maintains two more prediction loops at the encoder side to produce all possible predictions. Then, the two descriptions are generated from these two additional prediction loops in such a way that the drift can be prevented when only one description is received. By receiving both descriptions, the decoder can still combine these two descriptions in the central prediction loop to improve the video quality. Experimental results show that the proposed method can effectively prevent drift and improve the quality of single-description reconstruction by 0.2-0.8 dB as compared to the method in [A. Reibman et al., 1999].

Multiple description (MD) and layered coding (LC) are two source-coding approaches proposed for adaptive and robust video streaming over lossy networks. Both streaming methods aim to improve the streaming quality by transmitting the generated multiple sub-bitstreams over partially link-disjoint paths. However, the achieved qualities heavily depend on the media characteristics, path conditions and application requirements. In order to attain the highest quality, we study optimal multi-path selection methods for both MD and LC streaming. The simulations run over an overlay infrastructure show 9.0 - 12.5 dB peak signal-to-noise ratio (PSNR) improvement when the video is streamed over intelligently selected multiple paths instead of the shortest path or maximally link-disjoint paths. By the help of the proposed path selection methods, the end users estimate the expected qualities of MD and LC streaming for the given network conditions and application requirements prior to the streaming, which allows the users to make a choice accordingly.

Two-dimensional mesh-based models provide a good alternative to motion estimation and compensation. The estimation of the best node-point motion vectors is a challenging task. To this effect, Nakaya and Harashima (1994) proposed a hexagonal matching procedure. Toklu et al. improved the hexagonal search algorithm in terms of both motion-estimation accuracy and computational complexity by employing a hierarchy of regular meshes. Recognizing the limitations of regular meshes, Van Beek et al. (1997) extended Toklu et al.'s (1996) work by utilizing content-based meshes. Here, we provide an alternative hierarchical motion-estimation method with content-based meshes where hierarchical representations are employed for both the images and the irregular meshes in order to provide further improvements in computational complexity as well as motion accuracy. The comparison results are provided with real video sequences.

Methods for estimating motion in video sequences that are based on the optical flow equation (OFE) assume that the scene illumination is uniform and that the imaging optics are ideal. When these assumptions are appropriate, these methods can be very accurate, but when they are not, the accuracy of the motion field drops off accordingly. This paper extends the models upon which the OFE methods are based to include irregular, time-varying illumination models and models for imperfect optics that introduce vignetting, gamma, and geometric warping, such as are likely to be found with inexpensive PC cameras. The resulting optimization framework estimates the motion parameters, illumination parameters, and camera parameters simultaneously. In some cases these models can lead to nonlinear equations which must be solved iteratively; in other cases, the resulting optimization problem is linear. For the former case an efficient, hierarchical, iterative framework is provided that can be used to implement the motion estimator.

Face images that are captured by surveillance cameras usually have a very low resolution, which significantly limits the performance of face recognition systems. In the past, super-resolution techniques have been proposed to increase the resolution by combining information from multiple images. These techniques use super-resolution as a preprocessing step to obtain a high-resolution image that is later passed to a face recognition system. Considering that most state-of-the-art face recognition systems use an initial dimensionality reduction method, we propose to transfer the super-resolution reconstruction from pixel domain to a lower dimensional face space. Such an approach has the advantage of a significant decrease in the computational complexity of the super-resolution reconstruction. The reconstruction algorithm no longer tries to obtain a visually improved high-quality image, but instead constructs the information required by the recognition system directly in the low dimensional domain without any unnecessary overhead. In addition, we show that face-space super-resolution is more robust to registration errors and noise than pixel-domain super-resolution because of the addition of model-based constraints.

This paper presents a new bit-plane-wise unequal error protection algorithm for progressive bitstreams transmitted over lossy networks. The proposed algorithm protects a compressed embedded bitstream generated by a 3-D SPIHT algorithm by assigning an unequal amount of forward error correction (FEC) to each bit-plane. The proposed algorithm reduces the amount of side information needed to send the size of each code to the decoder by limiting the number of quality levels to the number of bit-planes to be sent while providing a graceful degradation of picture quality as packet losses increase. We also apply our proposed algorithm to transmission of JPEG 2000 coded images over the Internet. To get additional error-resilience at high packet loss rates, we extend our algorithm to multiple-substream unequal error protection. Simulation results show that the proposed algorithm is simple, fast and robust in hostile network conditions and, therefore, can provide reasonable picture quality for video applications under varying network conditions.

This correspondence studies antenna selection for wireless communications systems that employ multiple transmit and receive antennas. We assume that (1) the channel is characterized by quasi-static Rayleigh flat fading, and the subchannels fade independently, (2) the channel state information (CSI) is exactly known at the receiver, (3) the selection is available only at the receiver, and it is based on the instantaneous signal-to-noise ratio (SNR) at each receive antenna, and (4) space-time codes are used at the transmitter. We analyze the performance of such systems by deriving explicit upper bounds on the pairwise error probability (PEP). This performance analysis shows that (1) by selecting the set of antennas that observe the largest instantaneous SNR, one can achieve the same diversity gain as the one obtained by using all the receive antennas, provided that the underlying space-time code has full spatial diversity, and (2) in the case of rank-deficient space-time codes, the diversity gain may be dramatically reduced when antenna selection is used. However, we emphasize that in both cases the coding gain is reduced with antenna selection compared to the full complexity system. Based on the upper bounds derived, we describe code design principles suitable for antenna selection. Specifically, for systems with two transmit antennas, we design space-time codes that perform better than the known ones when antenna selection is employed. Finally, we present numerical examples and simulation results that validate our analysis and code design principles.

In this paper, we present a packet-loss resilient 3-D graphics transmission system that is scalable with respect to both channel bandwidth and channel error characteristics. The algorithm trades off source coding efficiency for increased bit-stream error resilience to optimize the decoded mesh quality on the client side. It uses the Compressed Progressive Mesh (CPM) algorithm to generate a hierarchical bit-stream representing different levels of details (LODs). We assign optimal forward error correction (FEC) code rates to protect different parts of the bit-stream differently. These optimal FEC code rates are determined theoretically via a distortion function that accounts for: the channel packet loss rate, the nature of the encoded 3-D mesh and the error protection bit-budget. We present experimental results, which show that with our unequal error protection (UEP) optimal approach, the decoded mesh quality degrades more gracefully (compared to either no error protection (NEP) or equal error protection (EEP) methods) as the packet loss rate increases.

Most commercial digital cameras have three types of color sensors (for red, green, and blue channels) that are placed on a detector surface according to a specific pattern. At the location of each pixel only one color sample is taken, and the values of the other colors must be interpolated using neighboring samples. This color plane interpolation is known as demosaicing; it is one of the important tasks in a digital camera pipeline. If demosaicing is not performed appropriately, images suffer from highly visible color artifacts. In this paper we present a new demosaicing technique that uses inter-channel correlation effectively in an alternating-projections scheme. We have compared this technique with various state-of-the-art demosaicing techniques, and it outperforms all of them, both visually and in terms of mean square error.

Historically, multiple description coding (MDC) and post-processing error concealment (ECN) algorithms for error-resilient video transmission have evolved separately. Although multiple description coding provides a way to make the video quality degrade gracefully in the presence of packet losses, the resulting quality may be unsatisfactory when only one description is used to reconstruct the signal. This paper proposes a joint description transform coding (MDTC) and statistical error concealment algorithm that adopts a maximum a posteriori (MAP) estimation approach to compensate for the loss of descriptions. This coordination results in significant improvements in reconstructed video quality over independent use of MDTC.

This paper presents a new bit-plane-wise unequal error protection algorithm for Internet video applications. The proposed algorithm protects a compressed embedded bitstream generated by a 3-D SPIHT algorithm by assigning unequal forward error correction (FEC) codes to each bit-plane. The proposed algorithm reduces the amount of side information needed to send the size of each code to the decoder by limiting the number of quality levels to the number of bit-planes to be sent while providing a graceful degradation of picture quality as packet losses increase. To get additional error-resilience at high packet loss rates, we extend our algorithm to multiple-substream unequal error protection. Simulation results show that the proposed algorithm is simple, fast and robust in hostile network conditions and therefore can provide reasonable picture quality for Internet video applications under varying network conditions.

Historically, multiple description coding (MDC) and post-processing error concealment (ECN) algorithms for video have evolved separately. Although multiple description coding provides a way to make the video quality degrade gracefully in the presence of packet losses, the resulting quality may be unsatisfactory when only one description is used to reconstruct the signal. This paper proposes a coordinated application of multiple description scalar quantizer (MDSQ) and error concealment, where the smoothness of the video signal helps to compensate for the loss of descriptions. This joint optimization results in significant improvements in reconstructed video quality over independent use of MDSQ.

In this paper, we present an unequal error protection method for packet-loss resilient transmission of progressively compressed 3D meshes. The proposed method is based on a source and channel coding approach where we set up a theoretical framework for the overall system by which the channel packet loss behavior and the channel bandwidth can be directly related to the decoded mesh quality at the receiver. In particular, we develop a statistical distortion measure and optimize it to compute the best combination of (i) the number of triangles to transmit, (ii) the total number of channel coding bits, and (iii) the distribution of these error-protection bits among the transmitted layers in order to maximize the expected decoded mesh quality at the receiver. The proposed method differs from the earlier approaches in two major aspects: (i) determination of the number of channel coding bits (C) and (ii) the approach of reducing the source rate in order to accommodate for channel coding bits. When the proposed method is used to transmit a typical 3D mesh over a channel with a 10% packet loss rate, the distortion (measured using the Hausdorff distance between the original and the decoded overly sampled meshes) is reduced by 50% compared to the case when no error protection is applied.

Face images that are captured by surveillance cameras usually have a very low resolution, which significantly limits the performance of face recognition systems. In the past, super-resolution techniques have been proposed that attempt to increase the resolution by combining information from multiple images. These techniques use super-resolution as a preprocessing system to obtain a high resolution image that can later be passed to a face recognition system. Considering that most state-of-the-art face recognition systems use an initial dimensionality reduction method, we propose embedding the super-resolution algorithm into the face recognition system so that super-resolution is not performed in the pixel domain, but is instead performed in a reduced dimensional domain. The advantage of such an approach is a significant decrease in the computational complexity of the super-resolution algorithm because the algorithm no longer tries to construct a visually improved high quality image, but instead constructs the information required by the recognition algorithm directly in the lower dimensional domain without any unnecessary overhead.

This paper presents a coordinated multiple-substream unequal error protection and error concealment algorithm for SPIHT-coded bitstreams transmitted over lossy channels. In the proposed scheme, we divide the video sequence corresponding to a group of pictures into two sub-sequences in the temporal domain and independently encode each sub-sequence with a 3-D SPIHT algorithm to generate two independent substreams. Each substream is protected by an FEC-based unequal error protection algorithm that assigns unequal forward error correction codes for each substream with bit-plane granularity. The information that is lost during transmission for one substream is estimated at the receiver by using the correlation between the two substreams and the smoothness of the video signal. Simulation results show that the proposed multiple-substream UEP algorithm is simple, fast, and robust in hostile network conditions, and that the proposed error concealment algorithm achieves about 1-3 dB PSNR gain over the case where there is no error concealment at high packet loss rates.

In this paper, we propose a very simple, but efficient two-stage MDC in the central prediction loop based on Reibman's three-prediction-loop structure (Reibman et al. (1999)). In our design, an additional multiple description scalar quantizer (MDSQ) is introduced into the central prediction loop. The encoding of the side information is modified in such a way that it can be used to improve the video quality when no drift occurs, but still maintain drift prevention capability. The simulation results with different video sequences at various packet loss rates demonstrate that the proposed two-stage MDC structure indeed enhances the quality by about 0.25 dB when the channel is error-free, yet effectively avoids drift when one description is completely lost.

We propose a protocol for efficient streaming of 3-D animations over lossy channels. In order to improve the expected quality on the client's side, we first transmit a crude model of the 3-D mesh, of its texture, and of its immediate evolution. We endow this data with significant error-protection against transmission error. Then we transmit a series of upgrades that refine the accuracy of the model and/or of the animation. These are encoded with lower levels of error protection that is proportional to their impact on the quality of the upgraded animation. We propose the following types of upgrade chunks: selection of a subset of vertices, adjustments of the accuracy of the positions of the selected vertices, connectivity refinements, motion adjustments of the selected subset of vertices, adjustments of the accuracy of the texture coordinates of the selected vertices, and upgrade of the quality of the texture. Finally, given the allocated bit-budget, we determine the optimal number of both source and channel coding bits assigned for each chunk to maximize the animation quality on the client's side. The optimization takes into account both the bandwidth and the error characteristics of the channel.

A packet-loss resilient, bandwidth-scalable 3D graphics streaming system is proposed. It uses the compressed progressive mesh (CPM) algorithm (see Pajarola, R. and Rossignac, J., IEEE Trans. on Visualization and Computer Graphics, vol.6, no.1, p.79-93, 2000) to generate a hierarchical bit-stream to represent different levels of details (LODs). We assign forward error correction (FEC) codes to each layer in the encoded bit-stream according to its importance. To this end, we develop a new distortion measure that quantifies the distortion in the reconstructed mesh when part of the bit-stream is lost. Experimental results with a collection of 3D meshes have been conducted to demonstrate the efficacy of our solution. They indicate that with our proposed unequal error protection (UEP), the decoded mesh quality degrades more gracefully, compared to either no error protection (NEP) or equal error protection (EEP) methods, as the packet loss rate increases.

Multiframe resolution enhancement ("superresolution") methods are becoming widely studied, but only a few procedures have been developed to work with compressed video, despite the fact that compression is a standard component of most image- and video-processing applications. One of these methods uses quantization-bound information to define convex sets and then employs a technique called "projections onto convex sets" (POCS) to estimate the original image. Another uses a discrete cosine transformation (DCT)-domain Bayesian estimator to enhance resolution in the presence of both quantization and additive noise. The latter approach is also capable of incorporating known source statistics and other reconstruction constraints to impose blocking artifact reduction and edge enhancement as part of the solution. We propose a spatial-domain Bayesian estimator that has advantages over both of these approaches.

There are a number of useful methods for creating high-quality video or still images from a lower quality video source. The best of these involve motion compensating a number of video frames to produce the desired video or still. These methods are formulated in the space domain and they require that the input be expressed in that format. More and more frequently, however, video sources are presented in a compressed format, such as MPEG, H.263, or DV. Ironically, there is important information in the compressed domain representation that is lost if the video is first decompressed and then used with a spatial-domain method. In particular, quantization information is lost once the video has been decompressed. Here, we propose a motion-compensated, transform-domain super-resolution procedure for creating high-quality video or still images that directly incorporates the transform-domain quantization information by working with the compressed bit stream. We apply this new formulation to MPEG-compressed video and demonstrate its effectiveness.

A major drawback of block-based still-image or video-compression methods at low rates is the visible block boundaries that are also known as blocking artifacts. Several methods have been proposed in the literature to reduce these artifacts. Most are single image methods, which do not distinguish between video and still images. However, video has a temporal dimension that can lead to better reconstruction if utilized effectively. We show how to combine information from multiple frames to reduce blocking artifacts. We derive constraint sets using motion between neighboring frames and quantization information that is available in the video bit stream. These multiframe constraint sets can be used to reduce blocking artifacts in an alternating-projections scheme. They can also be included in existing set-theoretic algorithms to improve their performance by narrowing down the feasibility set. Experimental results show the effectiveness of using these multiframe constraint sets.

Most commercial digital cameras use color filter arrays to sample red, green, and blue colors according to a specific pattern. At the location of each pixel only one color sample is taken, and the values of the other colors must be interpolated using neighboring samples. This color plane interpolation is known as demosaicing; it is one of the important tasks in a digital camera pipeline. If demosaicing is not performed appropriately, images suffer from highly visible color artifacts. In this paper we present a new demosaicing technique that uses inter-channel correlation effectively in an alternating-projections scheme. We have compared this technique with six state-of-the-art demosaicing techniques, and it outperforms all of them, both visually and in terms of mean square error.

Compressed video sequences are very vulnerable to channel disturbances when they are transmitted through an unreliable medium such as a wireless channel. Transmission errors not only corrupt the current decoded frame, but they may also propagate to succeeding frames. A number of post-processing error concealment (ECN) methods that exploit the spatial and/or temporal redundancy in the video signal have been proposed to combat channel disturbances. Although these approaches can effectively conceal lost or erroneous macroblocks (MBs), all of them only consider spatial and/or temporal correlation in a single frame (the corrupted one), which limits their ability to obtain an optimal recovery. Since the error propagates to the next few motion-compensated frames in the presence of lost MBs in an I or P frame, error concealment should simultaneously minimize the errors not only in the current decoded frame but also in the succeeding B and P frames that depend on the corrupted frame. We propose a novel multiframe recovery principle which analyzes the propagation of a lost MB into succeeding frames. Then, MPEG-compatible spatial and temporal error concealment approaches using this multiframe recovery principle are proposed, where the lost MBs are recovered in such a way that the error propagation is minimized.

Multiple description coding (MDC) provides error resilience by producing several descriptions of the source so that various quality reconstructions can be obtained from any subset of the descriptions. The use of correlating transforms within the MDC framework has been studied for "on-off" channels, such as packet-switched channels. Since this channel model is not appropriate for wireless systems, we introduce fading channel models, such as Rayleigh, Rician, or Nakagami channel, within the context of MDC. We further discuss the optimal correlating transforms.

Two-dimensional mesh-based models provide a good alternative to motion estimation and compensation. The estimation of the best motion vectors at the node-point motion vectors is a challenging task. To this effect, Nakaya et al. (1994), proposed a hexagonal matching procedure. Toklu et al. (1996) improved the hexagonal search algorithm in terms of both motion estimation accuracy and computational complexity by employing a hierarchy of regular meshes. Recognizing the limitations of regular meshes, Van Beek et al. (1999), extended Toklu's work by utilizing content-based meshes. Here, we provide an alternative hierarchical motion estimation method with content-based meshes where hierarchical representations are employed for images as well as meshes in order to provide further improvements in computational complexity as well as motion accuracy.

A major drawback of block-based still image or video compression methods at low rates are the visible block boundaries that are also known as blocking artifacts. Several methods have been proposed in the literature to reduce these artifacts for video sequences. However, most are simply adaptations of still image blocking artifact reduction methods, which do not exploit temporal information. We propose a novel multi-frame blocking artifact reduction method that incorporates temporal information effectively. This method uses the spatial correlations that exist between the successive frames to define constraint sets at multiple frames and provides a Projections Onto Convex Sets (POCS) solution. The proposed method operates solely on transform domain (DCT) data, and hence provides a solution that is compatible with the observed video. It does not need to make any spatial smoothness assumptions, which are typical with blocking artifact reduction algorithms for still images.

Resolution enhancement for video sequences has always been an attractive application in multimedia signal processing. "Superresolution" methods, that combine non-redundant information from a set of low-resolution images, are beginning to be applied to the most popular video compression standard, MPEG. Bayesian approaches, which are very successful for raw video, largely fail for MPEG video, since they do not incorporate the compression process into their models. This compression process introduces quantization noise, which is comparable to the additive noise that is used in the Bayesian models. We present an analytical derivation that combines the quantization and additive noises in a stochastic framework for MPEG-compressed video. This is a general framework in the sense that different video acquisition models, source statistics, implementation techniques can be used with it.

Compressed video sequences are very sensitive to channel disturbances when they are transmitted through an unreliable environment such as a wireless channel. The transmission errors not only corrupt the current decoded frame, but also propagate to the succeeding frames. Many error concealment methods have been proposed to recover the errors by exploiting temporal and spatial correlation. In this paper, we develop a multi-frame boundary matching algorithm (MF-BMA), which utilizes the boundary smoothness property not only in the decoded frame but also in the succeeding frames. The experimental results demonstrate that the proposed algorithm can provide better performance, especially when horizontal edges exist across the lost macroblocks.

The number of bits assigned to represent the color intensity at image pixels is usually referred as the bit depth. When the bit depth is not sufficient, images suffer from ridge-like structures known as false contours. Bit-depth limitations become important when low-contrast details are required, as in medical imaging, aerial/satellite photography, and high-quality scanning: applications. In this paper, we investigate a method for increasing bit depth. Specifically, we show that when a sequence of video frames is available, then it is possible to achieve a higher bit depth through a projections onto convex sets (POCS) based reconstruction method.

In this paper, we propose to improve the POCS-based super-resolution reconstruction (SRR) methods in two ways. First, the discretization of the continuous image formation model is improved to explicitly allow for higher order interpolation methods to be used. Second, the constraint sets are modified to reduce the amount of edge ringing present in the high resolution image estimate. This effectively regularizes the inversion process.

Almost all temporal video segmentation algorithms employ thresholds. Although the selection of thresholds considerably affects the algorithm performance a framework to choose them effectively does not exist. This paper introduces a framework for selecting statistically optimal thresholds in temporal video segmentation algorithms. Statistics for various temporal video events, such as shot boundary, zoom and pan events, are collected. Then, optimal thresholds are estimated so as to minimize a statistical cost function defined in terms of the complied statistics. Results with several real video sequences containing over 150,000 frames show improvements ranging from 5%-20% in true detection and false alarm rates.

With the plummeting cost of video capture equipment and the availability of free applications allowing MPEG encoding, hobbiests can now affordably capture and encode their home videos for storage and transmission in MPEG format. To produce a high-quality still image from this archived MPEG video, motion compensation must be applied to a number of video frames. A great deal of work has been done formulating this problem in the spatial domain, however, the classical spatial domain formulations are suboptimal for transform-coded video. Patti and Altunbasak (see Proc. IEEE Int. Conf. Image Processing, 1999) proposed a POCS solution that directly incorporates the transform domain quantization information by working with the compressed bit stream. We present an alternative maximum a posteriori (MAP) solution that not only incorporates quantization information, but can also impose additional blocking artifact reduction constraints. We apply this new formulation to MPEG compressed video and demonstrate its effectiveness.

In this paper, two-dimensional (2-D) motion estimation techniques based on the well-known optical-flow equation are treated. Cost functions are defined for various motion models that allow the optical flow equation (OFE) to be used in an optimization framework for the purpose of estimating the 2-D motion parameters. A method of extending these cost functions to include camera parameters is then proposed. The resulting sometimes non-linear optimization framework takes into account motion-model parameter estimation in the presence of geometric lens distortions. Linearization, when necessary, results in an OFE-like motion estimation method.

To date, there exist a number of worthy methods for creating a high quality still image from a video source. The best of these methods involve motion compensating a number of video frames to produce the desired still. These motion compensated methods are formulated in the space-domain. More and more, however, the video source is only available in compressed format such as MPEG, H.263 or DV. In this case there is important compressed domain information that is not utilized by the formulation. Most notably, the quantization information from the bitstream is not correctly taken into account. We propose a motion-compensated, transform domain formulation for creating high quality video stills that directly incorporates transform domain quantization information. We apply this new formulation to MPEG compressed video and demonstrate its effectiveness compared to a reasonable space-domain approach.

Creating high quality still pictures from video presents a challenging problem due to the low spatial resolution of most video signals. Many algorithms have been proposed in the literature that utilize multiple video frames to increase spatial resolution. These algorithms depend on two critical assumptions: first, that the scene does not change significantly in the temporal vicinity of the frame of interest, and second that the motion estimation between video frames is extremely accurate. Noting that panoramic views are not only visually pleasing, but also fit the aforementioned assumptions, we propose the use of a scene change detection algorithm to locate scenes containing mainly pan/tilt types of motion. Since many digital video sequences are compressed using MPEG, it is desirable to perform all computations with minimal decompression. To this end, we also propose methods to locate pans from MPEG-compressed video. Once the pan segments are located, a number of highly accurate motion estimation methods can be successfully applied to the video segment. Given the resulting accurate motion, there exist various methods of attacking the resolution enhancement problem and creating a panoramic still image. These, for the most part, are computationally expensive. Therefore, we propose a fast method of obtaining enhanced resolution panoramas from the lower resolution video signal.

This paper presents pixel-based and region-based parametric motion segmentation methods for robust motion segmentation with the goal of aligning motion boundaries with those of real objects in a scene. We first describe a two-step iterative procedure for parametric motion segmentation by either motion-vector or motion-compensated intensity matching. We next present a region-based extension of this method, whereby all pixels within a predefined spatial region are assigned the same motion label. These predefined regions may be fixed- or variable-size blocks or arbitrary-shaped areas defined by color or texture uniformity. A particular combination of these pixel-based and region-based methods is then proposed as a complete algorithm to obtain the best possible segmentation results on a variety of image sequences. Experimental results showing the benefits of the proposed scheme are provided.

This paper presents region-based affine motion parameter clustering methods by motion-vector and intensity matching to provide improved robustness and alignment of motion boundaries with real object boundaries. Regions may be formed as fixed- or variable-size rectangular blocks, triangular patches, or arbitrary-shaped areas defined by color or texture uniformity. A particular combination of these affine clustering methods is then proposed to obtain the best segmentation results on a variety of image sequences. Experimental results showing the benefits of the proposed scheme are provided.

This paper presents an object-based image coding scheme, where each image object is encoded individually, provided that their boundaries are specified. This allows object-based quality scalability, in addition to mixing lossy and lossless coding modes depending on the requirements of each image object. Furthermore, we propose a new object coding method using 2-D mesh-based image sampling in which the quadrilateral mesh patches are warped into square blocks, and encoded by traditional data/waveform coding methods. Experimental results on test images are provided.

This paper presents an object-based video modeling. Motion segmentation is performed at the initial frame to identify different coherently moving regions, called motion-objects. These regions are grouped to form objects. Each motion-object is fitted a content-based mesh, and tracked subsequently to the next frame via mesh motion estimation and compensation. The uncovered background region(s), which emerges when objects move, will be segmented so as to identify the new objects or occluded parts of already existing objects. The mesh model is modified to reflect the changes in object boundaries.

Video compression and retrieval have been treated as separate problems in the past. We present an object-based video representation that facilitates both compression and retrieval. Typically in retrieval applications, a video sequence is subdivided in time into a set of shorter segments each of which contains similar content. These segments are represented by 2-D representative images called "key-frames" that greatly reduce amount of data that is searched. However, key-frames do not describe the motions and actions of objects within the segment. We propose a representation that extends the ideas of the key-frame to further include what we define as "key-objects". These key-objects consist of regions within a key-frame that move with similar motion. Thus our key-objects allow a retrieval system to more efficiently present information to users and assist them in browsing and retrieving relevant video content.

This paper compares two-dimensional (2-D) and three-dimensional (3-D) object modeling in terms of their capabilities and performance (peak signal-to-noise-ratio and visual image quality) for very low bitrate video coding. We show that 2-D object-based coding with affine/perspective transformations and triangular mesh models can simulate almost all capabilities of 3-D object-based approaches using wireframe models at a fraction of the computational cost. Furthermore, experiments indicate that a 2-D mesh-based coder-decoder performs favorably compared to the new H.263 standard in terms of visual quality.

Motion compensation using two-dimensional (2-D) mesh models requires computation of the parameters of a spatial transformation for each mesh element (patch). It is well known that the parameters of an affine (bilinear or perspective) mapping can be uniquely estimated from three (four) point correspondences (at the vertices of a triangular or quadrilateral mesh element). On the other hand, overdetermined solutions using more than the required minimum number of point correspondences provide increased robustness against correspondence-estimation errors, however, this necessitates special consideration to preserve mesh-connectivity. This paper presents closed-form, overdetermined solutions for least squares estimation of affine motion parameters for a triangular mesh, which preserve mesh-connectivity using patch-based or node-based connectivity constraints. In particular, four new algorithms are presented: patch-constrained methods using point correspondences or spatio-temporal intensity gradients, and node-constrained methods using point correspondences or spatio-temporal intensity gradients. The methods using point correspondences can be viewed as postprocessing of a dense motion field for best representation in terms of a set of irregularly spaced samples. The methods that are based on spatio-temporal intensity gradients offer closed-form solutions for direct estimation of the best node-point motion vectors (equivalently the best transformation parameters). We show that the performance of the proposed closed-form solutions are comparable to those of the alternative search-based solutions at a fraction of the computational cost.

Two-dimensional (2-D) mesh-based motion compensation preserves neighboring relations (through connectivity of the mesh) as well as allowing warping transformations between pairs of frames; thus, it effectively eliminates blocking artifacts that are common in motion compensation by block matching. However, available 2-D mesh models, whether uniform or non-uniform, enforce connectivity everywhere within a frame, which is clearly not suitable across occlusion boundaries. To this effect, we hereby propose an occlusion-adaptive forward-tracking mesh model, where connectivity of the mesh elements (patches) across covered and uncovered region boundaries are broken. This is achieved by allowing no node points within the background to be covered (BTBC) and refining the mesh structure within the model failure (MF) region(s) at each frame. The proposed content-based mesh structure enables better rendition of the motion (compared to a uniform or a hierarchical mesh), while tracking is necessary to avoid transmission of all node locations at each frame. Experimental results show successful motion compensation and tracking.

We present a multistage affine motion segmentation method that combines the benefits of the dominant motion and block-based affine modeling approaches. In particular, we propose two key modifications to a recent motion segmentation algorithm developed by Wang and Adelson (1994). 1) The adaptive k-means clustering step is replaced by a merging step, whereby the affine parameters of a block which has the smallest representation error, rather than the respective cluster center, is used to represent each layer; and 2) we implement it in multiple stages, where pixels belonging to a single motion model are labeled at each stage. Performance improvement due to the proposed modifications is demonstrated on real video frames.

Available 2-D mesh models, whether uniform or adaptive, preserve neighboring relationships between the patches (connectivity) everywhere within the frame, which is clearly not suitable across motion and occlusion boundaries. To this effect, this paper proposes an occlusion-adaptive, content-based mesh design and forward tracking algorithm, where no node points are placed within the background to be covered (BTBC), and node points along the boundary of the uncovered background (UB) are allowed to split. Furthermore, the mesh is refined within the UB region at every frame. Experimental results show successful motion compensation and tracking.

Presents a new framework for combining maximum likelihood (ML) stereo-motion fusion with adaptive iterated extended Kalman filtering (IEKF) for 3-D motion tracking. The ML stereo-fusion step, with two stereo-pairs, generates observations of 3-D feature matches to be used by the IEKF step. The IEKF step, in turn, computes updated 3-D motion parameter estimates to be used by the ML stereo-motion fusion step. The covariance of the observation noise process is regulated by the value of the ML cost function to address occlusion related problems. The proposed simultaneous approach is compared with performing the 3-D feature correspondence estimation and the Kalman filtering separately using simulated stereo imagery.

We present a complete system for 2-D object-based video compression with a method for 2-D content-based triangular mesh design, two connectivity preserving affine motion parameterization schemes, two methods for temporal mesh propagation, a polygon-based adaptive model failure detection/coding scheme, and bit rate control strategies. The feasibility of the proposed methods has been demonstrated by experimental results.

Both motion/structure estimation from monocular video and disparity estimation from still-frame stereo are known to be ill-posed problems. Further, because disparity varies by depth, and motion parameters are different for independently moving objects, they can both benefit from scene segmentation. To this effect, we present a framework for simultaneous motion and disparity estimation including scene segmentation in stereo video. In this formulation, pairs of disparity and motion parameter vector values are segmented into K regions, where within each region a single set of motion parameters is defined, and the disparity field is allowed to vary smoothly. The algorithm iterates between computing the maximum a posteriori probability (MAP) estimates of the disparity and segmentation fields conditioned on the present motion parameter estimates, and the maximum likelihood (hit) estimates of the motion parameters via simulated annealing (SA). Simulation results are provided.>