Recent advancements in vision-language models (VLMs) have found important applications in medical imaging, particularly in digital pathology. VLMs demand large-scale datasets of image-caption pairs, which is often hard to obtain in medical domains. State-of-the-art VLMs in digital pathology have been pre-trained on datasets that are significantly smaller than their computer vision counterparts. Furthermore, the caption of a pathology slide often refers to a small sub-set of features in the image-an important point that is ignored in existing VLM pre-training schemes. Another important issue that is under-appericated is that the performance of state-of-the-art VLMs in zero-shot classification tasks can be sensitive to the choice of the prompts. In this paper, we first employ language rewrites using a large language model (LLM) to enrich a public pathology image-caption dataset and make it publicly available. Our extensive experiments demonstrate that by training with language rewrites, we can boost the performance of a state-of-the-art digital pathology VLM on downstream tasks such as zero-shot classification, and text-to-image and image-to-text retrieval. We further leverage LLMs to demonstrate the sensitivity of zero-shot classification results to the choice of prompts and propose a scalable approach to characterize this when comparing models. Finally, we present a novel context modulation layer that adjusts the image embeddings for better aligning with the paired text and use context-specific language rewrites for training this layer. In our results, we show that the proposed context modulation framework can further yield substantial performance gains.

The aim of multispectral image fusion is to combine object or scene features of images with different spectral characteristics to increase the perceptual quality. In this paper, we present a novel learning-based solution to image fusion problem focusing on infrared and visible spectrum images. The proposed solution utilizes only convolution and pooling layers together with a loss function using no-reference quality metrics. The analysis is performed qualitatively and quantitatively on various datasets. The results show better performance than state-of-the-art methods. Also, the size of our network enables real-time performance on embedded devices. Project codes can be found at https://github.com/ferhatcan/pyFusionSR.

The cost-effective visual representation and fast query-by-example search are two challenging goals that should be maintained for web-scale visual retrieval tasks on moderate hardware. This paper introduces a fast and robust method that ensures both of these goals by obtaining state-of-the-art performance for an image-to-video search scenario. Hence, we present critical enhancements to well-known indexing and visual representation techniques by promoting faster, better and moderate retrieval performance. We also boost the superiority of our method for some visual challenges by exploiting individual decisions of local and global descriptors at query time. For instance, local content descriptors represent copied/duplicated scenes with large geometric deformations such as scale, orientation and affine transformation. In contrast, the use of global content descriptors is more practical for near-duplicate and semantic searches. Experiments are conducted on a large-scale Stanford I2V dataset. The experimental results show that our method is useful in terms of complexity and query processing time for large-scale visual retrieval scenarios, even if local and global representations are used together. The proposed method is superior and achieves state-of-the-art performance based on the mean average precision (MAP) score of this dataset. Lastly, we report additional MAP scores after updating the ground annotations unveiled by retrieval results of the proposed method, and it shows that the actual performance.

Inverse problems in imaging such as denoising, deblurring, superresolution have been addressed for many decades. In recent years, convolutional neural networks (CNNs) have been widely used for many inverse problem areas. Although their indisputable success, CNNs are not mathematically validated as to how and what they learn. In this study, the authors prove that during training, CNN elements solve for inverse problems which are optimum solutions stored as CNN neuron filters. They discuss the necessity of mutual coherence between CNN layer elements in order for a network to converge to the optimum solution. They prove that required mutual coherence can be provided by the usage of residual learning and skip connections. They have set rules over training sets and depth of networks for better convergence, i.e. performance. They have experimentally validated theoretical assertions.

Data acquired from multichannel sensors are a highly valuable asset to interpret the environment for a variety of remote sensing applications. However, low spatial resolution is a critical limitation for previous sensors, and the constituent materials of a scene can be mixed in different fractions due to their spatial interactions. Spectral unmixing is a technique that allows us to obtain the material spectral signatures and their fractions from hyperspectral data. In this paper, we propose a novel endmember extraction and hyperspectral unmixing scheme, so-called EndNet, that is based on a two-staged autoencoder network. This well-known structure is completely enhanced and restructured by introducing additional layers and a projection metric [i.e., spectral angle distance (SAD) instead of inner product] to achieve an optimum solution. Moreover, we present a novel loss function that is composed of a Kullback-Leibler divergence term with SAD similarity and additional penalty terms to improve the sparsity of the estimates. These modifications enable us to set the common properties of endmembers, such as nonlinearity and sparsity for autoencoder networks. Finally, due to the stochastic-gradient-based approach, the method is scalable for large-scale data and it can be accelerated on graphical processing units. To demonstrate the superiority of our proposed method, we conduct extensive experiments on several well-known data sets. The results confirm that the proposed method considerably improves the performance compared to the state-of-the-art techniques in the literature.

In this paper, we propose a novel hyperspectral unmixing technique based on deep spectral convolution networks (DSCN). Particularly, three important contributions are presented throughout this paper. First, fully-connected linear operation is replaced with spectral convolutions to extract local spectral characteristics from hyperspectral signatures with a deeper network architecture. Second, instead of batch normalization, we propose a spectral normalization layer which improves the selectivity of filters by normalizing their spectral responses. Third, we introduce two fusion configurations that produce ideal abundance maps by using the abstract representations computed from previous layers. In experiments, we use two real datasets to evaluate the performance of our method with other baseline techniques. The experimental results validate that the proposed method outperforms baselines based on Root Mean Square Error (RMSE).

In this study, we propose a novel single image Bayesian super-resolution (SR) algorithm where the hyperspectral image (HSI) is the only source of information. The main contribution of the proposed approach is to convert the ill-posed SR reconstruction (SRR) problem in the spectral domain to a quadratic optimization problem in the abundance map domain. In order to do so, Markov Random Field (MRF) based energy minimization approach is proposed and proved that the solution is quadratic. The proposed approach consists of five main steps. First, the number of endmembers in the scene is determined using virtual dimensionality. Second, the endmembers and their low resolution abundance maps are computed using simplex identification via the splitted augmented Lagrangian (SISAL) and fully constrained least squares (FCLS) algorithms. Third, high resolution (HR) abundance maps are obtained using our proposed maximum a posteriori (MAP) based energy function. This energy function is minimized subject to smoothness, unity and boundary constraints. Fourth, the HR abundance maps are further enhanced with texture preserving methods. Finally, HR HSI is reconstructed using the extracted endmembers and the enhanced abundance maps. The proposed method is tested on three real HSI datasets; namely the Cave, Harvard and Hyperspectral Remote Sensing Scenes (HRSS) and compared to state-of-the-art alternative methods using peak signal to noise ratio, structural similarity, spectral angle mapper and relative dimensionless global error in synthesis metrics. It is shown that the proposed method outperforms the state of the art methods in terms of quality while preserving the spectral consistency.

Super-resolution Reconstruction (SRR) is technique to increase the spatial resolution of images. It is especially useful for hyperspectral images (HSI), which have good spectral resolution but low spatial resolution. In this study, we propose an improvement to our previous work and present a novel MAP-MRF (maximum a posteriori-Markov random Fields) based approach for the SRR of HSI. The key point of our approach is to find the abundance maps of an HSI and perform SRR on the abundance maps using MRF based energy minimization, without needing any other additional source of information. In order to do so, first, PCA is used to determine the endmembers. Second, SISAL and fully constraint least squares (FCLS) are used to estimate the abundance maps. Third, in order to find the high resolution abundance maps, the ill-posed inverse SRR problem for abundances is regularized with a MAP-MRF based approach. The MAP-MRF formulation is restricted with the constraints which are specific to the abundances. Using the non-linear programming (NLP) techniques, the convex MAP formulation is minimized and High Resolution (HR) abundance maps are obtained. Then, these maps are used to construct the HR HSI. This improved SRR method is verified on real data sets, and quantitative performance comparison is achieved using PSNR, SSIM and PSNR metrics. Our results indicate that this improved method gives very close results to the original high resolution images, keeps the spectral consistency, and performs better than the compared algorithms.

In this study a novel, spatio-temporal characteristics based video content analysis method is presented. The proposed method has been evaluated on different video quality assessment databases, which include videos with different characteristics and distortion types. Test results obtained on different databases demonstrate the robustness and accuracy of the proposed content analysis method. Moreover, this analysis method is employed in order to examine the performance improvement in audiovisual quality assessment when the video content is taken into consideration.

Among the human related factors, aggressive driving behavior is one of the major causes of traffic accidents [17]. On the other hand, detection and characterization of driver aggressiveness is a challenging task since there exist different psychological causes behind it. However, information about the driver behavior could be extracted from the data that is collected via different sensing devices. This paper presents a method to detect driver aggressiveness using only visual information provided by forward camera. The proposed method is based on detection of the road lines and the vehicles on the road and extracts information related with road lane departure rate, speed of the vehicle and possible forward collision time. Using these extracted features, a classifier is utilized in order to detect if driver shows an aggressive driving behavior. The proposed method is tested by a subjective testing method using 76 different driving sessions and achieved 90.4% success.

Accurate correction of atmospheric effects on data captured by an infrared (IR) camera is crucial for several applications such as vegetation monitoring, temperature monitoring, satellite images, hyperspectral imaging, numerical model simulations, surface properties characterization, and IR measurement interpretation. Atmospheric effects depend on the temporal changes, i.e., year, season, day, hour, etc., and on the geometry between the camera and the measured scene, i.e., line of sight. The orientation and the optical depth of the camera significantly affect the variation of the geometry across the pixels. In this paper, we propose a method to estimate the range and zenith angle of each pixel using only the Global Positioning System (GPS) coordinates of the camera and a point of interest in the scene. The estimated geometry and measured meteorological data are used to obtain the spectral atmospheric transmittance and path radiance. Furthermore, we propose an atmospheric effects removal, i.e., atmospheric correction, method that considers the spectral characteristics of the detector, lens, and filter. The proposed atmospheric correction process is analyzed in detail with the simultaneous measurements of two IR cameras. In this process, an enhanced temperature calibration method is developed and it is shown that the temperature accuracy for the dynamic range of the IR camera is very close to the noise equivalent temperature difference (NETD) value of the camera.

The recent improvements on internet technologies and video coding techniques cause an increase in copyright infringements especially for video. Frequently, image-based approaches appear as an essential solution due to the fact that joint usage of quantization-based indexing and weak geometric consistency stages give a capability to compare duplicate videos quickly. However, exploiting purely spatial content ignores the temporal variation of video. In this work, we propose a system that combines the state-of-the-art quantization-based indexing scheme with a novel trajectory-based geometric consistency on spatio-temporal features. This combination improves duplicate video matching task significantly. Briefly, spatial mean and variance of the trajectories are incorporated to establish a weak geometric consistency among pair of frames. To show the success of the proposed method, content-based video copy detection field is selected and TRECVID 2009 dataset is utilized. The experimental results show that constituting trajectory-based consistency on corresponding feature pairs outperforms the performances of merely utilizing spatiotemporal signature and visual signature with enhanced weak geometric consistency.

The increasing demand for streaming video raises the need for flexible and easily implemented Video Quality Assessment (VQA) metrics. Although there are different VQA metrics, most of these are either Full-Reference (FR) or Reduced-Reference (RR). Both FR and RR metrics bring challenges for on-the-fly multimedia systems due to the necessity of additional network traffic for reference data. No-Reference (NR) video metrics, on the other hand, as the name suggests, are much more flexible for user-end applications. This introduces a need for robust and efficient NR VQA metrics. In this paper, an NR VQA metric considering spatiotemporal information, bit rate, and packet loss rate characteristics of a video content is proposed. The proposed metric is evaluated on EPFL-PoliMI dataset, which includes different video content characteristics. The experimental results show that the proposed metric is a robust and accurate NR VQA metric towards diverse video content characteristics.

Need for automatic video copy detection is increased with the recent technical developments in the internet technologies and video recording. Even though image-based techniques with bag-of-word kind of representations are accepted as the best solution because of robustness and speed, they discard the convenient geometric relation which exists among interest points. In this work, we propose a novel geometric relation which computes a binary signature leveraging existence and non-existence of interest points in the neighborhood area. The experimental results on TRECVID 2009 content-based video copy detection dataset show that combination of our method with recently proposed quantization-based indexing and weak geometric consistency schemes outperforms classical representations.

Super‐resolution (SR) image reconstruction refers to methods where a higher resolution image is reconstructed using a set of overlapping aliased low‐resolution observations of the same scene. Although edge preservation has been a widely explored topic in SR literature, texture‐specific regularisation has recently gained interest. In this study, texture‐specific regularisation is handled as a post‐processing step. A two stage method is proposed, comprising multiple SR reconstructions with different regularisation parameters followed by a restoration step for preserving edges and textures. In the first stage, two maximum‐a‐posteriori estimators with two different amounts of regularisation are employed. In the second stage, pixel‐to‐pixel difference between these two estimates is post‐processed to restore edges and textures. Frequency selective characteristics of discrete cosine transform and Gabor filters are utilised in the post‐processing step. Experiments on synthetically generated images and real experiments demonstrate that the proposed methods give better results compared with the state‐of‐the‐art SR methods especially on textures and edges.

In this paper a novel fast circle detection algorithm is proposed which depends on the spatial properties of the connected components on the image. Two 1-D transforms of each connected component is obtained by taking the Radon Transform of the image for two different directions, which are in fact the integrations of the image through horizontal and vertical directions. Circles are detected using the similarities of detected peaks on the transformed functions and the characteristics of the values in between those peaks. The success of the method is analyzed using synthetic images and the performance of the method is presented and compared with Modified Hough Transform (MHT) using synthetic images.

Many researchers have been developing objective video quality assessment methods due to increasing demand for perceived video quality measurement results by end users to speed-up advancements of multimedia services. However, most of these methods are either Full-Reference (FR) metrics, which require the original video or Reduced-Reference (RR) metrics, which need some features extracted from the original video. No-Reference (NR) metrics, on the other hand, do not require any information about the original video; hence, are much more suitable for applications like video streaming. This paper presents a novel, objective, NR video quality assessment algorithm. The proposed algorithm is based on utilization of spatial extent of video, temporal extent of video using motion vectors, bit rate, and packet loss ratio. Test results obtained using LIVE video quality database demonstrate the accuracy and robustness of the proposed metric.

In this paper, we present a complete framework of an end-to-end error resilient transmission of 3D video over DVB-H and provide an analysis of transmission parameters. We perform the analysis for various layering, protection strategy and prediction structure using different contents and different channel conditions.

In order to speed up the wide-spread proliferation of the 3D video technologies (e.g., coding, transmission, display, etc), the effect of these technologies on 3D perception should be efficiently and reliably investigated. Using Full-Reference (FR) objective metrics for this investigation is not practical especially for “on the fly” 3D perception evaluation. Thus, a Reduced Reference (RR) metric is proposed to predict the depth perception of 3D video in this paper. The color-plus-depth 3D video representation is exploited for the proposed metric. Since the significant depth levels of the depth map sequences have great influence on the depth perception of users, they are considered as side information in the proposed RR metric. To determine the significant depth levels, the depth map sequences are abstracted using bilateral filter. Video Quality Metric (VQM) is utilized to predict the depth perception ensured by the significant depth levels due to its well correlation with the Human Visual System (HVS). The performance assessment results present that the proposed RR metric can be utilized in place of a FR metric to reliably measure the depth perception of 3D video with a low overhead.

This paper proposes a new maximum a posteriori (MAP) based super-resolution (SR) image reconstruction method targeting edges and textures in images. Unlike conventional MAP based SR image reconstruction methods a spatially varying image prior is employed which is updated according to the frequency content of the reconstructed image at each iteration at different locations. Two alternative methods based on discrete cosine transforms (DCT) and Gabor filters are proposed for determining the image prior. The proposed method is validated through simulations and real experiments which clearly demonstrates significant visual improvements especially on edges and textures compared to state-of-the-art SR methods.

This paper aims at providing an overview of the core technologies enabling the delivery of 3-D Media to next-generation mobile devices. To succeed in the design of the corresponding system, a profound knowledge about the human visual system and the visual cues that form the perception of depth, combined with understanding of the user requirements for designing user experience for mobile 3-D media, are required. These aspects are addressed first and related with the critical parts of the generic system within a novel user-centered research framework. Next-generation mobile devices are characterized through their portable 3-D displays, as those are considered critical for enabling a genuine 3-D experience on mobiles. Quality of 3-D content is emphasized as the most important factor for the adoption of the new technology. Quality is characterized through the most typical, 3-D-specific visual artifacts on portable 3-D displays and through subjective tests addressing the acceptance and satisfaction of different 3-D video representation, coding, and transmission methods. An emphasis is put on 3-D video broadcast over digital video broadcasting-handheld (DVB-H) in order to illustrate the importance of the joint source-channel optimization of 3-D video for its efficient compression and robust transmission over error-prone channels. The comparative results obtained identify the best coding and transmission approaches and enlighten the interaction between video quality and depth perception along with the influence of the context of media use. Finally, the paper speculates on the role and place of 3-D multimedia mobile devices in the future internet continuum involving the users in cocreation and refining of rich 3-D media content.

In this paper, we deal with layered predictive compression of animated meshes represented by series of 3D static meshes with same connectivity. We propose two schemes to improve the prediction. First improvement is using weighted spatial prediction rather than averaging neighbor vertices. The second improvement is a novel predictor based on rotation angle of incident triangles in current and previous frames. The experimental results show that around 6-10 % bitrate reduction can be achieved by replacing the spatial prediction in the reference coder with the proposed weighted spatial prediction and 9-18 % bitrate reduction is possible with the proposed angle based predictor using weighted spatial prediction, depending on the content and quantization level.

In this paper, we present a complete framework of an end-to-end error resilient transmission of 3D video over Digital Video Broadcasting - Handheld (DVB-H) and provide an extensive analysis of coding and transmission parameters. We perform the analysis for different coding and error resilience schemes using different contents coded at different bitrate levels. Throughout the experiments, we investigate the effects of video content type, video bitrate, coding method and unequal protection level for different channel conditions. The results show that Multi-view Coding (MVC) coding outperforms Simulcast and distribution of available bitrate between video quality and Forward Error Correction (FEC) protection is an important factor in different channel conditions.

In this paper, we study the effect of limited amplitude resolution (pixel depth) in super-resolution problem. The problem we address differs from the standard super-resolution problem in that amplitude resolution is considered as important as spatial resolution. We study the trade-off between the pixel depth and spatial resolution of low resolution (LR) images in order to obtain the best visual quality in the reconstructed high resolution (HR) image. The proposed framework reveals great flexibility in terms of pixel depth and number of LR images in super-resolution problem, and demonstrates that it is possible to obtain target visual qualities with different measurement scenarios including images with different amplitude and spatial resolutions.

In this paper a novel direction adaptive super-resolution (SR) image reconstruction method is proposed. The proposed maximum a-posteriori (MAP) based estimator uses gradient direction for optimal noise reduction while preserving the edges. Compared to the other edge-preserving methods, the proposed algorithm uses gradient direction in addition to the gradient amplitude for optimum regularization. The method comprises a gradient amplitude and direction estimation stage where a gradient direction map is obtained. This map guides the SR reconstruction stage through iterations. Three variations of the proposed method are compared against other edge-preserving super resolution methods. PSNR (Peak signal-to-noise-ratio), SSIM (Structural similarity index measure) values, and illustrations show that the proposed method has better performance especially on image pixel values where a strong gradient is present.

3D video based on multi-view representations is becoming widely popular. Real-time encoding/decoding of such video is an important concern as the number and resolution of views increase. We present systematic methods for design and optimization of real-time multi-view video encoding/decoding algorithms using multi-core processors and provide benchmark results. The proposed multi-core decoding architectures are fully compliant with the current JVT-MVC international standard, and enable multi-threaded processing with negligible loss of encoding efficiency. Benchmark results show that multi-core processors and multi-threading decoding is necessary for real-time multiview video decoding and display.

In this paper, we propose a distributed 3D dynamic mesh coding system. The system is based on Slepian and Wolf’s and Wyner and Ziv’s information-theoretic results. Our system extends the ideas in distributed video coding to 3D dynamic meshes with constant connectivity. The connectivity of the sequence and key frames are encoded and decoded by a conventional static mesh coder. The Wyner-Ziv frames are encoded independent of key frames but decoded jointly with decoded key frames. The joint decoding is performed by the low density parity check codes and the side information generated by linear interpolation of decoded key frames. Experimental results show that better rate-distortion performance is obtained compared to encoding each frame by a static mesh coder.

This paper presents an efficient joint source-channel coding scheme based on forward error correction (FEC) for three dimensional (3D) models. The system employs a wavelet based zero-tree 3D mesh coder based on Progressive Geometry Compression (PGC). Reed-Solomon (RS) codes are applied to the embedded output bitstream to add resiliency to packet losses. Two-state Markovian channel model is employed to model packet losses. The proposed method applies approximately optimal and unequal FEC across packets. Therefore the scheme is scalable to varying network bandwidth and packet loss rates (PLR). In addition, Distortion-Rate (D-R) curve is modeled to decrease the computational complexity. Experimental results show that the proposed method achieves considerably better expected quality compared to previous packet-loss resilient schemes.

In this paper, we propose a novel hybrid topology and interest based clustering to organize the overlay network in order to reduce startup latency and service interruption probability. Our method uses a clustering technique to organize the peers according to their interests and locality information. The proposed technique is compared with the current unstructured P2P architectures in terms of average hit time, hit ratio for searched content, and the maximum rate of information (in bits per second) that can be transmitted over P2P network caused by protocol and overlay- level connectivity. The simulation results show that the proposed system outperforms the current unstructured P2P architectures.

In this paper, we propose a new low-complexity retransmission based optimal video streaming and rate adaptation algorithm. The proposed OSRC (Optimal packet Scheduling and Rate Control) algorithm provides average reward optimal solution to the joint scheduling and rate control problem. The efficacy of the OSRC algorithm is demonstrated against optimal FEC based schemes and results are verified over TFRC (TCP Friendly Rate Control) transport with ns-2 simulations.

In this work, we present a multiple description coding (MDC) scheme for reliable transmission of compressed three dimensional (3-D) meshes. It trades off reconstruction quality for error resilience to provide the best expected reconstruction of 3-D mesh at the decoder side. The proposed scheme is based on multiresolution geometry compression achieved by using wavelet transform and modified SPIHT algorithm. The trees of wavelet coefficients are divided into sets. Each description contains the coarsest level mesh and a number of tree sets coded with different rates. The original 3-D geometry can be reconstructed with acceptable quality from any received description. More descriptions provide better reconstruction quality. The proposed algorithm provides flexible number of descriptions and is optimized for varying packet loss rates (PLR) and channel bandwidth.

We present a survey of transport methods for 3-D video ranging from early analog 3DTV systems to most recent digital technologies that show promise in designing 3DTV systems of tomorrow. Potential digital transport architectures for 3DTV include the DVB architecture for broadcast and the Internet Protocol (IP) architecture for wired or wireless streaming. There are different multiview representation/compression methods for delivering the 3-D experience, which provide a tradeoff between compression efficiency, random access to views, and ease of rate adaptation, including the "video-plus-depth" compressed representation and various multiview video coding (MVC) options. Commercial activities using these representations in broadcast and IP streaming have emerged, and successful transport of such data has been reported. Motivated by the growing impact of the Internet protocol based media transport technologies, we focus on the ubiquitous Internet as the network infrastructure of choice for future 3DTV systems. Current research issues in unicast and multicast mode multiview video streaming include network protocols such as DCCP and peer-to-peer protocols, effective congestion control, packet loss protection and concealment, video rate adaptation, and network/service scalability. Examples of end-to-end systems for multiview video streaming have been provided.

Research efforts on 3DTV technology have been strengthened worldwide recently, covering the whole media processing chain from capture to display. Different 3DTV systems rely on different 3D scene representations that integrate various types of data. Efficient coding of these data is crucial for the success of 3DTV. Compression of pixel-type data including stereo video, multiview video, and associated depth or disparity maps extends available principles of classical video coding. Powerful algorithms and open international standards for multiview video coding and coding of video plus depth data are available and under development, which will provide the basis for introduction of various 3DTV systems and services in the near future. Compression of 3D mesh models has also reached a high level of maturity. For static geometry, a variety of powerful algorithms are available to efficiently compress vertices and connectivity. Compression of dynamic 3D geometry is currently a more active field of research. Temporal prediction is an important mechanism to remove redundancy from animated 3D mesh sequences. Error resilience is important for transmission of data over error prone channels, and multiple description coding (MDC) is a suitable way to protect data. MDC of still images and 2D video has already been widely studied, whereas multiview video and 3D meshes have been addressed only recently. Intellectual property protection of 3D data by watermarking is a pioneering research area as well. The 3D watermarking methods in the literature are classified into three groups, considering the dimensions of the main components of scene representations and the resulting components after applying the algorithm. In general, 3DTV coding technology is maturating. Systems and services may enter the market in the near future. However, the research area is relatively young compared to coding of other types of media. Therefore, there is still a lot of room for improvement and new development of algorithms.

In this paper, we address the problem of 3D model transmission over error-prone channels using multiple description coding (MDC). The objective of MDC is to encode a source into multiple bitstreams, called descriptions, supporting multiple quality levels of decoding. Compared to layered coding techniques, each description can be decoded independently to approximate the model. In the proposed approach, the mesh geometry is compressed using multiresolution geometry compression. Then multiple descriptions are obtained by applying multiple description scalar quantization (MDSQ) to the obtained wavelet coefficients. Experimental results show that, the proposed approach achieves competitive compression performance compared with existing multiple description methods.

H.264 is the current state-of-the-art monoscopic video codec providing almost twice the coding efficiency with the same quality comparing the previous codecs. With the increasing interest in 3D TV, multi-view video sequences that are provided by multiple cameras capturing the three dimensional objects and/or scene are more widely used. Compressing multi-view sequences independently with H.264 (simulcast) is not efficient since the redundancy between the closer cameras is not exploited. In order to reduce these redundancies, we propose a multi-view video codec based on H.264 using disparity estimation/compensation as well as motion estimation/compensation. In order to effectively search for disparity/motion without increasing computational complexity, we modified the buffering structure of H.264 and implemented several referencing modes. Our results show that for closely located cameras, our codec outperforms simulcast H.264 coding. For sparsely located cameras, our method can still improve coding gain depending on the video characteristics.

Today, stereoscopic and multi-view video are among the popular research areas in the multimedia world. In this study, we have designed and built a platform consisting of stereo-view capturing, real-time transmission and display. At the display stage, end users view video in 3D by using polarized glasses. Multi-view video is compressed in an efficient way by using multi-view video coding techniques and streamed using standard real-time transport protocols. The entire system is built by modifying available open source systems whenever possible. Receiver can view the content of the video built from multiple channels as mono or stereo depending on its display and bandwidth capabilities

In order to compress stereo image pairs effectively, disparity compensation is the most widely used method. In this paper we examined the effects of using different disparity maps and their properties in an embedded JPEG2000 based disparity compensated stereo image coder. These properties include the block size, estimation method and the resulting entropy of the disparity map. Experimental results show that basic block matching gives better results than ground truth, especially on occluded regions and boundaries.

Transmission of high quality video over the Internet faces many challenges including unpredictable packet loss characteristics of the current Internet and the heterogeneity of receivers in terms of their bandwidth and processing capabilities. To address these challenges, we propose an architecture in this paper that is based on the temporally scalable and error resilient video coding mode of the H.263+ codec. In this architecture, the video frames are transported over a new generation IP network that supports differentiated services (Diffserv). We also propose a novel two rate three color promotion-capable marker (trTCPCM) to be used at the edge of the Diffserv network. Our simulation study demonstrates that an average of 30 dB can be achieved in the case of highly congested links.

A new feature based approach to frontal face recognition with Gabor wavelets is presented. The feature points are automatically extracted using the local characteristics of each individual face in order to decrease the effect of occluded features. There is no training as in neural network approaches, thus a single frontal face for each individual is enough as a reference. Experimental results show that the proposed method achieves a recognition ratio of over %95.

This paper presents a simple and effective preprocessing method developed for editing compressed video sequences. The proposed method involves extracting information about different video segments from the compressed bitstream. The algorithm is not designed to distinguish among types of segments but rather to indicate the position and duration. Since no decoding of the bitstream is done, the computational load of the algorithm is very low. Although the experimental results are shown on MPEG compressed video sequences, the algorithm can easily be applied to MJPEG, MPEG4 sequences given the header information.

This paper describes a feature based, two-pass algorithm for scene break detection in video sequences. In the first pass, the camera cuts are detected on the low resolution video using histogram comparison. The gradual changes are then detected on the high resolution video based on the editing characteristics. Experimental results indicate that the proposed discontinuous cut, dissolve and fade detection method is both computationally low and robust to local changes within the sequences.

We propose a new method for combined color image segmentation and edge linking. The image is first segmented based on color information only. The segmentation map is modeled by a Gibbs random field, to ensure formation of spatially contiguous regions. Next, spatial edge locations are determined using the magnitude of the gradient of the 3-channel image vector field. Finally, regions in the segmentation map are split and merged by a region-labeling procedure to enforce their consistency with the edge map. The boundaries of the final segmentation map constitute a linked edge map. Experimental results are reported.

This paper compares two-dimensional (2-D) and three-dimensional (3-D) object modeling in terms of their capabilities and performance (peak signal-to-noise-ratio and visual image quality) for very low bitrate video coding. We show that 2-D object-based coding with affine/perspective transformations and triangular mesh models can simulate almost all capabilities of 3-D object-based approaches using wireframe models at a fraction of the computational cost. Furthermore, experiments indicate that a 2-D mesh-based coder-decoder performs favorably compared to the new H.263 standard in terms of visual quality.

We present a multistage affine motion segmentation method that combines the benefits of the dominant motion and block-based affine modeling approaches. In particular, we propose two key modifications to a recent motion segmentation algorithm developed by Wang and Adelson (1994). 1) The adaptive k-means clustering step is replaced by a merging step, whereby the affine parameters of a block which has the smallest representation error, rather than the respective cluster center, is used to represent each layer; and 2) we implement it in multiple stages, where pixels belonging to a single motion model are labeled at each stage. Performance improvement due to the proposed modifications is demonstrated on real video frames.

We propose a new method for combined color image segmentation and edge linking. The image is first segmented based on color information only. The segmentation map is modeled by a Gibbs random field, to ensure formation of spatially contiguous regions. Next, spatial edge locations are determined using the magnitude of the gradient of the 3-channel image vector field. Finally, regions in the segmentation map are split and merged by a region-labeling procedure to enforce their consistency with the edge map. The boundaries of the final segmentation map constitute a linked edge map. Experimental results are reported.

Presents a new framework for combining maximum likelihood (ML) stereo-motion fusion with adaptive iterated extended Kalman filtering (IEKF) for 3-D motion tracking. The ML stereo-fusion step, with two stereo-pairs, generates observations of 3-D feature matches to be used by the IEKF step. The IEKF step, in turn, computes updated 3-D motion parameter estimates to be used by the ML stereo-motion fusion step. The covariance of the observation noise process is regulated by the value of the ML cost function to address occlusion related problems. The proposed simultaneous approach is compared with performing the 3-D feature correspondence estimation and the Kalman filtering separately using simulated stereo imagery.

We present a complete system for 2-D object-based video compression with a method for 2-D content-based triangular mesh design, two connectivity preserving affine motion parameterization schemes, two methods for temporal mesh propagation, a polygon-based adaptive model failure detection/coding scheme, and bit rate control strategies. The feasibility of the proposed methods has been demonstrated by experimental results.

An adaptive smoothing technique for speckle suppression in medical B-scan ultrasonic imaging is presented. The technique is based on filtering with appropriately shaped and sized local kernels. For each image pixel, a filtering kernel, which fits to the local homogeneous region containing the processed pixel, is obtained through a local statistics based region growing technique. The performance of the proposed filter has been tested on the phantom and tissue images. The results show that the filter effectively reduces the speckle while preserving the resolvable details. The simulation results are presented in a comparative way with two existing speckle suppression methods.

We address the problem of 3-D motion estimation in the context of knowledge-based coding of facial image sequences. The proposed method handles the global and local motion estimation and the adaptation of a generic wire-frame to a particular speaker simultaneously within an optical flow based framework including the photometric effects of motion. We use a flexible wire-frame model whose local structure is characterized by the normal vectors of the patches which are related to the coordinates of the nodes. Geometrical constraints that describe the propagation of the movement of the nodes are introduced, which are then efficiently utilized to reduce the number of independent structure parameters. A stochastic relaxation algorithm has been used to determine optimum global motion estimates and the parameters describing the structure of the wire-frame model. For the initialization of the motion and structure parameters, a modified feature based algorithm is used. Experimental results with simulated facial image sequences are given.>

Both motion/structure estimation from monocular video and disparity estimation from still-frame stereo are known to be ill-posed problems. Further, because disparity varies by depth, and motion parameters are different for independently moving objects, they can both benefit from scene segmentation. To this effect, we present a framework for simultaneous motion and disparity estimation including scene segmentation in stereo video. In this formulation, pairs of disparity and motion parameter vector values are segmented into K regions, where within each region a single set of motion parameters is defined, and the disparity field is allowed to vary smoothly. The algorithm iterates between computing the maximum a posteriori probability (MAP) estimates of the disparity and segmentation fields conditioned on the present motion parameter estimates, and the maximum likelihood (hit) estimates of the motion parameters via simulated annealing (SA). Simulation results are provided.>

Proposes a novel formulation where 3D global and local motion estimation and the adaptation of a generic wireframe model to a particular speaker are considered simultaneously within an optical flow based framework including the photometric effects of the motion. We use a flexible wireframe model whose local structure is characterized by the normal vectors of the patches which are related to the coordinates of the nodes. Geometrical constraints that describe the propagation of the movement of the nodes are introduced, which are then efficiently utilized to reduce the number of independent structure parameters. A stochastic relaxation algorithm has been used to determine optimum global motion estimates and the parameters describing the structure of the wireframe model. Results with both simulated and real facial image sequences are provided.>

In model-based coding of facial images, the accuracy of motion and depth parameter estimates strongly affects the coding efficiency. MBASIC (model-based analysis-synthesis image coding) is a simple and effective iterative algorithm recently proposed by Aizawa et el. (see Signal Processing: Image Communication, no.1, p.139-52, 1989) for 3-D motion and depth estimation when the initial depth estimates are relatively accurate. In this correspondence, we analyze its performance in the presence of errors in the initial depth estimates and propose a modification to MBASIC algorithm that significantly improves its robustness to random errors with only a small increase in the computational load.

An adaptive filter for smoothing images corrupted by signal dependent noise is presented. The filter is mainly developed for speckle suppression in medical B-scan ultrasonic imaging. The filter is based on mean filtering of the image using appropriately shaped and sized local kernels. Each filtering kernel, fitting to the local homogeneous region, is obtained through local statistics based region growing. Performance of the proposed scheme have been tested on a B-scan image of a standard tissue-mimicking ultrasound resolution phantom. The results indicate that the filter effectively reduces the speckle while preserving the resolvable details. The performance figures obtained through computer simulations on the phantom image are presented in a comparative way with some existing speckle suppression schemes.

The authors present a new technique for the display end of a holographic three-dimensional television system and describe the computer simulations. The technique is based on the reproduction of the desired pattern, in this case the hologram, using traveling surface waves. The proposed method is simpler and more efficient than the methods available in the literature and it solves the display resolution and refreshing rate problems completely. Simulations show that the proposed system will work as desired when implemented in real time.>