Multimodal streams of sensory information are naturally parsed and integrated by humans using signal-level feature extraction and higher level cognitive processes. Detection of attention-invoking audiovisual segments is formulated in this work on the basis of saliency models for the audio, visual, and textual information conveyed in a video stream. Aural or auditory saliency is assessed by cues that quantify multifrequency waveform modulations, extracted through nonlinear operators and energy tracking. Visual saliency is measured through a spatiotemporal attention model driven by intensity, color, and orientation. Textual or linguistic saliency is extracted from part-of-speech tagging on the subtitles information available with most movie distributions. The individual saliency streams, obtained from modality-depended cues, are integrated in a multimodal saliency curve, modeling the time-varying perceptual importance of the composite video stream and signifying prevailing sensory events. The multimodal saliency representation forms the basis of a generic, bottom-up video summarization algorithm. Different fusion schemes are evaluated on a movie database of multimodal saliency annotations with comparative results provided across modalities. The produced summaries, based on low-level features and content-independent fusion and selection, are of subjectively high aesthetic and informative quality.

We present a local feature detector that is able to detect regions of arbitrary scale and shape, without scale space construction. We compute a weighted distance map on image gradient, using our exact linear-time algorithm, a variant of group marching for Euclidean space. We find the weighted medial axis by extending residues, typically used in Voronoi skeletons. We decompose the medial axis into a graph representing image structure in terms of peaks and saddle points. A duality property enables reconstruction of regions using the same marching method. We greedily group regions taking both contrast and shape into account. On the way, we select regions according to our shape fragmentation factor, favoring those well enclosed by boundaries-even incomplete. We achieve state of the art performance in matching and retrieval experiments with reduced memory and computational requirements.

Several spatiotemporal feature point detectors have been used in video analysis for action recognition. Feature points are detected using a number of measures, namely saliency, cornerness, periodicity, motion activity etc. Each of these measures is usually intensity-based and provides a different trade-off between density and informativeness. In this paper, we use saliency for feature point detection in videos and incorporate color and motion apart from intensity. Our method uses a multi-scale volumetric representation of the video and involves spatiotemporal operations at the voxel level. Saliency is computed by a global minimization process constrained by pure volumetric constraints, each of them being related to an informative visual aspect, namely spatial proximity, scale and feature similarity (intensity, color, motion). Points are selected as the extrema of the saliency response and prove to balance well between density and informativeness. We provide an intuitive view of the detected points and visual comparisons against state-of-the-art space-time detectors. Our detector outperforms them on the KTH dataset using nearest-neighbor classifiers and ranks among the top using different classification frameworks. Statistics and comparisons are also performed on the more difficult Hollywood human actions (HOHA) dataset increasing the performance compared to current published results.

Detection of perceptually important video events is formulated here on the basis of saliency models for the audio, visual and textual information conveyed in a video stream. Audio saliency is assessed by cues that quantify multifrequency waveform modulations, extracted through nonlinear operators and energy tracking. Visual saliency is measured through a spatiotemporal attention model driven by intensity, color and motion. Text saliency is extracted from part-of-speech tagging on the subtitles information available with most movie distributions. The various modality curves are integrated in a single attention curve, where the presence of an event may be signified in one or multiple domains. This multimodal saliency curve is the basis of a bottom-up video summarization algorithm, that refines results from unimodal or audiovisual-based skimming. The algorithm performs favorably for video summarization in terms of informativeness and enjoyability.

Based on perceptual and computational attention modeling studies, we formulate measures of saliency for an audiovisual stream. Audio saliency is captured by signal modulations and related multi-frequency band features, extracted through nonlinear operators and energy tracking. Visual saliency is measured by means of a spatiotemporal attention model driven by various feature cues (intensity, color, motion). Audio and video curves are integrated in a single attention curve, where events may be enhanced, suppressed or vanished. The presence of salient events is signified on this audiovisual curve by geometrical features such as local extrema, sharp transition points and level sets. An audiovisual saliency-based movie summarization algorithm is proposed and evaluated. The algorithm is shown to perform very well in terms of summary informativeness and enjoyability for movie clips of various genres.

This paper describessalienShrink, a method to denoise images based on computing a map of salient coefficients in the wavelet domain and use it to improve common denoising algorithms. By salient, we refer to those coefficients that correspond mostly to pure signal and should therefore be preserved throughout the denoising procedure. We use a computationally efficient model to detect salient regions in the bands of the multiresolution wavelet transform. These regions are used to obtain a more accurate estimate of the noise level, improving the performance of existing well known shrinkage methods. Extensive experimental results on theBiShrinkmethod show that the proposed method effectively enhances PSNR and improves the visual quality of the denoised images.

A variety of approaches have been proposed in the literature for region-of-interest (ROI) estimation. In most of them the ROI definition is highly subjective, i.e., lacks scientific evidence in supporting the claim that the areas defined as ROIs are indeed perceptually interesting. In this paper we attempt to model ROIs as the visually attended areas indicated by a saliency map in order to lower as much as possible the subjectivity of selection. For evaluation purposes we follow a ROI-based video compression setup and present comparisons with state-of-the-art algorithms in terms of perceived visual quality and video compression improvement. Extended experiments concerning both MPEG-1 as well as low bit-rate MPEG-4 video encoding were conducted showing significant improvement in video compression efficiency without perceived deterioration in visual quality.

A saliency-based method for generating video summaries is presented, which exploits coupled audiovisual information from both media streams. Efficient and advanced speech and image processing algorithms to detect key frames that are acoustically and visually salient are used. Promising results are shown from experiments on a movie database.

In this paper we propose a novel saliency-based computational model for visual attention. This model processes both top-down (goal directed) and bottom-up information. Processing in the top-down channel creates the so called skin conspicuity map and emulates the visual search for human faces performed by humans. This is clearly a goal directed task but is generic enough to be context independent. Processing in the bottom-up information channel follows the principles set by Itti et al. but it deviates from them by computing the orientation, intensity and color conspicuity maps within a unified multi-resolution framework based on wavelet subband analysis. In particular, we apply a wavelet based approach for efficient computation of the topographic feature maps. Given that wavelets and multiresolution theory are naturally connected the usage of wavelet decomposition for mimicking the center surround process in humans is an obvious choice. However, our implementation goes further. We utilize the wavelet decomposition for inline computation of the features (such as orientation angles) that are used to create the topographic feature maps. The bottom-up topographic feature maps and the top-down skin conspicuity map are then combined through a sigmoid function to produce the final saliency map. A prototype of the proposed model was realized through the TMDSDMK642-0E DSP platform as an embedded system allowing real-time operation. For evaluation purposes, in terms of perceived visual quality and video compression improvement, a ROI-based video compression setup was followed. Extended experiments concerning both MPEG-1 as well as low bit-rate MPEG-4 video encoding were conducted showing significant improvement in video compression efficiency without perceived deterioration in visual quality.

In natural vision, we center our fixation on the most informative points in a scene in order to reduce our overall uncertainty about the scene and help interpret it. Even if we are looking for a specific stimulus around us, we face a great amount of uncertainty since that stimulus could be in any spatial location. Visual attention (VA) schemes have been proposed by researchers to account for the ability of the human eye to quickly fixate on informative regions. Recently, VA in images, and especially saliency-based VA, became an active research topic of the computer vision community. The proposed work provides an extension towards VA in video sequences by integrating spatiotemporal information. The potential applications include video classification, scene understanding, surveillance and segmentation

Motion information is essential in many computer vision and video analysis tasks. Since MPEG is still one of the most prevalent formats for representing, transferring and storing video data, the analysis of its motion field is important for real time video indexing and segmentation, event analysis and surveillance applications. Our work considers the problem of improving the optical flow field in MPEG sequences. We address the issues of robust, incremental, dense optical flow estimation by combining information from two different velocity fields, the available MPEG motion field and the one inferred by a multiresolution robust regularization technique applied on the DC coefficients. Thus, the regularization technique is based only on information that is directly available in the compressed stream, therefore avoiding time and memory consuming decompression. We extend standard techniques by adding a temporal continuity and an MPEG consistency constraint, both as mathematical constraints in the objective function and as hypothesis tests for the presence of motion discontinuities. Our approach is shown to perform well over a range of different motion scenarios and can serve as a basis for efficient video analysis tasks.

Palimpsests -twice written manuscripts- are of great interest since they may contain important hidden text underneath the visible one. Hyperspectral imaging may aid the expert to read the old script by enhancing the contrast between the under- and -overwriting. We present a hyperspectral imager (MUSIS), capable of acquiring 34 calibrated spectral bands in the range of 360-1150 nm (extended to 1550 nm when coupled with a photocathode tube). The potential of spectral imaging to improve the readability of manuscripts by employing common spectral analysis techniques is explored. Results on different manuscripts obtained by principal component analysis (PCA) and linear spectral mixture analysis (LSMA) prove that hyperspectral imaging tools has the potential to become an indispensable tool for the analysis of old manuscripts.

Bottom up approaches to visual attention (VA) have been applied successfully in a variety of applications, where no domain information exists, e.g. general purpose image and video segmentation. In face detection, humans perform conscious search; therefore, bottom up approaches are not so efficient. In this paper we introduce the inclusion of two channels in the VA architecture proposed by Itti et al. (1998) to account for motion and conscious search in a scene. Increasing the channels in the architecture requires an efficient way of combining the various maps that are produced. We solve this problem by using an innovative committee machine scheme which allows for dynamically changing the committee members (maps) and weighting the maps according to the confidence level of their estimation. The overall VA architecture achieves significantly better results compared with the simple skin based face detection as shown in the experimental results.

Several visual attention (VA) schemes have been proposed with the saliency-based ones being the most popular. The proposed work provides an extension towards VA in video sequences by incorporating the temporal dimension. The architecture is presented in detail and potential applications are investigated. We expect that the extended VA scheme can reveal interesting events across the sequence like occlusions and short occurrences of objects, providing a basis for video surveillance (e.g. intruder detection), segmentation, and summarization applications.

Efficient moving object tracking in cases of partial occlusion is a challenging task for the researchers in the fields of computer vision and video processing. In modern coding standards, like MPEG-4 and MPEG-7, the term of video objects is used to define moving objects in a video sequence. Automatic extraction of such objects is by no means trivial, and occlusion is one of the most important problems. This paper reformulates one of the most popular deformable templates for shape modeling and object tracking, the Snakes, in a probabilistic manner, in order to include providence for partial occlusion of the moving objects. Experiments of object tracking in partial occlusion, in complex natural sequences, where temporal clutter, abrupt motion and external lightning changes have been carried out, showing the efficiency of the proposed approach.

Assessment of the risk for the development of age related macular degeneration requires reliable detection of retinal abnormalities that are considered as precursors of the disease. A typical sign for the latter are the so-called drusen, which appear as abnormal white-yellow deposits on the retina. This paper presents a novel segmentation algorithm for automatic detection of abnormalities in images of the human eye's retina, acquired from a depth-vision camera. Conventional image processing techniques are sensitive to non-uniform illumination and nonhomogeneous background, which obstructs the derivation of reliable results for a large set of different images. Homomorphic filtering and a multilevel variant of histogram equalization are used for non-uniform illumination compensation and enhancement. We develop a novel segmentation technique, the histogram-teased adaptive local thresholding (HALT), to detect drusen in retina images by extracting the useful information without being affected by the presence of other structures. We provide experimental results from the application of our technique to real images, where certain abnormalities (drusen) have slightly different characteristics from the background and are hard to be segmented by other conventional techniques.