In this work, we present a probabilistic information fusion approach for the diagnosis of dementia from cross-sectional magnetic resonance (MR) images. The approach relies on first mapping the outputs of a support vector classifier (SVM) trained on image features to probabilities and then on combining these probabilities with the class-conditional distributions of neuropsychiatric test scores, such as the mini-mental state examination (MMSE). The SVM classifier is trained and tested on 121 subjects drawn from the Open Access Series of Imaging Studies (OASIS) database. Two independent sets of MMSE related statistics are estimated from data, one from the training set in OASIS and the other from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. The probabilistic fusion of image-based SVM decisions with no visual MMSE information exhibits very steep receiver operating characteristic curves on the test set; giving, at the equal error rate operating point, 92% accuracy.

Segmentation of brain structures from MR images is crucial in understanding the disease progress, diagnosis, and treatment monitoring. Atlases, showing the expected locations of the structures, are commonly used to start and guide the segmentation process. In many cases, the quality of the atlas may have a significant effect in the final result. In the literature, commonly used atlases may be obtained from one subject's data, only from the healthy, or depict only certain structures that limit their accuracy. Anatomical variations, pathologies, imaging artifacts all could aggravate the problems related to application of atlases. In this paper, we propose to use multiple atlases that are sufficiently different from each other as much as possible to handle such problems. To this effect, we have built a library of atlases and computed their similarity values to each other. Our study showed that the existing atlases have varying levels of similarity for different structures.

The aging population and the growing amount of medical data have increased the need for automated tools in the neurology departments. Although the researchers have been developing computerized methods to help the medical expert, these efforts have primarily emphasized to improve the effectiveness in single patient data, such as computing a brain lesion size. However, patient-to-patient comparison that should help improve diagnosis and therapy has not received much attention. To this effect, this paper introduces a fast and robust region-of-interest retrieval method for brain MR images. We make the following various contributions to the domains of brain MR image analysis, and search and retrieval system: 1) we show the potential and robustness of local structure information in the search and retrieval of brain MR images; 2) we provide analysis of two complementary features, local binary patterns (LBPs) and Kanade-Lucas-Tomasi feature points, and their comparison with a baseline method; 3) we show that incorporating spatial context in the features substantially improves accuracy; and 4) we automatically extract dominant LBPs and demonstrate their effectiveness relative to the conventional LBP approach. Comprehensive experiments on real and simulated datasets revealed that dominant LBPs with spatial context is robust to geometric deformations and intensity variations, and have high accuracy and speed even in pathological cases. The proposed method can not only aid the medical expert in disease diagnosis, or be used in scout (localizer) scans for optimization of acquisition parameters, but also supports low-power handheld devices.

This paper presents a new active contour-based, statistical method for simultaneous volumetric segmentation of multiple subcortical structures in the brain. In biological tissues, such as the human brain, neighboring structures exhibit co-dependencies which can aid in segmentation, if properly analyzed and modeled. Motivated by this observation, we formulate the segmentation problem as a maximum a posteriori estimation problem, in which we incorporate statistical prior models on the shapes and intershape (relative) poses of the structures of interest. This provides a principled mechanism to bring high level information about the shapes and the relationships of anatomical structures into the segmentation problem. For learning the prior densities we use a nonparametric multivariate kernel density estimation framework. We combine these priors with data in a variational framework and develop an active contour-based iterative segmentation algorithm. We test our method on the problem of volumetric segmentation of basal ganglia structures in magnetic resonance images. We present a set of 2-D and 3-D experiments as well as a quantitative performance analysis. In addition, we perform a comparison to several existent segmentation methods and demonstrate the improvements provided by our approach in terms of segmentation accuracy.

In the medical domain, experts usually look at specific anatomical structures to identify the cause of a pathology, and therefore they can largely benefit from automated tools that retrieve relevant slice(s) from a patient's image volume in diagnosis. Accordingly, this paper introduces a novel search and retrieval work for finding relevant slices in brain MR (magnetic resonance) volumes. As intensity is non-standard in MR we explore performance of two complementary intensity invariant features, local binary patterns and Kanade-Lucas-Tomasi feature points, their extended versions with spatial context, and a simple edge descriptor with spatial context. Experiments on real and simulated data showed that the local binary patterns with spatial context is fast, highly accurate, and robust to geometric deformations and intensity variations.

One of the hallmarks of Alzheimer's disease (AD) is the loss of neurons in the brain. In many cases, the medical experts use MR (magnetic resonance) images to qualitatively measure the neuronal loss by the shrinkage (atrophy) of the structures-of-interest, or sometimes more easily by the enlargement of the fluid-filled structures, such as the ventricles. For quantitative analysis, volume is the common choice. Volume, or area in 2D, is a gross measure and it cannot capture shape differences that can improve the diagnostic accuracy. Because most existing methods use complex and difficult- to-reproduce shape descriptors, the experts prefer more easily and robustly extractable area and volume in their diagnosis. In this paper, we introduce several novel and easily-extractable 2D shape descriptors for brain ventricle, and show that they and some of the well-known simple shape descriptors, such as perimeter, are better descriptors in the classification of AD patients and healthy controls.

When implemented in hardware, image-processing algorithms should be robust to memory limitations because some hardware architectures may not have memory size as large as the whole frame size. Although this is not generally a problem for low-level processing, higher-level understanding, such as object detection, demands novel solutions because the available information may, in some cases, be very local, e.g., only a partial view of the object could fit in the available memory size. In this paper, we propose a novel hardware-oriented overlaid text detection algorithm that can detect text with height as large as five times the memory size. The algorithm integrates a connected component (CC)-based algorithm with a texture-based machine learning approach. The CC-based algorithm uses character-level features in the horizontal direction whereas the texture-based algorithm extracts block-based features to integrate information from all directions. Furthermore, the texture-based algorithm employs a support vector machine (SVM) to benefit from the strength of machine learning tools. In order to detect text of large font size, we also propose a novel hardware-oriented, height-preserving multi-resolution analysis. Finally, the results of the two classifiers as well as color and edge cues are used for the final pixel-based text/non-text decision

Popular clustering algorithms, such as K-means (KM) and expectation maximization (EM), are sensitive to the initialization of cluster centers. In contrast, recently proposed K-harmonic means (KHM) algorithm is more robust to the randomness of the initialization. However, KHM works best when the dimensionality of the data (N) is small (usually less than 8). Because the dimensionality of features that are used for many clustering problems in image/video and speech processing is large, the benefits of KHM cannot be exploited. Based upon this observation, this paper proposes a novel method to employ KHM for high-dimensional data so as to realize initialization-independent clustering. The proposed method employs efficient spectral clustering techniques whereby the affinity matrix of the data is decomposed into its eigenvectors and k (total number of clusters) eigenvectors corresponding to the k largest eigenvalues are retained. That is, we represent N-D data by k-D transformed data when k < N and propose to employ KHM over this k-D transformed data. The assumption of k < N indeed encompasses a large number of significant video processing and computer vision problems where the use of KHM was not beneficial before. We demonstrate the effectiveness and the efficiency of the proposed algorithm for face clustering in the domains where the number of persons (k) is not large, such as anchorperson grouping, video-based speaker clustering in videoconferencing, and identity-based tracking in small office environments.

Video processing and computer vision communities usually employ shot-based or object-based structural video models and associate low-level (color, texture, shape, and motion) and semantic descriptions (textual annotations) with these structural (syntactic) elements. Database and information retrieval communities, on the other hand, employ entity-relation or object-oriented models to model the semantics of multimedia documents. This paper proposes a new generic integrated semantic-syntactic video model to include all of these elements within a single framework to enable structured video search and browsing combining textual and low-level descriptors. The proposed model includes semantic entities (video objects and events) and the relations between them. We introduce a new "actor" entity to enable grouping of object roles in specific events. This context-dependent classification of attributes of an object allows for more efficient browsing and retrieval. The model also allows for decomposition of events into elementary motion units and elementary reaction/interaction units in order to access mid-level semantics and low-level video features. The instantiations of the model are expressed as graphs. Users can formulate flexible queries that can be translated into such graphs. Alternatively, users can input query graphs by editing an abstract model (model template). Search and retrieval is accomplished by matching the query graph with those instantiated models in the database. Examples and experimental results are provided to demonstrate the effectiveness of the proposed integrated modeling and querying framework.

This paper introduces a novel generic framework for sports video processing by using the common feature of most sports: the dominant color of the field. This dominant field color is automatically detected and updated to compensate for the lighting and weather changes by a robust dominant color region detection algorithm. We also introduce new shot type classification algorithms for soccer and basketball. Finally, we use shot-based low-level features for domain-specific high-level applications. Specifically, the system detects soccer goals and summarizes soccer games in real-time, and it enables basketball fans to skip fouls, free throws, and time-out events by segmenting a basketball game into plays and breaks.

This paper proposes a novel automatic dominant color region detection algorithm that is robust to temporal variations in the dominant color due to field, weather, and lighting conditions throughout a sports video. The algorithm automatically learns the dominant color statistics of the field independent of the sports type, and updates color statistics throughout a sporting event by using two color spaces, a control space and a primary space. The robustness of the algorithm results from adaptation of the statistics of the dominant color in the primary space with drift protection using the control space, and fusion of the information from two spaces. We also propose novel and generic color-based algorithms for referee, player-of-interest, and play-break event detection in sports video. The efficiency of the proposed algorithms is demonstrated over a dataset of various sports video, including basketball, football, golf, and soccer video.

This paper proposes a single generic real-time (or near real-time) play-break event detection algorithm for multiple sports, which include football, tennis, basketball, and soccer. The proposed algorithm only uses shot-based generic cinematic features, such as shot type and shot length. Detected play-break events are employed for two purposes: 1) all plays in certain sports, such as football and tennis, are presented as summaries, and 2) play-break events, as part of a hierarchical event detection scheme, determine the segments-of-interest for the other event detection algorithms. An example of such event detection algorithms is given for soccer goal events, where the proposed soccer goal detection algorithm exploits the common cinematic techniques that are employed during the breaks that follow the goal plays. We demonstrate the genericity of the proposed play-break detection algorithm over football, tennis, basketball video and the effectiveness of the proposed soccer goal detection algorithm over a large data set.

We propose a fully automatic and computationally efficient framework for analysis and summarization of soccer videos using cinematic and object-based features. The proposed framework includes some novel low-level processing algorithms, such as dominant color region detection, robust shot boundary detection, and shot classification, as well as some higher-level algorithms for goal detection, referee detection, and penalty-box detection. The system can output three types of summaries: i) all slow-motion segments in a game; ii) all goals in a game; iii) slow-motion segments classified according to object-based features. The first two types of summaries are based on cinematic features only for speedy processing, while the summaries of the last type contain higher-level semantics. The proposed framework is efficient, effective, and robust. It is efficient in the sense that there is no need to compute object-based features when cinematic features are sufficient for the detection of certain events, e.g., goals in soccer. It is effective in the sense that the framework can also employ object-based features when needed to increase accuracy (at the expense of more computation). The efficiency, effectiveness, and robustness of the proposed framework are demonstrated over a large data set, consisting of more than 13 hours of soccer video, captured in different countries and under different conditions.

In this paper, we present the tools standardized by MPEG-7 for describing the semantics of multimedia. In particular, we focus on the abstraction model, entities, attributes and relations of MPEG-7 semantic descriptions. MPEG-7 tools can describe the semantics of specific instances of multimedia such as one image or one video segment but can also generalize these descriptions either to multiple instances of multimedia or to a set of semantic descriptions. The key components of MPEG-7 semantic descriptions are semantic entities such as objects and events, attributes of these entities such as labels and properties, and, finally, relations of these entities such as an object being the patient of an event. The descriptive power and usability of these tools has been demonstrated in numerous experiments and applications, these make them key candidates to enable intelligent applications that deal with multimedia at human levels.

This paper proposes to integrate text-based database models (commonly used in the database and information retrieval literature) and low-level video features, such as object-based motion features (commonly used in image processing and computer vision), within a single framework to describe video events for fast and effective browsing and retrieval using a mixture of textual and low-level descriptors. The instantiations of the model are expressed as graphs. Users can formulate flexible queries that can be translated into such graphs. Alternatively, users can input query graphs by editing an abstract model (model template) derived either from a model database or from an example scene in the media database. Search and retrieval is accomplished by matching the query graph with those instantiated models in the database. The proposed approach allows for integrated textual and low-level, descriptor-based video search, where the integration is enabled by the model. Examples and experimental results are provided to demonstrate the effectiveness of the proposed integrated modeling and querying framework.

This paper describes a hierarchical approach for object-based motion description of video in terms of object motions and object-to-object interactions. We present a temporal hierarchy for object motion description, which consists of low-level elementary motion units (EMU) and high-level action units (AU). Likewise, object-to-object interactions are decomposed into a hierarchy of low-level elementary reaction units (ERU) and high-level interaction units (IU). We then propose an algorithm for temporal segmentation of video objects into EMUs, whose dominant motion can be described by a single representative parametric model. The algorithm also computes a representative (dominant) affine model for each EMU. We also provide algorithms for identification of ERUs and for classification of the type of ERUs. Experimental results demonstrate that segmenting the life-span of video objects into EMUS and ERUs facilitates the generation of high-level visual summaries for fast browsing and navigation. At present, the formation of high-level action and interaction units is done interactively. We also provide a set of query-by-example results for low-level EMU retrieval from a database based on similarity of the representative dominant affine models.

We use Bayesian belief networks to statistically model the trends for event detection. We automatically detect non-rigid object trajectories for object motion units. Then, we use dominant and secondary trajectories of a single object in several consecutive motion units to understand semantic actions or those of more than one object to recognize semantic interactions between objects. We demonstrate sample Bayesian networks to detect events and extract the event descriptions, such as "catch the ball", "throw the ball" and "walk".

In our previous work, we developed a method to extract low-level elementary motion units (EMU) and elementary reaction units (ERU) for object-based event description, assuming perfect video object segmentation and tracking. This paper features the following contributions. (1) We propose a novel object tracking algorithm for a specific domain. (2) We evaluate the performance of our EMU and ERU extraction system with automatically-computed but imperfect video object information obtained by the proposed tracker. It is assumed that objects for which descriptors are sought for are interactively marked on the initial frame. (3) We extend the description of ERUs to enable better description of low-level reactions. Experimental results are provided to demonstrate each contribution.

Large scale deployment of digital multimedia applications requires effective and efficient image and video representations for indexing. We present a hierarchical representation of video objects based on their motion. The proposed representation enables both low and semantic level description of object motion. At the low level, we consider a parametric description of the dominant motion of video objects, and define elementary motion units (EMU) where the dominant motion of the object is coherent and does not undergo considerable change. A group of EMUs forms an action unit which carries semantic information. Interactions between objects are also considered by calculating the relative motion between objects in an object-based framework. Experimental results for retrieval of EMUs based on similarity of dominant object motion are demonstrated.