It is well-known that severe pain and powerful pain medications cause short- and long-term damage to the developing nervous system of newborns. Caregivers routinely use physiological vital signs [Heart Rate (HR), Respiration Rate (RR), Oxygen Saturation (SR)] to monitor post-surgical pain in the Neonatal Intensive Care Unit (NICU). Here we present a novel approach that combines continuous, non-invasive monitoring of these vital signs and Computer Vision/Deep Learning to make automatic neonate pain detection with an accuracy of 74% AUC, 67.59% mAP. Further, we report for the first time our Early Pain Detection (EPD) approach that explores prediction of the time to onset of post-surgical pain in neonates. Our EPD can alert NICU workers to postoperative neonatal pain about 5 to 10 minutes prior to pain onset. In addition to alleviating the need for intermittent pain assessments by busy NICU nurses via long-term observation, our EPD approach creates a time window prior to pain onset for the use of less harmful pain mitigation strategies. Through effective pain mitigation prior to spinal sensitization, EPD could minimize or eliminate severe post-surgical pain and the consequential need for powerful analgesics in post-surgical neonates.

Accurately detecting pain in infants remains a complex challenge. Conventional deep neural networks used for analyzing infant cry sounds typically demand large labeled datasets, substantial computational power, and often lack interpretability. In this work, we introduce a novel approach that leverages OpenAI's vision-language model, GPT-4(V), combined with mel spectrogram-based representations of infant cries through prompting. This prompting strategy significantly reduces the dependence on large training datasets while enhancing transparency and interpretability. Using the USF-MNPAD-II dataset, our method achieves an accuracy of 83.33% with only 16 training samples, in contrast to the 4,914 samples required in the baseline model. To our knowledge, this represents the first application of few-shot prompting with vision-language models such as GPT-4o for infant pain classification.

Although concept-based approaches are widely used to explain a model's behavior and assess the contributions of different concepts in decision-making, identifying relevant concepts can be challenging for non-experts. This paper introduces a novel method that simplifies concept selection by leveraging the capabilities of a state-of-the-art large Vision-Language Model (VLM). Our method employs a VLM to select textual concepts that describe the classes in the target dataset. We then transform these influential textual concepts into human-readable image concepts using a text-to-image model. This process allows us to explain the targeted network in a post-hoc manner. Further, we use directional derivatives and concept activation vectors to quantify the importance of the generated concepts. We evaluate our method on a neonatal pain classification task, analyzing the sensitivity of the model's output for the generated concepts. The results demonstrate that the VLM not only generates coherent and meaningful concepts that are easily understandable by non-experts but also achieves performance comparable to that of natural image concepts without the need for additional annotation costs.

Current AI-based methods for the classification of cellular features in microscopy images require time- and labor-intensive processes for training models. Specific limitations include the need for a large amount of image data and major time commitments from domain experts for accurate ground truthing. We present a solution for overcoming these limitations using a state-of-the-art vision language model. Our approach uses GPT-4, the Vision Language Model (VLM) from OpenAI, for the analysis and classification of Iba-1 immuno-stained microglia cells in tissue sections through the mouse hippocampus. We used GPT-4 to classify a dataset of low-power (20x) images of Iba-1 immuno-stained microglia cells from tissue sections treated with saline or a potent neurotoxin (tri-methyl-tin, TMT). Rather than training with images from each class, the GPT-4 input consists of minimal ground-truth prompts for visual question answering. We introduce a novel human-in-the-loop approach to automate the selection of example image-text pairs as input prompts and generate explanatory text as the basis for separating images into distinct classes. We assess test accuracy and efficiency compared to the baseline results using a convolutional neural net applied to the same dataset. Compared to the baseline, the equivalence in accuracy (91%) and substantial (86%) improvement in throughput efficiency with considerably lower needs for input data or domain experts highlight the effectiveness of our new method for automatic image classification. Unlike traditional methods focused on labeling images with one-to-two-word tags, our pipeline generates understandable ground truth by incorporating explanatory text for each image.

Background: The need for sharing large-scale datasets in training deep learning models, particularly in healthcare, raises significant data security and privacy concerns. To address these issues, methods such as data encryption or encoding are utilized. These techniques can encrypt the data and make it unreadable to humans, while still retaining its usefulness for training models.Method: In this study, we investigate various image encoding techniques designed to protect privacy by making images unrecognizable while still retaining their usefulness for model training. Our investigation utilized a publicly available facial database and focused on evaluating the trade-offs inherent in image encoding techniques, with a special emphasis on balancing privacy and model accuracy.Conclusion: This study navigate the balance between protecting sensitive data and meeting the data demands necessary for effective model training. It sheds light on the intricate trade-offs among different image encoding techniques and offers insights into finding an optimal balance between privacy protection and model performance.

The detection and segmentation of stained cells and nuclei are essential prerequisites for subsequent quantitative research for many diseases. Recently, deep learning has shown strong performance in many computer vision problems, including solutions for medical image analysis. Furthermore, accurate stereological quantification of microscopic structures in stained tissue sections plays a critical role in understanding human diseases and developing safe and effective treatments. In this article, we review the most recent deep learning approaches for cell (nuclei) detection and segmentation in cancer and Alzheimer's disease with an emphasis on deep learning approaches combined with unbiased stereology. Major challenges include accurate and reproducible cell detection and segmentation of microscopic images from stained sections. Finally, we discuss potential improvements and future trends in deep learning applied to cell detection and segmentation.

We propose a novel unsupervised meta-learning based segmentation algorithm for histopathology images. The proposed algorithm does not require any kind of patch-level annotations and relies solely on image labels, corresponding to any classification task, and direct feedback from a classifier. Furthermore, instead of simply segmenting histopathology images into different types of tissue, our algorithm determines the relative importance of each tissue region. After thresholding, the produced segmentations can also be used as regions of interest for various machine learning based diagnosis systems. We have tested our approach on Prostate cANcer graDe Assessment (PANDA) dataset and obtained 0.79 AUC, when testing the segmentation performance at patch-level, and 0.432 Dice coefficient, when testing precise segmentation, which is comparable to 0.446, described in related work which performed a supervised segmentation with U-Net. Note that no pixel level annotations were used.

Deep Learning (DL)-based solutions have shown promising performance in assessing neonatal pain. However, the occlusion of the visual modality (face and body) is common in clinical settings due to several factors, including a prone sleeping position, low light, or swaddling. In such scenarios, other pain signals, such as audio signals, can be used as the major behavioral signs of pain. Although DL-based methods are proposed to assess pain from audio, these methods lack transparency and explainability (black box), which can decrease the user's trust in the automated decision. In this work, we visualize the neonate's audio signal as a spectrogram image to classify it as pain or no pain and present an instance-based approach for explaining the decision of the black-box model. Further, this work provides an analysis of the most helpful and harmful training instances using an influence score followed by assessing their impact on pain prediction. Experimental results demonstrate that the proposed approach can detect and remove harmful instances, eventually leading to a compressed dataset. Our results also show that the proposed work can add explainability to the current DL-based pain detection methods, which can enhance users' trust and provide a viable approach toward pain assessment in clinical settings.

Across basic research studies, cell counting requires significant human time and expertise. Trained experts use thin focal plane scanning to count (click) cells in stained biological tissue. This computer-assisted process (optical disector) requires a well-trained human to select a unique best z-plane of focus for counting cells of interest. Though accurate, this approach typically requires an hour per case and is prone to inter-and intra-rater errors. Our group has previously proposed deep learning (DL)-based methods to automate these counts using cell segmentation at high magnification. Here we propose a novel You Only Look Once (YOLO) model that performs cell detection on multi-channel z-plane images (disector stack). This automated Multiple Input Multiple Output (MIMO) version of the optical disector method uses an entire z-stack of microscopy images as its input, and outputs cell detections (counts) with a bounding box of each cell and class corresponding to the z-plane where the cell appears in best focus. Compared to the previous segmentation methods, the proposed method does not require time-and labor-intensive ground truth segmentation masks for training, while producing comparable accuracy to current segmentation-based automatic counts. The MIMO-YOLO method was evaluated on systematic-random samples of NeuN-stained tissue sections through the neocortex of mouse brains (n=7). Using a cross validation scheme, this method showed the ability to correctly count total neuron numbers with accuracy close to human experts and with 100% repeatability (Test-Retest).

Infants receiving care in the Neonatal Intensive Care Unit (NICU) experience several painful procedures during their hospitalization. Assessing neonatal pain is difficult because the current standard for assessment is subjective, inconsistent, and discontinuous. The intermittent and inconsistent assessment can induce poor treatment and, therefore, cause serious long-term outcomes. In this paper, we present a comprehensive pain assessment system that utilizes facial expressions along with crying sounds, body movement, and vital sign changes. The proposed automatic system generates a standardized pain assessment comparable to those obtained by conventional nurse-derived pain scores. The system achieved 95.56 percent accuracy using decision fusion of different pain responses that were recorded in a challenging clinical environment. In addition to the decision fusion, we present the performance of multimodal assessment using other fusion schemes as well as a unimodal assessment approach. We also discuss the impact of different factors (e.g., gestational age) on pain, propose several group-specific models for pain assessment (e.g., pre-term and full-term models), and compare the performance of these models with the performance of general models. While further research is needed, our results show that the automatic assessment of neonatal pain is a viable and more efficient alternative to the manual assessment.

Spectrograms visualize the frequency components of a given signal which may be an audio signal or even a time-series signal. Audio signals have higher sampling rate and high variability of frequency with time. Spectrograms can capture such variations well. But, vital signs which are time-series signals have less sampling frequency and low-frequency variability due to which, spectrograms fail to express variations and patterns. In this paper, we propose a novel solution to introduce frequency variability using frequency modulation on vital signs. Then we apply spectrograms on frequency modulated signals to capture the patterns. The proposed approach has been evaluated on 4 different medical datasets across both prediction and classification tasks. Significant results are found showing the efficacy of the approach for vital sign signals. The results from the proposed approach are promising with an accuracy of 91.55% and 91.67% in prediction and classification tasks respectively.

Lung cancer is by far the leading cause of cancer death in the US. Recent studies have demonstrated the effectiveness of screening using low dose CT (LDCT) in reducing lung cancer related mortality. While lung nodules are detected with a high rate of sensitivity, this exam has a low specificity rate and it is still difficult to separate benign and malignant lesions. The ISBI 2018 Lung Nodule Malignancy Prediction Challenge, developed by a team from the Quantitative Imaging Network of the National Cancer Institute, was focused on the prediction of lung nodule malignancy from two sequential LDCT screening exams using automated (non-manual) algorithms. We curated a cohort of 100 subjects who participated in the National Lung Screening Trial and had established pathological diagnoses. Data from 30 subjects were randomly selected for training and the remaining was used for testing. Participants were evaluated based on the area under the receiver operating characteristic curve (AUC) of nodule-wise malignancy scores generated by their algorithms on the test set. The challenge had 17 participants, with 11 teams submitting reports with method description, mandated by the challenge rules. Participants used quantitative methods, resulting in a reporting test AUC ranging from 0.698 to 0.913. The top five contestants used deep learning approaches, reporting an AUC between 0.87 - 0.91. The team's predictor did not achieve significant differences from each other nor from a volume change estimate (p =.05 with Bonferroni-Holm's correction).

This paper presents the first investigation into the use of fully automated deep learning framework for assessing neonatal postoperative pain. It specifically investigates the use of Bilinear Convolutional Neural Network (B-CNN) to extract facial features during different levels of postoperative pain followed by modeling the temporal pattern using Recurrent Neural Network (RNN). Although acute and postoperative pain have some common characteristics (e.g., visual action units), postoperative pain has a different dynamic, and it evolves in a unique pattern over time. Our experimental results indicate a clear difference between the pattern of acute and postoperative pain. They also suggest the efficiency of using a combination of bilinear CNN with RNN model for the continuous assessment of postoperative pain intensity.

This paper introduces a game-based framework to compare the usability of authentication methods. The framework uses a dual-task interference technique to determine the usability of authentication methods. In the experiment, subjects participate in a multi-tasking game that simulates a conversation being interrupted by authentication requirements. By simulating a conversation, the goal is to reproduce a real use of authentication, and collect ecologically sound data. Participants also perform each authentication method in a standalone manner, which allows for comparison of the usability under two different cognitive loads. The authentication techniques evaluated represent each of the three main authentication factors, specifically password, fingerprint, and coauthentication. The three aspects of usability used to compare authentication techniques in this framework are efficiency, effectiveness, and satisfaction. An experiment with 43 participants enrolled was conducted to collect data pertaining to these aspects. The results show that fingerprint and coauthentication (both laptop and phone) are the more usable techniques evaluated.

The current standard for assessing neonatal pain is discontinuous and suffers from inter-observer variations, which can result in delayed intervention and inconsistent treatment of pain. Therefore, it is critical to address the shortcomings of the current standard and develop continuous and less subjective pain assessment tools. Convolutional Neural Networks have gained much popularity in the last decades due to the wide range of its successful applications in medical image analysis, object recognition, and emotion recognition. In this paper, we propose a Neonatal Convolutional Neural Network, designed and trained end-to-end to detect neonatal pain. We evaluated the proposed network in two data sets of neonates and compared its performance to the performance of ResNet architecture in the same data sets. Our proposed method outperformed ResNet in recognizing neonates' pain and achieved around 91.00% accuracy. While further research is needed, our preliminary results suggest that the presented network can be used for automatic pain assessment, and possibly similar applications. It also suggests that the automatic recognition of neonatal pain provides a viable and more efficient alternative to the current standard of pain assessment.

Here we investigate the application of convolutional neural networks (CNNs) to predict the survival time of patients with Glioblastoma Multiforme (GBM) brain tumor. Our dataset consists of T1-weighted high-resolution MRI images of just 68 GBM patients. We compare two analytic methods for predicting survival time. The first consists of training a small convolutional neural network (CNN) and the second uses extracted deep features from a pre-trained CNN. Our method is completely automated, except for tumor region segmentation. In addition, we utilize a snapshot ensemble approach to boost test accuracy when dealing with limited availability of medical images for CNN training purposes. Our approach achieves an accuracy of 72.06% using a trained small network and 66.18% using a pre-trained deep CNN. Our results compare favorably with the accuracy of 54.41% using histogram of oriented gradients (HOG) features and a non-neural network classifier.

Training deep learning models for unbiased stereology requires a large data set with associated ground truth. However manual ground truth annotation is tedious, time-consuming, and expert dependent. We propose an active deep learning method for automatic stereology counts using a snapshot ensemble approach. The method provides a confidence score for each mask in an unlabeled pool that reduces user verification to only images with high information content for training the deep learning model. The proposed method reduces the error rate to less than 1% for unbiased stereology cell counts on immunostained brain cells compared to manual stereology and requires ~25% less expert verification time compared to a previously proposed iterative deep learning approach.

Neonates do not have the ability to either articulate pain or communicate it non-verbally by pointing. The current clinical standard for assessing neonatal pain is intermittent and highly subjective. This discontinuity and subjectivity can lead to inconsistent assessment, and therefore, inadequate treatment. In this paper, we propose a multi-channel deep learning framework for assessing neonatal pain from videos. The proposed framework integrates information from two pain indicators or channels, namely facial expression and body movement, using convolutional neural network (CNN). It also integrates temporal information using a recurrent neural network (LSTM). The experimental results prove the efficiency and superiority of the proposed temporal and multi-channel framework as compared to existing similar methods.

Lack of enough labeled data is a major problem in building machine learning based models when the manual annotation (labeling) is error-prone, expensive, tedious, and time-consuming. In this paper, we introduce an iterative deep learning based method to improve segmentation and counting of cells based on unbiased stereology applied to regions of interest of extended depth of field (EDF) images. This method uses an existing machine learning algorithm called the adaptive segmentation algorithm (ASA) to generate masks (verified by a user) for EDF images to train deep learning models. Then an iterative deep learning approach is used to feed newly predicted and accepted deep learning masks/images (verified by a user) to the training set of the deep learning model. The error rate in unbiased stereology count of cells on an unseen test set reduced from about 3 % to less than 1 % after 5 iterations of the iterative deep learning based unbiased stereology process.

Lung cancer is the leading cause of cancer-related deaths globally, which makes early detection and diagnosis a high priority. Computed tomography (CT) is the method of choice for early detection and diagnosis of lung cancer. Radiomics features extracted from CT-detected lung nodules provide a good platform for early detection, diagnosis, and prognosis. In particular when using low dose CT for lung cancer screening, effective use of radiomics can yield a precise non-invasive approach to nodule tracking. Lately, with the advancement of deep learning, convolutional neural networks (CNN) are also being used to analyze lung nodules. In this study, our own trained CNNs, a pre-trained CNN and radiomics features were used for predictive analysis. Using subsets of participants from the National Lung Screening Trial, we investigated if the prediction of nodule malignancy could be further enhanced by an ensemble of classifiers using different feature sets and learning approaches. We extracted probability predictions from our different models on an unseen test set and combined them to generate better predictions. Ensembles were able to yield increased accuracy and area under the receiver operating characteristic curve (AUC). The best-known AUC of 0.96 and accuracy of 89.45% were obtained, which are significant improvements over the previous best AUC of 0.87 and accuracy of 76.79%.

Semantic features are common radiological traits used to characterize a lesion by a trained radiologist. These features have been recently formulated, quantified on a point scale in the context of lung nodules by our group. Certain radiological semantic traits have been shown to extremely predictive of malignancy [26]. Semantic traits observed by a radiologist at examination describe the nodules and the morphology of the lung nodule shape, size, border, attachment to vessel or pleural wall, location and texture etc. Deep features are numeric descriptors often obtained from a convolutional neural network (CNN) which are widely used for classification and recognition. Deep features may contain information about texture and shape, primarily. Lately, with the advancement of deep learning, convolutional neural networks (CNN) are also being used to analyze lung nodules. In this study, we relate deep features to semantic features by looking for similarity in ability to classify. Deep features were obtained using a transfer learning approach from both an ImageNet pre-trained CNN and our trained CNN architecture. We found that some of the semantic features can be represented by one or more deep features. In this process, we can infer that some deep feature(s) have similar discriminatory ability as semantic features.

Real-world datasets are often imbalanced, with an important class having many fewer examples than other classes. In medical data, normal examples typically greatly outnumber disease examples. A classifier learned from imbalanced data, will tend to be very good at the predicting examples in the larger (normal) class, yet the smaller (disease) class is typically of more interest. Imbalance is dealt with at the feature vector level (create synthetic feature vectors or discard some examples from the larger class) or by assigning differential costs to errors. Here, we introduce a novel method for over-sampling minority class examples at the image level, rather than the feature vector level. Our method was applied to the problem of Glioblastoma patient survival group prediction. Synthetic minority class examples were created by adding Gaussian noise to original medical images from the minority class. Uniform local binary patterns (LBP) histogram features were then extracted from the original and synthetic image examples with a random forests classifier. Experimental results show the new method (Image SMOTE) increased minority class predictive accuracy and also the AUC (area under the receiver operating characteristic curve), compared to using the imbalanced dataset directly or to creating synthetic feature vectors.

Mislabeled examples are difficult to avoid while building large scale datasets. In this paper we discuss an efficient approach for finding those mislabeled examples. Our approach involves selecting a small number of potentially mislabeled examples for review by an expert. We demonstrate the utility of our method by finding some mislabeled examples in one large scale dataset (ImageNet). We found 92 errors by automatically selecting 3607 examples to review out of 22951 images from 18 classes. This requires reviewing 9 times fewer examples than the random sampling method to find an equivalent number of mislabels.

Decisions about cervical cancer diagnosis and classification currently require microscopic examination of cervical tissue by an expert pathologist. In the present study, which focused on full automation of this approach, we solely use nucleus-level features to classify tissues as normal or cancer. We propose Adaptive Nucleus Shape Modeling (ANSM) algorithm for nucleus-level analysis which consists of two steps to capture the nucleus-level information: adaptive multilevel thresholding segmentation; and shape approximation by ellipse fitting. After applying the proposed algorithm, the features are extracted for tissue classification. Experiments show that ANSM can achieve an accuracy of 93.33% with a false negative rate of zero in classifying cancer and healthy cervical tissues using nucleus texture features. This provides evidence that nucleus-level analysis is valuable in cervical histology image analysis.

Current practices of assessing infants' pain depends on the observer's subjective and potentially inconsistent judgment and requires continuous monitoring by care providers. Therefore, pain may be misinterpreted or totally missed leading to misdiagnosis and over/under treatment. To address these shortcomings, current practices can be augmented with a machine-based assessment system that monitors various pain cues and provides an objective and continuous assessment of pain. Although several machine-based pain assessment approaches have been introduced, the majority of these approaches assess pain based on analysis of a single pain indicator (i.e., unimodal). In this paper, we propose an automated multimodal approach that utilizes a combination of both behavioral and physiological pain indicators to assess infants' pain. We also present a unimodal approach that depends on a single pain indicator for assessment. Recogsnizing pain using a single indicator yielded 88%, 85%, and 82% overall accuracies for facial expression, body movement, and vital signs, respectively. Combining facial expression, body movement, and changes in vital signs (i.e., the multimodal approach) for assessment achieved 95% overall accuracy. These preliminarily results indicate that utilizing both behavioral and physiological pain indicators could provide a better and more reliable assessment of infants' pain.

Finding appropriate feature representations from radiological images is a vital task for prediction and diagnosis. Deep convolutional neural networks have recently achieved state-of-the-art performance in classification problems from several different domains. Research has also shown the feasibility of using a pre-trained deep neural network as a feature extractor when only a small dataset is available. This paper proposes a novel image feature extraction method for predicting survival time from brain tumor magnetic resonance images using pretrained deep neural networks. Since all tumors are different sizes, we also explore different image resizing methods in the paper. We demonstrate that deep features can result in better survival time prediction with the highest accuracy of 95.45% versus conventional feature extraction methods from magnetic resonance images of the brain.

Computed tomography (CT) is widely used during diagnosis and treatment of Non-Small Cell Lung Cancer (NSCLC). Current computer-aided diagnosis (CAD) models, designed for the classification of malignant and benign nodules, use image features, selected by feature selectors, for making a decision. In this paper, we investigate automated selection of different image features informed by different nodule size ranges to increase the overall accuracy of the classification. The NLST dataset is one of the largest available datasets on CT screening for NSCLC. We used 261 cases as a training dataset and 237 cases as a test dataset. The nodule size, which may indicate biological variability, can vary substantially. For example, in the training set, there are nodules with a diameter of a couple millimeters up to a couple dozen millimeters. The premise is that benign and malignant nodules have different radiomic quantitative descriptors related to size. After splitting training and testing datasets into three subsets based on the longest nodule diameter (LD) parameter accuracy was improved from 74.68% to 81.01% and the AUC improved from 0.69 to 0.79. We show that if AUC is the main factor in choosing parameters then accuracy improved from 72.57% to 77.5% and AUC improved from 0.78 to 0.82. Additionally, we show the impact of an oversampling technique for the minority cancer class. In some particular cases from 0.82 to 0.87.

The goal of this paper is to show the use of data mining techniques to predict the Soft Tissue Sarcoma (STS) tumor progression. STS are cancers which occur in different parts of the body such as fat, muscle and nerves. The lack of effective treatments and the difficulty in predicting treatment response make them challenging for physicians, and has likely slowed the evolution of new therapeutic agents. To design a prediction model, we propose a novel quantitative histogram-based method to analyze the difference in histograms obtained from pre and post-treatment multi-modality magnetic resonance images. Here, we used Radiomics techniques as a non-invasive method for outcome prediction. This study could help physicians identify distinctive patterns within each tumor to find more patient-specific treatments. We demonstrated the new approach on two practical tasks: tumor recurrence prediction (metastasis) and rate of necrosis prediction. Our learned model shows 87.79% prediction accuracy for metastasis with a 0.73 AUC and 82.22% prediction accuracy for necrosis with a 0.65 AUC.

Lung cancer is caused by abnormal and uncontrolled growth of cells in the lungs and the mortality rate of lung cancer is the highest among all types of cancer. It can be identified and treated with the help of computed tomography (CT) images. For an automated classifier, identifying good features from an image is a key concern. Deep feature extraction using pre-trained convolutional neural networks has been successful for some image domains recently. In our study, we apply a pre-trained convolutional neural network (CNN) to extract deep features from lung cancer CT images and then train classifiers to predict short and long term survivors. The best accuracy of 77.5% was with a cropping approach using a decision tree classifier in a leave one out cross validation with ten features chosen using symmetric uncertainty feature ranking. We mixed extracted deep neural network features along with quantitative (traditional image) features and obtained the best accuracy of 82.5% with a nearest neighbor classifier in a leave one out cross validation using the symmetric uncertainty feature ranking algorithm.

An important problem in quantitative medical image analysis is a large number of features (often highly correlated) to instance ratio. To handle this, we developed a feature selector and an ensemble classifier based on a modified version of random subspace method. We propose using a fusion of feature selection concepts: ranking based, correlation based and random subspaces, to develop a concordance correlation coefficient based random subspace method (CCC RSM) feature selector. It forms random feature subsets with weakly correlated yet relevant features while the ensemble classification is achieved by training the base classifier with these feature subsets. Axillary lymph node (ALNs) metastases is one of the most important prognostic factors in breast cancer. We applied CCC RSM for four binary class classifications based on the number of metastatic ALNs: (i) = 1 vs 0 (ii) 1-3 vs 0 (iii) 1-3 vs = 4, and (iv) 4 vs 0. We extracted textural kinetics from habitats of fifty eight dynamic contrast enhanced magnetic resonance imaging breast tumors. We used three classifiers to compare the accuracies achieved by CCC RSM with random subspaces (RS), wrappers and correlation based feature selector (CFS). For each binary classification we achieved the best accuracy (= 78%) using CCC RSM.

Soft Tissue Sarcomas (STS) are malignant tumors which emanate from soft tissues of the body. They are challenging for physicians because of the infrequency of their occurrence and non-predictable outcomes. In this paper, we propose a novel framework to classify STS which focuses on radio logically defined sub-regions, so-called 'habitats'. The distinctive habitats are regions where tumor evolution may be observed. We assess T1 post- and pre-contrast gadolinium and T2 non-contrast Magnetic Resonance Images (MRIs) of 36 patients prior to treatment. This paper considers spatially distinct habitats, which may be helpful in clinical treatment, especially chemotherapy and radiation. Our approach contains three main steps: (1) intra-tumor segmentation into habitats based on pixel intensity, (2) texture analysis within each distinctive habitat to capture heterogeneity, and (3) prediction of metastatic and necrotic tumor. The experimental results show the individual cases were correctly classified as metastatic or non-metastatic disease with 86.11% accuracy based on 5 features and for necrosis =90% or necrosis <; 90% with 81.81% accuracy based on 4 features by using several meta-classifiers.

Lung segmentation in thoracic computed tomography (CT) scans is an important preprocessing step for computer-aided diagnosis (CAD) of lung diseases. This paper focuses on the segmentation of the lung field in thoracic CT images. Traditional lung segmentation is based on Gray level thresholding techniques, which often requires setting a threshold and is sensitive to image contrasts. In this paper, we present a fully automated method for robust and accurate lung segmentation, which includes a enhanced thresholding algorithm and a refinement scheme based on a texture-aware active contour model. In our thresholding algorithm, a histogram based image stretch technique is performed in advance to uniformly increase contrasts between areas with low Hounsfield unit (HU) values and areas with high HU in all CT images. This stretch step enables the following threshold-free segmentation, which is the Otsu algorithm with contour analysis. However, as a threshold based segmentation, it has common issues such as holes, noises and inaccurate segmentation boundaries that will cause problems in future CAD for lung disease detection. To solve these problems, a refinement technique is proposed that captures vessel structures and lung boundaries and then smooths variations via texture-aware active contour model. Experiments on 2,342 diagnosis CT images demonstrate the effectiveness of the proposed method. Performance comparison with existing methods shows the advantages of our method.

In this paper we propose a novel method for suppressing facial expressions in video sequences based on analysis of apparent strain in the face. The proposed method performs continuous optical strain analysis on a target face for all frames in a video. This analysis is used in conjunction with strain maps for every frame to counteract the elastic deformation of the facial tissue and keep the expression neutral. The method is capable of suppressing out all expression types without the need of training for specific expressions and is demonstrated on a publicly available data set. In addition, we test our method using publicly available expression (smile) recognition program that is included with OpenCV to quantify the suppression results. Our method shows an average reduction of expression detection confidence, the confidence of an expression present in a single frame, of nearly 90%.

The U.S. Defense Advanced Research Projects Agency's (DARPA) Neovision2 program aims to develop artificial vision systems based on the design principles employed by mammalian vision systems. Three such algorithms are briefly described in this paper. These neuromorphic-vision systems' performance in detecting objects in video was measured using a set of annotated clips. This paper describes the results of these evaluations including the data domains, metrics, methodologies, performance over a range of operating points and a comparison with computer vision based baseline algorithms.

Brain tumor heterogeneity is well recognized in clinical MRI imaging and it is a challenging problem to quantitatively explore the underlying variations. It is known that brain tumors in different patients can have remarkably diverse visual appearances. In this paper, we propose a novel concept to categorize brain tumors with emphasis on spatial "habitats": a tumor can be quantified into distinctive sub-regions where the potential dynamics of tumor evolution may be evident. Our work is aimed at discovering spatially distinctive habitats within the tumor region, which may be useful in clinical practice for image-guided therapy. In particular, the heterogeneity can be well captured by two main steps: (a) intra-tumor segmentation, (b) spatial mapping scheme from a multi-modality MRI imaging dataset (Tl-weighted, FLAIR and T2-weighted MRI slices). A tumor region is initially segmented into high and low signal groups and then a joint mapping scheme is used to consider the correlation between different input modalities. In addition, focusing on signal contrast, we propose a set of quantitative features to measure differences between sub-regions. We further examined the application of survival time prediction for patients with malignant Glioblastoma multiforme (GBM). Experimental results showed that these features enabled classifiers to predict survival groups.

In breast cancer, tumor heterogeneity is a reflection of differing tumor subtypes, which may display markedly different genotypes and clinical phenotypes. Although pathological and qualitative (based on contrast enhancement patterns) studies suggest the presence of clinical and molecular predictive tumor subregions, this has not been fully investigated. Our goal is to develop a novel algorithm to utilize the potential information available in different tumor subregions (periphery and core) by extracting textural kinetic features, for the purpose of estrogen receptor (ER) classification. We show that features from different tumor subregions, at appropriate scales and quantization levels, can be used to better classify ER subtypes than features averaged from the whole tumor. We analyzed representative two dimensional (2D) slices from twenty breast tumors with volumetric dynamic contrast enhanced magnetic resonance imaging (DCE-MRI) available by extracting multi-parametric textural kinetic features from the periphery, core and whole tumor. The utility of the features from different subregions are evaluated using six meta-classifiers (feature selector and classifier pairs), formed from two feature selectors and three classifiers. Classification accuracy approached 94%.

To classify cervical cells as normal or cancer, the histological image must be segmented. After segmentation mean nuclear volume can be used to distinguish between normal and cancer cells. Due to the rapid reproduction of cancer cells, they have higher mean nuclear volume than typical normal cells. We propose a large ensemble of segmentations which separate normal and cancer cases based on the single feature of mean nuclear volume. Four basic segmentors with different parameters generate the segmentations. The mean nuclear volume is extracted from the segmentations. The dataset used for this paper contained multiple images from 30 normal and 32 cancer patients. Hematoxylin and eosin (H&E) was used to stain archival tissue sections from the normal cervix and cervical cancers. Results show it is possible to predict class with greater than 84% accuracy.

Low-dose helical computed tomography (LDCT) has facilitated the early detection of lung cancer through pulmonary screening of patients. There have been a few attempts to develop a computer-aided diagnosis system for classifying pulmonary nodules using size and shape, with little attention to texture features. In this work, texture and shape features were extracted from pulmonary nodules selected from the LIDC data set. Several classifiers including Decision Trees, Nearest Neighbor, and Support Vector Machines (SVM) were used for classifying malignant and benign pulmonary nodules. An accuracy of 90.91% was achieved using a 5-nearest-neighbors algorithm and a data set containing texture features only. Laws and Wavelet features received the highest rank when using feature selection implying a larger contribution in the classification process. Considering the improvement in classification accuracy, the use of texture features appears to be a promising direction in computer-aided diagnosis of pulmonary nodules in LDCT.

Automated prediction of patient-specific disease progression can significantly contribute to clinical treatment. This paper presents a computer-assisted framework to tackle the survival time prediction problem. Inspired by the assumption that niche tumor regions may play a significant role in cancer diagnosis, we explore local visual variations from multiple MRI sequences. The research consists of three parts: 1) the extraction of multi-scale Local Binary Patterns (LBP) to describe the visual variations, 2) a supervised forward feature selection approach, called the Feature Ranking Model (FRM) which captures single feature predictive ability efficiently, and combines the top features to form a feature subset, 3) We cast the clinical survival time prediction task as a binary category classification problem. We tested the framework using a dataset of 32 cases collected from The Cancer Genome Atlas (TCGA). We obtained a 93.75% accuracy rate for the prediction of survival time.

Design-based (unbiased) stereology provides an accurate, precise, and efficient method to quantify morphological parameters of biological microstructures, such as the total number of three-dimensional (3D) objects (cells) in stained tissue sections. The current requirement for extensive user interaction with commercially available computerized stereology systems limits the throughput of data collection. To increase the efficiency of this process, an algorithm was developed to automate data collection from stained objects in thick, transparent tissue sections. We present a novel approach to extract, count and classify stained objects of interest in 3D by linking them through a z-stack of images. Skeletonization and erosion are used to further segment the under segmented (overlapping) cells resulting from the extraction of out of focus cells in conjunction with in focus cells. Finally, 3D shape features, computed from the re-linked cells, are used for final classification of counted objects into “cells” and “not-cells”. We achieve a classification accuracy of 85% using SVM in a leave one-out experiment. The results demonstrate the effectiveness of our algorithm to count cells in 3D from thick, transparent tissue sections.

Gene-expression microarray datasets often consist of a limited number of samples with a large number of gene-expression measurements, usually on the order of thousands. Therefore, dimensionality reduction is critical prior to any classification task. In this work, the iterative feature perturbation method (IFP), an embedded gene selector, is introduced and applied to four microarray cancer datasets: colon cancer, leukemia, Moffitt colon cancer, and lung cancer. We compare results obtained by IFP to those of support vector machine-recursive feature elimination (SVM-RFE) and the t-test as a feature filter using a linear support vector machine as the base classifier. Analysis of the intersection of gene sets selected by the three methods across the four datasets was done. Additional experiments included an initial pre-selection of the top 200 genes based on their p values. IFP and SVM-RFE were then applied on the reduced feature sets. These results showed up to 3.32% average performance improvement for IFP across the four datasets. A statistical analysis (using the Friedman/Holm test) for both scenarios showed the highest accuracies came from the t-test as a filter on experiments without gene pre-selection. IFP and SVM-RFE had greater classification accuracy after gene pre-selection. Analysis showed the t-test is a good gene selector for microarray data. IFP and SVM-RFE showed performance improvement on a reduced by t-test dataset. The IFP approach resulted in comparable or superior average class accuracy when compared to SVM-RFE on three of the four datasets. The same or similar accuracies can be obtained with different sets of genes.

Support vector machines are binary classifiers that can implement multi-class classifiers by creating a classifier for each possible combination of classes or for each class using a one class versus all strategy. Feature selection algorithms often search for a single set of features to be used by each of the binary classifiers. This ignores the fact that features that may be good discriminators for two particular classes might not do well for other class combinations. As a result, the feature selection process may not include these features in the common set to be used by all support vector machines. It is shown that by selecting features for each binary class combination, overall classification accuracy can be improved (as much as 2.1%), feature selection time can be significantly reduced (speed up of 3.2 times), and time required for training a multi-class support vector machine is reduced. Another benefit of this approach is that considerably less time is required for feature selection when additional classes are added to the training data. This is because the features selected for the existing class combinations are still valid, so that feature selection only needs to be run for the new class combinations created.

We propose a method for the automatic spotting (temporal segmentation) of facial expressions in long videos comprising of macro- and micro-expressions. The method utilizes the strain impacted on the facial skin due to the non-rigid motion caused during expressions. The strain magnitude is calculated using the central difference method over the robust and dense optical flow field observed in several regions (chin, mouth, cheek, forehead) on each subject's face. This new approach is able to successfully detect and distinguish between large expressions (macro) and rapid and localized expressions (micro). Extensive testing was completed on a dataset containing 181 macro-expressions and 124 micro-expressions. The dataset consists of 56 videos collected at USF, 6 videos from the Canal-9 political debates, and 3 low quality videos found on the internet. A spotting accuracy of 85% was achieved for macro-expressions and 74% of all micro-expressions were spotted.

A CT-scan is a vital tool for the diagnosis of lung cancer via tumor detection. Developing a classifier to make use of the information in CT-scan images could provide a non-invasive alternative to histopathological techniques such as needle biopsy to identify tumor types. Image features extracted from 74 lung tumor objects of CT-scan images are used in classifying tumor types. Classification is done into two major classes of non-small cell lung tumors, Adenocarcinoma and Squamous-cell Carcinoma, each constituting 30% of all lung tumors. In this first of its kind investigation, a large group of 2D and 3D image features which were hypothesized to be useful are evaluated for effectiveness in classifying the tumors. Classifiers including decision trees and support vector machines are used along with feature selection techniques (Wrappers and Relief-F) to build models for tumor classification. Results show that over the large feature space for both 2D and 3D features it is possible to recognize tumor classes with about 68% accuracy, showing new features may be of help. The accuracy achieved using 2D and 3D features is similar with 3D easier to use.

Typically, thousands of gene expression levels are recorded for a group of patients, leading to the situation where the number of features far exceeds the number of examples. To combat this, researchers would want to combine gene expression data collected at different sites into one data set to reduce the magnitude of the difference between the number of features (genes) and examples (samples). This makes gene selection a critical component of any process to build models using gene expression data. For instance, in the domain of ordering cancer patients based on survival time, one might assume that utilizing genes related to cancer development and progression will allow the best model to be built. In this paper, we explore two different gene selection techniques and examine how well the genes selected compare between methods. We also check gene set consistency between data sets collected using the same protocols at different research institutions. It is shown that gene selection can result in very different sets given different training data.

This paper develops a novel approach for fluid flow tracking and analysis. Specifically, the proposed algorithm is able to detect the traveling waves, compute the wave parameters and determine controlling film flow parameters for a liquid film flowing over a rotating disk. The input to the algorithm is an easily acquired video data. It is shown that under single light illumination, it is possible to track the specular portion of the reflected light on the traveling wave. Hence, wavy films can be tracked, and fluid flow parameters can be computed. The fluid flow parameters include wave velocities, wave inclination angles, and distances between consecutive waves. Once the parameters are computed, their accuracy is analyzed and compared with the solutions of the mathematical models based on the Navier–Stokes equations. The mathematical model predicts wave characteristics based on directly measured controlling parameters, such as disk rotation speed and fluid flow rate. It is shown that the calculated parameter values coincide with the predicted ones. The average computed parameters are within 5–10% of the predicted values. Next, the developed approach is generalized to model-based recovery of fluid flow controlling parameters: the rotation speed and the fluid-flow rate. The search in space for model parameters is performed to minimize the error between the flow characteristics predicted by the fluid dynamics model (e.g. distance between waves, wave inclination angles) and parameters recovered from video data. Results demonstrate that the speed of a disk and the flow rate, when compared to the ground truth available from direct observation, are recovered with the error less than 10%.

Scalable versions of the widely used fuzzy c-means clustering algorithm called single-pass fuzzy c-means and online fuzzy c-means have been recently introduced. Both algorithms facilitate scaling to very large numbers of examples while providing partitions that very closely approximate those one would obtain using fuzzy c-means. Both algorithms have been successfully applied to a number of datasets, most notably, magnetic resonance image volumes of the human brain. In this letter, we show that weighting examples in the fuzzy c-means algorithm does not cause a violation in its convergence proof, and we provide a separate proof of convergence that holds for any dataset.

Single pass fuzzy c-means and Online fuzzy c-means are two scalable versions of the widely used fuzzy c-means clustering algorithm. They both facilitate scaling to very large numbers of examples while providing partitions that very closely approximate those one would obtain using fuzzy c-means. They have been successfully applied to a number of data sets, most notably magnetic resonance image volumes of the human brain. In practice, the algorithms have converged on the data sets to which they been applied. Computers are of finite precision, which will allow real values to be converted to integers with minor loss of information. In this paper, we show that they will converge to local minima or saddle points of the modified objective function for any data set when weights are integers.

A novel wire detection algorithm for use by unmanned aerial vehicles (UAV) in low altitude urban reconnaissance is presented. This is of interest to urban search and rescue and military reconnaissance operations. Detection of wires plays an important role, because thin wires are hard to discern by tele-operators and automated systems. Our algorithm is based on identification of linear patterns in images. Most existing methods that search for linear patterns use a simple model of a line, which does not take into account the line surroundings. We propose the use of a robust Gaussian model to approximate the intensity profile of a line and its surroundings which allows effective discrimination of wires from other visually similar linear patterns. The algorithm is able to cope with highly cluttered urban backgrounds, moderate rain, and mist. Experimental results show a 17.7% detection improvement over the baseline.

This paper presents an algorithm that detects and tracks marine vessels in video taken by a nonstationary camera installed on an untethered buoy. The video is characterized by large inter-frame motion of the camera, cluttered background, and presence of compression artifacts. Our approach performs segmentation of ships in individual frames processed with a color-gradient filter. The threshold selection is based on the histogram of the search region. Tracking of ships in a sequence is enabled by registering the horizon images in one coordinate system and by using a multi-hypothesis framework. Registration step uses an area-based technique to correlate a processed strip of the image over the found horizon line. The results of evaluation of detection, localization, and tracking of the ships show significant increase in performance in comparison to the previously used technique.

In this paper, we propose a method to model the material constants (Young's modulus) of the skin in subregions of the face from the motion observed in multiple facial expressions and present its relevance to an image analysis task such as face verification. On a public database consisting of 40 subjects undergoing some set of facial motions associated with anger, disgust, fear, happy, sad, and surprise expressions, we present an expression invariant strategy to matching faces using the Young's modulus of the skin. Results show that it is indeed possible to match faces across expressions using the material properties of their skin.

Many genes and a small number of samples are problematic characteristics of microarray datasets. We investigated the impact on classification accuracy of gene selection approaches on filtered-to-200-gene datasets. Four datasets were used with 3 filters: Student's t-test, information gain, and reliefF. We applied Iterative Feature Perturbation (IFP) and Recursive Feature Elimination (SVM-RFE) for further gene selection. Both approaches resulted in accuracy improvement when used with the t-test-filtered datasets, but not when information gain or reliefF were used. An AUC analysis of the IFP and SVM-RFE accuracy curves indicated that both methods reached the highest AUC values after t-test filtering. A statistical analysis of accuracy across the best 50 genes using the Friedman/Holm test showed that IFP and SVM-RFE were significantly more accurate more often when applied to the t-test-filtered gene sets. Surprisingly, the simple t-test, applied as a filter, results in the best overall SVM accuracy and is at least as accurate as the other, more complicated filter methods.

Clustering large data has the problem of not having all the data fit in the memory at one time. It is a challenge to apply fuzzy clustering algorithms to get a partition in a timely manner. In this paper, we compare the online fuzzy clustering and single pass fuzzy clustering algorithms, which can be used to cluster very large data sets which might be treated as streaming data, with fuzzy c-means. We introduce more meaningful partition comparison measurements based on cluster center location instead of using the difference in Rmvalue. We obtained results on several large volumes of magnetic resonance images which indicate that the online FCM algorithm produces partitions which are very close to what you could get if you clustered all the data at one time. We also show online FCM outperforms single pass FCM and it can process streaming data as it comes without degradation in most cases.

Visual surveillance is an active research topic in image processing. Transit systems are actively seeking new or improved ways to use technology to deter and respond to accidents, crime, suspicious activities, terrorism, and vandalism. Human behavior-recognition algorithms can be used proactively for prevention of incidents or reactively for investigation after the fact. This paper describes the current state-of-the-art image-processing methods for automatic-behavior-recognition techniques, with focus on the surveillance of human activities in the context of transit applications. The main goal of this survey is to provide researchers in the field with a summary of progress achieved to date and to help identify areas where further research is needed. This paper provides a thorough description of the research on relevant human behavior-recognition methods for transit surveillance. Recognition methods include single person (e.g., loitering), multiple-person interactions (e.g., fighting and personal attacks), person-vehicle interactions (e.g., vehicle vandalism), and person-facility/location interactions (e.g., object left behind and trespassing). A list of relevant behavior-recognition papers is presented, including behaviors, data sets, implementation details, and results. In addition, algorithm's weaknesses, potential research directions, and contrast with commercial capabilities as advertised by manufacturers are discussed. This paper also provides a summary of literature surveys and developments of the core technologies (i.e., low-level processing techniques) used in visual surveillance systems, including motion detection, classification of moving objects, and tracking.

Red tides pose a significant environmental and economic threat in the Gulf of Mexico. Timely detection of red tides is important for understanding this phenomenon. In this paper, learning approaches based on k-nearest neighbors, random forests and support vector machines have been evaluated for red tide detection from MODIS satellite images. Detection results from our algorithms were compared with ground truth red tide data collected in situ. Our results show that red tide identification methods based on machine learning approaches outperform baseline algorithms based on bio-optical characterization.

This paper presents a novel method for automatic spotting (temporal segmentation) of facial expressions in long videos comprising of continuous and changing expressions. The method utilizes the strain impacted on the facial skin due to the non-rigid motion caused during expressions. The strain magnitude is calculated using the central difference method over the robust and dense optical flow field of each subjects face. Testing has been done on 2 datasets (which includes 100 macro-expressions) and promising results have been obtained. The method is robust to several common drawbacks found in automatic facial expression segmentation including moderate in-plane and out-of-plane motion. Additionally, the method has also been modified to work with videos containing micro-expressions. Micro-expressions are detected utilizing their smaller spatial and temporal extent. A subject's face is divided in to sub-regions (mouth, cheeks, forehead, and eyes) and facial strain is calculated for each of these regions. Strain patterns in individual regions are used to identify subtle changes which facilitate the detection of micro-expressions.

Common benchmark data sets, standardized performance metrics, and baseline algorithms have demonstrated considerable impact on research and development in a variety of application domains. These resources provide both consumers and developers of technology with a common framework to objectively compare the performance of different algorithms and algorithmic improvements. In this paper, we present such a framework for evaluating object detection and tracking in video: specifically for face, text, and vehicle objects. This framework includes the source video data, ground-truth annotations (along with guidelines for annotation), performance metrics, evaluation protocols, and tools including scoring software and baseline algorithms. For each detection and tracking task and supported domain, we developed a 50-clip training set and a 50-clip test set. Each data clip is approximately 2.5 minutes long and has been completely spatially/temporally annotated at the I-frame level. Each task/domain, therefore, has an associated annotated corpus of approximately 450,000 frames. The scope of such annotation is unprecedented and was designed to begin to support the necessary quantities of data for robust machine learning approaches, as well as a statistically significant comparison of the performance of algorithms. The goal of this work was to systematically address the challenges of object detection and tracking through a common evaluation framework that permits a meaningful objective comparison of techniques, provides the research community with sufficient data for the exploration of automatic modeling techniques, encourages the incorporation of objective evaluation into the development process, and contributes useful lasting resources of a scale and magnitude that will prove to be extremely useful to the computer vision research community for years to come.

Support vector machines (SVMs) can be trained to be very accurate classifiers and have been used in many applications. However, the training time and, to a lesser extent, prediction time of SVMs on very large data sets can be very long. This paper presents a fast compression method to scale up SVMs to large data sets. A simple bit-reduction method is applied to reduce the cardinality of the data by weighting representative examples. We then develop SVMs trained on the weighted data. Experiments indicate that bit-reduction SVM produces a significant reduction in the time required for both training and prediction with minimum loss in accuracy. It is also shown to typically be more accurate than random sampling when the data are not overcompressed.

We introduce a novel wire detection algorithm for use in low altitude urban aircraft reconnaissance. A line profile model is described and effectively used to discriminate wires from other linear patterns commonly found in urban scenes. The algorithm is able to cope with highly cluttered backgrounds, moderate rain and mist, and with no stabilization of camera. The studied domain is of particular interest to urban search and rescue and military reconnaissance operations. The algorithmpsilas receiver operating characteristic curve is shown, based on a multi-site dataset with 10160 wires spanning in 5576 frames. Encouraging results show up to 37% detection improvement over a previously published baseline algorithm for comparable false alarm rates.

This work presents a novel technique for automatic detection and tracking of marine vehicles in video of open sea. The source of video is a video camera mounted on a buoy platform in open sea. Such system is intended to work autonomously, taking video of the surrounding ocean surface and analyzing them on presence of marine vehicles. The proposed technique is based on detection of marine vehicles in individual video frames and tracking the detected targets through the video sequence with the help of a tracking algorithm. Several performance metrics are utilized for performance evaluation of the proposed approach. Accuracy of detection in 90% range is shown on a dataset of 30 short video sequences taken by a prototype of the system.

In surveillance situations, computer vision systems are often deployed to help humans perform their tasks more effectively. In a typical installation human observers are required to simultaneously monitor a number of video signals. Psychophysical research indicates that there are severe limitations in the ability of humans to monitor simultaneous signals. Do these same limitations extend to surveillance? We present a method for evaluating human surveillance performance in a situation that mimics demands of real world surveillance. A single computer monitor contained either nine display cells or four display cells. Each cell contained a stream of 2 to 4 moving objects. Observers were instructed to signal when a target event occurred - - when one of the objects entered a small square ldquoforbiddenrdquo region in the center of the display. Target events could occur individually or in groups of 2 or 3 temporally close events. The results indicate that the observers missed many targets (60%) when required to monitor 9 displays and many fewer when monitoring 4 displays (20%). Further, there were costs associated with target events occurring in close temporal succession. Understanding these limitations would help computer visions researchers to design algorithms and human-machine interfaces that result in improved overall performance.

We present a finite element modeling based approach to compute strain patterns caused by facial deformation during expressions in videos. A sparse motion field computed through a robust optical flow method drives the FE model. While the geometry of the model is generic, the material constants associated with an individualpsilas facial skin are learned at a coarse level sufficient for accurate strain map computation. Experimental results using the computational strategy presented in this paper emphasize the uniqueness and stability of strain maps across adverse data conditions (shadow lighting and face camouflage) making it a promising feature for image analysis tasks that can benefit from such auxiliary information.

In this paper, we propose a new edge-based text verification approach for video. Based on the investigation of the relation between candidate blocks and their neighbor areas, the proposed approach first detects background edges in candidate blocks, then erases them by an edge tracking technique, and finally the candidate blocks containing too few remaining edges are eliminated as false alarms. Three measures for text detection evaluation in video were used to assess the performance of the proposed text verification approach. Experimental results on 50 broadcast news video clips demonstrate the validity of our approach.

Microarray datasets are often limited to a small number of samples with a large number of gene expressions. Therefore, dimensionality reduction through a feature/gene selection process is highly important for classification purposes. In this paper, a feature perturbation method we previously introduced is applied to do gene selection from microarray data. A publicly available colon cancer dataset is used in our experiments. In comparison with SVM-RFE, our method is better with feature sets of between 10 and 80, however for less than 10 features SVM-RFE results in higher accuracy. An analysis of the area under the curve of the feature perturbation method for the top 50 and 25 features is performed, aiming to determine the proper amount of noise to be applied. We show that a good set of small features/genes can be found using the feature perturbation method.

Recently several algorithms for clustering large data sets or streaming data sets have been proposed. Most of them address the crisp case of clustering, which cannot be easily generalized to the fuzzy case. In this paper, we propose a simple single pass (through the data) fuzzy c means algorithm that neither uses any complicated data structure nor any complicated data compression techniques, yet produces data partitions comparable to fuzzy c means. We also show our simple single pass fuzzy c means clustering algorithm when compared to fuzzy c means produces excellent speed-ups in clustering and thus can be used even if the data can be fully loaded in memory. Experimental results using five real data sets are provided.

Before new medical treatments become available to the public, clinicians must conduct extensive trials to determine the efficacy of the novel therapy. In order for the clinical trial to be successful, a significant number of patients with an appropriate set of medical conditions must be accrued. We have implemented a web-based expert system at the H. Lee Moffitt Cancer Center & Research Institute in the Gastrointestinal Tumor Clinic (GITC) to help physicians screen patients for phase II trials. Our system allows physicians to screen a patient for multiple trials simultaneously. Our experiments have shown that adaptation of the system into a clinical environment and the success of the system are related to the amount of time physicians are willing to spend entering data. We also found significant regulatory issues (HIPAA) that make implementation challenging.

Clustering algorithms for streaming data sets are gaining importance due to the availability of large data streams from different sources. Recently a number of streaming algorithms have been proposed using crisp algorithms such as hard c means or its variants. The crisp cases may not be easily generalized to fuzzy cases as these two groups of algorithms try to optimize different objective functions. In this paper we propose a streaming variant of the fuzzy c means algorithm. At any stage during processing, a good streaming algorithm should be able to summarize data seen so far and also respond to evolving distributions. We study the tradeoff involved between summarization of data seen and response to an evolving distribution by varying the amount of history used by a streaming algorithm. Empirical evaluation of the performance of our algorithm using both artificial and real data sets under a noisy setting shows its effectiveness.

The success of forensic identification largely depends on the availability of strong evidence or traces that substantiate the prosecution hypothesis that a certain person is guilty of crime. In light of this, extracting subtle evidences which the criminals leave behind at the crime scene will be of valuable help to investigators. We propose a novel method of using strain pattern extracted from changing facial expressions in video as an auxiliary evidence for person identification. The strength of strain evidence is analyzed based on the increase in likelihood ratio it provides in a suspect population. Results show that strain pattern can be used as a supplementary biometric evidence in adverse operational conditions such as shadow lighting and face camouflage where pure intensity-based face recognition algorithms will fail

Registering features in multiple mammographic views is an important technique to improve breast cancer detection rate. However, nonrigid breast deformation during X-ray imaging poses a severe challenge to the conventional 2D registration methods. We present a method that utilizes a 3D model to facilitate two-view registration by predicting breast deformation. At first, a finite element model of a breast is constructed using its MRIs. The model is capable of simulating both compression and decompression. Feature registration is then accomplished through a series of projections and compression-decompression operations. Experiments using real patient data demonstrate that a mammographic feature can be successfully registered from one view to another.

Detecting a horizon in an image is an important part of many image related applications such as detecting ships on the horizon, flight control, and port security. Most of the existing solutions for the problem only use image processing methods to identify a horizon line in an image. This results in good accuracy for many cases and is fast in computation. However, for some images with difficult environmental conditions like a foggy or cloudy sky these image processing methods are inherently inaccurate in identifying the correct horizon. This paper investigates how to detect the horizon line in a set of images using a machine learning approach. The performance of the SVM, J48, and naive Bayes classifiers, used for the problem, has been compared. Accuracy of 90-99% in identifying horizon was achieved on image data set of 20 images

Combining multiple clustering solutions is important for obtaining a robust clustering solution, merging distributed clustering solutions, and scaling to large data sets. The combination of multiple clustering solutions within a scalable and robust framework for large data sets is discussed. A scalable framework requires both cluster ensemble creation and merging to be efficient in terms of time and memory complexity. We also introduce the concept of filtering malformed clusters from the ensemble. They result from unfortunate initialization or unbalanced data distribution or noise. Experimental results on real data sets show that this approach will scale and provide cluster partitions which are functionally better or equivalent when compared to clustering all the data at once and clustering solutions contained in the ensemble. We have also compared our algorithm with other ensemble merging and scalable algorithms to point out its strengths and limitations.

This paper presents a constrained genetic approach for reconstructing the material properties of elastic objects. The considered reconstruction problem is ill-posed and must be constrained properly so that a unique and stable numerical solution can be obtained. Qualitative prior information is incorporated using a rank-based scheme to constrain the admissible solutions. Experiments show that the proposed approach is robust when presented with noisy data and can reconstruct the elastic property accurately and reliably. In a comparison study with the deterministic Gauss-Newton methods, the constrained genetic approach also shows very consistent performance.

Support vector machines are very accurate classifiers and have been widely used in many applications. However, the training and to a lesser extent prediction time of support vector machines on very large data sets can be very long. This paper presents a fast compression method to scale up support vector machines to large data sets. A simple bit reduction method is applied to reduce the cardinality of the data by weighting representative examples. We then develop support vector machines which may be trained on weighted data. Experiments indicate that the bit reduction support vector machine produces a significant reduction in the time required for both training and prediction with minimum loss in accuracy. It is also shown to be more accurate than random sampling, when the data is not over-compressed.

An automatic human brain segmentation system for magnetic resonance images is presented. It has two main parts: a fuzzy clustering algorithm and a set of cluster combination rules. Images are segmented into ten classes by the unsupervised fuzzy c-means clustering algorithm. Then a knowledge-based system labels the clusters into the tissues of interest: cerebrospinal fluid, gray matter and white matter. This approach can process MRI data that comes from different scanners with different sequences and head coils, using several different spin-echo images (with different echo times) and different slice thickness. The system adapts without manual intervention. Segmented synthetic image data from the brainWeb simulated normal brain database resulted in a one voxel away accuracy of 90%. The results from real data from various magnetic resonance imagers were compared with a radiologist's segmentation and found to generally agree within 10%, the typical range of inter-rater radiologist agreement.

This paper presents an active learning method which reduces the labeling effort of domain experts in multi-class classification problems. Active learning is applied in conjunction with support vector machines to recognize underwater zooplankton from higher-resolution, new generation SIPPER II images. Most previous work on active learning with support vector machines only deals with two class problems. In this paper, we propose an active learning approach "breaking ties" for multi-class support vector machines using the one-vs-one approach with a probability approximation. Experimental results indicate that our approach often requires significantly less labeled images to reach a given accuracy than the approach of labeling the least certain test example and random sampling. It can also be applied in batch mode resulting in an accuracy comparable to labeling one image at a time and retraining.

A clinical trial is a study conducted on a group of patients to evaluate a new treatment procedure. Usually, clinicians manually select patients for a clinical trial; the choice of eligible patients is a labor-intensive process, and clinicians are often unable to identify sufficient number of patients, which delays the evaluation of new treatments. We have developed a Web-based system that helps clinicians to determine the eligibility of patients for multiple clinical trials. It uses probabilistic techniques that minimize the amount of manual data entry, by ordering the related data-entry steps. We describe the developed system and give the results of applying it to retrospective data of breast cancer patients at the Moffitt Cancer Center.

A modeling approach is presented for quantitative burn scar assessment. Emphases are given to: 1) constructing a finite-element model from natural image features with an adaptive mesh and 2) quantifying the Young's modulus of scars using the finite-element model and regularization method. A set of natural point features is extracted from the images of burn patients. A Delaunay triangle mesh is then generated that adapts to the point features. A three-dimensional finite-element model is built on top of the mesh with the aid of range images providing the depth information. The Young's modulus of scars is quantified with a simplified regularization functional, assuming that the knowledge of the scar's geometry is available. The consistency between the relative elasticity index and the physician's rating based on the Vancouver scale (a relative scale used to rate burn scars) indicates that the proposed modeling approach has high potential for image-based quantitative burn scar assessment.

We present a system to recognize underwater plankton images from the shadow image particle profiling evaluation recorder (SIPPER). The challenge of the SIPPER image set is that many images do not have clear contours. To address that, shape features that do not heavily depend on contour information were developed. A soft margin support vector machine (SVM) was used as the classifier. We developed a way to assign probability after multiclass SVM classification. Our approach achieved approximately 90% accuracy on a collection of plankton images. On another larger image set containing manually unidentifiable particles, it also provided 75.6% overall accuracy. The proposed approach was statistically significantly more accurate on the two data sets than a C4.5 decision tree and a cascade correlation neural network. The single SVM significantly outperformed ensembles of decision trees created by bagging and random forests on the smaller data set and was slightly better on the other data set. The 15-feature subset produced by our feature selection approach provided slightly better accuracy than using all 29 features. Our probability model gave us a reasonable rejection curve on the larger data set.

We present a methodology for correcting color images taken in practical indoor environments, such as laboratories, factories, and studios, that explicitly models illuminant location, surface reflectance and geometry, and camera responsivity. We explicitly model surfaces by taking our color images with corresponding registered three-dimensional (3-D) range images, which provide surface orientation and location information for every point in the scene. We automatically detect regions where color correction should not be applied, such as specularities, coarse texture regions, and jump edges. This correction results in objective color measures of the imaged surfaces. This kind of integrated, comprehensive system of color correction has not existed until now. i.e., it is the first of its kind in computer vision. We demonstrate results of applying this methodology to real images for applications in photorealistic rerendering, skin lesion detection, burn scar color measurement, and general color image enhancement. We also have tested the method under different lighting configurations and with three different range scanners.

When clinicians test a new treatment procedure, they need to identify and recruit patients with appropriate medical conditions. We have developed an expert system that helps clinicians select patients for experimental treatments, and to reduce the number and overall cost of related medical tests. We describe experiments on selecting patients for new treatments at the Moffitt Cancer Center. The experiments have shown that the system can increase the number of selected patients by a factor of three, and that it can also reduce the cost of the selection process.

We present a system to recognize underwater plankton images from the Shadow Image Particle Profiling Evaluation Recorder. As some images do not have clear contours, we developed several features that do not heavily depend on the contour information. A soft margin support vector machine (SVM) was used as the classifier. We developed a new way to assign probability after multi-class SVM classification. Our approach achieved approximately 90% accuracy on a collection of images with minimal noise. On another image set containing manually unidentifiable particles, it also provided promising results. Furthermore, our approach is more accurate on the two data sets than a C4.5 decision tree and a cascade correlation neural network at the 95% confidence level.

Type-1 diabetes is a life-long disorder that often occurs in children or young adults. Susceptibility to type-1 diabetes is believed to be dependent on both genetic and environmental causes. Genetic screening is becoming a common tool for evaluating risk, potentially before the onset of the disease. However, interpreting the results of genetic tests involves many rules taken from the literature on the correlation between risk and genetic markers. We introduce an expert system based approach in which the evaluation of risk for type-1 diabetes can be automated. This method can act as a tool for medical professionals and provides the crucial explanatory component, including citations to the relevant literature. The expert system was developed in conjunction with geneticists. As part of the corresponding clinical trial, the expert system correctly identified the at-risk genetic profiles in a set of 253 subjects evaluated by human experts.

Clustering is a useful approach in image segmentation, data mining, and other pattern recognition problems for which unlabeled data exist. Fuzzy clustering using fuzzy c-means or variants of it can provide a data partition that is both better and more meaningful than hard clustering approaches. The clustering process can be quite slow when there are many objects or patterns to be clustered. This paper discusses the algorithm brFCM, which is able to reduce the number of distinct patterns which must be clustered without adversely affecting the partition quality. The reduction is done by aggregating similar examples and then using a weighted exemplar in the clustering process. The reduction in the amount of clustering data allows a partition of the data to be produced faster. The algorithm is applied to the problem of segmenting 32 magnetic resonance images into different tissue types and the problem of segmenting 172 infrared images into trees, grass and target. Average speed-ups of as much as 59-290 times a traditional implementation of fuzzy c-means were obtained using brFCM, while producing partitions that are equivalent to those produced by fuzzy c-means.

In this paper we present a method to compute the egomotion of a range camera using the space envelope. The space envelope is a geometric model that provides more information than a simple segmentation for correspondences and motion estimation. We describe a novel variation of the maximal matching algorithm that matches surface normals to find correspondences. These correspondences are used to compute rotation and translation estimates of the egomotion. We demonstrate our methods on two image sequences containing 70 images. We also discuss the cases where our methods fail, and additional possible methods for exploiting the space envelope.

This paper presents a constrained genetic approach (CGA) for reconstructing the Young's modulus of elastic objects. Qualitative a priori information is incorporated using a rank based scheme to constrain the admissible solutions. Balance between the fitness function (adhesion to the measurement data) and the penalty function (fidelity to a priori knowledge) is achieved by a stochastic sort algorithm. The over-smoothing of Young's modulus discontinuity is avoided without the need of computing a deterministic weight coefficient. The experiment on synthetic data indicates that the proposed method not only reconstructed reliable Young's modulus from noisy data, but also expedited the convergence process significantly.

Segmentation of an image into regions and the labeling of the regions is a challenging problem. In this paper, an approach that is applicable to any set of multifeature images of the same location is derived. Our approach applies to, for example, medical images of a region of the body; repeated camera images of the same area; and satellite images of a region. The segmentation and labeling approach described here uses a set of training images and domain knowledge to produce an image segmentation system that can be used without change on images of the same region collected over time. How to obtain training images, integrate domain knowledge, and utilize learning to segment and label images of the same region taken under any condition for which a training image exists is detailed. It is shown that clustering in conjunction with image processing techniques utilizing an iterative approach can effectively identify objects of interest in images. The segmentation and labeling approach described here is applied to color camera images and two other image domains are used to illustrate the applicability of the approach.

We introduce a software framework called the Java Vision Toolkit (JVT) for teaching image processing and computer vision. The toolkit provides over 50 image operations and presents them to the user in a GUI that can render grayscale, color and 3D range images. The software is written in Java, enabling it to be integrated into HTML documents and interactive course materials. The framework is designed for extensibility using a source code template that supports the implementation of any new operation with a minimal amount of supporting code. For students, this framework encapsulates the GUI, file I/O and other trivial programming details and allows them the maximum amount of time to spend on understanding computer vision. We compare the JVT with other computer vision software frameworks that are used for teaching and research. We also discuss the use of the JVT in an undergraduate image processing course at the University of South Florida.

We present a methodology for calibrating multiple light source locations in 3D from images. The procedure involves the use of a novel calibration object that consists of three spheres at known relative positions. The process uses intensity images to find the positions of the light sources. We conducted experiments to locate light sources in 51 different positions in a laboratory setting. Our data shows that the vector from a point in the scene to a light source can be measured to within 2.7/spl plusmn/4/spl deg/ at /spl alpha/=.05 (6 percent relative) of its true direction and within 0.13/spl plusmn/.02 m at /spl alpha/=.05 (9 percent relative) of its true magnitude compared to empirically measured ground truth. Finally, we demonstrate how light source information is used for color correction.

Tracking both structure and motion of nonrigid objects from monocular images is an important problem in vision. In this paper, a hierarchical method which integrates local analysis (that recovers small details) and global analysis (that appropriately limits possible nonrigid behaviors) is developed to recover dense depth values and nonrigid motion from a sequence of 2D satellite cloud images without any prior knowledge of point correspondences. This problem is challenging not only due to the absence of correspondence information but also due to the lack of depth cues in the 2D cloud images (scaled orthographic projection). In our method, the cloud images are segmented into several small regions and local analysis is performed for each region. A recursive algorithm is proposed to integrate local analysis with appropriate global fluid model constraints, based on which a structure and motion analysis system, SMAS, is developed. We believe that this is the first reported system in estimating dense structure and nonrigid motion under scaled orthographic views using fluid model constraints. Experiments on cloud image sequences captured by meteorological satellites (GOES-8 and GOES-9) have been performed using our system, along with their validation and analyses. Both structure and 3D motion correspondences are estimated to subpixel accuracy. Our results are very encouraging and have many potential applications in earth and space sciences, especially in cloud models for weather prediction.

In our previous work, we used finite element models to determine nonrigid motion parameters and recover unknown local properties of objects given correspondence data recovered with snakes or other tracking models. In this paper, we present a novel multiscale approach to recovery of nonrigid motion from sequences of registered intensity and range images. The main idea of our approach is that a finite element (FEM) model incorporating material properties of the object can naturally handle both registration and deformation modeling using a single model-driving strategy. The method includes a multiscale iterative algorithm based on analysis of the undirected Hausdorff distance to recover correspondences. The method is evaluated with respect to speed and accuracy. Noise sensitivity issues are addressed. Advantages of the proposed approach are demonstrated using man-made elastic materials and human skin motion. Experiments with regular grid features are used for performance comparison with a conventional approach (separate snakes and FEM models). It is shown, however, that the new method does not require a sampling/correspondence template and can adapt the model to available object features. Usefulness of the method is presented not only in the context of tracking and motion analysis, but also for a burn scar detection application.

This paper presents an evaluation of edge detector performance. We use the task of structure from motion (SFM) as a "black box" through which to evaluate the performance of edge detection algorithms. Edge detector goodness is measured by how accurately the SFM could recover the known structure and motion from the edge detection of the image sequences. We use a variety of real image sequences with ground truth to evaluate eight different edge detectors from the literature. Our results suggest that ratings of edge detector performance based on pixel-level metrics and on the SFM are well correlated and that detectors such as the Canny detector and Heitger detector offer the best performance.

We present a methodology for calibrating multiple light source locations in 3D from images. The procedure involves the use of a novel calibration object that consists of either 2 or 3 spheres at known relative positions. There are two variants of the process: one which uses range and intensity imaging to find the positions of the light sources, and one that uses only the intensity image to locate the illuminants. We conducted experiments using both variations of the technique to locate light sources in 51 different positions in a laboratory setting. Our data shows that the vector from a point in the scene to a light source can be measured to within 3/spl deg/(6%) of its tote direction and within 0.13 m (9%) of its true magnitude compared to empirically measured ground truth. Finally, we demonstrate how light source information can be applied to burn scar color correction and color segmentation.

In this paper we present a novel multiscale approach to recovery of nonrigid motion from sequences of registered intensity and range images. The main idea of our approach is that a finite element (FEM) model can naturally handle both registration and deformation modeling using a single model-driving strategy. The method includes a multiscale iterative algorithm based on analysis of the undirected Hausdorff distance to recover correspondences. The method is evaluated with respect to speed, accuracy, and noise sensitivity. Advantages of the proposed approach are demonstrated using man-made elastic materials and human skin motion. Experiments with regular grid features are used for performance comparison with a conventional approach (separate snakes and FEM models). It is shown that the new method does not require a grid and can adapt the model to available object features.

We present a novel approach to estimate and analyze 3D fluid structure and motion of clouds from multi-spectrum 2D cloud image sequences. Accurate cloud-top structure and motion are very important for a host of meteorological and climate applications. However, due to the extremely complex nature of cloud fluid motion, classical nonrigid motion analysis methods are insufficient for solving this particular problem. In this paper, two spectra of satellite cloud images are utilized. The high-resolution visible channel is first used to perform cloud tracking by using a recursive algorithm which integrates local motion analysis with a set of global fluid constraints, defined according to the physical fluid dynamics. Then, the infrared channel (thermodynamic information) is incorporated to post-process the cloud tracking results in order to capture the cloud density variations and small details of cloud fluidity. Experimental results on GOES (Geostationary Operational Environmental Satellite) cloud image sequences are presented in order to validate and evaluate both the effectiveness and robustness of our algorithm.

We present a method for 3D non-rigid motion tracking and structure reconstruction from 2D points and curve segments from a sequence of perspective images. The 3D locations of features in the first frame are known. The 3D affine motion model is used to describe the nonrigid motion. The results from synthetic and real data are presented. The applications include: lip tracking, MPEG4 face player, and burn scar assessment. The results show that: 1) curve segments are more robust under noise (observed from synthetic data with different Gaussian noise level); and 2) using both feature yields a significant performance gain in real data.

Understanding oceanic primary production on a global scale can be enhanced by methods that are able to automatically track phytoplankton blooms from color satellite images. In the paper, unsupervised clustering and rule learning are combined to track green river, a plume of discolored water that forms every March-May offshore along the edge of the west Florida Shelf, from the Sea Viewing Wide Field of View Sensor which began flying in late 1997. Spatial information and sea surface temperature can be integrated into the approach to improve performance. Using cross-validation experiments over a series of 59 multi-spectral images, it is shown that the developed system is able to reliably discriminate between images with green river from those with no phytoplankton blooms or other kinds of blooms. It is also effective in identifying the region which the green river covers.

The success of nonrigid motion analysis using physical finite element model is dependent on the mesh that characterizes the object's geometric structure. We suggest a deformable mesh adapted to the natural features of images. The adaptive mesh requires much fewer number of nodes than the fixed mesh which was used in the work by Tsap et al. (1998). We demonstrate the higher efficiency of the adaptive mesh in the context of estimating burn scare elasticity relative to normal skin elasticity using the observed 2D image sequence. Our results show that the scar assessment method based on the physical model using natural feature adaptive mesh can be applied to images which do not have artificial markers.

This paper presents a framework for a customizable MPEG-4 face player using a FAPs (Facial Animation Parameters) sequence recovered from a real-time image sequence. First, the 3D nonrigid motion and structure of the facial features is recovered from a 2D image sequence and a "person-specific" model of the face. The model consists of the intensity and range image of the face. Then, the FAPs are computed from the recovered 3D structure. The customizable MPEG4 face animation is generated using a FAP sequence and a model of a specific person. The dataset of four face image sequences from three different face orientations equipped with range images are used. The GT (ground truth) results are generated using the 3D structure provided by range images. The results are evaluated quantitatively by comparing recovered FAP values, and qualitatively by comparing the generated MPEG-4 animations. FAPs are recovered up to 8% (relative) and 16 FAP units (absolute) accuracy and the animation is nearly identical to the MPEG-4 animation using GT FAPs.

In this paper we propose a method for increasing precision and reliability of elasticity analysis in complicated burn scar cases. The need for a technique that would help physicians by objectively assessing elastic properties of scars, motivated our original algorithm. This algorithm successfully employed active contours for tracking and finite element models for strain analysis. However, the previous approach considered only one normal area and one abnormal area within the region of interest, and scar shapes which were somewhat simplified. Most burn scars have rather complicated shapes and may include multiple regions with different elastic properties. Hence, we need a method capable of adequately addressing these characteristics. The new method can split the region into more than two localities with different material properties, select and quantify abnormal areas, and apply different forces if it is necessary for a better shape description of the scar. The method also demonstrates the application of scale and mesh refinement techniques in this important domain. It is accomplished by increasing the number of Finite Element Method (FEM) areas as well as the number of elements within the area. The method is successfully applied to elastic materials and real burn scar cases. We demonstrate all of the proposed techniques and investigate the behavior of elasticity function in a 3-D space. Recovered properties of elastic materials are compared with those obtained by a conventional mechanics-based approach. Scar ratings achieved with the method are correlated against the judgments of physicians.

A knowledge-guided approach to automatic classification of Coastal Zone Color images off the West Florida Shelf is described. The approach is used to identify red tides on the West Florida Shelf, as well as areas with high concentration of dissolved organic matter such as a river plume found seasonally along the West Florida coast over the middle of the shelf. The Coastal Zone Color images are initially segmented by the unsupervised Multistage Random Sampling Fuzzy c-Means algorithm. Then, a knowledge-guided system is applied to the centroid values of resultant clusters to label case I, case II waters, a dilute river plume ("green river"), and red tide. The domain knowledge base contains information on cluster distribution in feature space, as well as spatial information such as bathymetry data. Our knowledge base consists of a rule-guided system and an embedded neural network. From 60 images, after training the system, this procedure recognizes all 15 images which contained a river plume and 45 images without. The system can correctly classify 74% of the pixels that belong to the river plume, which provides a substantial advantage to users looking for offshore extensions of riverine influence. Red tides are also successfully identified in a time series of images for which ground truth confirmed the presence of a harmful bloom.

We propose new algorithms for accurate nonrigid motion tracking. Given an initial model representing general knowledge of the object, a set of sparse correspondences, and incomplete or missing information about geometry or material properties, we can recover dense motion vectors using finite element models. The method is based on the iterative analysis of the differences between the actual and predicted behaviors. Unknown parameters are recovered using an iterative descent search for the best nonlinear finite element model that approximates nonrigid motion of the given object. During this search process, we not only estimate material properties, but also infer dense point correspondences from our initial set of sparse correspondences. Thus, during tracking, the model is refined which, in turn, improves tracking quality. Experimental results demonstrate the success of the proposed algorithm. Our work demonstrates the possibility of accurate quantitative analysis of nonrigid motion in range image sequences with objects consisting of multiple materials and 3D volumes.

This paper presents a methodology and results of evaluating edge detection algorithms using an object recognition task. A dataset consisting of 37 real images with 5 different jeep-like vehicles is used. Five edge detectors are compared using ROC curve analysis. The Heitger detector gives the best results. The work is being extended to include more images and a train-and-test style evaluation.

Image sequences capturing Hurricane Luis through meteorological satellites (GOES-8 and GOES-9) are used to estimate hurricane-top heights (structure) and hurricane winds (motion). This problem is difficult not only due to the absence of correspondence but also due to the lack of depth cues in the 2D hurricane images (scaled orthographic projection). In this paper, we present a structure and motion analysis system, called SMAS. In this system, the hurricane images are first segmented into small square areas. We assume that each small area is undergoing similar nonrigid motion. A suitable nonrigid motion model for cloud motion is first defined. Then, non-linear least-square method is used to fit the nonrigid motion model for each area in order to estimate the structure, motion model, and 3D nonrigid motion correspondences. Finally, the recovered hurricane-top heights and winds are presented along with an error analysis. Both structure and 3D motion correspondences are estimated to subpixel accuracy. Our results are very encouraging, and have many potential applications in earth and space sciences, especially in cloud models for weather prediction.

This paper presents a task-oriented evaluation methodology for edge detectors. Performance is measured based on the task of structure from motion. Eighteen real image sequences from 2 different scenes varying in the complexity and scenery types are used. The task-level ground truth for each image sequence is manually specified in terms of the 3D motion and structure. An automated tool computes the accuracy of the motion and structure achieved using the set of edge maps. Parameter sensitivity and execution speed are also analyzed. Four edge detectors are compared. All implementations and data sets are publicly available.

In this paper we propose new algorithms for accurate nonrigid motion tracking. Given only a set of sparse correspondences and incomplete or missing information about geometry or material properties, we recover dense motion vectors using nonlinear finite element models. The method is based on the iterative analysis of the differences between the actual and predicted behavior. Large differences indicate that an object's properties are not captured properly by the model. Feedback from the images during the motion allows the refinement of the model by minimizing the error between the expected and true position of the object's points. Unknown parameters are recovered using an iterative descent search for the best model that approximates nonrigid motion of the given object. Thus, during tracking the model is refined which, in turn, improves tracking quality. The method was applied successfully to man-made elastic materials and human skin to recover unknown elasticity, to complex 3-D objects to find details of their geometry, and to a hand motion analysis application.

In this paper we present a novel method for tracking 3D nonrigid motion and 3D structure from a sequence of monocular, perspective images, where range data of the first frame is available. No a priori knowledge of point correspondences is assumed. A genneral approach and formulas for polynomial (second order) displacement functions are presented. An analysis of the number of equations versus the number of unknowns is given. Results for synthetic and real image sequences are presented.

We propose a new method for accurate nonrigid motion analysis when point correspondence data is not available. We construct nonlinear finite element models by integrating range data and prior knowledge about an object's properties. We attempt to recover the motion sequence given an initial alignment of the model with the first frame of the sequence. The main idea of the method is to find the forces that are responsible for the motion or shape deformation of the given object. The task is broken into subtasks of finding the forces for each frame. Both absolute values and directions of these forces are taken into consideration and iteratively varied not only for each frame, but also between the frames. Our work demonstrates the possibility of accurate nonrigid motion analysis and force recovery from range image sequences containing nonrigid objects and large motion without interframe point correspondences.

There is a growing need for expertise both in image analysis and in software engineering. To date, these two areas have been taught separately in an undergraduate computer and information science curriculum. However, we have found that introduction to image analysis can be easily integrated in data-structure courses without detracting from the original goal of teaching data structures. Some of the image processing tasks offer a natural way to introduce basic data structures such as arrays, queues, stacks, trees and hash tables. Not only does this integrated strategy expose the students to image related manipulations at an early stage of the curriculum but it also imparts cohesiveness to the data-structure assignments and brings them closer to real life. In this paper we present a set of programming assignments that integrates undergraduate data-structure education with image processing tasks. These assignments can be incorporated in existing data-structure courses with low time and software overheads. We have used these assignment sets thrice: once in a 10-week duration data-structure course at the University of California, Santa Barbara and the other two times in 15-week duration courses at the University of South Florida, Tampa.

The construction of a surface model from range data may be undertaken at any point in a continuum of scales that reflects the level of detail of the resulting model. This continuum relates the construction parameters to the scale of the model. We propose methods to dynamically reprocess range data at different scales. The construction result from a single scale is automatically evaluated, causing reconstruction at a different scale when user-defined criteria are not met. We demonstrate our methods in constructing a planar b-rep space envelope (a scene representation) for over 400 range images. The experiments demonstrate the ability to construct 100 percent valid models, with the scale of detail within specified requirements.

A system that automatically segments and labels glioblastoma-multiforme tumors in magnetic resonance images (MRI's) of the human brain is presented. The MRI's consist of T1-weighted, proton density, and T2-weighted feature images and are processed by a system which integrates knowledge-based (KB) techniques with multispectral analysis. Initial segmentation is performed by an unsupervised clustering algorithm. The segmented image, along with cluster centers for each class are provided to a rule-based expert system which extracts the intracranial region. Multispectral histogram analysis separates suspected tumor from the rest of the intracranial region, with region analysis used in performing the final tumor labeling. This system has been trained on three volume data sets and tested on thirteen unseen volume data sets acquired from a single MRI system. The KB tumor segmentation was compared with supervised, radiologist-labeled "ground truth" tumor volumes and supervised k-nearest neighbors tumor segmentations. The results of this system generally correspond well to ground truth, both on a per slice basis and more importantly in tracking total tumor volume during treatment over time.

In this paper a method for the objective assessment of burn scars is proposed. The quantitative measures developed in this research provide an objective way to calculate elastic properties of burn scars relative to the surrounding areas. The approach combines range data and the mechanics and motion dynamics of human tissues. Active contours are employed to locate regions of interest and to find displacements of feature points using automatically established correspondences. Changes in strain distribution over time are evaluated. Given images at two time instances and their corresponding features, the finite element method is used to synthesize strain distributions of the underlying tissues. This results in a physically based framework for motion and strain analysis. Relative elasticity of the burn scar is then recovered using iterative descent search for the best nonlinear finite element model that approximates stretching behavior of the region containing the burn scar. The results from the skin elasticity experiments illustrate the ability to objectively detect differences in elasticity between normal and abnormal tissue. These estimated differences in elasticity are correlated against the subjective judgments of physicians that are presently the practice.

This paper presents a novel technique for the estimation of global nonrigid motion without using point correspondences. The complete description of the nonrigid motion of an object involves specifying a displacement vector at each point of the object. Such a description provides a large amount of information which needs to be processed further in order to study the global characteristics of the deformation. Nonrigid motion can be studied hierarchically in terms of a global nonrigid motion and point-by-point local nonrigid motion. The technique presented in this paper gives a method for estimating a global affine or polynomial transformation between two objects. The novelty of the technique lies in the fact that it does not use any point correspondences. Our method uses hyperquadric models to model the data and estimate the global deformation. We show that affine or polynomial transformation between two datasets can be recovered from the hyperquadric parameters. The usefulness of the technique is two-fold. First, it paves the way for viewing nonrigid motion hierarchically in terms of global and local motion. Second, it can be used as a front end to other motion-analysis techniques that assume small motion. For instance, most nonrigid motion analyse's algorithms make some assumptions on the type of nonrigid motion (conformal motion, small motion, etc) that are not always satisfied in practice. When the motion between two datasets is large, our algorithm can be used to estimate the affine transformation (which includes scale and shear) or a polynomial transformation between the two datasets which can then be used to warp the first dataset closer to the second so as to satisfy the small motion assumption. We present experiment results with real and synthetic 2D and 3D data.

Proposes a method for the estimation of point correspondences between boundaries undergoing nonrigid motion. The algorithm works in two stages. In the first stage, a global estimate of the nonrigid motion is obtained using hyperquadric models. The second stage uses this estimate to remove the global nonrigid motion (scale, shear, etc.) and then compute point correspondences between the two datasets assuming small deformations. The global part of the nonrigid motion (scale, shear, rotation and translation) is estimated by modeling the object with hyperquadrics and estimating the transformation between the hyperquadric parameters. Point correspondences are then estimated by using differential geometric properties during small deformations. Experimental results with real data are presented.

Red tides on the west Florida shelf have significant economic and public health effects. Tracking the phytoplankton bloom, known as red tide, is important to understanding the phenomena. In this paper, a knowledge-based approach to automatic classification of Coastal Zone Color Scanner satellite images is developed. The Coastal Zone Color Scanner or CZCS images are initially segmented by the unsupervised mr-FCM algorithm then an expert system utilizes rules, and an iterative clustering process, to recognize case I (deep) water, case II (shallow) water and red tide by searching for expected features. The results show that this system is effective in recognizing images with red tide and segmenting the red tide.

A methodology for evaluating range image segmentation algorithms is proposed. This methodology involves (1) a common set of 40 laser range finder images and 40 structured light scanner images that have manually specified ground truth and (2) a set of defined performance metrics for instances of correctly segmented, missed, and noise regions, over- and under-segmentation, and accuracy of the recovered geometry. A tool is used to objectively compare a machine generated segmentation against the specified ground truth. Four research groups have contributed to evaluate their own algorithm for segmenting a range image into planar patches.

Knowledge about required functionality of an object can be used as an effective representation for a generic object category (e.g. "chair", "cup", or "hammer"). This approach to object representation and recognition has recently become an active area of research. We explore a scenario in which a robot senses the environment to obtain an initial partial shape model of an object. If the information in this initial model is not sufficient to hypothesize a possible function for the object, then additional view(s) may be suggested. Once a possible function is hypothesized, a plan is formulated for interacting with the object to confirm that its material properties are compatible with the hypothesized function. The module for reasoning about partial shape models has been evaluated on over 200 shape models acquired from range images. The module for carrying out a function verification plan has been evaluated in a simulated environment using the ThingWorld (TW) system.

Time-varying multispectral observations of clouds from meteorological satellites are used to estimate cloud-top heights (structure) and cloud winds (semi-fluid motion). Stereo image pairs over several time steps were acquired by two geostationary satellites with synchronized scanning instruments. Cloud-top height estimation from these image pairs is performed using an improved automatic stereo analysis algorithm on a massively parallel Maspar computer with 16 K processors. A new category of motion behavior known as semi-fluid motion is described for modeling cloud motions and an automatic algorithm for extracting semi-fluid motion is developed to track cloud winds. The time sequential dense estimates of cloud-top height depth maps in conjunction with intensity data are used to estimate local semi-fluid motion parameters for cloud tracking. Both stereo disparities and motion correspondences are estimated to sub-pixel accuracy. The Interactive Image SpreadSheet (IISS) is a new versatile visualization tool that was enhanced to analyze and visualize the results of the stereo analysis and semi-fluid motion estimation algorithms. Experimental results using time-varying data of the visible channel from two satellites in geosynchronous orbit is presented for the Hurricane Frederic.>

A new approach is presented for extracting an explicit 3D shape model from a single range image. One novel aspect is that the model represents both observed object surfaces, and surfaces which bound the volume of occluded space. Another novel aspect is that the approach does not require that the range image segmentation be perfect. The low-level segmentation may be such that the model-building process encounters topology versus geometry conflicts. The model-building process is designed to be "fail soft" in the face of such problems. The portion of the 3D model where a problem presents itself is "glued" together in a manner meant to minimize the disturbance in the 3D shape. The goal is to produce a valid boundary-representation which can be processed by higher-level routines. A third novel aspect of this work is that the implementation has been evaluated on over 200 real range images of polyhedral objects, with no operator intervention and all parameters held constant, and obtained a 97% success rate in creating valid b-reps.>

This paper discusses the applications of hyperquadric models in computer vision and focuses on their recovery from range data. Hyperquadrics are volumetric shape models that include superquadrics as a special case. A hyperquadric model can be composed of any number of terms and its geometric bound is an arbitrary convex polytope. Thus, hyperquadrics can model more complex shapes than superquadrics. Hyperquadrics also possess many other advantageous properties (compactness, semilocal control, and intuitive meaning). Our proposed algorithm starts with a rough fit using only six terms in 3D (four in 2D) and adds additional terms as necessary to improve fitting. Suitable constraints are used to ensure proper convergence. Experimental results with real 2D and 3D data are presented.>

The use of spherical harmonics for rigid and nonrigid shape representation is well known. This paper extends the method to surface harmonics defined on domains other than the sphere and to four-dimensional spherical harmonics. These harmonics enable us to represent shapes which cannot be represented as a global function in spherical coordinates, but can be in other coordinate systems. Prolate and oblate spheroidal harmonics and cylindrical harmonics are examples of surface harmonics which we find useful. Nonrigid shapes are represented as functions of space and time either by including the time-dependence as a separate factor or by using four-dimensional spherical harmonics. This paper compares the errors of fitting various surface harmonics to an assortment of synthetic and real data samples, both rigid and nonrigid. In all cases we use a linear least-squares approach to find the best fit to given range data. It is found that for some shapes there is a variation among geometries in the number of harmonics functions needed to achieve a desired accuracy. In particular, it was found that four-dimensional spherical harmonics provide an improved model of the motion of the left ventricle of the heart.>

Recovery of motion parameters and point correspondences is a fundamental problem in computer vision. Although a great deal of research has been done in solving rigid motion, nonrigid motion analysis has only recently been addressed and is gaining interest due to its wide range of applications. This paper introduces a novel method for estimating motion parameters and point correspondences between surfaces under small nonrigid deformations. It uses the changes in differential geometric properties of surface under motion. Simulations are performed by generating nonrigid motion on an ellipsoidal data to illustrate performance and accuracy of derived algorithms, Then, the algorithm is tested on the sequence of facial range images. The motion parameters generated by the algorithm has also been used do detect the abnormality in cardiac images.>

This paper presents a novel multiparadigm segmentation method based upon knowledge based clustering with reclustering. The techniques described enhance unsupervised classification and achieve pattern labeling. First domain knowledge is utilized to decide where and how a clustering algorithm is applied, then clustering is iteratively applied to focus-of-attention patterns with the knowledge of how many expected classes there are in those patterns and the prototypical patterns of a class. Examples showing clustering improvements are given from brain MRI's and satellite images.

We present a robust technique for the estimation of deformable hyperquadrics from images. Hyperquadrics are volumetric shape models that include superquadrics as a special case. Recovering hyperquadric parameters is difficult not only due to the existence of many local minima in the error function but also due to the existence of an infinite number of global minima (with zero error) that do not correspond to any meaningful shape. An algorithm that minimizes the error-of-fit function without using techniques similar to those presented here will often find itself stuck in "meaningless" minima, even with good initialization. Our algorithm exhibits good convergence behavior and is largely insensitive to initialization.

This paper describes a definition of the range image segmentation (of polyhedral scenes) problem, a data set to use in evaluation, a method for specifying ground truth, and a set of metrics to classify segmentation results against ground truths.>

Presents a new approach for the automatic tracking of SPAMM (Spatial Modulation of Magnetization) grid in cardiac MR images and consequent estimation of deformation parameters. The tracking is utilized to extract grid points from MR images and to establish correspondences between grid points in images taken at consecutive frames. These correspondences are used with a thin plate spline model to establish a mapping from one image to the next. This mapping is then used for motion and deformation estimation. Spatio-temporal tracking of SPAMM grid is achieved by using snakes-active contour models with an associated energy functional. The authors present a minimizing strategy which is suitable for tracking the SPAMM grid. By continuously minimizing their energy functionals, the snakes lock on to and follow the in-slice motion and deformation of the SPAMM grid. The proposed algorithm was tested with excellent results on 123 images (three data sets each a multiple slice 2D, 16 phase Cine study, three data sets each a multiple slice 2D, 13 phase Cine study and three data sets each a multiple slice 2D, 12 phase Cine study).

Superquadric is an implicit model which was recently introduced and successfully applied in computer vision research. The authors introduce its generalization, the use of the hyperquadric models, for computer vision applications, and focus on its utilization for shape recovery from range data. The hyperquadric model can be composed of any number of terms. Its geometric bound is an arbitrary convex polyhedron, and thus it can describe more complex shapes than the superquadric. A fitting method is proposed that starts with a rough fit with only two terms in the 2-D case or three terms in the 3-D case, and then adds additional terms to improve the fit. The experiments indicate that the use of hyperquadrics is a promising paradigm for shape representation and recovery in computer vision.>

Currently the aspect graph is computed from the theoretical standpoint of perfect resolution in object shape, the viewpoint and the projected image. This means that the aspect graph may include details that an observer could never see in practice. Introducing the notion of scale into the aspect graph framework provides a mechanism for selecting a level of detail that is "large enough" to merit explicit representation. This effectively allows control over the number of nodes retained in the aspect graph. This paper introduces the concept of the scale space aspect graph, defines three different interpretations of the scale dimension, and presents a detailed example for a simple class of objects, with scale defined in terms of the spatial extent of features in the image.>

A physically based modeling method that uses adaptive-size meshes to model surfaces of rigid and nonrigid objects is presented. The initial model uses an a priori determined mesh size. However, the mesh size increases or decreases dynamically during surface reconstruction to locate nodes near surface areas of interest (like high curvature points) and to optimize the fitting error. Further, presented with multiple 3-D data frames, the mesh size varies as the data surface undergoes nonrigid motion. This model is used to reconstruct 3-D surfaces, analyze the nonrigid motion, track the corresponding points in nonrigid motion, and create graphic animation and visualization. The method was tested on real range data, on simulated nonrigid motion, and on real data for the left ventricular motion.>

Presents a knowledge-based approach to automatic classification and tissue labeling of 2D magnetic resonance (MR) images of the human brain. The system consists of 2 components: an unsupervised clustering algorithm and an expert system. MR brain data is initially segmented by the unsupervised algorithm, then the expert system locates a landmark tissue or cluster and analyzes it by matching it with a model or searching in it for an expected feature. The landmark tissue location and its analysis are repeated until a tumor is found or all tissues are labeled. The knowledge base contains information on cluster distribution in feature space and tissue models. Since tissue shapes are irregular, their models and matching are specially designed: 1) qualitative tissue models are defined for brain tissues such as white matter; 2) default reasoning is used to match a model with an MR image; that is, if there is no mismatch between a model and an image, they are taken as matched. The system has been tested with 53 slices of MR images acquired at different times by 2 different scanners. It accurately identifies abnormal slices and provides a partial labeling of the tissues. It provides an accurate complete labeling of all normal tissues in the absence of large amounts of data nonuniformity, as verified by radiologists. Thus the system can be used to provide automatic screening of slices for abnormality. It also provides a first step toward the complete description of abnormal images for use in automatic tumor volume determination.

Currently the aspect graph is computed under the assumption of perfect resolution in the viewpoint, the projected image, and the object shape. Visual detail is represented that an observer might never see in practice. By introducing scale into this framework, a mechanism is provided for selecting levels of detail that are large enough to merit explicit representation, effectively allowing control over the size of the aspect graph. To this end the scale space aspect graph is introduced, and an interpretation of the scale dimension in terms of the spatial extent of image features is considered. A brief example is given for polygons in a plane.>

Adaptive-size physically based models suitable for nonrigid motion analysis are presented. The mesh size increases or decreases dynamically during the surface reconstruction process to locate nodes near surface areas of interests (like high curvature points) and to optimize the fitting error. A priori information about nonrigidity can be included so that the surface model deforms to fit moving data points while preserving some basic nonrigid constraints (e.g. isometry or conformality). Implementation of the proposed algorithm with and without isometric/conformal constraints is presented. Performance and accuracy of derived algorithms are demonstrated on data simulating deforming ellipsoidal and bending planar shapes. The algorithm is applied to the real range data for bending paper and to volumetric temporal left ventricular data.>

A method for the estimation of point correspondences on a surface undergoing nonrigid motion, based on changes in Gaussian curvature, is described. An approach for estimating the point correspondences and stretching of a surface undergoing conformal motion with constant (homothetic), linear, or polynomial stretching is proposed. Small motion assumption is utilized to hypothesize all possible point correspondences. Curvature changes are then computed for each hypothesis. The difference between computed curvature changes and the one predicted by the conformal motion assumption is calculated. The hypothesis with the smallest error gives point correspondences between consecutive time frames. Simulations performed on ellipsoidal data illustrate the performance and accuracy of derived algorithms. The algorithm is applied to volumetric CT data of the left ventricle of a dog's heart.>

A curvature-based approach to estimating nonrigid deformation of moving surfaces is described. Conformal motion can be characterized by stretching of the surface. At each point this stretching is equal in all directions, but is different for different points. This stretching function can be defined as an additional (with global translation and rotation) motion parameter. An algorithm for local stretching recovery from Gaussian curvature based on polynomial approximations of the stretching function is presented. These methods require point correspondences between time frames, but not the complete knowledge of nonrigid transformation.>

The authors describe an algorithm which uses a Gaussian and mean curvature profile for extracting special points on terrain and then use these points for recognition of particular regions of the terrain. The Gaussian and mean curvatures are chosen because they are invariant under rotation and translation. In the Gaussian and mean curvature image, the points of maximum and minimum curvature are extracted and used for matching. The stability of the position of those points in the presence of noise and with resampling is investigated. The input for this algorithm consists of 3-D digital terrain data. Curvature values are calculated from the data by fitting a quadratic surface over a square window and calculating directional derivatives of this surface. A method of surface fitting which is invariant to coordinate system transformation is suggested and implemented. The algorithm is tested with and without the presence of noise, and its performance is described.>

An algorithm is presented which uses Gaussian curvature for extracting special points on the terrain, and then uses these points for recognition of particular regions of the terrain. The Gaussian curvature is chosen because it is invariant under isometry, which includes rotation and translation. In the Gaussian curvature image, the points of maximum and minimum curvature are extracted and used for matching. The stability of the position of these points in the presence of noise with resampling is investigated. The Gaussian curvature is calculated from the 3-D digital terrain data by fitting a quadratic surface over a square window and calculating directional derivatives of this surface. A method of surface fitting which is invariant to coordinate system transformation is suggested and implemented. This method involves finding an optimal directional in which the fitting is performed.>

A description is given of a curvature-based approach to motion analysis of nonrigid surfaces. Based on changes in the mean and Gaussian curvatures during the motion, it is possible to classify the motion of a surface at each point as rigid, isometric, homothetic, conformal, and general (nonconformal). The general theory of curvature changes for all types of motions is presented. Then the formula specifying changes in the Gaussian curvature during homothetic motion is derived. Also, two special cases of nonrigid motion are considered. The first case deals with piecewise rigid motion. An algorithm is presented which separates a piecewise rigid surface into its rigid parts. The second case deals with homothetic motion. A method is given to isolate regions of the surface where the homothetic assumption is violated. Both algorithms are tested on simulated data and results.>