Fine-grained grocery object recognition is an important computer vision problem with broad applications in automatic checkout, in-store robotic navigation, and assistive technologies for the visually impaired. Existing datasets on groceries are mainly 2D images. Models trained on these datasets are limited to learning features from the regular 2D grids. While portable 3D sensors such as Kinect were commonly available for mobile phones, sensors such as LiDAR and TrueDepth, have recently been integrated into mobile phones. Despite the availability of mobile 3D sensors, there are currently no dedicated real-world large-scale benchmark 3D datasets for grocery. In addition, existing 3D datasets lack fine-grained grocery categories and have limited training samples. Furthermore, collecting data by going around the object versus the traditional photo capture makes data collection cumbersome. Thus, we introduce a large-scale grocery dataset called 3DGrocery100. It constitutes 100 classes, with a total of 87,898 3D point clouds created from 10,755 RGB-D single-view images. We benchmark our dataset on six recent stateof-the-art 3D point cloud classification models. Additionally, we also benchmark the dataset on few-shot and continual learning point cloud classification tasks. Project Page: https://bigdatavision.org/3DGrocery100/.

Despite the introduction of various attention methods for the answer grounding task, they often encounter three common challenges. Firstly, some designs lack the capability to utilize pre-trained networks and fail to benefit from extensive data pre-training. Secondly, certain custom designs are not based on well-established previous models, limiting the network’s learning potential. Lastly, complex designs that impede reimplementation or enhancement. To address these issues, this paper presents a novel architectural block, termed the Embedding Attention Block (EAB). This block re-calibrates channel-wise image feature-maps by explicitly modeling inter-dependencies between the image feature-maps and the image-question-answer embeddings. The visual demonstration showcases how this block filters out irrelevant feature-map channels based on embeddings. Our approach builds upon three key concepts: Semantic Region Proposal, Input Embedding, and Dynamic Region Fusion. We validate our method’s efficacy using the TextVQA-X, VQS, VQA-X, and VizWiz-VQA-Grounding datasets, and carry out several ablation studies to show the effectiveness of our design choices. Notably, our novel network ranked first place on the 2023 VizWiz-VQA-Grounding challenge leaderboard and won the challenge.

Stein’s paradox holds considerable sway in high-dimensional statistics, highlighting that the sample mean, traditionally considered the de facto estimator, might not be the most efficacious in higher dimensions. To address this, the James-Stein estimator proposes an enhancement by steering the sample means toward a more centralized mean vector. In this paper, first, we establish that normalization layers in deep learning use inadmissible estimators for mean and variance. Next, we introduce a novel method to employ the James-Stein estimator to improve the estimation of mean and variance within normalization layers. We evaluate our method on different computer vision tasks: image classification, semantic segmentation, and 3D object classification. Through these evaluations, it is evident that our improved normalization layers consistently yield superior accuracy across all tasks without extra computational burden. Moreover, recognizing that a plethora of shrinkage estimators surpass the traditional estimator in performance, we study two other prominent shrinkage estimators: Ridge and LASSO. Additionally, we provide visual representations to intuitively demonstrate the impact of shrinkage on the estimated layer statistics. Finally, we study the effect of regularization and batch size on our modified batch normalization. The studies show that our method is less sensitive to batch size and regularization, improving accuracy under various setups.

Answer grounding is the task of locating relevant visual evidence for the Visual Question Answering task. While a wide variety of attention methods have been introduced for this task, they suffer from the following three problems: designs that do not allow the usage of pre-trained networks and do not benefit from large data pre-training, custom designs that are not based on well-grounded previous designs, therefore limiting the learning power of the network, or complicated designs that make it challenging to re-implement or improve them. In this paper, we propose a novel architectural block, which we term Sentence Attention Block, to solve these problems. The proposed block re-calibrates channel-wise image feature-maps by explicitly modeling inter-dependencies between the image feature-maps and sentence embedding. We visually demonstrate how this block filters out irrelevant feature-maps channels based on sentence embedding. We start our design with a well-known attention method, and by making minor modifications, we improve the results to achieve state-of-the-art accuracy. The flexibility of our method makes it easy to use different pre-trained backbone networks, and its simplicity makes it easy to understand and be re-implemented. We demonstrate the effectiveness of our method on the TextVQA-X, VQS, VQA-X, and VizWiz-VQA-Grounding datasets. We perform multiple ablation studies to show the effectiveness of our design choices.

Data augmentations are commonly used to increase the robustness of deep neural networks. In most contemporary research, the networks do not decide the augmentations; they are task-agnostic, and grid search determines their magnitudes. Furthermore, augmentations applicable to lower-dimensional data do not easily extend to higher-dimensional data and vice versa. This paper presents an auto-augmenter for images and meshes (AIM) that easily incorporates into neural networks at training and inference times. It Jointly optimizes with the network to produce constrained, non-rigid deformations in the data. AIM predicts sample-aware deformations suited for a task, and our experiments confirm its effectiveness with various networks.

Recent deep learning-based contour detection studies show high accuracy in single-class boundary detection problems. However, this performance does not translate well in a multi-class scenario where continuous contours are required. Our research presents CU-Net, a U-Net-based network with residual-net encoders which can produce accurate and uninterrupted contour lines for multiple classes. The critical factor behind this concept is our continuity module, containing an interpolation layer and a novel activation function that converts discrete signals into smooth contours. We find the application of our approach in medical imaging problems like retinal layer segmentation from optical coherence tomography (OCT) scans. We applied our method to an expert annotated OCT dataset of children with sickle-cell disease. To compare with benchmarks, we evaluated our network on DME and HC-MS datasets. We achieved an overall mean absolute distance of 6.48 ± 2.04µM and 1.97 ± 0.89µM, respectively 1.03 and 1.4 times less than the current state-of-the-art.

The actin cytoskeleton plays a fundamental role in biological processes, and its shape continuously changes. Under different stimuli, the actin cytoskeleton can quickly polymerize and dissolve. Quantifying their changes among different time frames is necessary to understand the mechanism of actin filaments. Confocal microscopy makes it possible to image thin filaments, but quantifying the actin cytoskeletal organization’s changes remains challenging. This paper proposes deep learning facilitated pipeline to extract individual filaments from microscopic images. Then we model these extracted filaments as sequences of data points and adopt a curve clustering algorithm to cluster them by their shapes. We use the histogram of curve clusters to represent the actin cytoskeleton organization in each image and quantify the changes by measuring the distances between frequency distributions. Our experiments show that our method achieves better actin filament extraction results and provides an alternative way to reveal the shape-shifting of actin cytoskeleton organizations.

Studies show Visual Question Answering (VQA) systems are valuable tools for people with visual impairments to quickly obtain information from an image. In this poster, we present our ongoing work towards developing uses of live photos to mitigate quality issues in photography from people with visual impairments for VQA. New contributions building on our prior research include an expanded live photos dataset, a more in-depth analysis of VQA results, and an analysis of features of our live photos compared with existing data collected from people with visual impairments. We show that live photos are a promising method for improving accuracy of VQA and that our sample live photos mimic the types of images taken in real-world settings by people with visual impairments for the task of VQA.

Unlike images, meshes are irregular and unstructured. Thus, it is not trivial to extend existing image-based deep learning approaches for mesh analysis. In this paper, inspired by dilated convolutions for images, we proffer dilated convolutions for meshes. Our Dilated Mesh Convolution (DMC) unit inflates the kernels’ receptive field without increasing the number of learnable parameters. We also propose a Stacked Dilated Mesh Convolution (SDMC) block by stacking DMC units. It considers spatial regions around mesh faces’ at multiple scales while summarizing the neighboring contextual information. We accommodated SDMC in MeshNet to classify 3D meshes. Experimental results demonstrate that this redesigned model significantly improves classification accuracy on multiple data sets. Code is available at https://github.com/VimsLab/DMC.

Polygon meshes are a popular representation in computer graphics. They efficiently provide delineations of complex 3D shapes. However, their irregular structure hinders mesh analysis efforts in deep learning frameworks; few neural networks exist to describe meshes. MeshNet is a pioneer in this direction. In this paper, we propose a novel neural network that is substantially deeper than its MeshNet predecessor. This increase in depth is achieved through our specialized convolution and pooling blocks that operate on mesh faces. Our network named MeshNet++ learns local structures at multiple scales and is also robust to shortcomings of mesh decimation. We evaluated it for the shape classification task on various data sets, and results significantly higher than state-of-the-art were observed. In particular, results demonstrated that even a small number of examples suffice for training MeshNet++. Our code is available at https://github.com/VimsLab/MeshNet2.

The actin filament plays a fundamental role in numerous cellular processes such as cell growth, proliferation, migration, division, and locomotion. The actin cytoskeleton is highly dynamical and can polymerize and depolymerize in a very short time under different stimuli. To study the mechanics of actin filament, quantifying the length and number of actin filaments in each time frame of microscopic images is fundamental. In this paper, we adopt a Convolutional Neural Network (CNN) to segment actin filaments first, and then we utilize a modified Resnet to detect junctions and endpoints of filaments. With binary segmentation and detected keypoints, we apply a fast marching algorithm to obtain the number and length of each actin filament in microscopic images. We have also collected a dataset of 10 microscopic images of actin filaments to test our method. Our experiments show that our approach outperforms other existing approaches tackling this problem regarding both accuracy and inference time.

Analysis of sea ice image rasters presents exciting challenges to the computer vision community, and results from such visual analysis can offer novel insights to domain experts studying sea ice. In recent years, there is a surge in the production of geospatial data sets on sea ice along with the methods for their analysis. As numerous providers independently provision geospatial data sets, they are found in varying formats. Also, there is often limited collaboration between the intra-domain and inter-domain experts analyzing these data sets. To tackle these aforementioned challenges, we have developed the cPSITRES system using open-source technologies and libraries to serve cruise-based high-resolution spatiotemporal Big Data on sea ice along with relevant unstructured public data. The system enables its users to perform complex analysis on the provided data sets and also serves as a collaborative platform for the rapid discovery of open-source methods analyzing sea ice. We also showcase a few exemplary traditional and state-of-the-art computer vision methods for the analysis of data provided by cPSITRES.

Estimating physical activity (PA) intensity and energy expenditure (EE) is a problem that typically requires the use of wearable sensors such as a heart rate monitor, or accelerometer. We investigate the accuracy of a computer vision system using videos recorded from a pair of wearable video glasses to estimate PA strength and EE automatically using age, gender, speed, and activity cues. Age and gender are obtained using the Deep EXpectation network, while activity is estimated from joint angles and movement speed. We also present results on a study of 50 participants performing four different activities while measuring corresponding features of interest such as height, weight, age, sex, and ground truth EE and PA strength data collected via accelerometer. We present both the results of each computer vision subsystem and overall accuracy of the PA strength estimation (89.5%) and the average EE difference (1.96 kCal/min).

Multi-modal color-thermal matching is a difficult task due to innate appearance differences. Previous methods either use hand crafted features such as Histogram of Oriented Gradients (HOG), or use deep learning within a limited domain (faces, pedestrians). We improve on the state-of-the-art methods by creating a novel deep residual pseudo-siamese architecture trained on in-the-wild data. The siamese branches of the architecture learn to process each modality in a modular fashion, and the residual blocks allow for a deeper network without reducing branch dimensionality too quickly or making training infeasible. We compare our proposed network against other state-of-the-art networks, trained and tested on the CATS and KAIST dataset, respectively. Our experiments show an increase in performance when using the proposed network. The WILDCAT model, along with training and testing code, will be made available upon publication.

Motivation: Images convey essential information in biomedical publications. As such, there is a growing interest within the bio-curation and the bio-databases communities, to store images within publications as evidence for biomedical processes and for experimental results. However, many of the images in biomedical publications are compound images consisting of multiple panels, where each individual panel potentially conveys a different type of information. Segmenting such images into constituent panels is an essential first step toward utilizing images. Results: In this article, we develop a new compound image segmentation system, FigSplit, which is based on Connected Component Analysis. To overcome shortcomings typically manifested by existing methods, we develop a quality assessment step for evaluating and modifying segmentations. Two methods are proposed to re-segment the images if the initial segmentation is inaccurate. Experimental results show the effectiveness of our method compared with other methods. Availability and implementation: The system is publicly available for use at: https://www.eecis.udel.edu/~compbio/FigSplit. The code is available upon request. Contact: [email protected]. Supplementary information: Supplementary data are available online at Bioinformatics.

Shape registration is the process of aligning one 3D model to another. Most previous methods to align shapes with no known correspondences attempt to solve for both the transformation and correspondences iteratively. We present a shape registration approach that solves for the transformation using fuzzy correspondences to maximize the overlap between the given shape and the target shape. A coarse to fine approach with Levenberg-Marquardt method is used for optimization. We show our approach is robust and outperforms other state of the art methods when point clouds are noisy, sparse, and have non-uniform density. Our approach is generic, and we illustrate this by showing that it can also be used for 2D-3D alignment. Experiments show that our method is more robust to initialization and can handle larger changes in scale and rotation than other methods.

Stereo matching is a well researched area using visibleband color cameras. Thermal images are typically lower resolution, have less texture, and are noisier compared to their visible-band counterparts and are more challenging for stereo matching algorithms. Previous benchmarks for stereo matching either focus entirely on visible-band cameras or contain only a single thermal camera. We present the Color And Thermal Stereo (CATS) benchmark, a dataset consisting of stereo thermal, stereo color, and cross-modality image pairs with high accuracy ground truth (

Human age estimation plays an important role inhuman facial image analysis. Aging feature representation is one of the widely studied problems in this topic. Convolutional map (bio-inspired features, or BIF) has been proven to be the most successful framework, but its manual crafted filters cannot easily capture the complicated facial aging pattern. In this paper, we adopt this convolutional map framework but propose a novel feature learning approach based on convolutional sparse coding (CSC) that can automatically learn to characterize aging signatures. Compared to other popular feature learning approaches like deep convolutional neural network (CNN), we verify that our learning approach can extract localized subtle aging features like CNN, and also significantly reduce the model size. Moreover, we employ the standard deviation (STD)pooling to summarize the aging feature. Finally, the extracted features are fed into a discriminative manifold learning model to obtain more discriminative low-dimensional representations and further improve the computational efficiency. We evaluate our approach over the standard benchmark datasets. The experimental results demonstrate that our approach impressively out performs the state-of-the-art results. The proposed age estimation scheme also performs well in the cross-database age estimation task.

Face recognition under makeup changes is challenging. In this paper, we propose a new hierarchical feature learning framework for face recognition under makeup changes. We observe that hierarchical structures exist among heterogeneous gaps. That is, features tend to be more invariant on the higher level, and less invariant on the lower level. To better model the structures of domain gaps, we seek for transformations of multi-level features by jointly learning level-wise transformations. Specifically, we adopt both strategies of feature-augmentation and feature-transformation, and combine them in a hierarchical framework. For the feature-augmentation strategy, a rich representation is assembled by features from coarse-to-fine levels. For the feature-transformation strategy, transformations are jointly learned to minimize the domain gap in each level. Further, to preserve the hierarchical structure of domain gap, level-wise regularizations are introduced to model the diverse heterogeneous patterns among different feature levels. Experiments show the superiority of our proposed approach over the state-of-the-art methods.

Based on the dynamic structure, we design a system that can perform accurate stromule image segmentation, branch tip detection and tracking automatically. We substitute the user constraints in active contour segmentation by spatial fuzzy c-means clustering for providing more precise segmentation result. Based on the segmented contour after smoothing, we create a surface normal based feature that can accurately detect the branch tips. We further combine normal information together with tip position coordinate to apply ICP to track the branch tips moving path.

Automating the extraction and quantification of features from three-dimensional (3-D) image stacks is a critical task for advancing many applications, such as the analysis of biological structures. The quantification of biological resistance of a plant tissue to fungal infection through the analysis of attributes such as fungal penetration depth, fungal mass, and branching of the fungal network at large scales requires the union of 3D image acquisition and analysis. From an image processing perspective, these tasks reduce to segmentation of vessel-like structures and the extraction of features from their skeletonization. In order to sample multiple infection events for analysis, we have developed an approach we refer to as macroscopic microscopy. However, macroscopic microscopy produces high-resolution image stacks that pose challenges to routine approaches and are difficult for a human to annotate to obtain ground truth data. We present a synthetic hyphal network generator, a comparison of several vessel segmentation methods, and a minimum spanning tree method for connecting small gaps resulting from imperfections in imaging or incomplete skeletonization of hyphal networks. Qualitative results are shown for real microscopic data. We believe the comparison of vessel detectors on macroscopic microscopy data, the synthetic vessel generator, and the gap closing technique are beneficial to the image processing community.

Image tracking provides crucial insight for the image motion, which generates essential information for incremental structure-from-motion reconstruction and camera pose estimation. Typical usages, such as 3D reconstruction and visual odometry, all rely on robust and accurate local feature tracking through consecutive images. Current algorithms realize feature tracking through matching features extracted from discriminant textures in the images, for which distinctive image content is required to obtain accurate feature matching. For images with few textures, usually, an insufficient number of features are extracted to perform reliable tracking in a series of sequential images. We propose a method that makes use of a limited number of discriminate features to explore other features without strong discriminant power. We develop a feature integrating surrounding salient points distribution knowledge, raw pixel value, and coordinate information to discover a significant amount of features in weakly textured areas in an image. We also incorporate epipolar geometry in the feature correspondence calculation by taking the distance from the matching candidate to its corresponding point's epipolar line into account. To reduce the number of unreliable features, we project the estimated 3D points back to the images. The reprojection error is standardized according to the 3D point's depth, which reduces the bias introduced by the object distance to the camera. We conduct experiments on a large dataset of Arctic sea ice images, mainly composed by planes of ices and sea water. The experimental results demonstrate that our method can perform fast and accurate tracking in weakly textured images.

The measurement of variation in characteristics of an organism is referred to as phenotyping, and using image data to extract phenotypes is a rapidly developing area in biological research. For studying host-pathogen interactions, 3D microscopy data can provide useful information about mechanisms of infection and defense. Performing research on a fungal pathogen of a plant, we recently developed methods to image and combine multiple fields of view of microscopy data across a macroscopic scale. This study was focused on using macroscopic microscopy data to digitally extract the top epidermal cell layer of plant leaves and to count the number of fungal spores on the epidermis. This was achieved using an active surface approach to estimate the 3D position of the epidermis and a shape-template matching approach to detect spores. A compact shape representation is proposed to model spore shapes and generate candidate templates for detecting spores. Our experiments show results that indicate strong promise for the proposed approach in studying plant-fungal interactions.

We propose to represent the shape of an organ using a neural network classifier. The shape is represented by a function learned by a neural network. Radial Basis Function (RBF) is used as the activation function for each perceptron. The learned implicit function is a combination of radial basis functions, which can represent complex shapes. The organ shape representation is learned using classification methods. Our testing results show that the neural network shape provides the best representation accuracy. The use of RBF provides a rotation, translation and scaling invariant feature to represent the shape. Experiments show that our method can accurately represent the organ shape.

Recently, shape analysis has become of increasing interest in the medical community due to its potential in capturing the morphological variations across a population. The high quality 3D images captured can be used to extract 3D shape of the organs. 3D models of organs can also be used for training personnel, for visualization during image guided interventions and in simulations. A compact shape model that has implicit and explicit forms will aid in some of these medical use-cases. We propose a compact hybrid shape model as a combination of Extended Superquadrics (ESQ) [1] and Radial basis interpolation function (RBF). The hybrid shape model in its parametric form is given as ( f (θ ,φ)= h(θ ,φ)+g(θ ,φ)). h is the extended superquadric function and g is radial basis interpolation function. The points on the surface of the shape are given by

Material classification is an important area of research in computer vision. Typical algorithms use color and texture information for classification, but there are problems due to varying lighting conditions and diversity of colors in a single material class. In this work we study the use of long wave infrared (i.e. thermal) imagery for material classification. Thermal imagery has the benefit of relative invariance to color changes, invariance to lighting conditions, and can even work in the dark. We collect a database of 21 different material classes with both color and thermal imagery. We develop a set of features that describe water permeation and heating/cooling properties, and test several variations on these methods to obtain our final classifier. The results show that the proposed method outperforms typical color and texture features, and when combined with color information, the results are improved further.

Image-based localization is an essential complement to GPS localization. Current image-based localization methods are based on either 2D-to-3D or 3D-to-2D to find the correspondences, which ignore the real scene geometric attributes. The main contribution of our paper is that we use a 3D model reconstructed by a short video as the query to realize 3D-to-3D localization under a multi-task point retrieval framework. Firstly, the use of a 3D model as the query enables us to efficiently select location candidates. Furthermore, the reconstruction of 3D model exploits the correlations among different images, based on the fact that images captured from different views for SfM share information through matching features. By exploring shared information (matching features) across multiple related tasks (images of the same scene captured from different views), the visual feature's view-invariance property can be improved in order to get to a higher point retrieval accuracy. More specifically, we use multi-task point retrieval framework to explore the relationship between descriptors and the 3D points, which extracts the discriminant points for more accurate 3D-to-3D correspondences retrieval. We further apply multi-task learning (MTL) retrieval approach on thermal images to prove that our MTL retrieval framework also provides superior performance for the thermal domain. This application is exceptionally helpful to cope with the localization problem in an environment with limited light sources.

Images form a rich information source, which remains underutilized in biomedical document classification. We present here work that uses both image- and text-based features in order to identify articles of interest, in this case, pertaining to cis-regulatory modules in the context of gene-networks. Extending on our new idea, which we have recently introduced, of using OCR-based features to identify DNA contents in images, we combine image and text based classifiers to categorize documents as relevant or irrelevant to cis-regulatory modules. Using a set of hundreds of articles, marked by experts as relevant or irrelevant to cis-regulatory modules, we train/test image and text based classifiers, as well as classifiers integrating both. Our results indicate that the latter show the best performance with Recall, F-measure and Utility measures all above 0.9, demonstrating the significance of incorporating image data, and specifically OCR-based features, into the document categorization process. Moreover, the use of character distribution properties to represent images is directly relevant to other biomedical images containing text (e.g. RNA, proteins). Diagrams and other images containing text are also prevalent outside the biomedical domain, hence the work stands to be applicable and beneficial in other application areas.

Calibration of stereo cameras is important for accurate 3D reconstruction. For standard color cameras there are many available tools and algorithms for accurate calibration, such as detecting corners of chessboard patterns on planar calibration boards. When viewed in thermal imagery, these chessboard patterns are difficult to detect due to uniform temperature between the white and black squares. Previous techniques involve creating a custom calibration board using multiple materials. We propose improvements to a method that does not require a custom calibration board. Our method is made more reliable by using an iterative pre-processing technique to enhance contrast and a ceramic tile backing to retain heat longer. We present results which show our calibration board retains heat to reliably detect corners for over 10 minutes; our method performs well in real calibration trials.

Underwater objects behind a refractive surface pose problems for traditional 3D reconstruction techniques. Scenes where underwater objects are visible from the surface are commonplace, however the refraction of light causes 3D points in these scenes to project non-linearly. Refractive Stereo Ray Tracing allows for accurate reconstruction by modeling the refraction of light. Our approach uses techniques from ray tracing to compute the 3D position of points behind a refractive surface. This technique aims to reconstruct underwater structures in situations where access to the water is dangerous or cost prohibitive. Experimental results in real and synthetic scenes show this technique effectively handles refraction.

Human age provides key demographic information. It is also considered as an important soft biometric trait for human identification or search. Compared to other pattern recognition problems (e.g., object classification, scene categorization), age estimation is much more challenging since the difference between facial images with age variations can be more subtle and the process of aging varies greatly among different individuals. In this work, we investigate deep learning techniques for age estimation based on the convolutional neural network (CNN). A new framework for age feature extraction based on the deep learning model is built. Compared to previous models based on CNN, we use feature maps obtained in different layers for our estimation work instead of using the feature obtained at the top layer. Additionally, a manifold learning algorithm is incorporated in the proposed scheme and this improves the performance significantly. Furthermore, we also evaluate different classification and regression schemes in estimating age using the deep learned aging pattern (DLA). To the best of our knowledge, this is the first time that deep learning technique is introduced and applied to solve the age estimation problem. Experimental results on two datasets show that the proposed approach is significantly better than the state-of-the-art.

Techniques based on the well-studied approaches of structure from motion and bundle adjustment are very robust for scenes with texture. In scenes with little texture information these approaches can fail. Shape from shading determines the shape of an object up to a scale from a single image, and performs better than structure from motion methods in textureless regions. We propose using Gradient Constrained Interpolation to estimate a dense point cloud where holes are caused by regions of low texture during structure from motion reconstruction. Our technique is demonstrated to show good results in both synthetic and real data and outperforms methods which do not use image information.

Kinship verification has become a very active topic recently. Many kinship verification algorithms have been proposed, however, many problems still need to be solved, such as how to locate familial traits of two individuals with kinship and how to make use of facial geometry information to verify kinship relation. To solve these problems, we propose one feature matching scheme to locate familial traits between two facial images. We also advocate a method using geometry information for kinship verification. This approach achieves more than 17% improvement compared with the state of the art. In addition, preliminary results show that the proposed problems can be accurately addressed when fusing appearance and geometry feature.

Object recognition is an important problem with a wide range of applications. It is also a challenging problem, especially for animal categorization as the differences among breeds can be subtle. In this paper, based on statistical techniques for landmark-based shape representation, we propose to model the shape of dog breed as points on the Grassmann manifold. We consider the dog breed categorization as the classification problem on this manifold. The proposed scheme is tested on a dataset including 8,351 images of 133 different breeds. Experimental results demonstrate the advocated scheme outperforms state of the art approaches by nearly 20%.

We propose a pipelined methods to reduce memory consumption of large-scale Structure-from-Motion reconstruction with the use of unsorted images extracted from photo collection websites. Recent research is able to reconstruct cities based on extracted images from photo collection websites. SIFT feature is used to find the correspondences between two images. For the large-scale reconstruction with unsorted images, the system needs to store all the descriptors and feature points information in memory to search for correspondences. As each SIFT descriptor is a 128 dimensional real-value vector, storing all the descriptors would consume a significant amount of memory. Based on this limitation, we project the high dimensional features into a low-dimensional space using a learned projection matrix. After projection, the distance of the descriptors belonging to the same point in 3D space is decreased; the distance of the descriptors belonging to the different points is increased. Furthermore, we learn a mapping function, which maps the real-value descriptor into binary code. As Hamming descriptors contain only two value options per bit and the length of the descriptor is limited, there are usually multiple descriptors having the same Hamming distance to the query descriptor. In dealing with this problem, we give different weights to each dimension and rank each bit of the Hamming descriptor based on each dimensions discriminant power; this contributes to reduce the ambiguity in matching the descriptors. The experiments show that our method achieves dense reconstruction results with less than 10 percent of the original memory consumption.

In this work, we combine the complementary depth sensors Kinect and stereo image matching to obtain high quality correspondences. Our goal is to obtain a dense disparity map at the spatial and depth resolution of the stereo cameras (4-12 MP). We propose a global optimization scheme, where both the data and smoothness costs are derived using sensor confidences and low resolution geometry from Kinect. A spatially varying search range is used to limit the number of potential disparities at each pixel. The smoothness prior is Based on available low resolution depth from Kinect rather than image gradients, thus performing better in both textured areas with smooth depth and texture-less areas with depth gradient. We also propose a spatially varying smoothness weight to better handle occlusion areas, and the relative contribution of the two energy terms. We demonstrate how the two sensors can be effectively fused to obtain correct scene depth in ambiguous areas, as well as fine structural details in textured areas.

Textureless regions, though error prone in stereo, may contain shading information that may be exploited. Shape from shading (SFS) results relate to world coordinates by arbitrary scaling factors which are difficult to estimate. We propose a method for estimating dense disparities from sparse correspondences using SFS cues. We show that SFS can impose constraints on the gradient of disparity in textureless regions with constant albedo. Gradient Constrained Interpolation (GCI), which solves the estimation problem in one dimension, is presented. We efficiently generate paths between correspondences that cover the image and then use GCI to fill the pixels in between. Results are presented on real and synthetic images, and provide quantitative evaluations to show that the method outperforms baseline methods.

Automatic human age estimation has attracted a great deal of interest in the past few years. Although many advancements have been made by researchers, there are still many challenges: such as age estimation across different image acquisition methods, different expressions, gender and races. The influence due to race and gender seems to be the most common issue, because collecting a large amount of face images with comprehensive racial diversities seems impractical. The performance will degrade when estimating face images of races that differ from the training set. In this work, we present a new scheme to mitigate the influences of race and gender in the problem of age estimation. Our system will contribute a robust solution to solve the problem of age estimation across races and genders. This study is essential for developing a practical age estimation system (with mixture of races and gender.) To evaluate the performance of the proposed algorithm, we run comprehensive experiments on one widely used big database - MORPH-II, which contains more than 55, 000 images. On an average, an improvement of more than 20% has been achieved using the proposed scheme.

Scene flow is the motion of the 3D world, it is used in obstacle avoidance, slow motion interpolation, surveillance, studying human behavior, and much more. A mobile implementation of scene flow can greatly increase the flexibility of scene flow applications. Furthermore, combining multiple scene flows to one panoramic scene flow can aid in these tasks: allowing coverage of dead spots in surveillance, studying human motion from multiple views, or simply obtain a larger motion view of a scene. In this paper, a robust algorithm for building panoramic scene flow obtained from a mobile device is described. Since scene flow is estimated from a mobile device, observer motion is compensated for using least squares fitting over the entire scene. Furthermore, noise is reduced and outliers are eliminated from the 3D motion field using motion model fitting. The results demonstrate the effectiveness of the suggested algorithm for constructing a scene flow panorama from moving sources.

This paper proposes a novel scheme for face recognition from visible images to depth images. In our proposed technique, we adopt Partial Least Square (PLS) to handle correlation mapping between 2D to 3D. A considerable performance improvement is observed compared to using Canonical Correlation Analysis (CCA). To further improve the performance, a fusion scheme based on PLS and CCA is advocated. We evaluate the advocated approach on a popular face dataset-FRGCV2.0. Experimental results demonstrate that the proposed scheme is an effective approach to perform 2D-3D face recognition.

Structure-from-Motion reconstruction is to recover the 3 dimensional structure from 2 dimensional images. Recent research in this field demonstrates the ability to reconstruct cities based on images extracted from a photo collection website; SIFT feature is typically extracted to detect correspondences between images. For the reconstruction of large scale unsorted images, the system is required to store all features and points information in the memory to search for correspondences. As SIFT feature is a 128 dimensional real-valued vector, storing each descriptor would consume a significant amount of memory. Due to this limitation, we propose to project the high-dimensional feature into a lower-dimensional space by using a new learned projection matrix while still maintaining the property of the original features. Hence, the result of this projection will shorten the distance among descriptors of the same point while lengthening the distance among descriptors of different points. These projected descriptors use Hellinger distance for calculation of the similarity between features. Furthermore, we learn a mapping function, which will map the real-valued descriptor into binary code coping with the variation of correspondence searching method. Experiments demonstrate that our method achieve excellent results with limited memory requirement.

We present novel techniques for single-image vignetting correction based on symmetries of two forms of image gradients: semicircular tangential gradients (SCTG) and radial gradients (RG). For a given image pixel, an SCTG is an image gradient along the tangential direction of a circle centered at the presumed optical center and passing through the pixel. An RG is an image gradient along the radial direction with respect to the optical center. We observe that the symmetry properties of SCTG and RG distributions are closely related to the vignetting in the image. Based on these symmetry properties, we develop an automatic optical center estimation algorithm by minimizing the asymmetry of SCTG distributions, and also present two methods for vignetting estimation based on minimizing the asymmetry of RG distributions. In comparison to prior approaches to single-image vignetting correction, our methods do not rely on image segmentation and they produce more accurate results. Experiments show our techniques to work well for a wide range of images while achieving a speed-up of 3-5 times compared to a state-of-the-art method.

Marine radars have been employed to gather data in applications that require near-continuous monitoring and tracking of objects over a wide area from a single viewpoint, independent of weather and light conditions. However, little attention has been paid toward utilizing such systems for the study of long-term phenomena and detecting anomalous environmental events or hazards that may occur infrequently but have potentially significant impacts on coastal populations. In this paper, we concentrate on tracking features in seasonally ice-covered Arctic coastal ocean environments. We have developed tools for automated analysis of ground-based radar images of landfast ice and moving sea ice to extract ice-floe trajectories and velocity fields, delineate the boundary of stable landfast ice, detect events relevant to coastal populations and identify surface vessels. We employ dense and feature-based optical flow approaches to compute motion fields from the images, active contours for delineation of stable landfast ice, and Hidden Markov Models for machine learning based event detection. We present results from the analysis of sample images jointly with a quantitative evaluation of algorithm performance relative to operator-based assessments.

Contour extraction of Drosophila (fruit fly) embryos is an important step to build a computational system for matching expression pattern of embryonic images to assist the discovery of the nature of genes. Automatic contour extraction of embryos is challenging due to severe image variations, including 1) the size, orientation, shape, and appearance of an embryo of interest; 2) the neighboring context of an embryo of interest (such as nontouching and touching neighboring embryos); and 3) illumination circumstance. In this paper, we propose an automatic framework for contour extraction of the embryo of interest in an embryonic image. The proposed framework contains three components. Its first component applies a mixture model of quadratic curves, with statistical features, to initialize the contour of the embryo of interest. An efficient method based on imbalanced image points is proposed to compute model parameters. The second component applies active contour model to refine embryo contours. The third component applies eigen-shape modeling to smooth jaggy contours caused by blurred embryo boundaries. We test the proposed framework on a data set of 8,000 embryonic images, and achieve promising accuracy (88 percent), that is, substantially higher than the-state-of-the-art results.

The purpose of this paper is twofold: 1) to develop a high-resolution sea ice motion tracking system at the geospatial mesoscale (1-100 km2) and 2) to propose an algorithm that measures motion at close proximity to discontinuous regions. Here, we present a motion tracking system that computes differential motion at 400 m resolution and validate the accuracy/precision of this system via four studies. The first study measures the accuracy against displacements measured from in situ Global Positioning System (GPS) buoys deployed at the Sea-ice Experiment: Dynamic Nature of the Arctic (SEDNA) and the Surface HEat Budget of the Arctic Ocean (SHEBA) experiments. The estimates are found to be statistically comparable with GPS, with an average error of 361.9 and 600.6 m for the experiments, respectively. The second study compares the estimated displacements to those measured by the RADARSAT Geophysical Processing System. A precision error of 75.7 m is found between the two motion tracking systems. The third study uses intensity warping of randomly sampled measurements to evaluate discontinuous motion tracking. A one-tailed Wilcoxon signed rank test is used to validate these measurements at α = 0.01. Results from this paper prove that anisotropic smoothing produces significantly smaller errors at discontinuous locations (W = 4240 and p <; 0.001) over conventional isotropic smoothing. The fourth study compares displacements measured by anisotropic smoothing against manual measurements. This paper demonstrates an average reduction of the estimation error by 50 m with the use of anisotropic smoothing over the conventional isotropic smoothing.

Autonomous systems which learn and utilize a limited visual vocabulary have wide spread applications. Enabling such systems to segment a set of cluttered scenes into objects is a challenging vision problem owing to the non-homogeneous texture of objects and the random configurations of multiple objects in each scene. We present a solution to the following question: given a collection of images where each object appears in one or more images and multiple objects occur in each image, how best can we extract the boundaries of the different objects? The algorithm is presented with a set of stereo images, with one stereo pair per scene. The novelty of our work is the use of both color/texture and structure to refine previously determined object boundaries to achieve segmentation consistent with each of the input scenes presented. The algorithm populates an object library, which consists of a 3D model per object. Since an object is characterized both by texture and structure, for most purposes this representation is both complete and concise.

In this paper, we propose a method for estimating the optical center of a camera given only a single image with vignetting. This is accomplished by identifying the center of the vignetting effect in the image through an analysis of semicircular tangential gradients (SCTGs). For a given image pixel, the SCTG is the image gradient along the tangential direction of the circle centered at the currently estimated optical center and passing through the pixel. We show that for natural images with vignetting, the distribution of SCTGs is generally symmetric if the optical center is estimated accurately, but is skewed otherwise. By minimizing the asymmetry of the SCTG distribution with nonlinear optimization, our method is able to obtain reliable estimates of the optical center. Experiments on simulated and real vignetting images demonstrate the effectiveness of this technique.

We cast some new insights into solving the digital matting problem by treating it as a semi-supervised learning task in machine learning. A local learning based approach and a global learning based approach are then produced, to fit better the scribble based matting and the trimap based matting, respectively. Our approaches are easy to implement because only some simple matrix operations are needed. They are also extremely accurate because they can efficiently handle the nonlinear local color distributions by incorporating the kernel trick, that are beyond the ability of many previous works. Our approaches can outperform many recent matting methods, as shown by the theoretical analysis and comprehensive experiments. The new insights may also inspire several more works.

Contour extraction of Drosophila (fruit fly) embryos is an important step to build a computational system for matching expression pattern of embryonic images to assist the discovery of the nature of genes. Automatic contour extraction of embryos is challenging due to severe image variations, including i) the size, orientation, shape and appearance of an embryo of interest; ii) the neighboring context of an embryo of interest (such as non-touching and touching neighboring embryos); and iii) illumination circumstance. In this paper, we propose an automatic framework for contour extraction of the embryo of interest in an embryonic image. The proposed framework contains three components. Its first component applies a mixture model of quadratic curves, with statistical features, to initialize the contour of the embryo of interest. An efficient method based on imbalanced image points is proposed to compute model parameters. The second component applies active contour model to refine embryo contours. The third component applies eigen-shape modeling to smooth jaggy contours caused by blur embryo boundaries. We test the proposed framework on a dataset of 8000 embryonic images, and achieve promising accuracy (88%) that is substantially higher than the-state-of-the-art results.

This paper describes a hierarchical belief propagation implementation in which a `rough' disparity map calculation or motion estimation in higher levels is used to limit the search space and enable the calculation of the desired disparity map/set of motion vectors using a smaller search space than traditional belief propagation. We implement our algorithm on the GPU using the CUDA architecture and explore a number of implementation details with promising results; it is clear that the storage requirements of belief propagation can be significantly reduced using our method without too large of a sacrifice in the accuracy of the results. In addition, we take advantage of the interpolation capabilities built into the GPU in order to retrieve the computed disparities/motion vectors at sub-pixel accuracy without making any change in implementation.

Computer vision techniques have been applied for rapid and accurate structure recovery in many fields. Most methods perform poorly in areas containing little or no texture and in presence of repetitive patterns. We present a portable, cost-effective pattern projector system powered by the flash of a camera, to aid the reconstruction of such areas. No calibration is required between the camera-projector, projector-scene or pattern. We demonstrate the effectiveness of our system on various representative surfaces like metal, clay, porcelain and natural fibres with different inherent colors/textures. A pipeline is presented to automatically generate textured, true scale metric models, which can be used for quantitative studies or visualization. The practability of our system is explored in the specific area of digital archiving of historically significant objects.

In this paper, we propose a method for robustly determining the vignetting function given only a single image. Our method is designed to handle both textured and untextured regions in order to maximize the use of available information. To extract vignetting information from an image, we present adaptations of segmentation techniques that locate image regions with reliable data for vignetting estimation. Within each image region, our method capitalizes on the frequency characteristics and physical properties of vignetting to distinguish it from other sources of intensity variation. Rejection of outlier pixels is applied to improve the robustness of vignetting estimation. Comprehensive experiments demonstrate the effectiveness of this technique on a broad range of images with both simulated and natural vignetting effects. Causes of failures using the proposed algorithm are also analyzed.

In this paper, we propose an online interactive matting algorithm, which we call FuzzyMatte. Our framework is based on computing the fuzzy connectedness (FC) [20] from each unknown pixel to the known foreground and background. FC effectively captures the adjacency and similarity between image elements and can be efficiently computed using the strongest connected path searching algorithm. The final alpha value at each pixel can then be calculated from its FC. While many previous methods need to completely recompute the matte when new inputs are provided, FuzzyMatte effectively integrates these new inputs with the previously estimated matte by efficiently recomputing the FC value for a small subset of pixels. Thus, the computational overhead between each iteration of the refinement is significantly reduced. We demonstrate FuzzyMatte on a wide range of images. We show that FuzzyMatte updates the matte in an online interactive setting and generates high quality matte for complex images.

In this paper, we present a novel single-image vignetting method based on the symmetric distribution of the radial gradient (RG). The radial gradient is the image gradient along the radial direction with respect to the image center. We show that the RG distribution for natural images without vignetting is generally symmetric. However, this distribution is skewed by vignetting. We develop two variants of this technique, both of which remove vignetting by minimizing asymmetry of the RG distribution. Compared with prior approaches to single-image vignetting correction, our method does not require segmentation and the results are generally better. Experiments show our technique works for a wide range of images and it achieves a speed-up of 4’5 times compared with a state-of-the-art method.

Stereo algorithms for structure reconstruction demand accurate disparities with low mismatch errors and false positives. Mismatch errors in large textureless regions force most accurate algorithms to be sparse, with disparities known only in textured regions. We propose a novel method which uses characteristics of the multi-valued disparity to segregate image regions into unambiguous, occluded but textured and regions with low color variation. The disparity in the unambiguous region is calculated using stable matching with local disparity filtering. The disparity is interpolated into other regions by diffusion using unstructured triangulation and method of finite elements for rapid convergence. The boundary conditions for each of the region are appropriately modified so that accurate discontinuities in the disparity are preserved. A comparison of our method with some existing methods through experiments reveal that this algorithm indeed performs significantly better in producing dense accurate disparities.

In this paper, we propose an algorithm to resample coarse vector fields in order to obtain vector fields of a higher density. Unlike the typical linear interpolation scheme, our algorithm attempts to identify streamline characteristics in the flow field, and uses local polynomial parameterization of the flow to perform interpolation. Quantitative validation of the streamline oriented algorithm indicate that the Mean Square Error of the flow field obtained is more accurate than those obtained by bilinear interpolation schemes.

This paper delves into the problem of face recognition using color as an important cue in improving the accuracy of recognition. To perform recognition of color images, we use the characteristics of a 3D color tensor to generate a color LDA subspace, which in turn can be used to recognize a new probe image. To test the accuracy of our methodology, we computed the recognition rate across two color face databases. We observe that the use of the LDA color subspace significantly improves recognition accuracy over the standard gray scale approach without sacrificing computational efficiency.

This paper describes the modeling of a near real time geophysical motion analysis system for the Applied Physics Laboratory Ice Station (APLIS '07) that was held as part of the International Polar Year (IPY 2007-2009). One of the important aspects of the motion analysis system was to help field scientists to plan instrumentation based on the large scale dynamics taking place in the scene. This required that the data flow architecture be simultaneously efficient and accurate. Therefore, in order to provide for an efficient and robust method of analysis, the system was designed within the Unified Modeling Language framework and developed as modules that could be easily scaled to subsequent stages of research. This design-development framework helped in the successful use of the product for ground instrumentation deployment.

In this paper we propose a new framework to simultaneously segment and register lung and tumor in serial CT data. Our method assumes nonrigid transformation on lung deformation and rigid structure on the tumor. We use the B- Spline-based nonrigid transformation to model the lung deformation while imposing rigid transformation on the tumor to preserve the volume and the shape of the tumor. In particular, we set the control points within the tumor to form a control mesh and thus assume the tumor region follows the same rigid transformation as the control mesh. For segmentation, we apply a 2D graph-cut algorithm on the 3D lung and tumor datasets. By iteratively performing segmentation and registration, our method achieves highly accurate segmentation and registration on serial CT data. Finally, since our method eliminates the possible volume variations of the tumor during registration, we can further estimate accurately the tumor growth, an important evidence in lung cancer diagnosis. Initial experiments on five sets of patients ' serial CT data show that our method is robust and reliable.

The nonrigid motion of sea ice is an essential component when describing global climatology models. With the availability of sequential ERS-1 satellite imagery, we estimate high resolution motion and describe the differential characteristics of motion. This characterization is subsequently used to locate critical points, also known as coherent structures, in the flow, thereby reducing the large vector field into a collection of important feature points which essentially describe the flow pattern. In this paper, we build on previous work in an attempt to enhance the characterization of the flow field via visualization and quantification of these coherent structures. We show that the statistical quantification of these structures can provide a greater clarity in our understanding of the physical dynamics taking place within the sea ice

Classical Linear Discriminant Analysis (LDA) is not applicable for small sample size problems due to the singularity of the scatter matrices involved. Regularized LDA (RLDA) provides a simple strategy to overcome the singularity problem by applying a regularization term, which is commonly estimated via cross-validation from a set of candidates. However, cross-validation may be computationally prohibitive when the candidate set is large. An efficient algorithm for RLDA is presented that computes the optimal transformation of RLDA for a large set of parameter candidates, with approximately the same cost as running RLDA a small number of times. Thus it facilitates efficient model selection for RLDA. An intrinsic relationship between RLDA and Uncorrelated LDA (ULDA), which was recently proposed for dimension

Traditional stereo correspondence algorithms rely heavily on the Lambertian model of diffuse reflectance. While this diffuse assumption is generally valid for much of an image, processing of regions that contain specular reflections can result in severe matching errors. In this paper, We address the problem of binocular stereo dense matching in the presence of specular reflections by introducing a novel correspondence measurement which is robust to the specular reflections. Combining our novel correspondence measurement with various local or global optimatization methods, accurate depth can be estimated for both diffuse and specular regions. Unlike the previous works which seek to eliminate or avoid specular reflections using image preprocessing or multibaseline stereo, our approach works in its presence. Our approach is general and fits well in various stereo frameworks. Experiments with both synthetic and real images demonstrate the effectiveness and robustness of our approach.

2-D electrophoresis is the study of expressions of proteins. The technique produces images that contain protein spots, One of the tasks in the analysis of these images is matching of protein spots in two corresponding images for differential expression study. In this paper, we propose an algorithm which integrates a hierarchical based and energy based methods. The hierarchical based method initially finds corresponding pairs of spots. We formulate a new matching energy, which consists of local spot structure similarity, image similarity, and a spatial constraint. The proposed energy is minimized to find corresponding pairs of spots by a greedy based optimization algorithm. We extensively tested our method with synthetic images and real 2-D gel images from different biological experiments.

The study on how to represent appearance instances was the focus in most previous work in face recognition. Little attention, however, was given to the problem of how to select "good" instances for a gallery, which may be called the facial identity representation problem. This paper gives an evaluation of the identity representability of facial expressions. The identity representability of an expression is measured by the recognition accuracy achieved by using its samples as the gallery data. We use feature pixel distributions to represent appearance instances. A feature pixel distribution of an image is based on the number of occurrence of detected feature pixels (corners) in regular grids of an image plane. We propose imbalance oriented redundancy reduction for feature pixel detection. Our experimental evaluation indicates that certain facial expressions, such as the neutral, have stronger identity representability than other expressions, in various feature pixel distributions

In this paper, we present an adaptive appearance based face recognition framework that combines the efficiency of global approaches and the robustness of local approaches together. The framework uses a novel eye locator to select an appropriate scheme for appearance based recognition. The eye locator first locates eye candidates via a new strength assignment, determined by the dissimilarity between the local appearance of an image point and the appearance of its neighboring points. Then the eye locator applies a simple but flexible model (half-circle snake) to the local context of the eye candidates in order to either refine the location of an eye candidate or discard non-eye candidates. We show the performance of our framework by testing on challenging face datasets containing extreme expressions, severe occlusions, and varied lighting conditions

Mean shift based feature space analysis has been shown to be an elegant, accurate and robust technique. The elegance in this non-parametric algorithm is mainly due to its simplicity in performing gradient ascent to estimate the modes in a multidimensional data. One characteristic aspect of mean shift is that the mode estimation is performed at each data point. Since it is important to describe the data in as succinct manner as possible, it is important to focus on modal points in the data instead of every data point. In this paper, we attempt to tackle the mean shift problem through a "mode centric" approach using swarm intelligence. Here, the mode estimation is cast as a problem of goal seeking for the swarm as it moves through the multidimensional data space. Local maxima/minima and plateaus are avoided through information exchange between each member of the swarm, thereby converging at the mode values efficiently

In this paper, a framework of Discriminant Low-dimensional Subspace Analysis (DLSA) method is proposed to deal with the Small Sample Size (SSS) problem in face recognition area. Firstly, it is rigorously proven that the null space of the total covariance matrix, S t, is useless for recognition. Therefore, a framework of Fisher discriminant analysis in a low-dimensional space is developed by projecting all the samples onto the range space of S t. Two algorithms are proposed in this framework, i.e., Unified Linear Discriminant Analysis (ULDA) and Modified Linear Discriminant Analysis (MLDA). The ULDA extracts discriminant information from three subspaces of this lowdimensional space. The MLDA adopts a modified Fisher criterion which can avoid the singularity problem in conventional LDA. Experimental results on a large combined database have demonstrated that the proposed ULDA and MLDA can both achieve better performance than the other state-of-the-art LDA-based algorithms in recognition accuracy. 1

To solve the Small Sample Size (SSS) problem, the recent linear discriminant analysis using the 2D matrix-based data representation model has demonstrated its superiority over that using the conventional vector-based data representation model in face recognition [7]. But the explicit reason why the matrix-based model is better than vectorized model has not been given until now. In this paper, a framework of Generalized 2D Fisher Discriminant Analysis (G2DFDA) is proposed. Three contributions are included in this framework: 1) the essence of these ’2D ’ methods is analyzed and their relationships with conventional ’1D ’ methods are given, 2) a Bilateral and 3) a Kernel-based 2D Fisher Discriminant Analysis methods are proposed. Extensive experiment results show its excellent performance. 1

A general framework for 2D multiframe and 3D surface-to-surface motion estimation is presented in this paper. By viewing a 2D contour sequence as a pseudo 3D surface, we solve the motion estimation problem for 2D multiframe and 3D surface-to-surface in a general framework, by estimating the motion of a ”surface”. The deformation of a ”surface ” is modeled using spline-based motion. This spline-based motion model does not constrain the motion type in the temporal domain for 2D multiframe motion estimation. For 3D motion estimation, we focus on the relationship between the underlying nonrigid motion and 3D surface properties. The spline motion model provides our method certain advantages over other nonrigid shapebased methods. For example, we do not need approximation of the orthogonal parameterization. The small deformation constraint introduced by the previous surface-to-surface motion estimation methods is also relaxed in our method. Experiments on both synthetic and real motion are presented in this paper. 1

Most existing illuminant estimation algorithms work with assumptions of specific source types (e.g. directional light source or point light source). The assumptions bring up two main limitations which significantly restrict their applicabilities: first, the prior knowledge about source types is needed; second, it can not handle complex scenes where multiple source types co-exist. In this paper, we develop a general light source model which is designed to model arbitrary light sources. With the general source model, we are able to estimate multiple illuminants of different types within a single unified framework. We use a plastic sphere to probe both diffuse and specular reflections of a scene. With specular reflections, we estimate geometric parameters of the illuminants by a novel ray tracing and matching algorithm. The diffuse reflections are used to estimate photometric parameters of the illuminants. We also notice that common source types are degenerate cases of the general source model under certain conditions. Therefore, we can approximate light sources with common source types by checking the conditions. The approximations serve as type determinations of light sources, which makes our framework extremely useful for lighting related algorithms which need prior knowledge about light sources. Experiment results on a variety of real images demonstrate the efficiency and accuracy of our algorithm.

Visual media data such as an image is the raw data representation for many important applications. The biggest challenge in using visual media data comes from the extremely high dimensionality. We present a comparative study on spatial interest pixels (SIPs), including eight-way (a novel SIP miner), Harris, and Lucas-Kanade, whose extraction is considered as an important step in reducing the dimensionality of visual media data. With extensive case studies, we have shown the usefulness of SIPs as the low-level features of visual media data. A class-preserving dimension reduction algorithm (using GSVD) is applied to further reduce the dimension of feature vectors based on SIPs. The experiments showed its superiority over PCA.

This paper introduces a novel coarse-to-fine deformable contour optimization framework, which is composed of two main components. The first component uses scale-space and information theories to produce a coarser representation of the input image to be used in a coarse-to-fine optimization scheme. The employment of information theory ensures that maximal image information is propagated to the coarse images and employment of scale spaces provides a mechanism to change the image coarseness locally based on the deformable contour model definition. The second component of this framework uses a novel combination of dynamic programming and gradient descent methods to optimize the contour energy on coarser representations and then use the obtained coarse contour positions in finer optimizations. The motivation in using a combination of dynamic programming and gradient descent method is to take advantage of each method's efficiency and avoid their drawbacks. In order to verify the performance of this framework, we constructed a deformable contour model for the spatiotemporal tracking of closed contours and optimized the model energy under this framework. Experiments on this system performed using synthetic images and real world echocardiographic sequences demonstrated the effectiveness and practicality of this framework.

We present a novel technique for utilizing the Gaussian curvature information in 3D nonrigid motion estimation in the absence of known correspondence. Differential-geometric constraints derived in the paper allow one to estimate parameters of the local affine motion model given the values of Gaussian curvature before and after motion. These constraints can be further combined with the previously known constraints based on the unit normals before and after motion. Our experiments demonstrate that the resulting hybrid algorithm is more accurate than each of its constituents and more accurate than the classical ICP algorithm. We also present a technique for curvilinear orthogonalization of quadratic Monge patches that is essential in our derivation and useful in other applications.

Two novel systems computing dense three-dimensional (3-D) scene flow and structure from multiview image sequences are described in this paper. We do not assume rigidity of the scene motion, thus allowing for nonrigid motion in the scene. The first system, integrated model-based system (IMS), assumes that each small local image region is undergoing 3-D affine motion. Non-linear motion model fitting based on both optical flow constraints and stereo constraints is then carried out on each local region in order to simultaneously estimate 3-D motion correspondences and structure. The second system is based on extended gradient-based system (EGS), a natural extension of two-dimensional (2-D) optical flow computation. In this method, a new hierarchical rule-based stereo matching algorithm is first developed to estimate the initial disparity map. Different available constraints under a multiview camera setup are further investigated and utilized in the proposed motion estimation. We use image segmentation information to adopt and maintain the motion and depth discontinuities. Within the framework for EGS, we present two different formulations for 3-D scene flow and structure computation. One formulation assumes that initial disparity map is accurate, while the other does not. Experimental results on both synthetic and real imagery demonstrate the effectiveness of our 3-D motion and structure recovery schemes. Empirical comparison between IMS and EGS is also reported.

Many real-world datasets contain noise which could degrade the performances of learning algorithms. Motivated from the success of wavelet denoising techniques in image data, we explore a general solution to alleviate the effect of noisy data by wavelet preprocessing for medical/biological data classification. Our experiments are divided into two categories: one is of different classification algorithms on a specific database, and the other is of a specific classification algorithm (decision tree) on different databases. The experiment results show that the wavelet denoising of noisy data is able to improve the accuracies of those classification methods, if the localities of the attributes are strong enough.

We address the problem of non-rigid motion and correspondence estimation in 3D images in the absense of prior domain information. A generic framework is utilized in which a solution is approached by hypothesizing correspondence and evaluting the motion models constructed under each hypothesis. We present and evaluate experimentally ve algorithms that can be used in this approach. Our experiments were carried out on synthetic and real data with ground truth correspondence information. 1

A framework for segmenting multiple objects in an image based on deformable contours is proposed. In this framework, multiple snakes are applied with a new kind of energy called the "group energy." The group energy is introduced to handle sharing of properties across multiple objects in the image. Our framework allows contours of "strong objects" to guide contours of "weak objects" by utilizing deformable templates. We also automatically generate the necessary weighting parameters for energy minimization. A new approach for multiple snake optimization which is based on dynamic programming is also proposed. We applied our framework to the problem of image analysis of gene expression in microarrays. Comprehensive experiments were performed and comparisons were made between the individual energy based method and the proposed group energy based method. Our results are highly encouraging and have many potential applications in a variety of tracking scenarios.

In this paper, novel algorithms computing dense 3D scene flow from multiview image sequences are described. A new hierarchical rule-based stereo matching algorithm is presented to estimate the initial disparity map. Different available constraints under a multiview camera setup are investigated and then utilized in the proposed motion estimation algorithms. We show two different formulations for 3D scene flow computation. One formulation assumes that initial disparity map is accurate while the other does not make this assumption. Image segmentation information is used to maintain the motion and depth discontinuities. Iterative implementations are used to successfully compute 3D scene flow and structure at every point in the reference image. Novel hard constraints are introduced in this paper to make the algorithms more accurate and robust. Promising experimental results are seen by applying our algorithms to real imagery.

Tracking both structure and motion of nonrigid objects from monocular images is an important problem in vision. In this paper, a hierarchical method which integrates local analysis (that recovers small details) and global analysis (that appropriately limits possible nonrigid behaviors) is developed to recover dense depth values and nonrigid motion from a sequence of 2D satellite cloud images without any prior knowledge of point correspondences. This problem is challenging not only due to the absence of correspondence information but also due to the lack of depth cues in the 2D cloud images (scaled orthographic projection). In our method, the cloud images are segmented into several small regions and local analysis is performed for each region. A recursive algorithm is proposed to integrate local analysis with appropriate global fluid model constraints, based on which a structure and motion analysis system, SMAS, is developed. We believe that this is the first reported system in estimating dense structure and nonrigid motion under scaled orthographic views using fluid model constraints. Experiments on cloud image sequences captured by meteorological satellites (GOES-8 and GOES-9) have been performed using our system, along with their validation and analyses. Both structure and 3D motion correspondences are estimated to subpixel accuracy. Our results are very encouraging and have many potential applications in earth and space sciences, especially in cloud models for weather prediction.

Scene flow is the 3D motion field of points in the world. Given N (N>1) image sequences gathered with a N-eye stereo camera or N calibrated cameras, we present a novel system which integrates 3D scene flow and structure recovery in order to complement each other's performance. We do not assume rigidity of the scene motion, thus allowing for non-rigid motion in the scene. In our work, images are segmented into small regions. We assume that each small region is undergoing similar motion, represented by a 3D affine model. Nonlinear motion model fitting based on both optical flow constraints and stereo constraints is then carried over each image region in order to simultaneously estimate 3D motion correspondences and structure. To ensure the robustness, several regularization constraints are also introduced. A recursive algorithm is designed to incorporate the local and regularization constraints. Experimental results on both synthetic and real data demonstrate the effectiveness of our integrated 3D motion and structure analysis scheme.

Inferring both 3D structure and motion of nonrigid objects from monocular images is an important problem in computer vision. The challenges stem not only from the absence of point correspondences but also from the structure ambiguity. In this paper, a hierarchical method which integrates both local patch analysis and global shape descriptions is devised to solve the dual problem of structure and nonrigid motion recovery by using an elastic geometric model-extended superquadrics (ESQ). The nonrigid object of interest is segmented into many small areas and local analysis is performed to recover small details for each small area, assuming that each small area is undergoing similar nonrigid motion. Then, a recursive algorithm is proposed to guide and regularize local analysis with global information by using an appropriate global ESQ model. This local-global hierarchy enables us to capture both local and global deformations accurately and robustly. Experimental results on both simulation and real data are presented to validate and evaluate the effectiveness and robustness of the proposed approach.

We present a novel approach to estimate and analyze 3D fluid structure and motion of clouds from multi-spectrum 2D cloud image sequences. Accurate cloud-top structure and motion are very important for a host of meteorological and climate applications. However, due to the extremely complex nature of cloud fluid motion, classical nonrigid motion analysis methods are insufficient for solving this particular problem. In this paper, two spectra of satellite cloud images are utilized. The high-resolution visible channel is first used to perform cloud tracking by using a recursive algorithm which integrates local motion analysis with a set of global fluid constraints, defined according to the physical fluid dynamics. Then, the infrared channel (thermodynamic information) is incorporated to post-process the cloud tracking results in order to capture the cloud density variations and small details of cloud fluidity. Experimental results on GOES (Geostationary Operational Environmental Satellite) cloud image sequences are presented in order to validate and evaluate both the effectiveness and robustness of our algorithm.

This paper describes a novel system for 3D head tracking under partial occlusion from 2D monocular image sequences. In this system, the extended superquadric (ESQ) is used to generate a geometric 3D face model in order to reduce the shape ambiguity. Optical flow is then employed with this model to estimate the 3D rigid motion. To deal with occlusion, a new motion segmentation algorithm using motion residual error analysis is developed. The occluded areas are successfully detected and discarded as noise by the system. Also, accumulation error is heavily reduced by a new post-regularization process based on edge flow. This makes the system more stable over long occlusion image sequences. To show the accuracy, the system is applied on a synthetic occlusion sequence and comparisons with the ground truth are reported. To show the robustness, experiments on long occlusion image sequences, including synthetic and real ones, are reported.

Application of dynamic programming to the deformable contours has many advantages, such as guaranteed optimality and numerical stability. However, long execution times of these methods almost always force researchers to use dynamic programming in combination with multiresolution methods. Multiresolution methods shorten the execution time by subsampling the original images after an application of a smoothing filter. However, this speedup comes at the expense of contour optimality due to the loss of details in the decreased resolution. In this paper, we present a new multi-level framework for deformable contour optimization, which can achieve faster optimization times and performs better than current multiresolution methods. To form the new levels, this method uses a very efficient algorithm to segment the original images with respect to the deformable contour external energy instead of subsampling. An exhaustive search on these segments is carried out by dynamic programming. A novel gradient descent algorithm is employed to find optimal internal energy for large image segments, where the external energy remains constant due to segmentation. We also introduce a new algorithm to pass the contour information more precisely between the levels. We present an analysis of time and performance comparisons with the current multiresolution methods by the experiments done on variety of medical images, which confirmed efficiency and accuracy of our framework.

Superquadrics are a family of parametric shapes which can model a diverse set of objects. They have received significant attention because of their compact representation and robust methods for recovery of 3D models. However, their assumption of intrinsical symmetry fails in modeling numerous real-world examples such as human, body, animals, and other naturally occurring objects. In this paper, we present a novel approach, which is called extended superquadric to extend superquadric's representation power with exponent functions. An extended superquadric model can be deformed in any direction because it extends the exponents of superquadrics from constants to functions of the latitude and longitude angles in the spherical coordinate system. Thus extended superquadrics can model more complex shapes than superquadrics. It also maintains many desired properties of superquadrics such as compactness controllability, and intuitive meaning, which are all advantageous for shape modeling, recognition, and reconstruction. In this paper, besides the use of extended superquadrics for modeling, we also discuss our research into the recovery of extended superquadrics from 3D information (reconstruction). Experimental results of fitting extended superquadrics to 3D real data are presented. Our results are very encouraging and indicate that the use of extended superquadric is a promising paradigm for shape representation and recovery in computers vision and has potential benefits for the generation of synthetic images for computer graphics.

Image sequences capturing Hurricane Luis through meteorological satellites (GOES-8 and GOES-9) are used to estimate hurricane-top heights (structure) and hurricane winds (motion). This problem is difficult not only due to the absence of correspondence but also due to the lack of depth cues in the 2D hurricane images (scaled orthographic projection). In this paper, we present a structure and motion analysis system, called SMAS. In this system, the hurricane images are first segmented into small square areas. We assume that each small area is undergoing similar nonrigid motion. A suitable nonrigid motion model for cloud motion is first defined. Then, non-linear least-square method is used to fit the nonrigid motion model for each area in order to estimate the structure, motion model, and 3D nonrigid motion correspondences. Finally, the recovered hurricane-top heights and winds are presented along with an error analysis. Both structure and 3D motion correspondences are estimated to subpixel accuracy. Our results are very encouraging, and have many potential applications in earth and space sciences, especially in cloud models for weather prediction.

We propose a novel method for continuous 3D depth recovery and tracking using calibrated stereo. The method integrates stereo correspondence, surface reconstruction and tracking by using a new single deformable dual mesh optimization, resulting in simplicity, robustness and efficiency. In order to combine stereo correspondence and structure recovery, the method introduces an external energy function defined for a 3D volume based on cross-correlation between the stereo pairs. The internal energy functional of the deformable dual mesh imposes smoothness on the surfaces and it serves as a communication tool between the two meshes. Under the forces produced by the energy terms, the dual mesh deforms to recover and track the 3D surface. The newly introduced dual-mesh model, which is one of the main contributions of this paper, makes the system robust against local minima and yet it is efficient. A coarse-to-fine minimization approach makes the system even more efficient. Tracking is achieved by using the recovered surface as an initial position for the next time frame. Although the system can effectively utilize initial surface positions and disparity data, they are not needed for a successful operation, which makes this system applicable to a wide range of areas. We present the results of a number of experiments on stereo human face and cloud images, which proves that our new method is very effective.

Computerized analysis of the tongue surface movement can provide valuable information to speech and swallowing research. Ultrasound technology is currently the most attractive modality for the tongue imaging mainly because of its high video frame rate. However, problems with ultrasound imaging, such as noise and echo artifacts, refractions, and unrelated reflections pose significant challenges for computer analysis of the tongue images and hence specific methods must be developed. This paper presents a system that is developed for automatic extraction and tracking of the tongue surface movements from ultrasound image sequences. The ultrasound images are supplied by the head and transducer support system (HATS), which was developed in order to fix the head and support the transducer under the chin in a known position without disturbing speech. In this work, we propose a novel scheme for the analysis of the tongue images using deformable contours. We incorporate novel mechanisms to 1) impose speech related constraints on the deformations; 2) perform spatiotemporal smoothing using a contour postprocessing stage; 3) utilize optical flow techniques to speed up the search process; and 4) propagate user supplied information to the analysis of all image frames. We tested the system's performance qualitatively and quantitatively in consultation with speech scientists. Our system produced contours that are within the range of manual measurement variations. The results of our system are extremely encouraging and the system can be used in practical speech and swallowing research in the field of otolaryngology.

This paper presents a system for automatic extraction and tracking of 2D contours of the tongue surfaces from digital ultrasound image sequences. The input to the system is provided by a Head and Transducer Support System (HATS), which is developed for use in ultrasound imaging of the tongue movement. We developed a novel active contour (snakes) model that uses several temporally adjacent images during the extraction of the tongue surface contour for an image frame. The user supplies an initial contour model for a single image frame in the whole sequence. Using optical flow and multi-resolution methods, this initial contour is then used to find the candidate contour points in the temporally immediate adjacent images. Subsequently, the new snake mechanism is applied to estimate optimal contours for each image frame using these candidate points. In turn, the extracted contours are used as models for the extraction process of new adjacent frames. Finally, the system uses a novel postprocessing technique to refine the positions of the contours. We tested the system on 11 different speech sequences, each containing about 25 images. Visual inspection of the detected contours by the speech experts shows that the results are very promising and this system can be effectively employed in speech and swallowing research.

In this paper we present a novel method for tracking 3D nonrigid motion and 3D structure from a sequence of monocular, perspective images, where range data of the first frame is available. No a priori knowledge of point correspondences is assumed. A genneral approach and formulas for polynomial (second order) displacement functions are presented. An analysis of the number of equations versus the number of unknowns is given. Results for synthetic and real image sequences are presented.

We present a system that extracts and tracks 2D mid-sagital tongue surfaces from ultrasound image sequences produced by a head and transducer support system (HATS). Such a system can be a valuable and practical tool for speech and swallowing research. We extend our previous work by introducing a deformable contour model to impose restrictions of spatiotemporal coherency by assuming that the 2D mid-sagital tongue surface contours sweep a coherent 3D structure in spatiotemporal 3D space. Another novel contribution of this paper is a dynamic programming minimization method to optimize the new energy functional, which can be employed in general snake minimization tasks. We tested this system on a number of speech and swallowing sequences each containing 24 to 30 frames. We verified with speech experts and found that our system produces promising results, especially for the swallowing sequences, which are more problematic.

Proposes a method for the estimation of point correspondences between boundaries undergoing nonrigid motion. The algorithm works in two stages. In the first stage, a global estimate of the nonrigid motion is obtained using hyperquadric models. The second stage uses this estimate to remove the global nonrigid motion (scale, shear, etc.) and then compute point correspondences between the two datasets assuming small deformations. The global part of the nonrigid motion (scale, shear, rotation and translation) is estimated by modeling the object with hyperquadrics and estimating the transformation between the hyperquadric parameters. Point correspondences are then estimated by using differential geometric properties during small deformations. Experimental results with real data are presented.

Time-varying multispectral observations of clouds from meteorological satellites are used to estimate cloud-top heights (structure) and cloud winds (semi-fluid motion). Stereo image pairs over several time steps were acquired by two geostationary satellites with synchronized scanning instruments. Cloud-top height estimation from these image pairs is performed using an improved automatic stereo analysis algorithm on a massively parallel Maspar computer with 16 K processors. A new category of motion behavior known as semi-fluid motion is described for modeling cloud motions and an automatic algorithm for extracting semi-fluid motion is developed to track cloud winds. The time sequential dense estimates of cloud-top height depth maps in conjunction with intensity data are used to estimate local semi-fluid motion parameters for cloud tracking. Both stereo disparities and motion correspondences are estimated to sub-pixel accuracy. The Interactive Image SpreadSheet (IISS) is a new versatile visualization tool that was enhanced to analyze and visualize the results of the stereo analysis and semi-fluid motion estimation algorithms. Experimental results using time-varying data of the visible channel from two satellites in geosynchronous orbit is presented for the Hurricane Frederic.>

Recovery of motion parameters and point correspondences is a fundamental problem in computer vision. Although a great deal of research has been done in solving rigid motion, nonrigid motion analysis has only recently been addressed and is gaining interest due to its wide range of applications. This paper introduces a novel method for estimating motion parameters and point correspondences between surfaces under small nonrigid deformations. It uses the changes in differential geometric properties of surface under motion. Simulations are performed by generating nonrigid motion on an ellipsoidal data to illustrate performance and accuracy of derived algorithms, Then, the algorithm is tested on the sequence of facial range images. The motion parameters generated by the algorithm has also been used do detect the abnormality in cardiac images.>

A method for the estimation of point correspondences on a surface undergoing nonrigid motion, based on changes in Gaussian curvature, is described. An approach for estimating the point correspondences and stretching of a surface undergoing conformal motion with constant (homothetic), linear, or polynomial stretching is proposed. Small motion assumption is utilized to hypothesize all possible point correspondences. Curvature changes are then computed for each hypothesis. The difference between computed curvature changes and the one predicted by the conformal motion assumption is calculated. The hypothesis with the smallest error gives point correspondences between consecutive time frames. Simulations performed on ellipsoidal data illustrate the performance and accuracy of derived algorithms. The algorithm is applied to volumetric CT data of the left ventricle of a dog's heart.>