The studies of the network formed by social insects require the motion analysis of their interactions and movements in videos over an extended period of time. Automated detection is an important field of interest because it enables the motion analysis in large-scale experiments. When an automated detection method is applied to various insect types, the training task often involves the collection of a large number of labels provided by human experts. To save the experts' time and effort, unlabeled data have been recently employed to supplement the training. In this paper, we utilize the spatiotemporal connectivity of the unlabeled data to regulate the training of a detector on a new insect type. Our key contribution is integrating the spatiotemporal connectivity among the unlabeled samples to determine the weighting scheme of the existing classifiers from multiple sources. The evaluation on 3 data sets of social insects consisting of 6,000 samples demonstrates that a detector trained using our method can achieve comparable performance to previous approaches while reducing the training labels up to 16 times. Since the proposed method is based on regularizing the unlabeled samples based on their spatiotemporal connectivity, we refer to it as the Spatio-Temporally Regularized Adaptive Learning (STRAL).

In this work, we present an automated inspection approach to assist remote visual examinations of nuclear power plant components. An automated approach would require detecting often low contrast cracks that could be surrounded by or even within textures with similar appearances such as welding, scratches, and grind marks. We propose a crack segmentation method for remote visual examination videos by aggregating the pixel-level classification confidence from multiple frames consisting of different illumination conditions. A dataset of 685 pixel-level ground truth images consisting of 37 cracks from remote visual examination videos is used for evaluation. The results show that the proposed method provides a significant improvement over hand-crafted feature based segmentation and 9% over convolutional neural network based method.

Accurate measurement of left ventricular volumes and Ejection Fraction from cine MRI is of paramount importance to the evaluation of cardiovascular functions, yet it usually requires laborious and tedious work of trained experts to interpret them. To facilitate this procedure, numerous computer aided diagnosis (CAD) methods and tools have been proposed, most of which focus on the left or right ventricle segmentation. However, the identification of ES and ED frames from cardiac sequences is largely ignored, which is a key procedure in the automated workflow. This seemingly easy task is quite challenging, due to the requirement of high accuracy ( i.e. , precisely identifying specific frames from a sequence) and subtle differences among consecutive frames. Recently, with the rapid growth of annotated data and the increasing computational power, deep learning methods have been widely exploited in medical image analysis. In this paper, we propose a novel deep learning architecture, named as temporal regression network (TempReg-Net), to accurately identify specific frames from MRI sequences, by integrating the Convolutional Neural Network (CNN) with the Recurrent Neural Network (RNN). Specifically, a CNN encodes the spatial information of a cardiac sequence, and a RNN decodes the temporal information. In addition, we design a new loss function in our network to constrain the structure of predicted labels, which further improves the performance. Our approach is extensively validated on thousands of cardiac sequences and the average difference is merely 0.4 frames, comparing favorably with previous systems. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Robust inspection is important to ensure the safety of nuclear power plant components. An automated approach would require detecting often low contrast cracks that could be surrounded by or even within textures with similar appearances such as welding, scratches and grind marks. We propose a crack detection method for nuclear power plant inspection videos by fine tuning a deep neural network for detecting local patches containing cracks which are then grouped in spatial-temporal space for group-level classification. We evaluate the proposed method on a data set consisting of 17 videos consisting of nearly 150,000 frames of inspection video and provide comparison to prior methods.

Automated tracking methods greatly expedite the collection of data for studying social insects. However, due to the frequency of occlusions (due to interactions) and similarity in appearance and motion features, tracking could easily drift to the incorrect object as the affinity model is unable to discern between similar objects. Recently, a method to filter incorrect associations based on areas of occlusion was proposed. This method only filtered based on entrance and exit within a specific occlusion. In this paper, we propose to improve the tracking of multiple insects involving frequent occlusion by modeling the paths of possible movement within the occlusion tunnel which we call occlusion sub-tunnels. Using two datasets consisting of ants and termites, we demonstrate that the method filters 8% more incorrect associations on average resulting in a reduction of ID switches by 19% over all datasets.

The applicability of automated motion analysis is immense and continues to grow as our ability to record objects of interest becomes easier and less expensive. In the case of multi-object tracking, data association methods have been proposed to improve handling of occlusions. These methods are strongly affected by their ability to measure association affinities between fragmented object trajectories. Obtaining labeled training examples for learning how to measure these associations can be expensive and time-consuming. We propose an interactive training framework that utilizes an uncertainty based active sampling approach in combination with semi-supervised learning in order to reduce the number of labeled examples needed for training. Additionally, an affinity scoring function is learned with Random Forest to speed up learning affinity measures in order to make the interactive training framework possible. Experimental results on two 10,000 frame video sequences of ant colonies demonstrates a significant reduction in the amount of labeled examples needed over random sampling.

The automated tracking of social insects, such as ants, can efficiently provide unparalleled amounts of data for the of study complex group behaviors. However, a high level of occlusion along with similarity in appearance and motion can cause the tracking to drift to an incorrect ant. In this paper, we reduce drifting by using occlusion to identify incorrect ants and prevent the tracking from drifting to them. The key idea is that a set of ants enter occlusion, move through occlusion then exit occlusion. We do not attempt to track through occlusions but simply find a set of objects that enters and exits them. Knowing that tracking must stay within a set of ants exiting a given occlusion, we reduce drifting by preventing tracking to ants outside the occlusion. Using four 5000 frame video sequences of an ant colony, we demonstrate that the usage of occlusion tunnel reduces the tracking error of (1) drifting to another ant by 30% and (2) early termination of tracking by 7%.

The tracking of ants in video is important for the analysis of their complex group behavior. However, the manual analysis of these videos is tedious and time consuming. Automated tracking methods tend to drift due to frequent occlusions during their interactions and similarity in appearance. Semi-automated tracking methods enable corrections of tracking errors by incorporating user interaction. Although it is much lower than manual analysis, the required user time of the existing method is still typically 23 times the actual video length. In this paper, we propose a new semi-automated method that achieves similar accuracy while reducing the user interaction time by (1) mitigating user wait time by incorporating a data association tracking method to separate the tracking from user correction, and (2) minimizing the number of candidates visualized for user during correction. This proposed method is able to reduce the user interaction time by 67% while maintaining the accuracy within 3% of the previous semi-automated method [11].

The automated tracking of social insects, such as ants, could dramatically increase the fidelity and amount of analyzed data for studying complex group behaviors. Recently, data association based multiple object tracking methods have shown promise in improving handling of occlusions. However, the tracking of ants in a colony is still challenging as (1) their motion is often sporadic and irregular and (2) they are mostly present the entire duration of video. In this paper, we propose to improve the data association based tracking of multiple ants. First, we model the ant's motion using a set of irregular motion features including random walk model. Second, we use the convergence of particle filter based tracking to match tracklets with a long temporal gap. Testing results of two-fold cross validation on a 10,000 frame video shows that our proposed method was able to reduce the number of fragments by 61% and ID switches by 57%.

The pollen grains of different plant taxa exhibit various shapes and sizes. This structural diversity has made the identification and classification of pollen grains an important tool in many fields. Despite the myriad of applications, the classification of pollen grains is still a tedious and time-consuming process that must be performed by highly skilled specialists. In this paper, we propose an automatic classification method to discriminate pollen grains coming from a variety of taxonomic types. First, we develop a new feature that captures the spikes of pollen to improve the classification accuracy. Second, we take advantage of the classification rules extracted from the existing pollen types and apply them to the new types. Third, we introduce a new selection criterion to obtain the most valuable training samples from the unlabeled data and therefore reduce the number of needed training samples. Our experiment demonstrates that the proposed method reduces the training effort of a human expert up to 80% compared to other classification methods while achieving 92% accuracy in pollen classification.

Behavior analysis of social insects requires robust tracking over many frames. Automated tracking methods are not reliable for tracking over long video. And they are prone to a quick accumulation of error from one mis-tracking. However, searching and correcting of mis-tracking is time-consuming. In this paper, we present an efficient method for achieving robust tracking of multiple ants over a long video. First, our method minimizes the user wait time by speeding up tracking with a GPU. Second, it minimizes the amount of data the user needs to validate by automatically searching for potential errors and presenting them for user validation and correction. User studies with three participants on a 10,000 frame video demonstrates that (1) the speed of tracking is 16x faster with GPU optimization, (2) tracking accuracy was 96%, which is a 25% improvement over no user interaction, (3) users examined less than 0.6% of frames for validation and correction.

Motion and behavior analysis of social insects such as ants requires tracking many ants over time. This process is highly labor-intensive and tedious. Automatic tracking is challenging as ants often interact with one another, resulting in frequent occlusions that cause drifts in tracking. In addition, tracking many objects is computationally expensive. In this paper, we present a robust and efficient method for tracking multiple ants. We first prevent drifts by maximizing the coverage of foreground pixels at at global scale. Secondly, we improve speed by reducing markov chain length through dynamically changing the target proposal distribution for perturbed ant selection. Using a real dataset with ground truth, we demonstrate that our algorithm was able to improve the accuracy by 15% (resulting in 98% tracking accuracy) and the speed by 76%.

Studying and analyzing inter-communication among bees requires tracking of many bees. Manual labeling of bees over many frames is painstaking and time-consuming. Automated tracking is challenging because of the appearance change and unreliable features. This problem is magnified when tracking for a longer period of time is required. We present a method for tracking bees that minimizes the accumulation of error over time by using (1) static and adaptive appearance templates for handling appearance change, and (2) geometry-constrained resampling of particles for handling unreliable features. Evaluation against manually-labeled ground truth demonstrates that our method tracks bees with an RMSE of 8.7 pixels (typical bee length is >100 pixels), and 75% position and 58% angular error improvement over a particle filtering based tracking with Gaussian modeling of appearance.

The motion of leukocytes is significant in studying the inflammation response of the immune system. In inflammation conditions, some leukocytes slow down and eventually adhere to vessel walls. With many cells moving at a variety of speeds, collisions occur. These collisions result in abrupt changes in the motion and appearance of leukocytes. In this paper, we propose a novel method of tracking multiple cells undergoing collision by modeling the collision states of cells and testing multiple hypotheses of their motion and appearance. We demonstrate the results on in vivo intravital microscopy image sequences. The proposed method results in 20% more accurate cell position and 82% longer tracks than a previous approach.

Automated cell tracking using in vivo imagery is difficult, in general, due to the noise inherent in the imaging process, occlusions, varied cell appearance over time, motion of other tissue (distractors), and cells traveling in and out of the image plane. For certain types of cells these problems are exacerbated due to erratic motion patterns. In this paper, we introduce the Radial Flow Transform, which provides motion estimates for objects of interest in a scene without explicitly tracking each object. The transform is robust to misdetected objects, temporally-disjoint motion events, and can represent multiple directions of flow at a single location. We provide operations to convert to and from a vector field representation. This allows for intuitive reasoning about the motion patterns in a scene. We demonstrate results on synthetic data and in vivo microscopy video of a mouse liver.

University of Florida This paper presents a system which allows medical students to experience the interaction between a patient and a medical doctor using natural methods of interaction with a high level of immersion. We also present our experiences with a pilot group of medical and physician assistant students at various levels of training. They interacted with projector-based life-sized virtual characters using gestures and speech. We believe that natural interaction and a high level of immersion facilitates the education of communication skills. We present the system details as well as the participants ’ performance and opinions. The study confirmed that the level of immersion contributed significantly to the experience, and participants reported that the system is a powerful tool for teaching and training. Applying the system to formal communication skills evaluation and further scenario development will be the focus of future research and refinement.

Skin detection is an important process in many of computer vision algorithms. It usually is a process that starts at a pixel-level, and that involves a pre-process of colorspace transformation followed by a classification process. A colorspace transformation is assumed to increase separability between skin and non-skin classes, to increase similarity among different skin tones, and to bring a robust performance under varying illumination conditions, without any sound reasonings. In this work, we examine if the colorspace transformation does bring those benefits by measuring four separability measurements on a large dataset of 805 images with different skin tones and illumination. Surprising results indicate that most of the colorspace transformations do not bring the benefits which have been assumed.

This paper presents an evaluation of edge detector performance. We use the task of structure from motion (SFM) as a "black box" through which to evaluate the performance of edge detection algorithms. Edge detector goodness is measured by how accurately the SFM could recover the known structure and motion from the edge detection of the image sequences. We use a variety of real image sequences with ground truth to evaluate eight different edge detectors from the literature. Our results suggest that ratings of edge detector performance based on pixel-level metrics and on the SFM are well correlated and that detectors such as the Canny detector and Heitger detector offer the best performance.

We present a method for 3D non-rigid motion tracking and structure reconstruction from 2D points and curve segments from a sequence of perspective images. The 3D locations of features in the first frame are known. The 3D affine motion model is used to describe the nonrigid motion. The results from synthetic and real data are presented. The applications include: lip tracking, MPEG4 face player, and burn scar assessment. The results show that: 1) curve segments are more robust under noise (observed from synthetic data with different Gaussian noise level); and 2) using both feature yields a significant performance gain in real data.

This paper presents a framework for a customizable MPEG-4 face player using a FAPs (Facial Animation Parameters) sequence recovered from a real-time image sequence. First, the 3D nonrigid motion and structure of the facial features is recovered from a 2D image sequence and a "person-specific" model of the face. The model consists of the intensity and range image of the face. Then, the FAPs are computed from the recovered 3D structure. The customizable MPEG4 face animation is generated using a FAP sequence and a model of a specific person. The dataset of four face image sequences from three different face orientations equipped with range images are used. The GT (ground truth) results are generated using the 3D structure provided by range images. The results are evaluated quantitatively by comparing recovered FAP values, and qualitatively by comparing the generated MPEG-4 animations. FAPs are recovered up to 8% (relative) and 16 FAP units (absolute) accuracy and the animation is nearly identical to the MPEG-4 animation using GT FAPs.

This paper presents a methodology and results of evaluating edge detection algorithms using an object recognition task. A dataset consisting of 37 real images with 5 different jeep-like vehicles is used. Five edge detectors are compared using ROC curve analysis. The Heitger detector gives the best results. The work is being extended to include more images and a train-and-test style evaluation.

This paper presents a task-oriented evaluation methodology for edge detectors. Performance is measured based on the task of structure from motion. Eighteen real image sequences from 2 different scenes varying in the complexity and scenery types are used. The task-level ground truth for each image sequence is manually specified in terms of the 3D motion and structure. An automated tool computes the accuracy of the motion and structure achieved using the set of edge maps. Parameter sensitivity and execution speed are also analyzed. Four edge detectors are compared. All implementations and data sets are publicly available.