We investigate combinations of 3D-2D matches of points and lines for a single calibrated camera that imply a nontrivial constraint on the camera’s position. Specifically, we address the minimal cases for which n point matches and m line matches constrain the camera center to lie on an algebraic surface in $$\mathbb {R}^3$$ . For two points $$(m,n) = (0,2)$$ , the constraint is well-known to be a self-intersecting torus. We complete the classification problem by deriving explicit, general equations for the cases $$(m,n) \in \{ (2,0), (1, 1) \}$$ for two lines and one point, one line, respectively. We also report preliminary experiments investigating the suitability of these constraints for pose estimation tasks.

Abstract Video Object Segmentation (VOS) is a key component in computer vision applications, including surveillance, autonomous driving, and robotics. However, existing VOS models often struggle with generalization to new videos with complex, topologically transforming deformable objects (eg. cooking, assembling, state change), degraded environments and long video sequences, resulting in tracking drift, low recall and memory saturation. We developed Mu ltiple object VOS and tracking S mart Mem ory architecture (MuSMem), a generalizable approach that incorporates three key innovations: (i) fusing SAM with High-Quality masks alongside appearance-based candidate-selection to refine coarse segmentation masks, resulting in improved object boundaries; (ii) dynamic smart memory that manages a history of key frames based on a novel information preserving gain , combined with relevance and freshness spatio-temporal criteria; and (iii) explores the use of monocular depth maps for occlusion robustness. MuSMem significantly reduces memory usage, reduces drift, tracks complex object topological changes and improves long-term prediction performance. MuSMem can be integrated with Vision-Language Models (VLMs) for zero-shot generalization to unseen visual domains. Experiments using VOS benchmark datasets show that MuSMem ranks first on VOTSt-2024, Long Video Dataset and LVOS, and second on VOTS-2024, demonstrating the best generalizability and state-of-the-art performance across single-, multi-, and complex VOS tasks.

Hypoxic Ischemic Encephalopathy (HIE) represents a brain dysfunction, affecting approximately 1 to 5 per 1000 full-term neonates. The precise delineation and segmentation of HIE-related lesions in neonatal brain Magnetic Resonance Images (MRI) are pivotal in advancing outcome predictions, identifying patients at high risk, elucidating neurological manifestations, and assessing treatment efficacies. Despite its importance, the development of algorithms for segmenting HIE lesions from MRI volumes has been impeded by data scarcity. Addressing this critical gap, we organized the first BONBID-HIE challenge with diffusion MRI data (Apparent Diffusion Coefficient (ADC) maps) for HIE lesion segmentation, in conjunction with the MICCAI 2023. Totally 14 algorithms were submitted, employing a gamut of cutting-edge automatic machine-learning-based segmentation algorithms. Our comprehensive analysis of HIE lesion segmentation and submitted algorithms facilitates an in-depth evaluation of the current technological zenith, outlines directions for future advancements, and highlights persistent hurdles. To foster ongoing research and benchmarking, the annotated HIE dataset, developed algorithm dockers, and unified evaluation codes are accessible through a dedicated online platform (https://bonbid-hie2023.grand-challenge.org).

Abstract Discriminating salient moving objects against complex, cluttered backgrounds, with occlusions and challenging environmental conditions like weather and illumination, is essential for stateful scene perception in autonomous systems. We propose a novel deep architecture, named DeepFTSG, for robust moving object detection that incorporates single and multi-stream multi-channel USE-Net trellis asymmetric encoders extending U-Net with squeeze and excitation (SE) blocks and a single shared decoder network for fusing multiple motion and appearance cues. DeepFTSG is a deep learning based approach that builds upon our previous hand-engineered flux tensor split Gaussian (FTSG) change detection video analysis algorithm which won the CDNet CVPR Change Detection Workshop challenge competition. DeepFTSG generalizes much better than top-performing motion detection deep networks, such as the scene-dependent ensemble-based FgSegNet_v2, while using an order of magnitude fewer weights. Short-term motion and longer-term change cues are estimated using general-purpose unsupervised methods—flux tensor and multi-modal background subtraction, respectively. DeepFTSG was evaluated using the CDnet-2014 change detection challenge dataset, the largest change detection video sequence benchmark with 12.3 billion labeled pixels, and had an overall F-measure of 97%. We also evaluated the cross-dataset generalization capability of DeepFTSG trained solely on CDnet-2014 short video segments and then evaluated on unseen SBI-2015, LASIESTA and LaSOT benchmark videos. On the unseen SBI-2015 dataset, DeepFTSG had an F-measure accuracy of 87%, more than 30% higher compared to the top-performing deep network FgSegNet_v2 and outperforms the recently proposed KimHa method by 17%. On the unseen LASIESTA, DeepFTSG had an F-measure of 88% and outperformed the best recent deep learning method BSUV-Net2.0 by 3%. On the unseen LaSOT with axis-aligned bounding box ground-truth, network segmentation masks were converted to bounding boxes for evaluation, DeepFTSG had an F-Measure of 55%, outperforming KimHa method by 14% and FgSegNet_v2 by almost 1.5%. When a customized single DeepFTSG model is trained in a scene-dependent manner for comparison with state-of-the-art approaches, then DeepFTSG performs significantly better, reaching an F-Measure of 97% on SBI-2015 (+ 10%) and 99% on LASIESTA (+ 11%). The source code, pre-trained weights, and video demo for DeepFTSG are available at https://github.com/CIVA-Lab/DeepFTSG .

We present a pipeline for predicting mechanical properties of vertically-oriented carbon nanotube (CNT) forest images using a deep learning model for artificial intelligence (AI)-based materials discovery. Our approach incorporates an innovative data augmentation technique that involves the use of multi-layer synthetic (MLS) or quasi-2.5D images which are generated by blending 2D synthetic images. The MLS images more closely resemble 3D synthetic and real scanning electron microscopy (SEM) images of CNTs but without the computational cost of performing expensive 3D simulations or experiments. Mechanical properties such as stiffness and buckling load for the MLS images are estimated using a physics-based model. The proposed deep learning architecture, CNTNeXt, builds upon our previous CNTNet neural network, using a ResNeXt feature representation followed by random forest regression estimator. Our machine learning approach for predicting CNT physical properties by utilizing a blended set of synthetic images is expected to outperform single synthetic image-based learning when it comes to predicting mechanical properties of real scanning electron microscopy images. This has the potential to accelerate understanding and control of CNT forest self-assembly for diverse applications.

Collaborative Drone systems (CDS) have the potential to benefit a variety of application areas such as agriculture, military operations, surveillance, and disaster response. At the same time, CDS can pose challenges due to their limited flight time impacted by battery capacities, and constrained edge computation capabilities on-board the drones. Furthermore, an understudied subject relates to when drones in a CDS trust each other to accomplish a task, resulting in new vulnerabilities that can be exploited via cyber attacks. In this paper, we propose a novel trust quantification methodology in a CDS with intelligence-driven edge routing, which can help detect malicious nodes in a CDS that compromise communication and disrupt the functionality of packet forwarding. Our approach for trust quantification is guided by a CDS vulnerability analysis that characterizes impact due to the presence of two malicious threat agents viz., flooder node and faker node. Detection of these threat agents in a CDS is aided by trust quantification in the form of trust scores obtained by using a Bayesian Network model that allows for decision-making on CDS nodes’ trust levels. We validate our trust quantification methodology in ns-3 based simulation experiments and show how we can categorize nodes based on different thresholds of trust scores with varying sensitivities, which helps in the detection of CDS threat agents.

Most existing image privacy protection works focus mainly on the privacy of photo owners and their friends, but lack the consideration of other people who are in the background of the photos and the related location privacy issues. In fact, when a person is in the background of someone else’s photos, he/she may be unintentionally exposed to the public when the photo owner shares the photo online. Not only a single visited place could be exposed, attackers may also be able to piece together a person’s travel route from images. In this article, we propose a novel image privacy protection system, called LAMP, which aims to light up the location awareness for people during online image sharing. The LAMP system is based on a newly designed location-aware multi-party image access control model. Unlike previous works on small scales, the LAMP system is highly efficient and scalable as it can enforce privacy protection for billions of users on social networks in real time. The LAMP system automatically detects the user’s occurrences on photos regardless the user is the photo owner or not. Once a user is identified and the location of the photo is deemed sensitive according to the user’s privacy policy, the user’s face will be replaced with a synthetic face. A prototype of the system was implemented and evaluated to demonstrate its applicability in the real world.

3D city-scale models are useful in a number of applications, including education, city planning, navigation systems, artificial intelligence training, and simulations. However, final models need to be immersive and interactive, which requires a mixed reality (XR) environment design that combines e.g., a Cave Automatic Virtual Environment (CAVE) VR system with the Microsoft Hololens2 in a networked and multimodal setting. In this paper, we propose a pipeline to convert a city-scale point cloud into a finalized city-scale textured mesh in which, a number of XR devices can share the same environment and co-exist in a shared space for model interactions. Specifically, we use input point clouds obtained from wide area motion imagery systems or off-the-shelf drones pertaining to Albuquerque, New Mexico, but the pipeline is generalized so that other input can be used. Using four different traditional algorithms and an additional deep learning method, we create meshes for the model interactions. For each mesh produced, we map high-resolution textures onto them, producing a more accurate city, which is then passed into the shared/networked Unity environment. Ten participants provided their assessment of mesh quality and interactivity of the networked environment during exploration of different city reconstructions with the CAVE and laptop device modalities. Results on the perceptual immersive quality of the Point2Mesh deep learning meshes highlights the need for improvements to handle large city scale point clouds.

Modeling and simulation of large urban regions is beneficial for a range of applications including intelligent transportation, smart cities, infrastructure planning, and training artificial intelligence for autonomous navigation systems including ground vehicles and aerial drones. Immersive environments including virtual reality (VR), augmented reality (AR), mixed reality (MR or XR) can be used to explore city scale regions for planning, design, training and operations. Virtual environments are in the midst of rapid change as innovations in display tech-nologies, graphics processors and game engine software present new opportunities for incorporating modeling and simulation into engineering workflows. Game engine software like Unity with photorealistic rendering and realistic physics have plug-in support for a variety of virtual environments. In this paper, we explore the visualization of urban scale real world accurate meshes in virtual environments, including the Microsoft HoloLens head mounted display or the CAVE VR for multi-user interaction.

Current scientific experiments frequently involve control of specialized instruments (e.g., scanning electron microscopes), image data collection from those instruments, and transfer of the data for processing at simulation centers. This process requires a “human-in-the-loop” to perform those tasks manually, which besides requiring a lot of effort and time, could lead to inconsistencies or errors. Thus, it is essential to have an automated system capable of performing remote instrumentation to intelligently control and collect data from the scientific instruments. In this paper, we propose a Remote Instrumentation Science Environment (RISE) for intelligent image analytics that provides the infrastructure to securely capture images, determine process parameters via machine learning, and provide experimental control actions via automation, under the premise of “human-on-the-loop”. The machine learning in RISE aids an iterative discovery process to assist researchers to tune instrument settings to improve the outcomes of experiments. Driven by two scientific use cases of image analytics pipelines, one in material science, and another in biomedical science, we show how RISE automation leverages a cutting-edge integration of cloud computing, on-premise HPC cluster, and a Python programming interface available on a microscope. Using web services, we implement RISE to perform automated image data collection/analysis guided by an intelligent agent to provide real-time feedback control of the microscope using the image analytics outputs. Our evaluation results show the benefits of RISE for researchers to obtain higher image analytics accuracy, save precious time in manually controlling the microscopes, while reducing errors in operating the instruments.

Change impact analysis evaluates the changes that are made in the software and finds the ripple effects, in other words, finds the affected software components. Code changes and bug fixes can have a high impact on code quality by introducing new vulnerabilities or increasing their severity. A recent high-visibility example of this is the code changes in the log4j web software CVE-2021-45105 to fix known vulnerabilities by removing and adding method called change types. This bug fix process exposed further code security concerns. In this article, we analyze the most common set of bug fix change patterns to have a better understanding of the distribution of software changes and their impact on code quality. To achieve this, we implemented a tool that compares two versions of the code and extracts the changes that have been made. Then, we investigated how these changes are related to change impact analysis. In our case study, we identified the change types for bug-inducing and bug fix changes using the Quixbugs dataset. Furthermore, we used 13 of the projects and 621 bugs from Defects4J to identify the common change types in bug fixes. Then, to find the change types that cause an impact on the software, we performed an impact analysis on a subset of projects and bugs of Defects4J. The results have shown that, on average, 90% of the bug fix change types are adding a new method declaration and changing the method body. Then, we investigated if these changes cause an impact or a ripple effect in the software by performing a Markov chain-based change impact analysis. The results show that the bug fix changes had only impact rates within a range of 0.4–5%. Furthermore, we performed a statistical correlation analysis to find if any of the bug fixes have a significant correlation with the impact of change. The results have shown that there is a negative correlation between caused impact with the change types adding new method declaration and changing method body. On the other hand, we found that there is a positive correlation between caused impact and changing the field type.

Modeling and simulation of large urban regions is beneficial for a range of applications including intelligent transportation, smart cities, infrastructure planning, and training artificial intelligence for autonomous navigation systems including ground vehicles and aerial drones. Immersive environments including virtual reality (VR), augmented reality (AR), mixed reality (MR or XR) can be used to explore city scale regions for planning, design, training and operations. Virtual environments are in the midst of rapid change as innovations in display technologies, graphics processors and game engine software present new opportunities for incorporating modeling and simulation into engineering workflows. Game engine software like Unity with photorealistic rendering and realistic physics have plug-in support for a variety of virtual environments and typically model the scene as meshes. In this paper, we develop an end-to-end workflow for creating urban scale real world accurate synthetic environments that can be visualized in virtual environments including the Microsoft HoloLens head mounted display or the CAVE VR for multi-user interaction. Four meshing algorithms are evaluated for representation accuracy and city-scale meshes imported into Unity for assessing the quality of the immersive experience.

Change impact analysis analyzes the changes that are made in the software and finds the ripple effects, in other words, finds the affected software components. In this study, we analyze the bug fix change patterns to have a better understanding of what types of changes are common in fixing bugs. To achieve this, we implemented a tool that compares two versions of codes and detects the changes that are made. Then, we investigated how these changes are related to change impact analysis. In our case study, we used 13 of the projects and 621 bugs from Defects4J to identify the common change types in bug fixed. Then, to find the change types related to cause an impact in the software, we performed an impact analysis on a subset of projects and bugs of Defects4J. The results have shown that, on average, 90% of the bug fix change types are adding a new method declaration and changing the method body. Then, we investigated if these changes cause an impact or a ripple effect in the software by performing a Markov chain-based change impact analysis. The results show that the bug fix changes had only impact rates within a range of 0.4%-5%. Furthermore, we performed a statistical correlation analysis to find if any of the bug fixes have a significant correlation on the impact of change. The results have shown that there is a negative correlation between caused impact with the change types adding new method declaration and changing method body. On the other hand, we found that there is a positive correlation between caused impact and changing the field type.

We introduce a novel discrete cosine transform-based feature (DCTF) descriptor designed for both robustly matching features in aerial video and tracking features across wide-baseline oblique views in aerial wide area motion imagery (WAMI). Our DCTF descriptor preserves local structure more compactly in the frequency domain by utilizing the mathematical properties of the discrete cosine transform (DCT) and outperforms widely used the spatial-domain feature extraction methods, such as speeded up robust features (SURF) and scale-invariant feature transform (SIFT). The DCTF descriptor can be used in combination with other feature detectors, such as SURF and features from accelerated segment test (FAST), for which we provide experimental results. The performance of DCTF for image matching and feature tracking is evaluated on two city-scale aerial WAMI data sets (ABQ-215 and LA-351) and a synthetic aerial drone video data set digital imaging and remote sensing image generation (Rochester Institute of Technology (RIT)-DIRSIG). DCTF is a compact 120-D descriptor that is less than half the dimensionality of state-of-the-art deep learning-based approaches, such as SuperPoint, LF-Net, and DeepCompare, which requires no learning and is domain-independent. Despite its small size, the DCTF descriptor surprisingly produces the highest image matching accuracies ( F1= 0.76 and ABQ-215), the longest maximum and average feature track lengths, and the lowest tracking error (0.3 pixel, LA-351) compared with both handcrafted and deep learning features.

Computer-assisted algorithms have become a mainstay of biomedical applications to improve accuracy and reproducibility of repetitive tasks like manual segmentation and annotation. We propose a novel pipeline for red blood cell detection and counting in thin blood smear microscopy images, named RBCNet, using a dual deep learning architecture. RBCNet consists of a U-Net first stage for cell-cluster or superpixel segmentation, followed by a second refinement stage Faster R-CNN for detecting small cell objects within the connected component clusters. RBCNet uses cell clustering instead of region proposals, which is robust to cell fragmentation, is highly scalable for detecting small objects or fine scale morphological structures in very large images, can be trained using non-overlapping tiles, and during inference is adaptive to the scale of cell-clusters with a low memory footprint. We tested our method on an archived collection of human malaria smears with nearly 200,000 labeled cells across 965 images from 193 patients, acquired in Bangladesh, with each patient contributing five images. Cell detection accuracy using RBCNet was higher than 97 %. The novel dual cascade RBCNet architecture provides more accurate cell detections because the foreground cell-cluster masks from U-Net adaptively guide the detection stage, resulting in a notably higher true positive and lower false alarm rates, compared to traditional and other deep learning methods. The RBCNet pipeline implements a crucial step towards automated malaria diagnosis.

New virtualization technologies allow Infrastructure Providers (InPs) to lease their resources to Application Service Providers (ASPs) for highly scalable delivery of cloud services to end-users. However, existing literature lacks knowledge on Quality of Experience (QoE)-oriented cloud service orchestration algorithms that can guide ASPs on how to plan their budget to enhance satisfactory QoE delivery to end-users. In contrast to the InP's cloud service orchestration, the ASP's orchestration should not rely on expensive infrastructure control mechanisms such as Software-Defined Networking (SDN), or require aprioriknowledge on the number of services to be instantiated and their anticipated placement location within InP's infrastructure. In this paper, we address this issue of delivering satisfactory user QoE by synergistically optimizing both ASP's management and data planes. The optimization within the ASP management planefirst maximizes Service Level Objective (SLO) coverage of users when application services are being deployed, and are not yet operational. The optimization of the ASP data plane then enhances satisfactory user QoE delivery when applications services are operational with real user access. Our evaluation of QoE-oriented algorithms using realistic numerical simulations, real-world cloud testbed experiments with actual users and ASP case studies show notably improved performance over existing cloud service orchestration solutions.

In biomedical imaging using video microscopy, understanding large tissue structures at cellular and finer resolution poses many image acquisition challenges including limited field-of-view and tissue dynamics during imaging. Automated mosaicing or stitching of live tissue video microscopy enables the visualization and analysis of subtle morphological structures and large scale vessel network architecture in tissues like the mesentery. But mosacing can be challenging if there are deformable, motion-blurred, textureless, feature-poor frames. Feature-based methods perform poorly in such cases for the lack of distinctive keypoints. Standard single block correlation matching strategies might not provide robust registration due to deformable content. In addition, the panorama suffers if there is motion blur present in a sequence. To handle these challenges, we propose a novel algorithm, Deformable Normalized Cross Correlation (DNCC) image matching with RANSAC to establish robust registration. Besides, to produce seamless panorama from motion-blurred frames we present gradient blending method based on image edge information. The DNCC algorithm is applied on Frog Mesentery sequences. Our result is compared with PSS/AutoStitch [1, 2] to establish the efficiency and robustness of the proposed DNCC method.

Visual odometry has gained increasing attention due to the proliferation of unmanned aerial vehicles, self-driving cars, and other autonomous robotics systems. Landmark detection and matching are critical for visual localization. While current methods rely upon point-based image features or descriptor mappings we consider landmarks at the object level. In this paper, we propose LMNet a deep learning based landmark matching pipeline for city-scale, aerial images of urban scenes. LMNet consists of a Siamese network, extended with a multi-patch based matching scheme, to handle offcenter landmarks, varying landmark scales, and occlusions of surrounding structures. While there exist a number of landmark recognition benchmark datasets for ground-based and nadir aerial or satellite imagery, there is a lack of datasets and results for oblique aerial imagery. We use a unique unsupervised multi-view landmark image generation pipeline for training and testing the proposed matching pipeline using over 0.5 million real landmark patches. Results for aerial landmark matching across four cities show promising results.

RTip is a tool to quantify plant root growth velocity using high resolution microscopy image sequences at sub-pixel accuracy. The fully automated RTip tracker is designed for high-throughput analysis of plant phenotyping experiments with episodic perturbations. RTip is able to auto-skip past these manual intervention perturbation activity, i.e. when the root tip is not under the microscope, image is distorted or blurred. RTip provides the most accurate root growth velocity results with the lowest variance (i.e. localization jitter) compared to six tracking algorithms including the top performing unsupervised Discriminative Correlation Filter Tracker and the Deeper and Wider Siamese Network. RTip is the only tracker that is able to automatically detect and recover from (occlusion-like) varying duration perturbation events.

In this paper we introduce a novel end-to-end framework for generation of large, aerial, city-scale, realistic synthetic image sequences with associated accurate and precise camera metadata. The two main purposes for this data are (i) to enable objective, quantitative evaluation of computer vision algorithms and methods such as feature detection, description, and matching or full computer vision pipelines such as 3D reconstruction; and (ii) to supply large amounts of high quality training data for deep learning guided computer vision methods. The proposed framework consists of three main modules, a 3D voxel renderer for data generation, a deep neural network for artifact removal, and a quantitative evaluation module for Multi-View Stereo (MVS) as an example. The 3D voxel renderer enables generation of seen or unseen views of a scene from arbitrary camera poses with accurate camera metadata parameters. The artifact removal module proposes a novel edge-augmented deep learning network with an explicit edgemap processing stream to remove image artifacts while preserving and recovering scene structures for more realistic results. Our experiments on two urban, city-scale, aerial datasets for Albuquerque (ABQ), NM and Los Angeles (LA), CA show promising results in terms of structural similarity to real data and accuracy of reconstructed 3D point clouds.

Detection of moving objects is a critical component of many computer vision tasks. Recently, deep learning architectures have been developed for supervised learning based moving object change detection. Some top performing architectures, like FgSegNet are single frame spatial appearance cue-based detection and tend to overfit to the training videos. We propose a novel compact multi-cue autoencoder deep architecture, Motion U-Net (MU-Net) for robust moving object detection that generalizes much better than FgSegNet and requires nearly 30 times fewer weight parameters. Motion and change cues are estimated using a multi-modal background subtraction module combined with flux tensor motion estimation. MU-Net was trained and evaluated on the CDnet-2014 change detection challenge video sequences and had an overall F-measure of 0.9369. We used the unseen SBI-2015 video dataset to assess generalization capacity where MU-Net had an F-measure of 0.7625 while FgSegNet_v2 was 0.3519, less than half the MU-Net accuracy. The source code of the Motion U-Net is available at https://github.com/CIVA-Lab/Motion-U-Net.

Characterizing the spatial relationship between blood vessel and lymphatic vascular structures, in the mice dura mater tissue, is useful for modeling fluid flows and changes in dynamics in various disease processes. We propose a new deep learning-based approach to fuse a set of multi-channel single-focus microscopy images within each volumetric z-stack into a single fused image that accurately captures as much of the vascular structures as possible. The red spectral channel captures small blood vessels and the green fluorescence channel images lymphatics structures in the intact dura mater attached to bone. The deep architecture Multi-Channel Fusion U-Net (MCFU-Net) combines multi-slice regression likelihood maps of thin linear structures using max pooling for each channel independently to estimate a slice-based focus selection map. We compare MCFU-Net with a widely used derivative-based multi-scale Hessian fusion method [8]. The multi-scale Hessian-based fusion produces dark-halos, non-homogeneous backgrounds and less detailed anatomical structures. Perception based no-reference image quality assessment metrics PIQUE, NIQE, and BRISQUE confirm the effectiveness of the proposed method.

Cyber foraging has been shown to be especially effective for augmenting low-power Internet-of-Thing (IoT) devices by offloading video processing tasks to nearby edge/cloud computing servers. Factors such as dynamic network conditions, concurrent user access, and limited resource availability, cause offloading decisions that negatively impact overall processing throughput and end-user delays. Moreover, edge/cloud platforms currently offer both Virtual Machine (VM) and serverless computing pricing models, but many existing edge offloading approaches only investigate single VM-based offloading performance. In this paper, we propose a predictive (NP-complete) scheduling-based offloading framework and a heuristic-based counterpart that use machine learning to dynamically decide what combinations of functions or single VM needs to be deployed so that tasks can be efficiently scheduled. We collected over 10,000 network and device traces in a series of realistic experiments relating to a protest crowds incident management application. We then evaluated the practicality of our predictive cyber foraging approach using trace-driven simulations for up to 1000 devices. Our results indicate that predicting single VM offloading costs: (a) leads to near-optimal scheduling in 70% of the cases for service function chaining, and (b) offers a 40% gain in performance over traditional baseline estimation techniques that rely on simple statistics for estimations in the case of single VM-offloading. Considering a series of visual computing offloading scenarios, we also validate our approach benefits of using online versus offline machine learning models for predicting offloading delays.

Automated curvilinear image segmentation is a crucial step to characterize and quantify the morphology of blood vessels across scale. We propose a dual pipeline RF_OFB+U-NET that fuses U-Net deep learning features with a low level image feature filter bank using the random forests classifier for vessel segmentation. We modify the U-Net CNN architecture to provide a foreground vessel regression likelihood map that is used to segment both arteriole and venule blood vessels in mice dura mater tissues. The hybrid approach combining both hand-crafted and learned features was tested on 60 epifluores-cence microscopy images and improved the segmentation of thin vessel structures by nearly 5% using the Dice similarity coefficient compared to U-Net.

Regularization methods are used widely in image selective smoothing and edge preserving restoration of noisy images. Traditional methods utilize image gradients within regularization function for controlling the smoothing and can produce artifacts when noise levels are higher. In this paper, we consider a robust image adaptive exponent driven regularization for filtering noisy images with salient feature preservation. Our spatially adaptive variable exponent function depends on a continuous switch based on the eigenvalues of structure tensor which identifies noisy edges, and corners with higher accuracy. Structure tensor eigenvalues encode various image features and we consider a spatially varying continuous map which provides multiscale edge maps of natural images. By embedding the structure tensor-based exponent in a well-defined regularization model, we obtain denoising filters which are capable of obtaining good feature preserving image restoration. The GPU-based implementation computes the edge map in real time at 45-60 frames/s depending on the GPU card. Multiscale structure tensor-based spatially adaptive variable exponent provides reliable edge maps and compared with standard edge detectors it is robust under various noisy conditions. Moreover, filtering based on the multiscale variable exponent map method outperforms L0 sparse gradient-based image smoothing and related filters.

Multi-object tracking is a fundamental computer vision task with a wide variety of real-life applications ranging from surveillance and monitoring to biomedical video analysis. Multi-object tracking is a challenging task due to complications caused by object appearance changes, complex object dynamics, clutter in the environment, and partial or full occlusions. In this paper, we propose a time-efficient detection-based multi-object tracking system using a three-step cascaded data association scheme that combines a fast spatial distance only short-term data association, a robust tracklet linking step using discriminative object appearance models, and an explicit occlusion handling unit relying not only on tracked objects' motion patterns but also on environmental constraints such as presence of potential occlud-ers in the scene. Our experiments on UA-DETRAC multi-object tracking benchmark dataset consisting of challenging real-world traffic videos show promising results against state-of-the-art trackers.

A desirable smart traffic-monitoring and street-safety system can elicit and support the intervention of law enforcement agencies or medical staff. Recently, there has been a dramatically higher demand for such smart systems. To this end, the International Workshop on Traffic and Street Surveillance for Safety and Security (IWT4S) was organized in conjunction with the 15th IEEE International Conference on Advanced Video and Signal-based Surveillance (AVSS 2018). Our goal is to advance the state-of-the-art detection and tracking algorithms and provide a comprehensive performance evaluation for them. We evaluate 5 submitted detection and 7 submitted tracking methods on the large-scale UA-DETRAC benchmark, and the results are shared publicly on the website http://detrac-db. rit.albany.edu. We expect this challenge to advance the research and development of new detection and tracking methods for transportation applications.

Automatic image classification systems for indirect immunofluorescence (IIF) labeling of human epithelial (HEp-2) cell specimens are needed to improve the efficient management of autoimmune diseases. In this paper, we propose to classify HEp-2 cell specimen imagery using a combination of local features and deep learning features extracted from the IIF images. Two local descriptors are used to capture texture information, namely: Rotation Invariant Co-occurrence among Local Binary Patterns (RIC-LBP) extending the LBP descriptor and Joint Motif Labels (JML) based on the Peano scan motif concept. Deep learning features are then extracted using the VGG-19 image classification network. Finally, all descriptors are combined using a late fusion approach with a Random Forests (RF) classifier with seven output classes. Experimental results show that our proposed framework achieves a mean class accuracy of 92.11% with five-fold cross validation using the RF classifier with 1000 trees on the HEp-2 specimen benchmark dataset, which outperforms the state-of-the-art accuracy on this dataset.

New paradigms such as Mobile Edge Computing (MEC) are becoming feasible for use in, e.g., real-time decision-making during disaster incident response to handle the data deluge occurring in the network edge. However, MEC deployments today lack flexible IoT device data handling such as handling user preferences for real-time versus energy-efficient processing. Moreover, MEC can also benefit from a policy-based edge routing to handle sustained performance levels with efficient energy consumption. In this paper, we study the potential of MEC to address application issues related to energy management on constrained IoT devices with limited power sources, while also providing low-latency processing of visual data being generated at high resolutions. Using a facial recognition application that is important in disaster incident response scenarios, we propose a novel “offload decision-making” algorithm that analyzes the tradeoffs in computing policies to offload visual data processing (i.e., to an edge cloud or a core cloud) at low-to-high workloads. This algorithm also analyzes the impact on energy consumption in the decision-making under different visual data consumption requirements (i.e., users with thick clients or thin clients). To address the processing-throughput versus energy-efficiency tradeoffs, we propose a “Sustainable Policy-based Intelligence-Driven Edge Routing” algorithm that uses machine learning within Mobile Ad hoc Networks. This algorithm is energy aware and improves the geographic routing baseline performance (i.e., minimizes impact of local minima) for throughput performance sustainability, while also enabling flexible policy specification. We evaluate our proposed algorithms by conducting experiments on a realistic edge and core cloud testbed in the GENI Cloud infrastructure, and recreate disaster scenes of tornado damages within simulations. Our empirical results show how MEC can provide flexibility to users who desire energy conservation over low latency or vice versa in the visual data processing with a facial recognition application. In addition, our simulation results show that our routing approach outperforms existing solutions under diverse user preferences, node mobility, and severe node failure conditions.

Multiview learning has shown promising potential in many applications. However, most techniques are focused on either view consistency, or view diversity. In this paper, we introduce a novel multiview boosting algorithm, called Boost.SH, that computes weak classifiers independently of each view but uses a shared weight distribution to propagate information among the multiple views to ensure consistency. To encourage diversity, we introduce randomized Boost.SH and show its convergence to the greedy Boost.SH solution in the sense of minimizing regret using the framework of adversarial multiarmed bandits. We also introduce a variant of Boost.SH that combines decisions from multiple experts for recommending views for classification. We propose an expert strategy for multiview learning based on inverse variance, which explores both consistency and diversity. Experiments on biometric recognition, document categorization, multilingual text, and yeast genomic multiview data sets demonstrate the advantage of Boost.SH (85%) compared with other boosting algorithms like AdaBoost (82%) using concatenated views and substantially better than a multiview kernel learning algorithm (74%).

Multi-object tracking is an important computer vision task with wide variety of real-life applications from surveillance and monitoring to biomedical video analysis. Multi-object tracking is a challenging problem due to complications such as partial or full occlusions, factors affecting object appearance, object interaction dynamics, etc. and computational cost. In this paper, we propose a detection-based multi-object tracking system that uses a two-step data association scheme to ensure time efficiency while preserving tracking accuracy; a robust but discriminative object appearance model that compares object color attributes using a novel color correlation cost matrix; and a framework that handles occlusions through prediction. Our experiments on UA-DETRAC multi-object tracking benchmark dataset consisting of challenging real-world traffic videos show promising results against state-of-the-art trackers.

The rapid advances of transportation infrastructure have led to a dramatic increase in the demand for smart systems capable of monitoring traffic and street safety. Fundamental to these applications are a community-based evaluation platform and benchmark for object detection and multi-object tracking. To this end, we organize the AVSS2017 Challenge on Advanced Traffic Monitoring, in conjunction with the International Workshop on Traffic and Street Surveillance for Safety and Security (IWT4S), to evaluate the state-of-the-art object detection and multi-object tracking algorithms in the relevance of traffic surveillance. Submitted algorithms are evaluated using the large-scale UA-DETRAC benchmark and evaluation protocol. The benchmark, the evaluation toolkit and the algorithm performance are publicly available from the website http://detrac-db.rit.albany.edu.

Persistent detection and tracking of moving vehicles in airborne imagery provide indispensable information for many traffic surveillance applications including traffic monitoring and management, navigation systems, activity recognition and event detection. This paper presents a collaborative Spatial Pyramid Context-aware detection and Tracking system (SPCT) for moving vehicles in dense urban aerial imagery. The proposed system is composed of one master tracker that usually relies on visual object features and two auxiliary trackers based on object temporal motion information that will be called dynamically to assist master tracker. SPCT utilizes image spatial context at different level to make the video tracking system resistant to occlusion, background noise and improve target localization accuracy. We chose a pre-selected seven-channel complementary features including RGB color, intensity and spatial pyramid of HoG (PHoG) and exploit integral histogram as building block to meet the demands of real-time performance. The extensive experiments on ARGUS and ABQ wide aerial video and comparison with state-of-the-art single object trackers confirm that combining complementary tracking cues in an intelligent fusion framework is essential to address the challenges of persistent tracking in low frame rate Wide Aerial Motion Imagery (WAMI).

Segmenting curvilinear structures in retinal images is important in early diagnosing of some diseases and monitoring their progress. In this work, we proposed an automatic segmentation method to extract vascular network in CHASE data set. We utilized deep learning framework to build our layers that accept image patches as input and produce the segmented image as output. Our work characterized by its robust detection for the vascular tree in spite of challenged conditions in these images such as illumination and contrast variations, tiny and close vessels, crossing area and optic disk thick boundary. We achieved the highest sensitivity among all the state of the art methods in the literature with comparable specificity and better results in terms of accuracy. Our experimental results outperformed the best algorithm in sensitivity by 5%, as well as, we achieved an accuracy equal to 0.9630. In addition, we obtained better performance in terms of all evaluation methods after applying simple morphological operations to our final segmentation results.

Facial expressions play an important role in communication. Impaired facial expression is a common sign of numerous medical conditions, particularly neurological disorders. Accurate automated systems are needed to recognize facial expressions and to reveal valuable information that can be used for diagnosis and monitoring of neurological disorders. This paper presents a novel deep learning approach for automatic facial expression recognition. The proposed architecture first segments the facial components known to be important for facial expression recognition and forms an iconized image; then performs facial expression classification using the obtained iconized facial components image combined with the raw facial images. This approach integrates local part-based features with holistic facial information for robust facial expression recognition. Preliminary experimental results using the proposed system achieved 93.43% facial expression recognition accuracy, more than 6% accuracy improvement compared to facial expression recognition from raw input images. The goal of the proposed study is design of a noninvasive, objective, and quantitative facial expression recognition system to assist diagnosis and monitoring of neurological disorders affecting facial expressions.

This paper presents an Android-based mobile application named MUSeed that automatically computes seed morphometry utilizing image processing techniques. Unlike most of the existing tools, MUSeed does not impose restrictions on arrangement of seeds since it is capable of handling touching seed instances. First, RGB color space is converted to RG-chromaticity space to reduce the influence of shadows and illumination variations. Then K-means clustering is performed to segment the seeds from the background. To split touching seeds in the image, a Watershed with Edge-Augmented Markers (WEAM) algorithm and Concave Point Analysis (CPA) are developed. After that, a fitness function is performed to select the most appropriate result. MUSeed is benchmarked against other similar tools on 7 cultivars of American elderberry seeds and shows promising results.

Segmenting microvascular structures is an important requirement in understanding angioadaptation by which vascular networks remodel their morphological structures. Accurate segmentation for separating microvasculature structures is important in quantifying remodeling process. In this work, we utilize a deep convolutional neural network (CNN) framework for obtaining robust segmentations of microvasculature from epifluorescence microscopy imagery of mice dura mater. Due to the inhomogeneous staining of the microvasculature, different binding properties of vessels under fluorescence dye, uneven contrast and low texture content, traditional vessel segmentation approaches obtain sub-optimal accuracy. We consider a deep CNN for the purpose keeping small vessel segments and handle the challenges posed by epifluorescence microscopy imaging modality. Experimental results on ovariectomized - ovary removed (OVX) - mice dura mater epifluorescence microscopy images show that the proposed modified CNN framework obtains an highest accuracy of 99% and better than other vessel segmentation methods.

In the event of natural or man-made disasters, providing rapid situational awareness through video/image data collected at salient incident scenes is often critical to the first responders. However, computer vision techniques that can process the media-rich and data-intensive content obtained from civilian smartphones or surveillance cameras require large amounts of computational resources or ancillary data sources that may not be available at the geographical location of the incident. In this paper, we propose an incident-supporting visual cloud computing solution by defining a collection, computation, and consumption (3C) architecture supporting fog computing at the network edge close to the collection/consumption sites, which is coupled with cloud offloading to a core computation, utilizing software-defined networking (SDN). We evaluate our 3C architecture and algorithms using realistic virtual environment test beds. We also describe our insights in preparing the cloud provisioning and thin-client desktop fogs to handle the elasticity and user mobility demands in a theater-scale application. In addition, we demonstrate the use of SDN for on-demand compute offload with congestion-avoiding traffic steering to enhance remote user quality of experience in a regional-scale application. The optimization between fogs computing at the network edge with core cloud computing for managing visual analytics reduces latency, congestion, and increases throughput.

We describe a fast and efficient camera pose refinement and Structure from Motion (SfM) method for sequential aerial imagery with applications to georegistration and 3-D reconstruction. Inputs to the system are 2-D images combined with initial noisy camera metadata measurements, available from on-board sensors (e.g., camera, global positioning system, and inertial measurement unit). Georegistration is required to stabilize the ground-plane motion to separate camera-induced motion from object motion to support vehicle tracking in aerial imagery. In the proposed approach, we recover accurate camera pose and (sparse) 3-D structure using bundle adjustment for sequential imagery (BA4S) and then stabilize the video from the moving platform by analytically solving for the image-plane-to-ground-plane homography transformation. Using this approach, we avoid relying upon image-to-image registration, which requires estimating feature correspondences (i.e., matching) followed by warping between images (in a 2-D space) that is an error prone process for complex scenes with parallax, appearance, and illumination changes. Both our SfM (BA4S) and our analytical ground-plane georegistration method avoid the use of iterative consensus combinatorial methods like RANdom SAmple Consensus which is a core part of many published approaches. BA4S is very efficient for long sequential imagery and is more than 130 times faster than VisualSfM, 35 times faster than MavMap, and about 274 times faster than Pix4D. Various experimental results demonstrate the efficiency and robustness of the proposed pipeline for the refinement of camera parameters in sequential aerial imagery and georegistration.

Malaria is a major global health threat. The standard way of diagnosing malaria is by visually examining blood smears for parasite-infected red blood cells under the microscope by qualified technicians. This method is inefficient and the diagnosis depends on the experience and the knowledge of the person doing the examination. Automatic image recognition technologies based on machine learning have been applied to malaria blood smears for diagnosis before. However, the practical performance has not been sufficient so far. This study proposes a new and robust machine learning model based on a convolutional neural network (CNN) to automatically classify single cells in thin blood smears on standard microscope slides as either infected or uninfected. In a ten-fold cross-validation based on 27,578 single cell images, the average accuracy of our new 16-layer CNN model is 97.37%. A transfer learning model only achieves 91.99% on the same images. The CNN model shows superiority over the transfer learning model in all performance indicators such as sensitivity (96.99% vs 89.00%), specificity (97.75% vs 94.98%), precision (97.73% vs 95.12%), F1 score (97.36% vs 90.24%), and Matthews correlation coefficient (94.75% vs 85.25%).

Reducing the amount of user driven input for interactive image segmentation enables faster and more precise foreground extraction of objects. A sparse collection of labeled seed points sampled over image regions can be quickly provided by the user using a few mouse clicks. Seed points are used for training an Elastic Body Spline classifier mapping function. We evaluate the efficiency and accuracy of user defined point inputs compared to fully manual boundary drawing that can be time consuming and automatic image segmentation methods that may not have sufficient accuracy. We show that using an average of just 8 labeled pixels (i.e. sparse set of seed points) the proposed EBS foreground-background thresholding method can achieve 90 percent accuracy compared to manual ground truth on the Berkeley BSDS500 benchmark.

Most Wide Area Motion Imagery (WAMI) based trackers use motion based cueing for detecting and tracking moving objects. The results are very high false alarm rates in urban environments with tall structures due to parallax effects. This paper proposes an accurate moving object detection method using a precise orthorectification approach for ground stabilization combined with accurate multiview depth maps to reduce the number of false positives induced by parallax effects by 90 percent. Proposed hybrid moving vehicle detection approach for large scale aerial urban imagery is based on fusion of motion detection mask obtained from median-based background subtraction and tall structures height mask provided by image depth map information. Using buildings mask enables us to improve the object level detection accuracy in terms of F-measure by 57 percent from 22.2% to 79.2%.

Human epithelial (HEp-2) cell specimens are obtained from indirect immunofluorescence (IIF) imaging for diagnosis and management of autoimmune diseases. Analysis of HEp2 cells is important and in this work we consider automatic cell segmentation and classification using spatial and texture pattern features and random forest classifiers. In this paper, we summarize our efforts in classification and segmentation tasks proposed in ICPR 2016 contest. For the cell level staining pattern classification (Task 1), we utilized texture features such as rotational invariant co-occurrence (RIC) versions of the well-known local binary pattern (LBP), median binary pattern (MBP), joint adaptive median binary pattern (JAMBP), and motif labels (ML) along with other optimized features. We report the classification results utilizing different classifiers such as the k-nearest neighbors (kNN), support vector machine (SVM), and random forest (RF). We obtained the best accuracy of 94.26% for six cell classes with RIC-LBP combined with a motif pattern co-occurrence labels (MCL). For specimen level staining pattern classification (Task 2) we utilize a combination RIC-LBP with RF classifier and obtain 80% accuracy for seven classes. For cell segmentation (Task 4), we use our optimized multiscale spatial feature bank along with RF classifier for pixel-wise labeling to achieve an F-measure of 84.26% for 1008 images.

It is increasingly appreciated that the tumor stroma is an integral part of cancer initiation, growth, and progression. Recently it has been shown that the stromal elements of tumors hold prognostic as well as response-predictive information. This work proposes a multi-scale image analysis and machine learning pipeline for epithelial versus stromal tissue identification in images of H&E stained breast cancer specimens. Unlike many studies that perform pixel or block-based epithelium-stroma classification, this pipeline includes an explicit image segmentation module. We first partition the H&E stain images into coherent partitions/superpixels, then extract a number of regional color and texture features from these partitions, and finally use support vector machine classifiers to classify them into epithelium and stroma classes. We propose a multi-scale hierarchical fuzzy c-means (HFCM) approach for segmentation of the images. We also investigate multi-scale feature extraction and descriptors. Our experimental results on Stanford Tissue Microarray Database show that multi-scale regional feature descriptors outperform single-scale feature descriptors. Experimental results also show that when the same set of regional features are used, classification of HFCM-based partitions outperforms classification of both regular blocks and SLIC superpixels.

Collecting ground-truth or gold standard annotations from expert pathologists for developing histopathology analytic algorithms and computer-aided diagnosis for cancer grading is an expensive and time consuming process. Efficient visualization and annotation tools are needed to enable ground-truthing large whole-slide imagery. KOLAM is our scalable, cross-platform framework for interactive visualization of 2D, 2D+t and 3D imagery of high spatial, temporal and spectral resolution. In the current work KOLAM has been extended to support rapid interactive labelling and correction of automatic image classifier-based region labels of the tissue microenvironment by pathologists. Besides annotating regions-of-interest (ROIs), KOLAM enables extraction of the corresponding large polygonal image subregions for input into automatic segmentation algorithms, single-click region label reassignment and maintaining hierarchical image subregions. Experience indicates that clinicians prefer simple-to-use interfaces that support rapid labelling of large image regions with minimal effort. The incorporation of easy-to-use tissue annotation features in KOLAM makes it an attractive candidate for integration within a multi-stage histopathology image analysis pipeline supporting assisted segmentation and labelling to improve whole-slide imagery (WSI) analytics.

The importance of the internet-of-things (IOT) is now an established reality. With that backdrop, the phenomenal emergence of cameras/sensors mounted on unmanned aerial, ground and marine vehicles (UAVs, UGVs, UMVs) and body worn cameras is a notable new development. The swarms of cameras and real-time computing thereof are at the heart of new technologies like connected cars, drone-based city-wide surveillance and precision agriculture, etc. Smart computer vision algorithms (with or without dynamic learning) that enable object recognition and tracking, supported by baseline video content summarization or 2D/3D image reconstruction of the scanned environment are at the heart of such new applications. In this article, we summarize our recent innovations in this space. We focus primarily on algorithms and architectural design considerations for video summarization systems.

Elastic body splines (EBS) belong to a family of splines introduced for biomedical image registration. EBS models the elastic deformation of a homogeneous isotropic elastic body subjected to external forces. The task of interactive image segmentation is framed as a semi-supervised interpolation where the basis functions are learned using the user provided seed points to model and predict the labels for the unlabeled pixels. Seed points are sparse compared to other methods that may require scribbles and regions. The spline for interpolating labels is the EBS which we compare to our previous work using Gaussian Elastic Body splines (GEBS) [1] for the task of interactive image segmentation. Experimental results show that the EBS is about 14 percent better, in terms of accuracy, than GEBS and significantly better than random walk and graph cut based segmentation. EBS is also 2.5 times faster than GEBS.

It is estimated that less than five percent of videos are currently analyzed to any degree. In addition to petabyte-sized multimedia archives, continuing innovations in optics, imaging sensors, camera arrays, (aerial) platforms, and storage technologies indicates that for the foreseeable future existing and new applications will continue to generate enormous volumes of video imagery. Contextual video summarizations and activity maps offers one innovative direction to tackling this Big Data problem in computer vision. The goal of this work is to develop semi-automatic exploitation algorithms and tools to increase utility, dissemination and usage potential by providing quick dynamic overview geospatial mosaics and motion maps. We present a framework to summarize (multiple) video streams from unmanned aerial vehicles (UAV) or drones which have very different characteristics compared to structured commercial and consumer videos that have been analyzed in the past. Using both metadata geospatial characteristics of the video combined with fast low-level image-based algorithms, the proposed method first generates mini-mosaics that can then be combined into geo-referenced meta-mosaics imagery. These geospatial maps enable rapid assessment of hours long videos with arbitrary spatial coverage from multiple sensors by generating quick look imagery, composed of multiple mini-mosaics, summarizing spatiotemporal dynamics such as coverage, dwell time, activity, etc. The overall summarization pipeline was tested on several DARPA Video and Image Retrieval and Analysis Tool (VIRAT) datasets. We evaluate the effectiveness of the proposed video summarization framework using metrics such as compression and hours of viewing time.

A robust camera pose refinement approach for sequential wide-area airborne imagery is proposed in this letter. Image frames are sequentially acquired, and with each frame, its corresponding position and orientation are approximately available from airborne platform inertial measurement unit and GPS sensors. In the proposed structure from motion (SfM) approach the available approximation of camera parameters (from low-certainty sensors) is directly used in an optimization stage. The putative matches obtained from the sequential matching paradigm are also directly used in the optimization with no early stage filtering (e.g., no RANSAC). A robust function is proposed and used to deal with outliers (mismatches). The full pipeline has been run over a set of wide-area motion imagery data collected by an airplane flying over different cities in the U.S. The results prove the power and efficiency of the proposed pipeline. Effectiveness of the proposed robust function is compared with some popular robust functions such as Cauchy and Huber using synthetic data.

Edge preserving regularization using partial differential equation (PDE)-based methods although extensively studied and widely used for image restoration, still have limitations in adapting to local structures. We propose a spatially adaptive multiscale variable exponent-based anisotropic variational PDE method that overcomes current shortcomings, such as over smoothing and staircasing artifacts, while still retaining and enhancing edge structures across scale. Our innovative model automatically balances between Tikhonov and total variation (TV) regularization effects using scene content information by incorporating a spatially varying edge coherence exponent map constructed using the eigenvalues of the filtered structure tensor. The multiscale exponent model we develop leads to a novel restoration method that preserves edges better and provides selective denoising without generating artifacts for both additive and multiplicative noise models. Mathematical analysis of our proposed method in variable exponent space establishes the existence of a minimizer and its properties. The discretization method we use satisfies the maximum-minimum principle which guarantees that artificial edge regions are not created. Extensive experimental results using synthetic, and natural images indicate that the proposed multiscale Tikhonov-TV (MTTV) and dynamical MTTV methods perform better than many contemporary denoising algorithms in terms of several metrics, including signal-to-noise ratio improvement and structure preservation. Promising extensions to handle multiplicative noise models and multichannel imagery are also discussed.

Elastic body splines (EBS) belonging to the family of 3D splines were recently introduced to capture tissue deformations within a physical model-based approach for non-rigid biomedical image registration [1]. EBS model the displacement of points in a 3D homogeneous isotropic elastic body subject to forces. We propose a novel extension of using elastic body splines for learning driven figure-ground segmentation. The task of interactive image segmentation, with user provided foreground-background labeled seeds or samples, is formulated as learning an interpolating pixel classification function that is then used to assign labels for all unlabeled pixels in the image. The spline function we chose to model the supervised pixel classifier is the Gaussian elastic body spline (GEBS) which can use sparse scribbles from the user and has a closed form solution enabling a fast on-line implementation. Experimental results demonstrate the applicability of the GEBS approach for image segmentation. The GEBS method for interactive foreground image labeling shows promise and outperforms a previous approach using the thin-plate spline model.

This paper addresses the challenging problem of recognition and classification of textured surfaces under illumination variation, geometric transformations and noisy sensor measurements. We propose a new texture operator, Adaptive Median Binary Patterns (AMBP) that extends our previous Median Binary Patterns (MBP) texture feature. The principal idea of AMBP is to hash small local image patches into a binary pattern text on by fusing MBP and Local Binary Patterns (LBP) operators combined with using self-adaptive analysis window sizes to better capture invariant microstructure information while providing robustness to noise. The AMBP scheme is shown to be an effective mechanism for non-parametric learning of spatially varying image texture statistics. The local distribution of rotation invariant and uniform binary pattern subsets extended with more global joint information are used as the descriptors for robust texture classification. The AMBP is shown to outperform recent binary pattern and filtering-based texture analysis methods on two large texture corpora (CUReT and KTH_TIPS2-b) with and without additive noise. The AMBP method is slightly superior to the best techniques in the noiseless case but significantly outperforms other methods in the presence of impulse noise.

The National Library of Medicine (NLM) is developing a digital chest X-ray (CXR) screening system for deployment in resource constrained communities and developing countries worldwide with a focus on early detection of tuberculosis. A critical component in the computer-aided diagnosis of digital CXRs is the automatic detection of the lung regions. In this paper, we present a nonrigid registration-driven robust lung segmentation method using image retrieval-based patient specific adaptive lung models that detects lung boundaries, surpassing state-of-the-art performance. The method consists of three main stages: 1) a content-based image retrieval approach for identifying training images (with masks) most similar to the patient CXR using a partial Radon transform and Bhattacharyya shape similarity measure, 2) creating the initial patient-specific anatomical model of lung shape using SIFT-flow for deformable registration of training masks to the patient CXR, and 3) extracting refined lung boundaries using a graph cuts optimization approach with a customized energy function. Our average accuracy of 95.4% on the public JSRT database is the highest among published results. A similar degree of accuracy of 94.1% and 91.7% on two new CXR datasets from Montgomery County, MD, USA, and India, respectively, demonstrates the robustness of our lung segmentation approach.

Tuberculosis is a major health threat in many regions of the world. Opportunistic infections in immunocompromised HIV/AIDS patients and multi-drug-resistant bacterial strains have exacerbated the problem, while diagnosing tuberculosis still remains a challenge. When left undiagnosed and thus untreated, mortality rates of patients with tuberculosis are high. Standard diagnostics still rely on methods developed in the last century. They are slow and often unreliable. In an effort to reduce the burden of the disease, this paper presents our automated approach for detecting tuberculosis in conventional posteroanterior chest radiographs. We first extract the lung region using a graph cut segmentation method. For this lung region, we compute a set of texture and shape features, which enable the X-rays to be classified as normal or abnormal using a binary classifier. We measure the performance of our system on two datasets: a set collected by the tuberculosis control program of our local county's health department in the United States, and a set collected by Shenzhen Hospital, China. The proposed computer-aided diagnostic system for TB screening, which is ready for field deployment, achieves a performance that approaches the performance of human experts. We achieve an area under the ROC curve (AUC) of 87% (78.3% accuracy) for the first set, and an AUC of 90% (84% accuracy) for the second set. For the first set, we compare our system performance with the performance of radiologists. When trying not to miss any positive cases, radiologists achieve an accuracy of about 82% on this set, and their false positive rate is about half of our system's rate.

Visual feature-based tracking systems need to adapt to variations in the appearance of an object and in the scene for robust performance. Though these variations may be small for short time steps, they can accumulate over time and deteriorate the quality of the matching process across longer intervals. Tracking in aerial imagery can be challenging as viewing geometry, calibration inaccuracies, complex ight paths and background changes combined with illumination changes, and occlusions can result in rapid appearance change of objects. Balancing appearance adaptation with stability to avoid tracking non-target objects can lead to longer tracks which is an indicator of tracker robustness. The approach described in this paper can handle affine changes such as rotation by explicit orientation estimation, scale changes by using a multiscale Hessian edge detector and drift correction by using segmentation. We propose an appearance update approach that handles the 'drifting' problem using this adaptive scheme within a tracking environment that is comprised of a rich feature set and a motion model.

Muscle satellite cells, also known as myogenic precursor cells, are the dedicated stem cells responsible for postnatal skeletal muscle growth, repair, and hypertrophy. Biological studies aimed at describing satellite cell activity on their host myofiber using timelapse light microscopy enable qualitative study, but high-throughput automatic tracking of satellite cells translocating on myofibers is very difficult due to their complex motion across the three-dimensional surface of myofibers and the lack of discriminating cell features. Other complicating factors include inhomogeneous illumination, fixed focal plane, low contrast, and stage motion. We propose a semi-automated approach for satellite cell tracking on myofibers consisting of registration with illumination correction, background subtraction and particle filtering. Initial experimental results show the effectiveness of the approach.

The task of automated object tracking and performance assessment in low frame rate, persistent, wide spatial coverage motion imagery is an emerging research domain. The collection of hundreds to tens of thousands of dense trajectories produced by such automatic algorithms along with the subset of manually verified tracks across several coordinate systems require new tools for effective human computer interfaces and exploratory trajectory visualization. We describe an interactive visualization system that supports very large gig pixel per frame video, facilitates rapid, intuitive monitoring and analysis of tracking algorithm execution, provides visual methods for the inter comparison of very long manual tracks with multi segmented automatic tracker outputs, and a flexible KOLAM Tracking Simulator (KOLAM-TS) middleware that generates visualization data by automating the object tracker performance testing and benchmarking process.

We present a cell cycle analysis that aims towards improving our previous work by adding another channel and using one more dimension. The data we use is a set of 3D images of mouse cells captured with a spinning disk confocal microscope. All images are available in two channels showing the chromocenters and the fluorescently marked protein PCNA, respectively. In the present paper, we will describe our recent colocalization study in which we use Hessian-based blob detectors in combination with radial features to measure the degree of overlap between both channels. We show that colocalization performed in such a way provides additional discriminative power and allows us to distinguish between phases that we were not able to distinguish with a single 2D channel.

Graph partitioning active contours (GPAC) is a recently introduced approach that elegantly embeds the graph-based image segmentation problem within a continuous optimization framework. GPAC can be used within parametric snake-based or implicit level set-based active contour continuous paradigms for image partitioning. However, GPAC similar to many other graph-based approaches has quadratic memory requirements which severely limits the scalability of the algorithm to practical problem domains. An N xN image requires O(N(4)) computation and memory to create and store the full graph of pixel inter-relationships even before the start of the contour optimization process. For example, an 1024x1024 grayscale image needs over one terabyte of memory. Approximations using tile/block-based or superpixel-based multiscale grouping of the pixels reduces this complexity by trading off accuracy. This paper describes a new algorithm that implements the exact GPAC algorithm using a constant memory requirement of a few kilobytes, independent of image size.

The large amount of data produced by biological live cell imaging studies of cell behavior requires accurate automated cell segmentation algorithms for rapid, unbiased and reproducible scientific analysis. This paper presents a new approach to obtain precise boundaries of cells with complex shapes using ridge measures for initial detection and a modified geodesic active contour for curve evolution that exploits the halo effect present in phase-contrast microscopy. The level set contour evolution is controlled by a novel spatially adaptive stopping function based on the intensity profile perpendicular to the evolving front. The proposed approach is tested on human cancer cell images from LSDCAS and achieves high accuracy even in complex environments.

Computer assisted or automated histological grading of tissue biopsies for clinical cancer care is a long-studied but challenging problem. It requires sophisticated algorithms for image segmentation, tissue architecture characterization, global texture feature extraction, and high-dimensional clustering and classification algorithms. Currently there are no automatic image-based grading systems for quantitative pathology of cancer tissues. We describe a novel approach for tissue segmentation using fuzzy spatial clustering, vector-based multiphase level set active contours and nuclei detection using an iterative kernel voting scheme that is robust even in the case of clumped touching nuclei. Early results show that we can reach a 91% detection rate compared to manual ground truth of cell nuclei centers across a range of prostate cancer grades.

We present a new approach aimed at fast multiframe registration of airborne video collected by moving platforms such as unmanned aerial vehicles. Registration is used to enable separating the moving objects from the stationary background, which is similar to estimating the egomotion of the sensor. The proposed registration algorithm is used to match the moving background and remap video frames into a common co-ordinate system in order to stabilize that segment of video. The major modules include dense feature detection, sparse prominent edge and corner-based feature block identification, feature block region matching to extract control points, confidence weighted robust projective transformation estimation and image warping to register a segment of frames within a given temporal window. The proposed method is shown to produce good results with small image registration error.

Persistent object tracking in complex and adverse environments can be improved by fusing information from multiple sensors and sources. We present a new moving object detection and tracking system that robustly fuses infrared and visible video within a level set framework. We also introduce the concept of the flux tensor as a generalization of the 3D structure tensor for fast and reliable motion detection without eigen-decomposition. The infrared flux tensor provides a coarse segmentation that is less sensitive to illumination variations and shadows. The Beltrami color metric tensor is used to define a color edge stopping function that is fused with the infrared edge stopping function based on the grayscale structure tensor. The min fusion operator combines salient contours in either the visible or infrared video and drives the evolution of the multispectral geodesic active contour to refine the coarse initial flux tensor motion blobs. Multiple objects are tracked using correspondence graphs and a cluster trajectory analysis module that resolves incorrect merge events caused by under-segmentation of neighboring objects or partial and full occlusions. Long-term trajectories for object clusters are estimated using Kalman filtering and watershed segmentation. We have tested the persistent object tracking system for surveillance applications and demonstrate that fusion of visible and infrared video leads to significant improvements for occlusion handling and disambiguating clustered groups of objects

Searching for relevant knowledge across heterogeneous geospatial databases requires an extensive knowledge of the semantic meaning of images, a keen eye for visual patterns, and efficient strategies for collecting and analyzing data with minimal human intervention. In this paper, we present our recently developed content-based multimodal Geospatial Information Retrieval and Indexing System (GeoIRIS) which includes automatic feature extraction, visual content mining from large-scale image databases, and high-dimensional database indexing for fast retrieval. Using these underpinnings, we have developed techniques for complex queries that merge information from heterogeneous geospatial databases, retrievals of objects based on shape and visual characteristics, analysis of multiobject relationships for the retrieval of objects in specific spatial configurations, and semantic models to link low-level image features with high-level visual descriptors. GeoIRIS brings this diverse set of technologies together into a coherent system with an aim of allowing image analysts to more rapidly identify relevant imagery. GeoIRIS is able to answer analysts' questions in seconds, such as "given a query image, show me database satellite images that have similar objects and spatial relationship that are within a certain radius of a landmark."

In this paper, a new method for motion flow estimation that considers errors in all the derivative measurements is presented. Based on the total least squares (TLS) model, we accurately estimate the motion flow in the general noise case by combining noise model (in form of covariance matrix) with a parametric motion model. The proposed algorithm is tested on two different types of biological motion, a growing plant root and a gastrulating embryo, with sequences obtained microscopically. The local, instantaneous velocity field estimated by the algorithm reveals the behavior of the underlying cellular elements.

This paper presents a framework for the efficient retrieval of satellite imagery from large-scale databases. With the ever-expanding volume of imagery being acquired from satellite platforms it has become increasingly important to locate specific areas of interest within a large database of images. Identifying relevant areas within image databases can be thought of as finding the needle in the haystack problem; too often for a particular task or application there exist a small number of useful images hidden among millions of images. The motivation behind the work presented here is that through the use of geospatial image retrieval systems, the number of scenes that image analysts must manually examine may be decreased dramatically. By using a geospatial image retrieval system as a tool, analysts no longer must manually examine the entire database of imagery, but instead can limit their search to a subset identified by our retrieval system. The techniques that are introduced in this paper have been developed in our image retrieval system named GeoIRIS: Geospatial Information Retrieval and Indexing System.

A novel image interpolation method for resolution enhancement of still images is proposed. The approach consists of a nonlinear mapping from pixel index space to color space based on the low-resolution image so that nonintegral pixel indices in a super-sampling can be mapped to colors, i.e., interpolated. This nonlinear mapping is based on an extensor transformation of 2D pixel indices to an, N-dimensional vector space that encodes the Euclidean proximity interrelationships among a neighborhood of N pixels. Experimental results indicate that extensor-based interpolation yields results that are qualitatively superior to classical bilinear and bicubic interpolation, that has important advantages in comparison to edge-directed and optimal-recovery interpolation methods, and that has comparable or lower computational cost.

Tracking both structure and motion of nonrigid objects from monocular images is an important problem in vision. In this paper, a hierarchical method which integrates local analysis (that recovers small details) and global analysis (that appropriately limits possible nonrigid behaviors) is developed to recover dense depth values and nonrigid motion from a sequence of 2D satellite cloud images without any prior knowledge of point correspondences. This problem is challenging not only due to the absence of correspondence information but also due to the lack of depth cues in the 2D cloud images (scaled orthographic projection). In our method, the cloud images are segmented into several small regions and local analysis is performed for each region. A recursive algorithm is proposed to integrate local analysis with appropriate global fluid model constraints, based on which a structure and motion analysis system, SMAS, is developed. We believe that this is the first reported system in estimating dense structure and nonrigid motion under scaled orthographic views using fluid model constraints. Experiments on cloud image sequences captured by meteorological satellites (GOES-8 and GOES-9) have been performed using our system, along with their validation and analyses. Both structure and 3D motion correspondences are estimated to subpixel accuracy. Our results are very encouraging and have many potential applications in earth and space sciences, especially in cloud models for weather prediction.

A piecewise-affine image registration method is proposed to compute the displacement field in an image sequence of an aerodynamic object experiencing heavy winds in a wind tunnel. Our method is useful for tracking objects whose net 3-D motion is fully characterized by a non-rigid motion and a significant component that is common to all parts of the object. A set of control points has been introduced on the object surface to help divide each image into a set of triangles and enable tracking each triangular area. The computed velocity field is piecewise affine for each triangle and is continuous across the boundary between any two adjacent triangles. Of interest is the accuracy of the registration process, and its estimate based on local moments and shape distortion that can be computed from the original image and the motion compensated version of its registered pair. The method has been applied to several images and the experimental results are presented.

A novel hybrid wavelet video coding algorithm termed video significance-linked connected component analysis (VSLCCA) is developed for very low bit-rate applications. In the proposed VSLCCA codec, first, fine-tuned motion estimation based on the H.263 Recommendation is developed to reduce temporal redundancy, and exhaustive overlapped block motion compensation is utilized to ensure coherency in motion compensated error frames. Second, the wavelet transform is applied to each coherent motion compensated error frame to attain global energy compaction. Third, significant fields of wavelet-transformed error frames are organized and represented as significance-linked connected components so that both the within-subband clustering and the cross-scale dependency are exploited. Last, the horizontal and vertical components of motion vectors are encoded separately using adaptive arithmetic coding while significant wavelet coefficients are encoded in bit-plane order by using high order Markov source modeling and adaptive arithmetic coding. Experimental results on eight standard MPEG-4 test sequences show that for intraframe coding, on average the proposed codec exceeds H.263 and ZTE (zero-tree entropy) in peak signal-to-noise ratio by as much as 2.07 and 1.38 dB at 28 kbit/s, respectively. For entire sequence coding, VSLCCA is superior to H.263 and ZTE by 0.35 and 0.71 dB on average, respectively.

An automatic spatio-temporal video sequence segmentation algorithm is proposed. To address this very difficult computer vision problem, several novel algorithms have been developed which use both spatial and temporal information. First, a novel temporal segmentation algorithm is developed based on our previous work in motion estimation. Second, an iterative split-and-merge spatial segmentation scheme is proposed with an initial segmentation being provided by a recursive conditioned dilation operation merging pixels into homogeneous regions, followed by an iterative refinement algorithm to obtain the final spatial segmentation. Third, temporal and spatial segmentation are linked to form spatio-temporal segmentation with the shape of each object being simplified by a morphological close-opening operation to obtain the final segmentation. Performance evaluation shows that the developed algorithm can successfully segment moving objects without any human intervention.

An efficient image database system is developed. The most important features of the proposed system include compressed domain indexing, searching by using scalable features, and progressive image transmission. User interaction is involved both at the search refinement stage and in the display of the query results. The most important query types include query by color layout and query by wavelet-coefficient clustering information. The indexing and searching algorithms are tightly coupled with the underlying image compression algorithm by which means the images are stored in the database, reducing both the complexity and the storage requirements of the database management system. In this research, we utilize our previously developed (B.-B. Chai et al., 1997, 1998) high-performance wavelet image coding algorithm, termed "significance-linked connected component analysis", which not only renders a very high compression performance when compared to other top-ranked wavelet image coding algorithms and the JPEG standard, but also inherently supports scalable features and progressive transmission. Computer experiments demonstrate the efficiency of the developed system.

We present a new approach to the modeling and decomposition of Gaussian mixtures by using robust statistical methods. The mixture distribution is viewed as a contaminated Gaussian density. Using this model and the model-fitting (MF) estimator, we propose a recursive algorithm called the Gaussian mixture density decomposition (GMDD) algorithm for successively identifying each Gaussian component in the mixture. The proposed decomposition scheme has advantages that are desirable but lacking in most existing techniques. In the GMDD algorithm the number of components does not need to be specified a priori, the proportion of noisy data in the mixture can be large, the parameter estimation of each component is virtually initial independent, and the variability in the shape and size of the component densities in the mixture is taken into account. Gaussian mixture density modeling and decomposition has been widely applied in a variety of disciplines that require signal or waveform characterization for classification and recognition. We apply the proposed GMDD algorithm to the identification and extraction of clusters, and the estimation of unknown probability densities. Probability density estimation by identifying a decomposition using the GMDD algorithm, that is, a superposition of normal distributions, is successfully applied to automated cell classification. Computer experiments using both real data and simulated data demonstrate the validity and power of the GMDD algorithm for various models and different noise assumptions.

Time-varying multispectral observations of clouds from meteorological satellites are used to estimate cloud-top heights (structure) and cloud winds (semi-fluid motion). Stereo image pairs over several time steps were acquired by two geostationary satellites with synchronized scanning instruments. Cloud-top height estimation from these image pairs is performed using an improved automatic stereo analysis algorithm on a massively parallel Maspar computer with 16 K processors. A new category of motion behavior known as semi-fluid motion is described for modeling cloud motions and an automatic algorithm for extracting semi-fluid motion is developed to track cloud winds. The time sequential dense estimates of cloud-top height depth maps in conjunction with intensity data are used to estimate local semi-fluid motion parameters for cloud tracking. Both stereo disparities and motion correspondences are estimated to sub-pixel accuracy. The Interactive Image SpreadSheet (IISS) is a new versatile visualization tool that was enhanced to analyze and visualize the results of the stereo analysis and semi-fluid motion estimation algorithms. Experimental results using time-varying data of the visible channel from two satellites in geosynchronous orbit is presented for the Hurricane Frederic.>

Optic flow motion analysis represents an important family of visual information processing techniques in computer vision. Segmenting an optic flow field into coherent motion groups and estimating each underlying motion is a very challenging task when the optic flow field is projected from a scene of several independently moving objects. The problem is further complicated if the optic flow data are noisy and partially incorrect. In this paper, the authors present a novel framework for determining such optic flow fields by combining the conventional robust estimation with a modified genetic algorithm. The baseline model used in the development is a linear optic flow motion algorithm due to its computational simplicity. The statistical properties of the generalized linear regression (GLR) model are thoroughly explored and the sensitivity of the motion estimates toward data noise is quantitatively established. Conventional robust estimators are then incorporated into the linear regression model to suppress a small percentage of gross data errors or outliers. However, segmenting an optic flow field consisting of a large portion of incorrect data or multiple motion groups requires a very high robustness that is unattainable by the conventional robust estimators. To solve this problem, the authors propose a genetic partitioning algorithm that elegantly combines the robust estimation with the genetic algorithm by a bridging genetic operator called self-adaptation.