Myocardial perfusion imaging (MPI) is an essential tool for diagnosing and evaluating coronary artery disease (CAD). However, the diagnosis using MPI remains laborious as it involves multi-step readouts and meticulous image processing. These challenges impact current attempts at automating image interpretation of MPI. In this paper, we propose a multi-modality attention network (MMAN) that leverages information from clinical and MPI data for CAD diagnosis. Specifically, we propose an image-correlated cross-attention (ICCA) module that fuses information from both stress and rest MPI to enhance feature representation at the image level. Furthermore, we design a clinical data-guided attention (CDGA) module that integrates clinical data with image features to improve overall feature understanding for CAD evaluation. In addition, we employ self-learning for network pre-training, which further enhances the diagnostic performance using MPI on CAD. Experiments on a myocardial perfusion imaging dataset demonstrate that the proposed method is effective for CAD evaluation using myocardial perfusion imaging and clinical data.

Keypoints-based approaches have shown to be promising for retinal image registration, which superimpose two or more images from different views based on keypoint detection and description. However, existing approaches suffer from ineffective keypoint detector and descriptor training. Meanwhile, the non-linear mapping from 3D retinal structure to 2D images is often neglected. In this paper, we propose a novel learning-based junction detection approach for retinal image registration, which enhances both the keypoint detector and descriptor training. To improve the keypoint detection, it uses a multi-task vessel detection to regularize the model training, which helps to learn more representative features and reduce the risk of over-fitting. To achieve effective training for keypoints description, a new constrained negative sampling approach is proposed to compute the descriptor loss. Moreover, we also consider the non-linearity between retinal images from different views during matching. Experimental results on FIRE dataset show that our method achieves mean area under curve of 0.850, which is 12.6% higher than 0.755 by the state-of-the-art method. All the codes are available at https://github.com/samjcheng/SuperJunction.

In a clinical setting, the acquisition of certain medical image modality is often unavailable due to various considerations such as cost, radiation, etc. Therefore, unpaired cross-modality translation techniques, which involve training on the unpaired data and synthesizing the target modality with the guidance of the acquired source modality, are of great interest. Previous methods for synthesizing target medical images are to establish one-shot mapping through generative adversarial networks (GANs). As promising alternatives to GANs, diffusion models have recently received wide interests in generative tasks. In this paper, we propose a target-guided diffusion model (TGDM) for unpaired cross-modality medical image translation. For training, to encourage our diffusion model to learn more visual concepts, we adopted a perception prioritized weight scheme (P2W) to the training objectives. For sampling, a pre-trained classifier is adopted in the reverse process to relieve modality-specific remnants from source data. Experiments on both brain MRI-CT and prostate MRI-US datasets demonstrate that the proposed method achieves a visually realistic result that mimics a vivid anatomical section of the target organ. In addition, we have also conducted a subjective assessment based on the synthesized samples to further validate the clinical value of TGDM.

Segmentation of kidney on CT images is critical to computer-assisted surgical planning for kidney interventional therapy. Segmenting kidney manually is impractical in clinical, automatic segmentation is desirable. U-Net has been successful in medical image segmentation and is a promising candidate for the task. However, semantic gap still exists, especially when multiple phase images or multiple center images are involved. In this paper, we proposed an ULBNet to reduce the semantic gap and to improve segmentation performance. The proposed architecture includes new skip connections of local binary convolution (LBC). We also proposed a novel strategy of fast retraining a model for a new task without manually labelling required. We evaluated the network for kidney segmentation on multiple phase CT images. ULBNet resulted in an overall accuracy of 98.0% with comparison to Resunet 97.5%. Specifically, on the plain phase CT images, 98.1% resulted from ULBNet and 97.6% from Resunet; on the corticomedullay phase images, 97.8% from ULBNet and 97.2% from Resunet; on the nephrographic phase images, 97.6% from ULBNet and 97.4% from Resunet; on the excretory phase images, 98.1% from ULBNet and 97.4% from Resunet. The proposed network architecture performs better than Resunet on generalizing to multiple phase images.

This paper presents a new medical image segmentation method by using wavelet frame energy distribution, which is the sum of squares of the wavelet frame coefficients at each pixel. This work shows that the wavelet frame energy distribution contains the fine texture information extracted from images with low intensity contrast and complex structures using wavelet frame transform. Thus it is employed to enhance the segmentation quality under some challenge conditions such as low intensity contrast, weak/ambiguous boundaries, intensity inhomogeneity and heavy noise. Furthermore, this paper adopts convex relaxation approach to solve the corresponding optimization problem instead of classical level-set method, so the leading numerical computation is efficient and robust to initialization values. Experimental results also illustrate the efficiency of the proposed segmentation method for biomedical images under these extreme imaging conditions.

Brain extraction is an important preprocessing step for further analysis of brain MR images. Significant intensity inhomogeneity can be observed in rodent brain images due to the high-field MRI technique. Unlike most existing brain extraction methods that require bias corrected MRI, we present a high-order and L0regularized variational model for bias correction and brain extraction. The model is composed of a data fitting term, a piecewise constant regularization and a smooth regularization, which is constructed on a 3-D formulation for medical images with anisotropic voxel sizes. We propose an efficient multi-resolution algorithm for fast computation. At each resolution layer, we solve an alternating direction scheme, all subproblems of which have the closed-form solutions. The method is tested on three T2 weighted acquisition configurations comprising a total of 50 rodent brain volumes, which are with the acquisition field strengths of 4.7 Tesla, 9.4 Tesla and 17.6 Tesla, respectively. On one hand, we compare the results of bias correction with N3 and N4 in terms of the coefficient of variations on 20 different tissues of rodent brain. On the other hand, the results of brain extraction are compared against manually segmented gold standards, BET, BSE and 3-D PCNN based on a number of metrics. With the high accuracy and efficiency, our proposed method can facilitate automatic processing of large-scale brain studies.

An interactive surgical simulation system needs to meet three main requirements, speed, accuracy, and stability. In this paper, we present a stable and accurate method for animating mass-spring systems in real time. An integration scheme derived from explicit integration is used to obtain interactive realistic animation for a multiobject environment. We explore a predictor-corrector approach by correcting the estimation of the explicit integration in a poststep process. We introduce novel constraints on positions into the mass-spring model (MSM) to model the nonlinearity and preserve volume for the realistic simulation of the incompressibility. We verify the proposed MSM by comparing its deformations with the reference deformations of the nonlinear finite-element method. Moreover, experiments on porcine organs are designed for the evaluation of the multiobject deformation. Using a pair of freshly harvested porcine liver and gallbladder, the real organ deformations are acquired by computed tomography and used as the reference ground truth. Compared to the porcine model, our model achieves a 1.502 mm mean absolute error measured at landmark locations for cases with small deformation (the largest deformation is 49.109 mm) and a 3.639 mm mean absolute error for cases with large deformation (the largest deformation is 83.137 mm). The changes of volume for the two deformations are limited to 0.030% and 0.057%, respectively. Finally, an implementation in a virtual reality environment for laparoscopic cholecystectomy demonstrates that our model is capable to simulate large deformation and preserve volume in real-time calculations.

We propose a new variant of the Mumford-Shah model for simultaneous bias correction and segmentation of images with intensity inhomogeneity. First, based on the model of images with intensity inhomogeneity, we introduce an L0 gradient regularizer to model the true intensity and a smooth regularizer to model the bias field. In addition, we derive a new data fidelity using the local intensity properties to allow the bias field to be influenced by its neighborhood. Second, we use a two-stage segmentation method, where the fast alternating direction method is implemented in the first stage for the recovery of true intensity and bias field and a simple thresholding is used in the second stage for segmentation. Different from most of the existing methods for simultaneous bias correction and segmentation, we estimate the bias field and true intensity without fixing either the number of the regions or their values in advance. Our method has been validated on medical images of various modalities with intensity inhomogeneity. Compared with the state-of-art approaches and the well-known brain software tools, our model is fast, accurate, and robust with initializations.

This paper presents a novel discrete Mumford-Shah model for the simultaneous bias correction and image segmentation(SBCIS) for images with intensity inhomogeneity. The model is based on the assumption that an image can be approximated by a product of true intensities and a bias field. Unlike the existing methods, where the true intensities are represented as a linear combination of characteristic functions of segmentation regions, we employ L0gradient minimization to enforce a piecewise constant solution. We introduce a new neighbor term into the Mumford-Shah model to allow the true intensity of a pixel to be influenced by its immediate neighborhood. A two-stage segmentation method is applied to the proposed Mumford-Shah model. In the first stage, both the true intensities and bias field are obtained while the segmentation is done using the K-means clustering method in the second stage. Comparisons with the two-stage Mumford-Shah model show the advantages of our method in its ability in segmenting images with intensity inhomogeneity.

Image segmentation is to separate the image domain into sub-domains according to some optimization rule. Variational level-set image segmentation based on the Mumford-Shah model, together with level set formulation, leads to complicate implementation and expensive computational time. To overcome the difficulty in solving the complex coupled level set functions and handle any number of phases in the image, we improve the region-base active contour model in this paper. We formulate the image segmentation as a new multiphase minimization problem to handle images containing any integer phases of intensities. We also propose efficient numerical algorithms to solve such optimization problem without extra steps of convex relaxation. Numerical validation experiments on various images show better accuracy and robustness over existing methods.

As one of the most important image segmentation models, the Mumford-Shah functional was developed to pursue a piecewise smooth approximation of a given image based on the regularization on the total length of curves. In this paper, we modify the Mumford-Shah model using Euler's elastic a as the regularization. A two-stage segmentation method is applied the Euler's elastic a regularized Mumford-Shah model. The first stage is to find a smooth solution of the variant Mumford-Shah functional based on augmented Lagrangian method while a thresholding is performed in the second stage to obtain different phases for the segmentation. The K-means clustering method is used as the technique to find the thresholds for the segmentation. For intensity inhomogeneous images, we eliminate the effect of the bias field by bias-corrected fuzzy c-means method. Experimental results show that as the regularization, Euler's elastic a makes the Mumford-Shah model perform better for many kinds of images, including tubular and irregular shaped, CT Angiography (CTA) and MRI images in different noise level.

Image segmentation is to separate the image domain into local regions. Implicit active contour models via the level-set method can provide smooth and closed contours. The major category is based on the Mumford-Shah image component decomposition model. Approximating the image domain by a set of homogeneous regions, segmentation based on the piecewise constant (PC) models fail in intensity inhomogeneous images. Requiring intensity smoothness in local regions, segmentation based on the piecewise smooth (PS) models can perform better for intensity inhomogeneity than those PC based. Existing PS-based techniques are inefficient and not robust to complicate intensity scenarios with noise. Here we introduce a new functional model by decomposing an image into three parts: PS components, PC components and noise components. We convexify the decomposition model by incorporating with relaxation techniques and optimize the PS components over the whole image domain. New numerical algorithms are also proposed to implement the above approaches efficiently. Numerical validation experiments show that the proposed approaches can achieve much faster, more robust and more accurate image segmentation than existing arts.

Sleeping posture reveals important information for eldercare and patient care, especially for bed ridden patients. Traditionally, some works address the problem from either pressure sensor or video image. This paper presents a multimodal approach to sleeping posture classification. Features from pressure sensor map and video image have been proposed in order to characterize the posture patterns. The spatiotemporal registration of the two modalities has been considered in the design, and the joint feature extraction and data fusion is presented. Using multi-class SVM, experiment results demonstrate that the multimodal approach achieves better performance than the approaches using single modal sensing.

Due to the physical and mental impairments, frail elderly have to stay in the bed most of their time. So, serious health issues and physical injuries can occur to them if dangerous events and situations around the bed are not known in advance. This paper proposes continuous patient monitoring with situation awareness using multimodality sensors to recognize such dangerous events in and around the bed. With multimodal sensing intelligence, various contexts and situations of those bedridden elderly can be determined. By incorporating personalization and medical domain knowledge into detected events and activities, dangerous events can be recognized immediately. Then, timely intervention can be provided to the elderly with potential fall risks. So our solution supports situation aware patient monitoring and provides desirable personalized care at the institutions enabling to prevent potential health and well-being problems in and around the bed.

Efficiency and robustness are the two most important issues for multiobject tracking algorithms in real-time intelligent video surveillance systems. We propose a novel 2.5-D approach to real-time multiobject tracking in crowds, which is formulated as a maximum a posteriori estimation problem and is approximated through an assignment step and a location step. Observing that the occluding object is usually less affected by the occluded objects, sequential solutions for the assignment and the location are derived. A novel dominant color histogram (DCH) is proposed as an efficient object model. The DCH can be regarded as a generalized color histogram, where dominant colors are selected based on a given distance measure. Comparing with conventional color histograms, the DCH only requires a few color components (31 on average). Furthermore, our theoretical analysis and evaluation on real data have shown that DCHs are robust to illumination changes. Using the DCH, efficient implementations of sequential solutions for the assignment and location steps are proposed. The assignment step includes the estimation of the depth order for the objects in a dispersing group, one-by-one assignment, and feature exclusion from the group representation. The location step includes the depth-order estimation for the objects in a new group, the two-phase mean-shift location, and the exclusion of tracked objects from the new position in the group. Multiobject tracking results and evaluation from public data sets are presented. Experiments on image sequences captured from crowded public environments have shown good tracking results, where about 90% of the objects have been successfully tracked with the correct identification numbers by the proposed method. Our results and evaluation have indicated that the method is efficient and robust for tracking multiple objects (>or= 3) in complex occlusion for real-world surveillance scenarios.

Tracking objects using multiple sensors is more efficient than those using one sensor. In this paper, we proposed a method to fuse data from multiple sensors in Gaussian mixture probability hypothesis density filter. This method can avoid the data association problem in multi-sensor multi-object tracking. Moreover, it is more reliable and less computational than particle probability hypothesis density filter for multi-sensor multi-object tracking. We demonstrated the efficient of the approach by applications such as bearing and range tracking, and multiple speaker tracking.

Most methods for multiple object tracking in video represent the state of multi-object in a high dimensional joint state space. This leads to high computational complexity. This paper presents a method using the probability hypothesis density (PHD) filter to estimate the state of multiple objects in video. The method operates on the single object state space instead of the joint state space. A PHD recursion for visual observations with color measurements is proposed. Our method can track varying number of objects.

In this paper, we present an efficient method for tracking multiple speakers in a reverberant environment. The proposed method is based on the cardinalized probability hypothesis density (CPHD) filter. Because the CPHD filter can handle a large amount of clutter measurements, our method has a high reliability when tracking multiple speakers. Simulation experiments are presented to demonstrate the performance of the proposed method.

For applications such as behavior recognition it is important to maintain the identity of multiple targets, while tracking them in the presence of splits and merges, or occlusion of the targets by background obstacles. Here we propose an algorithm to handle multiple splits and merges of objects based on dynamic programming and a new geometric shape matching measure. We then cooperatively combine Kalman filter-based motion and shape tracking with the efficient and novel geometric shape matching algorithm. The system is fully automatic and requires no manual input of any kind for initialization of tracking. The target track initialization problem is formulated as computation of shortest paths in a directed and attributed graph using Dijkstra's shortest path algorithm. This scheme correctly initializes multiple target tracks for tracking even in the presence of clutter and segmentation errors which may occur in detecting a target. We present results on a large number of real world image sequences, where upto 17 objects have been tracked simultaneously in real-time, despite clutter, splits, and merges in measurements of objects. The complete tracking system including segmentation of moving objects works at 25 Hz on 352times288 pixel color image sequences on a 2.8-GHz Pentium-4 workstation

In this paper, we present a new scheme to robustly estimate the head pose from either video sequence or individual images. Developed from ISOMAP, we learn a person-independent and nonlinear embedding space (we call it a 2-D feature space) for different poses. A nonlinear interpolation is proposed to map new sequences or images into the 2-D feature space. Especially for video sequences, we propose an adaptive local fitting technique to filter unreasonable mappings. By exploring the intrinsic characteristics, we further estimate the head pose of that sequence or image. Experiments reported in this paper showed robust results.

In this paper, we present a new scheme to robustly detect a human attentive behavior, i.e., a frequent change in focus of attention (FCFA) from video sequences. The FCFA behavior can be easily perceived by people as temporal changes of human head pose. Here, we propose a non-linear head pose embedding and mapping algorithm to detect the pose in each frame of the sequence. Developed from ISOMAP, we learn a person-independent and non-linear embedding space (we call it a 2-D feature space) for different head poses. A non-linear interpolation mapping followed by an adaptive local fitting method is designed to map new frames into the 2-D feature space where head poses can be further obtained. An entropy classifier is then proposed on each sequence to detect the FCFA behavior. Experiments reported in this paper showed robust results.

Video-based surveillance systems have a wide range of applications for traffic monitoring, as they provide more information as compared to other sensors. In this paper, we present a rule-based framework for behavior and activity detection in traffic videos obtained from stationary video cameras. Moving targets are segmented from the images and tracked in real time. These are classified into different categories using a novel Bayesian network approach, which makes use of image features and image-sequence-based tracking results for robust classification. Tracking and classification results are used in a programmed context to analyze behavior. For behavior recognition, two types of interactions have mainly been considered. One is interaction between two or more mobile targets in the field of view (FoV) of the camera. The other is interaction between targets and stationary objects in the environment. The framework is based on two types of a priori information: 1) the contextual information of the camera's FoV, in terms of the different stationary objects in the scene and 2) sets of predefined behavior scenarios, which need to be analyzed in different contexts. The system can recognize behavior from videos and give a lexical output of the detected behavior. It also is capable of handling uncertainties that arise due to errors in visual signal processing. We demonstrate successful behavior recognition results for pedestrian-vehicle interaction and vehicle-checkpost interactions.

This letter describes a high-performance face recognition system by combining two recently proposed neural network models, namely Gabor wavelet network (GWN) and kernel associative memory (KAM), into a unified structure called Gabor wavelet associative memory (GWAM). GWAM has superior representation capability inherited from GWN and consequently demonstrates a much better recognition performance than KAM. Extensive experiments have been conducted to evaluate a GWAM-based recognition scheme using three popular face databases, i.e., FERET database, Olivetti-Oracle Research Lab (ORL) database and AR face database. The experimental results consistently show our scheme's superiority and demonstrate its very high-performance comparing favorably to some recent face recognition methods, achieving 99.3% and 100% accuracy, respectively, on the former two databases, exhibiting very robust performance on the last database against varying illumination conditions.

In this paper, we propose a real time system for tracking targets across blind regions of multiple cameras with non-overlapping fields of views (FOVs) using camera topology, and targets’ motion and shape information. Kalman filters are used to robustly track each target’s shape and motion in each camera view and the common ground plane view composed of all camera views. The target’s track in the blind region between cameras is obtained using Kalman filter predictions. For multi-camera correspondence matching we compute the Gaussian distributions of the tracking parameters across cameras for the target motion and position in the ground plane view. Matching of targets across camera views uses a graph based track initialization scheme, which accumulates information from occurrences of target in several consecutive frames of the video. Probabilistic matching is carried out by using the track parameters for new tracks obtained from the graph in a camera view with the parameters of the terminated tracks learnt by Kalman filters in the other camera views and ground plane view. We obtain 85% accuracy for corresponding matching while tracking vehicles observed from two cameras monitoring a highway.

This paper addresses the problem of background modeling for foreground object detection in complex environments. A Bayesian framework that incorporates spectral, spatial, and temporal features to characterize the background appearance is proposed. Under this framework, the background is represented by the most significant and frequent features, i.e., the principal features, at each pixel. A Bayes decision rule is derived for background and foreground classification based on the statistics of principal features. Principal feature representation for both the static and dynamic background pixels is investigated. A novel learning method is proposed to adapt to both gradual and sudden "once-off" background changes. The convergence of the learning process is analyzed and a formula to select a proper learning rate is derived. Under the proposed framework, a novel algorithm for detecting foreground objects from complex environments is then established. It consists of change detection, change classification, foreground segmentation, and background maintenance. Experiments were conducted on image sequences containing targets of interest in a variety of environments, e.g., offices, public buildings, subway stations, campuses, parking lots, airports, and sidewalks. Good results of foreground detection were obtained. Quantitative evaluation and comparison with the existing method show that the proposed method provides much improved results.

This paper proposes a novel method for detection and segmentation of foreground objects from a video which contains both stationary and moving background objects and undergoes both gradual and sudden "once-off" changes. A Bayes decision rule for classification of background and foreground from selected feature vectors is formulated. Under this rule, different types of background objects will be classified from foreground objects by choosing a proper feature vector. The stationary background object is described by the color feature, and the moving background object is represented by the color co-occurrence feature. Foreground objects are extracted by fusing the classification results from both stationary and moving pixels. Learning strategies for the gradual and sudden "once-off" background changes are proposed to adapt to various changes in background through the video. The convergence of the learning process is proved and a formula to select a proper learning rate is also derived. Experiments have shown promising results in extracting foreground objects from many complex backgrounds including wavering tree branches, flickering screens and water surfaces, moving escalators, opening and closing doors, switching lights and shadows of moving objects.

One of the important features for human machine interaction is its ability to recognize human faces. This paper presents a novel architecture suitable for real time robotic face recognition by learning a person's face incrementally, where the Gabor features at respective feature locations of a face are used to derive a similarity measurement. A face tracking followed by a clustering technique is used to learn a person's face appearance variance when the system interacts with the person. The recognition by learning proposed in this paper is similar to the partial memory incremental learning method, where we proposed a novel approach to the learning and updating process. Experiment shows significant improvement in the face recognition performance after learning over the time and with more interaction between a person and the system.

This paper proposes a novel method for tracking persons based on the principal colors of human objects. First, an efficient human object representation method, principal color representation (PCR), is proposed. Asymmetric similarity measures are then proposed based on the principal color representation. These asymmetric similarity measures could be used to evaluate the matching between two individuals as well as visual evident of an individual in a group. An efficient algorithm for tracking persons as individuals or in groups is then described. The method has been tested using image sequences containing multiple moving persons frequently gathering and separating. Our test results have shown that proposed method has successfully tracked both persons as individuals or in groups, and is robust to illumination changes.

This paper proposes a novel method for detecting foreground objects in nonstationary complex environments containing moving background objects. We derive a Bayes decision rule for classification of background and foreground changes based on inter-frame color co-occurrence statistics. An approach to store and fast retrieve color co-occurrence statistics is also established In the proposed method, foreground objects are detected in two steps. First, both foreground and background changes are extracted using background subtraction and temporal differencing. The frequent background changes are then recognized using the Bayes decision rule based on the learned color co-occurrence statistics. Both short-term and longterm strategies to learn the frequent background changes are proposed Experiments have shown promising results in detecting foreground objects from video containing wavering tree branches and flickering screens/water surface. The proposed method has shown better performance as compared with two existing methods.