Infrared and visible image fusion (IVIF) aims to create fused images that encompass the comprehensive features of both input images, thereby facilitating downstream vision tasks. However, existing methods often overlook illumination conditions in low-light environments, resulting in fused images where targets lack prominence. To address these shortcomings, we introduce the Illumination-Aware Infrared and Visible Image Fusion Network, abbreviated by IAIFNet. Within our framework, an illumination enhancement network initially estimates the incident illumination maps of input images, based on which the textural details of input images under low-light conditions are enhanced specifically. Subsequently, an image fusion network adeptly merges the salient features of illumination-enhanced infrared and visible images to produce a fusion image of superior visual quality. Our network incorporates a Salient Target Aware Module (STAM) and an Adaptive Differential Fusion Module (ADFM) to respectively enhance gradient and contrast with sensitivity to brightness. Extensive experimental results validate the superiority of our method over seven state-of-the-art approaches for fusing infrared and visible images on the public LLVIP dataset. Additionally, the lightweight design of our framework enables highly efficient fusion of infrared and visible images. Finally, evaluation results on the downstream multi-object detection task demonstrate the significant performance boost our method provides for detecting objects in low-light environments.

Image rain removal is an essential and challenging low-level task in computer vision. In recent years, although great progress has been achieved in the field of image and video deraining, the restored details may be incomplete in many cases, making it difficult to recover the clear images from diverse rain forms. Moreover, the features of layers may not be fully exploited, which may reduce the rain removal performance to some degree. To address the above issues, we propose a novel Mask-based Enhancement and Feature Aggregation Network (MEFA-Net). First, we develop a novel structure with enhancement branch, which can greatly improve the feature representation capability. The enhancement branch is implemented based on the random masking technique, which can generate complementary features and improve the robustness. Specially, we design a novel adaptive weighting block in the MEFA-Net, which can adaptively assign weights to the enhanced features for each specific image, effectively improving the generalization ability of MEFA-Net. Second, we propose a feature aggregation sub-module (FAS) for comprehensive representation by integrating the features from different layers. Experimental results demonstrate that the proposed MEFA-Net can achieve better performance than most state-of-the-art image deraining methods on four synthetic and one real-world datasets.

Region-based methods have become the state-of-art solution for monocular 6-DOF object pose tracking in recent years. However, two main challenges still remain: the robustness to heterogeneous configurations (both foreground and background), and the robustness to partial occlusions. In this paper, we propose a novel region-based monocular 3D object pose tracking method to tackle these problems. Firstly, we design a new strategy to define local regions, which is simple yet efficient in constructing discriminative local color histograms. Contrary to previous methods which define multiple circular regions around the object contour, we propose to define multiple overlapped, fan-shaped regions according to polar coordinates. This local region partitioning strategy produces much less number of local regions that need to be maintained and updated, while still being temporally consistent. Secondly, we propose to detect occluded pixels using edge distance and color cues. The proposed occlusion detection strategy is seamlessly integrated into the region-based pose optimization pipeline via a pixel-wise weight function, which significantly alleviates the interferences caused by partial occlusions. We demonstrate the effectiveness of the proposed two new strategies with a careful ablation study. Furthermore, we compare the performance of our method with the most recent state-of-art region-based methods in a recently released large dataset, in which the proposed method achieves competitive results with a higher average tracking success rate. Evaluations on two real-world datasets also show that our method is capable of handling realistic tracking scenarios.

Surgical resection is the main clinical method for the treatment of bone tumors. A critical procedure for bone tumor resection is to plan a set of cut planes that enable resecting the bone tumor with a safe margin while preserving the maximum amount of healthy bone. Currently, the surgeons rely on manual methods to plan the cut planes, which highly depend on the surgeons' experiences and have been demonstrated to be error-prone, and in turn, increase the recurrence rate or resect much healthy bone. This study targets on improving the precision of cut plane planning for the image guided pelvis tumor resection surgeries. A semi-automatic approach to cut plane planning was proposed via a coarse-to-fine strategy. It can efficiently identify a dangerous region in the 3D space, which contains the bone tumor and its surrounding normal tissue with a safe margin. By projecting the dangerous region into an appropriate 2D space, a segmented boundary-constrained linear regression method was leveraged to plan a set of 3D cut planes that ensure the minimum area of the resected specimen in the 2D space while having the dangerous region cleared. Further, a coarse-to-fine 3D cut plane planning method was developed by incorporating a 3D cut plane refinement scheme with our 2D planning method. Extensive experiments, on the surgical data from nine previous pelvis tumor resection surgeries, demonstrated that our proposed approach substantially improved the localization precision of cut planes ( ) and decreased the amount of resected specimen ( ), as compared to the manual method.

Estimating scene depth from a single image can be widely applied to understand 3D environments due to the easy access of the images captured by consumer-level cameras. Previous works exploit conditional random fields (CRFs) to estimate image depth, where neighboring pixels (superpixels) with similar appearances are constrained to share the same depth. However, the depth may vary significantly in the slanted surface, thus leading to severe estimation errors. In order to eliminate those errors, we propose a superpixel-based normal guided scale invariant deep convolutional field by encouraging the neighboring superpixels with similar appearance to lie on the same 3D plane of the scene. In doing so, a depth-normal multitask CNN is introduced to produce the superpixel-wise depth and surface normal predictions simultaneously. To correct the errors of the roughly estimated superpiexl-wise depth, we develop a normal guided scale invariant CRF (NGSI-CRF). NGSI-CRF consists of a scale invariant unary potential that is able to measure the relative depth between superpixels as well as the absolute depth of superpixels, and a normal guided pairwise potential that constrains spatial relationships between superpixels in accordance with the 3D layout of the scene. In other words, the normal guided pairwise potential is designed to smooth the depth prediction without deteriorating the 3D structure of the depth prediction. The superpixel-wise depth maps estimated by NGSI-CRF will be fed into a pixel-wise refinement module to produce a smooth fine-grained depth prediction. Furthermore, we derive a closed-form solution for the maximum a posteriori (MAP) inference of NGSI-CRF. Thus, our proposed network can be efficiently trained in an end-to-end manner. We conduct our experiments on various datasets, such as NYU-D2, KITTI, and Make 3D. As demonstrated in the experimental results, our method achieves superior performance in both indoor and outdoor scenes.

In this paper, we investigate the estimation of the maximum target registration error (TRE) magnitude of the target location while using point-based rigid registration in the image guided system. Under the uniform restriction of fiducial localization error (FLE) magnitude, we explicitly formulate the estimation as an optimization problem. Through analyzing the approximated problem which assumes the rigidity of the fiducial set holds with the perturbation of FLE, we present a strict lower bound for the maximum TRE magnitude. The simulations show that the lower bound is close to the actual maximum TRE magnitude for the target locations lying far away from the fiducial points. Unlike the expected TRE magnitude in which all fiducial points contribute, the lower bound is only related to the fiducial points serving as the vertices of the convex hull of the fiducial set. Our analysis provides a new perspective of investigating the problem of TRE estimation and is helpful for the surgeons to learn about the worst situation during using the image guided system.

Segmentation of infrared (IR) ship images is always a challenging task, because of the intensity inhomogeneity and noise. The fuzzy C-means (FCM) clustering is a classical method widely used in image segmentation. However, it has some shortcomings, like not considering the spatial information or being sensitive to noise. In this paper, an improved FCM method based on the spatial information is proposed for IR ship target segmentation. The improvements include two parts: 1) adding the nonlocal spatial information based on the ship target and 2) using the spatial shape information of the contour of the ship target to refine the local spatial constraint by Markov random field. In addition, the results of K -means are used to initialize the improved FCM method. Experimental results show that the improved method is effective and performs better than the existing methods, including the existing FCM methods, for segmentation of the IR ship images.

The Fuzzy C-Means clustering (FCM) and the possibility FCM (PFCM) are popular methods in MRI brain image segmentation. However, using the Euclidean squared-norm distance as the similarity criterion makes FCM and PFCM only suitable for clustering the hyperspherically distributed data groups. The MRI brain image does not distribute hyperspherically, which means FCM and PFCM have intrinsic deficiency for the segmentation of MRI brain image. The center-free FCM could segment the non-linearly separable data. But, it does not consider the spatial information and is very sensitive to noise. In order to segment the non-linearly separable data groups with noise, a center-free PFCM is proposed in this paper. Firstly, we modify the center-free FCM to deal with the non-linearly separable data. Then, we combine the improved center-free FCM with PFCM to make the new method less sensitive to noise. Experimental results on artificial datasets and MRI brain images show that our method is effective and outperforms the conventional FCM methods in the segmentation of the MRI brain images with noise.

Segmentation of infrared (IR) ship images is always a challenging task, because of the intensity inhomogeneity and noise. The Fuzzy C-Means (FCM) clustering is a classical method widely used in IR ship image segmentation. However, it has some shortcomings, like not considering the spatial information or being sensitive to noise. In this paper, an improved FCM algorithm based on the spatial information is proposed. The improvements include two parts: (1) adding the non-local spatial information based on the ship target; (2) using the spatial shape information of the contour of the ship target to refine the local spatial constraint by Markov Random Field (MRF). A preprocessing procedure and a target selection method are also used to further improve the performance of the segmentation result. Experimental results show that our method is very effective and performs better than the conventional FCM methods in segmentation of the infrared ship images.