Depth estimation is a cornerstone of 3D reconstruction and plays a vital role in minimally invasive endoscopic surgeries. However, most current depth estimation networks rely on traditional convolutional neural networks, which are limited in their ability to capture global information. Foundation models offer a promising approach to enhance depth estimation, but those models currently available are primarily trained on natural images, leading to suboptimal performance when applied to endoscopic images. In this work, we introduce a novel fine-tuning strategy for the Depth Anything Model and integrate it with an intrinsic-based unsupervised monocular depth estimation framework. Our approach includes a low-rank adaptation technique based on random vectors, which improves the model’s adaptability to different scales. Additionally, we propose a residual block built on depthwise separable convolution to compensate for the transformer’s limited ability to capture local features. Our experimental results on the SCARED dataset and Hamlyn dataset show that our method achieves state-of-the-art performance while minimizing the number of trainable parameters. Applying this method in minimally invasive endoscopic surgery can enhance surgeons’ spatial awareness, thereby improving the precision and safety of the procedures.

Unsupervised monocular depth estimation plays a vital role for endoscopy-based minimally invasive surgery (MIS). However, it remains challenging due to the distinctive imaging characteristics of endoscopy which disrupt the assumption of photometric consistency, a foundation relied upon by conventional methods. Distinct from recent approaches taking image pre-processing strategy, this paper introduces a pioneering solution through intrinsic image decomposition (IID) theory. Specifically, we propose a novel end-to-end intrinsic-based unsupervised monocular depth learning framework that is comprised of an image intrinsic decomposition module and a synthesis reconstruction module. This framework seamlessly integrates IID with unsupervised monocular depth estimation, and dedicated losses are meticulously designed to offer robust supervision for network training based on this novel integration. Noteworthy, we rely on the favorable property of the resulting albedo map of IID to circumvent the challenging images characteristics instead of pre-processing the input frames. The proposed method is extensively validated on SCARED and Hamlyn datasets, and better results are obtained than state-of-the-art techniques. Beside, its generalization ability and the effectiveness of the proposed components are also validated. This innovative method has the potential to elevate the quality of 3D reconstruction in monocular endoscopy, thereby enhancing the accuracy and robustness of augmented reality navigation technology in MIS.

Remote sensing image change captioning (RSICC) aims to articulate the changes in objects of interest within bitemporal remote sensing images using natural language. Given the limitations of current RSICC methods in expressing general features across multitemporal and spatial scenarios, and their deficiency in providing granular, robust, and precise change descriptions, we introduce a novel change captioning (CC) method based on the foundational knowledge and semantic guidance, which we term Semantic-CC. Semantic-CC alleviates the dependency of high-generalization algorithms on extensive annotations by harnessing the latent knowledge of foundation models, and it generates more comprehensive and accurate change descriptions guided by pixel-level semantics from change detection (CD). Specifically, we propose a bitemporal SAM-based encoder for dual-image feature extraction; a multitask semantic aggregation neck for facilitating information interaction between heterogeneous tasks; a straightforward multiscale CD decoder to provide pixel-level semantic guidance; and a change caption decoder based on the large language model (LLM) to generate change description sentences. Moreover, to ensure the stability of the joint training of CD and CC, we propose a three-stage training strategy that supervises different tasks at various stages. We validate the proposed method on the LEVIR-CC and LEVIR-CD datasets. The experimental results corroborate the complementarity of CD and CC, demonstrating that Semantic-CC can generate more accurate change descriptions and achieve optimal performance across both tasks.

Video semantic segmentation has achieved great success, which is significant for road scene understanding. However, semantic segmentation remains challenging in poor illumination and inclement weather. Thermal camera, highly invariant to light and highly penetrating to rain and fog, enables semantic segmentation to work under challenging conditions. Thus, this paper explores semantic segmentation in thermal videos to broaden the scope of the application of road scene understanding. We offer the first thermal video semantic segmentation dataset TVSS including 1695 thermal videos with 50850 frames in road scenes. It is available at:https://xzbai.buaa.edu.cn/datasets.html. TVSS is finely annotated by 17 categories at the frame rate of 1fps, with a labeled pixel density of 98.9%. Existing video semantic segmentation methods rely on the amount of labels and the representation power of backbones, which cannot achieve ideal results on thermal videos. Thus, we introduce a multi-granularity contrastive learning based thermal video semantic segmentation model (MGCL), which explores the abundant unlabeled frames to boost the supervised segmentation. Specifically, MGCL constructs multi-granularity self-supervised signals on unlabeled thermal videos by contrastive learning, including the intra-frame context generalization loss, the intra-clip temporal consistency loss, and the inter-video category discrimination loss. In addition, a hard anchor sampling strategy is introduced to focus on hard-classify pixels for further performance improvement. Extensive experiments on TVSS demonstrate the superior performance of MGCL in both accuracy and efficiency. Compared to the 12 state-of-the-art semantic segmentation methods, MGCL achieves 2.8% to 8.1% gains in mIoU performance while maintaining the inference speed.

Stroke is an acute cerebral vascular disease that is likely to cause long-term disabilities and death. Immediate emergency care with accurate diagnosis of computed tomographic (CT) images is crucial for dealing with a hemorrhagic stroke. However, due to the high variability of a stroke's location, contrast, and shape, it is challenging and time-consuming even for experienced radiologists to locate them. In this paper, we propose a U-net based deep learning framework to automatically detect and segment hemorrhage strokes in CT brain images. The input of the network is built by concatenating the flipped image with the original CT slice which introduces symmetry constraints of the brain images into the proposed model. This enhances the contrast between hemorrhagic area and normal brain tissue. Various Deep Learning topologies are compared by varying the layers, batch normalization, dilation rates, and pre-train models. This could increase the respective filed and preserves more information on lesion characteristics. Besides, the adversarial training is also adopted in the proposed network to improve the accuracy of the segmentation. The proposed model is trained and evaluated on two different datasets, which achieve the competitive performance with human experts with the highest location accuracy 0.9859 for detection, 0.8033 Dice score, and 0.6919 IoU for segmentation. The results demonstrate the effectiveness, robustness, and advantages of the proposed deep learning model in automatically hemorrhage lesion diagnosis, which make it possible to be a clinical decision support tool in stroke diagnosis.

This Innovative Practice Work-In-Progress presents an integrated signal processing experiment, which can cover most knowledge points of digital signal processing (DSP) course. Since the DSP course focuses on one dimension signal processing, voice signal has a great advantage. We provide an integrated voice signal processing experiment named as Phone Keypad Voice Recognition (PKVR), including the following parts: phone keypad voice collection, Discrete Fourier Transform (DFT) and analysis, filter design, digital query table establishment, number recognition of any keypad voice. Through the improvement of the practice training, the classroom teaching theory can be better understood in an interesting way for our students.

This Research to Practice WIP presents a UAVs visual navigation based comprehensive experiment to enhance multidisciplinary engineering ability in Aerospace engineering education. In traditional courses, aerospace-related disciplines are independently distributed in different courses, and there is rarely a hands-on platform which includes signal processing, control theory, and artificial intelligence into Aerospace engineering. Facing this problem, this paper designs a multidisciplinary comprehensive experiment, aiming to provide a hand-on platform and flexible project-based program to students of aerospace engineering professions. First of all, in order to let the students understand actual aerospace problems, a multidisciplinary simulation platform containing UAVs and remote objects scenarios is constructed for them to explore in the experiments. Second, the content of the experiment is designed into three stages including data acquisition and processing, conceptual design and simulation, in-flight validation, during which the multidisciplinary engineering ability runs through the whole process of the activities. Finally, Project Oriented Design Based Learning is also introduced here to combine engineering design education with innovation and creativity. Through the project demonstration and presentation at the end of the experiment, the multidisciplinary engineering ability of each student can be effectively evaluated. The UVN comprehensive experiment enables students to work on real-world aerospace engineering problems through a hardware-software integration framework, which may greatly stimulate their curiosity and interest in autonomously learning. It also provides students unprecedented opportunities to immerse themselves in projects that cross disciplinary boundaries, improve their professional ability and enhance their exploration competence in aerospace areas.

Fast and accurate ellipse detection is critical in certain computer vision tasks. In this paper, we propose an arc adjacency matrix-based ellipse detection (AAMED) method to fulfill this requirement. At first, after segmenting the edges into elliptic arcs, the digraph-based arc adjacency matrix (AAM) is constructed to describe their triple sequential adjacency states. Curvature and region constraints are employed to make the AAM sparse. Secondly, through bidirectionally searching the AAM, we can get all arc combinations which are probably true ellipse candidates. The cumulative-factor (CF) based cumulative matrices (CM) are worked out simultaneously. CF is irrelative to the image context and can be pre-calculated. CM is related to the arcs or arc combinations and can be calculated by the addition or subtraction of CF. Then the ellipses are efficiently fitted from these candidates through twice eigendecomposition of CM using Jacobi method. Finally, a comprehensive validation score is proposed to eliminate false ellipses effectively. The score is mainly influenced by the constraints about adaptive shape, tangent similarity, distribution compensation. Experiments show that our method outperforms the 12 state-of-the-art methods on 9 datasets as a whole, with reference to recall, precision, F-measure, and time-consumption.

Organ localization is an essential preprocessing step for many medical image analysis tasks such as image registration, organ segmentation and lesion detection. In this work, we propose an efficient method for multiple organ localization in CT image using 3D region proposal network. Compared with other convolutional neural network based methods that successively detect the target organs in all slices to assemble the final 3D bounding box, our method is fully implemented in 3D manner, thus can take full advantages of the spatial context information in CT image to perform efficient organ localization with only one prediction. We also propose a novel backbone network architecture that generates high-resolution feature maps to further improve the localization performance on small organs. We evaluate our method on two clinical datasets, where 11 body organs and 12 head organs (or anatomical structures) are included. As our results shown, the proposed method achieves higher detection precision and localization accuracy than the current state-of-theart methods with approximate 4 to 18 times faster processing speed. Additionally, we have established a public dataset dedicated for organ localization on http://dx. doi.org/10.21227/df8g-pq27. The full implementation of the proposed method have also been made publicly available on https://github.com/superxuang/caffe_3d_faster_rcnn.

Wavelet‐based segmentation approaches are widely used for texture segmentation purposes because of their ability to characterise different textures. In this study, the authors assess the influence of the chosen wavelet and propose to use the recently introduced empirical wavelets. We show that the adaptability of the empirical wavelet permits to reach better results than classic wavelets. To focus only on the textural information, they also propose to perform a cartoon + texture decomposition step before applying the segmentation algorithm. The proposed method is tested on six classic benchmarks, based on several popular texture images.

Pedestrian segmentation in infrared images is a challenging problem due to low SNR and inhomogeneous luminance distribution. In this paper, we first introduce background prior and object-center prior into infrared pedestrian segmentation, and propose a robust and efficient saliency-based scheme, which aims to obtain the accurate pedestrian object. First, background likelihood is developed to abstract the object representation based on the Gaussian mixture model soft decomposition. Second, by combining the shape information and the infrared character of the pedestrians, kernel density estimation-based foreground estimation is proposed to obtain the saliency iteratively with better adaptable for the fuzzy contour of the infrared object. Third, pedestrian boundary weight is employed to integrate the above two saliency maps for more intact and accurate results. Finally, pedestrians can be easily segmented from the infrared images, through any existing segmentation methods, as simple as the Otsu threshold method, on the obtained final saliency map. Extensive experiments on real infrared images captured by intelligent transportation systems demonstrate that our saliency algorithm consistently outperforms the state-of-the-art saliency detection methods, in terms of higher precision, F-measure, and lower mean absolute error. The effectiveness of our proposed segmentation algorithm is also evaluated by comparisons with the existing infrared segmentation methods and yields more precise and intact pedestrian regions.

Saliency detection is popular in image processing, but it is still a challenging problem for infrared pedestrian images. In this paper, an effective saliency detection method for infrared pedestrian images is proposed. Taking into consideration the characteristics of pedestrians including luminance and shape, the MSER-based local stableness (MLS) is firstly introduced. Then vertical edge-weighted contrast (VEC) is calculated. Finally, an intra-scale and inter-scale neighborhood based saliency propagation method is constructed to optimize and integrate the two features. Extensive experiments demonstrate the effectiveness of the proposed saliency method for infrared pedestrian images.

In this paper, a feasible simulator is proposed to predict the electromagnetic (EM) scattering from three-dimensional (3-D) electrically very large ship-sea models which are formulated accurately with surface integral equations. The EM coupling between the object and the rough surface is considered by the surface integral equation with the Green's function. In theory, standard the Method of Moments (MoM) can be used to solve the unknowns both on the object and the rough surface. However, the discretization of the rough surface significantly increases the computational resource requirements compared to calculating the scattering from the object alone. With an efficient parallelization of Multilevel Fast Multipole Algorithm (MLFMA) on computing platforms using distributed-memory architectures, the composite scattering characteristic of 3-D electrically very large ship-sea model is investigated, and a series of useful conclusions are also obtained.

Pedestrian segmentation in infrared images is a difficult problem for the defects of low SNR and inhomogeneous luminance distribution. In this paper, we propose a method which aims to obtain the accurate pedestrian segmentation through a background prior and boundary weight-based saliency. Background likelihood is firstly calculated as background prior to get an abstract representation for infrared pedestrian. Then, by considering the object-center prior, the object-biased Gaussian model is applied to derive the probability density estimation for pedestrians. Finally, the above two results are integrated with the boundary weight to obtain the final saliency map for infrared image, based on which pedestrians can be easily segmented. Experimental results on real infrared images captured by intelligent transportation systems demonstrate the effectiveness of the proposed approach against the state-of-the-art algorithms.

A novel infrared pedestrian segmentation algorithm based on the circular shortest path is proposed. The foreground containing pedestrians is estimated by saliency mapping and gray thresholding. In the foreground area, the human shape feature is introduced by a regional polar transformation. By adding the human shape coefficient to the object term of the cost function, the extracted contour can fit the human shape well while excluding most false alarms. The proposed algorithm performs better in areas with low contrast, and obtains good segmentation quantitatively and qualitatively.

Parameter estimation in Total Variation (TV) deblurring has been extensively studied in the literature during the last decade. However, few works have been done for parameter estimation in ℓp(0poutperforms TV and ℓ1in natural image deblurring. In this paper, by utilizing the Bayesian framework, we propose an adaptive fast iteratively reweighted least squares algorithm for ℓpregularized image deconvolution, which automatically estimates the unknown image and regularization parameter. Experiments show that the proposed method yields nearly optimal results and outperforms the state-of-the-art methods.

Saliency-based approaches has been well studied and successfully used in object detection for visible images. However, few researches have been done for saliency detection in infrared images, which are characterized with low resolution, SNR and contrast, fuzzy edge and lack of color features. In this paper, a contrast and distribution based saliency detection approach is proposed for infrared images. First, we develop an enhanced multi-scale saliency feature by improving the quality and contrast of the image in frequency domain. Second, luminance-distribution and gradient feature are explored to highlight the object with great gradient and compact distribution. Finally, by integrating the above two features, the final saliency map for infrared image were obtained. Experimental results on real infrared images demonstrate the effectiveness of the proposed approach against the state-of-the-art algorithms.

Blind image deconvolution involves two key objectives: 1) latent image and 2) blur estimation. For latent image estimation, we propose a fast deconvolution algorithm, which uses an image prior of nondimensional Gaussianity measure to enforce sparsity and an undetermined boundary condition methodology to reduce boundary artifacts. For blur estimation, a linear inverse problem with normalization and nonnegative constraints must be solved. However, the normalization constraint is ignored in many blind image deblurring methods, mainly because it makes the problem less tractable. In this paper, we show that the normalization constraint can be very naturally incorporated into the estimation process by using a Dirichlet distribution to approximate the posterior distribution of the blur. Making use of variational Dirichlet approximation, we provide a blur posterior approximation that considers the uncertainty of the estimate and removes noise in the estimated kernel. Experiments with synthetic and real data demonstrate that the proposed method is very competitive to the state-of-the-art blind image restoration methods.

Iteratively reweighted least squares (IRLS) is one of the most effective methods to minimize the lpregularized linear inverse problem. Unfortunately, the regularizer is nonsmooth and nonconvex when 0 <; p <; 1. In spite of its properties and mainly due to its high computation cost, IRLS is not widely used in image deconvolution and reconstruction. In this paper, we first derive the IRLS method from the perspective of majorization minimization and then propose an Alternating Direction Method of Multipliers (ADMM) to solve the reweighted linear equations. Interestingly, the resulting algorithm has a shrinkage operator that pushes each component to zero in a multiplicative fashion. Experimental results on both image deconvolution and reconstruction demonstrate that the proposed method outperforms state-of-the-art algorithms in terms of speed and recovery quality.

Blind image deconvolution (BID) is a severely ill-posed problem which requires prior information on the latent image to estimate the blur kernel. In this paper, a new observation that blurring always pushes the gradient of a local image region toward its mean value is introduced. And we formulate a novel function to measure the distance between the local gradient and its mean value. A novel regularizer associated with local gradient means is proposed. As it requires to segment the whole image into small regions, we propose an approximate method without any segmentation. Thanks to its simplicity the algorithm is fast and robust. Numerous experimental results on synthetic and real data demonstrate that our method is capable of removing various uniform blurs such as motion blur, atmospheric blur and out-of-focus blur.

Most existing methods for inpainting structure images fail in handling thick unknown regions, resulting in over-smoothing effects in the filled areas. In this paper, a novel inpainting algorithm, termed as isophote propagation, is presented to solve the problem. The algorithm propagates isophote information while maintaining the continuity of isophotes directions. It is simple and fast, and performs better in preserving sharp edges than existing algorithms. Effectiveness of the algorithm is demonstrated in experiments.