Semantic segmentation techniques are widely applied in various image analysis tasks. However, compared with natural images, medical image segmentation presents greater challenges. For instance, lesions often vary significantly in morphology, size, and structure, and are frequently accompanied by low contrast and blurred boundaries. To simultaneously preserve fine tissue structures when handling large-scale lesions and ensure the coherence of the divergent structures of vessels, tumors, and other organs while accurately segmenting adjacent cells, this paper proposes the concept of “Global Capture and Local Carving”. It introduces a model that integrates a hierarchical information fusion strategy, named CarveNet. CarveNet incorporates a carving mechanism at three levels: downsampling, feature transmission, and bottleneck processing. Structural Carving Pooling Module underpins the downsampling carving, deeply optimizing the information structure and morphology at different levels to maximize detail retention and minimize downsampling loss. Multi-window Carving ViT is employed for transmission carving, enhancing global information modeling while refining local feature representation. The bottleneck carving integrates a long-distance recurrent communication mechanism with grid-like spatial random shuffling to strengthen the robustness and diversity of feature extraction. Experiments conducted on eight medical image datasets demonstrate that CarveNet consistently delivers outstanding performance across all tasks, surpassing the second-best method in Dice coefficient by 1.136 %. This fully validates its effectiveness in terms of multi-lesion adaptability, accuracy, and generalization capability. The code is available at https://github.com/YF-W/CarveNet .

Evaluating large vision-language models (LVLMs) is very expensive, due to high computational cost and the wide variety of tasks. The good news is that if we already have some observed performance scores, we may be able to infer unknown ones. In this study, we propose a new framework for predicting unknown performance scores based on observed ones from other LVLMs or tasks. We first formulate the performance prediction as a matrix completion task. Specifically, we construct a sparse performance matrix $\boldsymbol{R}$, where each entry $R_{mn}$ represents the performance score of the $m$-th model on the $n$-th dataset. By applying probabilistic matrix factorization (PMF) with Markov chain Monte Carlo (MCMC), we can complete the performance matrix, i.e., predict unknown scores. Additionally, we estimate the uncertainty of performance prediction based on MCMC. Practitioners can evaluate their models on untested tasks with higher uncertainty first, which quickly reduces the prediction errors. We further introduce several improvements to enhance PMF for scenarios with sparse observed performance scores. Our experiments demonstrate the accuracy of PMF in predicting unknown scores, the reliability of uncertainty estimates in ordering evaluations, and the effectiveness of our enhancements for handling sparse data. Our code is available at https://github.com/Qinyu-Allen-Zhao/CrossPred-LVLM.

Feature shaping refers to a family of methods that exhibit state-of-the-art performance for out-of-distribution (OOD) detection. These approaches manipulate the feature representation, typically from the penultimate layer of a pre-trained deep learning model, so as to better differentiate between in-distribution (ID) and OOD samples. However, existing feature-shaping methods usually employ rules manually designed for specific model architectures and OOD datasets, which consequently limit their generalization ability. To address this gap, we first formulate an abstract optimization framework for studying feature-shaping methods. We then propose a concrete reduction of the framework with a simple piecewise constant shaping function and show that existing feature-shaping methods approximate the optimal solution to the concrete optimization problem. Further, assuming that OOD data is inaccessible, we propose a formulation that yields a closed-form solution for the piecewise constant shaping function, utilizing solely the ID data. Through extensive experiments, we show that the feature-shaping function optimized by our method improves the generalization ability of OOD detection across a large variety of datasets and model architectures. Our code is available at https://github.com/Qinyu-Allen-Zhao/OptFSOOD.

Recently, spatio-temporal fusion technologies have been rapidly developed and widely applied, which generally require one or more pairs of coarse- and fine-resolution images as reference data and a coarse-resolution image at the prediction time to produce a fine-resolution image at the forecast time. Consequently, most spatio-temporal fusion methods are phase-based and obtain temporal changing information from the coarse-resolution image pairs. Consequently, they usually have rigid constraints on reference data selection using the temporal interval criterion. However, due to the relatively long revisit cycle and cloud contamination, it is difficult to prepare adequate high-quality reference data with little change, especially for large-scale fusion tasks. Therefore, we propose a spatial-based fusion method, which only requires the coarse images at the prediction time and a fine reference image selected by a spatial-spectral-similarity criterion, named super-resolution based spatial fusion with the generative adversarial network (SRSF-GAN). SRSF-GAN uses a super-resolution (SR) module merely on the coarse images at the prediction time, and then perform a multi-scale fusion with the reference fine image. Moreover, the spatial attention mechanism is adopted to achieve dynamic weight tuning, i.e. assigning more weight to the SR image for changed areas and more weight to the fine reference image for unchanged areas. Comparison experiments based on three datasets show that our model can outperform the state-of-the-art methods, and the changes with different spatial and spectral ranges of variation can be recovered. The code will be uploaded to the following website: https://github.com/Zhaosir996/SRSFGAN.