Abstract Federated Learning (FL) typically operates in a closed-set setting where all test classes are known during training, limiting its applicability in real-world scenarios where models must handle emerging unknown classes. This leads to misclassification of unseen categories as known ones. To address this limitation, we introduce Federated Open-Set Recognition (FedOSR), a novel paradigm enabling distributed clients to collaboratively train models that classify known classes while detecting and rejecting unknown ones. However, FedOSR presents unique challenges: the inter-set interference between learning closed-set and open-set knowledge within each client, and the intra-set inconsistency arising from data heterogeneity across clients. These challenges fundamentally complicate the federated aggregation process, as divergent optimization objectives and heterogeneous data distributions lead to parameter misalignment during model aggregation. In this work, we propose FedPD++ , a parameter disentanglement guided framework that systematically addresses both challenges through coordinated client-server mechanisms. On the client side, Local Parameter Disentanglement (LPD) decouples each OSR model into task-specific closed-set and open-set subnetworks to prevent inter-set interference. We introduce a Dynamic Path Integral (DPI) score that robustly identifies task-relevant parameters by leveraging path integral stability, coupled with an Adaptive Soft Masking (ASM) strategy that creates flexible subnetworks with adaptive thresholds rather than rigid binary partitions. On the server side, Global Divide-and-Conquer Aggregation (GDCA) tackles intra-set inconsistency by partitioning each subnetwork into shared and specific components, then aligning corresponding parts across clients using optimal transport to eliminate parameter misalignment. To ensure stable aggregation, we integrate Sequential Batch-Norm Alignment (SBA) that leverages temporal batch normalization statistics from multiple clients. Extensive experiments on open-set classification and segmentation tasks demonstrate that FedPD++ consistently achieves significant performance improvements over state-of-the-art methods. Code is available at: https://github.com/CUHK-AIM-Group/FedPD

Fiber tract segmentation is crucial for clinical applications such as brain function interpretation and surgical planning. Existing methods typically adopt either a cortical-parcellation-based or fiber clustering approach, but fail to simultaneously integrate heterogeneous information (e.g., streamline shape, point position, anatomical priors). In this work, we propose Fiber HGNN, a novel heterogeneous graph neural network that explicitly models and integrates heterogeneous information of fibers for accurate fiber tract segmentation. We construct a heterogeneous graph comprising three types of nodes: streamline, fiber keypoint and anatomical region. Specifically, fiber keypoints are representative points sampled along each streamline to characterize local geometric features, while anatomical regions provide contextual priors derived from brain atlas. This design enables the network to jointly capture the complementary information of streamline shape, local geometry, and anatomical priors, thus facilitating the learning of more discriminative feature representations. To further leverage implicit anatomical connectivity, we design a Metapath-guided Heterogeneous Information Aggregation (MHIA) network. By analyzing the spatial relationships between streamline keypoints and anatomical regions, the heterogeneous graph is decomposed into anatomical subgraphs for each streamline. In each subgraph, heterogeneous information from metapath-linked nodes is aggregated to obtain the final fiber representation. We evaluate the effectiveness of our framework on the HCP105 and TractoInferno datasets. The experimental results demonstrate that our method significantly outperforms previous state-of-the-art methods. The source code is available at https://github.com/CUHK-AIM-Group/Fiber-HGNN.

U-Net has become a cornerstone in various visual applications such as image segmentation and diffusion probability models. While numerous innovative designs and improvements have been introduced by incorporating transformers or MLPs, the networks are still limited to linearly modeling patterns as well as the deficient interpretability. To address these challenges, our intuition is inspired by the impressive results of the Kolmogorov-Arnold Networks (KANs) in terms of accuracy and interpretability, which reshape the neural network learning via the stack of non-linear learnable activation functions derived from the Kolmogorov-Anold representation theorem. Specifically, in this paper, we explore the untapped potential of KANs in improving backbones for vision tasks. We investigate, modify and re-design the established U-Net pipeline by integrating the dedicated KAN layers on the tokenized intermediate representation, termed U-KAN. Rigorous medical image segmentation benchmarks verify the superiority of UKAN by higher accuracy even with less computation cost. We further delved into the potential of U-KAN as an alternative U-Net noise predictor in diffusion models, demonstrating its applicability in generating task-oriented model architectures.

Recent advances in generalizable 3D Gaussian Splatting have demonstrated promising results in real-time high-fidelity rendering without per-scene optimization, yet existing approaches still struggle to handle unfamiliar visual content during inference on novel scenes due to limited generalizability. To address this challenge, we introduce MonoSplat, a novel framework that leverages rich visual priors from pre-trained monocular depth foundation models for robust Gaussian reconstruction. Our approach consists of two key components: a Mono-Multi Feature Adapter that transforms monocular features into multi-view representations, coupled with an Integrated Gaussian Prediction module that effectively fuses both feature types for precise Gaussian generation. Through the Adapter’s lightweight attention mechanism, features are seamlessly aligned and aggregated across views while preserving valuable monocular priors, enabling the Prediction module to generate Gaussian primitives with accurate geometry and appearance. Through extensive experiments on diverse real-world datasets, we convincingly demonstrate that MonoSplat achieves superior reconstruction quality and generalization capability compared to existing methods while maintaining computational efficiency with minimal trainable parameters. Codes are available at https://github.com/CUHK-AIM-Group/MonoSplat.

As 3D Gaussian Splatting (3D-GS) emerges as a promising technique for 3D reconstruction and novel view synthesis, offering superior rendering quality and efficiency, it becomes crucial to ensure secure transmission and copyright protection of 3D assets in anticipation of widespread distribution. While steganography has advanced significantly in common 3D media like meshes and Neural Radiance Fields (NeRF), research into steganography for 3D- GS representations remains largely unexplored. To address this gap, we propose ConcealGS, a novel 3D steganography method that embeds implicit information into the explicit 3D representation of Gaussian Splatting. By introducing a consistency strategy for the decoder and a gradient optimization approach, ConcealGS overcomes limitations of NeRF-based models, enhancing both the robustness of implicit information and the quality of 3D reconstruction. Extensive evaluations across various potential application scenarios demonstrate that ConcealGS successfully recovers implicit information with negligible impact on rendering quality, offering a groundbreaking approach for embedding invisible yet recoverable information into 3D models. This work paves the way for advanced copyright protection and secure data transmission in the evolving landscape of 3D content creation and distribution. Code is available at https://github.com/zxk1212/ConcealGS.

With the rapid development of large generative models for 3D, especially the evolution from NeRF representations to more efficient Gaussian Splatting, the synthesis of 3D assets has become increasingly fast and efficient, enabling the large-scale publication and sharing of generated 3D objects. However, while existing methods can add watermarks or steganographic information to individual 3D assets, they often require time-consuming per-scene training and optimization, leading to watermarking overheads that can far exceed the time required for asset generation itself, making deployment impractical for generating large collections of 3D objects. To address this, we propose InstantSplamp a framework that seamlessly integrates the 3D steganography pipeline into large 3D generative models without introducing explicit additional time costs. Guided by visual foundation models,InstantSplamp subtly injects hidden information like copyright tags during asset generation, enabling effective embedding and recovery of watermarks within generated 3D assets while preserving original visual quality. Experiments across various potential deployment scenarios demonstrate that \model~strikes an optimal balance between rendering quality and hiding fidelity, as well as between hiding performance and speed. Compared to existing per-scene optimization techniques for 3D assets, InstantSplamp reduces their watermarking training overheads that are multiples of generation time to nearly zero, paving the way for real-world deployment at scale. Project page: https://gaussian-stego.github.io/.

As 4D extensions of 3D Gaussian Splatting (4D-GS) emerge as groundbreaking techniques for dynamic scene reconstruction and novel view synthesis in robotics and computer vision, ensuring the security and trustworthiness of these assets becomes crucial. While steganography has advanced significantly in 2D and 3D media, existing methods are inadequate for the complex, dynamic nature of 4D-GS representations. To address this gap, we propose Hide-in-Motion, a novel 4D steganography method for hiding information through deformation in Gaussian splatting. Our approach introduces a composite attribute and a Decouple Feature Field for coarse-to-fine deformation modeling and embedding implicit information, along with an Opacity-Guided Adaptive strategy. Hide-in-Motion overcomes the limitations of previous techniques, enhancing both the robustness of embedded information and the quality of 4D reconstruction. Extensive evaluations demonstrate that our method successfully embeds and recovers implicit information across various modalities while maintaining high rendering quality in dynamic scenes. This work not only advances copyright protection and secure data transmission for 4D assets but also paves the way for enhancing the security and integrity of 4D digital assets. Code is available at https://github.com/CUHK-AIM-Group/Hide-in-Motion.

Predicting individual-level non-neuroimaging phenotypes (e.g., fluid intelligence) using brain imaging data is a fundamental goal of neuroscience. Recent research has focused on utilizing high-cost functional magnetic resonance imaging (fMRI) to predict phenotypes seen during training. However, these methods 1) only consider predicting seen phenotypes, failing to achieve zero-shot inference for unseen phenotypes; 2) overlook the knowledge transfer from fMRI to structural MRI (sMRI), missing out on utilizing cost-effective sMRI for accurate predictions. To address these challenges, we propose a Foundational Model for Any Phenotype Prediction via fMRI to sMRI knowledge transfer (FM-APP), consisting of a Phenotypes Text Memory Bank (PTMB) module, Any Phenotype Prediction (APP) module, and fMRI to sMRI Knowledge Transfer (F2SKT) module. Our proposed FM-APP adapts to downstream tasks by generating regressor parameters instead of fine-tuning the model itself. Specifically, to retain important clues from seen phenotype descriptions, PTMB utilizes the BiomedCLIP model to store semantic features of seen phenotypes. To achieve any phenotype prediction, the APP introduces a regressor synthesizer for zero-shot inference. Additionally, to improve sMRI prediction accuracy while preserving its cost advantage, the F2SKT uses the PTMB to construct phenotype active maps, guiding adaptive knowledge transfer from fMRI to sMRI. Experiments on the Human Connectome Project (HCP) and HCP Aging datasets demonstrate our approach outperforms state-of-the-art methods, showcasing strong zero-shot inference capabilities and providing a novel framework for analyzing brain structure and phenotypes. Our code: https://github.com/ZhibinHe/FM-APP.

Reconstructing deformable anatomical structures from endoscopic videos is a pivotal and promising research topic that can enable advanced surgical applications and improve patient outcomes. While existing surgical scene reconstruction methods have made notable progress, they often suffer from slow rendering speeds due to using neural radiance fields, limiting their practical viability in real-world applications. To overcome this bottleneck, we propose EndoGaussian, a framework that integrates the strengths of 3D Gaussian Splatting representations, allowing for high-fidelity tissue reconstruction, efficient training, and real-time rendering. Specifically, we dedicate a Foundation Model-driven Initialization (FMI) module, which distills 3D cues from multiple vision foundation models (VFMs) to swiftly construct the preliminary scene structure for Gaussian initialization. Then, a Spatio-temporal Gaussian Tracking (SGT) is designed, efficiently modeling scene dynamics using the multi-scale HexPlane with spatio-temporal priors. Furthermore, to improve the dynamics modeling ability for scenes with large deformation, EndoGaussian integrates Motion-aware Frame Synthesis (MFS) to adaptively synthesize new frames as extra training constraints. Experimental results on public datasets demonstrate EndoGaussian's efficacy against prior state-of-the-art methods, including superior rendering speed (168 FPS, real-time), enhanced rendering quality (38.555 PSNR), and reduced training overhead (within 2 min/scene). These results underscore EndoGaussian's potential to significantly advance intraoperative surgery applications, paving the way for more accurate and efficient real-time surgical guidance and decision-making in clinical scenarios. Code is available at: https://github.com/CUHK-AIM-Group/EndoGaussian.

Despite previous endeavors to utilize Convolutional Neural Networks and Transformers as base networks for medical image analysis, their architectures still harbor inherent limitations: either an inability to model long-range dependencies or colossal computational consumption due to global self-attention. Recently, State Space Models (SSMs) have exhibited impressive capabilities in modeling long-term dependencies with satisfactory linear computational complexity. Nevertheless, extant medical visual SSMs are constrained by their limited capacity to capture inter-patch relationships and inefficient modeling due to the introduction of additional depth convolutions to handle high-dimensional data. In this paper, we propose a novel, Pure Visual State Space Model (PV-SSM) for high-dimensional medical data analysis. Different from prior medical visual SSMs, our proposed framework does not involve any convolutional or global attention operations while leverages a series of Pure-SSM blocks that employ a novel parallel-SSM mechanism to simultaneously extract feature data across different dimensions. Furthermore, we propose a learnable Parameterized Positional Encoding, which incorporates absolute positional information into patch features, effectively endowing inter-patch relationships with stronger inferential capabilities. We conducted extensive validation on various modalities of medical imaging data. Experimental results demonstrate superior performance and efficacy of our model against existing models. Our codes are available at https://github.com/chengwang96/PV-SSM

Generating diverse plausible outputs from a single input is crucial for addressing visual ambiguities, exemplified in medical imaging where experts may provide varying semantic segmentation annotations for the same image.Existing methods handles ambiguous segmentation relying on probabilistic modeling and extensive multi-output annotated data while often struggles with limited ambiguously labeled datasets common in real-world applications.To surmount the challenge, we propose P²SAM, a novel framework that leverages the Segment Anything Model (SAM)'s prior knowledge for ambiguous object segmentation. By transforming SAM's sensitivity to prompts into an advantage, we introduce a prior probabilistic space for prompts.Experimental results show that P²SAM significantly enhances medical segmentation precision and diversity using minimal ambiguously annotated samples. Benchmarking against state-of-the-art methods demonstrates superior performance with just 5.5% of the training data (+12% Dmax). This approach marks a significant advancement towards deploying probabilistic models in data-limited real-world scenarios.

Existing federated learning (FL) approaches are deployed under the unrealistic closed-set setting, with both training and testing classes belong to the same set, which makes the global model fail to identify the unseen classes as ‘unknown’. To this end, we aim to study a novel problem of federated open-set recognition (FedOSR), which learns an open-set recognition (OSR) model under federated paradigm such that it classifies seen classes while at the same time detects unknown classes. In this work, we propose a parameter disentanglement guided federated open-set recognition (FedPD) algorithm to address two core challenges of FedOSR: cross-client inter-set interference between learning closed-set and open-set knowledge and cross-client intra-set inconsistency by data heterogeneity. The proposed FedPD framework mainly leverages two modules, i.e., local parameter disentanglement (LPD) and global divide-and-conquer aggregation (GDCA), to first disentangle client OSR model into different subnetworks, then align the corresponding parts cross clients for matched model aggregation. Specifically, on the client side, LPD decouples an OSR model into a closed-set subnetwork and an open-set subnetwork by the task-related importance, thus preventing inter-set interference. On the server side, GDCA first partitions the two subnetworks into specific and shared parts, and subsequently aligns the corresponding parts through optimal transport to eliminate parameter misalignment. Extensive experiments on various datasets demonstrate the superior performance of our proposed method.

Point cloud segmentation is fundamental in many medical applications, such as aneurysm clipping and orthodontic planning. Recent methods mainly focus on designing powerful local feature extractors and generally overlook the segmentation around the boundaries between objects, which is extremely harmful to the clinical practice and degenerates the overall segmentation performance. To remedy this problem, we propose a GRAph-based Boundary-aware Network (GRAB-Net) with three paradigms, Graph-based Boundary-perception Module (GBM), Outer-boundary Context-assignment Module (OCM), and Inner-boundary Feature-rectification Module (IFM), for medical point cloud segmentation. Aiming to improve the segmentation performance around boundaries, GBM is designed to detect boundaries and interchange complementary information inside semantic and boundary features in the graph domain, where semantics-boundary correlations are modelled globally and informative clues are exchanged by graph reasoning. Furthermore, to reduce the context confusion that degenerates the segmentation performance outside the boundaries, OCM is proposed to construct the contextual graph, where dissimilar contexts are assigned to points of different categories guided by geometrical landmarks. In addition, we advance IFM to distinguish ambiguous features inside boundaries in a contrastive manner, where boundary-aware contrast strategies are proposed to facilitate the discriminative representation learning. Extensive experiments on two public datasets, IntrA and 3DTeethSeg, demonstrate the superiority of our method over state-of-the-art methods.

Endoscopy super-resolution (SR) plays an important role in improving diagnostic results and reducing the misdiagnosis rate. Even though recent studies have investigated the SR for endoscopy, these methods apply equal importance to the whole image and do not consider the relationship among pixels, especially the depth information, which can provide diagnosis-related information for clinicians. To address this problem, we propose a dynamic depth-aware network for endoscopy super-resolution, which represents the first effort to comprehensively integrate the depth information to the SR task for endoscopic images. It includes a depth-wise feature extracting branch (DW-B) and a depth-guided SR branch (DGSR-B). The DW-B aims to extract the representative feature for each depth level (i.e. depth matrix) further to provide auxiliary information and guide the super-resolution of texture under different depth levels. In DGSR-B, a depth-guided block (DGB) consisting of depth-focus normalization (DFN) is introduced to inject both the depth matrix and depth map into the LR image feature, so as to guide the image generation for each depth region. To adaptively super-resolve the regions under different depth levels, we devise a dynamic depth-aware loss to assign different trainable weights to each region for SR optimization. Extensive experiments have been conducted on two main publicly available datasets, i.e., the Kvasir dataset and the EndoScene dataset, and the superior performance verifies the effectiveness of our method for SR task and polyp segmentation. Source code is to be released.