Accurate segmentation of neurons in 3D fluorescence microscopy images is essential for advancing neuroscience. Prevalent methods split a volume into patches and process each patch separately due to computational resource limitations. However, they fail to capture global neuronal morphology across multiple patches, which results in discontinuous segmentation and poses a challenge for subsequent neuronal reconstruction. In this paper, we propose a dual U-Net architecture termed "Glancing Beyond Patch" Network (GBP-Net) to incorporate contextual information into segmentation. Specifically, GBP-Net encodes contextual and high-resolution information using two U-Nets, respectively, and facilitates their integration through a cross-scale context module (CSCM) and a cross-resolution fusion module (CRFM). CSCM utilizes a cross-attention mechanism to enable interaction of features from diverse fields of view after encoders, and CRFM employs Mamba to adaptively fuse high-resolution features after decoders. These two modules complement each other at different levels, enabling to capture both global neuronal structures and fine-grained details. Additionally, the proposed cross-network loss guides to focus on challenging samples by penalizing misclassified voxels from both networks, which further promotes the exploitation of contextual information. Experimental results on three datasets demonstrate that our method outperforms other advanced segmentation methods while maintaining computational efficiency. It keeps the global structure of neurons and achieves the highest F1 scores.

Accurate segmentation of organelles in electron microscopy (EM) volumes is essential for understanding intracellular organization. While promising, deep learning-based methods could be unstable and unreliable without sufficient annotations. Masked image modeling (MIM), a powerful pretraining technique, has proven effective in enhancing segmentation by extracting meaningful representations from large-scale unlabeled data. However, random masking strategies in classic MIMs could overlook the unique structural patterns of organelles and the spatial redundancy inherent in EM volumes, thus limiting pretraining efficiency. To address this issue, we propose OrgMIM, a dual-branch MIM framework that integrates complementary masking strategies to capture critical subcellular semantics and learn organelle-specific representations from EM data. Specifically, one branch is guided by static structural priors, leveraging visual foundation models to generate affinity maps that indicate organelle membranes as masking candidates. The other is driven by dynamic reconstruction feedback, using a self-guidance mechanism to compute average loss maps that highlight intricate organelle patterns for heuristic masking. Moreover, cross-branch consistency regularization is introduced for reliable representation learning across sparse semantic contexts. To support large-scale pretraining, we construct IsoOrg-1K, the first organelle-centric 3D EM dataset, comprising 928 informative volumes and over 120 billion voxels. Extensive evaluations on three public EM datasets with varied resolutions and appearances validate the superior performance of OrgMIM. Notably, OrgMIM pretraining on IsoOrg-1K boosts mIoU by 28.78% over training from scratch on the CMCC dataset with a Transformer-based model. All datasets, source codes, and pretrained weights are available at https://github.com/yanchaoz/OrgMIM.

Despite advances in ultrathin cutting, serial sections in electron microscopy (EM) still exhibit noticeable anisotropy, with much lower z-axis resolution compared with the other two axes. As a result, the imaged biovolume suffers from low connectivity smoothness in contextual structures, which makes the subcellular ultrastructural segmentation challenging. The recent 2.5D hybrid convolutions allow the direct learning of asymmetric semantics from anisotropic features. However, plain representations without the isotropic scale prior limit the performance of the upper bound. This paper presents a novel framework, referred to as ReIsoSeg, which aims at incorporating an isotropic scaling prior into anisotropic biovolumes. More precisely, ReIsoSeg consists of an anisotropic primary encoder, a pseudo-isotropic auxiliary module, and a weight-shared decoder. The auxiliary module implicitly deforms the anisotropic features from the primary encoder to align with the isotropic prior. The re-isotropic loss squeezes the pseudo-isotropic representations into the anisotropic space to reuse the anisotropic labels. The shared decoder ensures that the outputs of the anisotropic encoder converge towards the isotropic representations. During the inference process, the auxiliary module is excluded. Comprehensive experiments were conducted on the AC3/AC4, CREMI, and MitoEM subcellular ultrastructure datasets. The obtained results demonstrate the high performance of the proposed ReIsoSeg.

With an explosion in the uptake of volume electron microscopy (vEM) across neuroscience and fast-paced advances in imaging protocols, it is timely to introduce general-purpose automation for newly generated large-scale vEM datasets. Recent vision foundation models (e.g., SAM) set a new benchmark for the generalization of 2D segmentation. However, SAM has difficulty handling neurons that are densely packed into 3D volumes. To overcome this obstacle, we consider solutions from both data and model aspects. In terms of data, we introduce a data engine to optimize manual labeling, including (i) human-in-the-loop data cleansing and (ii) model-in-the-loop data unification. In terms of model, we present the SAEM2-SAvEM3with strategies, including (i) auxiliary learning, which predicts complementary representations for SAM masks and improves performance on dense instances; (ii) full-stage distillation, which integrates ViT embeddings into a 3D U-Net, achieves 2D-to-3D lifting and model slimming at the same time; and (iii) prompt-based graph partitioning, which reuses SAM prompts to assign weights of nodes and edges in the oversegmentation graph. According to evaluations of dense and large-scale sparse neurons, the out-of-distribution performance of our pretrained-distilled models is on par with the state-of-the-art supervised and semi-supervised methods. The overall pipeline provides a possible general-purpose solution for 3D neuron reconstruction in any new vEM data. Our code are available at https://github.com/JackieZhai/SAvEM3.

Superpixel aggregation is a powerful tool for automated neuron segmentation from electron microscopy (EM) volume. However, existing graph partitioning methods for superpixel aggregation still involve two separate stages-model estimation and model solving, and therefore model error is inherent. To address this issue, we integrate the two stages and propose an end-to-end aggregation framework based on deep learning of the minimum cost multicut problem called DeepMulticut. The core challenge lies in differentiating the NP-hard multicut problem, whose constraint number is exponential in the problem size. With this in mind, we resort to relaxing the combinatorial solver-the greedy additive edge contraction (GAEC)-to a continuous Soft-GAEC algorithm, whose limit is shown to be the vanilla GAEC. Such relaxation thus allows the DeepMulticut to integrate edge cost estimators, Edge-CNNs, into a differentiable multicut optimization system and allows a decision-oriented loss to feed decision quality back to the Edge-CNNs for adaptive discriminative feature learning. Hence, the model estimators, Edge-CNNs, can be trained to improve partitioning decisions directly while beyond the NP-hardness. Also, we explain the rationale behind the DeepMulticut framework from the perspective of bi-level optimization. Extensive experiments on three public EM datasets demonstrate the effectiveness of the proposed DeepMulticut.

BACKGROUND: Nanoscale connectomics, which aims to map the fine connections between neurons with synaptic-level detail, has attracted increasing attention in recent years. Currently, the automated reconstruction algorithms in electron microscope volumes are in great demand. Most existing reconstruction methodologies for cellular and subcellular structures are independent, and exploring the inter-relationships between structures will contribute to image analysis. The primary goal of this research is to construct a joint optimization framework to improve the accuracy and efficiency of neural structure reconstruction algorithms. RESULTS: In this investigation, we introduce the concept of connectivity consensus between cellular and subcellular structures based on biological domain knowledge for neural structure agglomeration problems. We propose a joint graph partitioning model for solving ultrastructural and neuronal connections to overcome the limitations of connectivity cues at different levels. The advantage of the optimization model is the simultaneous reconstruction of multiple structures in one optimization step. The experimental results on several public datasets demonstrate that the joint optimization model outperforms existing hierarchical agglomeration algorithms. CONCLUSIONS: We present a joint optimization model by connectivity consensus to solve the neural structure agglomeration problem and demonstrate its superiority to existing methods. The intention of introducing connectivity consensus between different structures is to build a suitable optimization model that makes the reconstruction goals more consistent with biological plausible and domain knowledge. This idea can inspire other researchers to optimize existing reconstruction algorithms and other areas of biological data analysis.

Biologists often need to handle numerous video-based home-cage animal behavior analysis tasks that require massive workloads. Therefore, we develop an AI-based multi-species tracking and segmentation system, SiamBOMB, for real-time and automatic home-cage animal behavioral analysis. In this system, a background-enhanced Siamese-based network with replaceable modular design ensures the flexibility and generalizability of the system, and a user-friendly interface makes it convenient to use for biologists. This real-time AI system will effectively reduce the burden on biologists.