Accurate segmentation of neurons in 3D fluorescence microscopy images is essential for advancing neuroscience. Prevalent methods split a volume into patches and process each patch separately due to computational resource limitations. However, they fail to capture global neuronal morphology across multiple patches, which results in discontinuous segmentation and poses a challenge for subsequent neuronal reconstruction. In this paper, we propose a dual U-Net architecture termed "Glancing Beyond Patch" Network (GBP-Net) to incorporate contextual information into segmentation. Specifically, GBP-Net encodes contextual and high-resolution information using two U-Nets, respectively, and facilitates their integration through a cross-scale context module (CSCM) and a cross-resolution fusion module (CRFM). CSCM utilizes a cross-attention mechanism to enable interaction of features from diverse fields of view after encoders, and CRFM employs Mamba to adaptively fuse high-resolution features after decoders. These two modules complement each other at different levels, enabling to capture both global neuronal structures and fine-grained details. Additionally, the proposed cross-network loss guides to focus on challenging samples by penalizing misclassified voxels from both networks, which further promotes the exploitation of contextual information. Experimental results on three datasets demonstrate that our method outperforms other advanced segmentation methods while maintaining computational efficiency. It keeps the global structure of neurons and achieves the highest F1 scores.

Accurate segmentation of organelles in electron microscopy (EM) volumes is essential for understanding intracellular organization. While promising, deep learning-based methods could be unstable and unreliable without sufficient annotations. Masked image modeling (MIM), a powerful pretraining technique, has proven effective in enhancing segmentation by extracting meaningful representations from large-scale unlabeled data. However, random masking strategies in classic MIMs could overlook the unique structural patterns of organelles and the spatial redundancy inherent in EM volumes, thus limiting pretraining efficiency. To address this issue, we propose OrgMIM, a dual-branch MIM framework that integrates complementary masking strategies to capture critical subcellular semantics and learn organelle-specific representations from EM data. Specifically, one branch is guided by static structural priors, leveraging visual foundation models to generate affinity maps that indicate organelle membranes as masking candidates. The other is driven by dynamic reconstruction feedback, using a self-guidance mechanism to compute average loss maps that highlight intricate organelle patterns for heuristic masking. Moreover, cross-branch consistency regularization is introduced for reliable representation learning across sparse semantic contexts. To support large-scale pretraining, we construct IsoOrg-1K, the first organelle-centric 3D EM dataset, comprising 928 informative volumes and over 120 billion voxels. Extensive evaluations on three public EM datasets with varied resolutions and appearances validate the superior performance of OrgMIM. Notably, OrgMIM pretraining on IsoOrg-1K boosts mIoU by 28.78% over training from scratch on the CMCC dataset with a Transformer-based model. All datasets, source codes, and pretrained weights are available at https://github.com/yanchaoz/OrgMIM.

Despite advances in ultrathin cutting, serial sections in electron microscopy (EM) still exhibit noticeable anisotropy, with much lower z-axis resolution compared with the other two axes. As a result, the imaged biovolume suffers from low connectivity smoothness in contextual structures, which makes the subcellular ultrastructural segmentation challenging. The recent 2.5D hybrid convolutions allow the direct learning of asymmetric semantics from anisotropic features. However, plain representations without the isotropic scale prior limit the performance of the upper bound. This paper presents a novel framework, referred to as ReIsoSeg, which aims at incorporating an isotropic scaling prior into anisotropic biovolumes. More precisely, ReIsoSeg consists of an anisotropic primary encoder, a pseudo-isotropic auxiliary module, and a weight-shared decoder. The auxiliary module implicitly deforms the anisotropic features from the primary encoder to align with the isotropic prior. The re-isotropic loss squeezes the pseudo-isotropic representations into the anisotropic space to reuse the anisotropic labels. The shared decoder ensures that the outputs of the anisotropic encoder converge towards the isotropic representations. During the inference process, the auxiliary module is excluded. Comprehensive experiments were conducted on the AC3/AC4, CREMI, and MitoEM subcellular ultrastructure datasets. The obtained results demonstrate the high performance of the proposed ReIsoSeg.

Superpixel aggregation represents a highly effective approach for automated neuron segmentation from electron microscopy (EM) volumes, which can be considered as a graph partitioning task on the region adjacency graph (RAG) of extracted superpixels. However, existing graph partitioning models for superpixel aggregation suffer from the modeling error due to insufficient model capacity. More specifically, the modeling error is caused by the simplification in formulating the real-world graph partitioning task (i.e., superpixel aggregation) into a mathematically well-defined optimization problem. To address this issue, we sidestep the explicit formulation and propose a fully end-to-end superpixel aggregation method based on deep learning of the graph partitioning task, called DeepPartitioning. The central challenge lies in characterizing the partitioning task involving combinatorial complexity. Hence, our method incorporates a line graph neural network (LGNN) to capture higher-order relational structures in RAGs. Specifically, the LGNN enables the propagation of second-order superpixel-pair features among adjacent edges in RAGs. In this way, the partitioning task can be implicitly transformed into the vanilla second-order multicut problem while maintaining higher-order structural information. Overall, our method integrates a second-order feature extractor, a higher-order feature integrator (i.e., the LGNN), and a differentiable approximation to a multicut solver into a unified, learnable framework. Extensive experiments on three public EM datasets demonstrate the effectiveness of the proposed DeepPartitioning within the neuron segmentation pipeline.

The multicut problem, also known as correlation clustering, is a classic combinatorial optimization problem that aims to optimize graph partitioning given only node (dis)similarities on edges. It serves as an elegant generalization for several graph partitioning problems and has found successful applications in various areas such as data mining and computer vision. However, the multicut problem with an exponentially large number of cycle constraints proves to be NP-hard, and existing solvers either suffer from exponential complexity or often give unsatisfactory solutions due to inflexible heuristics driven by hand-designed mechanisms. In this article, we propose a deep graph reinforcement learning method to solve the multicut problem within a combinatorial decision framework involving sequential edge contractions. The customized subgraph neural network adapts to the dynamically edge-contracted graph environment by extracting bilevel connected features from both contracted and original graphs. Our method can learn to infer feasible multicut solutions end-to-end toward optimization of the multicut objective in a data-driven manner. More specifically, by exploring the decision space adaptively, it implicitly gains heuristic knowledge from topological patterns of instances and thereby generates more targeted heuristics overcoming the short-sightedness inherent in the hand-designed ones. During testing, the learned heuristics iteratively contract graphs to construct high-quality solutions within polynomial time. Extensive experiments on synthetic and real-world multicut instances show the superiority of our method over existing combinatorial solvers, while also maintaining a certain level of out-of-distribution generalization ability.

With an explosion in the uptake of volume electron microscopy (vEM) across neuroscience and fast-paced advances in imaging protocols, it is timely to introduce general-purpose automation for newly generated large-scale vEM datasets. Recent vision foundation models (e.g., SAM) set a new benchmark for the generalization of 2D segmentation. However, SAM has difficulty handling neurons that are densely packed into 3D volumes. To overcome this obstacle, we consider solutions from both data and model aspects. In terms of data, we introduce a data engine to optimize manual labeling, including (i) human-in-the-loop data cleansing and (ii) model-in-the-loop data unification. In terms of model, we present the SAEM2-SAvEM3with strategies, including (i) auxiliary learning, which predicts complementary representations for SAM masks and improves performance on dense instances; (ii) full-stage distillation, which integrates ViT embeddings into a 3D U-Net, achieves 2D-to-3D lifting and model slimming at the same time; and (iii) prompt-based graph partitioning, which reuses SAM prompts to assign weights of nodes and edges in the oversegmentation graph. According to evaluations of dense and large-scale sparse neurons, the out-of-distribution performance of our pretrained-distilled models is on par with the state-of-the-art supervised and semi-supervised methods. The overall pipeline provides a possible general-purpose solution for 3D neuron reconstruction in any new vEM data. Our code are available at https://github.com/JackieZhai/SAvEM3.

Superpixel aggregation is a powerful tool for automated neuron segmentation from electron microscopy (EM) volume. However, existing graph partitioning methods for superpixel aggregation still involve two separate stages-model estimation and model solving, and therefore model error is inherent. To address this issue, we integrate the two stages and propose an end-to-end aggregation framework based on deep learning of the minimum cost multicut problem called DeepMulticut. The core challenge lies in differentiating the NP-hard multicut problem, whose constraint number is exponential in the problem size. With this in mind, we resort to relaxing the combinatorial solver-the greedy additive edge contraction (GAEC)-to a continuous Soft-GAEC algorithm, whose limit is shown to be the vanilla GAEC. Such relaxation thus allows the DeepMulticut to integrate edge cost estimators, Edge-CNNs, into a differentiable multicut optimization system and allows a decision-oriented loss to feed decision quality back to the Edge-CNNs for adaptive discriminative feature learning. Hence, the model estimators, Edge-CNNs, can be trained to improve partitioning decisions directly while beyond the NP-hardness. Also, we explain the rationale behind the DeepMulticut framework from the perspective of bi-level optimization. Extensive experiments on three public EM datasets demonstrate the effectiveness of the proposed DeepMulticut.

MOTIVATION: The registration of serial section electron microscope images is a critical step in reconstructing biological tissue volumes, and it aims to eliminate complex nonlinear deformations from sectioning and replicate the correct neurite structure. However, due to the inherent properties of biological structures and the challenges posed by section preparation of biological tissues, achieving an accurate registration of serial sections remains a significant challenge. Conventional nonlinear registration techniques, which are effective in eliminating nonlinear deformation, can also eliminate the natural morphological variation of neurites across sections. Additionally, accumulation of registration errors alters the neurite structure. RESULTS: This article proposes a novel method for serial section registration that utilizes an unsupervised optical flow network to measure feature similarity rather than pixel similarity to eliminate nonlinear deformation and achieve pairwise registration between sections. The optical flow network is then employed to estimate and compensate for cumulative registration error, thereby allowing for the reconstruction of the structure of biological tissues. Based on the novel serial section registration method, a serial split technique is proposed for long-serial sections. Experimental results demonstrate that the state-of-the-art method proposed here effectively improves the spatial continuity of serial sections, leading to more accurate registration and improved reconstruction of the structure of biological tissues. AVAILABILITY AND IMPLEMENTATION: The source code and data are available at https://github.com/TongXin-CASIA/EFSR.

Accurate and robust segmentation of mitochondria from electron microscopy images plays a critical role in understanding cellular functions. While effective, learning-based approaches require vast quantities of expert annotations. Manual efforts can be alleviated by transferring the knowledge learned from the source domain to unseen target domains, as known as unsupervised domain adaptation. In this work, we propose a two-stage pipeline for cross-dataset mitochondria segmentation, aiming to mitigate domain shift in a coarse-to-fine manner. In the first stage, we integrate the style transfer block and segmentation network into an end-to-end image alignment framework. Specifically, patch-wise contrastive learning is employed to guarantee the semantic fidelity of mitochondria, providing more reliable translated images for the segmentation network. In the second stage, a novel online self-training network updated with target images and co-evolving pseudo labels is proposed to fine-tune the segmentation network trained beforehand. Furthermore, to effectively utilize unlabeled data, consistency regularization is introduced in both stages to enforce stable predictions under various perturbations. Experimental results on public datasets demonstrate that the proposed approach outperforms the existing methods by a large margin in bidirectional adaptation for mitochondria segmentation and achieves comparable results with supervised methods.

Real noisy-clean pairs on a large scale are costly and difficult to obtain. Meanwhile, supervised denoisers trained on synthetic data perform poorly in practice. Self-supervised denoisers, which learn only from single noisy images, solve the data collection problem. However, self-supervised denoising methods, especially blindspot-driven ones, suffer sizable information loss during input or network design. The absence of valuable information dramatically reduces the upper bound of denoising performance. In this paper, we propose a simple yet efficient approach called Blind2Unblind to overcome the information loss in blindspot-driven denoising methods. First, we introduce a global-aware mask mapper that enables global perception and accelerates training. The mask mapper samples all pixels at blind spots on denoised volumes and maps them to the same channel, allowing the loss function to optimize all blind spots at once. Second, we propose a revisible loss to train the denoising network and make blind spots visible. The denoiser can learn directly from raw noise images without losing information or being trapped in identity mapping. We also theoretically analyze the convergence of the revisible loss. Extensive experiments on synthetic and real-world datasets demonstrate the superior performance of our approach compared to previous work. Code is available at https://github.com/demonsjin/Blind2Unblind.

BACKGROUND: Nanoscale connectomics, which aims to map the fine connections between neurons with synaptic-level detail, has attracted increasing attention in recent years. Currently, the automated reconstruction algorithms in electron microscope volumes are in great demand. Most existing reconstruction methodologies for cellular and subcellular structures are independent, and exploring the inter-relationships between structures will contribute to image analysis. The primary goal of this research is to construct a joint optimization framework to improve the accuracy and efficiency of neural structure reconstruction algorithms. RESULTS: In this investigation, we introduce the concept of connectivity consensus between cellular and subcellular structures based on biological domain knowledge for neural structure agglomeration problems. We propose a joint graph partitioning model for solving ultrastructural and neuronal connections to overcome the limitations of connectivity cues at different levels. The advantage of the optimization model is the simultaneous reconstruction of multiple structures in one optimization step. The experimental results on several public datasets demonstrate that the joint optimization model outperforms existing hierarchical agglomeration algorithms. CONCLUSIONS: We present a joint optimization model by connectivity consensus to solve the neural structure agglomeration problem and demonstrate its superiority to existing methods. The intention of introducing connectivity consensus between different structures is to build a suitable optimization model that makes the reconstruction goals more consistent with biological plausible and domain knowledge. This idea can inspire other researchers to optimize existing reconstruction algorithms and other areas of biological data analysis.

Registration of serial section electron microscopy (ssEM) images is essential for neural circuit reconstruction. Morphologies of neurite structure in adjacent sections are different. Thus, it is challenging to extract valid features in ssEM image registration. Convolutional neural networks (CNN) have made unprecedented progress in feature extraction of natural images. However, morphological differences need not be considered in the registration of natural images. Directly applying these methods will result in matching failure or over-registration. This paper proposes an unsupervised learning-based representation taking the morphological differences of ssEM images into account. CNN architecture was used to extract the feature. To train the network, the focused ion beam scanning electron microscope (FIB-SEM) images are used. The FIB-SEM images are in situ, so they are naturally registered. Sampling those images with a certain thickness can teach CNN to learn changes in neurite structure. The learned feature can be directly applied to existing ssEM image registration methods and reduce the negative effect of section thickness on registration accuracy. The experimental results show that the proposed feature outperforms the state-of-the-art method in matching accuracy and significantly improves the registration outcome when used in ssEM images.

Due to the limitation of the satellite attitude measurement accuracy and the system servo control error of the payload scanning mechanism, an optimal use of Micro-Wave Radiation Imager (MWRI) observations requires high geolocation accuracy. In the operational system, the MWRI geolocation accuracy reaches 1 pixel, and there still exists room for improvement. In this article, we improve upon the coastline inflection point method (CIM) and propose to assign the accurate correspondence by employing a nonrigid point set registration method. First, the method identifies a set of latent variables to recognize outliers and then applies nonparametric geometric constraints to the correspondence asa prioridistribution. Second, the maximuma posteriori(MAP) estimation is applied by the expectation–maximization (EM) algorithm to obtain correct inliers. The comparison with other methods demonstrates that the proposed method can provide more accurate estimation of geolocation bias. In addition, the pixel error and changes in spacecraft attitude with the long-term geolocation data in FY-3C MWRI before and after correction were analyzed during the period from April 1 to August 30, 2018. The results have shown that the geolocation errors are reduced from [0.50, 0.60] pixels to [0.20, 0.33] pixels in the along- and cross-track directions after the attitude correction. In addition, the reduction of the standard deviation shows that the geolocation quality of MWRI is improved.

Affine registration can fit the non-rigid deformation of slices effectively, and it is widely used in volume reconstruction of biological tissue. But most of the existing affine registration methods are registered in a given sequence, which results in the accumulation of errors. In this paper, a global optimized affine registration method is proposed, which can be used in volume reconstruction. To eliminate the cumulative error, the affine transformation of all images is estimated simultaneously based on an energy function. A soft penalty on affine transformation is added to restrict the shearing of images. Experiments show that our method provides a more reliable registration result compared with sequential affine registration. It can solve the problems caused by the accumulation of errors. The registration result fits the deformation of slices well and preserves the rigidity of images.

Biologists often need to handle numerous video-based home-cage animal behavior analysis tasks that require massive workloads. Therefore, we develop an AI-based multi-species tracking and segmentation system, SiamBOMB, for real-time and automatic home-cage animal behavioral analysis. In this system, a background-enhanced Siamese-based network with replaceable modular design ensures the flexibility and generalizability of the system, and a user-friendly interface makes it convenient to use for biologists. This real-time AI system will effectively reduce the burden on biologists.

Known as input in the numerical weather prediction (NWP) models, microwave radiation imager (MWRI) data have been widely distributed to the user community. Nevertheless, the current operational geolocation accuracy is still on the pixel scale due to the presence of geolocation uncertainty. In this letter, we propose a new method to estimate the geolocation errors in MWRI data. Compared to the traditional coastline inflection method (CIM), the proposed method has two innovations. First, we establish a surface fitting interpolation model by involving more observations to detect the coastline. Second, we employ the iterative closest point (ICP) algorithm to determine the correspondences between the detected coastline and the actual coastline. Simulated experimental results demonstrate that the proposed method can provide a more accurate geolocation error estimation than the CIM. By applying our method, we have processed an MWRI data set from January 1 to February 28 in 2016. The experimental results have shown that the operational FY-3C MWRI geolocation errors are 0.4813 and 0.4909 pixels in the along-track and cross-track directions, respectively, which can be significantly reduced to 0.1299 and 0.1497 pixels after the attitude correction. It means that the geolocation accuracy has an average improvement up to 70%.

Many effective image deformation methods have been proposed in recent years, but few of them use the contours of objects as the reference of deformation. However, the contour is an important factor, and contour‐based deformation can guarantee that the edge of the object after deformation is smoother compared to point‐based image deformation. This article presents an image deformation method based on contours using Moving Integral Least Squares (MILS) optimisation. First, the authors set the key points in the image to create control contours as required and then adjust the positions of these points to generate the desired contours. In order to warp these contours to their new positions, the image is deformed using MILS. They derive the affine, similarity, and rigid transformations in a general framework, and users can choose different curves according to their needs. The proposed method possesses two characteristics: (i) it is able to create detail‐preserving and intuitive deformations; and (ii) the solution of the deformation function has a simple closed form. They compare their method to the state‐of‐the‐art algorithm, which is modelled by rigid transformation. Experimental results show that their deformation is more vivid.

The microwave radiation imager (MWRI) located onboard the FengYun-3C (FY-3C) satellite provides a considerable amount of critical information for numerical weather predictions. Obtaining accurate geolocation results from the FY-3C MWRI data is of great importance. In this letter, we improve the traditional coastline inflection method (CIM) and propose an$\ell _{0}$sparse approximation model for geolocation error estimation and correction. Specifically, we propose using the jump point of the step function to estimate the true coastline point. This approach can characterize the geolocation errors more accurately than the CIM, which further improves the geolocation accuracy. In the theoretical part, we provide a complete solution to obtain the step function through an iterative blind deconvolution. For a practical use, we demonstrate the effectiveness of the proposed method for geolocation error estimation through quantitative results obtained on the FY-3C MWRI data. The experimental results show that the proposed method can achieve an improvement of up to 33.33% in the standard deviation of geolocation errors (approximately 0.00030) compared to the traditional CIM (approximately 0.00045). Furthermore, we also apply the proposed method to the FY-3C satellite and improve the geolocation accuracy of the MWRI data through geolocation error correction.

Image registration of serial electron microscopy (EM) sections is a feasible way to reveal 3D structure of the biological tissue. However, the image registration proves difficult, as it is hard to find reliable correspondences between the adjacent sections and the section distortion may occur during the sample preparation. In this paper, we propose a non-linear image registration method for serial EM sections, which is composed of pairwise correspondences extraction, correspondences position adjustment and image warping. The proposed method is highly automatic, and retains the morphology of the original electron microscopic images as much as possible. We demonstrate that our method outperforms the state-of-the-art approaches on several datasets of serial EM sections images including a synthetic test case.

BACKGROUND: The locations and shapes of synapses are important in reconstructing connectomes and analyzing synaptic plasticity. However, current synapse detection and segmentation methods are still not adequate for accurately acquiring the synaptic connectivity, and they cannot effectively alleviate the burden of synapse validation. RESULTS: We propose a fully automated method that relies on deep learning to realize the 3D reconstruction of synapses in electron microscopy (EM) images. The proposed method consists of three main parts: (1) training and employing the faster region convolutional neural networks (R-CNN) algorithm to detect synapses, (2) using the z-continuity of synapses to reduce false positives, and (3) combining the Dijkstra algorithm with the GrabCut algorithm to obtain the segmentation of synaptic clefts. Experimental results were validated by manual tracking, and the effectiveness of our proposed method was demonstrated. The experimental results in anisotropic and isotropic EM volumes demonstrate the effectiveness of our algorithm, and the average precision of our detection (92.8% in anisotropy, 93.5% in isotropy) and segmentation (88.6% in anisotropy, 93.0% in isotropy) suggests that our method achieves state-of-the-art results. CONCLUSIONS: Our fully automated approach contributes to the development of neuroscience, providing neurologists with a rapid approach for obtaining rich synaptic statistics.

Big data research has become a globally hot topic in recent years. One of the core problems in big data learning is how to extract effective information from the huge data. In this paper, we propose a Markov resampling algorithm to draw useful samples for handling coefficient-based regularized regression (CBRR) problem. The proposed Markov resampling algorithm is a selective sampling method, which can automatically select uniformly ergodic Markov chain (u.e.M.c.) samples according to transition probabilities. Based on u.e.M.c. samples, we analyze the theoretical performance of CBRR algorithm and generalize the existing results on independent and identically distributed observations. To be specific, when the kernel is infinitely differentiable, the learning rate depending on the sample size $m$ can be arbitrarily close to $\mathcal {O}(m^{-1})$ under a mild regularity condition on the regression function. The good generalization ability of the proposed method is validated by experiments on simulated and real data sets.

In this paper we propose an image super-resolution algorithm using wavelet-domain hidden Markov tree (HMT) model. Wavelet-domain HMT models the dependencies of multiscale wavelet coefficients through the state probabilities of wavelet coefficients, whose distribution densities can be approximated by the Gaussian mixture. Because wavelet-domain HMT accurately characterizes the statistics of real-world images, we reasonably specify it as the prior distribution and then formulate the image super-resolution problem as a constrained optimization problem. And the cycle-spinning technique is used to suppress the artifacts that may exist in the reconstructed high-resolution images. Quantitative error analyses are provided and several experimental images are shown for subjective assessment.