Superpixel aggregation is a powerful tool for automated neuron segmentation from electron microscopy (EM) volume. However, existing graph partitioning methods for superpixel aggregation still involve two separate stages-model estimation and model solving, and therefore model error is inherent. To address this issue, we integrate the two stages and propose an end-to-end aggregation framework based on deep learning of the minimum cost multicut problem called DeepMulticut. The core challenge lies in differentiating the NP-hard multicut problem, whose constraint number is exponential in the problem size. With this in mind, we resort to relaxing the combinatorial solver-the greedy additive edge contraction (GAEC)-to a continuous Soft-GAEC algorithm, whose limit is shown to be the vanilla GAEC. Such relaxation thus allows the DeepMulticut to integrate edge cost estimators, Edge-CNNs, into a differentiable multicut optimization system and allows a decision-oriented loss to feed decision quality back to the Edge-CNNs for adaptive discriminative feature learning. Hence, the model estimators, Edge-CNNs, can be trained to improve partitioning decisions directly while beyond the NP-hardness. Also, we explain the rationale behind the DeepMulticut framework from the perspective of bi-level optimization. Extensive experiments on three public EM datasets demonstrate the effectiveness of the proposed DeepMulticut.

MOTIVATION: The registration of serial section electron microscope images is a critical step in reconstructing biological tissue volumes, and it aims to eliminate complex nonlinear deformations from sectioning and replicate the correct neurite structure. However, due to the inherent properties of biological structures and the challenges posed by section preparation of biological tissues, achieving an accurate registration of serial sections remains a significant challenge. Conventional nonlinear registration techniques, which are effective in eliminating nonlinear deformation, can also eliminate the natural morphological variation of neurites across sections. Additionally, accumulation of registration errors alters the neurite structure. RESULTS: This article proposes a novel method for serial section registration that utilizes an unsupervised optical flow network to measure feature similarity rather than pixel similarity to eliminate nonlinear deformation and achieve pairwise registration between sections. The optical flow network is then employed to estimate and compensate for cumulative registration error, thereby allowing for the reconstruction of the structure of biological tissues. Based on the novel serial section registration method, a serial split technique is proposed for long-serial sections. Experimental results demonstrate that the state-of-the-art method proposed here effectively improves the spatial continuity of serial sections, leading to more accurate registration and improved reconstruction of the structure of biological tissues. AVAILABILITY AND IMPLEMENTATION: The source code and data are available at https://github.com/TongXin-CASIA/EFSR.

BACKGROUND: Nanoscale connectomics, which aims to map the fine connections between neurons with synaptic-level detail, has attracted increasing attention in recent years. Currently, the automated reconstruction algorithms in electron microscope volumes are in great demand. Most existing reconstruction methodologies for cellular and subcellular structures are independent, and exploring the inter-relationships between structures will contribute to image analysis. The primary goal of this research is to construct a joint optimization framework to improve the accuracy and efficiency of neural structure reconstruction algorithms. RESULTS: In this investigation, we introduce the concept of connectivity consensus between cellular and subcellular structures based on biological domain knowledge for neural structure agglomeration problems. We propose a joint graph partitioning model for solving ultrastructural and neuronal connections to overcome the limitations of connectivity cues at different levels. The advantage of the optimization model is the simultaneous reconstruction of multiple structures in one optimization step. The experimental results on several public datasets demonstrate that the joint optimization model outperforms existing hierarchical agglomeration algorithms. CONCLUSIONS: We present a joint optimization model by connectivity consensus to solve the neural structure agglomeration problem and demonstrate its superiority to existing methods. The intention of introducing connectivity consensus between different structures is to build a suitable optimization model that makes the reconstruction goals more consistent with biological plausible and domain knowledge. This idea can inspire other researchers to optimize existing reconstruction algorithms and other areas of biological data analysis.

Registration of serial section electron microscopy (ssEM) images is essential for neural circuit reconstruction. Morphologies of neurite structure in adjacent sections are different. Thus, it is challenging to extract valid features in ssEM image registration. Convolutional neural networks (CNN) have made unprecedented progress in feature extraction of natural images. However, morphological differences need not be considered in the registration of natural images. Directly applying these methods will result in matching failure or over-registration. This paper proposes an unsupervised learning-based representation taking the morphological differences of ssEM images into account. CNN architecture was used to extract the feature. To train the network, the focused ion beam scanning electron microscope (FIB-SEM) images are used. The FIB-SEM images are in situ, so they are naturally registered. Sampling those images with a certain thickness can teach CNN to learn changes in neurite structure. The learned feature can be directly applied to existing ssEM image registration methods and reduce the negative effect of section thickness on registration accuracy. The experimental results show that the proposed feature outperforms the state-of-the-art method in matching accuracy and significantly improves the registration outcome when used in ssEM images.

Affine registration can fit the non-rigid deformation of slices effectively, and it is widely used in volume reconstruction of biological tissue. But most of the existing affine registration methods are registered in a given sequence, which results in the accumulation of errors. In this paper, a global optimized affine registration method is proposed, which can be used in volume reconstruction. To eliminate the cumulative error, the affine transformation of all images is estimated simultaneously based on an energy function. A soft penalty on affine transformation is added to restrict the shearing of images. Experiments show that our method provides a more reliable registration result compared with sequential affine registration. It can solve the problems caused by the accumulation of errors. The registration result fits the deformation of slices well and preserves the rigidity of images.

Biologists often need to handle numerous video-based home-cage animal behavior analysis tasks that require massive workloads. Therefore, we develop an AI-based multi-species tracking and segmentation system, SiamBOMB, for real-time and automatic home-cage animal behavioral analysis. In this system, a background-enhanced Siamese-based network with replaceable modular design ensures the flexibility and generalizability of the system, and a user-friendly interface makes it convenient to use for biologists. This real-time AI system will effectively reduce the burden on biologists.

Many effective image deformation methods have been proposed in recent years, but few of them use the contours of objects as the reference of deformation. However, the contour is an important factor, and contour‐based deformation can guarantee that the edge of the object after deformation is smoother compared to point‐based image deformation. This article presents an image deformation method based on contours using Moving Integral Least Squares (MILS) optimisation. First, the authors set the key points in the image to create control contours as required and then adjust the positions of these points to generate the desired contours. In order to warp these contours to their new positions, the image is deformed using MILS. They derive the affine, similarity, and rigid transformations in a general framework, and users can choose different curves according to their needs. The proposed method possesses two characteristics: (i) it is able to create detail‐preserving and intuitive deformations; and (ii) the solution of the deformation function has a simple closed form. They compare their method to the state‐of‐the‐art algorithm, which is modelled by rigid transformation. Experimental results show that their deformation is more vivid.

Image registration of serial electron microscopy (EM) sections is a feasible way to reveal 3D structure of the biological tissue. However, the image registration proves difficult, as it is hard to find reliable correspondences between the adjacent sections and the section distortion may occur during the sample preparation. In this paper, we propose a non-linear image registration method for serial EM sections, which is composed of pairwise correspondences extraction, correspondences position adjustment and image warping. The proposed method is highly automatic, and retains the morphology of the original electron microscopic images as much as possible. We demonstrate that our method outperforms the state-of-the-art approaches on several datasets of serial EM sections images including a synthetic test case.

BACKGROUND: The locations and shapes of synapses are important in reconstructing connectomes and analyzing synaptic plasticity. However, current synapse detection and segmentation methods are still not adequate for accurately acquiring the synaptic connectivity, and they cannot effectively alleviate the burden of synapse validation. RESULTS: We propose a fully automated method that relies on deep learning to realize the 3D reconstruction of synapses in electron microscopy (EM) images. The proposed method consists of three main parts: (1) training and employing the faster region convolutional neural networks (R-CNN) algorithm to detect synapses, (2) using the z-continuity of synapses to reduce false positives, and (3) combining the Dijkstra algorithm with the GrabCut algorithm to obtain the segmentation of synaptic clefts. Experimental results were validated by manual tracking, and the effectiveness of our proposed method was demonstrated. The experimental results in anisotropic and isotropic EM volumes demonstrate the effectiveness of our algorithm, and the average precision of our detection (92.8% in anisotropy, 93.5% in isotropy) and segmentation (88.6% in anisotropy, 93.0% in isotropy) suggests that our method achieves state-of-the-art results. CONCLUSIONS: Our fully automated approach contributes to the development of neuroscience, providing neurologists with a rapid approach for obtaining rich synaptic statistics.