CT image reconstruction is a challenging inverse problem of estimating image intensities from sensor signals. While deep learning-based methods hold promise, they are typically classified into fully-supervised, self-supervised, and case-by-case approaches, each offering distinct advantages and limitations. Among these, case-by-case reconstruction techniques, exemplified by models like NeRP, provide the advantage of flexible data-sampling and image sizes, making them well-suited for the complex CT inverse problem. However, existing case-by-case models encounter difficulties in achieving precise reconstructions, particularly in sparse-view scenarios. To address this limitation, we propose tomo-NeRL, a novel implicit reconstruction model that incorporates tomographic information of individual pixels within the reconstruction network. By harnessing this tomo-graphic data for implicit inverse learning, tomo-NeRL enhances reconstruction quality while effectively mitigating artifacts. The core innovation of tomo-NeRL lies in defining and extracting tomographic insights, such as visual characteristics and spatial variations specific to tomographic imaging, from the sinogram data for individual pixels. Moreover, our work introduces an adaptive inversion framework that amalgamates tomographic information from diverse projection angles, constructing a latent feature space that enhances the reconstruction process. Extensive experiments unequivocally showcase tomo-NeRL’s outstanding reconstruction performance, excelling in artifact suppression and detail preservation, especially in sparse-view scenarios.

MOTIVATION: Intracellular organelle networks (IONs) such as the endoplasmic reticulum (ER) network and the mitochondrial (MITO) network serve crucial physiological functions. The morphology of these networks plays a critical role in mediating their functions. Accurate image segmentation is required for analyzing the morphology and topology of these networks for applications such as molecular mechanism analysis and drug target screening. So far, however, progress has been hindered by their structural complexity and density. RESULTS: In this study, we first establish a rigorous performance baseline for accurate segmentation of these organelle networks from fluorescence microscopy images by optimizing a baseline U-Net model. We then develop the multi-resolution encoder (MRE) and the hierarchical fusion loss (Lhf) based on two inductive components, namely low-level features and topological self-similarity, to assist the model in better adapting to the task of segmenting IONs. Empowered by MRE and Lhf, both U-Net and Pyramid Vision Transformer (PVT) outperform competing state-of-the-art models such as U-Net++, HR-Net, nnU-Net, and TransUNet on custom datasets of the ER network and the MITO network, as well as on public datasets of another biological network, the retinal blood vessel network. In addition, integrating MRE and Lhf with models such as HR-Net and TransUNet also enhances their segmentation performance. These experimental results confirm the generalization capability and potential of our approach. Furthermore, accurate segmentation of the ER network enables analysis that provides novel insights into its dynamic morphological and topological properties. AVAILABILITY AND IMPLEMENTATION: Code and data are openly accessible at https://github.com/cbmi-group/MRE.

How deep neural networks (DNNs) learn from noisy labels has been studied extensively in image classification but much less in image segmentation. So far, our understanding of the learning behavior of DNNs trained by noisy segmentation labels remains limited. In this study, we address this deficiency in both binary segmentation of biological microscopy images and multi-class segmentation of natural images. We generate extremely noisy labels by randomly sampling a small fraction (e.g., 10%) or flipping a large fraction (e.g., 90%) of the ground truth labels. When trained with these noisy labels, DNNs provide largely the same segmentation performance as trained by the original ground truth. This indicates that DNNs learn structures hidden in labels rather than pixel-level labels per se in their supervised training for semantic segmentation. We refer to these hidden structures in labels as meta-structures. When DNNs are trained by labels with different perturbations to the meta-structure, we find consistent degradation in their segmentation performance. In contrast, incorporation of meta-structure information substantially improves performance of an unsupervised segmentation model developed for binary semantic segmentation. We define meta-structures mathematically as spatial density distributions and show both theoretically and experimentally how this formulation explains key observed learning behavior of DNNs.

This paper focuses on the semi-supervised object detection (SSOD) which makes good use of unlabeled data to boost performance. We face the following obstacles when adapting the knowledge distillation (KD) framework in SSOD. (1) The teacher model serves a dual role as a teacher and a student, such that the teacher predictions on unlabeled images may limit the upper bound of the student. (2) The data imbalance issue caused by the large quantity of consistent predictions between the teacher and student hinders an efficient knowledge transfer between them. To mitigate these issues, we propose a novel SSOD model called Temporal Self-Ensembling Teacher (TSET). Our teacher model ensembles its temporal predictions for unlabeled images under stochastic perturbations. Then, our teacher model ensembles its model weights with those of the student model by an exponential moving average. These ensembling strategies ensure data and model diversity, and lead to better teacher predictions for unlabeled images. In addition, we adapt the focal loss to formulate the consistency loss for handling the data imbalance issue. Together with a thresholding method, the focal loss automatically reweights the inconsistent predictions, which preserves the knowledge for difficult objects to detect in the unlabeled images. The mAP of our model reaches 80.73% and 40.52% on the VOC2007 test set and the COCO2014minival5kset, respectively, and outperforms a strong fully supervised detector by 2.37% and 1.49%, respectively. Furthermore, the mAP of our model (80.73%) sets a new state-of-the-art performance in SSOD on the VOC2007 test set.

Objects are often organized in a semantic hierarchy of categories, where fine-level categories are grouped into coarse-level categories according to their semantic relations. While previous works usually only classify objects into the leaf categories, we argue that generating hierarchical labels can actually describe how the leaf categories evolved from higher level coarse-grained categories, thus can provide a better understanding of the objects. In this paper, we propose to utilize the CNN-RNN framework to address the hierarchical image classification task. CNN allows us to obtain discriminative features for the input images, and RNN enables us to jointly optimize the classification of coarse and fine labels. This framework can not only generate hierarchical labels for images, but also improve the traditional leaf-level classification performance due to incorporating the hierarchical information. Moreover, this framework can be built on top of any CNN architecture which is primarily designed for leaf-level classification. Accordingly, we build a high performance network based on the CNN-RNN paradigm which outperforms the original CNN (wider-ResNet) and also the current state-of-the-art. In addition, we investigate how to utilize the CNN-RNN framework to improve the fine category classification when a fraction of the training data is only annotated with coarse labels. Experimental results demonstrate that CNN-RNN can use the coarse-labeled training data to improve the classification of fine categories, and in some cases it even surpasses the performance achieved by fully annotated training data. This reveals that, CNN-RNN can alleviate the challenge of specialized and expensive annotation of fine labels.

Accurate segmentation of zebrafish from bright-field microscope images is crucial to many applications in the life sciences. Early zebrafish stages are used, and in these stages the zebrafish is partially transparent. This transparency leads to edge ambiguity as is typically seen in the larval stages. Therefore, segmentation of zebrafish objects from images is a challenging task in computational bio-imaging. Popular computational methods fail to segment the relevant edges, which subsequently results in inaccurate measurements and evaluations. Here we present a hybrid method to accomplish accurate and efficient segmentation of zebrafish specimens from bright-field microscope images. We employ the mean shift algorithm to augment the colour representation in the images. This improves the discrimination of the specimen to the background and provides a segmentation candidate retaining the overall shape of the zebrafish. A distance-regularised level set function is initialised from this segmentation candidate and fed to an improved level set method, such that we can obtain another segmentation candidate which preserves the explicit contour of the object. The two candidates are fused using heuristics, and the hybrid result is refined to represent the contour of the zebrafish specimen. We have applied the proposed method on two typical datasets. From experiments, we conclude that the proposed hybrid method improves both efficiency and accuracy of the segmentation of the zebrafish specimen. The results are going to be used for high-throughput applications with zebrafish.

In life sciences, light microscopy is used to study specimens. On the organism-level a bright-field representation present an overview for the whole shape of a specimen; the organ-level fluorescent staining representation supports in the interpretation of the detailed intrinsic structures. We present light microscopy axial-view imaging based on the Vertebrate Automated Screening Technology to acquire axial-view images for the organism and organs of zebrafish larvae. We obtain multi-modal 3D reconstruction using a profile-based method, from which we can derive the 3D measurements of volume and surface area. In this method, we employ a microscope camera calibration using voxel residual volume maximization algorithm. We intuitively align and fuse the obtained multi-models. Experimental results show natural visualization both for the whole organism and organ of zebrafish larvae; subsequently accurate 3D measurements are obtained. This method is very suitable for high-throughput research in which knowledge on size and shape is relevant to the understanding for development, effects of compounds or drugs.

In order to preserve sufficient fluorescence intensity and improve the quality of fluorescence images in optical projection tomography (OPT) imaging, a feasible acquisition solution is to temporally formalize the fluorescence and bright-field imaging procedure as two consecutive phases. To be specific, fluorescence images are acquired first, in a full axial-view revolution, followed by the bright-field images. Due to the mechanical drift, this approach, however, may suffer from a deviation of center of rotation (COR) for the two imaging phases, resulting in irregular 3D image fusion, with which gene or protein activity may be located inaccurately. In this paper, we address this problem and consider it into a framework based on sinogram unification so as to precisely fuse 3D images from different channels for CORs between channels that are not coincident or if COR is not in the center of sinogram. The former case corresponds to the COR deviation above; while the latter one correlates with COR alignment, without which artefacts will be introduced in the reconstructed results. After sinogram unification, inverse radon transform can be implemented on each channel to reconstruct the 3D image. The fusion results are acquired by mapping the 3D images from different channels into a common space. Experimental results indicate that the proposed framework gains excellent performance in 3D image fusion from different channels. For the COR alignment, a new automated method based on interest point detection and included in sinogram unification, is presented. It outperforms traditional COR alignment approaches in combination of effectiveness and computational complexity.

Image-based kinship recognition is an important problem in the reconstruction and analysis of social networks. Prior studies on image-based kinship recognition have focused solely on pair wise kinship verification, i.e. on the question of whether or not two people are kin. Such approaches fail to exploit the fact that many real-world photographs contain several family members, for instance, the probability of two people being brothers increases when both people are recognized to have the same father. In this work, we propose a graph-based approach that incorporates facial similarities between all family members in a photograph in order to improve the performance of kinship recognition. In addition, we introduce a database of group photographs with kinship annotations.