Cryo-EM in single particle analysis is known to have low SNR and requires to utilize several frames of the same particle sample to restore one high-quality image for visualizing that particle. However, the low SNR of cryo-EM movie and motion caused by beam striking make the task very challenging. Video enhancement algorithms in computer vision shed new light on tackling such tasks by utilizing deep neural networks. However, they are designed for natural images with high SNR. Meanwhile, the lack of ground truth in cryo-EM movie seems to be one major limiting factor of the progress. Hence, we present a synthetic cryo-EM movie generation pipeline, which can produce realistic diverse cryo-EM movie datasets with low-SNR movie frames and multiple ground truth values. Then we propose a deep spatio-temporal network (DST-Net) for cryo-EM movie frame enhancement trained on our synthetic data. Spatial and temporal features are first extracted from each frame. Spatio-temporal fusion and high-resolution re-constructor are designed to obtain the enhanced output. For evaluation, we train our model on seven synthetic cryo-EM movie datasets and infer on real cryo-EM data. The experimental results show that DST-Net can achieve better enhancement performance both quantitatively and qualitatively compared with others.

Cryo-EM movie in single particle analysis has extremely low SNR and requires aligning multiple frames to achieve signal enhancement. Currently, signal processing technique is adopted to estimate the motion vector between a pair of cryo-EM movie frames at patch-level, and the estimated motion vector is used as the reference for frame alignment, whose accuracy will determine the resolution of the reconstructed 3D structure of the particle. The patch-level motion may not well represent the beam-induced motion of particles since particles in a patch move towards different directions due to beam striking. However, the low SNR of cryo-EM movie makes it difficult to estimate the motion of particles at pixel-level. Meanwhile, existing optical flow estimation models only consider the ideal case where high-quality videos are provided, which fail to obtain optical flow from cryo-EM movie. In this paper, we diminish this limitation by proposing a model called NoiseFlow, a deep learning network for optical flow estimation from low SNR cryo-EM movie. NoiseFlow makes use of the multi-frame stacking module and the denoising module to extract noise-invariant features, and then computes the correlation volume from noise-invariant features to learn optical flow. For evaluation, we train our model on two synthetic cryo-EM movie datasets and infer on real cryo-EM data. The experimental results illustrate that NoiseFlow achieves state-of-the-art performance on both synthetic and real cryo-EM datasets.

Metastatic involvement of lymph nodes is one of the most important prognostic variables for many cancers. Several deep learning based algorithms have been developed to segment metastatic regions in pathological images to help predict prognosis. However, the training of these methods requires a large amount of annotated data, and the labeling task is an extremely time-consuming process for human annotators. In order to reduce the annotation burden, we for the first time propose a semi-automatic annotation method (PiPo-Net) for the labeling of pathological images. The method is comprised of two subnetworks, a pixel-wise segmentation network (Pi-Net) and a polygon-based annotation network (Po-Net). The Pi-Net adopts an improved encoder-decoder architecture and can effectively aggregate multi-scale image features. The Po-Net is built on the Pi-Net and leverages a two-layer recurrent neural network to generate tight-bounded polygons for the metastatic regions. Corresponding to the proposed network architecture, a loss function called PiPo-loss is introduced to help optimize the whole network. The main advantage of our method is that it integrates human annotators into the prediction loop, allowing to iteratively refine the predictions according to the suggestions from human annotators. We evaluate our method on Camelyon16 database and achieve a Dice score of 91% in the initial annotation attempt. We also demonstrate the effectiveness of the human-network collaborative annotation, which achieves promising labeling results, verifying the advantages of our proposed method.

Multiple instance learning (MIL) is a typical weakly-supervised learning method where the label is associated with a bag of instances instead of a single instance. Despite extensive research over past years, effectively deploying MIL remains an open and challenging problem, especially when the commonly assumed standard multiple instance (SMI) assumption is not satisfied. In this paper, we propose a multiple instance learning method based on deep graph convolutional network and feature selection (FS-GCN-MIL) for histopathological image classification. The proposed method consists of three components, including instance-level feature extraction, instance-level feature selection, and bag-level classification. We develop a self-supervised learning mechanism to train the feature extractor based on a combination model of variational autoencoder and generative adversarial network (VAE-GAN). Additionally, we propose a novel instance-level feature selection method to select the discriminative instance features. Furthermore, we employ a graph convolutional network (GCN) for learning the bag-level representation and then performing the classification. We apply the proposed method in the prediction of lymph node metastasis using histopathological images of colorectal cancer. Experimental results demonstrate that the proposed method achieves superior performance compared to the state-of-the-art methods.