Segmentation and reconstruction of arteries is important for a variety of medical and engineering fields, such as surgical planning and physiological modeling. However, manual methods can be laborious and subject to a high degree of human variability. In this work, we developed various convolutional neural network ( CNN ) architectures to segment Stanford type B aortic dissections ( TBADs ), characterized by a tear in the descending aortic wall creating a normal channel of blood flow called a true lumen and a pathologic channel within the wall called a false lumen. We introduced several variations to the two-dimensional ( 2D ) and three-dimensional (3 D ) U-Net, where small stacks of slices were inputted into the networks instead of individual slices or whole geometries. We compared these variations with a variety of CNN segmentation architectures and found that stacking the input data slices in the upward direction with 2D U-Net improved segmentation accuracy, as measured by the Dice similarity coefficient ( DC ) and point-by-point average distance ( AVD ), by more than 15\% . Our optimal architecture produced DC scores of 0.94, 0.88, and 0.90 and AVD values of 0.074, 0.22, and 0.11 in the whole aorta, true lumen, and false lumen, respectively. Altogether, the predicted reconstructions closely matched manual reconstructions.

We present a feature-free photogrammetric technique that enables quantitative 3D mesoscopic (mm-scale height variation) imaging with tens-of-micron accuracy from sequences of images acquired by a smartphone at close range (several cm) under freehand motion without additional hardware. Our end-to-end, pixel-intensity-based approach jointly registers and stitches all the images by estimating a coaligned height map, which acts as a pixel-wise radial deformation field that orthorectifies each camera image to allow plane-plus-parallax registration. The height maps themselves are reparameterized as the output of an untrained encoder-decoder convolutional neural network (CNN) with the raw camera images as the input, which effectively removes many reconstruction artifacts. Our method also jointly estimates both the camera’s dynamic 6D pose and its distortion using a nonparametric model, the latter of which is especially important in mesoscopic applications when using cameras not designed for imaging at short working distances, such as smartphone cameras. We also propose strategies for reducing computation time and memory, applicable to other multi-frame registration problems. Finally, we demonstrate our method using sequences of multi-megapixel images captured by an un-stabilized smartphone on a variety of samples (e.g., painting brushstrokes, circuit board, seeds).

This paper introduces a new method of data-driven microscope design for virtual fluorescence microscopy. We use a deep neural network (DNN) to effectively design optical patterns for specimen illumination that substantially improve upon the ability to infer fluorescence image information from unstained microscope images. To achieve this design, we include an illumination model within the DNN’s first layers that is jointly optimized during network training. We validated our method on two different experimental setups, with different magnifications and sample types, to show a consistent improvement in performance as compared to conventional microscope imaging methods. Additionally, to understand the importance of learned illumination on the inference task, we varied the number of illumination patterns being optimized (and thus the number of unique images captured) and analyzed how the structure of the patterns changed as their number increased. This work demonstrates the power of programmable optical elements at enabling better machine learning algorithm performance and at providing physical insight into next generation of machine-controlled imaging systems.

Recent machine learning techniques have dramatically changed how we process digital images. However, the way in which we capture images is still largely driven by human intuition and experience. This restriction is in part due to the many available degrees of freedom that alter the image acquisition process (lens focus, exposure, filtering, etc). Here we focus on one such degree of freedom - illumination within a microscope - which can drastically alter information captured by the image sensor. We present a reinforcement learning system that adaptively explores optimal patterns to illuminate specimens for immediate classification. The agent uses a recurrent latent space to encode a large set of variably-illuminated samples and illumination patterns. We train our agent using a reward that balances classification confidence with image acquisition cost. By synthesizing knowledge over multiple snapshots, the agent can classify on the basis of all previous images with higher accuracy than from naively illuminated images, thus demonstrating a smarter way to physically capture task-specific information.

Historically, sea ice concentration (SIC) has been measured through the use of passive microwave sensors, as well as human interpretation of synthetic aperture radar (SAR). Although passive microwave data are processed automatically, it suffers from poor spatial resolution and the higher frequency channels are sensitive to weather conditions. Deep learning has demonstrated its ability to perform complex and accurate analysis of images; here, we apply deep learning to estimate ice concentration from SAR scenes. We developed a deep convolutional neural network (CNN) that predicts SIC from SAR, trained upon passive microwave data. The model achieves a 5.24%/7.87% error on its train and test set, respectively. To assess the real-world applicability, we performed an independent validation on 18 SAR scenes (from two distinct geographical regions), not previously seen during training or test. Comparing against human-generated ice analysis charts, we achieved an L1 error of 0.2059, competitive with passive microwave (EL1= 0.1863) for the Canadian Arctic Archipelago. For the Gulf of Saint Lawrence region, we achieved an L1 error of 0.2653, significantly better than the passive microwave result (EL1= 0.3593). By using novel techniques for model training, as well as training entirely upon passive microwave data, we present an accessible and robust method of developing similar systems for processing SAR.1Our results suggest that with further postprocessing, CNNs are accurate and robust enough to be used for operational tasks.