Diffusion models have achieved state-of-the-art results on many modalities including images, speech, and video. However, existing models are not tailored to support remote sensing data, which is widely used in important applications including environmental monitoring and crop-yield prediction. Satellite images are significantly different from natural images -- they can be multi-spectral, irregularly sampled across time -- and existing diffusion models trained on images from the Web do not support them. Furthermore, remote sensing data is inherently spatio-temporal, requiring conditional generation tasks not supported by traditional methods based on captions or images. In this paper, we present DiffusionSat, to date the largest generative foundation model trained on a collection of publicly available large, high-resolution remote sensing datasets . As text-based captions are sparsely available for satellite images, we incorporate the associated metadata such as geolocation as conditioning information. Our method produces realistic samples and can be used to solve multiple generative tasks including temporal generation, multi-spectral superrresolution and in-painting. Our method outperforms previous state-of-the-art methods for satellite image generation and is the first large-scale _generative_ foundation model for satellite imagery. The project website can be found here: https://samar-khanna.github.io/DiffusionSat/

The limbus is the junction between the cornea and the sclera and harbors adult stem cells involved in corneal epithelial cells renewal, namely the limbal stem cells (LSCs). Damages to the limbus and/or the LSCs will lead to a loss of the limbal functions, defining limbal stem cell deficiency (LSCD). Clinically, LSCD presents with conjunctival cell invasion into the cornea, corneal scarring, neovascularization, and potential blindness. Recent guidelines have clarified the staging of the disease based on the area involved on the cornea and the use of additional imaging such as in vivo confocal microscopy (IVCM). Several biomarkers of LSC function are evaluated by IVCM on the central cornea, including the basal cell density and morphology, and subbasal nerve density. Evaluation of these biomarkers is time-consuming and remains subjective. Therefore, using a deep learning approach could enhance our diagnosis strategy and efficiency. The current paper demonstrates the performance of a deep learning-based pipeline to first identify clinically relevant images from IVCM volume scans and subsequently use them to classify LSCD severity based on a published grading system. The classification model achieved a test-time accuracy of 74%, and areas under the receiver operating characteristic curve (AUROC) of 0.93, 0.93, 0.94, and 0.87 for control, mild, moderate, and severe, respectively. In order to demonstrate the need to use both cell and nerve images, we also compare models that only use cell scans (68% accuracy) and only nerve scans (69% accuracy). This pilot study shows that the diagnosis of LSCD could be automated to improve diagnostic efficiency and to reduce inter- and intraclinician variability of the process.

Eric Mitchell, Joseph Noh, Siyan Li, Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, Christopher Manning. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Unsupervised pre-training methods for large vision models have shown to enhance performance on downstream supervised tasks. Developing similar techniques for satellite imagery presents significant opportunities as unlabelled data is plentiful and the inherent temporal and multi-spectral structure provides avenues to further improve existing pre-training strategies. In this paper, we present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE). To leverage temporal information, we include a temporal embedding along with independently masking image patches across time. In addition, we demonstrate that encoding multi-spectral data as groups of bands with distinct spectral positional encodings is beneficial. Our approach yields strong improvements over previous state-of-the-art techniques, both in terms of supervised learning performance on benchmark datasets (up to $\uparrow$ 7%), and transfer learning performance on downstream remote sensing tasks, including land cover classification (up to $\uparrow$ 14%) and semantic segmentation. Code and data are available on the project website: https://sustainlab-group.github.io/SatMAE/