Spectral Reflectance Indices (SRIs) have emerged as a powerful tool for unraveling and deciphering critical information about land cover, vegetation health, water quality, soil characteristics, etc. This approach provides valuable insights for sustainable resource management, agricultural optimizations, etc. However, vast list of available spectral indices makes it challenging to choose most relevant ones for a specific application. Identification of relevant SRIs for an application needs domain experts to analyze the information obtained from indices. Existing studies have used SRIs in a limited way - 1) as a single value for a location (county) and thus suffer from loss of spatial information; 2) only few popular spectral indices e.g. NDVI, SAVI, NDWI, etc. are exploited; 3) have been used for applications like crop yield and snow cover prediction. To circumvent these limitations, we embarked on three novel ideas - 1) compute spectral indices at pixel level of the satellite image resulting in an SRI image for a region; 2) introduce a selection method which selects SRIs relevant to a given application; 3) design a model called Spectral Indices Network, SpInN, utilizing two transformers ViViT and BERT. We use ViViT innovatively on different SRI images treating them as video frames for spatio-spectral index learning. We also use BERT to exploit temporal patterns of the spectral index time series. The proposed model is applied to various prediction and classification earth observation applications such as prediction of crop yield, solar energy, soil moisture, land cover classification, etc., and obtained state-of-the-art results.

Satellite data bolstered by their increasing accessibility is leading to many endeavors of automated monitoring of the earth's surface for various applications. Such applications demand high spatial resolution images at a temporal resolution of a few days which entails the challenge of processing a huge volume of image time series data. To overcome this computing bottleneck, we present PatchNet, a bespoke adaptation of beam search and attention mechanism. PatchNet is an automated patch selection neural network that requires only a partial spatial traversal of an image time series and yet achieves impressive results. Satellite systems face a trade-off between spatial and temporal resolutions due to budget/technical constraints e.g., Landsat-8/9 or Sentinel-2 have high spatial resolution whereas, MODIS has high temporal resolution. To deal with the limitation of coarse temporal resolution, we propose FuSITSNet, a twofold feature-based generic fusion model with multimodal learning in a contrastive setting. It produces a learned representation after fusion of two satellite image time series leveraging finer spatial resolution of Landsat and finer temporal resolution of MODIS. The patch alignment module of FuSITSNet aligns the PatchNet processed patches of Landsat-8 with the corresponding MODIS regions to incorporate its finer resolution temporal features. The untraversed patches are handled by the cross-modality attention which highlights additional hot spot features from the two modalities. We conduct extensive experiments on more than 2000 counties of US for crop yield, snow cover, and solar energy prediction and show that even one-fourth spatial processing of image time series produces state-of-the-art results. FuSITSNet outperforms the predictions of single modality and data obtained using existing generative fusion models and allows for monitoring of dynamic phenomena using freely accessible images, thereby unlocking new opportunities.

Satellite data provides valuable insights into environmental changes and natural resource management, such as monitoring deforestation, mapping land use changes, and identifying areas at risk of soil degradation. Landsat-8 and Sentine1-2 are the publicly available high spatial resolution satellites launched in recent years. But, both have a moderate temporal resolution which limits their use in the applications like precision agriculture, land cover mapping, disaster monitoring, etc. For such applications, daily or weekly monitoring is better suited. Fusing data from the two satellites can provide enhanced observations. Both Landsat-8 and Sentine1-2 satellites have the same geographic coordinate systems which makes them amiable for fusion. But, fusing data at the pixel level for these satellites is challenging as they visit the same location on different days. The proposed model `LSFuseNet’ effectively fuses data at the feature level. It is a dual-fusion model in which bi-directional cross-modal attention is used to identify and exchange the hotspot information in the two modalities. A feature alignment module learns the fine-grained features and mitigates the noise in the data. We have innovatively applied contrastive learning to improve the quality of the learned representations of the data from the two satellites. We evaluate our model for two applications - crop yield prediction and snow cover prediction. For crop yield prediction, we have taken two crops, viz. corn, and soybean, for approximately 500 counties in the US. For snow cover prediction, we considered approximately 1300 US counties. Our extensive experiments show that LSFuseNet outperforms competing models. Also, the benefit of fusing the data from two satellites over using the data from a single satellite is evident from the results of both applications. We have further modified the model to include meteorological and/or soil data (if applicable) to further enhance the performance of the model.

Early crop yield prediction is crucial in agriculture for making administrative plans to ensure food security, post harvest management and distribution of a crop. Remote sensing data captured using various satellites provide reliable phenological information for a crop through surface reflectance bands. Other important factors, affecting crop yield include meteorological and soil. The data which we have used for crop yield prediction is multimodal. It consists of spatiotemporal meteorological (numeric) and surface reflectance bands (satellite image), and temporally static soil (satellite image) data. We effectively utilize this multimodal data to develop the proposed multimodal deep learning model, CropYieldNet. The objective of the paper is to accurately predict crop yield using high resolution data obtained from recently launched satellites such as Landsat8 and Sentinel-2. We used contrastive learning in a supervised setting and data augmentation techniques to overcome the limited historical data available for training deep learning models.We introduce a depth-level selection module for effectively modelling the depth-variant information of soil data. We have also modified our model to perform in-season (early) crop yield prediction which is as accurate as end-season prediction. We evaluate our model for two crops, corn and soybean, on counties in US and districts in India using data from MODIS, Landsat8, and Sentinel-2 satellites. Our extensive experimentation show that our model outperforms competing models. Our experiments also show that CropYieldNet generalizes well when applied on different crops and geographies.