Mapping Greenland's subglacial topography is critical for projecting the future mass loss of the ice sheet and its contribution to global sea-level rise. However, the complex and sparse nature of observational data, particularly information about the bed topography under the ice sheet, significantly increases the uncertainty in model projections. Bed topography is traditionally measured by airborne ice-penetrating radars that measure the ice thickness directly underneath the aircraft, leaving data gaps of tens of kilometers in between flight lines. This study introduces a deep learning framework, DeepTopoNet, that integrates radar-derived ice thickness observations and BedMachine Greenland data through a novel dynamic loss-balancing mechanism. Among all efforts to reconstruct bed topography, BedMachine has emerged as one of the most widely used datasets, combining mass conservation principles and ice thickness measurements to generate high-resolution bed elevation estimates. The proposed loss function adaptively adjusts the weighting between radar and BedMachine's bed, ensuring robustness in areas with limited radar coverage while leveraging the high spatial resolution of BedMachine's bed estimates. Our approach incorporates gradient-based and trend surface features to enhance model performance and utilizes a convolutional neural network (CNN) architecture (i.e., BedTopoCNN) designed for subgrid-scale predictions. By systematically testing on the Upernavik Isstrøm) region in West Greenland, the model achieves high accuracy (MAE: 12.49 m, RMSE: 19.38 m, and R2: 0.99), outperforming baseline methods in reconstructing subglacial terrain. This work demonstrates the potential of deep learning in bridging observational gaps, providing a scalable and efficient solution to inferring subglacial topography. This framework paves the way for improved predictions of ice sheet flow and sea level rise.

Sea ice extent plays a crucial role in the Arctic system, and thus the study on causal relationships between sea ice extent and other climate variables comes to our interest to better understand the system. To find the causal relationship we applied various state-of-the-art causal discovery techniques from the time-independent and time-dependent domains. Then we employed several causal inference models to quantify the causal effects of different causal relationships in the Arctic system. The NSIDC Sea Ice Concentration observation data and the ERA-5 global reanalysis data were used in our study. Our analysis shows that the GES and VarLiNGAM from causal discovery methods and the conditional instrumental variable (CIV) causal inference model perform better on the Arctic Sea Ice time series dataset.

Ocean eddies play a significant role both at the sea surface and beneath it, contributing to the sustainability of marine ecosystems and influencing broader oceanic and climatic behaviors. Investigating ocean eddies is essential for monitoring changes in the Earth’s oceans and their impact on climate. This study focuses on benchmarking the performance of state-of-theart YOLO (You Only Look Once) models for locating small-scale (<20km) ocean eddies using satellite remote sensing images. We leverage AWS SageMaker for this evaluation, utilizing both single and multi-GPU configurations to explore the feasibility and efficiency of deploying AI applications in cloud-based environments. This research not only assesses the effectiveness of SageMaker in handling complex Earth science data but also provides insights into deployment challenges, resource management for large-scale data, and the overall user experience. The findings highlight the strengths and limitations of using SageMaker for remote sensing applications and suggest potential future research directions. Our code is open-sourced at https://shorturl.at/hcjmq.

The increasing complexity of multidimensional spatiotemporal data presents significant challenges for clustering techniques, particularly in capturing intricate temporal, spatial, and heterogeneous patterns. This paper proposes a novel Hybrid Ensemble Deep Graph Temporal Clustering (HEDGTC) algorithm that integrates homogeneous and heterogeneous ensemble clustering models, leveraging both traditional and deep learning-based clustering approaches. The algorithm utilizes graph neural networks (GNNs) to effectively combine the strengths of multiple clustering models and enhance the clustering performance. The ensemble models are designed to handle diverse data characteristics, while the deep learning components capture complex non-linear relationships within the data. GNNs are employed to derive the final clustering outcomes by preserving spatial and temporal dependencies, making the approach well-suited for complex multidimensional spatiotemporal data. Experimental results from three real-world multivariate spatiotemporal data demonstrate the effectiveness of HEDGTC in accurately clustering and analyzing spatiotemporal patterns, outperforming state of the art ensemble models as well as traditional and individual deep clustering methods in terms of clustering performance and accuracy. The proposed method offers a robust framework for a wide range of applications, including climate modeling, geospatial analysis, and dynamic system forecasting.

Accurate quantitative precipitation estimation (QPE) is essential for managing water resources, monitoring flash floods, creating hydrological models, and more. Traditional methods of obtaining precipitation data from rain gauges and radars have limitations such as sparse coverage and inaccurate estimates for different precipitation types and intensities. Symbolic regression, a machine learning method that generates mathematical equations fitting the data, presents a unique approach to estimating precipitation that is both accurate and interpretable. Using WSR-88D dual-polarimetric radar data from Oklahoma and Florida over three dates, we tested symbolic regression models involving genetic programming and deep learning, symbolic regression on separate clusters of the data, and the incorporation of knowledge-based loss terms into the loss function. We found that symbolic regression is both accurate in estimating rainfall and interpretable through learned equations. Accuracy and simplicity of the learned equations can be slightly improved by clustering the data based on select radar variables and by adjusting the loss function with knowledge-based loss terms. This research provides insights into improving QPE accuracy through interpretable symbolic regression methods.

Collocation of measurements from active and passive satellite sensors refers to the combination of data from two sensors that observe the same geographic area at nearly the same time but with differing spatial resolutions and viewing angles. This collocated data, often known as on-track data, comes with precise product labels from the active sensor but comprises only the pixels located directly on the path of an active satellite’s orbit. As a result, its spatial coverage is quite limited, especially when compared to the vast quantities of off-track data. Handling the abundant and information-dense off-track data is crucial for training machine learning models that can effectively integrate the unique features of this data along with on-track data. However, the sheer volume of off-track data presents significant challenges for these models. To address the challenges of large amounts of unlabeled off-track data in remote sensing applications, we introduce a self-supervised representation learning model with VAE and domain adaptation methods to learn a domain invariant classifier for the on-track and off-track data. The model’s performance is enhanced by pre-training off-track data with VAE generative model using off-track data, to learn a good representation that can be transferred to the down-streaming domain adaptation and classification tasks. The classifier is built on these representations to classify different cloud types in passive sensing data, with the goal of achieving higher accuracy in cloud property retrieval. Extensive quantitative and qualitative evaluation demonstrate our method achieves higher accuracy in cloud property retrieval for off-track remote sensing data.

Arctic amplification has altered the climate patterns both regionally and globally, resulting in more frequent and more intense extreme weather events in the past few decades. The essential part of Arctic amplification is the unprecedented sea ice loss as demonstrated by satellite observations. Accurately forecasting Arctic sea ice from sub-seasonal to seasonal scales has been a major research question with fundamental challenges at play. In addition to physics-based Earth system models, researchers have been applying multiple statistical and machine learning models for sea ice forecasting. Looking at the potential of data-driven approaches to study sea ice variations, we propose MT-IceNet – a UNet-based spatial and multi-temporal (MT) deep learning model for forecasting Arctic sea ice concentration (SIC). The model uses an encoder-decoder architecture with skip connections and processes multi-temporal input streams to regenerate spatial maps at future timesteps. Using bi-monthly and monthly satellite retrieved sea ice data from NSIDC as well as atmospheric and oceanic variables from ERA5 reanalysis product during 1979-2021, we show that our proposed model provides promising predictive performance for per-pixel SIC forecasting with up to 60% decrease in prediction error for a lead time of 6 months as compared to its state-of-the-art counterparts.

Atmospheric gravity waves are produced when gravity attempts to restore disturbances through stable layers in the atmosphere. They have a visible effect on many atmospheric phenomena such as global circulation and air turbulence. Despite their importance, however, little research has been conducted on how to detect gravity waves using machine learning algorithms. We faced two major challenges in our research: our raw data had a lot of noise and the labeled dataset was extremely small. In this study, we explored various methods of preprocessing and transfer learning in order to address those challenges. We pre-trained an autoencoder on unlabeled data before training it to classify labeled data. We also created a custom CNN by combining certain pre-trained layers from the InceptionV3 Model trained on ImageNet with custom layers and a custom learning rate scheduler. Experiments show that our best model outperformed the best performing baseline model by 6.36% in terms of test accuracy.

Spatial resolution is critical for observing and monitoring environmental phenomena. Acquiring high-resolution bathymetry data directly from satellites is not always feasible due to limitations on equipment, so spatial data scientists and researchers turn to single image super-resolution (SISR) methods that utilize deep learning techniques as an alternative method to increase pixel density. While super resolution residual networks (e.g., SR-ResNet) are promising for this purpose, several challenges still need to be addressed: (1) Earth data such as bathymetry is expensive to obtain and relatively limited in its data record amount; (2) certain domain knowledge needs to be complied with during model training; (3) certain areas of interest require more accurate measurements than other areas. To address these challenges, following the transfer learning principle, we study how to leverage an existing pre-trained super-resolution deep learning model, namely SR-ResNet, for high-resolution bathymetry data generation. We further enhance the SR-ResNet model to add corresponding loss functions based on domain knowledge. To let the model perform better for certain spatial areas, we add additional loss functions to increase the penalty of the areas of interest. Our experiments show our approaches achieve higher accuracy than most baseline models when evaluating using metrics including MSE, PSNR, and SSIM.

Domain adaptation techniques using deep neural networks have been mainly used to solve the distribution shift problem in homogeneous domains where data usually share similar feature spaces and have the same dimensionalities. Nevertheless, real world applications often deal with heterogeneous domains that come from completely different feature spaces with different dimensionalities. In our remote sensing application, two remote sensing datasets collected by an active sensor and a passive one are heterogeneous. In particular, CALIOP actively measures each atmospheric column. In this study, 25 measured variables/features that are sensitive to cloud phase are used and they are fully labeled. VIIRS is an imaging radiometer, which collects radiometric measurements of the surface and atmosphere in the visible and infrared bands. Recent studies have shown that passive sensors may have difficulties in prediction cloud/aerosol types in complicated atmospheres (e.g., overlapping cloud and aerosol layers, cloud over snow/ice surface, etc.). To overcome the challenge of the cloud property retrieval in passive sensor, we develop a novel VAE based approach to learn domain invariant representation that capture the spatial pattern from multiple satellite remote sensing data (VDAM), to build a domain invariant cloud property retrieval method to accurately classify different cloud types (labels) in the passive sensing dataset. We further exploit the weight based alignment method on the label space to learn a powerful domain adaptation technique that is pertinent to the remote sensing application. Experiments demonstrate our method outperforms other state-of-the-art machine learning methods and achieves higher accuracy in cloud property retrieval in the passive satellite dataset.

Critical natural resources and processes in the Arctic depend heavily on sea ice. Thus, accurate and timely predictions of Arctic sea ice changes is important. Arctic sea ice forecasting involves two connected tasks: predicting sea ice concentration (SIC) at each pixel and predicting overall sea ice extent (SIE). Instead of having two separate models for these two forecasting tasks, in this paper we study how to use multi-task learning techniques and leverage the connections between ice concentration and ice extent to improve accuracy for both forecasting tasks. Because of the spatiotemporal nature of the data, we designed two novel multi-task learning models based on the CNN and ConvLSTM, respectively. Further, in conjunction with multi-task models, we developed custom loss functions which train the models to ignore land pixels and optimize for both concentration and extent when making predictions. Our experiments show that multi-task models provide better accuracy for a 1-month lead time than models that predict sea ice extent and concentration separately. Our accuracies are better than or comparable to results in related state-of-the-art studies. Our best model in SIC prediction outperformed the best existing SIC prediction model in the literature with 1.78% less error, and our best model in SIE prediction outperformed the best existing SIE prediction model with 0.283 million km2less error.

Data-driven causality discovery is a common way to understand causal relationships among different components of a system. We study how to achieve scalable data-driven causality discovery on Amazon Web Services (AWS) and Microsoft Azure cloud and propose a causality discovery as a service (CDaaS) framework. With this framework, users can easily re-run previous causality discovery experiments or run causality discovery with different setups (such as new datasets or causality discovery parameters). Our CDaaS leverages Cloud Container Registry service and Virtual Machine service to achieve scalable causality discovery with different discovery algorithms. We further did extensive experiments and benchmarking of our CDaaS to understand the effects of seven factors (big data engine parameter setting, virtual machine instance number, type, subtype, size, cloud service, cloud provider) and how to best provision cloud resources for our causality discovery service based on certain goals including execution time, budgetary cost and cost-performance ratio. We report our findings from the benchmarking, which can help obtain optimal configurations based on each application’s characteristics. The findings show proper configurations could lead to both faster execution time and less budgetary cost.

Domain adaptation techniques have been developed to handle data from multiple sources or domains. Most existing domain adaptation models assume that source and target domains are homogeneous, i.e., they have the same feature space. Nevertheless, many real world applications often deal with data from heterogeneous domains that come from completely different feature spaces. In our remote sensing application, data in source domain (from an active spaceborne Lidar sensor CALIOP onboard CALIPSO satellite) contain 25 attributes, while data in target domain (from a passive spectroradiometer sensor VIIRS onboard Suomi-NPP satellite) contain 20 different attributes. CALIOP has better representation capability and sensitivity to aerosol types and cloud phase, while VIIRS has wide swaths and better spatial coverage but has inherent weakness in differentiating atmospheric objects on different vertical levels. To address this mismatch of features across the domains/sensors, we propose a novel end-to-end deep domain adaptation with domain mapping and correlation alignment (DAMA) to align the heterogeneous source and target domains in active and passive satellite remote sensing data. It can learn domain invariant representation from source and target domains by transferring knowledge across these domains, and achieve additional performance improvement by incorporating weak label information into the model (DAMA-WL). Our experiments on a collocated CALIOP and VIIRS dataset show that DAMA and DAMA-WL can achieve higher classification accuracy in predicting cloud types.

The rise of the Internet of Things (IoT) devices and the streaming platform has tremendously increased the data in motion or streaming data. It incorporates a wide variety of data, for example, social media posts, online gamers in-game activities, mobile or web application logs, online e-commerce transactions, financial trading, or geospatial services. Accurate and efficient forecasting based on real-time data is a critical part of the operation in areas like energy & utility consumption, healthcare, industrial production, supply chain, weather forecasting, financial trading, agriculture, etc. Statistical time series forecasting methods like Autoregression (AR), Autoregressive integrated moving average (ARIMA), and Vector Autoregression (VAR), face the challenge of concept drift in the streaming data, i.e., the properties of the stream may change over time. Another challenge is the efficiency of the system to update the Machine Learning (ML) models which are based on these algorithms to tackle the concept drift. In this paper, we propose a novel framework to tackle both of these challenges. The challenge of adaptability is addressed by applying the Lambda architecture to forecast future state based on three approaches simultaneously: batch (historic) data-based prediction, streaming (real-time) data-based prediction, and hybrid prediction by combining the first two. To address the challenge of efficiency, we implement a distributed VAR algorithm on top of the Apache Spark big data platform. To evaluate our framework, we conducted experiments on streaming time series forecasting with four types of data sets of experiments: data without drift (no drift), data with gradual drift, data with abrupt drift and data with mixed drift. The experiments show the differences of our three forecasting approaches in terms of accuracy and adaptability.

Dust plumes originating from the Earth's major arid and semi-arid areas can significantly affect the climate system and human health. Many existing methods have been developed to identify dust from non-dust pixels from a remote sensing point of view. However, these methods use empirical rules and therefore have difficulty detecting dust above or below the detectable thresholds. Supervised machine learning methods have also been applied to detect dust from satellite imagery, but these methods are limited especially when applying to areas outside the training data due to the inadequate amount of ground truth data. In this work, we proposed an automatic dust segmentation framework using semi-supervised machine learning, based on a collocated dataset using Visible Infrared Imaging Radiometer Suite (VIIRS) and Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO). The proposed method utilizes unsupervised machine learning for segmentation of VIIRS imagery, and leverages the guidance from the dust labels using the dust profile product of CALIPSO to determine the dust clusters as the final product. The dust clusters are determined based on the similarity of spectral signature from dust pixels along the CALIPSO tracks. Experiment results show that the accuracy of the proposed framework outperforms the traditional physical infrared method along CALIPSO tracks. In addition, the proposed method performs consistently over three different study areas, the North Atlantic Ocean, East Asia, and Northern Africa.

Predicting violent storms and dangerous weather conditions with current models can take a long time due to the immense complexity associated with weather simulation. Machine learning has the potential to classify tornadic weather patterns much more rapidly, thus allowing for more timely alerts to the public. To deal with class imbalance challenges in machine learning, different data augmentation approaches have been proposed. In this work, we examine the wall time difference between live data augmentation methods versus the use of preaugmented data when they are used in a convolutional neural network based training for tornado prediction. We also compare CPU and GPU based training over varying sizes of augmented data sets. Additionally we examine what impact varying the number of GPUs used for training will produce given a convolutional neural network.

The discretisation of an attribute refers to partitioning its continuous numerical values into intervals, each of which is associated a categorical label. The amount of such different categorical labels is called as target discretisation level of the continuous attribute. For data mining algorithms that can only work on discrete data, the discretisation will be necessary. At the same time, the discretisation can also make the original data more concise and interpretable. However, it is challenging to balance the target discretisation level and the information loss during the discretisation process. In this paper, we propose to use entropy of a continuous attribute as a benchmark to determine its target discretisation level for the first time. An entropy based naive unsupervised discretisation approach is also proposed and shows big advantages in terms of both data reduction and accuracy, which is evaluated by performing classifiers on the dataset whose continuous attributes are discretised based on the proposed approach. Our experiments on 28 datasets and 9 popular classifiers show that the accuracy of a discretisation approach will be largely affected when the target discretisation level of each continuous attribute is lower than the entropy benchmark. Meanwhile increasing the target discretisation level from the benchmark does not always improve the accuracy of the discretizer. These discoveries can provide valuable guidance to explore or optimise the approaches to the discretisation of continuous attributes.

Granger causality and its learning algorithms have been widely used in many disciplines to study cause-effect relationship among time series variables. In this paper, we address computing challenges of state-of-art Granger causality learning algorithms, specially when facing increasing dimensionality of available datasets. We study how to leverage gradient boosting meta machine learning techniques to achieve accurate causality discovery and big data parallel techniques for efficient causality discovery from large temporal datasets. We propose two main algorithms for gradient boosting based causality learning, and parallel gradient boosting based causality learning. Our experiments show our proposed algorithms can achieve efficient learning in distributed environments with good learning accuracy.

In a research community, the provenance sharing of scientific workflows can enhance distributed research cooperation, experiment reproducibility verification and experiment repeatedly doing. Considering that scientists in such a community are often in a loose relation and distributed geographically, traditional centralized provenance sharing architectures have shown their disadvantages in poor trustworthiness, reliabilities and efficiency. Additionally, they are also difficult to protect the rights and interests of data providers. All these have been largely hindering the willings of distributed scientists to share their workflow provenance. Considering the big advantages of blockchain in decentralization, trustworthiness and high reliability, an approach to sharing scientific workflow provenance based on blockchain in a research community is proposed. To make the approach more practical, provenance is handled on-chain and original data is delivered off-chain. A kind of block structure to support efficient provenance storing and retrieving is designed, and an algorithm for scientists to search workflow segments from provenance as well as an algorithm for experiments backtracking are provided to enhance the experiment result sharing, save computing resource and time cost by avoiding repeated experiments as far as possible. Analyses show that the approach is efficient and effective.

A self-organizing map (SOM) is a type of competitive artificial neural network, which projects the high-dimensional input space of the training samples into a low-dimensional space with the topology relations preserved. This makes SOMs supportive of organizing and visualizing complex data sets and have been pervasively used among numerous disciplines with different applications. Notwithstanding its wide applications, the self-organizing map is perplexed by its inherent randomness, which produces dissimilar SOM patterns even when being trained on identical training samples with the same parameters every time, and thus causes usability concerns for other domain practitioners and precludes more potential users from exploring SOM based applications in a broader spectrum. Motivated by this practical concern, we propose a deterministic approach as a supplement to the standard self-organizing map. In accordance with the theoretical design, the experimental results with satellite cloud data demonstrate the effective and efficient organization as well as simplification capabilities of the proposed approach.

Training Back-Propagation Neural Networks (BPNNs) on big datasets faces two challenges, the hight time cost and the possibility of getting trapped into local optimum. MapReduce has been introduced to improve the efficiency of BPNN training on big datasets in recent years. After each turn of BPNN training on each split of the dataset concurrently, lots of local BPNNs that are only convergent on the specific split will be produced, and a global BPNN candidate convergent on the whole dataset needs to be generated from them. This process is full of challenges because it has a high impact on the training efficiency as well as the training accuracy. The paper introduces the evolution of the local BPNNs into the MapReduce training of BPNN, and proposes a novel approach. Profiting from the advantages of EAs in global optimum searching, the approach can reduce the iterations to get the global convergent BPNN candidate and avoid the training process to get trapped into local optimum. Experiments show the approach can improve the training efficiency and accuracy remarkably. The approach has also been applied into a real-world big data application and verified it can work well on big and high dimension datasets.

With advances of big data technologies, there are many possible ways to program for each big data application. A challenge is to know the differences of the program approaches and decide which programming approach is the best for a particular big data application. In this paper, we use vehicle travel companion as a case study to explore four different programming approaches, including Spark RDD (with GroupBy or Join), Spark SQL with Hive and Hive on Hadoop, and tune the programmed big data applications. Our experiments show that the execution time of one programming approach could be more than 100-fold longer than that of another for the same application logic, which verifies that programming approach decision is important. We also explain the reasons for the differences. The findings could be applied to the selection of programming approach for other big data applications.

As sensors spread across almost every industry, the Internet of Things (IoT) is going to trigger an era of big data. However, the abundance of available sensing data causes new challenges when building IoT applications. One main challenge is how to select proper data from large amount of sensing data for learning useful information efficiently. Existing approaches require developers to manage data for each specific application, which is very time consuming since the developers may not have enough knowledge about the dynamic changing data quality of different sensors. In this paper, we propose a data management middleware to learn the correlations between time series sensor data without prior knowledge. The learned correlation is then applied to select the useful sensor and reconstruct the incorrect data. To generalize the correlation models for each application, we utilize the dynamic feedback from the application to update the data selection and reconstruction. We evaluate our data management middleware in smart grids. The evaluation results show that our middleware can achieve better application performance with the help of dynamic feedback, data reconstruction and data selection.

Human posture recognition has a wide range of applications including elderly care and video surveillance. This paper discusses how to recognize human postures using wearable devices. From real-world data, we analyze the challenges in terms of result performance, recognition efficiency and sensor selection. To deal with the challenges, we present our design with five techniques: i) oversampling and undersampling methods, ii) ensemble learning, iii) sensor selection, iv) stream data classification and v) post-processing techniques. We verify our design and show our findings through extensive experiments on real-world data, which shows our approach can achieve up to 91.5% overall weighted average accuracy for all three postures. We also discuss possible extensions of our work.

Ability to track provenance is a key feature of scientific workflows to support data lineage and reproducibility. The challenges that are introduced by the volume, variety and velocity of Big Data, also pose related challenges for provenance and quality of Big Data, defined as veracity. The increasing size and variety of distributed Big Data provenance information bring new technical challenges and opportunities throughout the provenance lifecycle including recording, querying, sharing and utilization. This paper discusses the challenges and opportunities of Big Data provenance related to the veracity of the datasets themselves and the provenance of the analytical processes that analyze these datasets. It also explains our current efforts towards tracking and utilizing Big Data provenance using workflows as a programming model to analyze Big Data.