The Observational Products for End-Users from Remote Sensing Analysis (OPERA) project at the Jet Propulsion Laboratory (JPL) will provide a near-global Radiometric Terrain Corrected synthetic aperture radar (SAR) backscatter from Sentinel-1 (RTC-S1) product. The OPERA RTC-S1 product will deliver map-projected burst-based radar images with a geographic scope that includes all land masses excluding Antarctica, and with temporal sampling coincident with the availability of Sentinel-1 interferometric wide (IW) single-look complex (SLC) data. This paper presents the OPERA RTC-S1 product, providing details about its layers, static layers, and metadata; describing the product’s processing workflow, based on the ISCE3 framework and using the same algorithms developed for the NASA-ISRO Synthetic Aperture Radar (NISAR) mission; and outlining the algorithm verification and the product validation plan. We also present a global mosaic of preliminary OPERA RTC-S1 products generated from a global end-to-end test run from a Sentinel-1A orbit cycle. The OPERA RTC-S1 product will be publicly distributed through the Alaska Satellite Facility (ASF) Distributed Active Archive Center (DAAC) free of charge, with a release date scheduled for September 2023 with forward stream production.

Big data analytics traditionally involves download of massive amounts of datasets to common server/cluster for processing. Analytic process gets slower with increasing size of required data and network conditions. Data scientists also need explicit access to data locations to download required data. Explicit access to required data may not always be granted due to security reasons. To simplify and accelerate the analytics process on distributed big data with security considerations, we proposed the Virtual Information Fabric Infrastructure (VIFI) for data driven discoveries. Instead of moving large amounts of data to a common place of processing, VIFI allows automatic transfer of required analytics programs to the distributed data locations for in-place processing of relevant data. VIFI allows data scientists to conduct and coordinate complex analytics processes on distributed data repositories using containerization technology and open-source workflow design tools. VIFI alleviates users from having detailed knowledge of distributed data locations, as well as required dependencies, installation and configuration of analytical libraries. In this paper, we demonstrate our current and future work to improve the VIFI architecture using previous and additional uses cases, data management layer that simplifies search of relevant data sets through addition of metadata, integration with security policies at different institutions with the proposed VIFI security layer, and the use of a user-friendly web interface to carry different VIFI activities.

In this paper, we describe an emergent tool called DAWN (short for "Distributed Analytics, Workflows and Numeric") which is a model for simulating, analyzing and optimizing system architectures for executing arbitrary data processing pipelines. As an example, we will apply DAWN to the investigation of a real-life Big Data use case in climate science: the evaluation of simulated rainfall characteristics using high-resolution observational data. We will show how DAWN can help in determining the optimal architecture, and science algorithms, to execute this case study analyzing distributed datasets, as a tradeoff between the overall time cost and the uncertainty of calculated metrics for model evaluation. We will also show how DAWN can guide architectural decisions for future research, specifically impacting how data should be generated and analyzed to cope with future projected data volumes.

This article describes the Earth System Grid Federation (ESGF) mission and an international integration strategy for data, database and computational architecture, and stable infrastructure highlighted by the authors (the ESGF Executive Committee). These highlights are key developments needed over the next five to seven years in response to large-scale national and international climate community projects that depend on ESGF for success. Quality assurance and baseline performance from laptop to high performance computing characterizes available and potential data streams and strategies. These are required for interactive data collections to remedy gaps in handling enormous international federated climate data archives. Appropriate cyber security ensures protection of data according to projects but still allows access and portability to different ESGF and individual groups and users. A timeline and plan for forecasting interoperable tools takes ESGF from a federated database archive to a robust virtual laboratory and concludes the article.

The Earth System Grid Federation (ESGF) is a multi-agency, international collaboration that aims at developing the software infrastructure needed to facilitate and empower the study of climate change on a global scale. The ESGF's architecture employs a system of geographically distributed peer nodes, which are independently administered yet united by the adoption of common federation protocols and application programming interfaces (APIs). The cornerstones of its interoperability are the peer-to-peer messaging that is continuously exchanged among all nodes in the federation; a shared architecture and API for search and discovery; and a security infrastructure based on industry standards (OpenID, SSL, GSI and SAML). The ESGF software is developed collaboratively across institutional boundaries and made available to the community as open source. It has now been adopted by multiple Earth science projects and allows access to petabytes of geophysical data, including the entire model output used for the next international assessment report on climate change (IPCC-AR5) and a suite of satellite observations (obs4MIPs) and reanalysis data sets (ANA4MIPs).

In production Grids for scientific applications, service and resource failures must be detected and addressed quickly. In this paper, we describe the monitoring infrastructure used by the Earth System Grid (ESG) project, a scientific collaboration that supports global climate research. ESG uses the Globus Toolkit Monitoring and Discovery System (MDS4) to monitor its resources. We describe how the MDS4 Index Service collects information about ESG resources and how the MDS4 Trigger Service checks specified failure conditions and notifies system administrators when failures occur. We present monitoring statistics for May 2006 and describe our experiences using MDS4 to monitor ESG resources over the last two years.

Understanding the Earth's climate system and how it might be changing is a preeminent scientific challenge. Global climate models are used to simulate past, present, and future climates, and experiments are executed continuously on an array of distributed supercomputers. The resulting data archive, spread over several sites, currently contains upwards of 100 TB of simulation data and is growing rapidly. Looking toward mid-decade and beyond, we must anticipate and prepare for distributed climate research data holdings of many petabytes. The Earth System Grid (ESG) is a collaborative interdisciplinary project aimed at addressing the challenge of enabling management, discovery, access, and analysis of these critically important datasets in a distributed and heterogeneous computational environment. The problem is fundamentally a Grid problem. Building upon the Globus toolkit and a variety of other technologies, ESG is developing an environment that addresses authentication, authorization for data access, large-scale data transport and management, services and abstractions for high-performance remote data access, mechanisms for scalable data replication, cataloging with rich semantic and syntactic information, data discovery, distributed monitoring, and Web-based portals for using the system.

In the emerging world of Grid Computing, shared computational, data, other distributed resources are becoming available to enable scientific advancement through collaborative research and collaboratories. This paper describes the increasing role of ontologies in the context of Grid Computing for obtaining, comparing and analyzing data. We present ontology entities and a declarative model that provide the outline for an ontology of scientific information. Relationships between concepts are also given. The implementation of some concepts described in this ontology is discussed within the context of the Earth System Grid II (ESG)[1].