We introduce the Atmospheric Radiation Measurement (ARM) Data Workbench, a web-based interface that combines access to atmospheric datasets and computational resources. It is developed to facilitate a interactive data exploration and data analysis for the research community. The ARM Data Workbench offers a comprehensive solution that combines data locality, user-friendly access, and collaborative capabilities within a single platform. To empower researchers with a familiar analysis and compute environment, the data workbench integrates a JupyterHub which is a popular platform for scientific computing. In this paper, we discuss the features of ARM Data Workbench through which users can access the ARM hosted scientific datasets, order them and get them delivered. We discuss different types of user accounts we provide in order to have access to ARM data and the compute platform to perform analysis. We explain the process of navigating to ARM’s Data Discovery interface to access datasets and JupyterHub platform for scientific users with an example case study. The platform targets new researchers, graduate students and established researchers to help them in their research in a collaborative environment.

The Atmospheric Radiation Measurement (ARM) is a multi-laboratory and multi-institutional U.S. Department of Energy (DOE) Office of Science National User Facility. The ARM Data Center (ADC), located at Oak Ridge National Laboratory, collects, archives, and shares vast atmospheric data crucial for climate research. The ADC manages over 7 PB of data from 460 instruments worldwide, processing it into more than 11,000 diverse data products using the Network Common Data Form (NetCDF) for machine-independent accessibility. The primary challenge addressed in this paper is the efficient management and distribution of vast and diverse datasets essential for the climate research community, enhancing accessibility through advanced tools like Data Discovery. The ADC has developed advanced infrastructure and software architecture to handle the continuous influx of heterogeneous data to enhance data discoverability, resulting in increased scientific collaboration. In 2023, users from over 34 countries downloaded and utilized ARM data, resulting in 1,455 publications. The ADC’s efforts have significantly improved the discoverability and usability of atmospheric data, fostering extensive scientific research and collaboration. This paper details the solutions implemented by the ADC team for efficient data discovery and distribution, and it demonstrates ARM’s capability of staging processed data for scientific analysis.

Scientific datasets are continuously growing with the amount of raw data being collected worldwide. This amount of data poses the biggest challenge to web search engines on how to retrieve them efficiently. This paper discusses how major scientific data centers are using popular open-source search platforms such as Solr [1] to retrieve structured data stored in data sources such as relational database management systems using its import handler mechanisms [2]. Additionally, we will also focus on how we can configure Solr to serve advanced full-text, faceted search capabilities, along with its key features, which simplify representing and delivering better performance to the scientific search interfaces.