Ice particle size, represented by maximum dimension, and its relation to particle mass are key quantities in physical precipitation retrievals. This mass-dimensional relation has primarily been derived from in situ imaging probe measurements. However, these relations are biased by a fundamental limitation of the imaging probes: the maximum dimension is determined from one to three 2-D projections of the 3-D particle geometry. Using a database of heuristically generated synthetic snow aggregates, we show that these instruments can produce substantial uncertainty in derived mass-dimensional relations. This uncertainty can be alleviated with more views at additional angles that provide independent information. Additionally, we show that the orientation distributions of the particle have a second-order effect on the errors in deriving the maximum dimensions, with more horizontally aligned particles producing greater uncertainties.

Geostationary satellite sensors have been widely used for precipitation retrieval, and numerous algorithms have been developed for precipitation retrieval using observations from geostationary satellite sensors. However, using the observations from a single geostationary satellite only offers a certain viewing angle and lacks the observation from a different perspective. In this research, we propose a deep learning (DL) framework for precipitation retrieval by leveraging the combined observations from the duo GOES satellites, namely, GOES-16 and GOES-18, as well as the Digital Elevation Model (DEM) information of the selected study domain. The experimental results show that the precipitation retrieval performance of the proposed framework is superior to the currently operational GOES RRQPE product and provides more accurate satellite-based precipitation retrieval.

Water clouds are composed of spherical water droplets, to which the Mie solution applies for their electromagnetic scattering. The essential variable of Mie solutions is$\xi\equiv mkD$(or equivalently mkr), where$m$is the complex index of refraction of water,$k=2\pi/\lambda$the angular wavenumber,$\lambda$the wavelength, and D (r) the diameter (radius) of the water droplet. Consequently, all water-droplet single-scattering properties, such as extinction and scattering efficiencies, i.e.,$Q_{e}$and$Q_{s}$, are functions of$\xi$, which provides a convenient pathway to parameterize water-droplet single-scattering properties across the electromagnetic spectrum or spectrum segments. In this paper, we first demonstrate that the numerical Mie solution indeed depends only on$\xi$. We then present our first attempt to parameterize extinction efficiency across the visible-infrared spectrum.

Deep Neural Networks (DNNs) have performed admirably in classification tasks. However, the characterization of their classification uncertainties, required for certain applications, has been lacking. In this work, we investigate the issue by assessing DNNs’ ability to estimate conditional probabilities and propose a framework for systematic uncertainty characterization. Denoting the input sample as x and the category as y, the classification task of assigning a category y to a given input x can be reduced to the task of estimating the conditional probabilities p(y|x), as approximated by the DNN at its last layer using the softmax function. Since softmax yields a vector whose elements all fall in the interval (0, 1) and sum to 1, it suggests a probabilistic interpretation to the DNN’s outcome. Using synthetic and real-world datasets, we look into the impact of various factors, e.g., probability density f(x) and inter-categorical sparsity, on the precision of DNNs’ estimations of p(y|x), and find that the likelihood probability density and the inter-categorical sparsity have greater impacts than the prior probability to DNNs’ classification uncertainty.

SpatioTemporal Adaptive Resolution Encoding (STARE) is an integer encoding scheme for spatial and temporal coordinates and volumes. The spatial element of STARE encodes the traversal through a recursive partitioning (quad-furcation) of spherical triangles (or trixels) on the unit sphere to index discrete solid angles. In addition, the index contains neighborhood information. The (approximate) resolution of a model grid cell or a satellite image pixel can thus be indicated together with its geolocation in one 64-bit integer number. The temporal element of STARE has a similar design with partitioning based on calendrical units rather than uniform quadfurcation. As such, pairs of STARE spatiotemporal indices index spatiotemporal volume intervals of various sizes. Therefore, not only can STARE optimize search and subsetting performance for geo-spatiotemporal data, but also facilitate spatiotemporal data placement alignment on distributed parallel computing resources to maximize scalability by minimizing costly unnecessary communication among these resources. In this paper, we focus our discussion on the application of STARE spatial encoding to NASA satellite swath data. This is accomplished by generating STARE companion or “sidecar” files, one file per data granule, which contain the STARE indices for the geolocations, together with approximate resolutions, of the data in the granule. The STARE sidecar files are stored as netCDF-4/HDF5 files. In the design of file metadata, we strive to ensure compatibility with the Climate and Forecast (CF) conventions, widely used in the geo-science community. Specifically, in accordance with the practices of CF conventions, variables are not identified by their names, but by including special attributes. This allows variable names to vary, which they invariably do, while ensuring that reading programs can properly parse the sidecar file.

Big Data technologies such as Cloud and parallel distributed computing and storage are necessary to treat Earth Science data volume. Yet the great diversity of Earth Science data renders it nearly impossible to organize that data on scalable platforms without costly data movement or undesired interpolation that straitjackets scientific research. The SpatioTemporal Adaptive Resolution Encoding (STARE) is an alternative geolocation and indexing scheme for harmonizing data for integrative analysis on scalable systems. STARE uses a hierarchical, recursive partitioning of space and time in which the index or coordinates of each node are integers from the same index space, usually allowing quick comparison without floating-point calculation. STARE is well suited to provide a unifying geo-semantics for arranging data in databases. In this work, we outline the technical principles underlying STARE and its application to SQLite as an example. The STARELite STARE-aware lightweight geo-database can be used to catalogue diverse data for geographical querying and integration on local resources and Cloud.

Scaling up volume and variety in Big Earth Science Data is particularly difficult when combining low-level, ungridded data, such as swath observations obtained with, for example, Moderate Resolution Imaging Spectroradiometers (MODIS). A unified way to index and combine data with different geo-spatiotemporal layouts and incomparable native array formatting is required for scalable integrative analyses based on data at its full instrument resolution, that is, without extra interpolation (or extrapolation) onto a common grid. The SpatioTemporal Adaptive Resolution Encoding (STARE) uses the Hierarchical Triangular Mesh (HTM) and the Hierarchical Calendrical Partitioning (HCP), recursive partitionings of solid angle and time into tree data structures, to encode spatiotemporal neighborhoods as sets of integers. Regions sharing common paths through the STARE tree hierarchy have similar index values, which can then serve as keys in algorithms and data structures supporting scalable integrative analyses. Thus, STARE co-aligns data in both physical (spatiotemporal) and cyber (memory) spaces, providing a means for marshalling computing resources and conducting analysis with minimum data movement, addressing volume scalability while simultaneously unifying diverse data for variety scaling. In this paper, we demonstrate how easy it is to use the Python STARE API (PySTARE) and the parallel programming platform Dask to integrate MODIS and Geostationary Operational Environmental Satellite (GOES) data, datasets with very different geo-spatiotemporal characteristics.

We highlight recent progress in and discuss future plans for the OpenSSP particle and scattering property database. Ongoing work has focused on expanding the types of particles to include polycrystals and melting snowflakes. Future expansion will include rimed particles, hail, and aligned snowflakes.

To address the acute challenge posed by the melting layer to accurate surface precipitation retrievals from space, we ensure the compositional consistency in ice, liquid, and total masses of synthetic melting hydrometeors with a method of stochastic compensation. The method is applied to simulated melting hydrometeors prior to calculating their scattering properties using the discrete dipole approximation (DDA). We investigate the impact of this stochastic compensation to calculated scattering properties by contrasting it with a naïve approach and report our findings.

Analysis incorporating geoscience data from different data sources entails dealing with their immense variety and volume. Until now, combining, for example, two or more different swath datasets from spaceborne observations has been a tedious, laborious process, limiting the scalability of potentially impactful integrative analyses. With the technologies developed in the NASA-funded SpatioTemporal Adaptive Resolution Encoding (STARE) project, we have made strides towards enabling scalable integrative analysis of diverse, voluminous geoscience data. Using STARE as a consistent geo-spatiotemporal indexing scheme to unify different data sources according to spatiotemporal colocation, spatiotemporal data co-alignment can thus be maintained on distrib-uted/parallel/Cloud resources to minimize costly and often unnecessary data transfer and communication, and to drastically improve scalability.

Using a three-dimensional radiative transfer model combined with cloud-resolving model output, we simulate active and passive sensor observations of clouds and precipitaiton. This combination of tools allows us to diagnose the contributions of various hydrometeor types. Radar multiple scattering is most closely associated with the presence of graupel. At W-band, massive amounts multiple scattering in deep convection can decorrelate the reflectivity profile from the vertical structure, but for less intense events, multiple scattering could be a useful indicator of riming. For passive sensors, polarization differences at 166 GHz indicate the presence of horizontally-aligned frozen particles with pronounced aspect ratios, while high concentrations of more isotropic aggregates and graupel dampen the polarization difference while also contributing to the lowest brightness temperature depressions. The insights into remote sensing measurements will facilitate the development of improved algorithms and advanced sensors.

We have leveraged STARE indexing to package partitioned data chunks from diverse datasets into netCDF files, distributed them on a cluster of 16 lightweight nodes with their placements spatiotemporally co-aligned, and demonstrated a few integrative analyses using netCDF parallel I/O and Python MPI, with single-user performance and scalability comparable to, or even better than, that of a parallel array database management system (ADBMS) such as SciDB. However, records of the node location and STARE index ranges for each data chunk, similar to the chunk maps of SciDB, must be maintained and consulted by the I/O and analysis code for coordinating the analytic operations in parallel, in order to achieve the good performance and scalability.

A Big Earth Data platform has been constructed based on a parallel distributed database management system, SciDB, to demonstrate visual analytics with interactive animation on diverse datasets. This high-performing capability is achieved by exploiting transparent multimodal parallelization, largely enabled by a unifying indexing scheme, STARE, that provides unparalleled variety scaling. Such a platform not only supports effortless interactive data exploration and analysis but also has the potential to systemize machine learning undertakings with diverse and voluminous Earth Science data.

An accurate representation of the electromagnetic (EM) behavior of precipitation particles requires modeling of realistic complex geometry and a numerically efficient technique to calculate averaged scattering properties over multiple random target orientations. The discrete dipole approximation is commonly used to compute scattering and absorption by snow particles, because of its geometry flexibility and numerical low cost. However, this method becomes inefficient when the scattering quantities need to be calculated for a large number of orientations. To overcome this limitation, we apply, in this paper, a direct solver-based method, known as the characteristic basis function method (CBFM), to the modeling of scattering by randomly oriented and complex-shaped snow particles. This domain decomposition technique is based on the generation of a new set of basis functions adapted to the geometry of the scatterer in order to significantly reduce the numerical size of the EM problem. This enables us to use a direct solver for the resolution of the final compressed system of linear equations, which is better adapted for multiple excitation problems. When applied to numerically large snow particles, our CBFM-based model, named Numerically Efficient Scattering by Complex Particles, has been shown to yield good results, which compare well with those obtained with discrete dipole scattering, while providing a dramatic reduction in the CPU time.

We have devised and implemented a key technology, SpatioTemporal Adaptive-Resolution Encoding (STARE), in an array database management system, i.e. SciDB, to achieve unparalleled variety scaling for Big Earth Data, enabling rapid-response visual analytics. STARE not only serves as a unifying data representation homogenizing diverse varieties of Earth Science Datasets, but also supports spatiotemporal data placement alignment of these datasets to optimize a major class of Earth Science data analyses, i.e. those requiring spatiotemporal coincidence. Using STARE, we tailor a data partitioning and distribution strategy for the data access patterns of our scientific analysis, leading to optimal use of distributed resources. With STARE, rapid-response visual analytics are made possible through a high-level query interface, allowing geoscientists to perform data exploration visually, intuitively and interactively. We envision a system based on these innovations to relieve geoscientists of most laborious data management chores so that they may focus better on scientific issues and investigations. A significant boost in scientific productivity may thus be expected. We demonstrate these advantages with a prototypical system including comparisons to alternatives.

We investigate the impact of data placement on two Big Data technologies, Spark and SciDB, with a use case from Earth Science where data arrays are multidimensional. Simultaneously, this investigation provides an opportunity to evaluate the performance of the technologies involved. Two datastores, HDFS and Cassandra, are used with Spark for our comparison. It is found that Spark with Cassandra performs better than with HDFS, but SciDB performs better yet than Spark with either datastore. The investigation also underscores the value of having data aligned for the most common analysis scenarios in advance on a shared nothing architecture. Otherwise, repartitioning needs to be carried out on the fly, degrading overall performance.

We have implemented a flexible User Defined Operator (UDO) for labeling connected components of a binary mask expressed as an array in SciDB, a parallel distributed database management system based on the array data model. This UDO is able to process very large multidimensional arrays by exploiting SciDB's memory management mechanism that efficiently manipulates arrays whose memory requirements far exceed available physical memory. The UDO takes as primary inputs a binary mask array and a binary stencil array that specifies the connectivity of a given cell to its neighbors. The UDO returns an array of the same shape as the input mask array with each foreground cell containing the label of the component it belongs to. By default, dimensions are treated as non-periodic, but the UDO also accepts optional input parameters to specify periodicity in any of the array dimensions. The UDO requires four stages to completely label connected components. In the first stage, labels are computed for each subarray or chunk of the mask array in parallel across SciDB instances using the weighted quick union (WQU) with half-path compression algorithm. In the second stage, labels around chunk boundaries from the first stage are stored in a temporary SciDB array that is then replicated across all SciDB instances. Equivalences are resolved by again applying the WQU algorithm to these boundary labels. In the third stage, relabeling is done for each chunk using the resolved equivalences. In the fourth stage, the resolved labels, which so far are “flattened” coordinates of the original binary mask array, are renamed with sequential integers for legibility. The UDO is demonstrated on a 3-D mask of 0(10n) elements, with 0(108) foreground cells and o(106) connected components. The operator completes in 19 minutes using 84 SciDB instances.

We have implemented an updated Hierarchical Triangular Mesh (HTM) as the basis for a unified data model and an indexing scheme for geoscience data to address the variety challenge of Big Earth Data. In the absence of variety, the volume challenge of Big Data is relatively easily addressable with parallel processing. The more important challenge in achieving optimal value with a Big Data solution for Earth Science (ES) data analysis, however, is being able to achieve good scalability with variety. With HTM unifying at least the three popular data models, i.e. Grid, Swath, and Point, used by current ES data products, data preparation time for integrative analysis of diverse datasets can be drastically reduced and better variety scaling can be achieved. HTM is also an indexing scheme, and when applied to all ES datasets, data placement alignment (or co-location) on the shared nothing architecture, which most Big Data systems are based on, is guaranteed and better performance is ensured. With HTM most geospatial set operations become integer interval operations with further performance advantages.

It is found that data placement on the networked nodes of a cluster based on the shared-nothing architecture (SNA) should align in the physical (i.e. spatiotemporal) space for most geoscience Big Data analysis systems in order to minimize data movements and thus achieve optimal performance and efficiency. This is due to the fact that data analysis in geosciences predominantly requires spatiotemporal coincidence. If individual datasets are considered separately in their placement on the cluster nodes, these systems often have to move data between nodes when an analysis involves two or more datasets. In this paper, we first report our discoveries from a data placement alignment experiment with two Big Data technologies, SciDB and Spark+HDFS, and then elucidate some of the far-reaching implications of this discovery.

Most of the climatological studies derived from reanalysis datasets to-date have been presence-based rather than event-based. We have gathered event-based climatological statistics from thirty-seven-plus (37+) years of blizzard-like snowstorms, identified and individually tracked, using hourly high-resolution datasets from the NASA's Modern Era Retrospective-analysis for Research and Applications (MERRA) data collection. We have not only extracted summary statistics for all storms, such as cumulative-probability density functions (CDFs, in percentiles) of storm duration and cumulative area coverage, but also per-event statistics for each storm, e.g. beginning/ending times, hourly locations, hourly mean snowfall intensities, and snowfall intensity probability distribution. In addition, we have constructed a web portal where users can view the hourly locations of each snowstorm annotated with conditions of the storm at that hour. Users with accounts on the portal can discover coincident data granules of relevant satellite remote-sensing observations by querying NASA metadata repository, i.e. EOS Clearing House (ECHO) or upcoming Common Metadata Repository (CMR) and create/share individualized virtual collections apposite to their research.

Feature extraction and tracking is a fundamental operation used in many geoscience applications. In this paper, we present a scalable method for computing and tracking features on distributed memory machines for large-scale geospatial data. We carefully apply new communication schemes to minimize the data exchanged among the computing nodes in building and updating the global connectivity information of features. We present a theoretical complexity analysis, and show that our method can significantly reduce the communication cost compared to the traditional method.

Our AES is an ideal example of a new generation of scientific analysis tools that are empowered by the rapid growth of facilities tailored for data-intensive computing. AES will greatly reduce the effort on the part of investigators to systematically search for interesting correlations and test hypotheses while also freeing researchers from the burden of managing the exploding volume of data. By supporting the ability to exchange event specifications and query results, AES greatly aids collaboration among investigators. We anticipate that AES will ultimately lead to entirely novel lines of investigation.

The retrieval of water cloud microphysics involving ensembles of spherical droplets has been firmly established on a solid theoretical foundation by using liquid water content, effective radius, and effective variance as characterizing parameters for the scattering properties of these ensembles. However, there is no such solid theoretical foundation for the retrieval of non-spherical or irregular hydrometeors. Based on scattering calculations performed on realistic crystals generated from an innovative numerical crystal growth model, it is found that, under the constraint of a fixed particle volume distribution, the single-scattering properties derived from ensembles of irregular particles made up with the same set of crystal habits are nearly invariant, if they also have at the same time the same total volume-to-surface and total volume-to-mean-projected-area ratios. Therefore we suggest the use of two effective radii, i.e effective projected-area radius and effective surface area radius, together with ice water content and particle volume distribution, as characterizing parameters for the scattering properties of particulate ensembles composed of non-spherical or irregular ice crystals.

This research is concerned with the problem of producing polar cloud masks for satellite imagery. The results presented are for Thematic Mapper (TM) data from the northern and southern polar regions, however, the techniques discussed will be applied to ASTER data when it becomes available. A series of classification techniques have been implemented and tested, the most promising of which is a neural network classifier. To use a neural network classifier, the pixels in the data must be transformed into feature vectors, some of which are used for training the network and the remainder of which are reserved for testing the final system. The first challenge is the identification of pure pixel samples from the imagery. The Interactive Visual Image Classification System (IVICS) was developed specifically for this project to make this task simpler for the human expert. After labeling the pixels, the feature vectors are generated. One hundred and forty potential vector elements, consisting of linear and nonlinear combinations of the satellite channel data, have been identified. Because smaller input vectors reduce the difficulty of training and can improve classification accuracy, the set of potential vector elements must be reduced. Two techniques have been tested: a histogram-based selection method and a fuzzy logic method. Both have proven effective for this task. Although the polar region is the only area considered in this work, a system that can produce cloud masks for all areas of the globe will be required. Thus, speed, extensibility, and flexibility requirements must be added to the accuracy constraint. To achieve these goals, a two-stage classification approach is used. The first stage uses a series of static and adaptive thresholds derived from statistical analysis of the polar scenes to reduce the set of possible classes to which a pixel may be assigned, once a cluster of classes has been selected, a neural network trained to distinguish between the classes in the cluster is used to make the ultimate classification.