Accurate provenance information facilitates improved understanding of Earth science data and scientific reproducibility and can serve as an indicator of data quality. Provenance capture is an integral part of many modern workflow systems but may not have been considered in the design of legacy data production systems. Furthermore, in addition to data lineage, it is also important to capture contextual information needed for understanding how a data set was produced. This paper describes our experience in retrofitting a legacy data system to support capture, storage, and dissemination of provenance. Data inputs and transformations are logged automatically, while broader context information describing science algorithms and ancillary files is manually compiled. Provenance and context information are integrated for interactive user access and embedded into data files as XML documents compliant with the “Lineage” specification for geographic metadata defined by the International Organization for Standardization in the ISO 19115-2 standard. Lessons learned from this approach can inform others who need to incorporate provenance into a data system after the fact.

There is a dearth of software tools that allow users to easily visualize, analyze and mine satellite imagery. The few tools that are available are expensive commercial packages that provide limited functionality. As part of a NASA funded project, a software tool named GLIDER is currently being developed to fill this void. GLIDER allows users to visualize and analyze satellite data in its native sensor view. Users can enhance the image by applying different image processing algorithms on the data. GLIDER provides the users with a full suite of pattern recognition and data mining algorithms that can be applied to the satellite imagery to extract thematic information. The suite of algorithms includes both supervised and unsupervised classification algorithms. In addition, users can project satellite imagery and analysis/mining results onto a 3D globe for visualization. GLIDER also allows users to add additional layers to the globe along with the projected image. Users can open multiple views within GLIDER to manage, visualize and analyze many data files all at once. This paper describes the features of GLIDER version 1.0.

This paper describes approaches and methodologies facilitating the analysis of large amounts of distributed scientific data. The existence of full-featured analysis tools, such as the Algorithm Development and Mining (ADaM) toolkit and online data repositories now provide easy access and analysis capabilities to large amounts of data. However, there are obstacles to getting the analysis tools and the data together in a workable environment. Does one bring the data to the tools or deploy the tools close to the data? The large size of many current Earth science datasets incurs significant overhead in network transfer for analysis workflows, even with the current advanced networking capabilities. We are developing two solutions for this problem that address different analysis scenarios. The first is a Data Center Deployment of the analysis services for large data selections, orchestrated by a remotely defined analysis workflow. The second is a Data Mining Center approach of providing a cohesive analysis solution for smaller subsets of data. The two approaches can be complementary and thus provide flexibility for researchers to exploit the best solution for their data requirements.

This paper presents a study on intelligent data thinning for satellite data. In particular, the focus is on the thinning of the Atmospheric Infrared Sounder (AIRS) profiles. A direct thinning method is first applied to a synthetic data set in order to identify optimal data selection strategies. Experiments on synthetic data suggest that a thinned data set should combine homogeneous samples, and high gradient and variance of gradient samples for optimal performance. This result leads to the modification of our previously developed Density Adjustment Data Thinning algorithm (DADT). The modified DADT (mDADT) algorithm is used to thin the AIRS profiles. Experiments are conducted to compare the thinning performances of mDADT with two simple thinning algorithms. Experiment results show that mDADT algorithm performs better than the two simple thinning algorithms, especially over the regions of significant atmospheric features.

The Southeastern Universities Research Association (SURA) coastal ocean observing and prediction (SCOOP) program is a SURA Coastal Research initiative that is deploying cutting edge information technology to advance the science of environmental prediction and hazard planning for our nation's coasts. SCOOP is a distributed program, incorporating heterogeneous data, software and hardware; thus the use of standards to enable interoperability is key to SCOOP's success. Standards activities range from internal coordination among SCOOP partners to participation in national standards efforts. As the lead partner in the SCOOP program for both data management and data translation, the University of Alabama in Huntsville (UAH) is developing a suite of advanced technologies to provide core data and information management services for scientific data, including the SCOOP Catalog and a suite of standards-based web services providing Catalog access. Currently under development is a web service that will export information on SCOOP data collections in a schema compliant with the Federal Geographic Data Committee's Content Standard for Digital Geospatial Metadata. SCOOP is also a participant in the OpenlOOS Interoperability Demonstration, which leverages open geospatial consortium (OGQ standards such as the Web map service (WMS) and Web Feature Service (WFS) protocols to display near real time coastal observations together with water level, wave, and surge forecasts. SCOOP partners are also active participants in several data and metadata standards efforts, including the national ocean sciences data management and communications metadata studies and the marine metadata interoperability project. Continued close cooperation between the IT and coastal science modeling communities is producing positive results toward a real-time modeling environment that will benefit coastal stakeholders through better predictive capabilities.

A phenomenon is defined as any state or process known through the senses rather than by intuition or reasoning, and thus is an observable event, especially something special or unusual. A geophysical phenomenon in the context of geoscience data can be characterized as a spatial region which is significantly different from the rest of the image; having higher/lower than average background intensity value; and having higher variation in intensity when compared to the remaining data points. This paper will describe two variations of the Phenomena Extraction Algorithm (PEA). The PEA consists of three components: a hierarchical splitting to efficiently decompose geoscience data into smaller regions; a set of statistical tests to determine whether decomposed region meets the definition of a geophysical phenomenon and an optimization algorithm to determine the best thresholds needed by these statistical tests. The two variations of the algorithm were tested on a synthetic dataset in a series of experiments. The results from these experiments will be presented in this paper. The use of PEA in a proof-of-concept effort within Linked Environment for Atmospheric Discovery (LEAD), a large NSF funded Information Technology Research project, will also be described.

Issues associated with failed data transfers can be complicated and inconvenient, not only for end users expecting timely products, but also for the data managers who are responsible for recovering from failures. Working in close collaboration with engineers at the University of Alabama in Huntsville's Information Technology and Systems Center, the AMSR-E SIPS has evolved a local implementation of the product delivery record server from a simple ECS-compliant data exchange system to a smart automated message handling system that also executes routine operational error recovery procedures. While automated data exchange mechanisms have been the norm for quite some time, the AMSR-E SIPS has deployed unique technologies in our local implementation of the PDRS by integrating it with and automating some routine data management functions, such as process scheduling, restaging of certain classes of failed data transfers, and statistics reporting. This paper discusses the evolution of the PDRS design and architecture, highlights the unique features of this flexible processing system that has contributed to reduced operating costs, and describes our plans to meet future requirements.

The SURA Coastal Ocean Observing & Prediction (SCOOP) program is building cyberinfrastructure (CI) to enable advanced real-time ensemble forecasting of the coastal impacts from storms and hurricanes. This prototype of a reliable, flexible, grid-enabled forecast system integrates real-time distributed data and computer models for the coasts of the southeastern United States. The SCOOP system employs a service-oriented architecture with archive and transport services, metadata catalog, resource management, and portal interfaces. Currently, the SCOOP system uses distributed HPC machines (SCOOP, SURAgrid, others) to meet on-demand requirements. Geospatial web services disseminate the forecast results.We provide the architecture overview and describe the currently deployed system for Hurricane Season 2006 as an example in which a storm advisory automatically initiates a workflow that delivers timely forecasts. The system generates a wind-ensemble and then configures, deploys, and analyzes a variety of water level and wave models across distributed HPC resources to deliver timely forecasts.

Processing requirements at NASA's AMSR-E SIPS * have evolved considerably throughout the life span of the mission, as the SIPS has moved from mission testing and early experimental data processing to full operations supporting both forward processing and reprocessing. Working in close collaboration with the University of Alabama in Huntsville's Information Technology and Systems Center (ITSC), the SIPS has developed a robust and flexible hardware and software framework that supports a variety of data streams and processing requirements. Today the SIPS maintains four operational processing environments in addition to development and integration test areas: • Routine operations environment for forward processing of AMSR-E data upon acquisition, • Late processing environment to handle occasional swaths of data that arrive after gridded products have been created for their time period, • Reprocessing environment for large-scale reprocessing efforts of individual data products or the entire product suite, • A special processing environment to manage special requests by the AMSR-E science team, such as evaluation of new algorithms with seasonal data. In addition, ITSC personnel have integrated science data subsetting and browse imagery generation into the SIPS processing flow, enabling the scientists to quickly and easily target parameters of interest. While similar in structure, each of these environments and their target data streams have unique requirements that would normally necessitate customized software. This paper will describe the evolution of the SIPS, the resulting processing and distribution framework, and plans to meet future requirements.

Fronts are significant meteorological phenomena of interest. The extraction of frontal systems from observations and model data can greatly benefit many kinds of research and applications in atmospheric sciences. Due to the huge amount of observational and model data available nowadays, automated extraction of front systems is necessary. This paper presents an automated method to detect frontal systems from numerical model-generated data. In this method, a frontal system is characterized by a vector of features, comprised of parameters derived from the model wind field. K-means clustering is applied to the generated sample set of the feature vectors to partition the feature space and to identify clusters representing the fronts. The probability that a model grid belongs to a front is estimated based on its feature vector. The probability image is generated corresponding to the model grids. A hierarchical thresholding technique is applied to the probability image to identify the frontal systems and a Gaussian Bayes classifier is trained to determine the proper threshold value. This is followed by post processing to filter out false signatures. Experiment results from this method are in good agreement with the ones identified by the domain experts.

Ensemble classifier methods have been developed to learn from streaming data, and adapt to concept drift. One strategy employed to adapt to concept drift is to rank the classifiers in the ensemble based on their performance on the most recent samples. However, this strategy is problematic when the samples are coming from different portions of the sample space during different times. Data streams with unevenly distributed samples can occur in a wide variety of applications including detection of seasonally occurring weather phenomena, processing of video streams and others. In applications such as these, the uneven distribution of the data in the sample space must be accounted for in order to derive classifiers with good performance. In order to address this requirement, the Coverage Based Ensemble Algorithm (CBEA) was developed. CBEA is an incremental anytime learning algorithm designed to work on streaming data where the samples are very unevenly distributed. The CBEA algorithm keeps information about the range of data used to train each classifier, and keeps classifiers based on two factors: their coverage of the range of possible values, and their age. The CBEA algorithm is described in detail and experimental results demonstrating its utility are presented.

Aurora study is a key area in understanding the connection and interaction of the solar-terrestrial system. Auroral events are monitored on the global scale at the Far Ultraviolet (FUV) spectrum by satellite-based sensors. However, the existence of dayglow emission significantly limits scientists is the ability to determine the location and the size of auroral ovals. A dynamic methodology to remove day airglaw emission from the LBHL band UVI images on the Polar satellite is presented in this paper. First, the methodology identifies the geomagnetic latitude bound of the night side auroral oval. Then, the maximum range of geomagnetic latitude bound of the auroral is inferred based on the domain knowledge. Using the non-auroral dayglow pixels, a multi-variable regression fit of the intensity as function of the cosine of the solar zenith angle and cosine of the satellite viewing zenith angle is obtained. The methodology then uses this fitting function to estimate the dayglow intensity and to remove its effect in the LBHL band FUV images. This methodology does not require an external source of input solar flux and the dayglow removal is solely based on individual images. Experiment results show that dayglow effect is removed significantly from the original LBHL FUV images. Using this methodology, the performance of auroral oval detection algorithms is significantly improved

Data preparation is an important part of the mining process. This work describes MIDAS, an agent framework for intelligent data processing. The objective framework is to provide end users automated data processing services such as subsetting and data format translation by coupling Earth science markup language (ESML) interchange technology and ontologies. These ontology driven agents guide the user through the process of data processing and are able to make decisions for them based on their output requirements. This work describes the design approach used to extend the ESML schema to incorporate semantics using ontologies. It explains the overall architecture including the infrastructure layer, organization and the agent design used in this framework. The subset of performatives derived from the knowledge query and manipulation language (KQML) used by the agents to interact will also be described.

An Interchange Technology facilitates seamless interactions between applications, tools and services with datasets in heterogeneous formats. The Information Technology and Systems Center at the University of Alabama in Huntsville is currently developing this technology focused on Earth Science data, in general, and particularly on the vast amounts of remotely sensed data. The Earth Science Markup Language (ESML) is one such interchange technology and is comprised of description files and a related library of programming utilities. ESML is based on the eXtensible Markup Language (XML/spl trade/) and allows data "descriptions" to be written in a standard fashion. ESML is unique in that it not only describes the content and structure of the data, but also provides semantic information for the end user application to utilize. Another advantage is that the effort involved to describe legacy data formats in ESML will be small. ESML and the associated library will allow wider interoperability of Earth Science services and tools, enabling Earth Scientists to author, discover, and interpret Earth Science information, and to work more easily with data in a variety of formats and structures. This interchange technology will facilitate the development of dataset-independent searches, visualization, and analysis tools without requiring data to be in any particular format. This paper will describe this interchange technology and provide examples of generic access to heterogeneous remotely sensed data sets.

The spread of invasive species is one of the most daunting environmental, economic, and human-health problems facing the United States and the World today. It is one of several grand challenge environmental problems being considered by NASA's Earth Science Vision for 2025. The invasive species problem is complex and presents many challenges. Developing an invasive species predictive capability could significantly advance the science and technology of ecological forecasting.

A new type of texture feature based on association rules is described. Association rules have been used in applications such as market basket analysis to capture relationships present among items in large data sets. It is shown that association rules can be adapted to capture frequently occurring local structures in images. The frequency of occurrence of these structures can be used to characterize texture. Methods for segmentation of textured images based on association rule features are described. Simulation results using images consisting of man made and natural textures show that association rule features perform well compared to other widely used texture features. Association rule features are used to detect cumulus cloud fields in GOES satellite images and are found to achieve higher accuracy than other statistical texture features for this problem.

A new type of texture feature based on association rules is proposed in this paper. Association rules have been used in applications such as market basket analysis to capture relationships present among items in large data sets. It is shown that association rules can be adapted to capture frequently occurring local structures in images. Association rules capture both structural and statistical information, and automatically identifies the structures that occur most frequently and relationships that have significant discriminative power. Methods for classification and segmentation of textured images using association rules as texture features are described. Simulation results using images consisting of man made and natural textures show that association rule features perform well compared to other widely used texture features. It is shown that association rule features can distinguish texture pairs with identical first, second, and third order statistics, and texture pairs that are not easily discriminable visually.

The paper addresses the problem of acquiring from scientific data, metadata that is descriptive of the actual content of the data. Scientists can use this content based metadata in subsequent archive searches to find data sets of interest. Such metadata would be especially useful in large scientific archives such as NASA's Earth Observing System Data and Information System (EOSDIS). The paper presents two generic approaches for content based metadata acquisition: target dependent and target independent. Both of these approaches are oriented toward characterizing datasets in terms of the scientific phenomena, such as mesoscale convective systems (severe storms) that they contain. In the target dependent approach, the archived data is mined for particular phenomena of interest and polygons representing the phenomena are stored in a spatial database where they can be used in the data search process. In the target independent approach, data is initially mined for deviations from normal and for trends. This data can then be used for subsequent searches for particular transient phenomena using the deviation data, or for phenomena related to trends. The paper describes results from implementing both of these approaches.