The 12 papers in this special issue represent the state of the art in geoscience data provenance research and applications.

The advancement of Earth observing sensors, data, and information systems enhances significantly the capabilities to access and process large volumes of geoscience data, which are often consumed by scientific workflows and processed in a distributed information environment. Consequently, data provenance becomes important since it allows users to determine the usability and reliability of data products. Motivation for capturing and sharing provenance also comes from the distributed data and information infrastructure that has been benefiting the Earth science community in the past decade, such as spatial data and information infrastructure, e-Science, and cyberinfrastructure. This paper provides an overview of geoscience data provenance in supporting provenance-aware geoscience data and information systems by summarizing state-of-the-art technologies and methodologies of geoscience data provenance and highlighting key considerations and possible solutions for geoscience data provenance.

Various remote sensing agencies of the world have created a data rich environment for research and applications over the last three decades. Especially over the last decade, the volume and variety of data useful for Earth system science have increased quite rapidly. One of the key purposes of collecting these data and generating useful digital products containing derived geophysical parameters is to study the long-term trends in the Earth's behavior. Long-term observational data and derived products are essential for validating results from models that predict the future behavior of the Earth system. Given the significant resources expended in gathering the observational data and developing the derived products, it is important to preserve them for the benefit of future generations of users. Preservation involves maintaining the bits with no loss (or loss within scientifically acceptable bounds) as they move across systems as well as over time, ensuring readability over time, and providing for long-term understandability and repeatability of previously obtained results. In order to ensure long-term understandability and repeatability, it is necessary to identify all items of content that must be preserved and plan for such preservation. This paper discusses the need for a standard enumerating and describing such content items and reports on the progress made by NASA and the Federation of Earth Science Information Partners (ESIP Federation) in the U.S. towards such a standard.

The authors review recent contributions to the state-of-thescience and benign proliferation of satellite remote sensing, spatial data infrastructure, near-real-time data acquisition, analysis on high performance computing platforms, sapient archiving, multi-modal dissemination and utilization for a wide array of scientific applications. The authors also address advances in Geoinformatics and its growing ubiquity, as evidenced by its inclusion as a focus area within the American Geophysical Union (AGU), European Geosciences Union (EGU), as well as by the evolution of the IEEE Geoscience and Remote Sensing Society's (GRSS) Data Archiving and Distribution Technical Committee (DAD TC).

Standard products from the five sensors on NASA's Earth Observing System's (EOS) Terra satellite are being used world-wide for earth science research and applications. This paper describes the evolution of the Terra data systems over the last decade in which the distributed systems that produce, archive and distribute high quality Terra data products were scaled by two orders of magnitude.

During 2005 through 2008, NASA defined and implemented a major evolutionary change in the Earth Observing system Data and Information System (EOSDIS) to modernize its capabilities. This implementation was based on a vision for 2015 developed during 2005. The “EOSDIS 2015 Vision” emphasizes increased end-to-end data system efficiency and operability; increased data usability; improved support for end users; and decreased operations costs. One key feature of the Evolution plan was achieving higher operational maturity (ingest, reconciliation, search and order, performance, error handling) for the NASA's Earth Observing System Clearinghouse (ECHO). The ECHO system is an operational metadata registry through which the scientific community can easily discover and exchange NASA's Earth science data and services. ECHO contains metadata for 2, 726 data collections comprising over 87 million individual data granules and 34 million browse images, consisting of NASA's EOSDIS Data Centers' and the United States Geological Survey's Landsat Project holdings. ECHO stores metadata from a variety of science disciplines and domains, including Climate Variability and Change, Carbon Cycle and Ecosystems, Earth Surface and Interior, Atmospheric Composition, Weather, and Water and Energy Cycle. ECHO provides a platform for the publication, discovery, understanding and access to NASA's Earth Observation resources (data, service and clients). In their native state, these data, service and client resources are not necessarily targeted for use beyond their original mission. However, with the proper interoperability mechanisms, users of these resources can expand their value, by accessing, combining and applying them in unforeseen ways. ECHO provides access to its capabilities through a set of services. These ECHO Applications Program Interfaces (APIs) are based on industry standards for performing web-based computing, specifically web services profile.

The 13 papers in this special issue focus on data archiving and distribution. The papers are grouped into five general categories, with some overlap among them: broad data system capabilities - development and evolution; processing; archiving and distribution; search and access; and long-term preservation.

NASA develops and utilizes a variety of large and small scale data systems to meet its Earth science objectives. Generally, the data systems are considered "Core Capabilities" providing the basic infrastructure needed to capture data from space-borne instruments as well as process, archive and distribute science data products to the user community. The Earth Observing System (EOS) Data and Information System (EOSDIS) is an example of NASA's Core Capabilities. Another class of capabilities, which may be called "Community Capabilities", is generally selected through peer review to support NASA's Earth science programs. Both Core and Community Capabilities are necessary not only to serve the basic needs of the broad and diverse community of over 200,000 users of NASA's Earth science data but also to improve and enhance the science insight for a variety of research and science applications. As information technology changes at a quick pace and new capabilities become readily available, users' expectations increase as well. Data systems' organizations need to be dynamic and adapt to these changes. While both Core and Community Capabilities must be adaptive, the pace at which adaptation occurs generally differs between the two. Core Capabilities typically have stricter operations and performance requirements serving a large community. These systems tend to be slower in changing whereas the Community Capabilities can be early adopters of new technologies since any impacts affect a much smaller community. Evolution of EOSDIS, based on a NASA-established vision for circa 2015 has been in implementation since early 2006. The vision emphasizes increased end-to-end data system efficiency and operability; increased data usability; improved support for end users; and decreased operations costs. Due to concurrent operations, the transition of operations to newer systems is being carried out incrementally. This paper provides a summary of the implementation, transition to operations and the lessons learned.

Numerous systems have been developed and have been operating in recent years for processing, archiving and distributing data from remote sensing satellites and in situ observations and increasing the utility of data by providing data products and/or information services to users. Among such data systems, this paper considers those that NASA develops and/or utilizes in order to fulfill one of its strategic objectives: "Study Earth from space to advance scientific understanding and meet societal needs." Such data systems are of various sizes and meet a variety of requirements. Some data systems constitute NASA's Core Capabilities, and provide the basic infrastructure for robust and reliable data capture, processing, archiving and distributing a set of data products to a large and diverse user community. The Earth Observing Data and Information System (EOSDIS) is one example of such a system. Another class of data systems, constituting Community Capabilities, provides specialized and innovative services to data users and/or research products offering new scientific insight. Such systems are generally supported by NASA through peer reviewed competition. This paper focuses on the community capabilities including projects under the Research, Education and Applications Solutions Network (REASoN). Well-established goals and metrics measuring progress relative to those goals are essential to assess effectiveness of any systems, programs or organizations. Detailed metrics are needed by system managers to observe performance and plan improvements. Metrics of interest to program managers tend to be oriented towards the utility of the systems in meeting strategic goals, and sizes of communities served. The metrics must be tailored to the types of community capabilities. For example, systems providing data and those providing services need to be assessed using different metrics. Metrics aggregated across systems are of interest in measuring program effectiveness supporting NASA'a science, applied science, and education objectives.

The earth observing system (EOS) data and information system (EOSDIS) has been serving a broad user community since 1994. Most of NASA's Earth science data are currently being archived, managed and distributed by EOSDIS. Also, EOSDIS commands and controls EOS spacecraft and instruments, captures data from the instruments and processes them into a set of standard products. As of March 2006, the archives of EOSDIS held over 4.3 petabytes of data from over 90 instruments and over 2000 distinct science products. The distribution of data to end users amounts to approximately 2 TB a day. The community receiving data from EOSDIS is on the order of 200,000 distinct users from a diverse set of organizations and scientific disciplines. While EOSDIS is effectively managing a large amount of data and successfully serving a broad user community, it is a system whose design and development originated more than 10 years ago during which many advances have occurred in information technology. Although there has been an on-going process of technology infusion, incremental improvements in processing and performance, and new functionality added in areas of user access, distribution, and archive management over the years, the underlying design has remained essentially the same. During this time frame, data volumes have grown dramatically and the science community has gained considerable experience in processing and analyzing their data. More recently, through examination of current operations and a series of lessons learned, there has been a desire to re-examine current operations for significant improvements in a variety of areas. The overall objectives of the EOSDIS evolution are to: increase end-to-end data system efficiency and autonomy while decreasing operations costs, increase data interoperability and usability by the science research, application, and modeling communities, improve data access and processing, and ensure safe stewardship.

Research in certain scientific disciplines - including Earth science, particle physics, and astrophysics - continually faces the challenge that the volume of data needed to perform valid scientific research can at times overwhelm even a sizable research community. The desire to improve utilization of this data gave rise to the Intelligent Archives project, which seeks to make data archives active participants in a knowledge building system capable of discovering events or patterns that represent new information or knowledge. Data mining can automatically discover patterns and events, but it is generally viewed as unsuited for large-scale use in disciplines like Earth science that routinely involve very high data volumes. Dozens of research projects have shown promising uses of data mining in Earth science, but all of these are based on experiments with data subsets of a few gigabytes or less, rather than the terabytes or petabytes typically encountered in operational systems. To bridge this gap, the Intelligent Archives project is establishing a testbed with the goal of demonstrating the use of data mining techniques in an operationally-relevant environment. This paper discusses the goals of the testbed and the design choices surrounding critical issues that arose during testbed implementation.

The present state in the NASA's Earth science data and information systems has evolved over the last two decades from a rather desperate and disconnected set of data providers through significant efforts by several organizations. In the 1980s NASA's Earth science data were generally held by principal investigators or at specialized, non-interoperable data systems. Considerable progress has been made during the last decade in data access, interoperability, and management of very large volumes of data. Today, a suite of Earth observing satellites, funded research investigations, and data systems are operating to provide an unprecedented amount of data and information products to a broad user community. Many lessons have been learned in the process of the evolution to-date and are being applied for the benefit of data systems' evolution in the near-term. Also, visions for the longer-term future are being studied. Embryonic technologies of today are relevant to support such visions. Future data systems with "Intelligent Archives" could support true knowledge building systems in support of the NASA's future Earth science goals.

The "trace" of a rectangular matrix is defined as the trace of a square matrix obtained by appending null rows (or columns) at the bottom (or right) end. The problem of maximizing the trace of a matrix, by permutations and mergers of rows and columns with constraints on the resulting size of the matrix, is of interest in comparison of maps and image-change detection. This correspondence presents an algorithm based on dynamic propramming for efficient maximization of trace.

Several geometric distortions are present in remotely sensed images depending on the type of sensors and the object being observed. It is often desirable to compensate for these distortions and store the images in reference to a standard coordinate system. Digital techniques for correction are versatile and introduce a minimum of radiometric errors. The main problems to be considered in this area are the determination of the corrective transformation, resampling, and the management of the large quantities of data. This correspondence considers the last problem. It is shown here that, by a judicious rearrangement of the input data, considerable reductions in the required memory capacity can be achieved. The rearrangement can be accomplished in several stages. The method presented here is amenable to pipeline implementation for processing a continuous stream of images. It can also be used on a relatively small computer with the staging being in time sequence rather than hardware. Examples are shown to illustrate the reductions in overall cost depending on the relative cost of memory and the peripheral devices.

This paper considers the problem of detecting the local similarity between templates in a given class and a given image using a hierarchically ordered sequential decision rule. When the given set consists of a large number of templates and the number of locations in the image matching any of the templates is small, it is wasteful to examine each of the templates at every location in the image for a match. Instead, it is proposed that the set of templates be partitioned and a ``representative template'' be defined for each of the partitions. Several levels of partitioning are defined. Elimination of mismatching locations and termination of computation can take place at each, level of detection. Each level of testing is over a more restrictive subset of the template class than the previous level. The paper presents a general formulation of this approach and gives criteria for selecting representative templates, the ordering of components of a template vector for error evaluation, and the threshold sequences to be used in deciding about a ``match.'' Suboptimal solutions are given satisfying these criteria. Illustrative examples are provided showing recognition of linear features in test patterns and photographs obtained by aerial and spaceborne sensors.

The purpose of this paper is to present an algorithm for transposing large rectangular matrices. This is basically a generalization of Eklundh's algorithm. Eklundh's method is designed to handle arrays in place of using powers-of-2 approach wherein partitions of the matrix have dimensions which are powers of 2. The algorithm presented here does not have this restriction. The choice of partitions is posed as an optimal control problem amenable to solution using Bellman's principle of optimality. The optimal partitioning of the matrix can be determined depending on the computer configuration (i.e., core size, whether moving-head or fixed-head disk drives are used, number of words per track, etc.) and the matrix dimensions. Simple modifications are noted for transposing complex/double precision and packed integer matrices. Experimental results are presented comparing the times needed for transposition using optimal partitioning and powers-of-2 partitioning. It is found in the cases tested that a speed gain of up to 5.55 is obtained by optimizing the partition sizes.