Modern High Performance Computing (HPC) applications, such as Earth science simulations, produce large amounts of data due to the surging of computing power, while big data applications have become more compute-intensive due to increasingly sophisticated analysis algorithms. The needs of both HPC and big data technologies for advanced HPC and big data applications create a demand for integrated system support. In this study, we introduce Scientific Data Processing (SciDP) to support both HPC and big data applications via integrated scientific data processing. SciDP can directly process scientific data stored on a Parallel File System (PFS), which is typically deployed in an HPC environment, in a big data programming environment running atop Hadoop Distributed File System (HDFS). SciDP seamlessly integrates PFS, HDFS, and the widely-used R data analysis system to support highly efficient processing of scientific data. It utilizes the merits of both PFS and HDFS for fast data transfer, overlaps computing with data accessing, and integrates R into the data transfer process. Experimental results show that SciDP accelerates analysis and visualization of a production NASA Center for Climate Simulation (NCCS) climate and weather application by 6x to 8x when compared to existing solutions.

Big-Data systems are increasingly important for solving the data-driven problems in many science domains including geosciences. However, existing big- data systems cannot support the efficient processing of self-describing data formats such as NetCDF which are commonly used by scientific communities for data distribution and sharing. This limitation presents a serious hurdle to the further adoption of big-data systems by science domains. This paper presents Kaleido, a solution to this problem by enabling big- data systems to efficiently store and process scientific data. Specifically, it enables Hadoop to directly store NetCDF data on HDFS, and process them in MapReduce using convenient APIs. It also enables Hive to support queries on NetCDF data, transparent to the users. Moreover, it employs optimizations tailored to scientific data, particularly dimension-aware layout which allows efficient execution of subset queries targeting any dimension of the multi- dimensional data. The paper presents a comprehensive evaluation of Kaleido using representative queries on a typical geoscientific dataset. The results show that Kaleido achieves substantial speedup and space saving compared to existing solutions for storing and processing NetCDF data on Hadoop, and it also substantially outperforms the state-of-the-art solutions for supporting subset queries on scientific data.

Machine learning and graph analytics typically process data in an iterative way, reading the same data multiple times and sharing intermediate results across the worker nodes in cluster. Hadoop MapReduce and Spark are two popular open source cluster compute frameworks for large scale data analytics. Apache Spark is currently the state-of-the-art in-memory computation model extending MapReduce by transforming data into RDDs stored in memory. One limitation of Spark, however, lies in the fact that data transformation and distribution is implicitly managed by HDFS. Data locality is not guaranteed for iterative machine learning algorithms which read the same data multiple times. For example, data needed for operations to one worker node might reside in RDDs stored in other worker nodes. The resulting data shuffling becomes a bottleneck when iteratively reading such RDDs. We propose YinMem, a parallel distributed indexed in-memory computation system, bridging the gap between Hadoop ecosystem and HPC by replacing MapReduce with MPI while obtaining the advantage of the distributed data storage. YinMem achieves fair load balancing prior to computation for large sparse matrix by scheduling and distributing indexed data from NoSQL database to the RAM of working nodes. YinMem explores Alluxio as the in-memory storage system and enables efficient data sharing of intermediate results. Preliminary results show that YinMem has achieved 3× speedup to Spark, for computing eigenvalue and eigenvectors of a 16-million scale sparse matrix.

Large data (over Terabyte) are produced by ultra high-resolution Earth science simulations with a long period of time. This creates a challenge to distribute and analyze in an effective, efficient, and scalable way. One key reason is that typical Earth science data are represented in NetCDF, which is not supported by the popular and powerful Hadoop Distribute File System (HDFS) and consequently cannot be analyzed with tools based on HDFS. In this paper, we report a system for visualizing and analyzing Earth science data based on Hadoop and Spark. It transforms data from the format of NetCDF to CSV (Comma Separated Value) that is supported by HDFS and indexes data appropriately to save storage space as well as manipulate flexibly through HIVE, Impala, and SparkSQL. Adaptive subsetting and visualization of cloud resolve model simulation data are used to validate and demonstrate the features of this system.

Big-data systems are increasingly important for solving the data-driven problems in many science domains including geosciences. However, existing big-data systems cannot support the self-describing data formats such as NetCDF which are commonly used by scientific communities for data distribution and sharing. This limitation presents a serious hurdle to the further adoption of big-data systems by science domains and prevents scientific users from leveraging these systems to improve their productivity. This paper presents a solution to this problem by enabling big-data systems to directly store and process scientific data. Specifically, it enables Hadoop to efficiently store NetCDF data on HDFS and process them in MapReduce using convenient APIs. It also enables Hive to support standard queries on NetCDF data, transparently to users. The paper also presents an evaluation of the proposed solution using several representative queries on a typical geoscientific dataset. The results show that the proposed approach achieves substantial speedup (up to 20 times) and space saving (83% reduction), compared to the traditional approach which has to convert NetCDF data to CSV format for Hadoop and Hive to use them.

NoSQL distributed databases have been devised to tackle the challenges resulting from volume, velocity and variety of big data. Graph representation of datasets requires efficient distributed linear algebra operations for large sparse matrix constructed from big data. Storing the transformed matrix into the database not only speeds up the big data analysis process but also facilitates the computation because of indexing. The Hadoop based approach does not natively support iterative algorithms due to data shuffling during each iteration. This paper presents a novel database-based distributed computation architecture bridging the gap between Hadoop and HPC. The novelty results from exploring the indexing capability of D4M (Dynamic Distributed Dimensional Data Model) to support linear algebra operations in a distributed computation environment. The idea is to store input data and intermediate results in associative array format inside Accumulo table to facilitate the data sharing among working nodes. pMatlab is deployed as the parallel computation engine. Our proposed architecture is proved to be lighter, easier and faster than MapReduce based approach. One example application is calculating top k eigenvalues and eigenvectors for large sparse matrix. Experiments on Graph500 benchmark datasets demonstrate 2X speedup of our architecture as compared to HEIGEN (An eigensolver for billion-scale matrices using MapReduce).

The success of the Hadoop MapReduce programming model has greatly propelled research in big data analytics. In recent years, there is a growing interest in the High Performance Computing (HPC) community to use Hadoop-based tools for processing scientific data. This interest is due to the facts that data movement becomes prohibitively expensive, highperformance data analytic becomes an important part of HPC, and Hadoop-based tools can perform large-scale data processing in a time and budget efficient manner. In this study, we propose PortHadoop, an enhanced Hadoop architecture that enables MapReduce applications reading data directly from HPC parallel file systems (PFS). PortHadoop saves HDFS storage space, and, more importantly, avoids the otherwise costly data copying. PortHadoop keeps all the semantics in the original Hadoop system and PFS. Therefore, Hadoop MapReduce applications can run on PortHadoop without code change except that the input file location is in PFS rather than HDFS. Our experimental results show that PortHadoop can operate effectively and efficiently with the PVFS2 and Ceph file systems.

With rapidly growing computing power, ultra high-resolution Earth science simulations with a long period of time are feasible. However, it is still very challenging to distribute and analyze a huge amount of simulation results, which could be over 100TB. One key reason is that typical Earth science data are represented in NetCDF, which is not supported by the popular and powerful Hadoop Distribute File System (HDFS) and consequently cannot be analyzed with tools based on HDFS. In this paper, we propose a Hadoop-based visualization and diagnosis framework for visualizing and analyzing Earth science data. It has a data model to transform data from the format of NetCDF to CSV (Comma Separated Value) that is supported by HDFS. With this model, data can be processed with the operations such as maximize, sum, and subset through HIVE and Cloudera Impala and, therefore, typical diagnoses can be performed. In addition, the framework has a technique to visualize and diagnose HDFS-resident data with the popular visualization and diagnosis tool, IDL. To speed up this process, a concurrent reader is developed to obtain HDFS-resident data. Moreover, a dynamic reader to transfer data from a parallel file system (PFS) to HDFS is developed to efficiently visualize and diagnose PFS-resident data. The cloud resolve mode simulations are used for testing and evaluating this framework.

The wide use of micro bloggers such as Twitter offers a valuable and reliable source of information during natural disasters. The big volume of Twitter data calls for a scalable data management system whereas the semi-structured data analysis requires full-text searching function. As a result, it becomes challenging yet essential for disaster response agencies to take full advantage of social media data for decision making in a near-real-time fashion. In this work, we use Lucene to empower HBase with full-text searching ability to build a scalable social media data analytics system for observing and analyzing human behaviors during the Hurricane Sandy disaster. Experiments show the scalability and efficiency of the system. Furthermore, the discovery of communities has the benefit of identifying influential users and tracking the topical changes as the disaster unfolds. We develop a novel approach to discover communities in Twitter by applying spectral clustering algorithm to retweet graph. The topics and influential users of each community are also analyzed and demonstrated using Latent Semantic Indexing (LSI).

Researchers in various fields are using optical microscopy to acquire very large images, 10000 - 200000 of pixels per side. Optical microscopes acquire these images as grids of overlapping partial images (thousands of pixels per side) that are then stitched together via software. Composing such large images is a compute and data intensive task even for modern machines. Researchers compound this difficulty further by obtaining time-series, volumetric, or multiple channel images with the resulting data sets now having or approaching terabyte sizes. We present a scalable hybrid CPU-GPU implementation of image stitching that processes large image sets at near interactive rates. Our implementation scales well with both image sizes and the number of CPU cores and GPU cards in a machine. It processes a grid of 42 × 59 tiles into a 17 k × 22 k pixels image in 43 s (end-to-end execution times) when using one NVIDIA Tesla C2070 card and two Intel Xeon E-5620 quad-core CPUs, and in 29 s when using two Tesla C2070 cards and the same two CPUs. It also composes and renders the composite image without saving it in 15 s. In comparison, ImageJ/Fiji, which is widely used by biologists, has an image stitching plugin that takes > 3.6 h for the same workload despite being multithreaded and executing the same mathematical operators, it composes and saves the large image in an additional 1.5 h. This implementation takes advantage of coarse-grain parallelism. It organizes the computation into a pipeline architecture that spans CPU and GPU resources and overlaps computation with data motion. The implementation achieves a nearly 10× performance improvement over our optimized non-pipeline GPU implementation and demonstrates near-linear speedup when increasing CPU thread count and increasing number of GPUs.

Exascale computing systems are soon to emerge, which will pose great challenges on the huge gap between computing and I/O performance. Many large-scale scientific applications play an important role in our daily life. The huge amounts of data generated by such applications require highly parallel and efficient I/O management policies. In this paper, we adopt a mission-critical scientific application, GEOS-5, as a case to profile and analyze the communication and I/O issues that are preventing applications from fully utilizing the underlying parallel storage systems. Through in-detail architectural and experimental characterization, we observe that current legacy I/O schemes incur significant network communication overheads and are unable to fully parallelize the data access, thus degrading applications' I/O performance and scalability. To address these inefficiencies, we redesign its I/O framework along with a set of parallel I/O techniques to achieve high scalability and performance. Evaluation results on the NASA discover cluster show that our optimization of GEOS- 5 with ADIOS has led to significant performance improvements compared to the original GEOS-5 implementation.

In the era of petascale computing, more scientific applications are being deployed on leadership scale computing platforms to enhance the scientific productivity. Many I/O techniques have been designed to address the growing I/O bottleneck on large-scale systems by handling massive scientific data in a holistic manner. While such techniques have been leveraged in a wide range of applications, they have not been shown as adequate for many mission critical applications, particularly in data postprocessing stage. One of the examples is that some scientific applications generate datasets composed of a vast amount of small data elements that are organized along many spatial and temporal dimensions but require sophisticated data analytics on one or more dimensions. Including such dimensional knowledge into data organization can be beneficial to the efficiency of data post-processing, which is often missing from exiting I/O techniques. In this study, we propose a novel I/O scheme named STAR (Spatial and Temporal AggRegation) to enable high performance data queries for scientific analytics. STAR is able to dive into the massive data, identify the spatial and temporal relationships among data variables, and accordingly organize them into an optimized multi-dimensional data structure before storing to the storage. This technique not only facilitates the common access patterns of data analytics, but also further reduces the application turnaround time. In particular, STAR is able to enable efficient data queries along the time dimension, a practice common in scientific analytics but not yet supported by existing I/O techniques. In our case study with a critical climate modeling application GEOS-5, the experimental results on Jaguar supercomputer demonstrate an improvement up to 73 times for the read performance compared to the original I/O method.

SUMMARY Unconventional multi‐ and many‐core processors (e.g. IBM ® Cell B.E. TM and NVIDIA ® GPU) have emerged as effective accelerators in trial climate and weather simulations. Yet these climate and weather models typically run on parallel computers with conventional processors (e.g. Intel ®, AMD ®, and IBM) using Message Passing Interface. To address challenges involved in efficiently and easily connecting accelerators to parallel computers, we investigated using IBM's Dynamic Application Virtualization TM (IBM DAV) software in a prototype hybrid computing system with representative climate and weather model components. The hybrid system comprises two Intel blades and two IBM QS22 Cell B.E. blades, connected with both InfiniBand ® (IB) and 1‐Gigabit Ethernet. The system significantly accelerates a solar radiation model component by offloading compute‐intensive calculations to the Cell blades. Systematic tests show that IBM DAV can seamlessly offload compute‐intensive calculations from Intel blades to Cell B.E. blades in a scalable, load‐balanced manner. However, noticeable communication overhead was observed, mainly due to IP over the IB protocol. Full utilization of IB Sockets Direct Protocol and the lower latency production version of IBM DAV will reduce this overhead. Copyright © 2011 John Wiley & Sons, Ltd.

It is our honor to serve as guest editors of this special section of the journal of Concurrency and Computation: Practice and Experience on Frontiers of GPU, Multi- and Many-Core Systems (FGMMS). We are pleased to present nine high-quality contributions in this special issue, where they were first presented at the Frontiers of GPU, Multi- and Many-Core Systems Workshop in conjunction with the 10 th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2010), held from May 17 to 20, 2010, in Melbourne, Victoria, Australia. The invited papers in this special issue represent augmented works drafted at the beginning of 2010 that addressed the issues below. Multicore and many-core microprocessors are being deployed in a broad spectrum of applications including clusters, clouds, and grids. Both conventional multicore and many-core processors, such as Intel Nehalem and IBM Power7 processors, and unconventional many-core processors, such as NVIDIA Tesla and AMD FireStream graphics processing units (GPUs), hold the promise of increasing performance through parallelism. However, GPU approaches in parallelism are distinctly different from those of conventional multicore and many-core processors, which raises new challenges. For example, how do we optimize applications for conventional multicore and many-core processors? How do we re-engineer applications to take advantage of GPUs’ tremendous computing power in a reasonable cost–benefit ratio? What are effective ways of using GPUs as accelerators? Enormous and rapid progress has been made in accelerator computing over the last two years, but nevertheless we believe the themes developed for the FGMMS workshop and that are represented here in this special issue are still very important ones. In the last two years we have seen the continued rise of GPU computing and indeed its uptake as multi-GPU systems now dominates the top 10 entries within the Top 500 supercomputing systems list 1. Although there has been something of a shakeout of the accelerator technologies that were prevalent three years ago, we have also seen the continued and steady rise of uptake of multicore conventional CPU devices, and an exciting future with combined multicore CPU and GPU devices seems likely. As the articles in this special issue suggest, there are still challenges ahead for application developers to make best use of these future and hybrid highly concurrent systems. There are however still many really important applications that will continue to drive interest, investment, and development of these technologies. The goals of this special issue are to discuss these and other issues and bring together developers of application algorithms and experts in utilizing multicore and many-core processors. We briefly introduce the articles as follows. El Zein and Rendell 2 explore the effect of different GPU programming options (e.g., memory type, memory access methods, and data types) on the performance of routine evaluating sparse matrix vector products and discuss the method for optimal performance. Playne and Hawick 3 report on their approach in accelerating finite-differencing applications using multiple GPU devices with a single CPU host and the asynchronous CPU/GPU communication. Kato and Hosino 4 present their algorithms for speeding up a k-nearest neighbor problem in the recommendation system through multiple GPUs. Ino et al. 5 discuss a cooperative multitasking method for concurrent execution of scientific and graphics applications on GPU and acceleration of compute unified device architecture-based applications using idle GPU cycles in the office. Gillan et al. 6 present a case study on how the instruction-level parallelism offered by three accelerator technologies — field-programmable gate array, GPU, and ClearSpeed — can be exploited in atomic physics with considerable differences in the implementation strategies. Barhen et al. 7 present an unconventional fast Fourier transform implementation scheme for the IBM Cell B.E. processors, named transverse vectorization, and provide the first results for multifast Fourier transform implementation and application on the novel, ultralow power Coherent LogixHyperX processor. Zhou et al. 8 investigate the software and system issues in accelerating climate and weather models in a prototype hybrid computing system, which comprises Intel blades and IBM Cell B.E. blades, connected with both InfiniBand and 1-Gigabit Ethernet and communicate with IBM's Dynamic Application Virtualization software. Qiu and Bae 9 present performance results of two significant bioinformatics applications, gene clustering and dimension reduction, on a Microsoft Windows cluster with up to 768 cores using Message Passing Interface and two variants of threading — Concurrency and Coordination Runtime and Task Parallel Library. Hackenberg et al. 10 present a tool for the graphical program flow analysis of hardware accelerated parallel programs. The tool monitors the hybrid program execution to record and visualize many performance relevant events along the way, and is exemplified through representative real-world applications written for both IBM Cell B.E. processor and NVIDIA Compute Unified Device Architecture (CUDA) API. This work was supported in part by a Microsoft CRMC grant. The guest editors of this special issue would like to express their deep gratitude to all authors, external reviewers, and Geoffrey Fox for their efforts in making this issue possible.

The goal of this special issue is to address such issues by assembling some of the latest researches on efficiently utilizing multicore and many-core processors in real-world applications, and their strategies for coping with those challenges. Multicore (e.g., Intel Westmere and IBM Power7) and many-core (e.g., NVIDIA Tesla and AMD FireStream graphics processing units (GPUs)) microprocessors are enabling more compute-intensive and data-intensive computation in desktop computers, clusters, and leadership supercomputers. However, efficient utilization of these microprocessors is still a very challenging issue. Their differing architectures require significantly different programming paradigms when adapting real-world applications. The actual porting costs are actively debated, as well as the relative performance between GPUs and CPUs. The invited papers in this special issue represent amplified works originally presented at the Frontiers of Multicore Computing Conference 2010, held at the University of Maryland, Baltimore County in August 2010. The selected papers cover representative research addressing the issues above. There are four papers addressing the issues related to CPUs and two on GPUs. Seelam et al. report their experiences in building and scaling enterprise business analytics benchmark, report generation, and rendering, on an IBM Power7 multicore system with eight Power7 cores and 32 hardware threads 1. The paper by Tracy and Brown presents a multithreaded physics software design to eliminate overhead associated with bodies at rest and consequently accelerate physics simulation in large, continuous virtual environments on Intel multicore processors 2. The paper by Hammond et al. presents multilevel performance analysis for the computational chemistry software, NW Chem, in Blue Gene/P and on two large-scale clusters 3. The paper by Simon et al. provides a performance evaluation and investigation of the astrophysics code, FLASH, for a variety of Intel multicore processors 4. The paper by Blattner and Yang presents the key steps in porting one data assimilation algorithm to GPU 5. The paper by Malik et al. examines the programming paradigms of Compute Unified Device Architecture (CUDA), OpenCL, The Portland Group Accelerator Compiler (PGI), and MATLAB through developing kernels from the Numerical Aerodynamics Simulation (NAS) parallel benchmarking suite 6. The guest editors of this special issue would like to express their deep gratitude to all authors, external reviewers, and Geoffrey Fox for their efforts in making this issue possible.

MapReduce, which was introduced by Google, provides two functional interfaces, Map and Reduce, for a user to write the user-specific code to process the large amount of data. It has been widely deployed in cloud computing systems. The parallel tasks, data partition, and data transit are automatically managed by its runtime system. This paper proposes a solution to optimize the MapReduce program model and demonstrate it with X10. We develop an adaptive load distribution scheme to balance the load on each node and consequently reduce across-node communication cost occurring in the Reduce function. In addition, we exploit shared-memory in each node to further reduce the communication cost with multi-core programming.

To maximize the performance of emerging multi- and many-core accelerators such as the IBM Cell B.E. and the NVIDIA GPU, a Memory Centric Kernel Framework (MCKF) was developed. MCKF allows a user to decompose the physical space of an application based on the available fast memory in the accelerators. In this way, reducing the communication cost in accessing data can maximize the extraordinary computing power of the accelerators. MCKF is both generic and flexible because it encapsulates hardware-specific characteristics. It has been implemented and tested for short-range inter-active particle simulation on IBM Cell B.E. blades.

Unconventional multi- and many-core processors (e.g., IBM® Cell B.E. ™ and NVIDIA® GPU) have emerged as effective accelerators in trial climate and weather simulations. Yet these climate and weather models typically run on parallel computers with conventional processors (e.g., Intel®, AMD®, and IBM) using Message Passing Interface (MPI). To address challenges involved in efficiently and easily connecting accelerators to parallel computers, we investigated using IBM's Dynamic Application Virtualization (DAV) software in a prototype hybrid computing system with representative climate and weather model components. The hybrid system comprises 2 Intel blades and 2 IBM QS22 Cell B.E. blades, connected with both InfiniBand® (IB) and 1-Gigabit Ethernet. The system significantly accelerates a solar radiation model component by offloading compute-intensive calculations to the Cell blades. Systematic tests show that DAV can seamlessly offload compute-intensive calculations from Intel blades to Cell B.E. blades in a scalable, load-balanced manner. However, noticeable communication overhead was observed, mainly due to IP over IB protocol. Full utilization of IB Sockets Direct Protocol (SDP) and the lower latency production version of DAV will reduce this overhead.

Cloud computing is emerging as an alternative computing platform to bridge the gap between scientists' growing computational demands and their computing capabilities. A scientist who wants to run HPC applications can obtain massive computing resources 'in the cloud' quickly (in minutes), as opposed to days or weeks it normally takes under traditional business processes. Due to the popularity of Amazon EC2, most HPC-in-the-cloud research has been conducted using EC2 as a target platform. Previous work has not investigated how results might depend upon the cloud platform used. In this paper, we extend previous research to three public cloud computing platforms. In addition to running classical benchmarks, we also port a 'full-size' NASA climate prediction application into the cloud, and compare our results with that from dedicated HPC systems. Our results show that 1) virtualization technology, which is widely used by cloud computing, adds little performance overhead; 2) most current public clouds are not designed for running scientific applications primarily due to their poor networking capabilities. However, a cloud with moderately better network (vs. EC2) will deliver a significant performance improvement. Our observations will help to quantify the improvement of using fast networks for running HPC-in-the-cloud, and indicate a promising trend of HPC capability in future private science clouds. We also discuss techniques that will help scientists to best utilize public cloud platforms despite current deficiencies.

Abstract The call for ever‐increasing model resolutions and physical processes in climate and weather models demands a continual increase in computing power. The IBM Cell processor's order‐of‐magnitude peak performance increase over conventional processors makes it very attractive to fulfill this requirement. However, the Cell's characteristics, 256 kB local memory per SPE and the new low‐level communication mechanism, make it very challenging to port an application. As a trial, we selected the solar radiation component of the NASA GEOS‐5 climate model, which: (1) is representative of column‐physics components (half of the total computational time), (2) has an extremely high computational intensity: the ratio of computational load to main memory transfers, and (3) exhibits embarrassingly parallel column computations. In this paper, we converted the baseline code (single‐precision Fortran) to C and ported it to an IBM BladeCenter QS20. For performance, we manually SIMDize four independent columns and include several unrolling optimizations. Our results show that when compared with the baseline implementation running on one core of Intel's Xeon Woodcrest, Dempsey, and Itanium2, the Cell is approximately 8.8x, 11.6x, and 12.8x faster, respectively. Our preliminary analysis shows that the Cell can also accelerate the dynamics component (∼25% total computational time). We believe these dramatic performance improvements make the Cell processor very competitive as an accelerator. Copyright © 2009 John Wiley & Sons, Ltd.

Recent trends in computer microprocessor development have shifted from a single powerful core to multi- and many-cores. As a result, the continuous improvement in computing power fueled by the exponentially increasing speed of a single processor could be over. With applications such as Earth and space sciences, higher resolutions and more sophisticated treatments of physical processes make models even more computationally intensive. Moreover, the drastic increase of data collected by various instruments requires a significant increase in computing power for data processing and analysis. Therefore, it is crucial for the computational science and technology community to evaluate the impacts of this shift on computationally intensive modeling and data processing applications and to develop appropriate solutions. It is known that the computing power of conventional processors is limited by memory bandwidth. Adding more cores to the processors worsens the problem. It is necessary to adapt computing algorithms to effectively utilize the computing power of those conventional multi- and many-core processors. In the last two years, there have emerged few unconventional processors: IBM's Cell Broadband Engine (hereafter referred to as Cell) and NVIDIA's Graphics Processing Unit (GPU). Intel and AMD are also developing competing Cell- or GPU-like processors, in addition to conventional multi- and many-core processors. It has been demonstrated that certain computationally intensive applications with moderate data communication can benefit from both Cell and GPU with a significant performance improvement. However, these emerging processors require new programming paradigms, which increase the porting costs and impede their effective utilization. To address such issues with the unconventional multi- and many-core processors, this special issue assembles some of the latest research on assessing the impacts of multi- and many-core processors, developing appropriate solutions, and taking advantage of the abundant computing power. The invited papers in this special issue represent augmented works originally presented at the Frontiers of Multicore Computing Conference 2008, held at the University of Maryland, Baltimore County in August 2008. Of course, it is not possible for a single issue to include all the topics addressed by the conference. However, the selected papers cover representative research addressing the issues above. There are four and two papers addressing the issues related to Cell and multi-core processors, respectively. The paper by Germann et al. presents their approaches and optimization methods in porting a short-range parallel molecular dynamics code to the first petaflops computer, Roadrunner, which uses Cells as accelerators 1. The paper by Woodward et al. reports their initial experience with porting and optimizing gas dynamics simulation on Roadrunner 2. The paper by Zhou et al. presents a study on the impact of Cell on the programming paradigm for climate and weather models 3. The paper by Brown et al. presents a contemporary artwork, the ‘Scalable City,’ accelerated with Cell 4. The paper by Wang and JaJa presents a study on interactive direct volume rendering with Intel Clovertown quad-core processors 5. The paper by Simmon and McGalliard provides benchmark results for a variety of high-performance computing applications on the Cray XT3 and XT4, which use dual- and quad-core AMD Opteron processors, and discusses and analyzes multi-core effects 6. The guest editors of this special issue would like to express their deep gratitude to all authors, external reviewers, and Geoffrey Fox for their efforts in making this issue possible.

Filesystem I/O continues to be a major performance bottleneck for many high-end computing (HEC) applications and in particular for Earth science models, which often generate a relatively large volume of data for a given amount of computational work. The severity of this I/O bottleneck rapidly increases with the number of processors utilized. Consequently, considerable computing resources are wasted, and the sustained performance of HEC applications such as climate and weather models is highly constrained. To alleviate much of this bottleneck, we have developed a lightweight software utility designed to improve performance of typical scientific applications by circumventing bandwidth limitations of typical HEC filesystems. The approach is to exploit the faster inter- processor bandwidth to move output data from compute nodes to designated I/O nodes as quickly as possible, thereby minimizing the I/O wait time. This utility has successfully demonstrated a significant performance improvement within a major NASA weather application.

Abstract Typical weather and climate models need a software tool to couple sub‐scale model components. The high‐performance computing requirements and a variety of model interfaces make the development of such a coupling tool very challenging. In this paper, we describe the approach of the Earth System Modeling Framework, in particular its component and coupling mechanism, and present the results of three cross‐organization model interoperability experiments. Copyright © 2006 John Wiley & Sons, Ltd.

Abstract High‐speed network and Grid computing have been actively investigated, and their capabilities are being demonstrated. However, their application to high‐end scientific computing and modeling is still to be explored. In this paper we discuss the related issues and present our prototype work on applying XCAT3 framework technology to geomagnetic data assimilation development with distributed computers, connected through an up to 10 Gigabit Ethernet network. Copyright © 2006 John Wiley & Sons, Ltd.

Filesystems continue to be a major performance bottleneck for many applications across a variety of hardware architectures. Most existing attempts to address this issue, e.g., PVFS, rely upon system resources which are not typically tuned for any specific user application, whereas others rely on special hardware capabilities such as shared-memory.We have developed an MPI-based parallel asynchronous I/O (PAIO) software package which enables applications to balance compute and I/O resources directly. PAIO uses a queueing mechanism to stage the data, sent over in parallel from compute nodes, on the reserved I/O nodes. Because the bandwidth of the inter-processor network greatly exceeds that of the filesystem, significant performance improvements can be achieved under a bursty I/O load provided sufficient memory is available for the I/O nodes. The results of PAIO for typical weather applications on an SGI Altix and other architectures will be reported in a poster board.

Abstract Typical Earth system models involve coupled model components in high‐performance computing (HPC) environments. In the last few years, several frameworks have been developed for HPC applications. Two of them are component‐based frameworks: the Earth System Modeling Framework (ESMF) defining a component interface for Earth system models and the Common Component Architecture (CCA) defining a generic component model. The purpose of this work is to investigate the relationship between ESMF and CCA to deploy the best features of ESMF and CCA into a ESMF–CCA prototype and examine the prototype through a representative coupled climate model. Copyright © 2005 John Wiley & Sons, Ltd.