Generative AI (GenAI) image tools are increasingly integrated into design workflows, prompting HCI research on their interaction methods and interfaces. We reviewed 37 such tools, including 28 HCI research systems and nine commercial systems (2022–July 2025), using three analytical frameworks: interaction methods, creative processes, and tool functionalities. We found that text prompts remain the dominant input method, while visual and attribute-based inputs—particularly in academic tools—are gaining traction and are often combined with text for refinement. Commercial systems emphasize parameter control, whereas academic tools focus on semantic attributes and visual organization. Most tools support ideation and exploration, but provide limited support for refinement and evaluation. Based on these findings, we identify nine design opportunities, including advanced visual interaction, simplified parameter control, precision editing, direct manipulation, workflow integration, default settings that support rapid exploration, and user guidance for later stages. We contribute a framework for analyzing GenAI interfaces and actionable directions for designing more usable, creativity-supportive GenAI image systems.

Generative AI (GenAI) image tools are increasingly used in design practice, enabling rapid ideation but offering limited support for refinement tasks such as adjusting layout, scale, or visual attributes. While text prompts and inpainting allow localized edits, they often remain inefficient or ambiguous for precise, in-context, and iterative refinement—motivating the exploration of alternative methods. This work examines how pen-based scribbles and annotations can enhance GenAI image refinement. A formative study with seven professional designers informed a prototype supporting three input modalities: text-only, visual-only, and combined prompting. A within-subjects study with 30 designers and design students compared these modalities across closed- and open-ended tasks, evaluating expressiveness, efficiency, workload, user experience, iteration, and multimodal strategies. Visual prompts improved clarity and speed for spatial edits while reducing workload, whereas text remained effective for semantic and global changes. The combined modality received the highest overall ratings, enabling complementary use, balancing spatial precision with semantic detail, and supporting smoother iteration. Task-specific preferences also emerged: adding new objects often required both modalities, while moving or modifying elements was typically handled through visual input. This work contributes (1) an empirical comparison of multimodal prompting for GenAI refinement, (2) a prototype integrating scribbles and annotations, and (3) insights into designers’ multimodal strategies to inform future GenAI interfaces that better support refinement in GenAI-supported design workflows.

As quantum hardware advances, integrating quantum processing units (QPUs) into HPC environments and managing diverse infrastructure and software stacks becomes increasingly essential. Pilot-Quantum addresses these challenges as a middleware designed to provide unified application-level management of resources and workloads across hybrid quantumclassical environments. It is built on a rigorous analysis of existing quantum middleware systems and application execution patterns. It implements the Pilot Abstraction conceptual model, originally developed for HPC, to manage resources, workloads, and tasks. It is designed for quantum applications that rely on task parallelism, including (i) Hybrid algorithms, such as variational approaches, and (ii) Circuit cutting systems, used to partition and execute large quantum circuits. Pilot-Quantum facilitates seamless integration of QPUs, classical CPUs, and GPUs, while supporting high-level programming frameworks like Qiskit and Pennylane. This enables users to efficiently design and execute hybrid workflows across diverse computing resources. The capabilities of Pilot-Quantum are demonstrated through mini-apps - simplified yet representative kernels focusing on critical performance bottlenecks. We demonstrate the capabilities of Pilot-Quantum through multiple mini-apps, including different circuit execution (e.g., using IBM's Eagle QPU and simulators), circuit-cutting, and quantum machine learning scenarios.

Large Language Models (LLMs) have experienced widespread adoption across scientific and industrial domains due to their versatility and utility for diverse tasks. Nevertheless, deploying and serving these models at scale with optimal throughput and latency remains a significant challenge, primarily because of LLMs’ high computational and memory demands. Specialized models optimized for specific tasks can be combined through a routing mechanism to address these challenges, creating a modular inference system. This paper introduces Expert Router, a scalable routing architecture that directs prompts to specialized expert models. We characterize multiple Expert Router configurations, including different LLama 3 models with quantized and non-quantized weights under up to 1,000 concurrent users. Our findings reveal that Expert Router introduces minimal latency overhead, with the configuration of expert models being a dominating factor in performance outcomes. High-parameter expert models deliver stable throughput and latency under moderate concurrency levels. In contrast, smaller expert models maintain competitive performance across a wider range of concurrent users compared to tensor-parallelized baseline models. This highlights the potential of Expert Router for efficient and scalable LLM deployment.

Large Language Model (LLM) routing architectures allow to manage and deploy multiple fine-tuned expert models through a single inference endpoint. However, a key challenge is that individually fine-tuned models may suffer from limited generation diversity and can produce hallucinations when responses extend beyond their specialization domains. To address this, we introduce the Dynamic Expert Router, an architecture that dynamically routes token generation across expert LLMs based on domain classification. The system monitors the generated output from expert LLMs and reroutes to a different model if the response begins to shift away from the relevant domain. We present preliminary experiments, in which we identify domain shifts during generation by analyzing the response sentences of a Llama 3 model. These findings guide our research questions on optimizing embedding techniques, clustering methods, and routing mechanisms.

Quantum generative modeling is a growing area of interest for industry-relevant applications. This work systematically compares a broad range of techniques to guide quantum computing practitioners when deciding which models and methods to use in their applications. We compare fundamentally different architectural ansatzes of parametric quantum circuits: (1) A continuous architecture, which produces continuous-valued data samples, and (2) a discrete architecture, which samples on a discrete grid. We also compare the performance of different data transformations: the min-max and the probability integral transforms. We use two popular training methods: (1) quantum circuit Born machines (QCBM), and (2) quantum generative adversarial networks (QGAN). We study their performance and tradeoffs as the number of model parameters increases, with a baseline comparison of similarly trained classical neural networks. The study is performed on six low-dimensional synthetic and two real financial data sets. Our two key findings are that: (1) For all data sets, our quantum models require similar or fewer parameters than their classical counterparts. In the extreme case, the quantum models require two orders of magnitude less parameters. (2) We empirically find that a variant of the discrete architecture, which learns the copula of the probability distribution, outperforms all other methods.

Generative Adversarial Networks (GANs) have shown potential for generating images, but have limitations when applied to complex datasets. To address these limitations, class-conditional training is employed, as it performs better and maintains a high level of semantic diversity. In this work, we propose a new method for training generative models on complex images by extracting rules defining the relationships between objects in the image, cropping significant sub-regions based on these rules, and training the models in a conditional setting using the extracted rules as labels. The proposed approach is evaluated and the results demonstrate its effectiveness by increasing the training dataset size, and then feeding it to conditional training. As a result, synthesized samples maintain asset fine-grained details and the visibility of small instances.

Privacy-enhancing technologies (PETs) are becoming increasingly crucial for addressing customer needs, security, privacy (e. g., enhancing anonymity and confidentiality), and regulatory requirements. However, applying PETs in organizations requires a precise understanding of use cases, technologies, and limitations. This paper investigates several industrial use cases, their characteristics, and the potential applicability of PETs to these. We conduct expert interviews to identify and classify uses cases, a gray literature review of relevant open-source PET tools, and discuss how the use case characteristics can be addressed using PETs’ capabilities. While we focus mainly on automotive use cases, the results also apply to other use case domains.

A vast and growing number of IoT applications connect physical devices, such as scientific instruments, technical equipment, machines, and cameras, across heterogenous infrastructure from the edge to the cloud to provide responsive, intelligent services while complying with privacy and security requirements. However, the integration of heterogeneous IoT, edge, and cloud technologies and the design of end-to-end applications that seamlessly work across multiple layers and types of infrastructures is challenging. A significant issue is resource management and the need to ensure that the right type and scale of resources is allocated on every layer to fulfill the application's processing needs. As edge and cloud layers are increasingly tightly integrated, imbalanced resource allocations and sub-optimally placed tasks can quickly deteriorate the overall system performance. This paper proposes an emulation approach for the investigation of task placements across the edge-to-cloud continuum. We demonstrate that emulation can address the complexity and many degrees-of-freedom of the problem, allowing us to investigate essential deployment patterns and trade-offs. We evaluate our approach using a machine learning-based workload, demonstrating the validity by comparing emulation and real-world experiments. Further, we show that the right task placement strategy has a significant impact on performance - in our experiments, between 5% and 65% depending on the scenario.

Summary We investigate the challenges of building an end‐to‐end cloud pipeline for real‐time intelligent visual inspection system for use in automotive manufacturing. Current methods of visual detection in automotive assembly are highly labor intensive, and thus prone to errors. An automated process is sought that can operate within the real‐time constraints of the assembly line and can reduce errors. Components of the cloud pipeline include capture of a large set of high‐definition images from a camera setup at the assembly location, transfer and storage of the images as needed, execution of object detection, and notification to a human operator when a fault is detected. The end‐to‐end execution must complete within a fixed time frame before the next car arrives in the assembly line. In this article, we report the design, development, and experimental evaluation of the tradeoffs of performance, accuracy, and scalability for a cloud system.

Industrial and scientific streaming applications require support for different types of processing and the management of heterogeneous infrastructure over a dynamic range of scales: from the edge to the cloud and HPC, and intermediate resources. Serverless is an emerging service that combines high-level middleware services, such as distributed execution engines for managing tasks, with low-level infrastructure. It offers the potential of usability and scalability but adds to the complexity of managing heterogeneous and dynamic resources. In response, we extend Pilot-Streaming to support serverless platforms. Pilot-Streaming provides a unified abstraction for resource management for HPC, cloud, and serverless, and allocates resource containers independent of the application workload removing the need to write resource-specific code. Understanding the performance and scaling characteristics of streaming applications and infrastructure presents another challenge. StreamInsight provides insight into the performance of streaming applications and infrastructure, their selection, configuration, and scaling behavior. Underlying StreamInsight is the universal scalability law, which permits the accurate quantification of scalability properties of streaming applications. Using experiments on HPC and AWS Lambda, we demonstrate that StreamInsight provides an accurate model for a variety of application characteristics, e. g., machine learning model sizes and resource configurations.

The Internet-of-Things (IoT) is growing in importance enabling an increasing number of scientific, industrial, and societal applications. At the same time, the computational capabilities of IoT and edge devices are rapidly improving making them viable for machine learning. Thus, the deployment of machine learning models on the edge is becoming a critical capability. However, such deployments are challenging as edge devices are more resource-constrained than clouds and cannot elastically scale on demand. Moving the application to the cloud can provide more computational power, but raises other challenges, such as security, reliability, and bandwidth. The integration of edge and cloud computing resources is often essential for many applications allowing them to choose the best configuration with respect to their requirements and characteristics, such as data rates and computational complexity. To understand the trade-offs between edge and cloud computing the assessment of different cloud and edge configurations is required. In this paper, we present EdgeInsight, a framework for characterizing and modeling of the inference performance of edge and cloud infrastructures. EdgeInsight enables system builders to size and fine-tune infrastructure parameters for different workloads and applications. We use EdgeInsight to qualitatively and quantitatively study edge and cloud deployment configuration for deep learning inference. Our evaluation shows, that edge inference can outperform cloud inference when model architecture and accuracy, inference framework and pre-processing parameters are carefully selected.

Artificial Intelligence (AI) and Deep Learning has been steadily gaining importance due it’s potential for a broad set of science and industry applications. The success of deep learning techniques has found many applications, e.g. in the domain of computer vision and natural language understanding. Developing AI applications is a complex task with many challenges related to data collection, model training, and deployment.In this paper, we evaluate architectures, models and deployment issues related to the usage of deep learning techniques in the automotive manufacturing domain. Particularly, we focus on different computer vision problems in automotive manufacturing processes, e.g., in logistics processes. We developed several deep learning models that help to improve the quality and efficiency of these processes. Finally, we provide an analysis of the architecture, datasets and models used, and provide performance metrics for each of the different models.

Deep learning models are associated with various deployment challenges. Inference of such models is typically very compute-intensive and memory-intensive. In this paper, we investigate the performance of deep learning models for a computer vision application used in the automotive manufacturing industry. This application has demanding requirements that are characteristic of Big Data systems, including high volume and high velocity. The application has to process a very large set of high-definition images in real-time with appropriate accuracy requirements using a deep learning-based object detection model. Meeting the run time, accuracy, and resource requirements require a careful consideration of the choice of model, model parameters, hardware, and environmental support. In this paper, we investigate the trade-offs of the most popular deep neural network-based object detection models on four hardware platforms. We report the trade-offs of resource consumption, run time, and accuracy for a realistic real-time application environment.

This paper presents a systematic evaluation of Amazon Kinesis and Apache Kafka for meeting highly demanding application requirements. Results show that Kinesis and Kafka can provide high reliability, performance and scalability. Cost and performance trade-offs of Kinesis and Kafka are presented for a variety of application data rates, resource utilization, and resource configurations.

An increasing number of scientific applications utilize stream processing to analyze data feeds of scientific instruments, sensors, and simulations. In this paper, we study the streaming and data processing requirements of light source experiments, which are projected to generate data at 20 GB/sec in the near future. As beamtimes available to users are typically short, it is essential that processing and analysis can be conducted in a streaming mode. The development and deployment of streaming applications is a complex task and requires the integration of heterogeneous, distributed infrastructure, frameworks, middleware and application components written in different languages and abstractions. Streaming applications may be extremely dynamic due to factors, such as variable data rates, network congestions, and application-specific characteristics, such as adaptive sampling techniques and the different processing techniques. Consequently, streaming system are often subject to back-pressures and instabilities requiring additional infrastructure to mitigate these issues. We propose Pilot-Streaming, a framework for supporting streaming applications and their resource management needs on HPC infrastructure. Underlying Pilot-Streaming is a unifying architecture that decouples important concerns and functions, such as message brokering, transport and communication, and processing. Pilot-Streaming simplifies the deployment of stream processing frameworks, such as Kafka and Spark Streaming, while providing a high-level abstraction for managing streaming infrastructure, e. g. adding/removing resources as required by the application at runtime. This capability is critical for balancing complex streaming pipelines. To address the complexity in the development of streaming applications, we present the Streaming Mini-Apps, which supports different plug-able algorithms for data generation and processing, e. g., for reconstructing light source images using different techniques. We use the streaming Mini-Apps to evaluate the Pilot-Streaming framework demonstrating its suitability for different use cases and workloads.

Different parallel frameworks for implementing data analysis applications have been proposed by the HPC and Big Data communities. In this paper, we investigate three task-parallel frameworks: Spark, Dask and RADICAL-Pilot with respect to their ability to support data analytics on HPC resources and compare them to MPI. We investigate the data analysis requirements of Molecular Dynamics (MD) simulations which are significant consumers of supercomputing cycles, producing immense amounts of data. A typical large-scale MD simulation of a physical system of O(100k) atoms over μsecs can produce from O(10) GB to O(1000) GBs of data. We propose and evaluate different approaches for parallelization of a representative set of MD trajectory analysis algorithms, in particular the computation of path similarity and leaflet identification. We evaluate Spark, Dask and RADICAL-Pilot with respect to their abstractions and runtime engine capabilities to support these algorithms. We provide a conceptual basis for comparing and understanding different frameworks that enable users to select the optimal system for each application. We also provide a quantitative performance analysis of the different algorithms across the three frameworks.

A complex and vast amount of data will be collected from on-board sensors of operational connected vehicles (CVs), infrastructure data sources such as roadway sensors and traffic signals, mobile data sources such as cell phones, social media sources such as Twitter, and news and weather data services. Unfortunately, these data will create a bottleneck at data centers for processing and retrievals of collected data, and will require the deployment of additional message transfer infrastructure between data producers and consumers to support diverse CV applications. In this paper, we present a strategy for creating an efficient and low-latency distributed message delivery system for CV applications using a distributed message delivery platform. This strategy enables large-scale ingestion, curation, and transformation of unstructured data (roadway traffic-related and roadway non-traffic-related data) into labeled and customized topics for a large number of subscribers or consumers, such as CVs, mobile devices, and data centers. We evaluate the performance of this strategy by developing a prototype infrastructure using Apache Kafka, an open source message delivery system, and compared its performance with the latency requirements of CV applications. We present experimental results of the message delivery infrastructure on two different distributed computing testbeds at Clemson University: the Holocron cluster and the Palmetto cluster. Experiments were performed to measure the latency of the message delivery system for a variety of testing scenarios. These experiments reveal that measured latencies are less than the U.S. Department of Transportation recommended latency requirements for CV applications, which prove the efficacy of the system for CV related data distribution and management tasks.

Common Crawl is a massive multi-petabyte dataset hosted by Amazon. It contains archived HTML web page data from 2008 to date. Common Crawl has been widely used for text mining purposes. Using data extracted from Common Crawl has several advantages over a direct crawl of web data, among which is removing the likelihood of a user's home IP address becoming blacklisted for accessing a given web site too frequently. However, Common Crawl is a data sample, and so questions arise about the quality of Common Crawl as a representative sample of the original data. We perform systematic tests on the similarity of topics estimated from Common Crawl compared to topics estimated from the full data of online forums. Our target is online discussions from a user forum for automotive enthusiasts, but our research strategy can be applied to other domains and samples to evaluate the representativeness of topic models. We show that topic proportions estimated from Common Crawl are not significantly different than those estimated on the full data. We also show that topics are similar in terms of their word compositions, and not worse than topic similarity estimated under true random sampling, which we simulate through a series of experiments. Our research will be of interest to analysts who wish to use Common Crawl to study topics of interest in user forum data, and analysts applying topic models to other data samples.

Summary A common feature across many science and engineering applications is the amount and diversity of data and computation that must be integrated to yield insights. Datasets are growing larger and becoming distributed; their location, availability, and properties are often time‐dependent. Collectively, these characteristics give rise to dynamic distributed data‐intensive applications. While “static” data applications have received significant attention, the characteristics, requirements, and software systems for the analysis of large volumes of dynamic, distributed data, and data‐intensive applications have received relatively less attention. This paper surveys several representative dynamic distributed data‐intensive application scenarios, provides a common conceptual framework to understand them, and examines the infrastructure used in support of applications.

Deep Learning refers to a set of machine learning techniques that utilize neural networks with many hidden layers for tasks, such as image classification, speech recognition, language understanding. Deep learning has been proven to be very effective in these domains and is pervasively used by many Internet services. In this paper, we describe different automotive uses cases for deep learning in particular in the domain of computer vision. We surveys the current state-of-the-art in libraries, tools and infrastructures (e. g. GPUs and clouds) for implementing, training and deploying deep neural networks. We particularly focus on convolutional neural networks and computer vision use cases, such as the visual inspection process in manufacturing plants and the analysis of social media data. To train neural networks, curated and labeled datasets are essential. In particular, both the availability and scope of such datasets is typically very limited. A main contribution of this paper is the creation of an automotive dataset, that allows us to learn and automatically recognize different vehicle properties. We describe an end-to-end deep learning application utilizing a mobile app for data collection and process support, and an Amazon-based cloud backend for storage and training. For training we evaluate the use of cloud and on-premises infrastructures (including multiple GPUs) in conjunction with different neural network architectures and frameworks. We assess both the training times as well as the accuracy of the classifier. Finally, we demonstrate the effectiveness of the trained classifier in a real world setting during manufacturing process.

Data is increasingly affecting the automotive industry, from vehicle development, to manufacturing and service processes, to online services centered around the connected vehicle. Connected, mobile and Internet of Things devices and machines generate immense amounts of sensor data. The ability to process and analyze this data to extract insights and knowledge that enable intelligent services, new ways to understand business problems, improvements of processes and decisions, is a critical capability. Hadoop is a scalable platform for compute and storage and emerged as de-facto standard for Big Data processing at Internet companies and in the scientific community. However, there is a lack of understanding of how and for what use cases these new Hadoop capabilities can be efficiently used to augment automotive applications and systems. This paper surveys use cases and applications for deploying Hadoop in the automotive industry. Over the years a rich ecosystem emerged around Hadoop comprising tools for parallel, in-memory and stream processing (most notable MapReduce and Spark), SQL and NOSQL engines (Hive, HBase), and machine learning (Mahout, MLlib). It is critical to develop an understanding of automotive applications and their characteristics and requirements for data discovery, integration, exploration and analytics. We then map these requirements to a confined technical architecture consisting of core Hadoop services and libraries for data ingest, processing and analytics. The objective of this paper is to address questions, such as: What applications and datasets are suitable for Hadoop? How can a diverse set of frameworks and tools be managed on multi-tenant Hadoop cluster? How do these tools integrate with existing relational data management systems? How can enterprise security requirements be addressed? What are the performance characteristics of these tools for real-world automotive applications? To address the last question, we utilize a standard benchmark (TPCx-HS), and two application benchmarks (SQL and machine learning) that operate on a dataset of multiple Terabytes and billions of rows.

We review the High Performance Computing Enhanced Apache Big Data Stack HPC-ABDS and summarize the capabilities in 21 identified architecture layers. These cover Message and Data Protocols, Distributed Coordination, Security & Privacy, Monitoring, Infrastructure Management, DevOps, Interoperability, File Systems, Cluster & Resource management, Data Transport, File management, NoSQL, SQL (NewSQL), Extraction Tools, Object-relational mapping, In-memory caching and databases, Inter-process Communication, Batch Programming model and Runtime, Stream Processing, High-level Programming, Application Hosting and PaaS, Libraries and Applications, Workflow and Orchestration. We summarize status of these layers focusing on issues of importance for data analytics. We highlight areas where HPC and ABDS have good opportunities for integration.

We study the two approaches, rHadoop and H2O, to intergate R, a popular statistical programming environment, into the Hadoop Big Data ecosystem. Using these approaches and the vanilla implementation of MapReduce to implement the solution to an analytic question for the on-time airline performance data set, we evaluate the differences in runtime performance and elaborate on the causes of these differences based on rHadoop and H2O's design principles.

The concept of Internet of Things (IoT) is rapidly moving from a vision to being pervasive in our everyday lives. This can be observed in the integration of connected sensors from a multitude of devices such as mobile phones, healthcare equipment, and vehicles. There is a need for the development of infrastructure support and analytical tools to handle IoT data, which are naturally big and complex. But, research on IoT data can be constrained by concerns about the release of privately owned data. In this paper, we present the design and implementation results of a synthetic IoT data generation framework. The framework enables research on synthetic data that exhibit the complex characteristics of original data without compromising proprietary information and personal privacy.

Current workflow abstractions in general lack: (a) an adequate approach to handle distributed data and (b) proper separation between logical tasks and data-flow from their mapping onto physical locations. As the complexity and dynamism of data and processing distribution have increased, optimized mapping of logical tasks to physical resources have become a necessity to avoid bottlenecks. We argue that the management of dynamic data and compute should become part of the runtime system of workflow engines to enable workflows to scale as necessary to address big data challenges and fully exploit distributed computing infrastructures (DCI). In this paper we explore how the P* model for pilot-abstractions, which proposes a clear separation between the logical compute and data units and their realization as a job or a file in some physical resource, could provide these capabilities for such a runtime environment. The Pilot-API provides a general-purpose interface to pilot-abstractions and the ability to assign compute and data resources to them. We share our experience of using the case study of a DNA sequencing pipeline, to re-implement the workflow using the Pilot-API. This first exercise, which resulted in a running application that is discussed here, illustrates the potential of this API to address (a) and (b). Our initial results indicate that the pilot abstractions (as captured by the P* model)offer an interesting approach to explore the design of a new generation of workflow management systems and runtime environments that are capable of intelligently deciding on application-aware late binding to physical resources.

Pilot-Jobs support effective distributed resource utilization, and are arguably one of the most widely-used distributed computing abstractions - as measured by the number and types of applications that use them, as well as the number of production distributed cyberinfrastructures that support them. In spite of broad uptake, there does not exist a well-defined, unifying conceptual model of Pilot-Jobs which can be used to define, compare and contrast different implementations. Often Pilot-Job implementations are strongly coupled to the distributed cyber-infrastructure they were originally designed for. These factors present a barrier to extensibility and interoperability. This paper is an attempt to (i) provide a minimal but complete model (P*) of Pilot-Jobs, (ii) establish the generality of the P* Model by mapping various existing and well known Pilot-Job frameworks such as Condor and DIANE to P*, (iii) derive an interoperable and extensible API for the P* Model (Pilot-API), (iv) validate the implementation of the Pilot-API by concurrently using multiple distinct Pilot-Job frameworks on distinct production distributed cyberinfrastructures, and (v) apply the P* Model to Pilot-Data.

Pilot-Jobs have become one of the most successful abstractions in distributed computing. In spite of extensive uptake, there does not exist a well defined, unifying conceptual model of pilot-jobs which can be used to define, compare and contrast different implementations. This presents a barrier to extensibility and interoperability. This paper is an attempt to, (i) provide a minimal but complete model (P*) of pilot-jobs, (ii) establish the generality of the P* Model by mapping various existing and well known pilot-jobs frameworks such as Condor and DIANE to P*, (iii) demonstrate the interoperable and concurrent usage of distinct pilot-job frameworks on different production distributed cyberinfrastructures via the use of an extensible API for the P* Model (Pilot-API).

The uptake of distributed infrastructures by scientific applications has been limited by the availability of extensible, pervasive and simple-to-use abstractions which are required at multiple levels development, deployment and execution stages of scientific applications. The Pilot-Job abstraction has been shown to be an effective abstraction to address many requirements of scientific applications. Specifically, Pilot-Jobs support the decoupling of workload submission from resource assignment; this results in a flexible execution model, which in turn enables the distributed scale-out of applications on multiple and possibly heterogeneous resources. Most Pilot-Job implementations however, are tied to a specific infrastructure. In this paper, we describe the design and implementation of a SAGA-based Pilot-Job, which supports a wide range of application types, and is usable over a broad range of infrastructures, i.e., it is general-purpose and extensible, and as we will argue is also interoperable with Clouds. We discuss how the SAGA-based Pilot-Job is used for different application types and supports the concurrent usage across multiple heterogeneous distributed infrastructure, including concurrent usage across Clouds and traditional Grids/Clusters. Further, we show how Pilot-Jobs can help to support dynamic execution models and thus, introduce new opportunities for distributed applications. We also demonstrate for the first time that we are aware of, the use of multiple Pilot-Job implementations to solve the same problem; specifically, we use the SAGA-based Pilot-Job on high-end resources such as the TeraGrid and the native Condor Pilot-Job (Glide-in) on Condor resources. Importantly both are invoked via the same interface without changes at the development or deployment level, but only an execution (run-time) decision

Azure is an emerging cloud platform developed and operated by Microsoft. It provides a range of abstractions and building blocks for creating scalable and reliable scientific applications. In this paper we investigate the applicability of the Azure abstractions to the well-known class of loosely coupled and ensemble-based applications. We propose the BigJob API as a novel abstraction for managing groups of Azure worker roles and for remotely executing tasks on them. We demonstrate that Azure enhanced with Big Job functionality provides performance comparable to other grid and cloud offerings loosely-coupled applications.

The ability to conduct advance reservations within grid environments is crucial for applications that want to utilize distributed resources in a predictable and efficient way. Advance reservations are essential for supporting deadline driven applications and the co-allocation of distributed resources. Further, advance reservations can significantly enhance the capabilities of resource brokers. Many Local Resource Management Systems (LRMSs), e.g. Torque/Maui, PBSPro and LSF, provide support for advance reservations. However, these capabilities are usually not accessible via the grid middleware. This paper presents and evaluates a job broker architecture that addresses this deficit and provides support for advance reservation on grid-level as well as a reservation-aware grid resource broker. A core component of the presented job broker architecture is the Earliest Start Time Estimator (ESE), which predicts queuing delays at grid resources. The job broker service is part of the Migol system - a grid middleware that addresses the fault tolerance of applications by offering capabilities such as automatic monitoring and recoveries.

There exists a class of scientific applications for which utilizing distributed resources is critical for reducing the time-to-solution. In this paper, we discuss a specific class of applications - Replica-Exchange simulations - where the orchestration of many distributed jobs in a dynamic and inherently unreliable distributed environment is essential for a successful completion. We describe the design, development and deployment of a unique framework for constructing fault-tolerant distributed simulations. The framework consists of two primary components - SAGA and Migol. SAGA is a high-level programmatic abstraction layer that provides a standardised interface for the primary distributed functionality required for application development. We present details of a newly developed functionality in SAGA - the Checkpoint and Recovery (CPR) API. Migol is an adaptive middleware, which supports the fault-tolerance of distributed applications by providing the capability to recover applications from checkpoint files transparently. In addition to describing the integration of SAGA-CPR with the Migol infrastructure, we outline our experiences with running a large scale, general-purpose Replica-Exchange application in a production distributed environment.

A major challenge in a service-oriented environment as a Grid is fault tolerance. The more resources and services involved, the more complicated and error-prone becomes the system. Migol (Luckow and Schnor, 2008) is a Grid middleware, which addresses the fault tolerance of Grid applications and services. Migol's core component is its registry service called application information service (AIS). To achieve fault tolerance and high availability the AIS is replicated on different sites. Since a registry is a stateful Web service, the replication of the AIS is no trivial task. In this paper, we present our concept for active replication of Grid services. Migol's Replication Service uses a token-based algorithm and certificate-based security to provide secure group communication. Further, we show in different experiments that active replication in a real Grid environment is feasible.

A major challenge in a dynamic Grid with thousands of machines connected to each other is fault tolerance. The more resources and components involved, themore complicated and error-prone becomes the system. Migol is an adaptive Grid middleware, which addresses the fault tolerance of Grid applications and services by providing the capability to recover applications from checkpoint files automatically. A critical aspect for an automatic recovery is the availability of checkpoint files: If a resource becomes unavailable, it is very likely that the associated storage is also unreachable, e. g. due to a network partition. A strategy to increase the availability of checkpoints isreplication.In this paper, we present the Checkpoint Replication Service. A key feature of this service is the ability to automatically replicate and monitor checkpoints in the Grid.