Proteins are complex biological information granules that play a crucial role in various cellular processes within living organisms. Processing 3D protein structures, which are the most informative from the biological point of view, is both intricate and time-consuming. In particular, performing 3D protein structure searches against large protein datasets involves identifying similarities and conducting structural alignments across numerous molecules (granules). This task demands advanced methods for matching identical and similar regions within protein structures and substantial computational resources to handle large collections of macromolecular data efficiently. In this paper, we present our parallel implementation of scalable 3D structural alignment on the Apache Spark big data platform. We describe a customized approach that leverages Spark data transformations within the data processing pipeline for the alignment process. Our experimental results demonstrate that this solution, tightly integrated with the Spark processing model, is both efficient and scalable, even with the increasing volume of protein structure data.

Detection and prediction of failures in Automated Guided Vehicles (AGV) are essential for the uninterrupted operation of production plants. Anomaly detection is usually achieved by comparing expected measurement values with actual observations. Thus, it is crucial to predict telemetry signals properly. In this paper, we research the prediction of energy consumption using state-of-the-art Artificial Neural Networks architectures (SCINet) compared with other Recurrent Neural Network (RNN) approaches on the data streams acquired from CoBotAGV. We especially focus on the possibility of applying feature weighting. We show that it can improve prediction capabilities. We also investigate resource utilization in terms of time to fit the embedded AGV environment.

In today’s world, smart factories are a coexisting element of smarticizing cities. Smart manufacturing of today relies on the automation of many component tasks of the production process. Automated guided vehicles (AGVs) that transport materials on the production lines are important elements of this automation. Appropriate management of a fleet of AGVs requires avoiding collisions. However, prediction and early detection of approaching collision points on the transportation routes not only prevent collisions but also enables adjusting the AGV operation and improving its flow. In this paper, we demonstrate the use of fuzzy sets and linguistic variables in collision prevention by processing AGV data streams with Apache Kafka. We extend the capabilities of Apache Kafka and ksqlDB towards fuzzy stream processing and use fuzzy KSQL queries to predict collisions. Our experiments prove that fuzzy querying against AGV data streams does not consume much time and computational resources, and we can successfully avoid collisions by predicting future positions of the AGV for various densities of data streams and widths of time windows.

Reverse k-nearest neighbor (RkNN) queries are prevalent in location-based services to find those locations that have the query point as one of their k nearest neighbors. However, such query requires users to disclose the location of the query point to a service provider who might be untrustworthy. Previous attempts to preserve the privacy of RkNN queries are either based on weaker notions of privacy such as location cloaking or not efficient when k > 1. In this paper, we propose novel solutions based on the private information retrieval (PIR) mechanism to preserve the privacy of RkNN query points. Our solutions include server-side data indexing and client-side query processing methods to facilitate PIR which is an inherently expensive data retrieval mechanism. We experimentally evaluate our approach using real-world datasets and show that it preserves the location privacy of queries with reasonable computation and storage overhead.

Extracting patterns and deriving insights from spatio-temporal data finds many target applications in various domains, such as in urban planning and computational sustainability. Due to their inherent capability of simultaneously modeling the spatial and temporal aspects of multiple instances, tensors have been successfully used to analyze such spatio-temporal data. However, standard tensor factorization approaches often result in components that are highly overlapping, which hinders the practitioner's ability to interpret them without advanced domain knowledge. In this work, we tackle this challenge by proposing a tensor factorization framework, called CP-ORTHO, to discover distinct and easily-interpretable patterns from multi-modal, spatio-temporal data. We evaluate our approach on real data reflecting taxi drop-off activity. CP-ORTHO provides more distinct and interpretable patterns than prior art, as measured via relevant quantitative metrics, without compromising the solution's accuracy. We observe that CP-ORTHO is fast, in that it achieves this result in 5x less time than the most accurate competing approach.

One of the greatest challenges in spatial crowdsourcing is determining the veracity of reports from multiple users about a particular event or phenomenon. In this paper, we address the difficulties of truth discovery in spatio-temporal tasks and present a new method based on recursive Bayesian estimation (BE) from multiple reports of users. Our method incorporates a reliability model for users, which improves as more reports arrive while increasing the accuracy of the model in labeling the state of the event. The model is further improved by Kalman estimation (BE+KE) that models the spatio-temporal correlations of the events and predicts the next state of an event and is corrected when new reports arrive. The methods are tested in a simulated environment, as well as using real-world data. Experimental results show that our methods are adaptable to the available data, can incorporate previous beliefs, and outperform existing truth discovery methods of spatio-temporal events.

Predictive range queries retrieve objects in a certain spatial region at a (future) prediction time. Processing predictive range queries on large moving object databases is expensive. Thus effective pruning is important, especially for long-term predictive queries since accurately predicting long-term future behaviors of moving objects is challenging and expensive. In this work, we propose a pruning method that effectively reduces the candidate set for predictive range queries based on (high-order) Markov chain models learned from historical trajectories. The key to our method is to devise compressed representations for sparse multi-dimensional matrices, and leverage efficient algorithms for matrix computations. Experimental evaluations show that our approach significantly outperforms other pruning methods in terms of efficiency and precision.

The rapid development of positioning techniques has enabled information to be widely collected on continuously moving objects, such as vehicles and mobile device users. Database systems supporting massive updates and predictive spatio-temporal queries on the moving objects are essential for modern location-based services. On the other hand, information (e.g. Location and velocity) that describes the motion of millions of moving objects can be accommodated in main memory with modern computing machines. Thus, in-memory indexing techniques are gaining popularity among moving object databases. Recent works indicate that a simple uniform grid is one of the best choices for indexing moving objects in main memory. However, to the best of our knowledge, relevant works in the literature consider only grids in the location space but disregard the velocity information of the moving objects, which we believe can be exploited to further improve query performance. In this paper, we propose D-Grid, an in-memory dual space grid index for moving objects. Specifically, it indexes moving objects using grid structures in both location and velocity spaces, which improves query performance by almost an order of magnitude. We also propose a lazy deletion and garbage cleaning mechanism that can be applied to both our dual space and existing location space uniform grid based indexes and further improve update performance. Extensive experiments demonstrate that our approach significantly outperforms existing uniform grid based in-memory indexes.

We propose to demonstrate STAC, a tool for spatial task assignment with cloaked locations in crowd sensing applications. The need for systems such as STAC becomes critical when participants of crowd sensing applications hesitate to share their locations due to privacy concerns. In such applications, STAC enables effective task assignment capabilities without compromising the location privacy of workers and volunteers. STAC implements novel algorithms that handle uncertain task assignment, which is a challenging problem involving uncertain locations of participants as a result of location cloaking. We will demonstrate, from both synthetic and real data, that our methods achieve high sensing coverage with low cost using cloaked participant locations. We will also demonstrate a location-based mobile Q&A application that utilizes STAC for task assignment.

Indexing moving objects has been extensively studied in the past decades. In most real world applications, the moving objects exhibit particular patterns on their velocities. For example, velocities of vehicles in city road networks usually show patterns on both directions and values. Velocity-based partitioning techniques have been proved effective in improving query performances of moving object indexes. This demo presents VPIndexer, a toolkit for visualizing comparison of three velocity-based partitioning algorithms: VMBR-based partitioning, DVA-based partitioning and our recently proposed speed-based partitioning techniques. VPIndexer uses the Bx-tree and the TPR*-tree as the baseline approaches.

Cloud computing is now a mainstream technology in many application domains and user constituencies. For scientific high-performance applications in academic and research settings however, the trade-offs between cost and elasticity on the one hand, and performance and access on the other, is not always clear. We discuss our experiences with comparing cost and utility for a hemodynamics computational fluid dynamics code on three typical platform-types available to researchers: IaaS clouds, grids, and on-premise local resources. To rank the tested platforms, we introduce a simple utility function describing the value of a completed computational task to the user as a function of the wait time and the cost of the computation. Our results suggest that IaaS clouds can be a convenient choice for the considered class of CFD simulations, providing a valuable trade-off between cost and task completion time.

Distributed mobile crowd sensing is becoming a valuable paradigm, enabling a variety of novel applications built on mobile networks and smart devices. However, this trend brings several challenges, including the need for crowd sourcing platforms to manage interactions between applications and the crowd (participants or workers). One of the key functions of such platforms is spatial task assignment which assigns sensing tasks to participants based on their locations. Task assignment becomes critical when participants are hesitant to share their locations due to privacy concerns. In this paper, we examine the problem of spatial task assignment in crowd sensing when participants utilize spatial cloaking to obfuscate their locations. We investigate methods for assigning sensing tasks to participants, efficiently managing location uncertainty and resource constraints. We propose a novel two-stage optimization approach which consists of global optimization using cloaked locations followed by a local optimization using participants' precise locations without breaching privacy. Experimental results using both synthetic and real data show that our methods achieve high sensing coverage with low cost using cloaked locations.

Monitoring web browsing behavior has benefited many data mining applications, such as top-K discovery and anomaly detection. However, releasing private user data to the greater public would concern web users about their privacy, especially after the incident of AOL search log release where anonymization was not correctly done. In this paper, we adopt differential privacy, a strong, provable privacy definition, and show that differentially private aggregates of web browsing activities can be released in real-time while preserving the utility of shared data. Our proposed algorithms utilize the rich correlation of the time series of aggregated data and adopt a state-space approach to estimate the underlying, true aggregates from the perturbed values by the differential privacy mechanism. We evaluate our algorithms with real-world web browsing data. Utility evaluations with three metrics demonstrate that the quality of the private, released data by our solutions closely resembles that of the original, unperturbed aggregates.

Sharing aggregate statistics of private data can be of great value when data mining can be performed in real-time to understand important phenomena such as influenza outbreaks or traffic congestion. However, to this date there have been no tools for releasing real-time aggregated data with differential privacy, a strong and provable privacy guarantee. We propose FAST, a real-time system that allows differentially private aggregate sharing and time-series analytics. FAST employs a set of novel, adaptive strategies to improve the utility of shared/released data while guaranteeing the user-specified level of differential privacy. We will demonstrate the challenges and our solutions in the context of prepared data sets as well as live participation data dynamically collected among the SIGMOD'13 attendees.

High-performance applications are often developed for a specific class of target platforms and executing them on the increasing variety of Cloud and grid environments requires substantial adjustments and reconciliation. The aim of our framework, called ADAPT (Adaptive Application and Platform Translation), is to allow adaptation of applications for execution on various computational resources. The overall objective is to enhance usability of cyber-infrastructure platforms by providing automated adaptations of applications and conditioning of target environments so that greater cross-utilization is achieved. This paper presents a proof-of-concept experiment of a possible use of ADAPT, viz. executing a C/MPI application originally written for clusters on the Microsoft Azure infrastructure. This MPI application-to-cloud adjustment is based on automatic identification of the application programming paradigm followed by applying an application transformation to enable execution on an alternative target.

Our on-going project, Unibus, aims to facilitate provisioning and aggregation of multifaceted resources from resource providers and end-users' perspectives. To achieve that, Unibus proposes (1) the Capability Model and mediators (resource drivers) to virtualize access to diverse resources, and (2) soft and successive conditioning to enable automatic and user-transparent resource provisioning. In this paper we examine the Unibus concepts and prototype in a real situation of aggregation of two commercial clouds and execution of benchmarks on aggregated resources. We also present and discuss benchmarks' results.

High-end machines at modern HPC centers are constantly undergoing hardware and system software upgrades - necessitating frequent rebuilds of application codes. The number of possible combinations of compilers, libraries, application build configurations, differing hardware architectures, etc, makes the process of building applications very onerous, requiring expert build knowledge from different domains. Our ongoing Harness Workbench Toolkit (HWT) project aims to foster and streamline the entire build process on heterogeneous computational platforms. This paper focuses on a key research issue of the HWT that regards facilitating and enhancement portability of build systems across multifarious machines, with particular respect to scientific software commonly used in the HPC community. The article presents a novel HWT approach based on the concept of generic build systems and profiles which encapsulate build knowledge provided independently by relevant experts. The paper describes profiles, the logistics of storing and retrieving build information, and interfacing to user-guided builds. We also report on experiences with applying the HWT approach to two scientific production codes (CPMD, GAMESS) on Cray XT4.

The paper addresses the build problem in the HPC arena that results from heterogeneity in hardware architectures, system software, and application build systems. We propose a new approach that generalizes current build systems, and enhances their portability across heterogeneous high-end platforms. Although the original application build system needs to be modified it is a one-time activity that allows us to extract target platform specific information and store it into profiles. Our toolkit exploits profiles to retrieve appropriate target-specific information during the actual build process. Our developed mechanism termed late binding enables dynamic concretization of platform-specific variables from profiles. This approach simplifies the build process for heterogeneous environments, promotes profile reuse, and improves its portability. In order to verify our approach in practice we have applied our methodology to a production molecular dynamics code (the CPMD application).

Fast hardware turnover in supercomputing centers, stimulated by rapid technological progress, results in high heterogeneity among HPC platforms, and necessitates that applications are ported and adapted frequently. The cutting-edge nature of the hardware mandates customized performance tuning, which, coupled with continuously growing application complexity, makes the process inherently and increasingly challenging. In this paper, we analyze build procedures of a representative set of HPC applications, and attempt to identify commonalities that can be exploited to enhance cross-platform portability. We then propose a novel method for reducing non-portabilities while preserving high performance. The approach, based on profiles that capture and isolate non-portable features at various levels, requires only a moderate amount of changes to existing makefiles. It leverages the expertise of system designers and administrators, and reduces burdens placed on application scientists. As a proof of concept, we discuss the application of our methodology to enhancing portability of the Mile application across heterogeneous HPC platforms.

Heterogeneous distributed computing platforms continue to require substantial effort in terms of software installation and management for efficient use, often necessitating manual intervention by resource providers and end-users. In this paper we propose a novel model of resource sharing that is a viable alternative to that commonly adopted in the grid community. Our model shifts the resource virtualization and aggregation responsibilities to software at the client side, taking these burdens away from resource providers. Following this model, we have designed the Zero-Force MPI toolkit that automates installation, run, and post-processing stages of HPC applications, thus allowing application scientists to focus on using computational resources instead of managing them. Through the provided console, MPI can be soft-installed across a set of distributed resources with just a few commands. The prototype implementation of the ZF-MPI toolkit is based on FTMPI. To evaluate the toolkit, we conducted experiments with deploying and executing the NAS Parallel Benchmarks. Results demonstrate that the run-not-install approach exemplified by ZF-MPI is effective and may substantially increase productivity of application execution.

The capability to announce and discover resources is a foundation for heterogeneous computing systems. Independent projects have adopted custom implementations of information services, which are not interoperable and induce substantial maintenance costs. In this paper, we propose an alternative methodology. We suggest that it is possible to reuse existing naming service deployments and combine them into complex, scalable, hierarchical, distributed federations, by using appropriate client-side integration middleware that unifies service access and hides heterogeneity behind a common API. We investigate a JNDI-based approach, and describe in detail two newly implemented JNDI service providers, which enable unified access to 1) Jini lookup services, and 2) harness distributed naming services. We claim that these two technologies, along with others already accessible through JNDI such as e.g. DNS and LDAP, offer features suitable for use in hierarchical heterogeneous information systems.

Shared HPC platforms continue to require substantial effort for software installation and management, often necessitating manual intervention and tedious procedures. We propose a novel model of resource sharing that shifts resource virtualization and aggregation responsibilities to client-side software, thus reducing the burdens on resource providers.The Zero-Force MPI toolkit automates the installation, build, run, and post-processing stages of HPC applications, thus allowing application scientists to focus on using resources instead of managing them. Through a provided console, MPI runtime systems, support libraries, application executables, and needed datafiles can be soft-installed across distributed resources with just a few commands. Built-in data synchronization capabilities simplify common HPC development tasks, saving end-user time and effort. To evaluate ZF-MPI, we conducted experiments with the NAS Parallel Benchmarks. Results demonstrate that the proposed run-not-install approach is effective and may substantially increase overall productivity.

Abstract When the critical path of a communication session between end points includes the actions of operating system kernels, there are attendant overheads. Along with other factors, such as functionality and flexibility, such overheads motivate and favor the implementation of communication protocols in user space. When implemented with threads, such protocols may hold the key to optimal communication performance and functionality. Based on implementations of reliable user‐space protocols supported by a threads framework, we focus on our experiences with internal threads' scheduling techniques and their potential impact on performance. We present scheduling strategies that enable threads to do both application‐level and communication‐related processing. With experiments performed on a Sun SPARC‐5 LAN environment, we show how different scheduling strategies yield different levels of application‐processing efficiency, communication latency and packet‐loss. This work forms part of a larger study on the implementation of multiple thread‐based protocols in a single address space, and the benefits of coupling protocols with applications. Copyright © 2005 John Wiley & Sons, Ltd.

Summary form only given. Remote method invocation is one of the most popular communication paradigms in distributed computing. However, its synchronous nature may affect application performance. Asynchronous, or nonblocking, RMI addresses the issue by implicitly introducing concurrency into an application and allowing it to interleave computations with communications. We analyze certain semantic issues raised by asynchronous RMI, such as execution order, exception handling, cancellation, data consistency, and more. Solutions are proposed, and a realization of asynchronous RMI within our multiprotocol, extensible communication framework for Java (termed RMIX), is presented.

Summary form only given. Service-based modalities are becoming more prevalent in metacomputing and grid computing. The absence of preexisting client-server relationships imply the use of standardized, self-describing protocols, which typically suffer from performance deficiencies. The RMIX framework is designed to permit heterogeneous entities to interact via a service-based paradigm while enabling protocol selection and dynamic protocol switching. In particular, an RPC-compliant communication provider enables multilanguage heterogeneous clients to communicate with servers and leverage well-established RPC/XDR mechanisms to support interoperability without sacrificing performance. We describe the RMIX-RPCX communication substrate, and presents preliminary results that suggest its potential value in enhancing service-oriented heterogeneous metacomputing.

We propose the notion of 'collaborative peer groups', defined as peer-to-peer overlay networks with controlled membership and multiway communication primitives that offer well-defined semantics. Peers join such groups subject to symmetric acceptance, typically based on functional commonalities and, optionally, group-specific authentication. Collaborative peer group networks share the same properties as other peer-to-peer networks, including full decentralization, symmetric abilities, and dynamism. In addition, however, an extensible set of multiway communication primitives, especially appropriate for such peer groups, is provided and supports operations such as reliable message delivery to proximal group members or a subset thereof, message aggregation from peers, and discovery of peers supporting specific functional attributes. Based on several current and emerging application scenarios, we motivate and present the proposed collaborative peer group model, outline the group management architecture, and describe the initial set of communication primitives to be supported. A discussion of the toolkit development methodology and preliminary experiences concludes the paper.

This paper describes the Harness-PVM emulation module that makes it possible to run legacy PVM applications within the Harness metacomputing system environment. We show benchmark results suggesting that component- and Java-based frameworks may provide substantial benefits without significantly compromising performance. Augmenting or implementing grids using such component based systems can thus be advantageous in a number of situations.

Although there have been several attempts to create grid systems, there is no clear definition for grids. In this paper, a formal approach is presented for defining elementary functionalities of grid systems. A comparative analysis with conventional distributed systems shows that there are semantical differences, not just technical ones. The resulting abstract working model for grids can serve as a framework for defining new systems or analyzing existing ones.

Software infrastructures that support metacomputing are evolving from traditional monolithic, platform-specific systems to component and service-based frameworks. We demonstrate that contrary to popular belief, such modular software systems are capable of delivering good to excellent performance, support legacy as well as new application programming paradigms, and deliver enhanced functionality. The Harness system, a software backplane enabling reconfigurable distributed concurrent computing is used to emulate the PVM programming environment. Numerical kernel benchmarks show that application performance results using the emulator and native versions are within a few percent of each other. Coupled with the ability to leverage pre-existing and specialized modules, our experiences suggest that service-oriented computational grids may be constructed rapidly and effectively via such component-based architectural frameworks that deliver full functionality, without compromising efficiency.

A novel component-based, service-oriented framework for distributed metacomputing is described. Adopting a provider-centric view of resource sharing, this project emphasizes lightweight software infrastructures that maintain a minimal state, and interface to current and emerging distributed computing standards. Resource owners host a software backplane onto which owners, clients, or third-party, resellers may load components or component-suites that deliver value added services without compromising owner security or control. Standards-based descriptions of services facilitate publication and discovery via established schemes. The architecture of the container framework, design of components, security and access control schemes, and preliminary experiences are described in this paper.

This paper presents schemes that enable high level communications protocols and applications to adapt to connectivity loss and quality degradation in metropolitan area wireless networks. We postulate that the majority of these problem areas or trouble spots, which are intrinsic to wireless networks, are related to location and environmental factors. Based on this premise, we propose a mechanism that gathers semantic data pertaining to trouble spots; prior knowledge of such locations can be used by higher-level communication protocols to preemptively adapt, thereby avoiding undesirable effects at the application level. To facilitate the detection and categorization of trouble spots, we propose a list of metrics to analyze the status of a wireless service. We report on our experiences with using these metrics to identify trouble spots and present initial results from an experimental evaluation of their effectiveness.

In this paper we present a novel approach to fault-tolerant group membership for use predominantly in collaborative computing environments. As an exemplar, we use the Collaborative Computing Transport Layer which offers reliable atomic multicast capabilities for use in collaborative environments such as the Collaborative Computing Frameworks (CCF). Specific design goals of the approach are the elimination of processing overhead due to heartbeats, support for partial failures and extensibility These goals are satisfied in an approach which uses an IP multicast failure detector and two election based algorithms. By basing failure detection on IP multicast, the need for explicit keep-alive packets is removed, thus in the absence of failures the approach imposes no overhead.

Harness is an Java-centric metacomputing system based on a principle of dynamic reconfigurability not only in terms of participating computing resources, but also the capabilities of the virtual machine itself. The central feature of the system is a “plug-in” mechanism built upon Java dynamic class loading services enabling integration of new functionality in the run time. In this paper we describe new flexible, framework based Java class loading techniques and their application in the Harness system. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In order to solve the emerging debugging issues in the field of metacomputing we defined the fundamental principles of an adaptive and integrated debugging and visualization tool: a novel metadebugger. The current prototype has been implemented in the Harness metacomputing framework.

This paper discusses an approach which aims to provide legacy message passing libraries with Java-like portability in a heterogeneous, metacomputing environment. The results of such portability permit distributed computing components to be ‘soft-loaded’ or ‘soft-installed’ in a dynamic fashion, onto cooperating resources for concurrent, synchronized parallel execution. This capability provides researchers with the ability to tap into a much larger resource pool and to utilize highly tuned codes for achievingperformance. Necessarily, the Java programming language is a significant component. The Java Native Interface (JNI) is used to wrap message passing libraries written in other languages, and the bytecode which is generated for the front-end may be analyzed in order to completely determine the needs of the code which it wraps. This characterization allows the pre-configuration of a remote environment so as to be able to support execution. The usefulness of the portability gained by our approach is illustrated through examples showing the soft-installation of a process using an MPI computational substrate and the soft-installation of a process which requires a C-based communication library based upon the efficient multi-cast communication package, CCTL. The examples show that significant gains in performance can be achieved while allowing message passing execution to still exhibit high levels of portability. Copyright © 2000 John Wiley & Sons, Ltd.

Article MPI and Java-MPI: contrasts and comparisons of low-level communication performance Share on Authors: Vladimir Getov School of Computer Science, University of Westminster, Harrow HA1 3TP, UK School of Computer Science, University of Westminster, Harrow HA1 3TP, UKView Profile , Paul Gray Dept. of Mathematics and Computer Science, Emory University, Atlanta, GA Dept. of Mathematics and Computer Science, Emory University, Atlanta, GAView Profile , Vaidy Sunderam Dept. of Mathematics and Computer Science, Emory University, Atlanta, GA Dept. of Mathematics and Computer Science, Emory University, Atlanta, GAView Profile Authors Info & Claims SC '99: Proceedings of the 1999 ACM/IEEE conference on SupercomputingJanuary 1999 Pages 21–eshttps://doi.org/10.1145/331532.331553Published:01 January 1999 14citation711DownloadsMetricsTotal Citations14Total Downloads711Last 12 Months1Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Workstation and PC clusters interconnected by SCI (scalable coherent interface) are very promising technologies for high-performance cluster computing. Using commercial SBus to SCI interface cards and system software and drivers, a two-workstation cluster has been constructed for initial testing and evaluation. The PVM system has been adapted to operate on this cluster using both raw channel and shared-memory access to the SCI interconnect, and preliminary communications performance tests have been carried out. To achieve mutual exclusion in accessing shared-memory segments, two protocols were used. Our preliminary results indicate that communications throughput in the range of 17.7 Mbytes/s, and round-trip latencies of 80 μs using the first and 140 μs using the second protocol, can be obtained on SCI clusters. These figures are significantly better (by a factor of 2 to 4) for small and large messages than those attainable on Fast Ethernet LANs. Since these performance results are very encouraging, we expect that, in the very near future, SCI networks will be capable of delivering several tens of Mbytes/s bandwidth and a few tens of microseconds latencies, and will significantly enhance the viability of cluster computing. Copyright © 1999 John Wiley & Sons, Ltd.

We describe our vision, goals and plans for HARNESS, a distributed, reconfigurable and heterogeneous computing environment that supports dynamically adaptable parallel applications. HARNESS builds on the core concept of the personal virtual machine as an abstraction for distributed parallel programming, but fundamentally extends this idea, greatly enhancing dynamic capabilities. HARNESS is being designed to embrace dynamics at every level through a pluggable model that allows multiple distributed virtual machines (DVMs) to merge, split and interact with each other. It provides mechanisms for new and legacy applications to collaborate with each other using the HARNESS infrastructure, and defines and implements new plug-in interfaces and modules so that applications can dynamically customize their virtual environment. HARNESS fits well within the larger picture of computational grids as a dynamic mechanism to hide the heterogeneity and complexity of the nationally distributed infrastructure. HARNESS DVMs allow programmers and users to construct personal subsets of an existing computational grid and treat them as unified network computers, providing a familiar and comfortable environment that provides easy-to-understand scoping.

A group of chemists each direct their Web browsers to a page containing an embedded virtual reality (VR) representation of a collection of molecules. Each chemist is represented in the VR world by an "avatar" reflecting position and view. The molecule is not a simple ball and stick model; depicted is the solvent accessible surface (SAS) surrounding a protein that the chemists fold in real time. They manipulate and measure the structures using tools from a variety of different vendors. As more collaborators join, performance actually improves as each Web browser loads an applet contributing to the SAS computation. Agents on the server coordinate computations amongst the dynamic pool of clients and act as interfaces to legacy software. ToolSpace is a Collaborative Virtual Environment (CVE) software framework with: a visualization client that runs in a standard Web browser equipped with a VRML plugin, a state sharing server that runs under any operating system with a Java virtual machine, and a set of Java classes for developing applications. The goal of the project is to prototype a next generation computing environment that is: multi-user collaborative, distributed, component and object based, platform independent, three dimensional, scalable with respect to computer hardware and input/output devices, and above all, easy to use and develop for. Such an environment has natural applications in fields ranging from chemistry, to crime scene analysis, to military war games.

Application sharing is an important aspect of collaborative computing. One mechanism for sharing applications is a multiplexor, a tool that broadcasts input/output messages from a single client to multiple displays. We describe the design and implementation of a multiplexor for X windows that supports a heterogeneous environment and works effectively across both local and wide area networks. This X-multiplexor is composed of two distinct components, the Collaborative Computing Frameworks X-Multiplexor (CCFX) which acts as a pseudo server for the client, and the Collaborative Computing Session Manager (CCSM) which renders the client images and supplies the user interface to the system. In effect, CCSM acts as a distributed window manager, ensuring that all participants have the same view of the shared clients. This two headed approach offers significant gains in flexibility and resource usage. CCFX and CCSM can communicate using any reliable transport protocol. This can result in significant reduction of bandwidth, decreased latency, and avoids some TCP/IP communication required by the X protocol. In addition, this architecture facilitates the creation of a virtual X environment. Using this virtual environment, instead of an actual X server's environment, and then translating the virtual properties to a specific server's properties, allows for more robust support of heterogeneous systems.

CCF (Collaborative Computing Frameworks) is a suite of software systems, communications protocols, and tools that enable collaborative, computer-based cooperative work. CCF constructs a virtual work environment on multiple computer systems connected over the Internet, to form a Collaboratory. In this setting, participants interact with each other, simultaneously access and operate computer applications, refer to global data repositories or archives, collectively create and manipulate documents or other artifacts, perform computational transformations, and conduct a number of other activities via telepresence. Research issues addressed in this project include problem solving environments and methodologies for laboratory and instrument-based scientific disciplines, and computer science issues in heterogeneous distributed systems. New approaches are being investigated and developed for fast multiway communication, robust geographically distributed data management methodologies, high-performance computational transforms inlined within collaboration sessions, and related auxiliary issues such as active documents, security, archival storage, and experiment management and control. In this paper, we discuss the design philosophy and systems rationale behind CCF, describe the major subsystems of the collaborative computing environment, and discuss the salient features of the system.

Distributed concurrent computing based on lightweight processes can potentially address performance and functionality limits in heterogeneous systems. The TPVM framework, based on the notion of ‘exportable services’, is an extension to the PVM message-passing system, but uses threads as units of computing, scheduling, and parallelism. TPVM facilitates and supports three different distributed concurrent programming paradigms: (a) the traditional, task based, explicit message-passing model; (b) a data-driven instantiation model that enables straightforward specification of computation based on data dependencies; and (c) a partial shared-address space model via remote memory access, with naming and typing of distributed data areas. The latter models offer significantly different computing paradigms for network-based computing, while maintaining a close resemblance to, and building upon, the conventional PVM infrastructure in the interest of compatibility and ease of transition. The TPVM system comprises three basic modules: a library interface that provides access to thread-based distributed concurrent computing facilities, a portable thread interface module which abstracts the required thread-related services, and a thread server module which performs scheduling and system data management. System implementation as well as applications experiences have been very encouraging, indicating the viability of the proposed models, the feasibility of portable and efficient threads systems for distributed computing, and the performance improvements that result from multithreaded concurrent computing. © 1998 John Wiley & Sons, Ltd.

This paper discusses how the aspects unique to the Java programming language can be combined with complementary and unique aspects of other languages such as C and Fortran. This combining of the strong features of Java, such as portability and platform independence, with packages and legacy codes written in traditional languages such as C and Fortran results in a program blend which exhibits portability and speed not realizable by any of these languages individually. One area where this confluence of previously disparate language features has strong potential is in the area of distributed, concurrent computing over heterogeneous platforms and across local network and filesystem boundaries – the setting addressed within this paper. Also addressed in this paper are the pivotal aspects of the Java bytecode representation of a class object which makes the porting of shared libraries across network boundaries, filesystems, and architectures possible. © 1998 John Wiley & Sons, Ltd.

Network computing has evolved into a popular and effective mode of high performance computing. Network computing environments have fundamental differences from hardware multiprocessors, involving a different approach to measuring and characterizing performance, monitoring an application's progress and understanding program behavior. In this paper, we present the design and implementation of PVaniM, an experimental visualization environment we have developed for the PVM network computing system. PVaniM supports a two-phase approach whereby on-line visualization focuses on large-grained events that are influenced by and relate to the dynamic network computing environment, and postmortem visualization provides for detailed program analysis and tuning. PVaniM's capabilities are illustrated via its use on several applications and a comparison with single-phase visualization environments developed for network computing. Our experiences indicate that, for several classes of applications, the two-phase visualization scheme can provide valuable insight into the behavior, efficiency and operation of distributed and parallel programs in network computing environments. © 1998 John Wiley & Sons, Ltd.

The Collaborative Computing Transport Layer (CCTL) is a communication substrate consisting of a suite of multiparty protocols, providing varying service qualities among process groups. CCTL explicitly supports distributed collaborative and multimedia applications. CCTL is based on a two-level group hierarchy. Logical interconnections among entities, called channels, define an efficient and light-weight group mechanism. Channels support a variety of service qualities such as reliability and message ordering. Related channels can also be combined to form sessions, heavy-weight groups which provide a default atomic multicast service. CCTL supports membership protocols tailored to the quality of service offered by a channel, including a relaxed form of virtual synchrony. In this paper, we present three membership protocols and compare and relate our implementations to alternatives. Our two-level architecture allows simple and efficient implementation of the membership protocols.

Metacomputing, or distributed processing on networks (local networks, intranets or the Internet), has re-emerged as a technology with tremendous promise and potential, owing in part to the emergence of the Java language and programming system. Java both influences and is influenced by the requisite and dynamic aspects of network programming. However, its viability as a programming language for the scientific community is yet to be established. This paper describes IceT, a novel framework for collaborative and high-performance distributed computing which has been built on a Java substrate. The IceT system exploits the portability and scripting advantages of Java and incorporates well-established distributed and concurrent computing techniques into the framework, while retaining the ability to access specialized processing capabilities and precompiled code. The result is a dynamic and efficient distributed computing environment upon which data and processes are highly portable amongst heterogeneous platforms and multiple users. © 1997 John Wiley & Sons, Ltd.

Local or wide-area heterogeneous workstation clusters are relatively cheap and highly effective, though inherently unstable operating environments for long-running distributed computations. We found this to be the case in early experiments with a prototype of the EcliPSe system, a software toolkit for replicative applications on heterogeneous workstation clusters. Hardware or network failures in computations that executed for over a day were not uncommon. In this work, a variety of features for the incorporation of failure resilience in the EcliPSe system are described. Key characteristics of this fault-tolerant system are ease of use, low state-saving cost, system scalability and good performance. We present results of some experiments demonstrating low state-saving overheads and small system-recovery times, as a function of the amount of state saved.

The TPVM (Threads-oriented PVM) system, is an experimental auxiliary subsystem for the PVM distributed system, which supports the use of lightweight processes or "threads" as the basic unit of parallelism and scheduling. TPVM provides a library interface which presents both a traditional, task based, explicit message passing model, as well as a data-driven scheduling model that enables straightforward specification of computation based on data dependencies. Our system design is still under development, but a prototype implementation has allowed us to perform a number of preliminary experiments. These have provided strong evidence that TPVM can offer improved performance, processor utilization, and load balance to several application categories. Through our experiments we have also determined that the current TPVM design is not very well suited to certain types of applications, most notably highly synchronous, SPMD-style algorithms.

Visualization and animation tools may become extremely important aids in the understanding, verification, and performance tuning of parallel computations. Presently, however, the use of visualization has had only a limited use for enhancing parallel computation. We hypothesize that one of the primary reasons for the limited use of visualization tools in parallel program development is the difficulty of acquiring the information necessary to drive the visual display. Our approach to this impediment focuses on integrating visualization support directly into a distributed computing system. Central to this integration is the addition of a logical clock that prevents the timestamps of events from violating causality. The implementation requires the "piggybacking" of a negligible amount of extra header information on system messages and the impact on performance is minimal. This results in a system that produces useful visualizations with no extra effort required by the applications programmer. Also integrated into the distributed system is support which simplifies the creation of programmer-defined, application-specific visualizations, unique to each new parallel program developed.

Barrier synchronization is a commonly used primitive in parallel processing. In this paper, we present different algorithms for barrier synchronization on the widely prevalent multiaccess bus network, and derive analytical performance metrics for each of the proposed schemes, which are then compared against simulation results.>

Local or wide-area connected workstation cluster-based computation systems are inherently failure-prone, particularly for long running computations. In this work we introduce a variety of features for failure resilience in the EcliPSe system for replicative applications. Key characteristics of fault-tolerant EcliPSe are ease of use, low statesaving costs, system scalability and good performance.

Abstract In concurrent computing environments based on heterogeneous processing elements interconnected by general‐purpose networks, several classes of overheads contribute to lowered performance. The most obvious limitations are network throughput and latency, but certain other factors also play a significant role. In an attempt to gain some insight into the nature of these overheads, and to propose strategies to alleviate them, empirical measurements of native communication performance as well as application execution performance were conducted, using the PVM network computing system. These experiments and our analyses have identified load imbalance, the parallelism model adopted, communication delay and throughput, and within‐host overheads as the primary factors affecting performance in cluster environments. Interestingly, we find that agenda parallelism and load balancing strategies contribute significantly more to better performance than improved communications or system tuning. Drawing general conclusions on how these inefficiencies may be overcome is inadvisable because of the tremendous variability of many parameters in general purpose network environments; we therefore propose several potential approaches, including model selection criteria, partitioning strategies, and software system heuristics, to reduce overheads and enhance performance in network based environments.

The authors present the results of an experiment on massively parallel stochastic simulation of polymer chains for measuring scale-invariant phenomena at critical temperatures. The parallelization is achieved through the EcliPSe toolkit and conducted on a flexible, tree-structured virtual machine made up of arbitrary and heterogeneous computing nodes dispersed across the country. These nodes cooperate to perform the simulation and pool results together in real time at a central node which initiates the parallel simulation. The advantage of the tree-structure is that it allows for a fault-resilient, flexible environment for long-running parallel simulations. The authors present results on sheer performance, price performance, and toolkit-based parallelization by monitoring the same computations on a CRAY Y-MP and making detailed comparisons. Most significant are the excellent price-performance ratios given by the heterogeneous computing environment.>

Abstract Concurrent computing environments based on loosely coupled networks have proven effective as resources for multiprocessing. Experiences with and enhancements to version 1.0 of PVM (Parallel Virtual Machine) are described in this paper. PVM is a software package that allows the utilization of a heterogeneous network of parallel and serial computers as a single computational resource. This report also describes an interactive graphical interface to PVM, and porting and performance results from production applications.

The EcliPSe system is a prototype toolkit for concurrent stochastic simulation. The toolkit provides high-level simulation-oriented primitives which an application programmer uses to describe the simulation process using a sequential computation model. The system transparently replicates appropriate sections of the simulation when the program is executed on a concurrent system. By binding to machine-dependent versions of the EcliPSe library, the simulation program can execute without modification on a variety of architectures, including uniprocessors, hypercubes, shared-memory machines, and loosely coupled networks, leading, in those experiments conducted, to near-linear speedup. The main design aspects of the EcliPSe system, salient implementation features, and performance figures for some simulation applications are presented.>

The design of an inclusive session level protocol targeted at distributed applications on local networks is presented. The session protocol is motivated by the observation that application requirements, as well as network characteristics, for current and future distributed systems are not well matched to available protocol suites. The nature of services provided in the proposed protocol are derived from typical application requirements, and include group communications, synchronization and recovery, and integrated distributed primitives such as mutual exclusion and consensus. The protocol is also influenced by the characteristics of typical local networks that support sequenced delivery with data integrity, have low latency and high throughput, and are well suited to global addressing schemes. Initial experiences with a test implementation indicate that high level service support is valuable and can be provided with good performance.

Abstract The PVM system is a programming environment for the development and execution of large concurrent or parallel applications that consist of many interacting, but relatively independent, components. It is intended to operate on a collection of heterogeneous computing elements interconnected by one or more networks. The participating processors may be scalar machines, multiprocessors, or special‐purpose computers, enabling application components to execute on the architecture most appropriate to the algorithm. PVM provides a straightforward and general inferface that permits the description of various types of algorithms (and their interactions), while the underlying infrastructure permits the execution of applications on a virtual computing environment that supports multiple parallel computation models. PVM contains facilities for concurrent, sequential or conditional execution of application components, is portable to a variety of architectures, and supports certain forms of error detection and recovery.