Controlling the spread of infectious diseases in large populations is an important societal challenge. Mathematically, the problem is best captured as a certain class of reaction-diffusion processes (referred to as contagion processes) over appropriate synthesized interaction networks. Agent-based models have been successfully used in the recent past to study such contagion processes. We describe EpiSimdemics, a highly scalable, parallel code written in Charm++ that uses agent-based modeling to simulate disease spreads over large, realistic, co-evolving interaction networks. We present a new parallel implementation of EpiSimdemics that achieves unprecedented strong and weak scaling on different architectures - Blue Waters, Cori and Mira. EpiSimdemics achieves five times greater speedup than the second fastest parallel code in this field. This unprecedented scaling is an important step to support the long term vision of realtime epidemic science. Finally, we demonstrate the capabilities of EpiSimdemics by simulating the spread of influenza over a realistic synthetic social contact network spanning the continental United States (~280 million nodes and 5.8 billion social contacts).

Data movement in high-performance computing systems accelerated by graphics processing units (GPUs) remains a challenging problem. Data communication in popular parallel programming models, such as the Message Passing Interface (MPI), is currently limited to the data stored in the CPU memory space. Auxiliary memory systems, such as GPU memory, are not integrated into such data movement standards, thus providing applications with no direct mechanism to perform end-to-end data movement. We introduce MPI-ACC, an integrated and extensible framework that allows end-to-end data movement in accelerator-based systems. MPI-ACC provides productivity and performance benefits by integrating support for auxiliary memory spaces into MPI. MPI-ACC supports data transfer among CUDA, OpenCL and CPU memory spaces and is extensible to other offload models as well. MPI-ACC's runtime system enables several key optimizations, including pipelining of data transfers, scalable memory management techniques, and balancing of communication based on accelerator and node architecture. MPI-ACC is designed to work concurrently with other GPU workloads with minimum contention. We describe how MPI-ACC can be used to design new communication-computation patterns in scientific applications from domains such as epidemiology simulation and seismology modeling, and we discuss the lessons learned. We present experimental results on a state-of-the-art cluster with hundreds of GPUs; and we compare the performance and productivity of MPI-ACC with MVAPICH, a popular CUDA-aware MPI solution. MPI-ACC encourages programmers to explore novel application-specific optimizations for improved overall cluster utilization.

Cost estimation is crucial in the performance modeling of parallel algorithms and allocation of computational resources on distributed systems. This paper presents a novel methodology for estimating the cost of constrained producer-consumer (CPC) algorithms. In CPC algorithms, the computation is performed by classes of nodes (tasks), separated in time. The methodology combines data flow analysis with communication latencies to determine the production and consumption of data on different processors, which helps in determining the amount of computations and communication. The cost metric that we develop in this paper uses computational imbalances and communication load, and determines a single cost value. The resulting metric is unique, as it provides the first model that targets CPC algorithms. It has wide application in Genetic Algorithms, molecular dynamics, scheduling schemes and computational epidemiology. We provide a general method for determining the application-specific constants of the cost metric. As an example, we extract the constants for EpiSimdemics (a highly scalable contagion simulator), and give guidelines for applying the procedure to other CPC algorithms. Our evaluations show that the cost metric estimated the execution times of a contagion simulator with less than a 6.5% error. The metric can be used in optimal assignment of computational resources.

Cost estimation is crucial for optimizing the performance of parallel algorithms on distributed systems. This paper performs load analysis and cost estimation of constrained producer-consumer (CPC) parallel algorithms. In CPC algorithms, the computation is performed by classes of nodes (tasks), separated in time. For any given CPC problem, the cost can be modelled as a linear combination of computational imbalance components and a communication load. These components can be determined from analysis of the inter-task communication of the algorithm.

Analysis of structural properties and dynamics of networks is currently a central topic in many disciplines including Social Sciences, Biology and Business. CINET, a cyber infrastructure for such studies, introduced the concept of supporting network analysis as a service. The basic idea is to allow experts in various disciplines to focus on obtaining domain-specific insights from the results of network analyses instead of worrying about programming details and allocation of computational resources needed to carry out the analyses. A basic version of CINET was released in May 2012. This paper discusses CINET 2.0, a significantly enhanced version that supports complex network analyses through a web portal. CINET 2.0 has already been used for teaching courses related to Network Science at several US universities. In this paper, we discuss how CINET 2.0 significantly extends CINET 1.0 through enhancements to some components and the addition of new components.

Fine-grained communication in supercomputing applications often limits performance through high communication overhead and poor utilization of network bandwidth. This paper presents Topological Routing and Aggregation Module (TRAM), a library that optimizes fine-grained communication performance by routing and dynamically combining short messages. TRAM collects units of fine-grained communication from the application and combines them into aggregated messages with a common intermediate destination. It routes these messages along a virtual mesh topology mapped onto the physical topology of the network. TRAM improves network bandwidth utilization and reduces communication overhead. It is particularly effective in optimizing patterns with global communication and large message counts, such as all-to-all and many-to-many, as well as sparse, irregular, dynamic or data dependent patterns. We demonstrate how TRAM improves performance through theoretical analysis and experimental verification using benchmarks and scientific applications. We present speedups on petascale systems of 6x for communication benchmarks and up to 4x for applications.

Modeling dynamical systems represents an important application class covering a wide range of disciplines including but not limited to biology, chemistry, finance, national security, and health care. Such applications typically involve large-scale, irregular graph processing, which makes them difficult to scale due to the evolutionary nature of their workload, irregular communication and load imbalance. EpiSimdemics is such an application simulating epidemic diffusion in extremely large and realistic social contact networks. It implements a graph-based system that captures dynamics among co-evolving entities. This paper presents an implementation of EpiSimdemics in Charm++ that enables future research by social, biological and computational scientists at unprecedented data and system scales. We present new methods for application-specific processing of graph data and demonstrate the effectiveness of these methods on a Cray XE6, specifically NCSA's Blue Waters system.

We describe ISIS, a high-performance-computing-based application to support computational epidemiology of infectious diseases. ISIS has been developed over the last seven years in close coordination with public health and policy experts. It has been used in a number of important federal planning and response exercises. ISIS grew out of years of experience in developing and using HPC-oriented models of complex socially coupled systems. This identified the guiding principle that complex models will be used by domain experts only if they can do realistic analysis without becoming computing experts.

Socially coupled systems are comprised of interdependent social, organizational, economic, infrastructure and physical networks. Today's urban regions serve as an excellent example of such systems. People and institutions confront the implications of the increasing scale of information becoming available due to a combination of advances in pervasive computing, data acquisition systems as well as high performance computing. Integrated modeling and decision making environments are necessary to support planning, analysis and counter factual experiments to study these complex systems. Here, we describe SIMFRASTRUCTURE - a flexible coordination middleware that supports high performance computing oriented decision and analytics environments to study socially coupled systems. Simfrastructure provides a multiplexing mechanism by which simple and intuitive user-interfaces can be plugged in as front-end systems, and high-end computing resources can be plugged in as back-end systems for execution. This makes the computational complexity of the simulations completely transparent to the users. The decoupling of user interfaces and data repository from simulation execution allows users to access simulation results asynchronously and enables them to add new datasets and simulation models dynamically. Simfrastructure enables implementation of a simple yet powerful modeling environment with built-in analytics-as-a service platform, which provides seamless access to high end computational resources, through an intuitive interface for studying socially coupled systems. We illustrate the applicability of Simfrastructure in the context of an integrated modeling environment to study public health epidemiology.

Differentiation of CD4+ T cells into effector or regulatory phenotypes is tightly controlled by the cytokine milieu, complex intracellular signaling networks and numerous transcriptional regulators. We combined experimental approaches and computational modeling to investigate the mechanisms controlling differentiation and plasticity of CD4+ T cells in the gut of mice. Our computational model encompasses the major intracellular pathways involved in CD4+ T cell differentiation into T helper 1 (Th1), Th2, Th17 and induced regulatory T cells (iTreg). Our modeling efforts predicted a critical role for peroxisome proliferator-activated receptor gamma (PPARγ) in modulating plasticity between Th17 and iTreg cells. PPARγ regulates differentiation, activation and cytokine production, thereby controlling the induction of effector and regulatory responses, and is a promising therapeutic target for dysregulated immune responses and inflammation. Our modeling efforts predict that following PPARγ activation, Th17 cells undergo phenotype switch and become iTreg cells. This prediction was validated by results of adoptive transfer studies showing an increase of colonic iTreg and a decrease of Th17 cells in the gut mucosa of mice with colitis following pharmacological activation of PPARγ. Deletion of PPARγ in CD4+ T cells impaired mucosal iTreg and enhanced colitogenic Th17 responses in mice with CD4+ T cell-induced colitis. Thus, for the first time we provide novel molecular evidence in vivo demonstrating that PPARγ in addition to regulating CD4+ T cell differentiation also plays a major role controlling Th17 and iTreg plasticity in the gut mucosa.

This paper presents ENISI Visual, an agent-based simulator for modeling gut immunity to enteric pathogens. Gastrointestinal systems are important for in-taking food and other nutritions and gut immunity is an important part of human immune system. ENISI Visual provides quality visualizations and users can control initial cell concentrations and the simulation speed, take snapshots, and record videos. The cells are represented with different icons and the icons change colors as their states change. Users can observe real-time immune responses, including cell recruitment, cytokine and chemokine secretion and dissipation, random or chemotactic movement, cell-cell interactions, and state changes. The case study clearly shows that users can use ENISI Visual to develop models and run novel and insightful in silico experiments.

Networks are an effective abstraction for representing real systems. Consequently, network science is increasingly used in academia and industry to solve problems in many fields. Computations that determine structure properties and dynamical behaviors of networks are useful because they give insights into the characteristics of real systems. We introduce a newly built and deployed cyberinfrastructure for network science (CINET) that performs such computations, with the following features: (i) it offers realistic networks from the literature and various random and deterministic network generators; (ii) it provides many algorithmic modules and measures to study and characterize networks; (iii) it is designed for efficient execution of complex algorithms on distributed high performance computers so that they scale to large networks; and (iv) it is hosted with web interfaces so that those without direct access to high performance computing resources and those who are not computing experts can still reap the system benefits. It is a combination of application design and cyberinfrastructure that makes these features possible. To our knowledge, these capabilities collectively make CINET novel. We describe the system and illustrative use cases, with a focus on the CINET user.

Here we present the ENteric Immunity Simulator (ENISI), a modeling system for the inflammatory and regulatory immune pathways triggered by microbe-immune cell interactions in the gut. With ENISI, immunologists and infectious disease experts can test and generate hypotheses for enteric disease pathology and propose interventions through experimental infection of an in silico gut. ENISI is an agent based simulator, in which individual cells move through the simulated tissues, and engage in context-dependent interactions with the other cells with which they are in contact. The scale of ENISI is unprecedented in this domain, with the ability to simulate $10^7$ cells for 250 simulated days on 576 cores in one and a half hours, with the potential to scale to even larger hardware and problem sizes. In this paper we describe the ENISI simulator for modeling mucosal immune responses to gastrointestinal pathogens. We then demonstrate the utility of ENISI by recreating an experimental infection of a mouse with Helicobacter pylori 26695. The results identify specific processes by which bacterial virulence factors do and do not contribute to pathogenesis associated with H. pylori strain 26695. These modeling results inform general intervention strategies by indicating immunomodulatory mechanisms such as those used in inflammatory bowel disease may be more appropriate therapeutically than directly targeting specific microbial populations through vaccination or by using antimicrobials.

Clinical symptoms of gastrointestinal (GI) infections are often caused by the inflammatory response elicited to eliminate the invading microbe. Here we present ENISI, a simulator of GI immune mechanisms in response to resident commensal bacteria as well as invading pathogens and the effect on host clinical symptoms. ENISI is a tool for identifying treatment strategies that reduce inflammation-induced damage and, at the same time, ensure pathogen removal by allowing one to test plausibility of in vitro observed behavior as explanations for observations in vivo, propose behaviors not yet tested in vitro that could explain these tissue-level observations, and conduct low-cost, preliminary experiments of proposed interventions/ treatments. An example of such application is shown in which we simulate dysentery resulting from B. hyodysenteriae infection and identify aspects of the host immune pathways that lead to continued inflammation-induced tissue damage even after pathogen elimination.

Modeling environments to study epidemic outbreaks can be used as decision support tools by decision makers to support public health policies. However, although the current high performance simulation engines have become adept at rapidly simulating disease diffusion, computational environments for exploring complex mitigation strategies are fairly rudimentary. Interactive simulations provide a natural way to study the complicated co-evolution of disease dynamics and public policies. In this paper, we formalize the problem of interactive simulations and present experimental results based on an interactive simulation platform that we have recently developed.

Large scale realistic simulations of malware on mobile wireless networks have recently become an increasingly important application of high-performance computing. We propose EpiCure - an individual-based, scalable high performance computing oriented modeling environment to study malware propagation over realistic mobile networks. It is designed specifically to work on commodity cluster architectures. EpiCure runs extremely fast for realistic instances that involve: (i) large time-varying networks consisting of millions of heterogeneous individuals with time varying interaction neighborhoods, (ii) dynamic interactions between disease propagation, device behavior, and the exogenous interventions, and (iii) large number of replicated runs necessary for statistically sound estimates about the stochastic epidemic evolution. We find that EpiCure runs several orders of magnitude faster than another comparable simulation tool while delivering similar results. Beyond simple compute speed, EpiCure has been designed so that analysts can easily represent a range of interventions leading to improved human productivity and ease of use. This is an increasingly important metric in high performance computing. We illustrate EpiCure using three case studies that bring out the novel features of EpiCure.

To respond to the serious threat of pandemics (e.g. 2009 H1N1 influenza) to human society, we developed Indemics (Interactive Epidemic Simulation), an interactive, data intensive, high performance modeling environment for realtime pandemic planning, situation assessment, and course of action analysis. Indemics was built upon a model of interactive data intensive scientific computation, supporting online interactions between users and simulations and enabling epidemic simulations over detailed social contact networks and realistic representations of complex public policies and intervention strategies.

Large scale realistic epidemic simulations have recently become an increasingly important application of high-performance computing. We propose a parallel algorithm, EpiFast, based on a novel interpretation of the stochastic disease propagation in a contact network. We implement it using a master-slave computation model which allows scalability on distributed memory systems.

Preventing and controlling outbreaks of infectious diseases such as pandemic influenza is a top public health priority. We describe EpiSimdemics - a scalable parallel algorithm to simulate the spread of contagion in large, realistic social contact networks using individual-based models. EpiSimdemics is an interaction-based simulation of a certain class of stochastic reaction-diffusion processes. Straightforward simulations of such process do not scale well, limiting the use of individual-based models to very small populations. EpiSimdemics is specifically designed to scale to social networks with 100 million individuals. The scaling is obtained by exploiting the semantics of disease evolution and disease propagation in large networks. We evaluate an MPI-based parallel implementation of EpiSimdemics on a mid-sized HPC system, demonstrating that EpiSimdemics scales well. EpiSimdemics has been used in numerous sponsor defined case studies targeted at policy planning and course of action analysis, demonstrating the usefulness of EpiSimdemics in practical situations.