Demo Data from EDBT 2026 / reviewers in the wild / expert
Rupak Biswas
dblp:17/2584
· DBLP profile ↗
57ranked-venue papers
7as first author
1since 2021 · last 2024
0009-0006-6787-6033ORCID · corroborated
Domains — the database's venue-derived domains; a paper can count in several
Systems, architecture and hardware · 54 · 7 first-author · 1 since 2021Applied, interdisciplinary, general and emerging computing · 2Theory of computation · 1
Expertise — from the expertise taxonomy: the topics of the expert's papers under the CCF categories. A weight counts papers with recency: 1 for a paper about the topic, 0.3 when the topic is its context, halved every five years.
| Computer architecture, parallel and distributed computing, and storage systems
11 papers |
Performance modeling and evaluation · 35% Parallel and multicore computing · 15% High-performance computing · 13% |
Topics — the 30 heaviest of 33, each with the papers that count most for it
| Topic | Weight | Papers | Last | Evidence papers |
|---|---|---|---|---|
Performance modeling and evaluation
benchmarking |
0.4 | 4 | 2013 | An early performance evaluation of many integrated core architecture based SGI rackable computing system · SC 2013 Early performance evaluation of a "Nehalem" cluster using scientific and engineering applications · SC 2009 An Application-Based Performance Characterization of the Columbia Supercluster · SC 2005 |
Performance modeling and evaluation › parallel performance evaluation
scientific application performance |
0.1 | 2 | 2005 | An Application-Based Performance Characterization of the Columbia Supercluster · SC 2005 Evaluation of Cache-based Superscalar and Cacheless Vector Architectures for Scientific Computations · SC 2003 |
Performance modeling and evaluation › parallel system performance
cluster performance analysis |
0.1 | 1 | 2009 | Early performance evaluation of a "Nehalem" cluster using scientific and engineering applications · SC 2009 |
Processor architecture and microarchitecture
multicore design |
0.1 | 1 | 2009 | Early performance evaluation of a "Nehalem" cluster using scientific and engineering applications · SC 2009 |
Memory systems
non-uniform memory access |
0.1 | 1 | 2009 | Early performance evaluation of a "Nehalem" cluster using scientific and engineering applications · SC 2009 |
Performance modeling and evaluation › benchmarking
benchmark suite analysis |
0.1 | 1 | 2008 | Scientific application-based performance comparison of SGI Altix 4700, IBM POWER5+, and SGI ICE 8200 supercomputers · SC 2008 |
High-performance computing
performance optimization at scale |
0.1 | 2 | 2013 | An early performance evaluation of many integrated core architecture based SGI rackable computing system · SC 2013 Early performance evaluation of a "Nehalem" cluster using scientific and engineering applications · SC 2009 |
Performance modeling and evaluation
workload characterization |
0.1 | 1 | 2005 | An Application-Based Performance Characterization of the Columbia Supercluster · SC 2005 |
Parallel and multicore computing › parallel programming models
message passing |
0.1 | 2 | 2000 | Parallelization of a Dynamic Unstructured Algorithm Using Three Leading Programming Paradigms · IEEE Trans. Parallel Distributed Syst. 2000 Parallelization of a Dynamic Unstructured Application using Three Leading Paradigms · SC 1999 |
Parallel and multicore computing
MPI |
0.1 | 2 | 2000 | Parallelization of a Dynamic Unstructured Algorithm Using Three Leading Programming Paradigms · IEEE Trans. Parallel Distributed Syst. 2000 Parallelization of a Dynamic Unstructured Application using Three Leading Paradigms · SC 1999 |
Processor architecture and microarchitecture
multithreading |
0.1 | 2 | 2000 | Parallelization of a Dynamic Unstructured Algorithm Using Three Leading Programming Paradigms · IEEE Trans. Parallel Distributed Syst. 2000 Parallelization of a Dynamic Unstructured Application using Three Leading Paradigms · SC 1999 |
Parallel and multicore computing
parallel programming models |
0.1 | 2 | 2000 | Parallelization of a Dynamic Unstructured Algorithm Using Three Leading Programming Paradigms · IEEE Trans. Parallel Distributed Syst. 2000 Parallelization of a Dynamic Unstructured Application using Three Leading Paradigms · SC 1999 |
Memory systems
shared memory |
0.1 | 2 | 2000 | Parallelization of a Dynamic Unstructured Algorithm Using Three Leading Programming Paradigms · IEEE Trans. Parallel Distributed Syst. 2000 Parallelization of a Dynamic Unstructured Application using Three Leading Paradigms · SC 1999 |
Parallel and multicore computing › load balancing
dynamic load balancing |
0.0 | 2 | 2001 | Parallel Processing of Adaptive Meshes with Load Balancing · IEEE Trans. Parallel Distributed Syst. 2001 Global Load Balancing with Parallel Mesh Adaption on Distributed-Memory Systems · SC 1996 |
Distributed systems
grid computing |
0.0 | 1 | 2003 | Job Superscheduler Architecture and Performance in Computational Grid Environments · SC 2003 |
Cloud and datacenter computing
job scheduling |
0.0 | 1 | 2003 | Job Superscheduler Architecture and Performance in Computational Grid Environments · SC 2003 |
Cloud and datacenter computing
resource management |
0.0 | 1 | 2003 | Job Superscheduler Architecture and Performance in Computational Grid Environments · SC 2003 |
Processor architecture and microarchitecture
vector processor |
0.0 | 1 | 2003 | Evaluation of Cache-based Superscalar and Cacheless Vector Architectures for Scientific Computations · SC 2003 |
High-performance computing › scientific computing systems
adaptive mesh refinement |
0.0 | 1 | 2001 | Parallel Processing of Adaptive Meshes with Load Balancing · IEEE Trans. Parallel Distributed Syst. 2001 |
Parallel and multicore computing
programming models |
0.0 | 1 | 2000 | A Comparison of Three Programming Models for Adaptive Applications on the Origin2000 · SC 2000 |
High-performance computing
unstructured mesh computation |
0.0 | 2 | 2001 | Global Load Balancing with Parallel Mesh Adaption on Distributed-Memory Systems · SC 1996 Parallel Processing of Adaptive Meshes with Load Balancing · IEEE Trans. Parallel Distributed Syst. 2001 |
Parallel and multicore computing › parallel programming models › hybrid programming models
hybrid MPI/OpenMP |
0.0 | 1 | 2008 | Scientific application-based performance comparison of SGI Altix 4700, IBM POWER5+, and SGI ICE 8200 supercomputers · SC 2008 |
Memory systems
memory bandwidth |
0.0 | 1 | 2008 | Scientific application-based performance comparison of SGI Altix 4700, IBM POWER5+, and SGI ICE 8200 supercomputers · SC 2008 |
Parallel and multicore computing
parallel programming models and runtimes |
0.0 | 1 | 2008 | Scientific application-based performance comparison of SGI Altix 4700, IBM POWER5+, and SGI ICE 8200 supercomputers · SC 2008 |
Memory systems › non-uniform memory access
CC-NUMA |
0.0 | 1 | 1999 | Parallelization of a Dynamic Unstructured Application using Three Leading Paradigms · SC 1999 |
High-performance computing
scientific computing systems |
0.0 | 3 | 2000 | Parallelization of a Dynamic Unstructured Algorithm Using Three Leading Programming Paradigms · IEEE Trans. Parallel Distributed Syst. 2000 A Comparison of Three Programming Models for Adaptive Applications on the Origin2000 · SC 2000 Parallelization of a Dynamic Unstructured Application using Three Leading Paradigms · SC 1999 |
High-performance computing
supercomputing |
0.0 | 1 | 2005 | An Application-Based Performance Characterization of the Columbia Supercluster · SC 2005 |
Parallel and multicore computing
load balancing |
0.0 | 1 | 1996 | Global Load Balancing with Parallel Mesh Adaption on Distributed-Memory Systems · SC 1996 |
Cloud and datacenter computing › cluster resource management and scheduling
cluster resource management |
0.0 | 1 | 2003 | Job Superscheduler Architecture and Performance in Computational Grid Environments · SC 2003 |
Processor architecture and microarchitecture
superscalar processor |
0.0 | 1 | 2003 | Evaluation of Cache-based Superscalar and Cacheless Vector Architectures for Scientific Computations · SC 2003 |
Methods — techniques the papers use, named apart from their topics
NAS Parallel Benchmarks · 0.4MPI · 0.2HPCC benchmarks · 0.2microbenchmarks · 0.2OpenMP · 0.2microbenchmarking · 0.1turbo boost · 0.1SMT · 0.1HPC Challenge benchmarks · 0.1vectorization · 0.0
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2024 | Assessing and advancing the potential of quantum computing: A NASA case study
Eleanor Gilbert Rieffel, Ata Akbari Asanjan, M. Sohaib Alam, Namit Anand, David E. Bernal, Sophie Block, Lucas T. Brady, Steve Cotton, Zoe Gonzalez Izquierdo, Shon Grabbe, Erik Gustafson, Stuart Hadfield, Paul Aaron Lott, Filip B. Maciejewski, Salvatore Mandrà, Jeffrey Marshall, Gianni Mossi, Humberto Munoz Bauza, Jason Saied, Nishchay Suri, Davide Venturelli, Zhihui Wang 0012, Rupak Biswas |
Future Gener. Comput. Syst. | 23 |
| 2020 | Quantum Annealing Applied to De-Conflicting Optimal Trajectories for Air Traffic ManagementabstractWe present the mapping of a class of simplified air traffic management problems (strategic conflict resolution) to quadratic unconstrained Boolean optimization problems. The mapping is performed through an original representation of the conflict-resolution problem in terms of a conflict graph, where the nodes of the graph represent flights and the edges represent a potential conflict between flights. The representation allows a natural decomposition of a real-world instance related to wind-optimal trajectories over the Atlantic Ocean into smaller subproblems that can be discretized and are amenable to be programmed in quantum annealers. In this paper, we tested the new programming techniques, and we benchmark the hardness of the instances using both classical solvers and the D-Wave 2X and D-Wave 2000Q quantum chip. The preliminary results show that for reasonable modeling choices, the most challenging subproblems which are programmable in the current devices are solved to optimality with 99% of probability within a second of annealing time. Tobias Stollenwerk, Bryan O'Gorman, Davide Venturelli, Salvatore Mandrà, Olga Rodionova, Hokkwan Ng, Banavar Sridhar, Eleanor Gilbert Rieffel, Rupak Biswas |
IEEE Trans. Intell. Transp. Syst. | 9 |
| 2017 | High-End Computing for Next-Generation Scientific Discovery
Rupak Biswas, David Donofrio, Leonid Oliker |
Parallel Comput. | 1 |
| 2017 | A NASA perspective on quantum computing: Opportunities and challenges
Rupak Biswas, Zhang Jiang, Kostya Kechezhi, Sergey Knysh, Salvatore Mandrà, Bryan O'Gorman, Alejandro Perdomo-Ortiz, Andre Petukhov, John Realpe-Gomez, Eleanor Gilbert Rieffel, Davide Venturelli, Fedir Vasko, Zhihui Wang 0012 |
Parallel Comput. | 1 |
| 2016 | Performance evaluation of Amazon Elastic Compute Cloud for NASA high-performance computing applicationsabstractAbstract Cloud computing environments are now widely available and are being increasingly utilized for technical computing. They are also being touted for high‐performance computing (HPC) applications in science and engineering. For example, Amazon Elastic Compute Cloud (EC2) Services offers specialized Cluster Compute instance types to run HPC applications. In this paper, we compare the performance characteristics of two Amazon EC2 HPC instance types with that of National Aeronautics and Space Administration's (NASA) Pleiades supercomputer, an SGI® ICE™ cluster. For this study, we utilized the HPC Challenge kernels and the NAS Parallel Benchmarks along with four full‐scale applications from the repertoire of codes that are being used by NASA scientists and engineers. We compare the total runtime of these codes for varying number of cores. We also break out the computation and communication times for a subset of these applications to explore the effect of interconnect differences on the two systems. In general, the single node performance of the two platforms is equivalent. However, for most of the codes when scaling to larger core counts, the performance of the EC2 HPC instances generally lags that of Pleiades because of worse network performance of the former. In addition to analyzing application performance, we also briefly touch upon the overhead due to virtualization and the usability of cloud environments such as Amazon EC2. Published 2013. This article is a U.S. Government work and is in the public domain in the U.S.A. Piyush Mehrotra, M. Jahed Djomehri, Steve Heistand, Robert Hood, Haoqiang Jin, Arthur Lazanoff, Subhash Saini, Rupak Biswas |
Concurr. Comput. Pract. Exp. | 8 |
| 2013 | An early performance evaluation of many integrated core architecture based SGI rackable computing systemabstractIntel recently introduced the Xeon Phi coprocessor based on the Many Integrated Core architecture featuring 60 cores with a peak performance of 1.0 Tflop/s. NASA has deployed a 128-node SGI Rackable system where each node has two Intel Xeon E2670 8-core Sandy Bridge processors along with two Xeon Phi 5110P coprocessors. We have conducted an early performance evaluation of the Xeon Phi. We used microbenchmarks to measure the latency and bandwidth of memory and interconnect, I/O rates, and the performance of OpenMP directives and MPI functions. We also used OpenMP and MPI versions of the NAS Parallel Benchmarks along with two production CFD applications to test four programming modes: offload, processor native, coprocessor native and symmetric (processor plus coprocessor). In this paper we present preliminary results based on our performance evaluation of various aspects of a Phi-based system. Subhash Saini, Haoqiang Jin, Dennis C. Jespersen, Huiyu Feng, M. Jahed Djomehri, William Arasin, Robert Hood, Piyush Mehrotra, Rupak Biswas |
SC | 9 |
| 2012 | I/O performance characterization of Lustre and NASA applications on PleiadesabstractIn this paper we study the performance of the Lustre file system using five scientific and engineering applications representative of NASA workload on large-scale supercomputing systems such as NASA's Pleiades. In order to facilitate the collection of Lustre performance metrics, we have developed a software tool that exports a wide variety of client and server-side metrics using SGI's Performance Co-Pilot (PCP), and generates a human readable report on key metrics at the end of a batch job. These performance metrics are (a) amount of data read and written, (b) number of files opened and closed, and (c) remote procedure call (RPC) size distribution (4 KB to 1024 KB, in powers of 2) for I/O operations. RPC size distribution measures the efficiency of the Lustre client and can pinpoint problems such as small write sizes, disk fragmentation, etc. These extracted statistics are useful in determining the I/O pattern of the application and can assist in identifying possible improvements for users' applications. Information on the number of file operations enables a scientist to optimize the I/O performance of their applications. Amount of I/O data helps users choose the optimal stripe size and stripe count to enhance I/O performance. In this paper, we demonstrate the usefulness of this tool on Pleiades for five production quality NASA scientific and engineering applications. We compare the latency of read and write operations under Lustre to that with NFS by tracing system calls and signals. We also investigate the read and write policies and study the effect of page cache size on I/O operations. We examine the performance impact of Lustre stripe size and stripe count along with performance evaluation of file per process and single shared file accessed by all the processes for NASA workload using parameterized IOR benchmark. Subhash Saini, Jason Rappleye, Johnny Chang, David Barker, Piyush Mehrotra, Rupak Biswas |
HiPC | 6 |
| 2011 | The impact of hyper-threading on processor resource utilization in production applicationsabstractIntel provides Hyper-Threading (HT) in processors based on its Pentium and Nehalem micro-architecture such as the Westmere-EP. HT enables two threads to execute on each core in order to hide latencies related to data access. These two threads can execute simultaneously, filling unused stages in the functional unit pipelines. To aid better understanding of HT-related issues, we collect Performance Monitoring Unit (PMU) data (instructions retired; unhalted core cycles; L2 and L3 cache hits and misses; vector and scalar floating-point operations, etc.). We then use the PMU data to calculate a new metric of efficiency in order to quantify processor resource utilization and make comparisons of that utilization between single-threading (ST) and HT modes. We also study performance gain using unhalted core cycles, code efficiency of using vector units of the processor, and the impact of HT mode on various shared resources like L2 and L3 cache. Results using four full-scale, production-quality scientific applications from computational fluid dynamics (CFD) used by NASA scientists indicate that HT generally improves processor resource utilization efficiency, but does not necessarily translate into overall application performance gain. Subhash Saini, Haoqiang Jin, Robert Hood, David Barker, Piyush Mehrotra, Rupak Biswas |
HiPC | 6 |
| 2011 | Performance Analysis of CFD Application Cart3D Using MPInside and Performance Monitor Unit Data on Nehalem and Westmere Based SupercomputersabstractCart3D is a computational fluid dynamics (CFD) application aimed at conceptual and preliminary design of aerospace vehicles with complex geometries. It is widely used by design engineers at NASA, Department of Defense and aerospace companies in the USA. We present detailed performance analysis of Cart3D using two tools SGI MPInside and op_scope that collects hardware counter data from Intel Performance Monitoring Unit (PMU) on supercomputers based on Nehalem micro-architecture. Using these tools, we have done dynamic profiling of Cart3D (compute time, communication time and I/O time), along with dynamic profiling of MPI functions (MPI_Sendrecv, MPI_Bcast, MPI_Isend, MPI_Irecv, MPI_Allreduce, MPI_Barrier, etc.) with respect to message size of each rank and time consumed by each function. MPI communication is further analyzed by studying the performance of MPI functions used in this application as a function of message size and number of cores. Using these tools we have also studied efficiency of the processor to measure its effective utilization, efficiency of the floating-point units, percentage of vectorization and percentage of data coming from L2 cache, L3 cache, and main memory. This study was performed on two computing sub-systems based on quad-core Nehalem-EP and hex-core West mere-EP processors that are part of Pleiades an SGI Altix ICE at NASA Ames Research Center. Subhash Saini, Piyush Mehrotra, Kenichi Taylor, Michael J. Aftosmis, Rupak Biswas |
HPCC | 5 |
| 2011 | High performance computing using MPI and OpenMP on multi-core parallel systems
Haoqiang Jin, Dennis C. Jespersen, Piyush Mehrotra, Rupak Biswas, Lei Huang 0006, Barbara M. Chapman |
Parallel Comput. | 4 |
| 2011 | Emerging programming paradigms for large-scale scientific computing
Leonid Oliker, Rajesh Nishtala, Rupak Biswas |
Parallel Comput. | 3 |
| 2010 | Performance Analysis of Scientific and Engineering Applications Using MPInside and TAUabstractIn this paper, we present performance analysis of two NASA applications using performance tools like Tuning and Analysis Utilities (TAU) and SGI MP Inside. MITgcmUV and OVERFLOW are two production-quality applications used extensively by scientists and engineers at NASA. MITgcmUV is a global ocean simulation model, developed by the Estimating the Circulation and Climate of the Ocean (ECCO) Consortium, for solving the fluid equations of motion using the hydrostatic approximation. OVERFLOW is a general-purpose Navier-Stokes solver for computational fluid dynamics (CFD) problems. Using these tools, we analyze the MPI functions (MPI_Sendrecv, MPI_Bcast, MPI_Reduce, MPI_Allreduce, MPI_Barrier, etc.) with respect to message size of each rank, time consumed by each function, and how ranks communicate. MPI communication is further analyzed by studying the performance of MPI functions used in these two applications as a function of message size and number of cores. Finally, we present the compute time, communication time, and I/O time as a function of the number of cores. Subhash Saini, Piyush Mehrotra, Kenichi Taylor, Sameer Shende, Rupak Biswas |
HPCC | 5 |
| 2010 | Performance impact of resource contention in multicore systemsabstractResource sharing in commodity multicore processors can have a significant impact on the performance of production applications. In this paper we use a differential performance analysis methodology to quantify the costs of contention for resources in the memory hierarchy of several multicore processors used in high-end computers. In particular, by comparing runs that bind MPI processes to cores in different patterns, we can isolate the effects of resource sharing. We use this methodology to measure how such sharing affects the performance of four applications of interest to NASA-OVERFLOW, MITgcm, Cart3D, and NCC. We also use a subset of the HPCC benchmarks and hardware counter data to help interpret and validate our findings. We conduct our study on high-end computing platforms that use four different quad-core microprocessors-Intel Clovertown, Intel Harpertown, AMD Barcelona, and Intel Nehalem-EP. The results help further our understanding of the requirements these codes place on their production environments and also of each computer's ability to deliver performance. Robert Hood, Haoqiang Jin, Piyush Mehrotra, Johnny Chang, M. Jahed Djomehri, Sharad Gavali, Dennis C. Jespersen, Kenichi Taylor, Rupak Biswas |
IPDPS | 9 |
| 2009 | Early performance evaluation of a "Nehalem" cluster using scientific and engineering applicationsabstractIn this paper, we present an early performance evaluation of a 624-core cluster based on the Intel® Xeon® Processor 5560 (code named "Nehalem-EP", and referred to as Xeon 5560 in this paper)---the third-generation quad-core architecture from Intel. This is the first processor from Intel with a non-uniform memory access (NUMA) architecture managed by on-chip integrated memory controller. It employs a point-to-point interconnect called the Intel® QuickPath Interconnect (QPI) between processors and to the input/output (I/O) hub. It also introduces to a quad-core architecture both Intel's hyper-threading technology (or simultaneous multi-threading, "SMT") and Intel® Turbo Boost Technology ("Turbo mode") that automatically allow processor cores to run faster than the base operating frequency if the processor is operating below rated power, temperature, and current specification limits. It can be engaged with any number of cores or logical processors enabled and active. We critically evaluate these features using the High Performance Computing Challenge (HPCC) benchmarks, NAS Parallel Benchmarks (NPB), and four full-scale scientific applications. We compare and contrast the results of a cluster based on the Xeon 5560 with an SGI® Altix® ICE 8200EX cluster of quad-core Intel® Xeon® 5472 Processor ("Xeon 5472" from here on) and another cluster of Intel® Xeon® 5462 Processor ("Xeon 5462"; the Xeon 5400 Series Processors are previous generation quad-core Intel processors and were code named Harpertown). Subhash Saini, Andrey Naraikin, Rupak Biswas, David Barkai, Timothy Sandstrom |
SC | 3 |
| 2009 | Revolutionary technologies for acceleration of emerging petascale applications
Rupak Biswas, Leonid Oliker, Jeffrey S. Vetter |
Parallel Comput. | 1 |
| 2008 | Scientific application-based performance comparison of SGI Altix 4700, IBM POWER5+, and SGI ICE 8200 supercomputersabstractThe suitability of next-generation high-performance computing systems for petascale simulations will depend on various performance factors attributable to processor, memory, local and global network, and input/output characteristics. In this paper, we evaluate performance of new dual-core SGI Altix 4700, quad-core SGI Altix ICE 8200, and dual-core IBM POWER5+ systems. To measure performance, we used micro-benchmarks from High Performance Computing Challenge (HPCC), NAS Parallel Benchmarks (NPB), and four real-world applications- three from computational fluid dynamics (CFD) and one from climate modeling. We used the micro-benchmarks to develop a controlled understanding of individual system components, then analyzed and interpreted performance of the NPBs and applications. We also explored the hybrid programming model (MPI+OpenMP) using multi-zone NPBs and the CFD application OVERFLOW-2. Achievable application performance is compared across the systems. For the ICE platform, we also investigated the effect of memory bandwidth on performance by testing 1, 2, 4, and 8 cores per node. Subhash Saini, Dale Talcott, Dennis C. Jespersen, M. Jahed Djomehri, Haoqiang Jin, Rupak Biswas |
SC | 6 |
| 2006 | Design and Performance of a Heterogeneous Grid Partitioner
Daniel J. Harvey, Sajal K. Das 0001, Rupak Biswas |
Algorithmica | 3 |
| 2005 | Integrated Performance Monitoring of a Cosmology Application on Leading HEC PlatformsabstractThe cosmic microwave background (CMB) is an exquisitely sensitive probe of the fundamental parameters of cosmology. Extracting this information is computationally intensive, requiring massively parallel computing and sophisticated numerical algorithms. In this work we present MADbench, a lightweight version of the MADCAP CMB power spectrum estimation code that retains the operational complexity and integrated system requirements. In addition, to quantify communication behavior across a variety of architectural platforms, we introduce the integrated performance monitoring (IPM) package: a portable, lightweight, and scalable tool for effectively extracting MPI message-passing overheads. A performance characterization study is conducted on some of the world's most powerful supercomputers, including the superscalar Seaborg (IBM Power3+) and CC-NUMA Columbia (SGIAltix), as well as the vector-based Earth Simulator (NEC SX-6 enhanced) and Phoenix (Cray XI) systems. In-depth analysis shows that in order to bridge the gap between theoretical and sustained system performance, it is critical to gain a clear understanding of how the distinct parts of large-scale parallel applications interact with the individual subcomponents of HEC platforms. Julian Borrill, Jonathan Carter 0002, Leonid Oliker, David Skinner, Rupak Biswas |
ICPP | 5 |
| 2005 | An Application-Based Performance Characterization of the Columbia SuperclusterabstractColumbia is a 10,240-processor supercluster consisting of 20 Altix nodes with 512 processors each, and currently ranked as one of the fastest computers in the world. In this paper, we present the performance characteristics of Columbia obtained on up to four computing nodes interconnected via the InfiniBand and/or NUMAlink4 communication fabrics. We evaluate floatingpoint performance, memory bandwidth, message passing communication speeds, and compilers using a subset of the HPC Challenge benchmarks, and some of the NAS Parallel Benchmarks including the multi-zone versions. We present detailed performance results for three scientific applications of interest to NASA, one from molecular dynamics, and two from computational fluid dynamics. Our results show that both the NUMAlink4 and In- finiBand interconnects hold promise for multi-node application scaling to at least 2048 processors. Rupak Biswas, M. Jahed Djomehri, Robert Hood, Haoqiang Jin, Cetin C. Kiris, Subhash Saini |
SC | 1 |
| 2005 | Performance evaluation of the SX-6 vector architecture for scientific computationsabstractThe growing gap between sustained and peak performance for scientific applications is a well-known problem in high-performance computing. The recent development of parallel vector systems offers the potential to reduce this gap for many computational science codes and deliver a substantial increase in computing capabilities. This paper examines the intranode performance of the NEC SX-6 vector processor, and compares it against the cache-based IBM Power3 and Power4 superscalar architectures, across a number of key scientific computing areas. First, we present the performance of a microbenchmark suite that examines many low-level machine characteristics. Next, we study the behavior of the NAS Parallel Benchmarks. Finally, we evaluate the performance of several scientific computing codes. Overall results demonstrate that the SX-6 achieves high performance on a large fraction of our application suite and often significantly outperforms the cache-based architectures. However, certain classes of applications are not easily amenable to vectorization and would require extensive algorithm and implementation reengineering to utilize the SX-6 effectively. Copyright © 2005 John Wiley & Sons, Ltd. Leonid Oliker, Andrew Canning, Jonathan Carter 0002, John Shalf, David Skinner, Stéphane Ethier, Rupak Biswas, M. Jahed Djomehri, Rob F. Van der Wijngaart |
Concurr. Pract. Exp. | 7 |
| 2003 | A Hierarchical and Distributed Approach for Mapping Large Applications to Heterogeneous Grids Using Genetic AlgorithmsabstractIn this paper, we propose a distributed approach for mapping a single large application to a heterogeneous grid environment. To minimize the execution time of the parallel application, we distribute the mapping overhead to the available nodes of the grid. This approach not only provides a fast mapping of tasks to resources but is also scalable. We adopt a hierarchical grid model and accomplish the job of mapping tasks to this topology using a scheduler tree. Results show that our three-phase algorithm provides high quality mappings, and is fast and scalable. Soumya Sanyal 0002, Sajal K. Das 0001, Rupak Biswas |
CLUSTER | 4 |
| 2003 | Performance Analysis of a Hybrid Overset Multi-block Application on Multiple Architectures
M. Jahed Djomehri, Rupak Biswas |
HiPC | 2 |
| 2003 | Designing an Efficient Partitioning Algorithm for Grid Environments with Application to N-body Problems
Daniel J. Harvey, Sajal K. Das 0001, Rupak Biswas |
ICCSA (2) | 3 |
| 2003 | Performance of a Heterogeneous Grid Partitioner for N-body ApplicationsabstractAn important characteristic of distributed grids is that they allow geographically separated multicomputers to be tied together in a transparent virtual environment to solve large-scale computational problems. However, many of these applications require effective runtime load balancing for the resulting solutions to be viable. Recently, we developed a latency tolerant partitioner, called MinEX, specifically for use in distributed grid environments. We compare the performance of MinEX to that of METIS using simulated heterogeneous grid configurations. A solver for the classical N-body problem is implemented to provide a benchmark for the comparisons. Simulation results show that MinEX provides superior quality partitions while being competitive to METIS in speed of execution. Daniel J. Harvey, Sajal K. Das 0001, Rupak Biswas |
ICPP | 3 |
| 2003 | Evaluation of Cache-based Superscalar and Cacheless Vector Architectures for Scientific ComputationsabstractThe growing gap between sustained and peak performance for scientific applications is a well-known problem in high end computing. The recent development of parallel vector systems offers the potential to bridge this gap for many computational science codes and deliver a substantial increase in comput-ing capabilities. This paper examines the intranode performance of the NEC SX-6 vector processor and the cache-based IBM Power3/4 superscalar architectures across a number of scientific computing areas. First, we present the performance of a microbenchmark suite that examines low-level machine characteristics. Next, we study the behavior of the NAS Parallel Benchmarks. Finally, we evaluate the performance of several scientific computing codes. Results demonstrate that the SX-6 achieves high performance on a large fraction of our applications and often significantly outperforms the cache-based architectures. However, certain applications are not easily amenable to vectorization and would require extensive algorithm and implementation reengineering to utilize the SX-6 effectively. Leonid Oliker, Andrew Canning, Jonathan Carter 0002, John Shalf, David Skinner, Stéphane Ethier, Rupak Biswas, M. Jahed Djomehri, Rob F. Van der Wijngaart |
SC | 7 |
| 2003 | Job Superscheduler Architecture and Performance in Computational Grid EnvironmentsabstractComputational grids hold great promise in utilizing geographically separated heterogeneous resources to solve large-scale complex scientific problems. However, a number of major technical hurdles, including distributed resource management and effective job scheduling, stand in the way of realizing these gains. In this paper, we propose a novel grid superscheduler architecture and three distributed job migration algorithms. We also model the critical interaction between the superscheduler and autonomous local schedulers. Extensive performance comparisons with ideal, central, and local schemes using real workloads from leading computational centers are conducted in a simulation environment. Additionally, synthetic workloads are used to perform a detailed sensitivity analysis of our superscheduler. Several key metrics demonstrate that substantial performance gains can be achieved via smart superscheduling in distributed computational grids. Hongzhang Shan, Leonid Oliker, Rupak Biswas |
SC | 3 |
| 2003 | High-performance Java codes for computational fluid dynamicsabstractAbstract The computational science community is reluctant to write large‐scale computationally‐intensive applications in Java due to concerns over Java's poor performance, despite the claimed software engineering advantages of its object‐oriented features. Naive Java implementations of numerical algorithms can perform poorly compared to corresponding Fortran or C implementations. To achieve high performance, Java applications must be designed with good performance as a primary goal. This paper presents the object‐oriented design and implementation of two real‐world applications from the field of computational fluid dynamics (CFD): a finite‐volume fluid flow solver (LAURA, from NASA Langley Research Center) and an unstructured mesh adaptation algorithm (2D_TAG, from NASA Ames Research Center). This work builds on our previous experience with the design of high‐performance numerical libraries in Java. We examine the performance of the applications using the currently available Java infrastructure and show that the Java version of the flow solver LAURA performs almost within a factor of 2 of the original procedural version. Our Java version of the mesh adaptation algorithm 2D_TAG performs within a factor of 1.5 of its original procedural version on certain platforms. Our results demonstrate that object‐oriented software design principles are not necessarily inimical to high performance. Copyright © 2003 John Wiley & Sons, Ltd. Christopher Riley, Siddhartha Chatterjee, Rupak Biswas |
Concurr. Comput. Pract. Exp. | 3 |
| 2003 | Performance enhancement strategies for multi-block overset grid CFD applications
M. Jahed Djomehri, Rupak Biswas |
Parallel Comput. | 2 |
| 2003 | Message passing and shared address space parallelism on an SMP cluster
Hongzhang Shan, Jaswinder Pal Singh, Leonid Oliker, Rupak Biswas |
Parallel Comput. | 4 |
| 2002 | MinEX: a latency-tolerant dynamic partitioner for grid computing applications
Sajal K. Das 0001, Daniel J. Harvey, Rupak Biswas |
Future Gener. Comput. Syst. | 3 |
| 2002 | Adaptive Load-Balancing Algorithms Using Symmetric Broadcast Networks
Sajal K. Das 0001, Daniel J. Harvey, Rupak Biswas |
J. Parallel Distributed Comput. | 3 |
| 2002 | A Comparison of Three Programming Models for Adaptive Applications on the Origin2000
Hongzhang Shan, Jaswinder Pal Singh, Leonid Oliker, Rupak Biswas |
J. Parallel Distributed Comput. | 4 |
| 2001 | Latency Hiding in Dynamic Partitioning and Load Balancing of Grid Computing ApplicationsabstractThe Information Power Grid (IPG) concept developed by NASA is aimed to provide a metacomputing platform for large-scale distributed computations, by hiding the intricacies of a highly heterogeneous environment and yet maintaining adequate security. We propose a latency-tolerant partitioning scheme that dynamically balances processor workloads on the IPG, and minimizes data movement and runtime communication. By simulating an unsteady adaptive mesh application on a wide area network, we study the performance of our load balancer under the Globus environment. The number of IPG nodes, the number of processors per node, and the interconnect speeds are parameterized to derive conditions under which the IPG would be suitable for parallel distributed processing of such applications. Experimental results demonstrate that effective solutions are achieved when. The IPG nodes are connected by a high-speed asynchronous interconnection network. Sajal K. Das 0001, Daniel J. Harvey, Rupak Biswas |
CCGRID | 3 |
| 2001 | A Latency-Tolerant Partitioner for Distributed Computing on the Information Power GridabstractNASA's Information Power Grid (IPG) is an infrastructure designed to harness the power of geographically distributed computers, databases and human expertise, in order to solve large-scale realistic computational problems. This type of a metacomputing environment is necessary to present a unified virtual machine to application developers that hides the intricacies of a highly heterogeneous environment and yet maintains adequate security. In this paper, we present a novel partitioning scheme, called MinEX, that dynamically balances processor workloads while minimizing data movement and runtime communication, for applications that are executed in a parallel distributed fashion on the IPG. Experimental results show that MinEX is an effective load balancer in a distributed IPG environment. Sajal K. Das 0001, Daniel J. Harvey, Rupak Biswas |
IPDPS | 3 |
| 2001 | Message Passing Vs. Shared Address Space on a Clusters of SMPsabstractThe emergence of scalable computer architectures using clusters of PCs (or PC-SMPs) with commodity networking has made them attractive platforms for high-end scientific computing. Currently, message passing (MP) and shared address space (SAS) are the two leading programming paradigms for these systems. MP has been standardized with MPI, and is the most common and mature parallel programming approach. However, MP code development can be extremely difficult, especially for irregularly structured computations. SAS offers substantial ease of programming, but may suffer from performance limitations due to poor spatial locality and high protocol overhead. In this paper, we compare the performance of and programming effort required for six applications under both programming models on a 32-CPU PC-SMP cluster. Our application suite consists of codes that typically do not exhibit scalable performance under shared-memory programming due to their high communication-to-computation ratios and complex communication patterns. Results indicate that SAS can achieve about half the parallel efficiency of MPI for most of our applications; however on certain classes of problems, SAS performance is competitive with MPI. Hongzhang Shan, Jaswinder Pal Singh, Leonid Oliker, Rupak Biswas |
IPDPS | 4 |
| 2001 | Parallel Processing of Adaptive Meshes with Load BalancingabstractMany scientific applications involve grids that lack a uniform underlying structure. These applications are often also dynamic in nature in that the grid structure significantly changes between successive phases of execution. In parallel computing environments, mesh adaptation of unstructured grids through selective refinement/coarsening has proven to be an effective approach. However, achieving load balance while minimizing interprocessor communication and redistribution costs is a difficult problem. Traditional dynamic load balancers are mostly inadequate because they lack a global view of system loads across processors. In this paper, we propose a novel and general-purpose load balancer that utilizes symmetric broadcast networks (SBN) as the underlying communication topology and compare its performance with a successful global load balancing environment, called PLUM, specifically created to handle adaptive unstructured applications. Our experimental results on an IBM SP2 demonstrate that the SBN-based load balancer achieves lower redistribution costs than that under PLUM by overlapping processing and data migration. Sajal K. Das 0001, Daniel J. Harvey, Rupak Biswas |
IEEE Trans. Parallel Distributed Syst. | 3 |
| 2000 | Parallel and Distributed Computational Fluid Dynamics: Experimental Results and Challenges
M. Jahed Djomehri, Rupak Biswas, Rob F. Van der Wijngaart, Maurice Yarrow |
HiPC | 2 |
| 2000 | A Comparison of Three Programming Models for Adaptive Applications on the Origin2000
Hongzhang Shan, Jaswinder Pal Singh, Leonid Oliker, Rupak Biswas |
SC | 4 |
| 2000 | Self-Avoiding Walks over Adaptive Unstructured GridsabstractIn this paper, we present self-avoiding walks as a novel technique to ‘linearize’ an unstructured mesh. Unlike space-filling curves which are based on a geometric embedding, our strategy is combinatorial since it uses the mesh connectivity only. We formulate a linear time-complexity algorithm for the construction of these self-avoiding walks over a triangular mesh. We also show how the concept can be easily modified for adaptive grids that are generated in a hierarchical manner based on a set of simple rules, and made amenable for efficient parallelization. We suggest a metric that might be used to evaluate the quality of such walks and present some sample results. The proposed locality-enhancing approach should be very useful in the runtime partitioning and load balancing of adaptive unstructured grids. Copyright © 2000 John Wiley & Sons, Ltd. Gerd Heber, Rupak Biswas, Guang R. Gao |
Concurr. Pract. Exp. | 2 |
| 2000 | Graph partitioning and parallel computing
Rupak Biswas, Bruce Hendrickson, George Karypis |
Parallel Comput. | 1 |
| 2000 | Parallel tetrahedral mesh adaptation with dynamic load balancing
Leonid Oliker, Rupak Biswas, Harold N. Gabow |
Parallel Comput. | 2 |
| 2000 | Parallelization of a Dynamic Unstructured Algorithm Using Three Leading Programming ParadigmsabstractThe success of parallel computing in solving real-life computationally intensive problems relies on their efficient mapping and execution on large-scale multiprocessor architectures. Many important applications are both unstructured and dynamic in nature, making their efficient parallel implementation a daunting task. This paper presents the parallelization of a dynamic unstructured mesh adaptation algorithm using three popular programming paradigms on three leading supercomputers. We examine an MPI message-passing implementation on the Cray T3E and the SGI Origin2000, a shared-memory implementation using the cache coherent nonuniform memory access (CC-NUMA) feature of the Origin2000, and a multithreaded version on the newly released Tera Multithreaded Architecture (MTA). We compare several critical factors of this parallel code development, including runtime, scalability, programmability, portability, and memory overhead. Our overall results demonstrate that multithreaded systems offer tremendous potential for quickly and efficiently solving some of the most challenging real-life problems on parallel computers. Leonid Oliker, Rupak Biswas |
IEEE Trans. Parallel Distributed Syst. | 2 |
| 1999 | Parallelization of a Dynamic Unstructured Application using Three Leading ParadigmsabstractThe success of parallel computing in solving real-life computationally-intensive problems relies on their efficient mapping and execution on large-scale multiprocessor architectures.Many important applications are both unstructured and dynamic in nature, making their efficient parallel implementation a daunting task.This paper presents the parallelization of a dynamic unstructured mesh adaptation algorithm using three popular programming paradigms on three leading supercomputers.We examine an MPI message-passing implementation on the Cray T3E and the SGI Origin2000, a shared-memory implementation using cache coherent nonuniform memory access (CC-NUMA) of the Origin2000, and a multithreaded version on the newly-released Tera Multithreaded Architecture (MTA).We compare several critical factors of this parallel code development, including runtime, scalability, programmability, and memory overhead.Our overall results demonstrate that multithreaded systems offer tremendous potential for quickly and efficiently solving some of the most challenging real-life problems on parallel computers. Leonid Oliker, Rupak Biswas |
SC | 2 |
| 1998 | Performance Analysis and Portability of the PLUM Load Balancing System
Leonid Oliker, Rupak Biswas, Harold N. Gabow |
Euro-Par | 2 |
| 1998 | Parallel Processing of Adaptive Meshes with Load BalancingabstractMany scientific applications involve grids that lack a uniform underlying structure. These applications are often also dynamic in, nature in the sense that the grid structure significantly changes between successive phases of execution. In parallel computing environments, mesh adaptation of unstructured grids through selective refinement/coarsening has proven to be an effective approach. However, achieving load balance while minimizing interprocessor communication and redistribution costs is a difficult problem. Traditional dynamic load balancers are mostly inadequate because they lack a global view of system loads across processors. In this paper, we present a novel, general-purpose load balancer that utilizes symmetric broadcast networks (SBN) as the underlying communication topology. The experimental results on the IBM SP2 demonstrate that performance of the SBN-based load balancer is comparable to results achieved under PLUM, a global load balancing environment created to handle adaptive unstructured applications. Sajal K. Das 0001, Daniel J. Harvey, Rupak Biswas |
ICPP | 3 |
| 1998 | Dynamic Load Balancing for Adaptive Meshes Using Symmetric Broadcast NetworksabstractArticle Free Access Share on Dynamic load balancing for adaptive meshes using symmetric broadcast networks Authors: Sajal K. Das Department of Computer Sciences, University of North Texas, Denton, TX Department of Computer Sciences, University of North Texas, Denton, TXView Profile , Daniel J. Harvey Department of Computer Sciences, University of North Texas, Denton, TX Department of Computer Sciences, University of North Texas, Denton, TXView Profile , Rupak Biswas MRJ Technology Solutions, NASA Ames Research Center, Moffett Field, CA MRJ Technology Solutions, NASA Ames Research Center, Moffett Field, CAView Profile Authors Info & Claims ICS '98: Proceedings of the 12th international conference on SupercomputingJuly 1998 Pages 417–424https://doi.org/10.1145/277830.277934Published:13 July 1998Publication History 0citation321DownloadsMetricsTotal Citations0Total Downloads321Last 12 Months13Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF Sajal K. Das 0001, Daniel J. Harvey, Rupak Biswas |
International Conference on Supercomputing | 3 |
| 1998 | PLUM: Parallel Load Balancing for Adaptive Unstructured Meshes
Leonid Oliker, Rupak Biswas |
J. Parallel Distributed Comput. | 2 |
| 1998 | HARP: A Dynamic Spectral Partitioner
Horst D. Simon, Andrew Sohn, Rupak Biswas |
J. Parallel Distributed Comput. | 3 |
| 1997 | Design of Novel Load-Balancing Algorithms with Implementations on an IBM SP1
Sajal K. Das 0001, Daniel J. Harvey, Rupak Biswas |
Euro-Par | 3 |
| 1997 | Load balancing sequences of unstructured adaptive gridsabstractMesh adaption is a powerful tool for efficient unstructured grid computations but causes load imbalance on multiprocessor systems. To address this problem, we have developed PLUM, an automatic portable framework for performing adaptive large scale numerical computations in a message passing environment. The paper makes several important additions to our previous work. First, a new remapping cost model is presented and empirically validated on an SP2. Next, our load balancing strategy is applied to sequences of dynamically adapted unstructured grids. Results indicate that our framework is effective on many processors for both steady and unsteady problems with several levels of adaption. Additionally, we demonstrate that a coarse starting mesh produces high quality load balancing, at a fraction of the cost required for a fine initial mesh. Finally, we show that the data remapping overhead can be significantly reduced by applying our heuristic processor reassignment algorithm. Rupak Biswas, Leonid Oliker |
HiPC | 1 |
| 1997 | Adaptive load-balancing algorithms using symmetric broadcast networks: performance study on an IBM SP2abstractIn a distributed-computing environment, it is important to ensure that the processor work loads are adequately balanced. Among numerous load-balancing algorithms, a unique approach due to Das and Prasad defines a symmetric broadcast network (SBN) that provides a robust communication pattern among the processors in a topology-independent manner. In this paper, we propose and analyze three SBN-based load-balancing algorithms, and implement them on an SP2. A thorough experimental study with Poisson-distributed synthetic loads demonstrates that these algorithms are very effective in balancing system load while minimizing processor idle time. They also compare favorably with several existing techniques. Sajal K. Das 0001, Daniel J. Harvey, Rupak Biswas |
ICPP | 3 |
| 1997 | Efficient Load Balancing and Data Remapping for Adaptive Grid CalculationsabstractMesh adaption is a powerful tool for efficient unstructuredgrid computations but causes load imbalance among processors on a parallel machine, We present a novel method to dynamically balance the processor workloads with a global view.This paper presents, for the first time, the implementation and integration of all major components within our dynamic load balancing strategy for adaptive grid calculations.Mesh adaption, repartitioning, processor assignment, and remapping are critical components of the framework that must be accomplished rapidly and efficiently so ss not to cause a significant overhead to the numerical simulation.Previous results indicated that mesh repartitioning and data remapping are potential bottlenecks for performing large-scale scientific calculations.We resolve these issues and demonstrate that our framework remains viable on a huge number of processors. Leonid Oliker, Rupak Biswas |
SPAA | 2 |
| 1997 | HARP: A Fast Spectral PartitionerabstractPartitioningunstructured graphsis central to the parallel solution of computational science and engineering problems, Spectral partitioners, such recursive spectral bisection (RSB), have proven effective in generating high-quality partitions of realistically-sized meshes.The major problem which hindered their widespread use was their long execution times.This paper presents a new inertial spectral partitioned, called HARP.The main objective of the proposed approach is to quickly partition the meshes at runtime in a manner that works efficiently for real applications in the context of distributed-memory machines.The underlying principle of HARP is to find the eigenvectors of the unpartitioned vertices and then project them onto the eigenvectors of the originaJ mesh.Results for various meshes ranging in size from 1000 to 100,000 vertices indicate that HARP can indeed partition meshes rapidly at runtime, Experimental results show that our largest mesh can be partitioned sequentially in only a few seconds on an SP2 which is several times faster than other spectral partitioners while maintaining the solution quality of the proven RSB method.A parallel MPI version of HARP has also been implemented on IBM SP2 and Cray T3E.PrrralIel HARP, running on 64 processors SP2 and T3E, can partition a mesh containing more than 100,000 vertices into 64 subgrids in about half a second.These results indicate that graph partitioning can now be truly embedded in dynamically-changing real-world applications, Horst D. Simon, Andrew Sohn, Rupak Biswas |
SPAA | 3 |
| 1997 | Special Issue on Dynamic Load Balancing: Guest Editors' Introduction
Andrew Sohn, Rupak Biswas |
J. Parallel Distributed Comput. | 2 |
| 1996 | Satisfiability Test with Synchronous Simulated Annealing on the Fujitsu AP1000 Massively-Parallel MultiprocessorabstractSolving the hard Satisfiability Problem is time consuming even for modest-sized problem instances. Solving the Random L-SAT Problem is especially difficult due to the ratio of clauses to variables. This report presents a parallel synchronous simulated annealing method for solving the Random L-SAT Problem on a large-scale distributed-memory multiprocessor. In particular, we use a parallel synchronous simulated annealing procedure, called Generalized Speculative Computation, which guarantees the same decision sequence as sequential simulated annealing. To demonstrate the performance of the parallel method, we have selected problem instances varying in size from 100-variables/425-clauses to 5000-variables/21,250-clauses. Experimental results on the AP1000 multiprocessor indicate that our approach can satisfy 99.9 percent of the clauses while giving almost a 70-fold speedup on 500 processors. Andrew Sohn, Rupak Biswas |
International Conference on Supercomputing | 2 |
| 1996 | Global Load Balancing with Parallel Mesh Adaption on Distributed-Memory SystemsabstractDynamic mesh adaption on unstructured grids is a powerful tool for efficiently computing unsteady problems to resolve solution features of interest. Unfortunately, this causes load imbalance among processors on a parallel machine. This paper describes the parallel implementation of a tetrahedral mesh adaption scheme and a new global load balancing method. A huristic remapping algorithm is presented that assigns partitions to processors such that the redistribution cost is minimized. Results indicate that the paralel performance of the mesh adaption code depends on the nature of the adaption region and show a 35.5X speedup on 64 processors when about 35% of the mesh is randomly adapted. For large-scale scientific computations, our load balancing strategy gives almost a sixfold reduction in solver execution times over non-balanced loads. Furthermore, our heuristic remapper yields processor assignments that are less than 3% off the optimal solutions but requries only 1% of the computational time. Rupak Biswas, Leonid Oliker, Andrew Sohn |
SC | 1 |
| 1996 | A Dynamic Load Balancing Framework for Unstructured Adaptive Computations on Distributed-Memory MultiprocessorsabstractThe computational requirements for an adaptive solution Andrew Sohn, Rupak Biswas, Horst D. Simon |
SPAA | 2 |