Quantum computing has potential to provide exponential speedups over classical computing for many important applications. However, today's quantum computers are in their early stages, and hardware quality issues hinder the scale of program execution. Benchmarking and simulation of quantum circuits on classical computers is therefore essential to advance the understanding of how quantum computers and programs operate, enabling both algorithm discovery that leads to high-impact quantum computation and engineering improvements that deliver to more powerful quantum systems. Unfortunately, the nature of quantum information causes simulation complexity to scale exponentially with problem size.

The emergence of quantum computers as a new computational paradigm has been accompanied by speculation concerning the scope and timeline of their anticipated revolutionary changes. While quantum computing is still in its infancy, the variety of different architectures used to implement quantum computations make it difficult to reliably measure and compare performance. This problem motivates our introduction of SupermarQ, a scalable, hardware-agnostic quantum benchmark suite which uses application-level metrics to measure performance. SupermarQ is the first attempt to systematically apply techniques from classical benchmarking methodology to the quantum domain. We define a set of feature vectors to quantify coverage, select applications from a variety of domains to ensure the suite is representative of real workloads, and collect benchmark results from the IBM, IonQ, and AQT@LBNL platforms. Looking forward, we envision that quantum benchmarking will encompass a large cross-community effort built on open source, constantly evolving benchmark suites. We introduce SupermarQ as an important step in this direction.