The SPEC SERT suite is the industry-standard benchmark for evaluating server energy efficiency and is widely adopted in government regulations and certification programs. Current certification rules require every CPU memory channel to be populated with at least one Dual In-line Memory Module (DIMM). In real-world deployments, however, servers are sometimes configured with fewer DIMMs, leaving some channels unpopulated. This discrepancy introduces a significant gap between certified efficiency scores and the actual efficiency of deployed systems. To address this issue, there are ongoing discussions about certifying servers with partially populated memory channels. However, the impact of memory configuration on SPEC SERT results has not been systematically studied. In this paper, we present a comprehensive analysis of how different memory configurations affect performance, power consumption, and energy efficiency on two state-of-the-art server systems using the SPEC SERT 2 suite. We vary both the number of populated channels and the type of DIMMs. Our experimental results show that server efficiency scores can be up to 3.4 times lower with only one channel populated and still up to 1.3 times lower with half the channels, compared to fully populated configurations. Detailed analysis of individual SPEC SERT 2 worklets reveals that some CPU worklets are highly sensitive to memory bandwidth, and that the impact of memory configuration is dependent on both server architecture and workload intensity. These findings underscore the need to reconsider certification criteria and highlight the importance of memory configuration for accurate energy efficiency assessment.

With the rising demand for information technology comes an increase in energy consumption to power it. Namely, cloud computing is constantly growing, in part due to a rising number of devices using the cloud to provide certain functionality. This growth leads to an increase in energy consumption in data centers and is estimated to climb to over 1PWh in 2030. Hardware manufacturers counter the rising demand for energy in cloud data centers by providing techniques to make servers more energy-efficient. However, the advances cannot fully compensate for the growth. To further increase energy efficiency, software needs to be addressed as well but is often neglected by the developers.In this paper, we compare six sorting algorithms, a common task in most programs, against each other in terms of energy efficiency to allow developers to select the best solution to their problem. We selected well-known algorithms in two variants, and two implementation languages, C and Python. We ran each algorithm on two, out of four, state-of-the-art server systems with different CPUs.

Innovation needs a competitive and fair playing field on which products can be compared and informed choices can be made. Standard benchmarks are a necessity to create such a level playing field among competitors in the server market for more energy-efficient servers. That, in turn, motivates their engineers to design more energy-efficient hardware. The SPECpower_ssj 2008 benchmark drove the increase of server energy efficiency by 113 times for single CPU servers, or 19 times on average. Yet, with added functionality and load, they are expected to consume a rising amount of energy. Additionally, server usage in data centers has changed over time with new application types. To continue the effort of increasing server energy efficiency, a new version, SPECpowerNext, is under development. In this work, after a short introduction to SPECpower_ssj 2008, we present the new implementation of SPECpowerNext together with the standardized way to collect server information in heterogeneous data centers. We also give insight, including preliminary measurements, into two of SPECpowerNext new workloads, the Wiki and the APA workload, in addition to an overview of both workloads.

Attacks on software systems are becoming more and more frequent, aggressive and sophisticated. With the changing threat landscape, in 2018, organizations are looking at when they will be attacked, not if. Intrusion Detection Systems (IDSs) can help in defending against these attacks. The systems that host IDSs require extensive computing resources as IDSs tend to detect attacks under overloaded conditions wrongfully. With the end of Moore's law and the growing adoption of Internet of Things, designers of security systems can no longer expect processing power to keep up the pace with them. This limitation requires ways to increase the performance of these systems without adding additional compute power. In this work, we present two dynamic and a static approach to bypass IDS for traffic deemed benign. We provide its prototype implementation and evaluate our solution. Our evaluation shows promising results. Performance is increased up to the level of a system without an IDS. Attack detection is within the margin of error from the 100% rate. However, our findings show that dynamic approaches perform best when using software switches. The use of a hardware switch reduces the detection rate and performance significantly.

General Purpose Graphics Processing Units (GPGPUs) are becoming more and more common in current servers and data centers, which in turn consume a significant amount of electrical power. Measuring and benchmarking this power consumption is important as it helps with optimization and selection of these servers. However, benchmarking and comparing the energy efficiency of GPGPU workloads is challenging as standardized workloads are rare and standardized power and efficiency measurement methods and metrics do not exist. In addition, not all GPGPU systems run at maximum load all the time. Systems that are utilized in transactional, request driven workloads, for example, can run at lower utilization levels. Existing benchmarks for GPGPU systems primarily consider performance and are intended only to run at maximum load. They do not measure performance or energy efficiency at other loads. In turn, server energy-efficiency benchmarks that consider multiple load levels do not address GPGPUs.

Benchmarking of energy efficiency is important as it helps researchers, customers, and developers to evaluate and compare the energy efficiency of software and hardware solutions. Developing and deploying energy-efficiency benchmarking workloads are challenging tasks, as work must be able to be executed in a power measurement environment using an energy-efficiency measurement methodology.The existing SPEC Chauffeur Worklet Development Kit (WDK) enables the development and use of custom workloads (called worklets) within a standardized power measurement methodology. However, it features no integration in development environments, making building and deployment of workloads challenging. We address this challenge by proposing Autopilot, a plugin for the Eclipse IDE. Autopilot enables fast and easy building and deployment of a workload under development on a system for testing. It also enables benchmark execution directly from the development environment.

Experimental analysis of computer systems' power consumption has become an integral part of system performance evaluation, efficiency management, and model-based analysis. As with all measurements, repeatability and reproducibility of power measurements are a major challenge. Nominally identical systems can have different power consumption running the same workload under otherwise identical conditions. This behavior can also be observed for individual system components. Specifically, CPU power consumption can vary amongst different samples of nominally identical CPUs. This in turn has a significant impact on the overall system power, considering that a system's processor is the largest and most dynamic power consumer of the overall system. The concrete impact of CPU sample power variations is unknown, as comprehensive studies about differences in power consumption for nominally identical systems are currently missing. We address this lack of studies by conducting measurements on four different processor types from two different architectures. For each of these types, we compare up to 30 physical processor samples with a total sum of 90 samples over all processor types. We analyze the variations in power consumption for the different samples using six different workloads over five load levels. Additionally, we analyze how these variations change for different processor core counts and architectures. The results of this paper show that selection of a processor sample can have a statistically significant impact on power consumption. With no correlation to performance, power consumption for nominally identical processors can differ as much as 29.6% in idle and 19.5% at full load. We also show that these variations change over different architectures and processor types.

Energy efficiency of servers has become a significant issue over the last years. Load distribution plays a crucial role in the improvement of energy efficiency as (un-)balancing strategies can be leveraged to distribute load over one or multiple systems in a way in which resources are utilized at high performance, yet low overall power consumption. This can be achieved on multiple levels, from load distribution on single CPU cores to machine level load balancing on distributed systems. With modern day server architectures providing load balancing opportunities at several layers, answering the question of optimal load distribution has become non-trivial. Work has to be distributed hierarchically in a fashion that enables maximum energy efficiency at each level. Current approaches balance load based on generalized assumptions about the energy efficiency of servers. These assumptions are based either on very machine-specific or highly generalized observations that may or may not hold true over a variety of systems and configurations. In this paper, we use a modified version of the SPEC SERT suite to measure the energy efficiency of a variety of hierarchical load distribution strategies on single and multi-node systems. We introduce a new strategy and evaluate energy efficiency for homogeneous and heterogeneous workloads over different hardware configurations. Our results show that the selection of a load distribution strategy depends heavily on workload, system utilization, as well as hardware. Used in conjunction with existing strategies, our new load distribution strategy can reduce a single system's power consumption by up to 10.7%.

Standardized benchmarks have become widely accepted tools for the comparison of products and evaluation of methodologies. These benchmarks are created by consortia like SPEC and TPC under confidentiality agreements which provide little opportunity for outside observers to get a look at the processes and concerns that are prevalent in benchmark development. This paper introduces the primary concerns of benchmark development from the perspectives of SPEC and TPC committees. We provide a benchmark definition, outline the types of benchmarks, and explain the characteristics of a good benchmark. We focus on the characteristics important for a standardized benchmark, as created by the SPEC and TPC consortia. To this end, we specify the primary criteria to be employed for benchmark design and workload selection. We use multiple standardized benchmarks as examples to demonstrate how these criteria are ensured.

Energy efficiency of servers has become a significant research topic over the last years, as server energy consumption varies depending on multiple factors, such as server utilization and workload type. Server energy analysis and estimation must take all relevant factors into account to ensure reliable estimates and conclusions. Thorough system analysis requires benchmarks capable of testing different system resources at different load levels using multiple workload types. Server energy estimation approaches, on the other hand, require knowledge about the interactions of these factors for the creation of accurate power models. Common approaches to energy-aware workload classification categorize workloads depending on the resource types used by the different workloads. However, they rarely take into account differences in workloads targeting the same resources. Industrial energy-efficiency benchmarks typically do not evaluate the system's energy consumption at different resource load levels, and they only provide data for system analysis at maximum system load.

The Server Efficiency Rating Tool (SERT) has been developed by the Standard Performance Evaluation Corporation (SPEC) at the request of the US Environmental Protection Agency (EPA). Almost 3% of all electricity consumed within the US in 2010 went to running datacenters. With this in mind, the EPA released Version 2.0 of the ENERGY STAR for Computer Servers program in early 2013 to include the mandatory use of the SERT. Other governments world-wide that are also concerned by growing power consumption of servers and datacenters are considering the adoption of the SERT.

The driving philosophy for the Standard Performance Evaluation Corporation (SPEC) is to ensure that the marketplace has a fair and useful set of metrics to differentiate systems is by providing standardized benchmark suites. This paper gives an overview of SPEC and its continuous drive for better benchmarks.

An overview of the SPEC PTDaemon and SPEC's Benchmark Methodology

Overview and future outlook of the #1 industry standard virtualization benchmark, SPECvirt_sc2010.

The Server Efficiency Rating Tool (SERT) has been developed by Standard Performance Evaluation Corporation (SPEC) at the request of the US Environmental Protection Agency (EPA), prompted by concerns that US datacenters consumed almost 3% of all energy in 2010. Since the majority was consumed by servers and their associated heat dissipation systems the EPA launched the ENERGY STAR Computer Server program, focusing on providing projected power consumption information to aid potential server users and purchasers. This program has now been extended to a world-wide audience.

SPECpower_ssj2008 [1] is the first industry-standard SPEC [2] benchmark that evaluates the power and performance characteristics of volume server-class and multi-node class computers. This poster-paper gives an overview of the benchmark that defines the server power measurement standards [8] in the same way SPEC have done for performance.

The Server Efficiency Rating Tool (SERT) [1] has been developed by Standard Performance Evaluation Corporation (SPEC) [2] at the request of the US Environmental Protection Agency (EPA) [3], prompted by concerns that US datacenters consumed almost 3% of all energy in 2010. This poster-paper gives an overview of the SERT.

According to the United States Environmental Protection Agency (US EPA) almost 3% of all electricity consumed within the US in 2010 goes to running datacenters, with the majority of that powering servers and the associated air conditioning systems dedicated to eliminating the heat they produce. The EPA launched the ENERGY STAR® Computer Server program in May 2009, intended to deliver information to better enable server purchasing decisions based on projected power consumption.The Server Efficiency Rating Tool (SERT) has been developed by the Standard Performance Evaluation Corporation (SPEC) SPECpower committee to address the EPA requirements for Version 2 of the ENERGY STAR server program. Unlike many tools sourced from the SPEC organization the SERT is not intended to be a benchmark, and for Version 2 does not offer a single score model. Instead it produces detailed information regarding the influence of CPU, memory, network and storage I/O configurations on overall server power consumption.This paper describes the design and development of the SERT, including discussion of the collaborative nature of working with the EPA and the various industry stakeholders involved in the design, review and development process. Many of the core ideas behind SERT were derived from theSPECpower_ssj2008 and other SPEC-developed benchmarks, and this paper illustrates where ideas and code were shared, as well as where new thinking resulted in entirely new solutions. It also includes thoughts for the future, as the ENERGY STAR server program continues to evolve and the SERT will evolve with it.