Benchmarks

Benchmarks

• When you buy or evaluate a computer, you want to know how well it will perform for your work. If you ran exactly the same programs every day, you could simply test one computer, then test another and compare how long your programs took. But in the real world, most users don't run exactly the same tasks all the time.

• So instead of relying on your specific workload, we use benchmarks ‒ these are carefully selected programs that attempt to reflect typical work. We run these on computers and use the results to compare performance, hoping the results relate to how the computer will perform on your real tasks.

• SPEC (System Performance Evaluation Cooperative) is an organization that gathers computer vendors and users to create standard benchmark suites. The SPEC benchmarks are standardized programs used to measure and compare the performance and power efficiency of different computer systems. They act as a common workload that lets users predict how well a computer will handle typical, day‒to‒day tasks, even if the user doesn't have the exact programs to test.

• SPEC is critical because it helps solve the problem of comparing different processors and systems in a fair and standardized way.

• The original SPEC CPU benchmark (called SPEC89) measured processor performance. It consists of two main suites:

1. CINT2006 (Integer Benchmarks): A set of 12 programs covering tasks that rely heavily on integer arithmetic and memory access, like compiling code, running a chess program, or basic simulations.

2. CFP2006 (Floating‒Point Benchmarks): A set of 17 programs covering tasks that require complex, high‒precision calculations, such as fluid dynamics, molecular modelling and finite element analysis.

• These benchmarks attempt to stress the CPU, memory subsystem, compilers, etc., but they generally do not focus on many other parts of a computer system (such as network I/O or graphics).

1. SPECratio

• To provide a simple, single number for comparison, SPEC uses the SPECratio :

■ Normalization: The execution time of a measured computer is compared against the execution time of a reference processor (a standard baseline machine).

■ The Ratio: SPECratio = Execution Time on Reference Computer / Execution Time on Measured Computer.

■ Interpretation: A larger SPECratio means a faster computer. For example, a SPECratio of 50 means the measured computer is 50 times faster than the reference machine.

■ 3200 Single summary: The overall score for CINT2006 or CFP2006 is found by taking the geometric mean of the SPECratios for all programs in the suite. This mathematical method ensures the final relative comparison between any two computers remains consistent, regardless of which computer was chosen as the initial reference.

2. SPEC Power Benchmark: Measuring Efficiency

• With energy consumption becoming a major concern, the SPEC Power benchmark measures the energy efficiency of server systems under varying loads.

• It runs a Java‒based business application workload (originally SPECJBB2005) that stresses the processor, memory and caches. The system's performance (throughput, measured in "ssj_ops") and its electrical power consumption (in watts) are measured at 10 different workload levels, from 100% load down to 0% (idle).

• The data is condensed into a single metric that represents overall efficiency :

• Interpretation: This metric is the ratio of the total performance achieved across all ten load levels to the total power consumed at those same levels.

• Result: A higher "overall ssj_ops per watt" value indicates a more energy‒efficient server. It means the server delivers more processing work for every watt of power it consumes, which is crucial for data centers aiming to reduce electricity costs.

Review Questions

1. What is the purpose of the SPEC benchmarks?

2. Differentiate between CINT2006 and CFP2006 benchmark suites of SPEC

3. Define SPECratio and explain how it is used to compare computer systems.

4. Describe how the SPEC power benchmark measures the energy efficiency of a system.