We finally get to the practical part of this article. As usual, I propose a different benchmark as a practical example, only this time we will be seeing a more complex benchmark, in fact a CPU benchmark suite: we'll use the latest version of nbench-byte (version 2.1) as our example. You can download it from Uwe Mayer's new Web site or from the Linux Benchmarking Project
What we are going to measure this time can be described as "general CPU performance". So: this is not processor performance for matrix operations, this is not MMX performance, this is not the ability of a processor to decode an MPEG stream. Also, this is not a measure of a processor interrupt response time, peak MIPS, etc.
Now let's take a look at a correct procedure, following all the steps recommended in section 1.1
For this short example we just want to compare the performance of two different CPUs: the AMD K6 and the Cyrix 6x86MX. This is comparative benchmarking, so we should keep all conditions fixed and vary just this single variable: the CPU.
This is not too ambitious and I have no bias for/against any of these two chips. Also, since both CPUs are widely available at reasonable prices, such comparative benchmarking may be of interest to GNU/Linux users wanting to upgrade and/or put together their next CPU.
Nbench-byte is an improved, updated version of the BYTEmark benchmark suite developed at BYTE magazine by Rick Grehan. Uwe F. Mayer did the port to Linux and is its present maintainer/developer. The latest version is 2.1, dated December 97.
Similarly to SPEC95, this modern CPU benchmark suite uses 10 different algorithms that are representative of common CPU-intensive tasks (the file bdoc.txt included with the source has a description of each algorithm). Note that Rick has stopped development of BYTEmark (neither Uwe nor myself managed to contact him), but you can see that this is not a committee-designed benchmark; in this respect its lineage fits quite well the GNU/Linux style of development.
Nbench-byte 2.1 also goes one step beyond SPEC95 in that it generates three index figures: an Integer Index, a Floating-Point Index and a Memory Index. The Memory Index reflects the fact that on most modern CPUs, the memory subsystem represents a major performance bottleneck. You can check the Web site for STREAM a new benchmark specifically created to address this issue, for more information on this topic.
One of nbench-byte nicest features is that it calibrates itself. For each of the tests it determines a minimum amount of work that needs to be done to be able to accurately measure the time needed. Then it runs that test five times and does a statistical analysis (using the student-t distribution) to see if the results are consistent (meaning that the probability is at least 95% that the true mean of the results is within 5% of the calculated mean of the results). If not, then nbench runs the test up to twenty-five times more and does the statistical analysis after each additional test run. If consistency cannot be achieved within a total of thirty runs, a warning will be issued when the score gets reported.
In terms of raw data statistical processing, nbench-byte 2.1 goes beyond all the other benchmarks I have ever come across.
Another very interesting feature of this benchmark suite is its portability across a wide range of OS's and platforms. However, because of fundamental differences in compiler/libraries/memory management in different OSes, this benchmark should not be carelessly used to compare results across platforms. This is not an OS benchmark, it's a CPU benchmark (see the pitfalls subsection below). You have been warned.
We are doing comparative benchmarking, so we will be using exactly the same hardware for our benchmark runs. All that will change between runs is:
Also note that we are using the precompiled nbench executable, as shipped in the tar.gz package.
To describe our hardware setup, we resort to the Linux Benchmarking Toolkit Report Form:
LINUX BENCHMARKING TOOLKIT REPORT FORM
CPU
===
Vendor: AMD/Cyrix
Model: K6-166/6x86MX-PR200
Core clock:166 MHz (2.5 x 66MHz)
Motherboard vendor: ASUS
Mbd. model: P55T2P4
Mbd. chipset: Intel HX
Bus type: PCI
Bus clock: 33 MHz
Cache total: 512 Kb
Cache type/speed: Pipeline burst 6 ns
SMP (number of processors): 1
RAM
===
Total: 32 MB
Type: EDO SIMMs
Speed: 60 ns
Disk
====
Vendor: IBM
Model: IBM-DCAA-34430
Size: 4.3 GB
Interface: EIDE
Driver/Settings: Bus Master DMA mode 2
Video board
===========
Vendor: Generic S3
Model: Trio64-V2
Bus: PCI
Video RAM type: 60 ns EDO DRAM
Video RAM total: 2 MB
X server vendor: XFree86
X server version: 3.3
X server chipset choice: S3 accelerated
Resolution/vert. refresh rate: 1152x864 @ 70 Hz
Color depth: 16 bits
Kernel
======
Version: 2.0.29
Swap size: 64 MB
gcc
===
Version: 2.7.2.3
Options: (default nbench)
libc version: 5.4.38
Test notes
==========
Two processors tested. The 6x86MX was configured with a special rc.cyrix file.
RESULTS
========
Linux kernel 2.0.0 Compilation Time: N/A
Whetstone Double Precision (FPU) INDEX: N/A
UnixBench 4.10 system INDEX: N/A
Xengine: N/A
nbench-byte integer INDEX: 6x86MX - 0.686; K6 - 0.713
nbench-byte memory INDEX: 6x86MX - 0.753; K6 - 0.793
nbench-byte floating-point INDEX: 6x86MX - 0.655; K6 - 0.802
Comments
=========
With the CPU case open, it took me 30 minutes to run nbench-byte on the two processors!
One can get very detailed benchmark results with nbench-byte 2.1 by specifying the -v option. However, here we are only showing the normal output from a standard run, first on the 6x86MX, then on the K6:
6x86MX results:
BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)
TEST : Iterations/sec. : Old Index : New Index
: : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT : 80.681 : 2.07 : 0.68
STRING SORT : 11.107 : 4.96 : 0.77
BITFIELD : 2.1997e+07 : 3.77 : 0.79
FP EMULATION : 8.5349 : 4.10 : 0.95
FOURIER : 881.21 : 1.00 : 0.56
ASSIGNMENT : 0.71582 : 2.72 : 0.71
IDEA : 147.28 : 2.25 : 0.67
HUFFMAN : 58.095 : 1.61 : 0.51
NEURAL NET : 0.70897 : 1.14 : 0.48
LU DECOMPOSITION : 27.869 : 1.44 : 1.04
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX : 2.861
FLOATING-POINT INDEX: 1.181
Baseline (MSDOS*) : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
C compiler : gcc version 2.7.2.3
libc : libc.so.5.4.38
MEMORY INDEX : 0.753
INTEGER INDEX : 0.686
FLOATING-POINT INDEX: 0.655
Baseline (LINUX) : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.
K6 results:
BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)
TEST : Iterations/sec. : Old Index : New Index
: : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT : 82.229 : 2.11 : 0.69
STRING SORT : 10.57 : 4.72 : 0.73
BITFIELD : 2.0672e+07 : 3.55 : 0.74
FP EMULATION : 6.4842 : 3.11 : 0.72
FOURIER : 1117.1 : 1.27 : 0.71
ASSIGNMENT : 0.93388 : 3.55 : 0.92
IDEA : 158.42 : 2.42 : 0.72
HUFFMAN : 81.407 : 2.26 : 0.72
NEURAL NET : 1.0764 : 1.73 : 0.73
LU DECOMPOSITION : 26.521 : 1.37 : 0.99
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX : 2.990
FLOATING-POINT INDEX: 1.445
Baseline (MSDOS*) : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
C compiler : gcc version 2.7.2.3
libc : libc.so.5.4.38
MEMORY INDEX : 0.793
INTEGER INDEX : 0.713
FLOATING-POINT INDEX: 0.802
Baseline (LINUX) : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.
We will concentrate on the Linux data, for obvious reasons. As we can see, whereas the 6x86MX outperforms the K6 on some tests by a narrow margin (approx. 6%), the K6 vastly outperforms the 6x86MX on other tests.
On our synthetic test nbench-byte version 2.1, the K6 has shown slightly better performance than the 6x86MX, running at the same 166MHz (2.5 x 66MHz) clock rate on exactly the same hardware.
The basic pitfall that one should be warned against concerning nbench-byte applies similarly to all benchmarks: one should not to try to use this tool for something it was not designed for. Since this is a CPU benchmark, do not use it to test OS performance, video bandwidth, or any other feature that implies I/O activity. Also, it is not an adequate tool for comparing compilers and/or C and math libraries.
This is less obvious than it seems at first. For an accurate, thorough, documented discussion of this particular pitfall, you are referred to one of Uwe's excellent pages on benchmarking
Another pitfall would have been to compare the two processors running on widely different machines. Motherboard, cache and RAM timing setup can skew results by as much as 10%. Compilation options and libraries can also skew results by 25% or more.