Topic: Benchmarking VMs

Those of you who know me know that I've had a keen interest in virtual machines and even Wine.  The recent discussions here about qemu, VMware and Win4BSD prompted me to do some benchmarking to see how they perform. 

That turns out not to be so easy, particularly for the graphics benchmarks in which I was interested.  That is, my "seat-of-the-pants" meter told me that Win4BSD (and by extension, qemu) had pretty lousy graphics performance.  But finding a benchmark that worked for both VMware and Win4BSD proved to be very difficult.

I finally found CrystalMark, which appears to be a reasonable benchmark although it seems to be most used in the overclocking community.  So let me present the results, and then I will go over them. 

Here's how they are read: CrystalMark tests various components to computer performance, assigns them a number, and those numbers are added to give a system rating.  There were some anomalies, so I left out the overall CrystalMark.  When a raw score is give (like frames per second, or FPS) the actual value is stated, and the CrystalMark is shown in parentheses.  I've listed native results for both a single and dual CPUs to show how the benchmark scales.

                  Win4BSD         VMware          XP (1 CPU)      XP (2 CPU)

ALU                 5342                5438               6690            13100
Fibonacci          1974                1995               2432             4639
Napierian           835                  845               1050             2085
Eratosthenes     1018                1047               1290             2540
QuickSort          1493                1529               1894             3780

FPU                   20898               7008               8659            16830
MikoFPU                   0                   829              1030              2050
RandMeanSS         1947                 3516              4350              8500
FFT                      1469                 1510              1860             3470
Mandelbrot          17460 (!)           1131              1400             2790

MEM                     1149               1385               1815            3050
Read, MB/s             140 (140)         208 (207)        290 (290)     560 
Write, MB/s            230 (230)         325 (324)        460 (457)     523 
R/W, MB/s              131 (130)         194 (193)        260 (257)     483 
Cache, MB/s          6276 (627)        6392 (639)      7900 (789)  14700

HDD                      7243              20015              7920            7840
Read, MB/s                40 (1608)       187 (5730)       58 (2304)     57 (2296)
Write, MB/s                32 (1260)        90 (3613)       51 (2051)     51 (2045)
RandomR512K, MB/s    40 (1600)      152 (5034)       35 (1414)     35 (1409)
RandomW512K, MB/s   25 (1018)        38 (1505)       28 (1120)     28 (1104)
RandomR64K, MB/s      27 (1078)        76 (3052)        9 (356)        9 (352)
RandomW64K, MB/s     17 (679)          27 (1081)      17 (679)       16 (632)

GDI                          272                1163              9341           9170
Text                            3                  148              2630           2620
Square                       78                  226              2980           2920
Circle                       129                  514              2570           2542
BitBlt                         62                  275              1160           1090

D2D                          678                1153              4190          4100
Sprite 10 , FPS             14 (1)              39 (3)           227 (22)      226 (22)
Sprite 100, FPS            12 (12)            38 (38)          216 (216)    215 (215)
Sprite 500 , FPS           12 (58)            28 (142)        150 (747)    147 (734)
Sprite 1000, FPS          10 (100)           21 (212)         92 (921)      90 (901)
Sprite 5000 , FPS           5 (232)            7 (361)         23 (1124)    22 (1100)
Sprite 10000, FPS          3 (275)            4 (397)         12 (1156)    11 (1130)

OGL                             0                  379            15040        15900
Scene 1 Score                                 (352)          (11110)       (12000)
   Lines (x1000)                             29949          2107400     2540000
   Scene 2 CPU                                     8                 256           512
Scene 2 Score                                   (27)             (3930)       (3850)
   Polyg (x1000)                                 200           221350       217000
Scene 2 CPU                                         1                128            128

It seems to be impossible to get these to line up!  I will post them to my web site tomorrow and follow up here to let you know.

The ALU scores are identical for Win4BSD and VMware.  The FPU scores have oddnesses; it seems that they can be considered roughly comparable.  Memory access favors VMware by 50%, though the cache scores are about the same.  VMware has an intelligent disk caching routine that seems to confuse the benchmarks -- it has performed comparably in other benchmarks I have tried.  That said, Win4BSD seems to perform pretty well.

The graphics scores paint another picture.  VMware performs better than Win4BSD by a factor of 3 to 5 depending on the test.  For some reason the OpenGL routines would not run on Win4BSD, but I did not examine that further since I don't use OGL on Windows.  The set-of-the-pants meter is vindicated!

It seems to me that Win4BSD, and by extension, qemu, would make good VMs for servers if the Ethernet interface were not so slow.  Major work needs to be done on the graphics routines -- right now it is just not competitive with even a very old version of VMware.  That is where development efforts should focus.

Particulars: Win4BSD ran Windows 2000; VMware ran XP Pro, as did the native machine.  That machine is my old dual Athlon (2800+) with 2.5GB memory.  The VMs had at least 750MB assigned to them; they were run out of Gnome with no other active tasks running.  The VMs were stored on a Maxtor 10K V SCSI drive; the system drive is a Seagate 15K.3 SCSI.  The video card is an nVidia Quadro FX2000.  The OS is FreeBSD 6.2-R; all the VMs used the same MP kernel and a single processor by setting  hint.apic.0.disabled="1" in device.hints.  No SCSI disk write-through changes were made to the native XP (also on a 15K.3).

Comments are of course welcomed.

Re: Benchmarking VMs

Nice benchmark, yes qemu graphics subsystem need a lot of work, since generally its fast if it goes to pure CPU speed.

Some time ago I have done qemu OpenGL tests, but not with very advanced 3D scsne but only with very simple 3D scene, take a look about these:
http://vermaden.proplayer.pl/gfx/screen … opengl.jpg
http://vermaden.proplayer.pl/gfx/photos/qemu-opengl.avi

"religions, worst damnation of mankind"
[color=Blue]Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for

Re: Benchmarking VMs

Hey DrJ you got a mention in the DragonFly BSD Digest

http://www.shiningsilence.com/dbsdlog/i … /2107.html

Re: Benchmarking VMs

Cool!

Re: Benchmarking VMs

Who's Justin ? One of us ?

<wintellect> NetBSD users are smart enough to accept that there's no 3D support tongue

Re: Benchmarking VMs

here is nice benchmark also:
http://www.linux-gamers.net/modules/sma … ?itemid=56

"religions, worst damnation of mankind"
[color=Blue]Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for