The PC architecture is going to completely change during the next months:
new chipsets, new CPUs, new motherboard formats, a new kind of memory,
a new BUS, etc...
It might become more and more difficult to find hardware fully
compatible with OS/2 or eComStation.
On the other side, even today machines have enough power to run multiple
operating systems via hardware emulation and those new machines will be
even more and more powerful.
So running eComStation and/or other operating systems within a virtualized
machine acting as a sort of hardware abstraction layer might soon become
an actractive solution for a lot of different problems.
Since I already have a fairly powerful machine and as more and more
virtual machine solutions are becoming available, I decided to make
a few tests to check how the system performances are affected when eComStation
runs in a virtual machine, rather than in a real machine.
At first I tried to use some of the most common benchmark applications for
OS/2, but, unfortunately I could not run all tests as some of them just
crashed when executed within a virtual machine.
Besides that, I soon realized
that the benchmark results were meaningless, not usable for a
comparison with a real machine or a different
kind of virtual machine, as the virtual machine timer is more or less out
of synch with the real timer.
For those reasons I wrote a
small suite of benchmarks
and used a chronometer to get correct, although aproximate, results when
the benchmark program was executed in a virtual machine.
The benchmark suite consists in a PM program to test the graphic
performances, measuring the time needed to perform the most common
graphic operations and visually checking how those operations are performed,
and in a command line program for testing the CPU and disk performances.
Besides running those benchmarks I also measured the time needed to install the operating
system, the boot time, the time needed to install and to start Star Office 5.1a,
and provided some comments about its usage within the virtual machines.
The real machine configuration is:
CPU
Intel P4 2.6 GHz (running at 2.87 GHz)
motherboard
Asus P4P800
system memory
2 x DDR 400 - 512 MB Kingston modules (total 1GB)
graphic adapter
ATI 9200 SE
screen resolution
1280 x 1024 pixels 16 million of colors
hard disk
2 x Maxtor DiamondMax Plus 9 60 GB ATA 133
operating system
eComStation 1.1
The virtual machines that have been tested are
Connectix Virtual PC 5.2 for Windows and Serenity Virtual Station 1.0 preview
running within a Windows 2000 (plus service pack 3) host.
The configuration of the virtual machines is:
The same install options were used in both the virtual machines:
feature
description
operating system
eComStation 1.1
install media
CD ISO image
install mode
advanced installation
hardware options
basic vga, IBM com.sys, no APM, fast time slicing
program options
no Dos-Win16 support, no problem determination tools,
no desktop productivity tools, no new view, no IBM mozilla based web browser,
no java 1.3
network options
no network support
The following tables report the time needed to install eComStation 1.1 in Virtual PC
and Virtual Station.
The time needed for each installation phase is reported.
The time has been
measured since the boot from hard disk / boot from CD selection
up to the reboot dialog.
In installation phase 4 the time has been measured up to the
completion of multimedia installation (i.e. when the message box appears).
phase 1 (seconds - less is better)
Virtual PC
184
Virtual Station
220
phase 2 (seconds - less is better)
Virtual PC
120
Virtual Station
227
phase 3 (seconds - less is better)
Virtual PC
58
Virtual Station
107
phase 4 (seconds - less is better)
Virtual PC
25
Virtual Station
58
total installation time (seconds - less is better)
Virtual PC
387
Virtual Station
612
Virtual PC was quite faster than Virtual Station, probably because
it makes a better use of the host disk cache.
That was also confirmed by the boot time, which I checked on the first reboot,
after having completed the installation and configuration of the graphic
and mouse drivers:
time needed to boot to the desktop (seconds - less is better)
This test measures the accuracy of the virtual machine timer.
The program checks the system uptime to calculate the elapsed time
(in milliseconds). The timer is started and stopped by pressing a keyboard
key and the elapsed time is compared with a chronometer (the value closer
to 10 are the better):
timer accuracy (seconds)
Real machine
10
Virtual PC
10
Virtual Station
4
While the inaccuracy of the Virtual PC timer has always been in the 2-300
milliseconds range (about as much as the human body reaction times),
Virtual Station timer is too much out of synch.
This test measures the speed of execution of the basic integer operations
(sum, subtraction, multiplication, division).
The unit is million of operations/second.
integer operations (Mop/sec)
Real machine
369
Virtual PC
339
Virtual Station
349
This test measures the speed of execution of the basic floating point
operations (sum, subtraction, multiplication, division).
The unit is million of operations/second.
floating point operations (Mop/sec)
Real machine
188
Virtual PC
175
Virtual Station
180
This test measures the speed of execution of trigonometric operations
(sin, cosin, tangent and logarithm and square root).
The unit is million of operations/second.
trigonometric operations (Mop/sec)
Real machine
16.3
Virtual PC
15.0
Virtual Station
15.8
These tests show that both Virtual PC and Virtual Station do a god job
with CPU emulation, at least regarding the basic mathematical operations,
with Virtual Station performing slightly better than Virtual PC.
All the disk tests do not write directly to the disk, but use the file
system cache since the purpose is to check application performance, not
hardware performance.
The real machine uses a quite large disk cache for JFS (128 MB) and just
a 2 MB cache for HPFS and 1 MB for the FAT partition. The FAT partition
used for the tests is the last partition of the disk which is usually slower.
The first test, multiple file write, measures the performances
by creating 100 directories,
each directory containing 2 subdirectories and each subdirectory containing
32 text files of size ranging from 1KB to 16 KB. The test is repeated 3
times. A total number of 300 directories, 600 subdirectories and 19200
text files (167 MB of data) are created.
multiple file write (MB/sec)
Real machine (FAT)
2.08
Real machine (HPFS)
6.73
Real machine (JFS)
4.06
Virtual PC
4.28
Virtual Station
2.79
The second test, large file write, measures the performances
by writing a large text file
(128 MB). The test is repeated 3 times. In total 403 MB of data are
written to the disk.
large file write (MB/sec)
Real machine (FAT)
12.5
Real machine (HPFS)
15.5
Real machine (JFS)
15.2
Virtual PC
18.3
Virtual Station
13.0
This test, multiple file read, measures the performance by reading
multiple files (the files created during the multiple file write
test). A total of 167 MB of data is read.
multiple file read (MB/sec)
Real machine (FAT)
4.7
Real machine (HPFS)
10.6
Real machine (JFS)
11.0
Virtual PC
9.3
Virtual Station
4.2
With the large file read test, a 128 MB file is read in 256 steps.
The read operation is repeated 12 times. A total of 1.6 GB of data is read
from the disk.
large file read (MB/sec)
Real machine (FAT)
26.4
Real machine (HPFS)
18.3
Real machine (JFS)
44.5
Virtual PC
57.5
Virtual Station
30.4
The multiple file copy tests consists in copying a tree of directories
and subdirectories multiple times.
25 directories containing 50 subdirectories and 1600 files are copied 3 times.
The total size of copied data is 41 MB.
multiple file copy (MB/sec)
Real machine (FAT)
0.41
Real machine (HPFS)
0.56
Real machine (JFS)
1.07
Virtual PC
0.87
Virtual Station
0.41
During the large file copy test a 128 MB file is copied 3 times.
large file copy (MB/sec)
Real machine (FAT)
6.7
Real machine (HPFS)
7.4
Real machine (JFS)
12.8
Virtual PC
22.4
Virtual Station
15.5
The sequential file zip test measure the performance by calling
for each file of a tree ZIP.EXE with the maximum compression rate.
The files are compressed individually, but are added to the same archive.
During this test 384 files are compressed for a total size of about 3.3 MB.
sequential file zip (MB/sec)
Real machine (FAT)
0.071
Real machine (HPFS)
0.157
Real machine (JFS)
0.106
Virtual PC
0.043
Virtual Station
0.025
The multiple file zip test compresses multiple times a tree of files
containing 16 directories, 32 subdirectories and 1024 files.
ZIP.EXE is executed with the maximum compression rate and with the
recursive option (i.e. it is executed once for each test repetition).
The total size of compressed data is about 134 MB.
multiple file zip (MB/sec)
Real machine (FAT)
1.89
Real machine (HPFS)
2.47
Real machine (JFS)
4.62
Virtual PC
0.92
Virtual Station
0.55
The large file zip test measures the performance by zipping a
128 MB file multiple times.
The test is repeated 3 times and the total size of the processed data is
about 403 MB.
large file zip (MB/sec)
Real machine (FAT)
4.20
Real machine (HPFS)
4.41
Real machine (JFS)
4.78
Virtual PC
4.11
Virtual Station
3.95
The file tree unzip test uncompress an archive containing a tree
of files multiple times overwriting the existing files.
The archive was previously compressed with the higher rate of compression.
The total size of processed data is about 107 MB.
file tree unzip (MB/sec)
Real machine (FAT)
0.68
Real machine (HPFS)
1.59
Real machine (JFS)
2.87
Virtual PC
1.22
Virtual Station
0.60
The last test, file tree deletion removes the data created during
the previous tests.
The total size of the data is about 413 MB
file tree deletion (MB/sec)
Real machine (FAT)
37.5
Real machine (HPFS)
25.6
Real machine (JFS)
44.9
Virtual PC
68.8
Virtual Station
29.5
All this tests provide consistent results with Virtual PC performing
better than Virtual Station and in some tests even better than the
real machine.
This in mainly due to the ability to make a better use of the host
disk cache (thanks to the -usehostdiskcache option).
The host disk cache, anyway, helps little when executing operations
which require both a lot of disk access and heavy CPU usage, like
in most of the ZIP/UNZIP tests.
The character statistics test just counts the occurrences of the
various characters contained in a 2 MB text file.
The test is repeated 6360 times. A total of about 453 MB of data is processed.
character statistics (MB/sec)
Real machine
15.1
Virtual PC
13.3
Virtual Station
14.2
During the word parsing test a 2 MB text file is scanned and the
words are extracted and stored in an array of text strings.
The total size of the processed data is about 208 MB.
word parsing (MB/sec)
Real machine
6.79
Virtual PC
6.11
Virtual Station
6.49
The case insensitive word sort test sorts the array of text words
created during the previous test.
The test is repeated 87 time and the total size of the processed data is
about 176 MB.
case insensitive word sort (MB/sec)
Real machine
5.91
Virtual PC
5.18
Virtual Station
5.50
The case sensitive word sort test sorts the array of text words
created during the word parsing test .
The test is repeated 129 time and the total size of the processed data is
about 261 MB.
case sensitive word sort (MB/sec)
Real machine
8.52
Virtual PC
7.68
Virtual Station
8.16
During the uppercase conversion test a 2 MB text file is converted
to uppercase for 2160 times. The total size of the processed data is
about 235 MB.
uppercase conversion (MB/sec)
Real machine
7.86
Virtual PC
7.12
Virtual Station
7.58
During the lowercase conversion test a 2 MB text file is converted
to lowercase for 2160 times. The total size of the processed data is
about 235 MB.
lowercase conversion (MB/sec)
Real machine
7.84
Virtual PC
7.34
Virtual Station
7.83
The case insensitive word search test performs a case insensitive
linear search of a text word in a 2 MB file, counting the occurrences.
The test is repeated for 1050 times. The total size of the processed data is
about 2.2 GB.
case insensitive word search (MB/sec)
Real machine
73.3
Virtual PC
66.7
Virtual Station
71.0
The case sensitive word search test performs a case sensitive
linear search of a text word in a 2 MB file, counting the occurrences.
The test is repeated for 450 times. The total size of the processed data is
about 944 MB.
case sensitive word search (MB/sec)
Real machine
31.5
Virtual PC
28.6
Virtual Station
30.4
All these tests confirm Virtual Station performances with CPU emulation
as previously showed by the mathematical operations tests.
Virtual PC is slightly
behind Virtual Station, but both virtual machines work almost as fast as the
real one.
The smoothness test provides a visual feedback of the ability
of the virtual machine to render animations.
The animation is created by drawing a bitmap in a loop, offsetting it of
2 pixels for each iteration. After each draw operation, the process is
suspended for 1 millisecond, so the speed result is also affected by the
accuracy of the emulated timer of the virtual machine.
Virtual PC is almost as fast as the real machine, but every 2-3 seconds
the animation stops for a fraction of seconds. The speed gain is achieved
by skipping a few screen frames. Virtual Station performance is quite
unsatisfactory.
animation smoothness test (operations / sec)
Real machine
512
Virtual PC
517
Virtual Station
181
The speed test measures the speed of the draw bitmap operation.
The speed of the virtual machine graphic adapter is better than that of
the real machine, but the result is only achieved by skipping screen
frames.
The real machine, skipping much less frames (depending on the frequency
of the vertical refresh of your monitor) clearly shows the track followed
by the ball, while the virtual machines seem to show multiple balls at
random places. Virtual Station, although providing the best numerical
results, gives the worst visual result.
draw bitmap test (operations / sec)
Real machine
6561
Virtual PC
20556
Virtual Station
92500
The strech bitmap speed test, makes quite use of the CPU power and
skipping frames is of little help to the virtual machines in this case.
Virtual Station is faster than Virtual PC, but Virtual PC animation is
generally smoother.
The horizontal/vertical lines test draws squares of different colors
and decreasing size on the screen.
Virtual Station in this test is faster than the real machine, while
Virtual PC is much slower as it probably doesn't skip many frames when
rendering horizontal/vertical lines.
horizontal/vertical lines (lines / sec)
Real machine
25004
Virtual PC
8780
Virtual Station
37895
The rectangles test draws a series of rectangles of decreasing sizes
and different colors.
Both Virtual PC and Virtual Station are quite faster than the real machine
as probably both of them recur to the trick of screen frame skipping.
rectangles (rectangles / sec)
Real machine
2964
Virtual PC
5040
Virtual Station
5355
The text characters test draws a series of text lines of different
colors.
Screen frame skipping did not help much with this test and both Virtual PC
and Virtual Station are quite slower than the real machine.
The load dialog test, just measures how long it takes to load a
dialog populated with a rich set of controls: 4 group boxes, 1 multiline
edit control, 7 radio buttons, 10 push buttons and 1 list box.
Of course some system options affect the dialog loading performance.
The following table shows the dialog loading speed when titlebar and
pushbuttons enhancements are active, the next table shows the performance
of a system without any enhancements.
Even in the worst case, the difference in loading a dialog with or without
the aesthetic enhancements is just 190 versus 133 milliseconds, hardly
noticeable in the real usage.
The move dialog test is not much different from the bitmap drawing
test as the dialog bitmap is just copied to a different position on the
screen.
Both Virtual PC and Virtual Station score better than the real PC, but
what is shown on the screen is quite different as 3/4 of the screen frames
are just skipped.
move dialog (motion / sec)
Real machine
1019
Virtual PC
1304
Virtual Station
1304
The size dialog test is executed by increasing the dialog width and
height by 1 pixel until they reach the screen size, then decreasing
and increasing again.
A lot of calculations are involved as all the dialog controls are moved
and resized according to the new dialog size.
The screen frame skipping cannot be applied in this case and so
both the virtual machines are quite slower than the real one.
A consistent performance increase comes from disabling the titlebar
and push buttons enhancements since all the calculations performed to draw
the shades for the titlebar and the pushbutton are no longer needed.
All these tests show that the graphic performances are the real bottleneck
of the virtual machines. The windowing tests anyway show that although the
virtual machines perform quite slower than the real one, still most
applications are likely to run at an acceptable speed when executed
in a virtual machine.
Generally virtual machines are not yet suitable for running games (although
I found that Windows Pinball runs quite smoothly in Virtual PC).
Although Virtual Station provide better scores in some tests, Virtual PC
provides better visual results, probably thanks to a more accurate
virtualization of the PC timer.
As a last test I wanted to check the behaviour and usability of a real
application. I choosed Star Office 5.1a as it is quite resource
demanding.
The first thing I checked was the installation time.
Again, as in the operating system installation test, Virtual PC was quite
faster than Virtual Station (shorter bars indicate better performance):
Star Office 5.1a installation (seconds)
Virtual PC
90
Virtual Station
233
As a second test, since Star Office has always been
one of the slowest applications regarding load time (at least before
Mozilla came out) I tested how long it takes to get the application
window since the icon is double clicked.
The first table shows the time needed to start the program for the first
time, the second table shows the impact of the disk cache when reloading
the program just after it was closed:
Star Office 5.1a load time (seconds)
Virtual PC
9
Virtual Station
25
Star Office 5.1a re-load time (seconds)
Virtual PC
6
Virtual Station
23
As a last test I checked how the applicaton feels in real usage.
Since graphic performance is the virtual machines weakest point
I created a 512x512 pixel image and started to play with the aerograph tool.
Virtual PC behaved quite good in this test providing the same responsivenes
of the real machine to the point to make it hard to tell the real machine
from the Virtual PC one. Virtual Station, instead, was more unresponsive
featuring a small yet noticeable and boring delay between the mouse movements
and the track being painted on the screen.
What surprised me more was the lack of responsiveness of the application
menu. On the first access, with Virtual Station, any submenu of Star Office
takes a few seconds before appearing on the screen, giving the feeling
of a broken application.
While the benchmarks showed mixed results, with Virtual Station apparently
scoring better in many tests, Virtual PC provided a better feeling
in real usage, probably thanks to various factors like the ability to make
a better use of the host disk cache and of synchronizing its virtual clock
with the real clock.
Virtual Station, in order to work as a sort of hardware abstraction layer,
allowing to use eComStation (and/or other operating systems) as
guests, must take care of those limitations.
In any case, considering that Virtual Station is not yet
a mature product, while Virtual PC has been developed since much longer,
Virtual Station is at least quite promising and I'm confident that in the
near future its developers will be able to make it outperform Virtual PC.
Regarding the virtual machine management program, i.e. the application
which runs on the host PC, currently Virtual PC
is still better than Virtual Station both as user interface and as features
availability.
On the other hand, Virtual PC has been taken by the deadly embrace of
the Microsoft snake, so future support for both guest and host operating
systems is limited to the various versions of Windows.
Virtual Station, on the contrary, has always had as a goal the support
of a much wider range of operating systems. It is marketed and developed
by small yet dynamic companies which are much more likely to listen to the
users requests and various features, not yet present even
in Virtual PC 2004, (e.g. USB ports), will be available within few months.
So Virtual PC is currently a better solution. If you have already purchased
it, go on using it until it satisfies your needs,
but if you plan to buy a virtual machine application now or
in the near future I suggest you to wait for the next version of Virtual
Station.
To conclude here is a short list of features I would like to see in a
future version of Virtual Station (I did not mention audio and USB support
are those are already under development):
mouse driver with scroll wheel support,
support for more than 256 MB of memory,
support for more hard drives,
ability to save the state of the guest operating system rather than
having to shut it down.