benchmarking Poly/ML

[Stephen Weeks]

I just did some benchmarking of the latest release of Poly/ML
(http://www.polyml.org/), which is their first release to be SML 97 compliant
and to have the standard basis library.  Here are the timings for SML/NJ,
Poly/ML, and MLton on the benchmarks that Poly/ML could handle (it barfed on all
the other usual ones that aren't here).  These were run on a 733 MhZ Pentium
III.  Poly/ML's compile times are very fast (< 3 seconds for all benchmarks
here).  OTOH, for its running times, Poly/ML is insanely bad on the floating
point benchmarks (fft, mandelbrot, matrix-multiply, nucleic).  Poly/ML is also
pretty bad on the symbolic and integer benchmarks.  It does manage to beat MLton
on the procedure call benchmarks (fib, tak) and on merge, which is basically an
allocation/gc benchmark.  Hopefully we'll have fib and tak up to snuff with the
native backend -- IIRC the differences between MLton and Poly/ML are about the
same differences as between MLton C and MLton native.  As to merge, I'd be
interested to hear how MLton C compares to MLton native.  I don't think merge
made it into Matthew's latest round of benchmarks.

absolute running time in seconds
			SML/NJ	Poly/ML MLton
count-graphs		 15.9	 31.7	 6.4
fft			 31.8	249.6	19.8
fib			  7.0	  4.6	 5.9
knuth-bendix		 28.1	 37.9	 9.1
lexgen			 31.9	 25.8	16.7
mandelbrot		 32.2	458.9	 9.7
matrix-multiply		 33.8	 68.0	 4.8
merge			351.5	 42.3	49.6
mlyacc			 29.6	108.5	11.8
nucleic			 39.9	214.0	14.3
ratio-regions		 77.9	 65.3	11.7
simple			 18.9	 65.8	 6.0
smith-normal-form	142.1	 20.1	 1.2
tak			 21.8	 10.1	13.7
vliw			 19.2	 14.2	 9.2

ratio of running time to MLton
			SML/NJ	Poly/ML
count-graphs		  2.5	 5.0
fft			  1.6	12.6
fib			  1.2	 0.8
knuth-bendix		  3.1	 4.2
lexgen			  1.9	 1.5
mandelbrot		  3.3	47.3
matrix-multiply		  7.0	14.2
merge			  7.1	 0.9
mlyacc			  2.5	 9.2
nucleic			  2.8	15.0
ratio-regions		  6.7	 5.6
simple			  3.1	11.0
smith-normal-form	118.4	16.8
tak			  1.6	 0.7
vliw			  2.1	 1.5