performance page
Stephen Weeks
MLton@sourcelight.com
Tue, 10 Jul 2001 15:08:50 -0700
--BKXRV8ZmZi
Content-Type: text/plain; charset=us-ascii
Content-Description: message body text
Content-Transfer-Encoding: 7bit
Henry, could you please replace the performance page with the following. It has
new numbers for the mlkit based on running with the unlimited stack size. It
also has the zern numbers. Thanks.
--BKXRV8ZmZi
Content-Type: text/html
Content-Transfer-Encoding: 7bit
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>MLton's Performance</title>
</head>
<body>
This page compares the performance of the following SML compilers on a range
of benchmarks.
<ul>
<li> <a href = "http://www.sourcelight.com/MLton/">MLton</a> 20010708
<li> <a href = "http://www.itu.dk/research/mlkit/">ML Kit</a> 3.9.1
<li> <a href = "http://www.dina.kvl.dk/~sestoft/mosml.html">Moscow ML</a>
2.00
<li> <a href =
"http://cm.bell-labs.com/cm/cs/what/smlnj/index.html">SML/NJ</a>
110.33
</ul>
All of the benchmarks are available <a href = "benchmarks/">here</a>.
Some of the benchmarks were obtained from
<a href="ftp://ftp.research.bell-labs.com/dist/smlnj/benchmarks/110-benchmarks.tar.Z">here</a>.
Some of the benchmarks use data files in the
<a href="benchmarks/DATA/"><tt>DATA</tt></a> subdirectory.
<h1>Setup</h1>
All benchmarks were compiled and run on a 733MhZ P6 with 512M of physical
memory. The benchmarks were compiled with the default settings for all the
compilers, except for Moscow ML, which was passed the <tt>-orthodox
-standalone -toplevel</tt> switches. The SML/NJ executables were produced
by wrapping the entire program in a <tt>local</tt> declaration whose body
performs an <tt>exportFn</tt>.
<h1>Run time ratio</h1>
<p>
The following table gives the ratio of the run time of each benchmark
to the run time when compiled by MLton. A * in an entry means that the
compiler failed to compile the benchmark or that the benchmark failed to
run.
<p>
<table border=2 cellpadding=5 cellspacing=2>
<tr>
<th align = left>benchmark</th>
<th align = right>ML Kit</th>
<th align = right>Moscow ML</th>
<th align = right>SML/NJ</th>
</tr>
<tr>
<td align = left>barnes-hut</td>
<td align = right>3.5</td>
<td align = right>12.2</td>
<td align = right>0.8</td>
</tr>
<tr>
<td align = left>checksum</td>
<td align = right>*</td>
<td align = right>*</td>
<td align = right>3.3</td>
</tr>
<tr>
<td align = left>count-graphs</td>
<td align = right>6.7</td>
<td align = right>21.9</td>
<td align = right>1.5</td>
</tr>
<tr>
<td align = left>fft</td>
<td align = right>8.1</td>
<td align = right>26.3</td>
<td align = right>1.0</td>
</tr>
<tr>
<td align = left>fib</td>
<td align = right>0.9</td>
<td align = right>4.8</td>
<td align = right>1.1</td>
</tr>
<tr>
<td align = left>hamlet</td>
<td align = right>*</td>
<td align = right>9.2</td>
<td align = right>1.4</td>
</tr>
<tr>
<td align = left>knuth-bendix</td>
<td align = right>4.8</td>
<td align = right>12.6</td>
<td align = right>2.1</td>
</tr>
<tr>
<td align = left>lexgen</td>
<td align = right>2.4</td>
<td align = right>5.6</td>
<td align = right>1.6</td>
</tr>
<tr>
<td align = left>life</td>
<td align = right>6.9</td>
<td align = right>20.0</td>
<td align = right>0.9</td>
</tr>
<tr>
<td align = left>logic</td>
<td align = right>*</td>
<td align = right>3.5</td>
<td align = right>0.6</td>
</tr>
<tr>
<td align = left>mandelbrot</td>
<td align = right>4.2</td>
<td align = right>32.7</td>
<td align = right>1.4</td>
</tr>
<tr>
<td align = left>matrix-multiply</td>
<td align = right>10.1</td>
<td align = right>35.3</td>
<td align = right>3.0</td>
</tr>
<tr>
<td align = left>md5</td>
<td align = right>*</td>
<td align = right>*</td>
<td align = right>3.0</td>
</tr>
<tr>
<td align = left>merge</td>
<td align = right>*</td>
<td align = right>*</td>
<td align = right>10.7</td>
</tr>
<tr>
<td align = left>mlyacc</td>
<td align = right>*</td>
<td align = right>7.7</td>
<td align = right>1.7</td>
</tr>
<tr>
<td align = left>mpuz</td>
<td align = right>8.8</td>
<td align = right>47.9</td>
<td align = right>3.0</td>
</tr>
<tr>
<td align = left>nucleic</td>
<td align = right>*</td>
<td align = right>15.8</td>
<td align = right>0.7</td>
</tr>
<tr>
<td align = left>peek</td>
<td align = right>5.4</td>
<td align = right>28.4</td>
<td align = right>1.9</td>
</tr>
<tr>
<td align = left>psdes-random</td>
<td align = right>8.7</td>
<td align = right>*</td>
<td align = right>2.3</td>
</tr>
<tr>
<td align = left>ratio-regions</td>
<td align = right>12.4</td>
<td align = right>47.0</td>
<td align = right>5.7</td>
</tr>
<tr>
<td align = left>ray</td>
<td align = right>*</td>
<td align = right>11.8</td>
<td align = right>0.9</td>
</tr>
<tr>
<td align = left>raytrace</td>
<td align = right>*</td>
<td align = right>*</td>
<td align = right>2.3</td>
</tr>
<tr>
<td align = left>simple</td>
<td align = right>2.3</td>
<td align = right>12.5</td>
<td align = right>1.3</td>
</tr>
<tr>
<td align = left>smith-normal-form</td>
<td align = right>*</td>
<td align = right>*</td>
<td align = right>96.4</td>
</tr>
<tr>
<td align = left>tak</td>
<td align = right>1.7</td>
<td align = right>7.8</td>
<td align = right>1.4</td>
</tr>
<tr>
<td align = left>tensor</td>
<td align = right>*</td>
<td align = right>*</td>
<td align = right>5.9</td>
</tr>
<tr>
<td align = left>tsp</td>
<td align = right>2.4</td>
<td align = right>14.4</td>
<td align = right>1.6</td>
</tr>
<tr>
<td align = left>vector-concat</td>
<td align = right>11.5</td>
<td align = right>20.5</td>
<td align = right>7.1</td>
</tr>
<tr>
<td align = left>vector-rev</td>
<td align = right>12.4</td>
<td align = right>39.1</td>
<td align = right>23.1</td>
</tr>
<tr>
<td align = left>vliw</td>
<td align = right>2.6</td>
<td align = right>7.8</td>
<td align = right>1.3</td>
</tr>
<tr>
<td align = left>wc-input1</td>
<td align = right>13.4</td>
<td align = right>*</td>
<td align = right>7.3</td>
</tr>
<tr>
<td align = left>wc-scanStream</td>
<td align = right>8.0</td>
<td align = right>*</td>
<td align = right>1.6</td>
</tr>
<tr>
<td align = left>zebra</td>
<td align = right>11.8</td>
<td align = right>23.3</td>
<td align = right>6.9</td>
</tr>
<tr>
<td align = left>zern</td>
<td align = right>*</td>
<td align = right>*</td>
<td align = right>1.5</td>
</tr>
</table>
<h1>Compile time</h1>
The following table gives the compile time of each benchmark in seconds.
A * in an entry means that the compiler failed to compile the benchmark.
<p>
<table border=2 cellpadding=5 cellspacing=2>
<tr>
<th align = left>benchmark</th>
<th align = right>ML Kit</th>
<th align = right>MLton</th>
<th align = right>Moscow ML</th>
<th align = right>SML/NJ</th>
</tr>
<tr>
<td align = left>barnes-hut</td>
<td align = right>8.1</td>
<td align = right>2.6</td>
<td align = right>0.8</td>
<td align = right>1.7</td>
</tr>
<tr>
<td align = left>checksum</td>
<td align = right>*</td>
<td align = right>0.7</td>
<td align = right>*</td>
<td align = right>0.2</td>
</tr>
<tr>
<td align = left>count-graphs</td>
<td align = right>2.6</td>
<td align = right>1.8</td>
<td align = right>0.2</td>
<td align = right>1.1</td>
</tr>
<tr>
<td align = left>fft</td>
<td align = right>2.1</td>
<td align = right>1.5</td>
<td align = right>0.2</td>
<td align = right>1.0</td>
</tr>
<tr>
<td align = left>fib</td>
<td align = right>1.0</td>
<td align = right>0.7</td>
<td align = right>0.1</td>
<td align = right>0.2</td>
</tr>
<tr>
<td align = left>hamlet</td>
<td align = right>*</td>
<td align = right>52.3</td>
<td align = right>43.2</td>
<td align = right>91.6</td>
</tr>
<tr>
<td align = left>knuth-bendix</td>
<td align = right>5.3</td>
<td align = right>2.5</td>
<td align = right>0.4</td>
<td align = right>2.2</td>
</tr>
<tr>
<td align = left>lexgen</td>
<td align = right>10.2</td>
<td align = right>5.8</td>
<td align = right>0.8</td>
<td align = right>4.8</td>
</tr>
<tr>
<td align = left>life</td>
<td align = right>2.8</td>
<td align = right>1.5</td>
<td align = right>0.2</td>
<td align = right>0.7</td>
</tr>
<tr>
<td align = left>logic</td>
<td align = right>6.8</td>
<td align = right>7.5</td>
<td align = right>0.4</td>
<td align = right>2.0</td>
</tr>
<tr>
<td align = left>mandelbrot</td>
<td align = right>1.1</td>
<td align = right>0.8</td>
<td align = right>0.1</td>
<td align = right>0.2</td>
</tr>
<tr>
<td align = left>matrix-multiply</td>
<td align = right>1.2</td>
<td align = right>0.8</td>
<td align = right>0.1</td>
<td align = right>0.3</td>
</tr>
<tr>
<td align = left>md5</td>
<td align = right>*</td>
<td align = right>2.4</td>
<td align = right>*</td>
<td align = right>2.1</td>
</tr>
<tr>
<td align = left>merge</td>
<td align = right>1.0</td>
<td align = right>0.8</td>
<td align = right>0.1</td>
<td align = right>0.2</td>
</tr>
<tr>
<td align = left>mlyacc</td>
<td align = right>60.4</td>
<td align = right>19.4</td>
<td align = right>7.4</td>
<td align = right>24.1</td>
</tr>
<tr>
<td align = left>mpuz</td>
<td align = right>1.4</td>
<td align = right>1.0</td>
<td align = right>0.1</td>
<td align = right>0.4</td>
</tr>
<tr>
<td align = left>nucleic</td>
<td align = right>28.2</td>
<td align = right>4.5</td>
<td align = right>2.1</td>
<td align = right>3.0</td>
</tr>
<tr>
<td align = left>peek</td>
<td align = right>1.1</td>
<td align = right>1.1</td>
<td align = right>0.1</td>
<td align = right>0.2</td>
</tr>
<tr>
<td align = left>psdes-random</td>
<td align = right>1.1</td>
<td align = right>0.8</td>
<td align = right>*</td>
<td align = right>0.3</td>
</tr>
<tr>
<td align = left>ratio-regions</td>
<td align = right>4.4</td>
<td align = right>3.1</td>
<td align = right>0.4</td>
<td align = right>2.1</td>
</tr>
<tr>
<td align = left>ray</td>
<td align = right>3.7</td>
<td align = right>3.6</td>
<td align = right>0.2</td>
<td align = right>1.1</td>
</tr>
<tr>
<td align = left>raytrace</td>
<td align = right>*</td>
<td align = right>10.1</td>
<td align = right>*</td>
<td align = right>6.8</td>
</tr>
<tr>
<td align = left>simple</td>
<td align = right>14.6</td>
<td align = right>7.4</td>
<td align = right>0.9</td>
<td align = right>4.5</td>
</tr>
<tr>
<td align = left>smith-normal-form</td>
<td align = right>*</td>
<td align = right>8.1</td>
<td align = right>*</td>
<td align = right>3.5</td>
</tr>
<tr>
<td align = left>tak</td>
<td align = right>1.0</td>
<td align = right>0.7</td>
<td align = right>0.1</td>
<td align = right>0.2</td>
</tr>
<tr>
<td align = left>tensor</td>
<td align = right>*</td>
<td align = right>3.1</td>
<td align = right>*</td>
<td align = right>3.5</td>
</tr>
<tr>
<td align = left>tsp</td>
<td align = right>2.7</td>
<td align = right>1.8</td>
<td align = right>0.3</td>
<td align = right>0.8</td>
</tr>
<tr>
<td align = left>vector-concat</td>
<td align = right>1.0</td>
<td align = right>0.8</td>
<td align = right>0.1</td>
<td align = right>0.2</td>
</tr>
<tr>
<td align = left>vector-rev</td>
<td align = right>1.0</td>
<td align = right>0.8</td>
<td align = right>0.1</td>
<td align = right>0.2</td>
</tr>
<tr>
<td align = left>vliw</td>
<td align = right>36.2</td>
<td align = right>12.1</td>
<td align = right>3.0</td>
<td align = right>18.8</td>
</tr>
<tr>
<td align = left>wc-input1</td>
<td align = right>1.1</td>
<td align = right>1.7</td>
<td align = right>0.1</td>
<td align = right>0.3</td>
</tr>
<tr>
<td align = left>wc-scanStream</td>
<td align = right>1.1</td>
<td align = right>1.8</td>
<td align = right>0.1</td>
<td align = right>0.3</td>
</tr>
<tr>
<td align = left>zebra</td>
<td align = right>2.8</td>
<td align = right>5.0</td>
<td align = right>0.2</td>
<td align = right>0.8</td>
</tr>
<tr>
<td align = left>zern</td>
<td align = right>*</td>
<td align = right>1.1</td>
<td align = right>*</td>
<td align = right>0.8</td>
</tr>
</table>
<h1>Code size</h1>
The following table gives the code size of each benchmark in bytes. The
size for MLton and the ML Kit is the sum of text and data for the standalone
executable as reported by <tt>size</tt>. The size for Moscow ML is the size
in bytes of the executable <tt>a.out</tt>. The size for SML/NJ is the size
of the heap file created by <tt>exportFn</tt> and does not include the size
of the SML/NJ runtime system (approximately 95K). A * in an entry means that
the compiler failed to compile the benchmark.
<p>
<table border=2 cellpadding=5 cellspacing=2>
<tr>
<th align = left>benchmark</th>
<th align = right>ML Kit</th>
<th align = right>MLton</th>
<th align = right>Moscow ML</th>
<th align = right>SML/NJ</th>
</tr>
<tr>
<td align = left>barnes-hut</td>
<td align = right>179,980</td>
<td align = right>45,835</td>
<td align = right>94,990</td>
<td align = right>331,768</td>
</tr>
<tr>
<td align = left>checksum</td>
<td align = right>*</td>
<td align = right>22,994</td>
<td align = right>*</td>
<td align = right>332,504</td>
</tr>
<tr>
<td align = left>count-graphs</td>
<td align = right>109,948</td>
<td align = right>42,418</td>
<td align = right>84,575</td>
<td align = right>355,376</td>
</tr>
<tr>
<td align = left>fft</td>
<td align = right>107,284</td>
<td align = right>33,114</td>
<td align = right>84,095</td>
<td align = right>332,808</td>
</tr>
<tr>
<td align = left>fib</td>
<td align = right>68,588</td>
<td align = right>22,842</td>
<td align = right>79,878</td>
<td align = right>310,968</td>
</tr>
<tr>
<td align = left>hamlet</td>
<td align = right>*</td>
<td align = right>977,217</td>
<td align = right>277,168</td>
<td align = right>1,263,816</td>
</tr>
<tr>
<td align = left>knuth-bendix</td>
<td align = right>115,252</td>
<td align = right>62,139</td>
<td align = right>88,439</td>
<td align = right>321,504</td>
</tr>
<tr>
<td align = left>lexgen</td>
<td align = right>227,444</td>
<td align = right>132,090</td>
<td align = right>104,883</td>
<td align = right>390,136</td>
</tr>
<tr>
<td align = left>life</td>
<td align = right>100,220</td>
<td align = right>39,642</td>
<td align = right>83,390</td>
<td align = right>305,120</td>
</tr>
<tr>
<td align = left>logic</td>
<td align = right>136,068</td>
<td align = right>150,658</td>
<td align = right>87,252</td>
<td align = right>331,744</td>
</tr>
<tr>
<td align = left>mandelbrot</td>
<td align = right>101,612</td>
<td align = right>22,906</td>
<td align = right>81,341</td>
<td align = right>311,992</td>
</tr>
<tr>
<td align = left>matrix-multiply</td>
<td align = right>118,348</td>
<td align = right>23,466</td>
<td align = right>81,879</td>
<td align = right>338,632</td>
</tr>
<tr>
<td align = left>md5</td>
<td align = right>*</td>
<td align = right>36,187</td>
<td align = right>*</td>
<td align = right>331,792</td>
</tr>
<tr>
<td align = left>merge</td>
<td align = right>68,796</td>
<td align = right>23,962</td>
<td align = right>80,091</td>
<td align = right>307,904</td>
</tr>
<tr>
<td align = left>mlyacc</td>
<td align = right>525,404</td>
<td align = right>425,226</td>
<td align = right>148,286</td>
<td align = right>700,456</td>
</tr>
<tr>
<td align = left>mpuz</td>
<td align = right>89,500</td>
<td align = right>27,970</td>
<td align = right>82,381</td>
<td align = right>320,184</td>
</tr>
<tr>
<td align = left>nucleic</td>
<td align = right>234,220</td>
<td align = right>60,066</td>
<td align = right>207,154</td>
<td align = right>354,288</td>
</tr>
<tr>
<td align = left>peek</td>
<td align = right>77,420</td>
<td align = right>30,539</td>
<td align = right>81,618</td>
<td align = right>311,016</td>
</tr>
<tr>
<td align = left>psdes-random</td>
<td align = right>84,540</td>
<td align = right>24,178</td>
<td align = right>*</td>
<td align = right>313,016</td>
</tr>
<tr>
<td align = left>ratio-regions</td>
<td align = right>111,308</td>
<td align = right>59,378</td>
<td align = right>87,485</td>
<td align = right>335,856</td>
</tr>
<tr>
<td align = left>ray</td>
<td align = right>123,356</td>
<td align = right>73,377</td>
<td align = right>89,860</td>
<td align = right>384,072</td>
</tr>
<tr>
<td align = left>raytrace</td>
<td align = right>*</td>
<td align = right>180,902</td>
<td align = right>*</td>
<td align = right>510,056</td>
</tr>
<tr>
<td align = left>simple</td>
<td align = right>194,364</td>
<td align = right>170,906</td>
<td align = right>94,397</td>
<td align = right>641,056</td>
</tr>
<tr>
<td align = left>smith-normal-form</td>
<td align = right>*</td>
<td align = right>143,386</td>
<td align = right>*</td>
<td align = right>483,400</td>
</tr>
<tr>
<td align = left>tak</td>
<td align = right>68,356</td>
<td align = right>22,866</td>
<td align = right>79,928</td>
<td align = right>306,872</td>
</tr>
<tr>
<td align = left>tensor</td>
<td align = right>*</td>
<td align = right>53,802</td>
<td align = right>*</td>
<td align = right>342,048</td>
</tr>
<tr>
<td align = left>tsp</td>
<td align = right>115,732</td>
<td align = right>39,067</td>
<td align = right>86,140</td>
<td align = right>322,552</td>
</tr>
<tr>
<td align = left>vector-concat</td>
<td align = right>77,660</td>
<td align = right>23,426</td>
<td align = right>80,191</td>
<td align = right>317,128</td>
</tr>
<tr>
<td align = left>vector-rev</td>
<td align = right>77,860</td>
<td align = right>23,482</td>
<td align = right>80,073</td>
<td align = right>317,128</td>
</tr>
<tr>
<td align = left>vliw</td>
<td align = right>418,076</td>
<td align = right>264,234</td>
<td align = right>135,386</td>
<td align = right>618,576</td>
</tr>
<tr>
<td align = left>wc-input1</td>
<td align = right>144,652</td>
<td align = right>43,075</td>
<td align = right>86,900</td>
<td align = right>311,992</td>
</tr>
<tr>
<td align = left>wc-scanStream</td>
<td align = right>145,100</td>
<td align = right>45,675</td>
<td align = right>87,076</td>
<td align = right>313,016</td>
</tr>
<tr>
<td align = left>zebra</td>
<td align = right>85,404</td>
<td align = right>111,131</td>
<td align = right>83,419</td>
<td align = right>310,256</td>
</tr>
<tr>
<td align = left>zern</td>
<td align = right>*</td>
<td align = right>27,041</td>
<td align = right>*</td>
<td align = right>337,944</td>
</tr>
</table>
<hr>
<!-- Created: Tue Jun 2 16:05:24 EDT 1998 -->
<!-- hhmts start -->
Last modified: Tue Jul 10 15:05:24 PDT 2001
<!-- hhmts end -->
</body>
</html>
--BKXRV8ZmZi--