MLton 20070826 Performance
Home  Index  
This page compares the performance of a number of SML compilers on a range of benchmarks. For a [WWW]performance comparison of many different languages, including [WWW]MLton, see the [WWW]Computer Language Shootout.

This page compares the following SML compiler versions.

There are tables for run time, code size, and compile time.

Setup

All benchmarks were compiled and run on a 2.6 GHz Celeron with 2G of RAM. The benchmarks were compiled with the default settings for all the compilers, except for Moscow ML, which was passed the -orthodox -standalone -toplevel switches. The Poly/ML executables were produced by use'ing the file, followed by a PolyML.commit. The SML/NJ executables were produced by wrapping the entire program in a local declaration whose body performs an SMLofNJ.exportFn.

For more details, or if you want to run the benchmarks yourself, please see the [WWW]benchmark directory of our Sources.

All of the benchmarks are available for download from this page. Some of the benchmarks were obtained from the SML/NJ benchmark suite. Some of the benchmarks expect certain input files to exist in the [WWW]DATA subdirectory.

Run-time ratio

The following table gives the ratio of the run time of each benchmark when compiled by another compiler to the run time when compiled by MLton. That is, the larger the number, the slower the generated code runs. A number larger than one indicates that the corresponding compiler produces code that runs more slowly than MLton. If an entry is *, that means that the corresponding compiler failed to compile the benchmark or that the benchmark failed to run.

benchmark MLton ML-Kit MosML Poly/ML SML/NJ
[WWW]barnes-hut 1.0 * * * 1.6
[WWW]boyer 1.0 * 10.1 1.9 3.1
[WWW]checksum 1.0 * * * *
[WWW]count-graphs 1.0 7.3 60.7 4.2 3.8
[WWW]DLXSimulator 1.0 * * * *
[WWW]fft 1.0 1.2 * 24.2 0.8
[WWW]fib 1.0 0.9 5.0 1.2 1.3
[WWW]flat-array 1.0 2.2 35.0 1041.6 13.4
[WWW]hamlet 1.0 * * * 3.1
[WWW]imp-for 1.0 2.8 63.0 5.1 5.6
[WWW]knuth-bendix 1.0 * 19.8 4.8 4.6
[WWW]lexgen 1.0 2.5 5.0 1.7 1.5
[WWW]life 1.0 1.7 30.6 7.7 1.4
[WWW]logic 1.0 * 9.4 1.2 2.1
[WWW]mandelbrot 1.0 4.2 34.0 51.1 1.3
[WWW]matrix-multiply 1.0 8.3 42.5 13.2 5.3
[WWW]md5 1.0 * * * *
[WWW]merge 1.0 * * 1.1 7.9
[WWW]mlyacc 1.0 1.5 8.2 1.2 2.2
[WWW]model-elimination 1.0 * * * 2.6
[WWW]mpuz 1.0 2.3 78.2 4.6 4.1
[WWW]nucleic 1.0 * * 23.5 0.8
[WWW]output1 1.0 30.7 61.4 16.2 14.4
[WWW]peek 1.0 15.2 176.9 17.9 11.3
[WWW]psdes-random 1.0 5.0 * * 2.7
[WWW]ratio-regions 1.0 2.0 34.7 2.1 5.4
[WWW]ray 1.0 * 14.8 22.3 0.8
[WWW]raytrace 1.0 * * * 3.3
[WWW]simple 1.0 1.7 19.3 7.3 2.4
[WWW]smith-normal-form 1.0 * * * >1000
[WWW]tailfib 1.0 1.0 51.9 3.2 1.4
[WWW]tak 1.0 1.2 17.0 1.3 2.0
[WWW]tensor 1.0 * * * 7.4
[WWW]tsp 1.0 3.4 31.8 * 17.7
[WWW]tyan 1.0 * 15.7 1.0 1.6
[WWW]vector-concat 1.0 1.2 20.4 2.0 20.4
[WWW]vector-rev 1.0 2.2 41.9 2.3 152.4
[WWW]vliw 1.0 * * * 2.5
[WWW]wc-input1 1.0 11.1 * 7.5 17.2
[WWW]wc-scanStream 1.0 22.1 * 203.7 11.5
[WWW]zebra 1.0 3.9 30.2 3.4 8.5
[WWW]zern 1.0 * * * 2.6

Note: for SML/NJ, the smith-normal-form benchmark was killed after running for over 25,000 seconds.

Code size

The following table gives the code size of each benchmark in bytes. The size for MLton and the ML Kit is the sum of text and data for the standalone executable as reported by size. The size for Moscow ML is the size in bytes of the executable a.out. The size for Poly/ML is the difference in size of the database before the session start and after the commit. The size for SML/NJ is the size of the heap file created by exportFn and does not include the size of the SML/NJ runtime system (approximately 100K). A * in an entry means that the compiler failed to compile the benchmark.

benchmark MLton ML-Kit MosML Poly/ML SML/NJ
[WWW]barnes-hut 103,231 * * * 433,216
[WWW]boyer 138,518 163,204 116,300 122,880 526,376
[WWW]checksum 52,794 * * * *
[WWW]count-graphs 66,838 84,124 84,613 98,304 454,776
[WWW]DLXSimulator 129,398 * * * *
[WWW]fft 64,797 80,240 84,046 65,536 434,256
[WWW]fib 47,738 18,588 79,892 49,152 415,488
[WWW]flat-array 47,762 23,820 80,034 49,152 410,680
[WWW]hamlet 1,256,813 * * * 1,412,360
[WWW]imp-for 47,626 19,372 80,040 57,344 400,424
[WWW]knuth-bendix 109,126 93,400 88,439 180,224 431,144
[WWW]lexgen 203,559 208,332 104,883 196,608 501,824
[WWW]life 66,130 78,084 83,390 65,536 414,760
[WWW]logic 106,614 116,880 87,251 114,688 440,360
[WWW]mandelbrot 47,690 77,004 81,340 57,344 404,520
[WWW]matrix-multiply 49,181 87,016 82,417 57,344 435,256
[WWW]md5 77,646 * * * *
[WWW]merge 49,318 24,296 80,090 49,152 400,432
[WWW]mlyacc 507,431 473,748 148,286 2,850,816 820,336
[WWW]model-elimination 638,084 * * * 1,009,880
[WWW]mpuz 50,594 73,232 82,382 81,920 408,616
[WWW]nucleic 199,181 258,552 * 221,184 487,480
[WWW]output1 80,720 63,336 80,187 49,152 399,400
[WWW]peek 76,302 62,092 81,621 57,344 403,544
[WWW]psdes-random 48,402 25,196 * * 421,944
[WWW]ratio-regions 73,914 95,924 87,482 73,728 443,448
[WWW]ray 183,243 108,848 89,859 147,456 493,712
[WWW]raytrace 265,332 * * * 636,112
[WWW]simple 222,914 192,032 94,396 475,136 756,840
[WWW]smith-normal-form 181,686 * * 131,072 558,224
[WWW]tailfib 47,434 18,804 79,943 57,344 399,400
[WWW]tak 47,818 18,580 79,908 57,344 411,392
[WWW]tensor 97,677 * * * 450,672
[WWW]tsp 82,190 97,716 86,146 * 425,024
[WWW]tyan 134,910 137,800 91,586 196,608 477,272
[WWW]vector-concat 49,018 23,924 80,194 49,152 410,680
[WWW]vector-rev 48,246 24,104 80,078 57,344 410,680
[WWW]vliw 393,762 * * * 731,304
[WWW]wc-input1 101,850 129,212 85,771 49,152 404,520
[WWW]wc-scanStream 109,106 129,708 85,947 49,152 405,544
[WWW]zebra 141,146 41,532 83,422 90,112 419,896
[WWW]zern 91,087 * * * 479,384

Compile time

The following table gives the compile time of each benchmark in seconds. A * in an entry means that the compiler failed to compile the benchmark.

benchmark MLton ML-Kit MosML Poly/ML SML/NJ
[WWW]barnes-hut 8.28 * * * 1.37
[WWW]boyer 8.14 8.99 0.39 0.12 3.20
[WWW]checksum 5.45 * * * *
[WWW]count-graphs 6.12 2.06 0.14 0.05 0.90
[WWW]DLXSimulator 9.81 * * * *
[WWW]fft 5.95 1.32 0.11 0.05 0.69
[WWW]fib 5.45 0.60 0.05 0.02 0.22
[WWW]flat-array 5.33 0.61 0.04 0.01 0.25
[WWW]hamlet 85.70 * * * 88.87
[WWW]imp-for 5.37 0.73 0.05 0.01 0.25
[WWW]knuth-bendix 7.09 4.11 0.19 0.12 1.60
[WWW]lexgen 11.02 7.21 0.40 0.26 3.63
[WWW]life 5.84 2.16 0.10 0.04 0.64
[WWW]logic 7.02 4.82 0.22 0.09 1.68
[WWW]mandelbrot 5.41 0.75 0.06 0.02 0.29
[WWW]matrix-multiply 5.39 0.77 0.06 0.01 0.30
[WWW]md5 6.01 * * * *
[WWW]merge 5.41 0.62 0.06 0.02 0.26
[WWW]mlyacc 24.70 40.69 3.35 1.08 18.04
[WWW]model-elimination 25.04 * * * 28.79
[WWW]mpuz 5.41 1.07 0.07 0.03 0.45
[WWW]nucleic 14.24 24.79 * 0.36 2.78
[WWW]output1 6.05 0.68 0.05 0.01 0.23
[WWW]peek 6.04 0.70 0.05 0.02 0.25
[WWW]psdes-random 5.39 0.75 * * 64.13
[WWW]ratio-regions 6.63 4.02 0.21 0.11 1.50
[WWW]ray 9.51 3.02 0.15 0.08 1.03
[WWW]raytrace 13.92 * * * 5.08
[WWW]simple 11.40 13.19 0.43 0.21 3.76
[WWW]smith-normal-form 8.90 * * 0.10 2.25
[WWW]tailfib 5.35 0.64 0.05 0.02 0.24
[WWW]tak 5.36 0.62 0.05 0.01 0.22
[WWW]tensor 8.75 * * * 2.81
[WWW]tsp 6.50 1.93 0.15 * 0.66
[WWW]tyan 8.86 6.25 0.30 0.17 2.28
[WWW]vector-concat 5.52 0.68 0.05 0.01 0.25
[WWW]vector-rev 5.33 0.64 0.05 0.02 0.26
[WWW]vliw 18.28 * * * 13.12
[WWW]wc-input1 6.85 0.68 0.07 0.02 0.27
[WWW]wc-scanStream 7.07 0.69 0.06 0.02 0.29
[WWW]zebra 8.57 2.30 0.09 0.04 0.78
[WWW]zern 6.20 * * * 0.65


Last edited on 2006-11-02 17:55:30 by MatthewFluet.