This page compares the performance of a number of SML compilers on a range of benchmarks.

This page compares the following SML compiler versions.

  • MLton 20171211 (git 79d4a623c)

  • ML Kit 4.3.12 (20171210)

  • Moscow ML 2.10.1 ++ (git f529b33bb, 20170711)

  • Poly/ML 5.7.2 Testing (git 5.7.1-35-gcb73407a)

  • SML/NJ 110.81 (20170501)

There are tables for run time, code size, and compile time.

Setup

All benchmarks were compiled and run on a 2.6 GHz Core i7-5600U with 16G of RAM. The benchmarks were compiled with the default settings for all the compilers, except for Moscow ML, which was passed the -orthodox -standalone -toplevel switches. The Poly/ML executables were produced using polyc. The SML/NJ executables were produced by wrapping the entire program in a local declaration whose body performs an SMLofNJ.exportFn.

For more details, or if you want to run the benchmarks yourself, please see the benchmark directory of our Sources.

All of the benchmarks are available for download from this page. Some of the benchmarks were obtained from the SML/NJ benchmark suite. Some of the benchmarks expect certain input files to exist in the DATA subdirectory.

Run-time ratio

The following table gives the ratio of the run time of each benchmark when compiled by another compiler to the run time when compiled by MLton. That is, the larger the number, the slower the generated code runs. A number larger than one indicates that the corresponding compiler produces code that runs more slowly than MLton. A * in an entry means the compiler failed to compile the benchmark or that the benchmark failed to run.

benchmark MLton ML-Kit MosML Poly/ML SML/NJ

barnes-hut.sml

1.00

10.11

19.36

2.98

1.24

boyer.sml

1.00

*

7.87

1.22

1.75

checksum.sml

1.00

30.79

*

10.94

9.08

count-graphs.sml

1.00

6.51

40.42

2.34

2.32

DLXSimulator.sml

1.00

0.97

*

0.60

*

even-odd.sml

1.00

0.50

11.50

0.42

0.42

fft.sml

1.00

7.35

81.51

4.03

1.19

fib.sml

1.00

1.41

10.94

1.25

1.17

flat-array.sml

1.00

7.19

68.33

5.28

13.16

hamlet.sml

1.00

4.97

22.85

1.58

*

imp-for.sml

1.00

4.99

57.84

3.34

4.67

knuth-bendix.sml

1.00

*

18.43

3.18

3.06

lexgen.sml

1.00

2.76

7.94

3.19

*

life.sml

1.00

1.80

20.19

0.89

1.50

logic.sml

1.00

5.10

11.06

1.15

1.27

mandelbrot.sml

1.00

3.50

25.52

1.33

1.28

matrix-multiply.sml

1.00

29.40

183.02

7.41

15.19

md5.sml

1.00

95.18

*

32.61

47.47

merge.sml

1.00

1.42

*

0.74

3.24

mlyacc.sml

1.00

1.83

8.45

0.84

*

model-elimination.sml

1.00

4.03

12.42

1.70

2.25

mpuz.sml

1.00

3.73

57.44

2.05

3.22

nucleic.sml

1.00

3.96

*

1.73

1.20

output1.sml

1.00

6.26

30.85

7.82

5.99

peek.sml

1.00

9.37

44.78

2.18

2.15

psdes-random.sml

1.00

*

*

2.79

3.59

ratio-regions.sml

1.00

5.68

165.56

3.92

37.52

ray.sml

1.00

12.05

25.08

8.73

1.75

raytrace.sml

1.00

*

*

2.11

3.33

simple.sml

1.00

2.95

24.03

3.67

1.93

smith-normal-form.sml

1.00

*

*

1.04

*

string-concat.sml

1.00

1.88

28.01

0.70

2.67

tailfib.sml

1.00

1.58

23.57

0.90

1.04

tak.sml

1.00

1.69

15.90

1.57

2.01

tensor.sml

1.00

*

*

*

2.07

tsp.sml

1.00

2.19

66.76

3.27

1.48

tyan.sml

1.00

*

19.43

1.08

1.03

vector32-concat.sml

1.00

13.85

*

1.80

12.48

vector64-concat.sml

1.00

*

*

*

13.92

vector-rev.sml

1.00

7.88

68.85

9.39

68.80

vliw.sml

1.00

2.46

15.39

1.43

1.55

wc-input1.sml

1.00

6.00

*

29.25

9.54

wc-scanStream.sml

1.00

80.43

*

19.45

8.71

zebra.sml

1.00

4.62

35.56

1.68

9.97

zern.sml

1.00

*

*

*

1.60

Note: for SML/NJ, the smith-normal-form.sml benchmark was killed after running for over 51,000 seconds.

Code size

The following table gives the code size of each benchmark in bytes. The size for MLton and the ML Kit is the sum of text and data for the standalone executable as reported by size. The size for Moscow ML is the size in bytes of the executable a.out. The size for Poly/ML is the difference in size of the database before the session start and after the commit. The size for SML/NJ is the size of the heap file created by exportFn and does not include the size of the SML/NJ runtime system (approximately 100K). A * in an entry means that the compiler failed to compile the benchmark.

benchmark MLton ML-Kit MosML Poly/ML SML/NJ

barnes-hut.sml

180,788

810,267

199,503

148,120

402,480

boyer.sml

250,246

*

248,018

196,984

496,664

checksum.sml

122,422

225,274

*

106,088

406,560

count-graphs.sml

151,878

250,126

187,048

144,032

428,136

DLXSimulator.sml

223,073

827,483

*

272,664

*

even-odd.sml

122,350

87,586

181,415

106,072

380,928

fft.sml

145,008

237,230

186,228

131,400

418,896

fib.sml

122,310

87,402

181,312

106,088

380,928

flat-array.sml

121,958

104,102

181,464

106,072

394,256

hamlet.sml

1,503,849

2,280,691

407,219

2,249,504

*

imp-for.sml

122,078

89,346

181,470

106,088

381,952

knuth-bendix.sml

193,145

*

192,659

161,080

400,408

lexgen.sml

308,296

826,819

213,128

268,272

*

life.sml

141,862

721,419

186,463

118,552

384,024

logic.sml

211,086

782,667

188,908

198,408

409,624

mandelbrot.sml

122,086

700,075

183,037

106,104

386,048

matrix-multiply.sml

124,398

280,006

184,328

110,232

416,784

md5.sml

150,497

271,794

*

122,624

399,416

merge.sml

123,846

100,858

181,542

106,136

381,960

mlyacc.sml

678,920

1,233,587

263,721

576,728

*

model-elimination.sml

846,779

1,432,283

297,108

777,664

985,304

mpuz.sml

124,126

229,078

184,440

114,584

392,232

nucleic.sml

298,038

507,186

*

475,808

456,744

output1.sml

157,973

699,003

181,680

118,800

380,928

peek.sml

156,401

201,138

183,438

110,456

385,072

psdes-random.sml

126,486

106,166

*

106,088

393,256

ratio-regions.sml

150,174

265,694

190,088

184,536

414,760

ray.sml

260,863

736,795

195,064

198,976

512,160

raytrace.sml

384,905

*

*

446,424

623,824

simple.sml

365,578

895,139

197,765

1,051,952

708,696

smith-normal-form.sml

286,474

*

*

262,616

547,984

string-concat.sml

119,102

140,626

183,249

106,088

390,160

tailfib.sml

122,110

87,890

181,369

106,072

381,952

tak.sml

122,246

87,402

181,349

106,088

376,832

tensor.sml

186,545

*

*

*

421,984

tsp.sml

163,033

722,571

188,634

126,984

393,264

tyan.sml

235,449

*

195,401

184,816

478,296

vector32-concat.sml

123,790

104,398

*

106,200

394,256

vector64-concat.sml

123,846

*

*

*

405,552

vector-rev.sml

122,982

104,614

181,534

106,072

394,256

vliw.sml

538,074

1,182,851

249,884

580,792

749,752

wc-input1.sml

186,152

699,459

191,347

127,200

386,048

wc-scanStream.sml

196,232

700,131

191,539

127,232

387,072

zebra.sml

230,433

128,354

186,322

127,048

390,184

zern.sml

156,902

*

*

*

453,768

Compile time

The following table gives the compile time of each benchmark in seconds. A * in an entry means that the compiler failed to compile the benchmark.

benchmark MLton ML-Kit MosML Poly/ML SML/NJ

barnes-hut.sml

2.70

0.89

0.15

0.29

0.20

boyer.sml

2.87

*

0.14

0.20

0.41

checksum.sml

2.21

0.24

*

0.07

0.05

count-graphs.sml

2.28

0.34

0.04

0.11

0.21

DLXSimulator.sml

2.93

1.01

*

0.27

*

even-odd.sml

2.23

0.20

0.01

0.07

0.04

fft.sml

2.35

0.28

0.03

0.09

0.10

fib.sml

2.16

0.19

0.01

0.07

0.04

flat-array.sml

2.16

0.20

0.01

0.07

0.04

hamlet.sml

12.28

19.25

23.75

6.44

*

imp-for.sml

2.14

0.20

0.01

0.08

0.04

knuth-bendix.sml

2.48

*

0.08

0.14

0.23

lexgen.sml

3.31

0.75

0.15

0.22

*

life.sml

2.25

0.32

0.03

0.09

0.10

logic.sml

2.72

0.57

0.07

0.17

0.21

mandelbrot.sml

2.14

0.24

0.01

0.07

0.04

matrix-multiply.sml

2.14

0.24

0.01

0.08

0.05

md5.sml

2.31

0.39

*

0.12

0.27

merge.sml

2.15

0.21

0.01

0.07

0.04

mlyacc.sml

7.07

4.53

2.05

0.80

*

model-elimination.sml

6.78

4.76

1.20

1.65

4.78

mpuz.sml

2.14

0.28

0.02

0.08

0.07

nucleic.sml

3.96

2.12

*

0.37

0.49

output1.sml

2.30

0.22

0.01

0.07

0.04

peek.sml

2.26

0.20

0.01

0.07

0.04

psdes-random.sml

2.12

0.22

*

9.83

12.55

ratio-regions.sml

2.59

0.47

0.07

0.16

0.24

ray.sml

2.95

0.46

0.05

0.17

0.14

raytrace.sml

3.93

*

*

0.45

0.74

simple.sml

3.42

1.23

0.30

0.32

0.53

smith-normal-form.sml

3.23

*

*

0.15

0.32

string-concat.sml

2.25

0.28

0.01

0.08

0.05

tailfib.sml

2.24

0.21

0.01

0.08

0.05

tak.sml

2.23

0.20

0.01

0.08

0.05

tensor.sml

2.73

*

*

*

0.44

tsp.sml

2.42

0.38

0.05

0.11

0.11

tyan.sml

2.93

*

0.10

0.27

0.31

vector32-concat.sml

2.23

0.22

*

0.07

0.04

vector64-concat.sml

2.18

*

*

*

0.04

vector-rev.sml

2.23

0.22

0.01

0.08

0.05

vliw.sml

5.25

2.93

0.63

0.94

1.85

wc-input1.sml

2.46

0.24

0.01

0.08

0.05

wc-scanStream.sml

2.61

0.25

0.01

0.08

0.05

zebra.sml

2.99

0.35

0.03

0.09

0.11

zern.sml

2.31

*

*

*

0.11