Team PLClub ICFP entry -- comparing the performance of OCAML and SML

[Stephen Weeks]

> |> So, you are correct for Real.Math.sqrt -- the SML/NJ
> |> implementation is about 10 times slower than the C math library.  I
> |> then computed a rough estimate for the number of seconds spent by
> |> SML/NJ in sqrt for each benchmark by taking the number of sqrts
> |> divided by SML/NJ's rate for sqrts.  Here are the numbers for each
> |> benchmark, along with the original running time measurements.
> |>                                                 
> |>               number of    number SML/NJ   Total running time
> |>                calls to        of   sqrt 
> |>               Real.Math     sqrts   time  OCAML  MLton SML/NJ
> |> -----------   ---------  --------  -----  -----  -----  -----
> |> holes            252374    252359    0.6    1.8    3.2    3.9
> |> fov              416558    416545    0.9    1.5    1.8    3.2
> |> intercyl         789548    735426    1.6    1.6    2.1    4.3
> |> snowgoon         470715    362464    0.8    2.9    3.3    5.1
> |> dice            1263757    925063    2.0    3.9    4.9    8.8
> |> golf             193489    130088    0.3    1.5    2.4    3.1
> |> cone-fractal     603638    455931    1.0    3.7    4.3    6.5
> |> large             73500     73487    0.6    4.3    3.0    6.7
> |> pipe            1087559    983314    2.2    5.4    4.6    7.9
> |> chess           1192926   1030668    2.3   16.0   15.5   21.6
> |> fractal         2296066   2229266    4.9   12.2    8.5   45.4
> |> 
> |> From this table, I conclude that the performance penalty for using
> |> sqrt in SML/NJ is a significant cause for why SML/NJ is slower than
> |> OCAML or MLton, but not the only cause.
> |> 
> 
> Thanks for all the experiments. But I would not make any conclusion
> this quickly. For example, I am surprised that SML/NJ only spent
> 10% of its time executing sqrt on chess and fractal if "sqrt" is
> indeed a hot spot. 

You're welcome.  To be clear, the 4.9 was not based on direct measurement.  It
was computed as the ratio of 2,229,266 and the number of calls to sqrt that
SML/NJ made per second in a simple loop I wrote, roughly 460,000.  I did not say
that sqrt was a hot spot in terms of running time.  What I concluded (from the
numbers above) was that sqrt was the dominant function in terms of number of
calls to Real.Math functions.  For example, on fractal, sqrt was responsible for
2,229,266 out of 2,296,066 Real.Math calls.  I do not understand why the fact
that 4.9 is roughly 10% of 45.4 gives a reason to doubt the conclusion that sqrt
was a significant cause for why SML/NJ is slower than OCAML or MLton, but not
the only cause.  To me, that 4.5 is such a small part of the difference between
45.4 and 8.5 or 12.2 supports the conclusion that there are other causes besides
sqrt.

> SML/NJ also implements many of its Real primitives in ML itself
> (e.g., floor, trunc, round, cell, ...), so they may also contribute
> to the slowdown. 

Agreed.  It is unfortuntate for now that using Real and Real.Math in SML/NJ pays
such a penalty.   As Dave mentioned, SML/NJ is moving to an infrastructure where
sqrt (and presumably other Real.Math functions) is natively supported.  That
will be a good thing when it happens.