shootout scorecard

[Stephen Weeks]

> > In particular, mlton's average CPU is 6, while ocamlb's is 10.
> 
> That's the average CPU Rank on the language summary pages.  The rank
> is the distance from the top position on the tests.  Then I just take
> the arithmetic mean.  That number is fairly meaningless, but I wanted
> something to allow me to colorize the ranks that were probably in need
> of improvement.  

Right.  I like that info.  It helps to see what's relatively slow.  I was mostly
stating that to contrast with the following.

> > Further mlton beats ocamlb in all but two of the benchmarks (reversefile,
> > wordfreq), yet is only slightly ahead of it in the scoring.  Am I misleading
> > myself?
> 
> Well, the scoring algorithm (as described on the bottom of the craps
> page) may reflect which language is faster, but it doesn't reflect at
> all how much faster.
> 
> For CPU (for example), the score on the scorecard is relative to the
> highest and lowest scores.  For instance if language A takes 1 second,
> language B takes 2 seconds, and language C takes 200 seconds, then the
> base score is 200 + 1 = 201.  The scores for each language are calculated
> by subtracting the CPU from the base, then normalizing to between 0 and
> 10.  So the score for A is 10 * (201 - 1) / 201 = 9.95.  score for B is
> 9.90, and the score for C is 0.05.  These scores are then multiplied
> by the weight of each test, and then summed over all tests.
> 
> I've been thinking about replacing this algorithm ... I just haven't
> spent the time on it yet.

Aha.  Now I get it.  The differences between the faster implementations are
minimized whenever a really slow one causes there to be a large normalizing
factor.  That does seem like it should be replaced.

Thanks for the explanation.