[MLton] mlton.org content migration

[Matthew Fluet]

I've continued to research alternative hosting solutions for the
mlton.org content.  My preference is still to migrate back to
SourceForge.net.  I've done some preliminary work and experiments, and
I think that most of the transition would be straightforward.

There were some concerns about SourceForge being "slow".  My
experiments show that it is actually faster than the current mlton.org
hosting provider.  Here are three experiments:

1) I exported the MLton Subversion repository and imported it into
mlton.svn.sourceforge.net.  I performed a "/usr/bin/time svn checkout"
of mlton/trunk at HEAD from both, with the following results over five
trials:

mlton.org:
       54.62 real         1.41 user         2.62 sys
       67.31 real         1.41 user         2.62 sys
       47.28 real         1.41 user         2.64 sys
       42.38 real         1.40 user         2.60 sys
       52.38 real         1.41 user         2.61 sys
mlton.svn.sourceforge.net:
       18.18 real         1.86 user         2.59 sys
       19.40 real         1.87 user         2.61 sys
       20.37 real         1.86 user         2.61 sys
       19.45 real         1.87 user         2.58 sys
       20.33 real         1.87 user         2.54 sys

So, the slowest SourceForge checkout is half the time of the fastest
mlton.org checkout.

2) I copied the mlton-svnroot.tar.bz2 file (48M) to
mlton.sourceforge.net.  I performed a "/usr/bin/time wget" of the file
from both, with the following results:

mlton.org (one large):
       40.98 real         0.11 user         0.80 sys
       41.50 real         0.11 user         0.79 sys
       40.33 real         0.10 user         0.79 sys
       40.98 real         0.11 user         0.79 sys
       41.48 real         0.11 user         0.81 sys
mlton.sourceforge.net (one large):
       41.19 real         0.06 user         0.68 sys
       42.30 real         0.06 user         0.68 sys
       40.84 real         0.05 user         0.57 sys
       40.78 real         0.06 user         0.61 sys
       42.21 real         0.06 user         0.63 sys

So, nearly identical performance for downloading one large file,
indicating that the bandwidth for web content is about the same.

3) I copied the guide/ directory from mlton.org to
mlton.sourceforge.net.  These are static HTML snapshots of the wiki
from the past three releases (plus corresponding .tar.gz files),
totaling 7.4M over 401 files.  I performed a "/usr/bin/time wget -r
-np -l inf" of this directory from both, with the following results:

mlton.org (many small):
       39.50 real         0.12 user         0.27 sys
       40.67 real         0.12 user         0.27 sys
       41.05 real         0.12 user         0.27 sys
       41.20 real         0.12 user         0.27 sys
       41.12 real         0.12 user         0.28 sys
mlton.sourceforge.net (many small):
       17.56 real         0.12 user         0.21 sys
       20.64 real         0.11 user         0.20 sys
       16.87 real         0.12 user         0.20 sys
       15.92 real         0.11 user         0.20 sys
       15.99 real         0.11 user         0.20 sys

So, the slowest SourceForge fetch is about half the fastest mlton.org
fetch, indicating that the latency for web content is much better on
SF.

Of course, what that doesn't test is speed of dynamic content
generation, which might or might not be important.  Currently, all of
the wiki content of mlton.org is dynamic content, but it isn't clear
that it would be easy or worthwhile to install MoinMoin on MLton's
SourceForge project web.


The mlton-{devel,user}@lists.sourceforge.net mailing lists still
exist, though had been purged of subscribers.  I worked through all of
the MailMan admin screens and configured them to reject all spam (as
identified by the SourceForge.net spam filters), to hold all
non-subscriber posts for moderation, to accept all subscriber posts,
and to admin approve subscription requests.  [There are also
mlton-{announce,commit}@lists.sourceforge.net mailing lists,
configured as read-only.]  As I mentioned previously, I think that
this is the simplest configuration.  Certainly simpler than the
configuration currently on mlton.org, with its whitelist and no
automatic spam filtering.  As I noted previously, we can also import
our existing archives, although they end up not as pipermail archives,
but as SF's "forum archives" [which I admit are significantly less
navigable than the pipermail archives].  It is also trivially to copy
our existing pipermail archives (they are just static web content)
over to mlton.sourceforge.net.  In a perfect world, it would be nice
to continue to add new messages to the pipermail archives.  Since one
can access the mbox archives of SF mailing lists for backup purposes,
it might be possible to do so, though non-trivial as pipermail seems
to be fairly tightly integrated with mailman.


That leaves the wiki content.  As noted before, SF provides MediaWiki,
but at a rather deeper URL than just mlton.sourceforge.net.  Also, as
noted before, we would lose many of our wiki macros for pulling in
content directly from the Subversion repository.  Also, we would lose
syntax highlighting of SML code (whether directly in the page or
pulled from the Subversion repository).

One option is to move to a completely static site via a static site
generator.  There seem to be quite a number of tools for generating
HTML from simplified markup.  I don't imagine that it would be too
difficult to migrate the MoinMoin markup to a different markup.
Ideally, we would use a tool that could be extended with analogs of
our existing wiki macros and could make use of highlighting of SML
code (i.e., using enscript or Pygments).  My "vision" would be to
place the markup pages plus any plugins/extensions to the base tool
under version control, which would ensure that anyone could rebuild
the site.  Updating the web content from the version control content
would be a nightly cron job (or possibly triggered by commits).


Those are my thoughts.  Migrating the Subversion repository is very
quick (a couple of hours).  Migrating the mailing lists would be a
couple of days: freezing the lists; requesting the import of mbox
archives; allowing @mlton.org MX record to propagate through DNS.
Migrating the web content would be quick (once there existed a means
to generate the web content), plus the time to allow *.mlton.org CNAME
records to propagate through DNS.