[MLton-commit] r7500

[Matthew Fluet]

Fix bug in comment-handling in lexer for mlyacc's input language.

[Submitted by Michael Norrish.]

In particular, if a comment was successfully lexed, it returned a
BOGUS_VALUE token, which then messed up the higher-level parsing.
This was due to the use of the idiom

  continue() before YYBEGIN SOMESTATE

making the return value of the continue the return value for the lexer
at that point.

This bug prevented the use of comments in code sections entirely.

It was somewhat masked in the prefix section by the fact that most
supposed comments there were never lexed as such anyway.  This could
be observed by a file that included %% in a comment in the prefix
section.


----------------------------------------------------------------------

U   mlton/trunk/mlyacc/src/yacc.lex

----------------------------------------------------------------------

Modified: mlton/trunk/mlyacc/src/yacc.lex
===================================================================
--- mlton/trunk/mlyacc/src/yacc.lex	2011-01-17 08:34:17 UTC (rev 7499)
+++ mlton/trunk/mlyacc/src/yacc.lex	2011-01-17 20:56:39 UTC (rev 7500)
@@ -75,11 +75,11 @@
 qualid ={id}".";
 %%
 <INITIAL>"(*"   => (Add yytext; YYBEGIN COMMENT; commentLevel := 1;
-                    continue() before YYBEGIN INITIAL);
+                    continue(); YYBEGIN INITIAL; continue());
 <A>"(*"         => (YYBEGIN EMPTYCOMMENT; commentLevel := 1; continue());
 <CODE>"(*"      => (Add yytext; YYBEGIN COMMENT; commentLevel := 1;
-                    continue() before YYBEGIN CODE);
-<INITIAL>[^%\013\n]+ => (Add yytext; continue());
+                    continue(); YYBEGIN CODE; continue());
+<INITIAL>[^(%\013\n]+ => (Add yytext; continue());
 <INITIAL>"%%"    => (YYBEGIN A; HEADER (concat (rev (!text)),pos yypos,pos yypos));
 <INITIAL,CODE,COMMENT,F,EMPTYCOMMENT>{eol}  => (Add yytext; incLineNum yypos; continue());
 <INITIAL>.       => (Add yytext; continue());