tools/re2c/re2c.1 - toolchain/yasm - Git at Google

 .ds re \fBre2c\fP
 .ds le \fBlex\fP
 .ds rx regular expression
 .ds lx \fIl\fP-expression
 .TH RE2C 1 "8 April 1994" "Version 0.5"
 \"$Log: re2c.1,v $
 \"Revision 1.1  2002/04/07 22:27:06  peter
 \"Initial revision
 \"
 \"Revision 1.2  1994/04/16  15:50:32  peterr
 \"Fix bug in simple example.
 \"
 \"Revision 1.1  1994/04/08  15:39:09  peterr
 \"Initial revision
 \"
 .SH NAME
 re2c \- convert regular expressions to C/C++

 .SH SYNOPSIS
 \*(re [\fB-esb\fP] \fIname\fP

 .SH DESCRIPTION
 \*(re is a preprocessor that generates C-based recognizers from regular
 expressions.
 The input to \*(re consists of C/C++ source interleaved with
 comments of the form \fC/*!re2c\fP ... \fC*/\fP which contain
 scanner specifications.
 In the output these comments are replaced with code that, when
 executed, will find the next input token and then execute
 some user-supplied token-specific code.

 For example, given the following code

 .in +3
 .nf
 #define NULL            ((char*) 0)
 char *scan(char *p){
 char *q;
 #define YYCTYPE         char
 #define YYCURSOR        p
 #define YYLIMIT         p
 #define YYMARKER        q
 #define YYFILL(n)
 /*!re2c
         [0-9]+          {return YYCURSOR;}
         [\\000-\\377]     {return NULL;}
 */
 }
 .fi
 .in -3

 \*(re will generate

 .in +3
 .nf
 /* Generated by re2c on Sat Apr 16 11:40:58 1994 */
 #line 1 "simple.re"
 #define NULL            ((char*) 0)
 char *scan(char *p){
 char *q;
 #define YYCTYPE         char
 #define YYCURSOR        p
 #define YYLIMIT         p
 #define YYMARKER        q
 #define YYFILL(n)
 {
         YYCTYPE yych;
         unsigned int yyaccept;
         goto yy0;
 yy1:    ++YYCURSOR;
 yy0:
         if((YYLIMIT - YYCURSOR) < 2) YYFILL(2);
         yych = *YYCURSOR;
         if(yych <= '/') goto yy4;
         if(yych >= ':') goto yy4;
 yy2:    yych = *++YYCURSOR;
         goto yy7;
 yy3:
 #line 10
         {return YYCURSOR;}
 yy4:    yych = *++YYCURSOR;
 yy5:
 #line 11
         {return NULL;}
 yy6:    ++YYCURSOR;
         if(YYLIMIT == YYCURSOR) YYFILL(1);
         yych = *YYCURSOR;
 yy7:    if(yych <= '/') goto yy3;
         if(yych <= '9') goto yy6;
         goto yy3;
 }
 #line 12

 }
 .fi
 .in -3

 .SH OPTIONS
 \*(re provides the following options:
 .TP
 \fB-e\fP
 Cross-compile from an ASCII platform to an EBCDIC one.
 .TP
 \fB-s\fP
 Generate nested \fCif\fPs for some \fCswitch\fPes.  Many compilers need this
 assist to generate better code.
 .TP
 \fB-b\fP
 Implies \fB-s\fP.  Use bit vectors as well in the attempt to coax better
 code out of the compiler.  Most useful for specifications with more than a
 few keywords (e.g. for most programming languages).

 .SH "INTERFACE CODE"
 Unlike other scanner generators, \*(re does not generate complete scanners:
 the user must supply some interface code.
 In particular, the user must define the following macros:
 .TP
 \fCYYCHAR\fP
 Type used to hold an input symbol.
 Usually \fCchar\fP or \fCunsigned char\fP.
 .TP
 \fCYYCURSOR\fP
 \*(lx of type \fC*YYCHAR\fP that points to the current input symbol.
 The generated code advances \fCYYCURSOR\fP as symbols are matched.
 On entry, \fCYYCURSOR\fP is assumed to point to the first character of the
 current token.  On exit, \fCYYCURSOR\fP will point to the first character of
 the following token.
 .TP
 \fCYLIMIT\fP
 Expression of type \fC*YYCHAR\fP that marks the end of the buffer
 (\fCYLIMIT[-1]\fP is the last character in the buffer).
 The generated code repeatedly compares \fCYYCURSOR\fP to \fCYLIMIT\fP
 to determine when the buffer needs (re)filling.
 .TP
 \fCYYMARKER\fP
 \*(lx of type \fC*YYCHAR\fP.
 The generated code saves backtracking information in \fCYYMARKER\fP.
 .TP
 \fCYYFILL(\fP\fIn\fP\fC)\fP
 The generated code "calls" \fCYYFILL\fP when the buffer needs
 (re)filling:  at least \fIn\fP additional characters should
 be provided.  \fCYYFILL\fP should adjust \fCYYCURSOR\fP, \fCYYLIMIT\fP and
 \fCYYMARKER\fP as needed.  Note that for typical programming languages
 \fIn\fP will be the length of the longest keyword plus one.

 .SH "SCANNER SPECIFICATIONS"
 Each scanner specification consists of a set of \fIrules\fP and name
 definitions.
 Rules consist of a regular expression along with a block of C/C++ code that
 is to be executed when the associated regular expression is matched.
 Name definitions are of the form
 ``\fIname\fP \fC=\fP \fIregular expression\fP\fC;\fP''.

 .SH "SUMMARY OF RE2C REGULAR EXPRESSIONS"
 .TP
 \fC"foo"\fP
 the literal string \fCfoo\fP.
 ANSI-C escape sequences can be used.
 .TP
 \fC[xyz]\fP
 a "character class"; in this case,
 the \*(rx matches either an '\fCx\fP', a '\fCy\fP', or a '\fCz\fP'.
 .TP
 \fC[abj-oZ]\fP
 a "character class" with a range in it;
 matches an '\fCa\fP', a '\fCb\fP', any letter from '\fCj\fP' through '\fCo\fP',
 or a '\fCZ\fP'.
 .TP
 \fIr\fP\fC\e\fP\fIs\fP
 match any \fIr\fP which isn't an \fIs\fP. \fIr\fP and \fIs\fP must be regular expressions
 which can be expressed as character classes.
 .TP
 \fIr\fP\fC*\fP
 zero or more \fIr\fP's, where \fIr\fP is any regular expression
 .TP
 \fC\fIr\fP\fC+\fP
 one or more \fIr\fP's
 .TP
 \fC\fIr\fP\fC?\fP
 zero or one \fIr\fP's (that is, "an optional \fIr\fP")
 .TP
 name
 the expansion of the "name" definition (see above)
 .TP
 \fC(\fP\fIr\fP\fC)\fP
 an \fIr\fP; parentheses are used to override precedence
 (see below)
 .TP
 \fIrs\fP
 an \fIr\fP followed by an \fIs\fP ("concatenation")
 .TP
 \fIr\fP\fC|\fP\fIs\fP
 either an \fIr\fP or an \fIs\fP
 .TP
 \fIr\fP\fC/\fP\fIs\fP
 an \fIr\fP but only if it is followed by an \fIs\fP. The s is not part of
 the matched text. This type of \*(rx is called "trailing context".
 .LP
 The regular expressions listed above are grouped according to
 precedence, from highest precedence at the top to lowest at the bottom.
 Those grouped together have equal precedence.

 .SH "A LARGER EXAMPLE"
 .LP
 .in +3
 .nf
 #include <stdlib.h>
 #include <stdio.h>
 #include <fcntl.h>
 #include <string.h>

 #define ADDEQ   257
 #define ANDAND  258
 #define ANDEQ   259
 #define ARRAY   260
 #define ASM     261
 #define AUTO    262
 #define BREAK   263
 #define CASE    264
 #define CHAR    265
 #define CONST   266
 #define CONTINUE        267
 #define DECR    268
 #define DEFAULT 269
 #define DEREF   270
 #define DIVEQ   271
 #define DO      272
 #define DOUBLE  273
 #define ELLIPSIS        274
 #define ELSE    275
 #define ENUM    276
 #define EQL     277
 #define EXTERN  278
 #define FCON    279
 #define FLOAT   280
 #define FOR     281
 #define FUNCTION        282
 #define GEQ     283
 #define GOTO    284
 #define ICON    285
 #define ID      286
 #define IF      287
 #define INCR    288
 #define INT     289
 #define LEQ     290
 #define LONG    291
 #define LSHIFT  292
 #define LSHIFTEQ        293
 #define MODEQ   294
 #define MULEQ   295
 #define NEQ     296
 #define OREQ    297
 #define OROR    298
 #define POINTER 299
 #define REGISTER        300
 #define RETURN  301
 #define RSHIFT  302
 #define RSHIFTEQ        303
 #define SCON    304
 #define SHORT   305
 #define SIGNED  306
 #define SIZEOF  307
 #define STATIC  308
 #define STRUCT  309
 #define SUBEQ   310
 #define SWITCH  311
 #define TYPEDEF 312
 #define UNION   313
 #define UNSIGNED        314
 #define VOID    315
 #define VOLATILE        316
 #define WHILE   317
 #define XOREQ   318
 #define EOI     319

 typedef unsigned int uint;
 typedef unsigned char uchar;

 #define BSIZE   8192

 #define YYCTYPE         uchar
 #define YYCURSOR        cursor
 #define YYLIMIT         s->lim
 #define YYMARKER        s->ptr
 #define YYFILL(n)       {cursor = fill(s, cursor);}

 #define RET(i)  {s->cur = cursor; return i;}

 typedef struct Scanner {
     int                 fd;
     uchar               *bot, *tok, *ptr, *cur, *pos, *lim, *top, *eof;
     uint                line;
 } Scanner;

 uchar *fill(Scanner *s, uchar *cursor){
     if(!s->eof){
         uint cnt = s->tok - s->bot;
         if(cnt){
             memcpy(s->bot, s->tok, s->lim - s->tok);
             s->tok = s->bot;
             s->ptr -= cnt;
             cursor -= cnt;
             s->pos -= cnt;
             s->lim -= cnt;
         }
         if((s->top - s->lim) < BSIZE){
             uchar *buf = (uchar*)
                 malloc(((s->lim - s->bot) + BSIZE)*sizeof(uchar));
             memcpy(buf, s->tok, s->lim - s->tok);
             s->tok = buf;
             s->ptr = &buf[s->ptr - s->bot];
             cursor = &buf[cursor - s->bot];
             s->pos = &buf[s->pos - s->bot];
             s->lim = &buf[s->lim - s->bot];
             s->top = &s->lim[BSIZE];
             free(s->bot);
             s->bot = buf;
         }
         if((cnt = read(s->fd, (char*) s->lim, BSIZE)) != BSIZE){
             s->eof = &s->lim[cnt]; *(s->eof)++ = '\\n';
         }
         s->lim += cnt;
     }
     return cursor;
 }

 int scan(Scanner *s){
         uchar *cursor = s->cur;
 std:
         s->tok = cursor;
 /*!re2c
 any     = [\\000-\\377];
 O       = [0-7];
 D       = [0-9];
 L       = [a-zA-Z_];
 H       = [a-fA-F0-9];
 E       = [Ee] [+-]? D+;
 FS      = [fFlL];
 IS      = [uUlL]*;
 ESC     = [\\\\] ([abfnrtv?'"\\\\] | "x" H+ | O+);
 */

 /*!re2c
         "/*"                    { goto comment; }

         "auto"                  { RET(AUTO); }
         "break"                 { RET(BREAK); }
         "case"                  { RET(CASE); }
         "char"                  { RET(CHAR); }
         "const"                 { RET(CONST); }
         "continue"              { RET(CONTINUE); }
         "default"               { RET(DEFAULT); }
         "do"                    { RET(DO); }
         "double"                { RET(DOUBLE); }
         "else"                  { RET(ELSE); }
         "enum"                  { RET(ENUM); }
         "extern"                { RET(EXTERN); }
         "float"                 { RET(FLOAT); }
         "for"                   { RET(FOR); }
         "goto"                  { RET(GOTO); }
         "if"                    { RET(IF); }
         "int"                   { RET(INT); }
         "long"                  { RET(LONG); }
         "register"              { RET(REGISTER); }
         "return"                { RET(RETURN); }
         "short"                 { RET(SHORT); }
         "signed"                { RET(SIGNED); }
         "sizeof"                { RET(SIZEOF); }
         "static"                { RET(STATIC); }
         "struct"                { RET(STRUCT); }
         "switch"                { RET(SWITCH); }
         "typedef"               { RET(TYPEDEF); }
         "union"                 { RET(UNION); }
         "unsigned"              { RET(UNSIGNED); }
         "void"                  { RET(VOID); }
         "volatile"              { RET(VOLATILE); }
         "while"                 { RET(WHILE); }

         L (L|D)*                { RET(ID); }

         ("0" [xX] H+ IS?) | ("0" D+ IS?) | (D+ IS?) |
         (['] (ESC|any\\[\\n\\\\'])* ['])
                                 { RET(ICON); }

         (D+ E FS?) | (D* "." D+ E? FS?) | (D+ "." D* E? FS?)
                                 { RET(FCON); }

         (["] (ESC|any\\[\\n\\\\"])* ["])
                                 { RET(SCON); }

         "..."                   { RET(ELLIPSIS); }
         ">>="                   { RET(RSHIFTEQ); }
         "<<="                   { RET(LSHIFTEQ); }
         "+="                    { RET(ADDEQ); }
         "-="                    { RET(SUBEQ); }
         "*="                    { RET(MULEQ); }
         "/="                    { RET(DIVEQ); }
         "%="                    { RET(MODEQ); }
         "&="                    { RET(ANDEQ); }
         "^="                    { RET(XOREQ); }
         "|="                    { RET(OREQ); }
         ">>"                    { RET(RSHIFT); }
         "<<"                    { RET(LSHIFT); }
         "++"                    { RET(INCR); }
         "--"                    { RET(DECR); }
         "->"                    { RET(DEREF); }
         "&&"                    { RET(ANDAND); }
         "||"                    { RET(OROR); }
         "<="                    { RET(LEQ); }
         ">="                    { RET(GEQ); }
         "=="                    { RET(EQL); }
         "!="                    { RET(NEQ); }
         ";"                     { RET(';'); }
         "{"                     { RET('{'); }
         "}"                     { RET('}'); }
         ","                     { RET(','); }
         ":"                     { RET(':'); }
         "="                     { RET('='); }
         "("                     { RET('('); }
         ")"                     { RET(')'); }
         "["                     { RET('['); }
         "]"                     { RET(']'); }
         "."                     { RET('.'); }
         "&"                     { RET('&'); }
         "!"                     { RET('!'); }
         "~"                     { RET('~'); }
         "-"                     { RET('-'); }
         "+"                     { RET('+'); }
         "*"                     { RET('*'); }
         "/"                     { RET('/'); }
         "%"                     { RET('%'); }
         "<"                     { RET('<'); }
         ">"                     { RET('>'); }
         "^"                     { RET('^'); }
         "|"                     { RET('|'); }
         "?"                     { RET('?'); }


         [ \\t\\v\\f]+           { goto std; }

         "\\n"
             {
                 if(cursor == s->eof) RET(EOI);
                 s->pos = cursor; s->line++;
                 goto std;
             }

         any
             {
                 printf("unexpected character: %c\\n", *s->tok);
                 goto std;
             }
 */

 comment:
 /*!re2c
         "*/"                    { goto std; }
         "\\n"
             {
                 if(cursor == s->eof) RET(EOI);
                 s->tok = s->pos = cursor; s->line++;
                 goto comment;
             }
         any                     { goto comment; }
 */
 }

 main(){
     Scanner in;
     int t;
     memset((char*) &in, 0, sizeof(in));
     in.fd = 0;
     while((t = scan(&in)) != EOI){
 /*
         printf("%d\\t%.*s\\n", t, in.cur - in.tok, in.tok);
         printf("%d\\n", t);
 */
     }
     close(in.fd);
 }
 .fi
 .in -3

 .SH "SEE ALSO"
 .LP
 flex(1), lex(1).

 .SH FEATURES
 .LP
 \*(re does not provide a default action:
 the generated code assumes that the input
 will consist of a sequence of tokens.
 Typically this can be dealt with by adding a rule such as the one for
 unexpected characters in the example above.
 .LP
 The user must arrange for a sentinel token to appear at the end of input
 (and provide a rule for matching it):
 \*(re does not provide an \fC<<EOF>>\fP expression.
 If the source is from a null-byte terminated string, a
 rule matching a null character will suffice.  If the source is from a
 file then the approach taken in the example can be used: pad the input with
 a newline (or some other character that can't appear within another token);
 upon recognizing such a character check to see if it is the sentinel
 and act accordingly.
 .LP
 \*(re does not provide start conditions:  use a separate scanner
 specification for each start condition (as illustrated in the above example).
 .LP
 No [^x].  Use difference instead.
 .SH BUGS
 .LP
 Only fixed length trailing context can be handled.
 .LP
 The maximum value appearing as a parameter \fIn\fP to \fCYYFILL\fP is not
 provided to the generated code (this value is needed for constructing
 the interface code).
 Note that this value is usually relatively small: for
 typical programming languages \fIn\fP will be the length of the longest
 keyword plus one.
 .LP
 Difference only works for character sets.
 .LP
 The \*(re internal algorithms need documentation.

 .SH AUTHOR
 .LP
 Please send bug reports, fixes and feedback to:
 .LP
 .nf
 Peter Bumbulis
 Computer Systems Group
 University of Waterloo
 Waterloo, Ontario
 N2L 3G1
 Internet:  [email protected]
 .fi
	.ds re \fBre2c\fP
	.ds le \fBlex\fP
	.ds rx regular expression
	.ds lx \fIl\fP-expression
	.TH RE2C 1 "8 April 1994" "Version 0.5"
	\"$Log: re2c.1,v $
	\"Revision 1.1 2002/04/07 22:27:06 peter
	\"Initial revision
	\"
	\"Revision 1.2 1994/04/16 15:50:32 peterr
	\"Fix bug in simple example.
	\"
	\"Revision 1.1 1994/04/08 15:39:09 peterr
	\"Initial revision
	\"
	.SH NAME
	re2c \- convert regular expressions to C/C++

	.SH SYNOPSIS
	\*(re [\fB-esb\fP] \fIname\fP

	.SH DESCRIPTION
	\*(re is a preprocessor that generates C-based recognizers from regular
	expressions.
	The input to \*(re consists of C/C++ source interleaved with
	comments of the form \fC/!re2c\fP ... \fC/\fP which contain
	scanner specifications.
	In the output these comments are replaced with code that, when
	executed, will find the next input token and then execute
	some user-supplied token-specific code.

	For example, given the following code

	.in +3
	.nf
	#define NULL ((char*) 0)
	char scan(char p){
	char *q;
	#define YYCTYPE char
	#define YYCURSOR p
	#define YYLIMIT p
	#define YYMARKER q
	#define YYFILL(n)
	/*!re2c
	[0-9]+ {return YYCURSOR;}
	[\\000-\\377] {return NULL;}
	*/
	}
	.fi
	.in -3

	\*(re will generate

	.in +3
	.nf
	/* Generated by re2c on Sat Apr 16 11:40:58 1994 */
	#line 1 "simple.re"
	#define NULL ((char*) 0)
	char scan(char p){
	char *q;
	#define YYCTYPE char
	#define YYCURSOR p
	#define YYLIMIT p
	#define YYMARKER q
	#define YYFILL(n)
	{
	YYCTYPE yych;
	unsigned int yyaccept;
	goto yy0;
	yy1: ++YYCURSOR;
	yy0:
	if((YYLIMIT - YYCURSOR) < 2) YYFILL(2);
	yych = *YYCURSOR;
	if(yych <= '/') goto yy4;
	if(yych >= ':') goto yy4;
	yy2: yych = *++YYCURSOR;
	goto yy7;
	yy3:
	#line 10
	{return YYCURSOR;}
	yy4: yych = *++YYCURSOR;
	yy5:
	#line 11
	{return NULL;}
	yy6: ++YYCURSOR;
	if(YYLIMIT == YYCURSOR) YYFILL(1);
	yych = *YYCURSOR;
	yy7: if(yych <= '/') goto yy3;
	if(yych <= '9') goto yy6;
	goto yy3;
	}
	#line 12

	}
	.fi
	.in -3

	.SH OPTIONS
	\*(re provides the following options:
	.TP
	\fB-e\fP
	Cross-compile from an ASCII platform to an EBCDIC one.
	.TP
	\fB-s\fP
	Generate nested \fCif\fPs for some \fCswitch\fPes. Many compilers need this
	assist to generate better code.
	.TP
	\fB-b\fP
	Implies \fB-s\fP. Use bit vectors as well in the attempt to coax better
	code out of the compiler. Most useful for specifications with more than a
	few keywords (e.g. for most programming languages).

	.SH "INTERFACE CODE"
	Unlike other scanner generators, \*(re does not generate complete scanners:
	the user must supply some interface code.
	In particular, the user must define the following macros:
	.TP
	\fCYYCHAR\fP
	Type used to hold an input symbol.
	Usually \fCchar\fP or \fCunsigned char\fP.
	.TP
	\fCYYCURSOR\fP
	\(lx of type \fCYYCHAR\fP that points to the current input symbol.
	The generated code advances \fCYYCURSOR\fP as symbols are matched.
	On entry, \fCYYCURSOR\fP is assumed to point to the first character of the
	current token. On exit, \fCYYCURSOR\fP will point to the first character of
	the following token.
	.TP
	\fCYLIMIT\fP
	Expression of type \fC*YYCHAR\fP that marks the end of the buffer
	(\fCYLIMIT[-1]\fP is the last character in the buffer).
	The generated code repeatedly compares \fCYYCURSOR\fP to \fCYLIMIT\fP
	to determine when the buffer needs (re)filling.
	.TP
	\fCYYMARKER\fP
	\(lx of type \fCYYCHAR\fP.
	The generated code saves backtracking information in \fCYYMARKER\fP.
	.TP
	\fCYYFILL(\fP\fIn\fP\fC)\fP
	The generated code "calls" \fCYYFILL\fP when the buffer needs
	(re)filling: at least \fIn\fP additional characters should
	be provided. \fCYYFILL\fP should adjust \fCYYCURSOR\fP, \fCYYLIMIT\fP and
	\fCYYMARKER\fP as needed. Note that for typical programming languages
	\fIn\fP will be the length of the longest keyword plus one.

	.SH "SCANNER SPECIFICATIONS"
	Each scanner specification consists of a set of \fIrules\fP and name
	definitions.
	Rules consist of a regular expression along with a block of C/C++ code that
	is to be executed when the associated regular expression is matched.
	Name definitions are of the form
	``\fIname\fP \fC=\fP \fIregular expression\fP\fC;\fP''.

	.SH "SUMMARY OF RE2C REGULAR EXPRESSIONS"
	.TP
	\fC"foo"\fP
	the literal string \fCfoo\fP.
	ANSI-C escape sequences can be used.
	.TP
	\fC[xyz]\fP
	a "character class"; in this case,
	the \*(rx matches either an '\fCx\fP', a '\fCy\fP', or a '\fCz\fP'.
	.TP
	\fC[abj-oZ]\fP
	a "character class" with a range in it;
	matches an '\fCa\fP', a '\fCb\fP', any letter from '\fCj\fP' through '\fCo\fP',
	or a '\fCZ\fP'.
	.TP
	\fIr\fP\fC\e\fP\fIs\fP
	match any \fIr\fP which isn't an \fIs\fP. \fIr\fP and \fIs\fP must be regular expressions
	which can be expressed as character classes.
	.TP
	\fIr\fP\fC*\fP
	zero or more \fIr\fP's, where \fIr\fP is any regular expression
	.TP
	\fC\fIr\fP\fC+\fP
	one or more \fIr\fP's
	.TP
	\fC\fIr\fP\fC?\fP
	zero or one \fIr\fP's (that is, "an optional \fIr\fP")
	.TP
	name
	the expansion of the "name" definition (see above)
	.TP
	\fC(\fP\fIr\fP\fC)\fP
	an \fIr\fP; parentheses are used to override precedence
	(see below)
	.TP
	\fIrs\fP
	an \fIr\fP followed by an \fIs\fP ("concatenation")
	.TP
	\fIr\fP\fC\|\fP\fIs\fP
	either an \fIr\fP or an \fIs\fP
	.TP
	\fIr\fP\fC/\fP\fIs\fP
	an \fIr\fP but only if it is followed by an \fIs\fP. The s is not part of
	the matched text. This type of \*(rx is called "trailing context".
	.LP
	The regular expressions listed above are grouped according to
	precedence, from highest precedence at the top to lowest at the bottom.
	Those grouped together have equal precedence.

	.SH "A LARGER EXAMPLE"
	.LP
	.in +3
	.nf
	#include <stdlib.h>
	#include <stdio.h>
	#include <fcntl.h>
	#include <string.h>

	#define ADDEQ 257
	#define ANDAND 258
	#define ANDEQ 259
	#define ARRAY 260
	#define ASM 261
	#define AUTO 262
	#define BREAK 263
	#define CASE 264
	#define CHAR 265
	#define CONST 266
	#define CONTINUE 267
	#define DECR 268
	#define DEFAULT 269
	#define DEREF 270
	#define DIVEQ 271
	#define DO 272
	#define DOUBLE 273
	#define ELLIPSIS 274
	#define ELSE 275
	#define ENUM 276
	#define EQL 277
	#define EXTERN 278
	#define FCON 279
	#define FLOAT 280
	#define FOR 281
	#define FUNCTION 282
	#define GEQ 283
	#define GOTO 284
	#define ICON 285
	#define ID 286
	#define IF 287
	#define INCR 288
	#define INT 289
	#define LEQ 290
	#define LONG 291
	#define LSHIFT 292
	#define LSHIFTEQ 293
	#define MODEQ 294
	#define MULEQ 295
	#define NEQ 296
	#define OREQ 297
	#define OROR 298
	#define POINTER 299
	#define REGISTER 300
	#define RETURN 301
	#define RSHIFT 302
	#define RSHIFTEQ 303
	#define SCON 304
	#define SHORT 305
	#define SIGNED 306
	#define SIZEOF 307
	#define STATIC 308
	#define STRUCT 309
	#define SUBEQ 310
	#define SWITCH 311
	#define TYPEDEF 312
	#define UNION 313
	#define UNSIGNED 314
	#define VOID 315
	#define VOLATILE 316
	#define WHILE 317
	#define XOREQ 318
	#define EOI 319

	typedef unsigned int uint;
	typedef unsigned char uchar;

	#define BSIZE 8192

	#define YYCTYPE uchar
	#define YYCURSOR cursor
	#define YYLIMIT s->lim
	#define YYMARKER s->ptr
	#define YYFILL(n) {cursor = fill(s, cursor);}

	#define RET(i) {s->cur = cursor; return i;}

	typedef struct Scanner {
	int fd;
	uchar bot, tok, ptr, cur, pos, lim, top, eof;
	uint line;
	} Scanner;

	uchar fill(Scanner s, uchar *cursor){
	if(!s->eof){
	uint cnt = s->tok - s->bot;
	if(cnt){
	memcpy(s->bot, s->tok, s->lim - s->tok);
	s->tok = s->bot;
	s->ptr -= cnt;
	cursor -= cnt;
	s->pos -= cnt;
	s->lim -= cnt;
	}
	if((s->top - s->lim) < BSIZE){
	uchar buf = (uchar)
	malloc(((s->lim - s->bot) + BSIZE)*sizeof(uchar));
	memcpy(buf, s->tok, s->lim - s->tok);
	s->tok = buf;
	s->ptr = &buf[s->ptr - s->bot];
	cursor = &buf[cursor - s->bot];
	s->pos = &buf[s->pos - s->bot];
	s->lim = &buf[s->lim - s->bot];
	s->top = &s->lim[BSIZE];
	free(s->bot);
	s->bot = buf;
	}
	if((cnt = read(s->fd, (char*) s->lim, BSIZE)) != BSIZE){
	s->eof = &s->lim[cnt]; *(s->eof)++ = '\\n';
	}
	s->lim += cnt;
	}
	return cursor;
	}

	int scan(Scanner *s){
	uchar *cursor = s->cur;
	std:
	s->tok = cursor;
	/*!re2c
	any = [\\000-\\377];
	O = [0-7];
	D = [0-9];
	L = [a-zA-Z_];
	H = [a-fA-F0-9];
	E = [Ee] [+-]? D+;
	FS = [fFlL];
	IS = [uUlL]*;
	ESC = [\\\\] ([abfnrtv?'"\\\\] \| "x" H+ \| O+);
	*/

	/*!re2c
	"/*" { goto comment; }

	"auto" { RET(AUTO); }
	"break" { RET(BREAK); }
	"case" { RET(CASE); }
	"char" { RET(CHAR); }
	"const" { RET(CONST); }
	"continue" { RET(CONTINUE); }
	"default" { RET(DEFAULT); }
	"do" { RET(DO); }
	"double" { RET(DOUBLE); }
	"else" { RET(ELSE); }
	"enum" { RET(ENUM); }
	"extern" { RET(EXTERN); }
	"float" { RET(FLOAT); }
	"for" { RET(FOR); }
	"goto" { RET(GOTO); }
	"if" { RET(IF); }
	"int" { RET(INT); }
	"long" { RET(LONG); }
	"register" { RET(REGISTER); }
	"return" { RET(RETURN); }
	"short" { RET(SHORT); }
	"signed" { RET(SIGNED); }
	"sizeof" { RET(SIZEOF); }
	"static" { RET(STATIC); }
	"struct" { RET(STRUCT); }
	"switch" { RET(SWITCH); }
	"typedef" { RET(TYPEDEF); }
	"union" { RET(UNION); }
	"unsigned" { RET(UNSIGNED); }
	"void" { RET(VOID); }
	"volatile" { RET(VOLATILE); }
	"while" { RET(WHILE); }

	L (L\|D)* { RET(ID); }

	("0" [xX] H+ IS?) \| ("0" D+ IS?) \| (D+ IS?) \|
	(['] (ESC\|any\\[\\n\\\\'])* ['])
	{ RET(ICON); }

	(D+ E FS?) \| (D* "." D+ E? FS?) \| (D+ "." D* E? FS?)
	{ RET(FCON); }

	(["] (ESC\|any\\[\\n\\\\"])* ["])
	{ RET(SCON); }

	"..." { RET(ELLIPSIS); }
	">>=" { RET(RSHIFTEQ); }
	"<<=" { RET(LSHIFTEQ); }
	"+=" { RET(ADDEQ); }
	"-=" { RET(SUBEQ); }
	"*=" { RET(MULEQ); }
	"/=" { RET(DIVEQ); }
	"%=" { RET(MODEQ); }
	"&=" { RET(ANDEQ); }
	"^=" { RET(XOREQ); }
	"\|=" { RET(OREQ); }
	">>" { RET(RSHIFT); }
	"<<" { RET(LSHIFT); }
	"++" { RET(INCR); }
	"--" { RET(DECR); }
	"->" { RET(DEREF); }
	"&&" { RET(ANDAND); }
	"\|\|" { RET(OROR); }
	"<=" { RET(LEQ); }
	">=" { RET(GEQ); }
	"==" { RET(EQL); }
	"!=" { RET(NEQ); }
	";" { RET(';'); }
	"{" { RET('{'); }
	"}" { RET('}'); }
	"," { RET(','); }
	":" { RET(':'); }
	"=" { RET('='); }
	"(" { RET('('); }
	")" { RET(')'); }
	"[" { RET('['); }
	"]" { RET(']'); }
	"." { RET('.'); }
	"&" { RET('&'); }
	"!" { RET('!'); }
	"~" { RET('~'); }
	"-" { RET('-'); }
	"+" { RET('+'); }
	"" { RET(''); }
	"/" { RET('/'); }
	"%" { RET('%'); }
	"<" { RET('<'); }
	">" { RET('>'); }
	"^" { RET('^'); }
	"\|" { RET('\|'); }
	"?" { RET('?'); }


	[ \\t\\v\\f]+ { goto std; }

	"\\n"
	{
	if(cursor == s->eof) RET(EOI);
	s->pos = cursor; s->line++;
	goto std;
	}

	any
	{
	printf("unexpected character: %c\\n", *s->tok);
	goto std;
	}
	*/

	comment:
	/*!re2c
	"*/" { goto std; }
	"\\n"
	{
	if(cursor == s->eof) RET(EOI);
	s->tok = s->pos = cursor; s->line++;
	goto comment;
	}
	any { goto comment; }
	*/
	}

	main(){
	Scanner in;
	int t;
	memset((char*) &in, 0, sizeof(in));
	in.fd = 0;
	while((t = scan(&in)) != EOI){
	/*
	printf("%d\\t%.*s\\n", t, in.cur - in.tok, in.tok);
	printf("%d\\n", t);
	*/
	}
	close(in.fd);
	}
	.fi
	.in -3

	.SH "SEE ALSO"
	.LP
	flex(1), lex(1).

	.SH FEATURES
	.LP
	\*(re does not provide a default action:
	the generated code assumes that the input
	will consist of a sequence of tokens.
	Typically this can be dealt with by adding a rule such as the one for
	unexpected characters in the example above.
	.LP
	The user must arrange for a sentinel token to appear at the end of input
	(and provide a rule for matching it):
	\*(re does not provide an \fC<<EOF>>\fP expression.
	If the source is from a null-byte terminated string, a
	rule matching a null character will suffice. If the source is from a
	file then the approach taken in the example can be used: pad the input with
	a newline (or some other character that can't appear within another token);
	upon recognizing such a character check to see if it is the sentinel
	and act accordingly.
	.LP
	\*(re does not provide start conditions: use a separate scanner
	specification for each start condition (as illustrated in the above example).
	.LP
	No [^x]. Use difference instead.
	.SH BUGS
	.LP
	Only fixed length trailing context can be handled.
	.LP
	The maximum value appearing as a parameter \fIn\fP to \fCYYFILL\fP is not
	provided to the generated code (this value is needed for constructing
	the interface code).
	Note that this value is usually relatively small: for
	typical programming languages \fIn\fP will be the length of the longest
	keyword plus one.
	.LP
	Difference only works for character sets.
	.LP
	The \*(re internal algorithms need documentation.

	.SH AUTHOR
	.LP
	Please send bug reports, fixes and feedback to:
	.LP
	.nf
	Peter Bumbulis
	Computer Systems Group
	University of Waterloo
	Waterloo, Ontario
	N2L 3G1
	Internet: [email protected]
	.fi