@q Copyright (c) 1990 The Regents of the University of California. @> @q All rights reserved. @> @q This code is derived from software contributed to Berkeley by @> @q Vern Paxson. @> @q The United States Government has rights in this work pursuant @> @q to contract no. DE-AC03-76SF00098 between the United States @> @q Department of Energy and the University of California. @> @q This file is part of SPLinT. @> @q Redistribution and use in source and binary forms, with or without @> @q modification, are permitted provided that the following conditions @> @q are met: @> @q 1. Redistributions of source code must retain the above copyright @> @q notice, this list of conditions and the following disclaimer. @> @q 2. Redistributions in binary form must reproduce the above copyright @> @q notice, this list of conditions and the following disclaimer in the @> @q documentation and/or other materials provided with the distribution. @> @q Neither the name of the University nor the names of its contributors @> @q may be used to endorse or promote products derived from this software @> @q without specific prior written permission. @> @q THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR @> @q IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED @> @q WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR @> @q PURPOSE. @> @**The \ifbootstrapmode flex \else\flex\ \fi parser stack. \ifbootstrapmode % this is a bootstrap run to generate the tables \input limbo.sty \def\optimization{5} \input yy.sty \modebootstrap \else % otherwise set up the pretty printing of tokens \let\hostparsernamespace\flexnamespace \fi The scanner generator, \flex, uses \bison\ to produce a parser for its input language. Its lexer is output by \flex\ itself so both are reused to generate the parser and the scanner for pretty printing \flex\ input. This task is made somewhat complicated by the dependence of the \flex\ input scanner on the correctly placed whitespace\footnote{For example, each regular expression definition in section~1 must start at the beginning of the line.}, as well as the reliance of the said scanner on rather involved state switching. Therefore, making subparsers for different fragments of \flex\ input involves not only choosing an appropriate subset of grammar rules to correctly process the grammatic constructs but also setting up the correct lexer states. The first subparser is designed to process a complete \flex\ file. This parser is not currently part of any parser stack and is only used for testing. This is the only parser that does not rely on any custom adjustments to the lexer state to operate correctly. @(fip.yy@>= @G %{@> @ @=%} @> @ @= %union {@> @=} %{@> @ @=%} @> @@= %% @> @ @= %% @g @ The selection of options for \bison\ parsers suitable for \splint\ have been discussed \ifbootstrapmode\else\locallink{bison.options}earlier \endlink\fi so we list them here without further comments. @= @G %token-table %debug %start goal @g @ A parser for section~1 (definitions and declarations). This parser requires a custom lexer, as discussed above, to properly set up the state. Short of this, the lexer may produce the wrong kind of tokens or even generate an error. @(ddp.yy@>= @G %{@> @ @=%} @> @ @= %union {@> @=} %{@> @ @=%} @> @@= %% @> @ @= @> @ @= %% @ A parser for section~2 (rules and actions). This subparser must also use a custom set up for its lexer as discussed above. @(rap.yy@>= @G %{@> @ @=%} @> @ @= %union {@> @=} %{@> @ @=%} @> @@= %% @> @ @= @> @ @= %% @g @ A parser for just the regular expression syntax. A custom lexer initialization must precede the use of this parser, as well. @(rep.yy@>= @G %{@> @ @=%} @> @ @= %union {@> @=} %{@> @ @=%} @> @@= %% @> @ @= @> @ @= %% @*1 Token and state declarations for the \eatone{flex}\flex\ input scanner. Needless to say, the original grammar used by \flex\ was not designed with pretty printing in mind (and why would it be?). Instead, efficiency was the goal which resulted in a number of lexical constructs being processed `on the fly', as the lexer encounters them. Such syntax fragments never reach the parser, and would not have a chance to be displayed by our routines, unless some grammar extensions and alterations were introduced. To make the pretty printing possible, a number of new tokens have been introduced below that are later used in a few altered or entirely new grammar productions. @= @G %token CHAR NUMBER SECTEND SCDECL XSCDECL NAME PREVCCL EOF_OP %token OPTION_OP OPT_OUTFILE OPT_PREFIX OPT_YYCLASS OPT_HEADER OPT_EXTRA_TYPE %token OPT_TABLES %token CCE_ALNUM CCE_ALPHA CCE_BLANK CCE_CNTRL CCE_DIGIT CCE_GRAPH %token CCE_LOWER CCE_PRINT CCE_PUNCT CCE_SPACE CCE_UPPER CCE_XDIGIT %token CCE_NEG_ALNUM CCE_NEG_ALPHA CCE_NEG_BLANK CCE_NEG_CNTRL CCE_NEG_DIGIT CCE_NEG_GRAPH %token CCE_NEG_LOWER CCE_NEG_PRINT CCE_NEG_PUNCT CCE_NEG_SPACE CCE_NEG_UPPER CCE_NEG_XDIGIT %left CCL_OP_DIFF CCL_OP_UNION @ We introduce an additional option type to capture all the non-parametric options used by the \flex\ lexer. The original lexer processes these options at the point of recognition, while the typesetting parser needs to be aware of them. @= @G %token TOP_OP POINTER_OP ARRAY_OP DEF_OP RE_DEF OPT_OTHER OPT_DEPRECATED @ {\em \POSIX\ and \.{AT\&T} \lex\ place the precedence of the repeat operator, \.{\{\}}, below that of concatenation. Thus, \.{ab\{3\}} is\/ \.{ababab}. Most other \POSIX\ utilities use an {\rm Extended Regular Expression (ERE)} precedence that has the repeat operator higher than concatenation. This causes \.{ab\{3\}} to yield\/ \.{abbb}. In order to support the \POSIX\ and \.{AT\&T} precedence and the \flex\ precedence we define two token sets for the begin and end tokens of the repeat operator, \prodstyle{BEGIN_REPEAT_POSIX} and\/ \prodstyle{END_REPEAT_POSIX}. The lexical scanner chooses which tokens to return based on whether {\let\it\itbold\prodstyle{posix\_compat} or \prodstyle{lex\_compat} are specified. Specifying either \prodstyle{posix\_compat} or \prodstyle{lex\_compat}} will cause \flex\ to parse scanner files as per the \.{AT\&T} and \POSIX-mandated behavior.} @= @G %token BEGIN_REPEAT_POSIX END_REPEAT_POSIX BEGIN_REPEAT_FLEX END_REPEAT_FLEX @*1 The grammar for \eatone{flex}\flex\ input. The original grammar has been carefully split into sections to facilitate the assembly of various subparsers in the \flex's stack. Neither the \flex\ parser nor its scanner are part of the bootstrap procedure which simplifies both the input file organization, as well as the macro design. Some amount of preprocessing is still necessary, however, to extract the state names from the lexer file (see \ifbootstrapmode\else\locallink{state.grabbing}above \endlink\fi for the explanation). We can nevertheless get away with an empty \Cee\ preamble. @= @ @= @G @t}\vb{\inline\flatten}{@> goal: initlex sect1 sect1end sect2 initforrule {@> @ @=} ; sect1end: SECTEND {@> @ @=} ; initlex: {@> @=} ; @g @ @= @[TeX_( "/yy0{/the/yy(2)/the/yy(4)}" );@]@; @ @= @@; @@; @ @= @G @t}\vb{\inline\flatten}{@> goal: sect1 {@> @ @=} ; @ @= @[TeX_( "/table/expandafter{/the/yy(1)}" );@]@; @ @= @G sect1: sect1 startconddecl namelist1 {@> @ @=} | sect1 options {@> @ @=} | {@> @ @=} | error {@> @ @=} ; startconddecl: SCDECL {@> @ @=} | XSCDECL {@> @ @=} ; namelist1: namelist1 NAME {@> @ @=} | NAME {@> @ @=} | error {@> @ @=} ; @g @ @= @[TeX_( "/yy0{/the/yy(1)/nx/flscondecl/the/yy(2){/the/yy(3)}}" );@]@; @ @= @[TeX_( "/yy0{/the/yy(1)/the/yy(2)}" );@]@; @ @= @[TeX_( "/yy0{}" );@]@; @ @= @[TeX_( "/yyerror" );@]@; @ @= @[TeX_( "/yy0{{s}/the/yy(1)}" );@]@; @ @= @[TeX_( "/yy0{{x}/the/yy(1)}" );@]@; @ @= @[TeX_( "/yy0{/the/yy(1)/nx/flnamesep{}{}/nx/flname/the/yy(2)}" );@]@; @ @= @[TeX_( "/yy0{/nx/flname/the/yy(1)}" );@]@; @ @= @[TeX_( "/yyerror" );@]@; @ @= @G options: OPTION_OP optionlist {@> @ @=} | POINTER_OP {@> @ @=} | ARRAY_OP {@> @ @=} | TOP_OP '\n' {@> @ @=} | DEF_OP RE_DEF {@> @ @=} | OPT_DEPRECATED {@> @ @=} ; @t}\vb{\inline\flatten}{@> optionlist: optionlist option {@> @ @=} | {@> @ @=} ; @t}\vb{\resetf}{@> option: OPT_OUTFILE '=' NAME {@> @ @=} | OPT_EXTRA_TYPE '=' NAME {@> @ @=} | OPT_PREFIX '=' NAME {@> @ @=} | OPT_YYCLASS '=' NAME {@> @ @=} | OPT_HEADER '=' NAME {@> @ @=} | OPT_TABLES '=' NAME {@> @ @=} | OPT_OTHER {@> @ @=} ; @g @ @= @[TeX_( "/yy0{/nx/floptions{/the/yy(2)}}" );@]@; @ @= @[TeX_( "/yy0{/nx/flptropt/the/yy(1)}" );@]@; @ @= @[TeX_( "/yy0{/nx/flarrayopt/the/yy(1)}" );@]@; @ @= @[TeX_( "/yy0{/nx/fltopopt/the/yy(1)/the/yy(2)}" );@]@; @ @= @[TeX_( "/yy0{/nx/flredef/the/yy(1)/the/yy(2)}" );@]@; @ @= @[TeX_( "/yy0{/the/yy(1)/the/yy(2)}" );@]@; @ @= @[TeX_( "/yy0{}" );@]@; @ @= @[TeX_( "/yy0{/nx/flopt{file}/the/yy(3)}" );@]@; @ @= @[TeX_( "/yy0{/nx/flopt{xtype}/the/yy(3)}" );@]@; @ @= @[TeX_( "/yy0{/nx/flopt{prefix}/the/yy(3)}" );@]@; @ @= @[TeX_( "/yy0{/nx/flopt{yyclass}/the/yy(3)}" );@]@; @ @= @[TeX_( "/yy0{/nx/flopt{header}/the/yy(3)}" );@]@; @ @= @[TeX_( "/yy0{/nx/flopt{tables}/the/yy(3)}" );@]@; @ @= @[TeX_( "/yy0{/nx/flopt{other}/the/yy(1)}" );@]@; @ @= @[TeX_( "/yy0{/nx/flopt{deprecated}/the/yy(1)}" );@]@; @ @= @G goal: sect2 {@> @ @=} ; @g @ @= @[TeX_( "/table/yy(1)" );@]@; @ This portion of the grammar was changed to make it possible to read the action code. @= @G sect2: sect2 scon initforrule flexrule '\n' '\n' {@> @ @=} | sect2 scon '{' sect2 '}' {@> @ @=} | {@> @ @=} | sect2 '\n' {@> @ @=} ; @t}\vb{\inline\flatten}{@> initforrule: {@> @[TeX_( "/flin@@ruletrue/yylexnext" );@] @=} ; @g @ @= @[TeX_( "/ifflcontinued@@action" );@]@; @[TeX_( " /toksb{/flactionc}" );@]@; @[TeX_( "/else" );@]@; @[TeX_( " /toksb{/flaction}" );@]@; @[TeX_( "/fi" );@]@; @[TeX_( "/toksa/expandafter{/astformat@@flaction}" );@]@; /* capture the formatting action */ @[TeX_( "/yypushx{/the/yy(1)/the/toksb{/the/yy(2)}{/the/yy(4)}/the/yy(5)/the/yy(6){/the/toksa}}/on/astarray" );@]@; @[TeX_( "/yy0{/astarraylastcs}" );@]@; @[TeX_( "/let/astformat@@flaction/empty" );@]@; /* reset the format */ @ @= @[TeX_( "/yy0{/the/yy(1)/nx/flactiongroup{/the/yy(2)}/the/yy(3){/the/yy(4)}/the/yy(5)}" );@]@; @ @= @[TeX_( "/yy0{}" );@]@; @ @= @[TeX_( "/yy0{/the/yy(1)/nx/flbareaction/the/yy(2)}" );@]@; @ @= @G @t}\vb{\inline\flatten}{@> scon_stk_ptr: {@> @=} ; @t}\vb{\resetf}{@> scon: '<' scon_stk_ptr namelist2 '>' {@> @ @=} | '<' '*' '>' {@> @ @=} | {@> @ @=} ; namelist2: namelist2 ',' sconname {@> @ @=} | sconname {@> @ @=} | error {@> @ @=} ; @t}\vb{\inline\flatten}{@> sconname: NAME {@> @ @=} ; @ @= @[TeX_( "/yy0{/nx/flsconlist{/the/yy(1)}{/the/yy(3)}{/the/yy(4)}}" );@]@; @ @= @[TeX_( "/yy0{/nx/flsconuniv/the/yy(3)}" );@]@; @ @= @[TeX_( "/yy0{}" );@]@; @ @= @[TeX_( "/yy0{/the/yy(1)/nx/flnamesep/the/yy(2)/the/yy(3)}" );@]@; @ @= @@; @ @= @[TeX_( "/yyerror" );@]@; @ @= @[TeX_( "/yy0{/nx/flname/the/yy(1)}" );@]@; @ @= @@; @ @= @G goal: flexrule {@> @ @=} ; @g @ The parsed regular expression is output in the \.{\\table} register. It is important to ensure that whenever this parser is used inside another parser that uses \.{\\table} for output, the changes to this register stay local. The \.{\\frexproc} macro in \.{yyunion.sty} ensures that all the changes are local to the parsing macro. @= @[TeX_( "/table/yy(1)" );@]@; @ @= @G flexrule: '^' rule {@> @ @=} | rule {@> @ @=} | EOF_OP {@> @ @=} | error {@> @ @=} ; @ @= @[TeX_( "/toksa/expandafter{/astformat@@flrule}" );@]@; @[TeX_( "/let/astformat@@flrule/empty" );@]@; @[TeX_( "/yy0{/nx/flbolrule{/the/yy(2)}{/the/toksa}}" );@]@; @ @= @[TeX_( "/toksa/expandafter{/astformat@@flrule}" );@]@; @[TeX_( "/let/astformat@@flrule/empty" );@]@; @[TeX_( "/yy0{/nx/flrule{/the/yy(1)}{/the/toksa}}" );@]@; @ @= @[TeX_( "/yy0{/nx/fleof/the/yy(1)}" );@]@; @ @= @[TeX_( "/yyerror" );@]@; @ @= @G rule: re2 re {@> @ @=} | re2 re '$' {@> @ @=} | re '$' {@> @ @=} | re {@> @ @=} ; re: re '|' series {@> @ @=} | series {@> @ @=} ; @t}\vb{\inline\flatten}{@> re2: re '/' {@> @ @=} ; @g @ @= @[TeX_( "/getsecond{/yy(1)}/to/toksa/getthird{/yy(1)}/to/toksb" );@]@; @[TeX_( "/yy0{/nx/flretrail{/the/toksa}{/the/toksb}{/the/yy(2)}}" );@]@; @ @= @[TeX_( "/yyerror" );@]@; @ @= @[TeX_( "/yy0{/nx/flreateol{/the/yy(1)}/the/yy(2)}" );@]@; @ @= @@; @ @= @[TeX_( "/yy0{/the/yy(1)/nx/flor/the/yy(2)/the/yy(3)}" );@]@; @ @= @@; @ @= @[TeX_( "/yy0{/nx/fltrail{/the/yy(1)}{/the/yy(2)}}" );@]@; @ @= @G series: series singleton {@> @ @=} | singleton {@> @ @=} | series BEGIN_REPEAT_POSIX NUMBER ',' NUMBER END_REPEAT_POSIX {@> @ @=} | series BEGIN_REPEAT_POSIX NUMBER ',' END_REPEAT_POSIX {@> @ @=} | series BEGIN_REPEAT_POSIX NUMBER END_REPEAT_POSIX {@> @ @=} ; @g @ @= @[TeX_( "/yy0{/the/yy(1)/the/yy(2)}" );@]@; @ @= @@; @ @= @@; @ @= @@; @ @= @@; @ @= @G singleton: singleton '*' {@> @ @=} | singleton '+' {@> @ @=} | singleton '?' {@> @ @=} | singleton BEGIN_REPEAT_FLEX NUMBER ',' NUMBER END_REPEAT_FLEX {@> @ @=} | singleton BEGIN_REPEAT_FLEX NUMBER ',' END_REPEAT_FLEX {@> @ @=} | singleton BEGIN_REPEAT_FLEX NUMBER END_REPEAT_FLEX {@> @ @=} | '.' {@> @ @=} | fullccl {@> @ @=} | PREVCCL {@> @ @=} | '"' string '"' {@> @ @=} | '(' re ')' {@> @ @=} | CHAR {@> @ @=} ; @g @ @= @[TeX_( "/yy0{/nx/flrepeat{/the/yy(1)}}" );@]@; @ @= @[TeX_( "/yy0{/nx/flrepeatstrict{/the/yy(1)}}" );@]@; @ @= @[TeX_( "/yy0{/nx/flrepeatonce{/the/yy(1)}}" );@]@; @ @= @[TeX_( "/yy0{/nx/flrepeatnm{/the/yy(1)}{/the/yy(3)}{/the/yy(5)}}" );@]@; @ @= @[TeX_( "/yy0{/nx/flrepeatgen{/the/yy(1)}{/the/yy(3)}}" );@]@; @ @= @[TeX_( "/yy0{/nx/flrepeatn{/the/yy(1)}{/the/yy(3)}}" );@]@; @ @= @[TeX_( "/yy0{/nx/fldot/the/yy(1)}" );@]@; @ @= @@; @ @= @@; @ @= @[TeX_( "/yy0{/nx/flstring{/the/yy(1)}{/the/yy(2)}{/the/yy(3)}}" );@]@; @ @= @[TeX_( "/toksa/expandafter{/astformat@@flparens}" );@]@; @[TeX_( "/let/astformat@@flparens/empty" );@]@; @[TeX_( "/yy0{/nx/flparens{/the/yy(1)}{/the/yy(2)}{/the/yy(3)}{/the/toksa}}" );@]@; @ @= @[TeX_( "/yy0{/nx/flchar/the/yy(1)}" );@]@; @ @= @G fullccl: fullccl CCL_OP_DIFF braceccl {@> @ @=} | fullccl CCL_OP_UNION braceccl {@> @ @=} | braceccl {@> @ @=} ; braceccl: '[' ccl ']' {@> @ @=} | '[' '^' ccl ']' {@> @ @=} ; ccl: ccl CHAR '-' CHAR {@> @ @=} | ccl CHAR {@> @ @=} | ccl ccl_expr {@> @ @=} | {@> @ @=} ; @g @ @= @[TeX_( "/yy0{/nx/flccldiff{/the/yy(1)}{/the/yy(3)}}" );@]@; @ @= @[TeX_( "/yy0{/nx/flcclunion{/the/yy(1)}{/the/yy(3)}}" );@]@; @ @= @@; @ @= @[TeX_( "/yy0{/nx/flbraceccl{/the/yy(1)}{/the/yy(2)}{/the/yy(3)}}" );@]@; @ @= @[TeX_( "/yy0{/nx/flbracecclneg{/the/yy(1)}{/the/yy(3)}{/the/yy(4)}}" );@]@; @ @= @[TeX_( "/yy0{/the/yy(1)/nx/flcclrnge{/nx/flchar/the/yy(2)}{/nx/flchar/the/yy(4)}}" );@]@; @ @= @[TeX_( "/yy0{/the/yy(1)/nx/flchar/the/yy(2)}" );@]@; @ @= @[TeX_( "/yy0{/the/yy(1)/nx/flcclexpr/the/yy(2)}" );@]@; @ @= @[TeX_( "/yy0{}" );@]@; @ @= @G ccl_expr: @t}\vb{\flatten}{@> CCE_ALNUM {@> @ @=} | CCE_ALPHA {@> @ @=} | CCE_BLANK {@> @ @=} | CCE_CNTRL {@> @ @=} | CCE_DIGIT {@> @ @=} | CCE_GRAPH {@> @ @=} @t}\vb{\fold\flatten}{@> | CCE_LOWER {@> @ @=} | CCE_PRINT {@> @ @=} | CCE_PUNCT {@> @ @=} | CCE_SPACE {@> @ @=} | CCE_XDIGIT {@> @ @=} | CCE_UPPER {@> @ @=} @t}\vb{\fold\flatten}{@> | CCE_NEG_ALNUM {@> @ @=} | CCE_NEG_ALPHA {@> @ @=} | CCE_NEG_BLANK {@> @ @=} | CCE_NEG_CNTRL {@> @ @=} | CCE_NEG_DIGIT {@> @ @=} | CCE_NEG_GRAPH {@> @ @=} @t}\vb{\fold\flatten}{@> | CCE_NEG_PRINT {@> @ @=} | CCE_NEG_PUNCT {@> @ @=} | CCE_NEG_SPACE {@> @ @=} | CCE_NEG_XDIGIT {@> @ @=} | CCE_NEG_LOWER {@> @ @=} | CCE_NEG_UPPER {@> @ @=} ; @t}\vb{\inline}{@> string: string CHAR {@> @ @=} | {@> @ @=} ; @g @ @= @[TeX_( "/yy0{/the/yy(1)}" );@]@; @ @= @[TeX_( "/yy0{/the/yy(1)/nx/flchar/the/yy(2)}" );@]@; @ @= @[TeX_( "/yy0{}" );@]@; @ This is needed to get the |yytoknum| array. A trivial declaration suffices. @= #define YYPRINT(file, type, value) @[yyprint (file, type, value)@] static void yyprint (FILE *file, int type, YYSTYPE value){}