Return to ss3 CVS log

Up to [CSRG BSD Unix] / 43BSDReno / share / doc / ps1 / 15.yacc

Annotation of 43BSDReno/share/doc/ps1/15.yacc/ss3, revision 1.1

1.1     ! root        1: .\"    @(#)ss3 6.1 (Berkeley) 5/8/86
        !             2: .\"
        !             3: .SH
        !             4: 3: Lexical Analysis
        !             5: .PP
        !             6: The user must supply a lexical analyzer to read the input stream and communicate tokens
        !             7: (with values, if desired) to the parser.
        !             8: The lexical analyzer is an integer-valued function called
        !             9: .I yylex .
        !            10: The function returns an integer, the
        !            11: .I "token number" ,
        !            12: representing the kind of token read.
        !            13: If there is a value associated with that token, it should be assigned
        !            14: to the external variable
        !            15: .I yylval .
        !            16: .PP
        !            17: The parser and the lexical analyzer must agree on these token numbers in order for
        !            18: communication between them to take place.
        !            19: The numbers may be chosen by Yacc, or chosen by the user.
        !            20: In either case, the ``# define'' mechanism of C is used to allow the lexical analyzer
        !            21: to return these numbers symbolically.
        !            22: For example, suppose that the token name DIGIT has been defined in the declarations section of the
        !            23: Yacc specification file.
        !            24: The relevant portion of the lexical analyzer might look like:
        !            25: .DS
        !            26: yylex(){
        !            27:        extern int yylval;
        !            28:        int c;
        !            29:        . . .
        !            30:        c = getchar();
        !            31:        . . .
        !            32:        switch( c ) {
        !            33:                . . .
        !            34:        case \'0\':
        !            35:        case \'1\':
        !            36:          . . .
        !            37:        case \'9\':
        !            38:                yylval = c\-\'0\';
        !            39:                return( DIGIT );
        !            40:                . . .
        !            41:                }
        !            42:        . . .
        !            43: .DE
        !            44: .PP
        !            45: The intent is to return a token number of DIGIT, and a value equal to the numerical value of the
        !            46: digit.
        !            47: Provided that the lexical analyzer code is placed in the programs section of the specification file,
        !            48: the identifier DIGIT will be defined as the token number associated
        !            49: with the token DIGIT.
        !            50: .PP
        !            51: This mechanism leads to clear,
        !            52: easily modified lexical analyzers; the only pitfall is the need
        !            53: to avoid using any token names in the grammar that are reserved
        !            54: or significant in C or the parser; for example, the use of
        !            55: token names
        !            56: .I if
        !            57: or
        !            58: .I while
        !            59: will almost certainly cause severe
        !            60: difficulties when the lexical analyzer is compiled.
        !            61: The token name
        !            62: .I error
        !            63: is reserved for error handling, and should not be used naively
        !            64: (see Section 7).
        !            65: .PP
        !            66: As mentioned above, the token numbers may be chosen by Yacc or by the user.
        !            67: In the default situation, the numbers are chosen by Yacc.
        !            68: The default token number for a literal
        !            69: character is the numerical value of the character in the local character set.
        !            70: Other names are assigned token numbers
        !            71: starting at 257.
        !            72: .PP
        !            73: To assign a token number to a token (including literals),
        !            74: the first appearance of the token name or literal
        !            75: .I
        !            76: in the declarations section
        !            77: .R
        !            78: can be immediately followed by
        !            79: a nonnegative integer.
        !            80: This integer is taken to be the token number of the name or literal.
        !            81: Names and literals not defined by this mechanism retain their default definition.
        !            82: It is important that all token numbers be distinct.
        !            83: .PP
        !            84: For historical reasons, the endmarker must have token
        !            85: number 0 or negative.
        !            86: This token number cannot be redefined by the user; thus, all
        !            87: lexical analyzers should be prepared to return 0 or negative as a token number
        !            88: upon reaching the end of their input.
        !            89: .PP
        !            90: A very useful tool for constructing lexical analyzers is
        !            91: the
        !            92: .I Lex
        !            93: program developed by Mike Lesk.
        !            94: .[
        !            95: Lesk Lex
        !            96: .]
        !            97: These lexical analyzers are designed to work in close
        !            98: harmony with Yacc parsers.
        !            99: The specifications for these lexical analyzers
        !           100: use regular expressions instead of grammar rules.
        !           101: Lex can be easily used to produce quite complicated lexical analyzers,
        !           102: but there remain some languages (such as FORTRAN) which do not
        !           103: fit any theoretical framework, and whose lexical analyzers
        !           104: must be crafted by hand.

This archive runs on limited infrastructure. Preserving old code on modern bandwidth. Automated agents are requested to crawl responsibly.