![[BACK]](/cvsweb/icons/back.gif){kind=icon}
Return to awk CVS log ![[TXT]](/cvsweb/icons/text.gif){kind=icon}
![[DIR]](/cvsweb/icons/dir.gif){kind=icon}
Up to [CSRG BSD Unix] / 43BSDReno / share / doc / usd / 19.awk|
Return to awk CVS log | Up to [CSRG BSD Unix] / 43BSDReno / share / doc / usd / 19.awk |
1.1 root 1: .\" @(#)awk 6.1 (Berkeley) 5/22/86
2: .\"
3: .EH 'USD:19-%''Awk \(em A Pattern Scanning and Processing Language'
4: .OH 'Awk \(em A Pattern Scanning and Processing Language''USD:19-%'
5: .\" .fp 3 G no G on APS (use gb) or Dandelion Printer (use CW)
6: .\" the .T is only a ditroff feature...
7: .if '\*.T'dp' .fp 3 El
8: .if '\*.T'aps' .fp 3 gB
9: ....TM "78-1271-12, 78-1273-6" 39199 39199-11
10: .ND "September 1, 1978"
11: ....TR 68
12: .\".RP
13: . \" macros here
14: .tr _\(em
15: .if t .tr ~\(ap
16: .tr |\(or
17: .tr *\(**
18: .de UC
19: \&\\$3\s-1\\$1\\s0\&\\$2
20: ..
21: .de IT
22: .if n .ul
23: \&\\$3\f2\\$1\fP\|\\$2
24: ..
25: .de UL
26: .if n .ul
27: \&\\$3\f3\\$1\fP\&\\$2
28: ..
29: .de P1
30: .DS I 3n
31: .nf
32: .if n .ta 5 10 15 20 25 30 35 40 45 50 55 60
33: .if t .ta .3i .6i .9i 1.2i
34: .if t .tr -\-'\(fm*\(**
35: .if t .tr _\(ul
36: .ft 3
37: .lg 0
38: .ss 18
39: . \"use first argument as indent if present
40: ..
41: .de P2
42: .ps \\n(PS
43: .vs \\n(VSp
44: .ft R
45: .ss 12
46: .if n .ls 2
47: .tr --''``^^!!
48: .if t .tr _\(em
49: .fi
50: .lg
51: .DE
52: ..
53: .hw semi-colon
54: .hy 14
55: . \"2=not last lines; 4= no -xx; 8=no xx-
56: . \"special chars in programs
57: .de WS
58: .sp \\$1
59: ..
60: . \" end of macros
61: .TL
62: Awk \(em A Pattern Scanning and Processing Language
63: .br
64: (Second Edition)
65: .AU "MH 2C-522" 4862
66: Alfred V. Aho
67: .AU "MH 2C-518" 6021
68: Brian W. Kernighan
69: .AU "MH 2C-514" 7214
70: Peter J. Weinberger
71: .AI
72: .MH
73: .AB
74: .IT Awk
75: is a programming language whose
76: basic operation
77: is to search a set of files
78: for patterns, and to perform specified actions upon lines or fields of lines which
79: contain instances of those patterns.
80: .IT Awk
81: makes certain data selection and transformation operations easy to express;
82: for example, the
83: .IT awk
84: program
85: .sp
86: .ce
87: .ft 3
88: length > 72
89: .ft
90: .sp
91: prints all input lines whose length exceeds 72 characters;
92: the program
93: .ce
94: .sp
95: .ft 3
96: NF % 2 == 0
97: .ft R
98: .sp
99: prints all lines with an even number of fields;
100: and the program
101: .ce
102: .sp
103: .ft 3
104: { $1 = log($1); print }
105: .ft R
106: .sp
107: replaces the first field of each line by its logarithm.
108: .PP
109: .IT Awk
110: patterns may include arbitrary boolean combinations of regular expressions
111: and of relational operators on strings, numbers, fields, variables, and array elements.
112: Actions may include the same pattern-matching constructions as in patterns,
113: as well as
114: arithmetic and string expressions and assignments,
115: .UL if-else ,
116: .UL while ,
117: .UL for
118: statements,
119: and multiple output streams.
120: .PP
121: This report contains a user's guide, a discussion of the design and implementation of
122: .IT awk ,
123: and some timing statistics.
124: ....It supersedes TM-77-1271-5, dated September 8, 1977.
125: .AE
126: .CS 6 1 7 0 1 4
127: .if n .ls 2
128: .nr PS 9
129: .nr VS 11
130: .NH
131: Introduction
132: .if t .2C
133: .PP
134: .IT Awk
135: is a programming language designed to make
136: many common
137: information retrieval and text manipulation tasks
138: easy to state and to perform.
139: .PP
140: The basic operation of
141: .IT awk
142: is to scan a set of input lines in order,
143: searching for lines which match any of a set of patterns
144: which the user has specified.
145: For each pattern, an action can be specified;
146: this action will be performed on each line that matches the pattern.
147: .PP
148: Readers familiar with the
149: .UX
150: program
151: .IT grep\|
152: .[
153: unix program manual
154: .]
155: will recognize
156: the approach, although in
157: .IT awk
158: the patterns may be more
159: general than in
160: .IT grep ,
161: and the actions allowed are more involved than merely
162: printing the matching line.
163: For example, the
164: .IT awk
165: program
166: .P1
167: {print $3, $2}
168: .P2
169: prints the third and second columns of a table
170: in that order.
171: The program
172: .P1
173: $2 ~ /A\||B\||C/
174: .P2
175: prints all input lines with an A, B, or C in the second field.
176: The program
177: .P1
178: $1 != prev { print; prev = $1 }
179: .P2
180: prints all lines in which the first field is different
181: from the previous first field.
182: .NH 2
183: Usage
184: .PP
185: The command
186: .P1
187: awk program [files]
188: .P2
189: executes the
190: .IT awk
191: commands in
192: the string
193: .UL program
194: on the set of named files,
195: or on the standard input if there are no files.
196: The statements can also be placed in a file
197: .UL pfile ,
198: and executed by the command
199: .P1
200: awk -f pfile [files]
201: .P2
202: .NH 2
203: Program Structure
204: .PP
205: An
206: .IT awk
207: program is a sequence of statements of the form:
208: .P1
209: .ft I
210: pattern { action }
211: pattern { action }
212: ...
213: .ft 3
214: .P2
215: Each line of input
216: is matched against
217: each of the patterns in turn.
218: For each pattern that matches, the associated action
219: is executed.
220: When all the patterns have been tested, the next line
221: is fetched and the matching starts over.
222: .PP
223: Either the pattern or the action may be left out,
224: but not both.
225: If there is no action for a pattern,
226: the matching line is simply
227: copied to the output.
228: (Thus a line which matches several patterns can be printed several times.)
229: If there is no pattern for an action,
230: then the action is performed for every input line.
231: A line which matches no pattern is ignored.
232: .PP
233: Since patterns and actions are both optional,
234: actions must be enclosed in braces
235: to distinguish them from patterns.
236: .NH 2
237: Records and Fields
238: .PP
239: .IT Awk
240: input is divided into
241: ``records'' terminated by a record separator.
242: The default record separator is a newline,
243: so by default
244: .IT awk
245: processes its input a line at a time.
246: The number of the current record is available in a variable
247: named
248: .UL NR .
249: .PP
250: Each input record
251: is considered to be divided into ``fields.''
252: Fields are normally separated by
253: white space \(em blanks or tabs \(em
254: but the input field separator may be changed, as described below.
255: Fields are referred to as
256: .UL "$1, $2,"
257: and so forth,
258: where
259: .UL $1
260: is the first field,
261: and
262: .UL $0
263: is the whole input record itself.
264: Fields may be assigned to.
265: The number of fields in the current record
266: is available in a variable named
267: .UL NF .
268: .PP
269: The variables
270: .UL FS
271: and
272: .UL RS
273: refer to the input field and record separators;
274: they may be changed at any time to any single character.
275: The optional command-line argument
276: \f3\-F\fIc\fR
277: may also be used to set
278: .UL FS
279: to the character
280: .IT c .
281: .PP
282: If the record separator is empty,
283: an empty input line is taken as the record separator,
284: and blanks, tabs and newlines are treated as field separators.
285: .PP
286: The variable
287: .UL FILENAME
288: contains the name of the current input file.
289: .NH 2
290: Printing
291: .PP
292: An action may have no pattern,
293: in which case the action is executed for
294: all
295: lines.
296: The simplest action is to print some or all of a record;
297: this is accomplished by the
298: .IT awk
299: command
300: .UL print .
301: The
302: .IT awk
303: program
304: .P1
305: { print }
306: .P2
307: prints each record, thus copying the input to the output intact.
308: More useful is to print a field or fields from each record.
309: For instance,
310: .P1
311: print $2, $1
312: .P2
313: prints the first two fields in reverse order.
314: Items separated by a comma in the print statement will be separated by the current output field separator
315: when output.
316: Items not separated by commas will be concatenated,
317: so
318: .P1
319: print $1 $2
320: .P2
321: runs the first and second fields together.
322: .PP
323: The predefined variables
324: .UL NF
325: and
326: .UL NR
327: can be used;
328: for example
329: .P1
330: { print NR, NF, $0 }
331: .P2
332: prints each record preceded by the record number and the number of fields.
333: .PP
334: Output may be diverted to multiple files;
335: the program
336: .P1
337: { print $1 >"foo1"; print $2 >"foo2" }
338: .P2
339: writes the first field,
340: .UL $1 ,
341: on the file
342: .UL foo1 ,
343: and the second field on file
344: .UL foo2 .
345: The
346: .UL >>
347: notation can also be used:
348: .P1
349: print $1 >>"foo"
350: .P2
351: appends the output to the file
352: .UL foo .
353: (In each case,
354: the output files are
355: created if necessary.)
356: The file name can be a variable or a field as well as a constant;
357: for example,
358: .P1
359: print $1 >$2
360: .P2
361: uses the contents of field 2 as a file name.
362: .PP
363: Naturally there is a limit on the number of output files;
364: currently it is 10.
365: .PP
366: Similarly, output can be piped into another process
367: (on
368: .UC UNIX
369: only); for instance,
370: .P1
371: print | "mail bwk"
372: .P2
373: mails the output to
374: .UL bwk .
375: .PP
376: The variables
377: .UL OFS
378: and
379: .UL ORS
380: may be used to change the current
381: output field separator and output
382: record separator.
383: The output record separator is
384: appended to the output of the
385: .UL print
386: statement.
387: .PP
388: .IT Awk
389: also provides the
390: .UL printf
391: statement for output formatting:
392: .P1
393: printf format expr, expr, ...
394: .P2
395: formats the expressions in the list
396: according to the specification
397: in
398: .UL format
399: and prints them.
400: For example,
401: .P1
402: printf "%8.2f %10ld\en", $1, $2
403: .P2
404: prints
405: .UL $1
406: as a floating point number 8 digits wide,
407: with two after the decimal point,
408: and
409: .UL $2
410: as a 10-digit long decimal number,
411: followed by a newline.
412: No output separators are produced automatically;
413: you must add them yourself,
414: as in this example.
415: The version of
416: .UL printf
417: is identical to that used with C.
418: .[
419: C programm language prentice hall 1978
420: .]
421: .NH 1
422: Patterns
423: .PP
424: A pattern in front of an action acts as a selector
425: that determines whether the action is to be executed.
426: A variety of expressions may be used as patterns:
427: regular expressions,
428: arithmetic relational expressions,
429: string-valued expressions,
430: and arbitrary boolean
431: combinations of these.
432: .NH 2
433: BEGIN and END
434: .PP
435: The special pattern
436: .UL BEGIN
437: matches the beginning of the input,
438: before the first record is read.
439: The pattern
440: .UL END
441: matches the end of the input,
442: after the last record has been processed.
443: .UL BEGIN
444: and
445: .UL END
446: thus provide a way to gain control before and after processing,
447: for initialization and wrapup.
448: .PP
449: As an example, the field separator
450: can be set to a colon by
451: .P1
452: BEGIN { FS = ":" }
453: .ft I
454: \&... rest of program ...
455: .ft 3
456: .P2
457: Or the input lines may be counted by
458: .P1
459: END { print NR }
460: .P2
461: If
462: .UL BEGIN
463: is present, it must be the first pattern;
464: .UL END
465: must be the last if used.
466: .NH 2
467: Regular Expressions
468: .PP
469: The simplest regular expression is a literal string of characters
470: enclosed in slashes,
471: like
472: .P1
473: /smith/
474: .P2
475: This
476: is actually a complete
477: .IT awk
478: program which
479: will print all lines which contain any occurrence
480: of the name ``smith''.
481: If a line contains ``smith''
482: as part of a larger word,
483: it will also be printed, as in
484: .P1
485: blacksmithing
486: .P2
487: .PP
488: .IT Awk
489: regular expressions include the regular expression
490: forms found in
491: the
492: .UC UNIX
493: text editor
494: .IT ed\|
495: .[
496: unix program manual
497: .]
498: and
499: .IT grep
500: (without back-referencing).
501: In addition,
502: .IT awk
503: allows
504: parentheses for grouping, | for alternatives,
505: .UL +
506: for ``one or more'', and
507: .UL ?
508: for ``zero or one'',
509: all as in
510: .IT lex .
511: Character classes
512: may be abbreviated:
513: .UL [a\-zA\-Z0\-9]
514: is the set of all letters and digits.
515: As an example,
516: the
517: .IT awk
518: program
519: .P1
520: /[Aa]ho\||[Ww]einberger\||[Kk]ernighan/
521: .P2
522: will print all lines which contain any of the names
523: ``Aho,'' ``Weinberger'' or ``Kernighan,''
524: whether capitalized or not.
525: .PP
526: Regular expressions
527: (with the extensions listed above)
528: must be enclosed in slashes,
529: just as in
530: .IT ed
531: and
532: .IT sed .
533: Within a regular expression,
534: blanks and the regular expression
535: metacharacters are significant.
536: To turn of the magic meaning
537: of one of the regular expression characters,
538: precede it with a backslash.
539: An example is the pattern
540: .P1
541: /\|\e/\^.\^*\e//
542: .P2
543: which matches any string of characters
544: enclosed in slashes.
545: .PP
546: One can also specify that any field or variable
547: matches
548: a regular expression (or does not match it) with the operators
549: .UL ~
550: and
551: .UL !~ .
552: The program
553: .P1
554: $1 ~ /[jJ]ohn/
555: .P2
556: prints all lines where the first field matches ``john'' or ``John.''
557: Notice that this will also match ``Johnson'', ``St. Johnsbury'', and so on.
558: To restrict it to exactly
559: .UL [jJ]ohn ,
560: use
561: .P1
562: $1 ~ /^[jJ]ohn$/
563: .P2
564: The caret ^ refers to the beginning
565: of a line or field;
566: the dollar sign
567: .UL $
568: refers to the end.
569: .NH 2
570: Relational Expressions
571: .PP
572: An
573: .IT awk
574: pattern can be a relational expression
575: involving the usual relational operators
576: .UL < ,
577: .UL <= ,
578: .UL == ,
579: .UL != ,
580: .UL >= ,
581: and
582: .UL > .
583: An example is
584: .P1
585: $2 > $1 + 100
586: .P2
587: which selects lines where the second field
588: is at least 100 greater than the first field.
589: Similarly,
590: .P1
591: NF % 2 == 0
592: .P2
593: prints lines with an even number of fields.
594: .PP
595: In relational tests, if neither operand is numeric,
596: a string comparison is made;
597: otherwise it is numeric.
598: Thus,
599: .P1
600: $1 >= "s"
601: .P2
602: selects lines that begin with an
603: .UL s ,
604: .UL t ,
605: .UL u ,
606: etc.
607: In the absence of any other information,
608: fields are treated as strings, so
609: the program
610: .P1
611: $1 > $2
612: .P2
613: will perform a string comparison.
614: .NH 2
615: Combinations of Patterns
616: .PP
617: A pattern can be any boolean combination of patterns,
618: using the operators
619: .UL \||\||
620: (or),
621: .UL &&
622: (and), and
623: .UL !
624: (not).
625: For example,
626: .P1
627: $1 >= "s" && $1 < "t" && $1 != "smith"
628: .P2
629: selects lines where the first field begins with ``s'', but is not ``smith''.
630: .UL &&
631: and
632: .UL \||\||
633: guarantee that their operands
634: will be evaluated
635: from left to right;
636: evaluation stops as soon as the truth or falsehood
637: is determined.
638: .NH 2
639: Pattern Ranges
640: .PP
641: The ``pattern'' that selects an action may also
642: consist of two patterns separated by a comma, as in
643: .P1
644: pat1, pat2 { ... }
645: .P2
646: In this case, the action is performed for each line between
647: an occurrence of
648: .UL pat1
649: and the next occurrence of
650: .UL pat2
651: (inclusive).
652: For example,
653: .P1
654: /start/, /stop/
655: .P2
656: prints all lines between
657: .UL start
658: and
659: .UL stop ,
660: while
661: .P1
662: NR == 100, NR == 200 { ... }
663: .P2
664: does the action for lines 100 through 200
665: of the input.
666: .NH 1
667: Actions
668: .PP
669: An
670: .IT awk
671: action is a sequence of action statements
672: terminated by newlines or semicolons.
673: These action statements can be used to do a variety of
674: bookkeeping and string manipulating tasks.
675: .NH 2
676: Built-in Functions
677: .PP
678: .IT Awk
679: provides a ``length'' function
680: to compute the length of a string of characters.
681: This program prints each record,
682: preceded by its length:
683: .P1
684: {print length, $0}
685: .P2
686: .UL length
687: by itself is a ``pseudo-variable'' which
688: yields the length of the current record;
689: .UL length(argument)
690: is a function which yields the length of its argument,
691: as in
692: the equivalent
693: .P1
694: {print length($0), $0}
695: .P2
696: The argument may be any expression.
697: .PP
698: .IT Awk
699: also
700: provides the arithmetic functions
701: .UL sqrt ,
702: .UL log ,
703: .UL exp ,
704: and
705: .UL int ,
706: for
707: square root,
708: base
709: .IT e
710: logarithm,
711: exponential,
712: and integer part of their respective arguments.
713: .PP
714: The name of one of these built-in functions,
715: without argument or parentheses,
716: stands for the value of the function on the
717: whole record.
718: The program
719: .P1
720: length < 10 || length > 20
721: .P2
722: prints lines whose length
723: is less than 10 or greater
724: than 20.
725: .PP
726: The function
727: .UL substr(s,\ m,\ n)
728: produces the substring of
729: .UL s
730: that begins at position
731: .UL m
732: (origin 1)
733: and is at most
734: .UL n
735: characters long.
736: If
737: .UL n
738: is omitted, the substring goes to the end of
739: .UL s .
740: The function
741: .UL index(s1,\ s2)
742: returns the position where the string
743: .UL s2
744: occurs in
745: .UL s1 ,
746: or zero if it does not.
747: .PP
748: The function
749: .UL sprintf(f,\ e1,\ e2,\ ...)
750: produces the value of the expressions
751: .UL e1 ,
752: .UL e2 ,
753: etc.,
754: in the
755: .UL printf
756: format specified by
757: .UL f .
758: Thus, for example,
759: .P1
760: x = sprintf("%8.2f %10ld", $1, $2)
761: .P2
762: sets
763: .UL x
764: to the string produced by formatting
765: the values of
766: .UL $1
767: and
768: .UL $2 .
769: .NH 2
770: Variables, Expressions, and Assignments
771: .PP
772: .IT Awk
773: variables take on numeric (floating point)
774: or string values according to context.
775: For example, in
776: .P1
777: x = 1
778: .P2
779: .UL x
780: is clearly a number, while in
781: .P1
782: x = "smith"
783: .P2
784: it is clearly a string.
785: Strings are converted to numbers and
786: vice versa whenever context demands it.
787: For instance,
788: .P1
789: x = "3" + "4"
790: .P2
791: assigns 7 to
792: .UL x .
793: Strings which cannot be interpreted
794: as numbers in a numerical context
795: will generally have numeric value zero,
796: but it is unwise to count on this behavior.
797: .PP
798: By default, variables (other than built-ins) are initialized to the null string,
799: which has numerical value zero;
800: this eliminates the need for most
801: .UL BEGIN
802: sections.
803: For example, the sums of the first two fields can be computed by
804: .P1
805: { s1 += $1; s2 += $2 }
806: END { print s1, s2 }
807: .P2
808: .PP
809: Arithmetic is done internally in floating point.
810: The arithmetic operators are
811: .UL + ,
812: .UL \- ,
813: .UL \(** ,
814: .UL / ,
815: and
816: .UL %
817: (mod).
818: The C increment
819: .UL ++
820: and
821: decrement
822: .UL \-\-
823: operators are also available,
824: and so are the assignment operators
825: .UL += ,
826: .UL \-= ,
827: .UL *= ,
828: .UL /= ,
829: and
830: .UL %= .
831: These operators may all be used in expressions.
832: .NH 2
833: Field Variables
834: .PP
835: Fields in
836: .IT awk
837: share essentially all of the properties of variables _
838: they may be used in arithmetic or string operations,
839: and may be assigned to.
840: Thus one can
841: replace the first field with a sequence number like this:
842: .P1
843: { $1 = NR; print }
844: .P2
845: or
846: accumulate two fields into a third, like this:
847: .P1
848: { $1 = $2 + $3; print $0 }
849: .P2
850: or assign a string to a field:
851: .P1
852: { if ($3 > 1000)
853: $3 = "too big"
854: print
855: }
856: .P2
857: which replaces the third field by ``too big'' when it is,
858: and in any case prints the record.
859: .PP
860: Field references may be numerical expressions,
861: as in
862: .P1
863: { print $i, $(i+1), $(i+n) }
864: .P2
865: Whether a field is deemed numeric or string depends on context;
866: in ambiguous cases like
867: .P1
868: if ($1 == $2) ...
869: .P2
870: fields are treated as strings.
871: .PP
872: Each input line is split into fields automatically as necessary.
873: It is also possible to split any variable or string
874: into fields:
875: .P1
876: n = split(s, array, sep)
877: .P2
878: splits the
879: the string
880: .UL s
881: into
882: .UL array[1] ,
883: \&...,
884: .UL array[n] .
885: The number of elements found is returned.
886: If the
887: .UL sep
888: argument is provided, it is used as the field separator;
889: otherwise
890: .UL FS
891: is used as the separator.
892: .NH 2
893: String Concatenation
894: .PP
895: Strings may be concatenated.
896: For example
897: .P1
898: length($1 $2 $3)
899: .P2
900: returns the length of the first three fields.
901: Or in a
902: .UL print
903: statement,
904: .P1
905: print $1 " is " $2
906: .P2
907: prints
908: the two fields separated by `` is ''.
909: Variables and numeric expressions may also appear in concatenations.
910: .NH 2
911: Arrays
912: .PP
913: Array elements are not declared;
914: they spring into existence by being mentioned.
915: Subscripts may have
916: .ul
917: any
918: non-null
919: value, including non-numeric strings.
920: As an example of a conventional numeric subscript,
921: the statement
922: .P1
923: x[NR] = $0
924: .P2
925: assigns the current input record to
926: the
927: .UL NR -th
928: element of the array
929: .UL x .
930: In fact, it is possible in principle (though perhaps slow)
931: to process the entire input in a random order with the
932: .IT awk
933: program
934: .P1
935: { x[NR] = $0 }
936: END { \fI... program ...\fP }
937: .P2
938: The first action merely records each input line in
939: the array
940: .UL x .
941: .PP
942: Array elements may be named by non-numeric values,
943: which gives
944: .IT awk
945: a capability rather like the associative memory of
946: Snobol tables.
947: Suppose the input contains fields with values like
948: .UL apple ,
949: .UL orange ,
950: etc.
951: Then the program
952: .P1
953: /apple/ { x["apple"]++ }
954: /orange/ { x["orange"]++ }
955: END { print x["apple"], x["orange"] }
956: .P2
957: increments counts for the named array elements,
958: and prints them at the end of the input.
959: .NH 2
960: Flow-of-Control Statements
961: .PP
962: .IT Awk
963: provides the basic flow-of-control statements
964: .UL if-else ,
965: .UL while ,
966: .UL for ,
967: and statement grouping with braces, as in C.
968: We showed the
969: .UL if
970: statement in section 3.3 without describing it.
971: The condition in parentheses is evaluated;
972: if it is true, the statement following the
973: .UL if
974: is done.
975: The
976: .UL else
977: part is optional.
978: .PP
979: The
980: .UL while
981: statement is exactly like that of C.
982: For example, to print all input fields one per line,
983: .P1
984: i = 1
985: while (i <= NF) {
986: print $i
987: ++i
988: }
989: .P2
990: .PP
991: The
992: .UL for
993: statement is also exactly that of C:
994: .P1
995: for (i = 1; i <= NF; i++)
996: print $i
997: .P2
998: does the same job as the
999: .UL while
1000: statement above.
1001: .PP
1002: There is an alternate form of the
1003: .UL for
1004: statement which is suited for accessing the
1005: elements of an associative array:
1006: .P1
1007: for (i in array)
1008: \fIstatement\f3
1009: .P2
1010: does
1011: .ul
1012: statement
1013: with
1014: .UL i
1015: set in turn to each element of
1016: .UL array .
1017: The elements are accessed in an apparently random order.
1018: Chaos will ensue if
1019: .UL i
1020: is altered, or if any new elements are
1021: accessed during the loop.
1022: .PP
1023: The expression in the condition part of an
1024: .UL if ,
1025: .UL while
1026: or
1027: .UL for
1028: can include relational operators like
1029: .UL < ,
1030: .UL <= ,
1031: .UL > ,
1032: .UL >= ,
1033: .UL ==
1034: (``is equal to''),
1035: and
1036: .UL !=
1037: (``not equal to'');
1038: regular expression matches with the match operators
1039: .UL ~
1040: and
1041: .UL !~ ;
1042: the logical operators
1043: .UL \||\|| ,
1044: .UL && ,
1045: and
1046: .UL ! ;
1047: and of course parentheses for grouping.
1048: .PP
1049: The
1050: .UL break
1051: statement causes an immediate exit
1052: from an enclosing
1053: .UL while
1054: or
1055: .UL for ;
1056: the
1057: .UL continue
1058: statement
1059: causes the next iteration to begin.
1060: .PP
1061: The statement
1062: .UL next
1063: causes
1064: .IT awk
1065: to skip immediately to
1066: the next record and begin scanning the patterns from the top.
1067: The statement
1068: .UL exit
1069: causes the program to behave as if the end of the input
1070: had occurred.
1071: .PP
1072: Comments may be placed in
1073: .IT awk
1074: programs:
1075: they begin with the character
1076: .UL #
1077: and end with the end of the line,
1078: as in
1079: .P1
1080: print x, y # this is a comment
1081: .P2
1082: .NH
1083: Design
1084: .PP
1085: The
1086: .UX
1087: system
1088: already provides several programs that
1089: operate by passing input through a
1090: selection mechanism.
1091: .IT Grep ,
1092: the first and simplest, merely prints all lines which
1093: match a single specified pattern.
1094: .IT Egrep
1095: provides more general patterns, i.e., regular expressions
1096: in full generality;
1097: .IT fgrep
1098: searches for a set of keywords with a particularly fast algorithm.
1099: .IT Sed\|
1100: .[
1101: unix programm manual
1102: .]
1103: provides most of the editing facilities of
1104: the editor
1105: .IT ed ,
1106: applied to a stream of input.
1107: None of these programs provides
1108: numeric capabilities,
1109: logical relations,
1110: or variables.
1111: .PP
1112: .IT Lex\|
1113: .[
1114: lesk lexical analyzer cstr
1115: .]
1116: provides general regular expression recognition capabilities,
1117: and, by serving as a C program generator,
1118: is essentially open-ended in its capabilities.
1119: The use of
1120: .IT lex ,
1121: however, requires a knowledge of C programming,
1122: and a
1123: .IT lex
1124: program must be compiled and loaded before use,
1125: which discourages its use for one-shot applications.
1126: .PP
1127: .IT Awk
1128: is an attempt
1129: to fill in another part of the matrix of possibilities.
1130: It
1131: provides general regular expression capabilities
1132: and an implicit input/output loop.
1133: But it also provides convenient numeric processing,
1134: variables,
1135: more general selection,
1136: and control flow in the actions.
1137: It
1138: does not require compilation or a knowledge of C.
1139: Finally,
1140: .IT awk
1141: provides
1142: a convenient way to access fields within lines;
1143: it is unique in this respect.
1144: .PP
1145: .IT Awk
1146: also tries to integrate strings and numbers
1147: completely,
1148: by treating all quantities as both string and numeric,
1149: deciding which representation is appropriate
1150: as late as possible.
1151: In most cases the user can simply ignore the differences.
1152: .PP
1153: Most of the effort in developing
1154: .I awk
1155: went into deciding what
1156: .I awk
1157: should or should not do
1158: (for instance, it doesn't do string substitution)
1159: and what the syntax should be
1160: (no explicit operator for concatenation)
1161: rather
1162: than on writing or debugging the code.
1163: We have tried
1164: to make the syntax powerful
1165: but easy to use and well adapted
1166: to scanning files.
1167: For example,
1168: the absence of declarations and implicit initializations,
1169: while probably a bad idea for a general-purpose programming language,
1170: is desirable in a language
1171: that is meant to be used for tiny programs
1172: that may even be composed on the command line.
1173: .PP
1174: In practice,
1175: .IT awk
1176: usage seems to fall into two broad categories.
1177: One is what might be called ``report generation'' \(em
1178: processing an input to extract counts,
1179: sums, sub-totals, etc.
1180: This also includes the writing of trivial
1181: data validation programs,
1182: such as verifying that a field contains only numeric information
1183: or that certain delimiters are properly balanced.
1184: The combination of textual and numeric processing is invaluable here.
1185: .PP
1186: A second area of use is as a data transformer,
1187: converting data from the form produced by one program
1188: into that expected by another.
1189: The simplest examples merely select fields, perhaps with rearrangements.
1190: .NH
1191: Implementation
1192: .PP
1193: The actual implementation of
1194: .IT awk
1195: uses the language development tools available
1196: on the
1197: .UC UNIX
1198: operating system.
1199: The grammar is specified with
1200: .IT yacc ;
1201: .[
1202: yacc johnson cstr
1203: .]
1204: the lexical analysis is done by
1205: .IT lex ;
1206: the regular expression recognizers are
1207: deterministic finite automata
1208: constructed directly from the expressions.
1209: An
1210: .IT awk
1211: program is translated into a
1212: parse tree which is then directly executed
1213: by a simple interpreter.
1214: .PP
1215: .IT Awk
1216: was designed for ease of use rather than processing speed;
1217: the delayed evaluation of variable types
1218: and the necessity to break input
1219: into fields makes high speed difficult to achieve in any case.
1220: Nonetheless,
1221: the program has not proven to be unworkably slow.
1222: .PP
1223: Table I below shows the execution (user + system) time
1224: on a PDP-11/70 of
1225: the
1226: .UC UNIX
1227: programs
1228: .IT wc ,
1229: .IT grep ,
1230: .IT egrep ,
1231: .IT fgrep ,
1232: .IT sed ,
1233: .IT lex ,
1234: and
1235: .IT awk
1236: on the following simple tasks:
1237: .IP "\ \ 1."
1238: count the number of lines.
1239: .IP "\ \ 2."
1240: print all lines containing ``doug''.
1241: .IP "\ \ 3."
1242: print all lines containing ``doug'', ``ken'' or ``dmr''.
1243: .IP "\ \ 4."
1244: print the third field of each line.
1245: .IP "\ \ 5."
1246: print the third and second fields of each line, in that order.
1247: .IP "\ \ 6."
1248: append all lines containing ``doug'', ``ken'', and ``dmr''
1249: to files ``jdoug'', ``jken'', and ``jdmr'', respectively.
1250: .IP "\ \ 7."
1251: print each line prefixed by ``line-number\ :\ ''.
1252: .IP "\ \ 8."
1253: sum the fourth column of a table.
1254: .LP
1255: The program
1256: .IT wc
1257: merely counts words, lines and characters in its input;
1258: we have already mentioned the others.
1259: In all cases the input was a file containing
1260: 10,000 lines
1261: as created by the
1262: command
1263: .IT "ls \-l" ;
1264: each line has the form
1265: .P1
1266: -rw-rw-rw- 1 ava 123 Oct 15 17:05 xxx
1267: .P2
1268: The total length of this input is
1269: 452,960 characters.
1270: Times for
1271: .IT lex
1272: do not include compile or load.
1273: .PP
1274: As might be expected,
1275: .IT awk
1276: is not as fast as the specialized tools
1277: .IT wc ,
1278: .IT sed ,
1279: or the programs in the
1280: .IT grep
1281: family,
1282: but
1283: is faster than the more general tool
1284: .IT lex .
1285: In all cases, the tasks were
1286: about as easy to express as
1287: .IT awk
1288: programs
1289: as programs in these other languages;
1290: tasks involving fields were
1291: considerably easier to express as
1292: .IT awk
1293: programs.
1294: Some of the test programs are shown in
1295: .IT awk ,
1296: .IT sed
1297: and
1298: .IT lex .
1299: .[
1300: $LIST$
1301: .]
1302: .1C
1303: .TS
1304: center;
1305: c c c c c c c c c
1306: c c c c c c c c c
1307: c|n|n|n|n|n|n|n|n|.
1308: Task
1309: Program 1 2 3 4 5 6 7 8
1310: _
1311: \fIwc\fR 8.6
1312: \fIgrep\fR 11.7 13.1
1313: \fIegrep\fR 6.2 11.5 11.6
1314: \fIfgrep\fR 7.7 13.8 16.1
1315: \fIsed\fR 10.2 11.6 15.8 29.0 30.5 16.1
1316: \fIlex\fR 65.1 150.1 144.2 67.7 70.3 104.0 81.7 92.8
1317: \fIawk\fR 15.0 25.6 29.9 33.3 38.9 46.4 71.4 31.1
1318: _
1319: .TE
1320: .sp
1321: .ce
1322: \fBTable I.\fR Execution Times of Programs. (Times are in sec.)
1323: .sp 2
1324: .2C
1325: .PP
1326: The programs for some of these jobs are shown below.
1327: The
1328: .IT lex
1329: programs are generally too long to show.
1330: .LP
1331: AWK:
1332: .LP
1333: .P1
1334: 1. END {print NR}
1335: .P2
1336: .P1
1337: 2. /doug/
1338: .P2
1339: .P1
1340: 3. /ken|doug|dmr/
1341: .P2
1342: .P1
1343: 4. {print $3}
1344: .P2
1345: .P1
1346: 5. {print $3, $2}
1347: .P2
1348: .P1
1349: 6. /ken/ {print >"jken"}
1350: /doug/ {print >"jdoug"}
1351: /dmr/ {print >"jdmr"}
1352: .P2
1353: .P1
1354: 7. {print NR ": " $0}
1355: .P2
1356: .P1
1357: 8. {sum = sum + $4}
1358: END {print sum}
1359: .P2
1360: .LP
1361: SED:
1362: .LP
1363: .P1
1364: 1. $=
1365: .P2
1366: .P1
1367: 2. /doug/p
1368: .P2
1369: .P1
1370: 3. /doug/p
1371: /doug/d
1372: /ken/p
1373: /ken/d
1374: /dmr/p
1375: /dmr/d
1376: .P2
1377: .P1
1378: 4. /[^ ]* [ ]*[^ ]* [ ]*\e([^ ]*\e) .*/s//\e1/p
1379: .P2
1380: .P1
1381: 5. /[^ ]* [ ]*\e([^ ]*\e) [ ]*\e([^ ]*\e) .*/s//\e2 \e1/p
1382: .P2
1383: .P1
1384: 6. /ken/w jken
1385: /doug/w jdoug
1386: /dmr/w jdmr
1387: .P2
1388: .LP
1389: LEX:
1390: .LP
1391: .P1
1392: 1. %{
1393: int i;
1394: %}
1395: %%
1396: \en i++;
1397: . ;
1398: %%
1399: yywrap() {
1400: printf("%d\en", i);
1401: }
1402: .P2
1403: .P1
1404: 2. %%
1405: ^.*doug.*$ printf("%s\en", yytext);
1406: . ;
1407: \en ;
1408: .P2
This archive runs on limited infrastructure. Preserving old code on modern bandwidth. Automated agents are requested to crawl responsibly.