15. A Library of awk Functions

This chapter presents a library of useful awk functions. The sample programs presented later (see section Practical awk Programs) use these functions. The functions are presented here in a progression from simple to complex.

Extracting Programs from Texinfo Source Files, presents a program that you can use to extract the source code for these example library functions and programs from the Texinfo source for this Info file. (This has already been done as part of the gawk distribution.)

If you have written one or more useful, general purpose awk functions, and would like to contribute them for a subsequent edition of this Info file, please contact the author. See section Reporting Problems and Bugs, for information on doing this. Don't just send code, as you will be required to either place your code in the public domain, publish it under the GPL (see section GNU GENERAL PUBLIC LICENSE), or assign the copyright in it to the Free Software Foundation.

15.1 Simulating gawk-specific Features		What to do if you don't have gawk.
15.2 Implementing nextfile as a Function		Two implementations of a nextfile function.
15.3 Assertions		A function for assertions in awk programs.
15.4 Rounding Numbers		A function for rounding if sprintf does not do it correctly.
15.5 Translating Between Characters and Numbers		Functions for using characters as numbers and vice versa.
15.6 Merging an Array Into a String		A function to join an array into a string.
15.7 Turning Dates Into Timestamps		A function to turn a date into a timestamp.
15.8 Managing the Time of Day		A function to get formatted times.
15.9 Noting Data File Boundaries		A function for handling data file transitions.
15.10 Processing Command Line Options		A function for processing command line arguments.
15.11 Reading the User Database		Functions for getting user information.
15.12 Reading the Group Database		Functions for getting group information.
15.13 Naming Library Function Global Variables		How to best name private global variables in library functions.

15.1 Simulating gawk-specific Features

The programs in this chapter and in Practical awk Programs, freely use features that are specific to gawk. This section briefly discusses how you can rewrite these programs for different implementations of awk.

Diagnostic error messages are sent to `/dev/stderr'. Use `| "cat 1>&2"' instead of `> "/dev/stderr"', if your system does not have a `/dev/stderr', or if you cannot use gawk.

A number of programs use nextfile (see section The nextfile Statement), to skip any remaining input in the input file. Implementing nextfile as a Function, shows you how to write a function that will do the same thing.

Finally, some of the programs choose to ignore upper-case and lower-case distinctions in their input. They do this by assigning one to IGNORECASE. You can achieve the same effect by adding the following rule to the beginning of the program:

# ignore case { $0 = tolower($0) }

Also, verify that all regexp and string constants used in comparisons only use lower-case letters.

15.2 Implementing nextfile as a Function

The nextfile statement presented in The nextfile Statement, is a gawk-specific extension. It is not available in other implementations of awk. This section shows two versions of a nextfile function that you can use to simulate gawk's nextfile statement if you cannot use gawk.

Here is a first attempt at writing a nextfile function.

# nextfile --- skip remaining records in current file # this should be read in before the "main" awk program function nextfile() { _abandon_ = FILENAME; next } _abandon_ == FILENAME { next }

This file should be included before the main program, because it supplies a rule that must be executed first. This rule compares the current data file's name (which is always in the FILENAME variable) to a private variable named _abandon_. If the file name matches, then the action part of the rule executes a next statement, to go on to the next record. (The use of `_' in the variable name is a convention. It is discussed more fully in Naming Library Function Global Variables.)

The use of the next statement effectively creates a loop that reads all the records from the current data file. Eventually, the end of the file is reached, and a new data file is opened, changing the value of FILENAME. Once this happens, the comparison of _abandon_ to FILENAME fails, and execution continues with the first rule of the "real" program.

The nextfile function itself simply sets the value of _abandon_ and then executes a next statement to start the loop going.(20)

This initial version has a subtle problem. What happens if the same data file is listed twice on the command line, one right after the other, or even with just a variable assignment between the two occurrences of the file name?

In such a case, this code will skip right through the file, a second time, even though it should stop when it gets to the end of the first occurrence. Here is a second version of nextfile that remedies this problem.

# nextfile --- skip remaining records in current file # correctly handle successive occurrences of the same file # Arnold Robbins, arnold@gnu.org, Public Domain # May, 1993 # this should be read in before the "main" awk program function nextfile() { _abandon_ = FILENAME; next } _abandon_ == FILENAME { if (FNR == 1) _abandon_ = "" else next }

The nextfile function has not changed. It sets _abandon_ equal to the current file name and then executes a next satement. The next statement reads the next record and increments FNR, so FNR is guaranteed to have a value of at least two. However, if nextfile is called for the last record in the file, then awk will close the current data file and move on to the next one. Upon doing so, FILENAME will be set to the name of the new file, and FNR will be reset to one. If this next file is the same as the previous one, _abandon_ will still be equal to FILENAME. However, FNR will be equal to one, telling us that this is a new occurrence of the file, and not the one we were reading when the nextfile function was executed. In that case, _abandon_ is reset to the empty string, so that further executions of this rule will fail (until the next time that nextfile is called).

If FNR is not one, then we are still in the original data file, and the program executes a next statement to skip through it.

An important question to ask at this point is: "Given that the functionality of nextfile can be provided with a library file, why is it built into gawk?" This is an important question. Adding features for little reason leads to larger, slower programs that are harder to maintain.

The answer is that building nextfile into gawk provides significant gains in efficiency. If the nextfile function is executed at the beginning of a large data file, awk still has to scan the entire file, splitting it up into records, just to skip over it. The built-in nextfile can simply close the file immediately and proceed to the next one, saving a lot of time. This is particularly important in awk, since awk programs are generally I/O bound (i.e. they spend most of their time doing input and output, instead of performing computations).

15.3 Assertions

When writing large programs, it is often useful to be able to know that a condition or set of conditions is true. Before proceeding with a particular computation, you make a statement about what you believe to be the case. Such a statement is known as an "assertion." The C language provides an <assert.h> header file and corresponding assert macro that the programmer can use to make assertions. If an assertion fails, the assert macro arranges to print a diagnostic message describing the condition that should have been true but was not, and then it kills the program. In C, using assert looks this:

#include <assert.h> int myfunc(int a, double b) { assert(a <= 5 && b >= 17); ... }

If the assertion failed, the program would print a message similar to this:

prog.c:5: assertion failed: a <= 5 && b >= 17

The ANSI C language makes it possible to turn the condition into a string for use in printing the diagnostic message. This is not possible in awk, so this assert function also requires a string version of the condition that is being tested.

# assert --- assert that a condition is true. Otherwise exit. # Arnold Robbins, arnold@gnu.org, Public Domain # May, 1993 function assert(condition, string) { if (! condition) { printf("%s:%d: assertion failed: %s\n", FILENAME, FNR, string) > "/dev/stderr" _assert_exit = 1 exit 1 } } END { if (_assert_exit) exit 1 }

The assert function tests the condition parameter. If it is false, it prints a message to standard error, using the string parameter to describe the failed condition. It then sets the variable _assert_exit to one, and executes the exit statement. The exit statement jumps to the END rule. If the END rules finds _assert_exit to be true, then it exits immediately.

The purpose of the END rule with its test is to keep any other END rules from running. When an assertion fails, the program should exit immediately. If no assertions fail, then _assert_exit will still be false when the END rule is run normally, and the rest of the program's END rules will execute. For all of this to work correctly, `assert.awk' must be the first source file read by awk.

You would use this function in your programs this way:

function myfunc(a, b) { assert(a <= 5 && b >= 17, "a <= 5 && b >= 17") ... }

If the assertion failed, you would see a message like this:

mydata:1357: assertion failed: a <= 5 && b >= 17

There is a problem with this version of assert, that it may not be possible to work around with standard awk. An END rule is automatically added to the program calling assert. Normally, if a program consists of just a BEGIN rule, the input files and/or standard input are not read. However, now that the program has an END rule, awk will attempt to read the input data files, or standard input (see section Startup and Cleanup Actions), most likely causing the program to hang, waiting for input.

15.4 Rounding Numbers

The way printf and sprintf (see section Using printf Statements for Fancier Printing) do rounding will often depend upon the system's C sprintf subroutine. On many machines, sprintf rounding is "unbiased," which means it doesn't always round a trailing `.5' up, contrary to naive expectations. In unbiased rounding, `.5' rounds to even, rather than always up, so 1.5 rounds to 2 but 4.5 rounds to 4. The result is that if you are using a format that does rounding (e.g., "%.0f") you should check what your system does. The following function does traditional rounding; it might be useful if your awk's printf does unbiased rounding.

# round --- do normal rounding # # Arnold Robbins, arnold@gnu.org, August, 1996 # Public Domain function round(x, ival, aval, fraction) { ival = int(x) # integer part, int() truncates # see if fractional part if (ival == x) # no fraction return x if (x < 0) { aval = -x # absolute value ival = int(aval) fraction = aval - ival if (fraction >= .5) return int(x) - 1 # -2.5 --> -3 else return int(x) # -2.3 --> -2 } else { fraction = x - ival if (fraction >= .5) return ival + 1 else return ival } } # test harness { print $0, round($0) }

15.5 Translating Between Characters and Numbers

One commercial implementation of awk supplies a built-in function, ord, which takes a character and returns the numeric value for that character in the machine's character set. If the string passed to ord has more than one character, only the first one is used.

The inverse of this function is chr (from the function of the same name in Pascal), which takes a number and returns the corresponding character.

Both functions can be written very nicely in awk; there is no real reason to build them into the awk interpreter.

# ord.awk --- do ord and chr # # Global identifiers: # _ord_: numerical values indexed by characters # _ord_init: function to initialize _ord_ # # Arnold Robbins # arnold@gnu.org # Public Domain # 16 January, 1992 # 20 July, 1992, revised BEGIN { _ord_init() } function _ord_init( low, high, i, t) { low = sprintf("%c", 7) # BEL is ascii 7 if (low == "\a") { # regular ascii low = 0 high = 127 } else if (sprintf("%c", 128 + 7) == "\a") { # ascii, mark parity low = 128 high = 255 } else { # ebcdic(!) low = 0 high = 255 } for (i = low; i <= high; i++) { t = sprintf("%c", i) _ord_[t] = i } }

Some explanation of the numbers used by chr is worthwhile. The most prominent character set in use today is ASCII. Although an eight-bit byte can hold 256 distinct values (from zero to 255), ASCII only defines characters that use the values from zero to 127.(21) At least one computer manufacturer that we know of uses ASCII, but with mark parity, meaning that the leftmost bit in the byte is always one. What this means is that on those systems, characters have numeric values from 128 to 255. Finally, large mainframe systems use the EBCDIC character set, which uses all 256 values. While there are other character sets in use on some older systems, they are not really worth worrying about.

function ord(str, c) { # only first character is of interest c = substr(str, 1, 1) return _ord_[c] } function chr(c) { # force c to be numeric by adding 0 return sprintf("%c", c + 0) } #### test code #### # BEGIN \ # { # for (;;) { # printf("enter a character: ") # if (getline var <= 0) # break # printf("ord(%s) = %d\n", var, ord(var)) # } # }

An obvious improvement to these functions would be to move the code for the _ord_init function into the body of the BEGIN rule. It was written this way initially for ease of development.

There is a "test program" in a BEGIN rule, for testing the function. It is commented out for production use.

15.6 Merging an Array Into a String

When doing string processing, it is often useful to be able to join all the strings in an array into one long string. The following function, join, accomplishes this task. It is used later in several of the application programs (see section Practical awk Programs).

Good function design is important; this function needs to be general, but it should also have a reasonable default behavior. It is called with an array and the beginning and ending indices of the elements in the array to be merged. This assumes that the array indices are numeric--a reasonable assumption since the array was likely created with split (see section Built-in Functions for String Manipulation).

# join.awk --- join an array into a string # Arnold Robbins, arnold@gnu.org, Public Domain # May 1993 function join(array, start, end, sep, result, i) { if (sep == "") sep = " " else if (sep == SUBSEP) # magic value sep = "" result = array[start] for (i = start + 1; i <= end; i++) result = result sep array[i] return result }

An optional additional argument is the separator to use when joining the strings back together. If the caller supplies a non-empty value, join uses it. If it is not supplied, it will have a null value. In this case, join uses a single blank as a default separator for the strings. If the value is equal to SUBSEP, then join joins the strings with no separator between them. SUBSEP serves as a "magic" value to indicate that there should be no separation between the component strings.

It would be nice if awk had an assignment operator for concatenation. The lack of an explicit operator for concatenation makes string operations more difficult than they really need to be.

15.7 Turning Dates Into Timestamps

The systime function built in to gawk returns the current time of day as a timestamp in "seconds since the Epoch." This timestamp can be converted into a printable date of almost infinitely variable format using the built-in strftime function. (For more information on systime and strftime, see section Functions for Dealing with Time Stamps.)

An interesting but difficult problem is to convert a readable representation of a date back into a timestamp. The ANSI C library provides a mktime function that does the basic job, converting a canonical representation of a date into a timestamp.

It would appear at first glance that gawk would have to supply a mktime built-in function that was simply a "hook" to the C language version. In fact though, mktime can be implemented entirely in awk.(22)

Here is a version of mktime for awk. It takes a simple representation of the date and time, and converts it into a timestamp.

The code is presented here intermixed with explanatory prose. In Extracting Programs from Texinfo Source Files, you will see how the Texinfo source file for this Info file can be processed to extract the code into a single source file.

The program begins with a descriptive comment and a BEGIN rule that initializes a table _tm_months. This table is a two-dimensional array that has the lengths of the months. The first index is zero for regular years, and one for leap years. The values are the same for all the months in both kinds of years, except for February; thus the use of multiple assignment.

# mktime.awk --- convert a canonical date representation # into a timestamp # Arnold Robbins, arnold@gnu.org, Public Domain # May 1993 BEGIN \ { # Initialize table of month lengths _tm_months[0,1] = _tm_months[1,1] = 31 _tm_months[0,2] = 28; _tm_months[1,2] = 29 _tm_months[0,3] = _tm_months[1,3] = 31 _tm_months[0,4] = _tm_months[1,4] = 30 _tm_months[0,5] = _tm_months[1,5] = 31 _tm_months[0,6] = _tm_months[1,6] = 30 _tm_months[0,7] = _tm_months[1,7] = 31 _tm_months[0,8] = _tm_months[1,8] = 31 _tm_months[0,9] = _tm_months[1,9] = 30 _tm_months[0,10] = _tm_months[1,10] = 31 _tm_months[0,11] = _tm_months[1,11] = 30 _tm_months[0,12] = _tm_months[1,12] = 31 }

The benefit of merging multiple BEGIN rules (see section The BEGIN and END Special Patterns) is particularly clear when writing library files. Functions in library files can cleanly initialize their own private data and also provide clean-up actions in private END rules.

The next function is a simple one that computes whether a given year is or is not a leap year. If a year is evenly divisible by four, but not evenly divisible by 100, or if it is evenly divisible by 400, then it is a leap year. Thus, 1904 was a leap year, 1900 was not, but 2000 will be.

# decide if a year is a leap year function _tm_isleap(year, ret) { ret = (year % 4 == 0 && year % 100 != 0) || (year % 400 == 0) return ret }

This function is only used a few times in this file, and its computation could have been written in-line (at the point where it's used). Making it a separate function made the original development easier, and also avoids the possibility of typing errors when duplicating the code in multiple places.

The next function is more interesting. It does most of the work of generating a timestamp, which is converting a date and time into some number of seconds since the Epoch. The caller passes an array (rather imaginatively named a) containing six values: the year including century, the month as a number between one and 12, the day of the month, the hour as a number between zero and 23, the minute in the hour, and the seconds within the minute.

The function uses several local variables to precompute the number of seconds in an hour, seconds in a day, and seconds in a year. Often, similar C code simply writes out the expression in-line, expecting the compiler to do constant folding. E.g., most C compilers would turn `60 * 60' into `3600' at compile time, instead of recomputing it every time at run time. Precomputing these values makes the function more efficient.

# convert a date into seconds function _tm_addup(a, total, yearsecs, daysecs, hoursecs, i, j) { hoursecs = 60 * 60 daysecs = 24 * hoursecs yearsecs = 365 * daysecs total = (a[1] - 1970) * yearsecs # extra day for leap years for (i = 1970; i < a[1]; i++) if (_tm_isleap(i)) total += daysecs j = _tm_isleap(a[1]) for (i = 1; i < a[2]; i++) total += _tm_months[j, i] * daysecs total += (a[3] - 1) * daysecs total += a[4] * hoursecs total += a[5] * 60 total += a[6] return total }

The function starts with a first approximation of all the seconds between Midnight, January 1, 1970,(23) and the beginning of the current year. It then goes through all those years, and for every leap year, adds an additional day's worth of seconds.

The variable j holds either one or zero, if the current year is or is not a leap year. For every month in the current year prior to the current month, it adds the number of seconds in the month, using the appropriate entry in the _tm_months array.

Finally, it adds in the seconds for the number of days prior to the current day, and the number of hours, minutes, and seconds in the current day.

The result is a count of seconds since January 1, 1970. This value is not yet what is needed though. The reason why is described shortly.

The main mktime function takes a single character string argument. This string is a representation of a date and time in a "canonical" (fixed) form. This string should be "year month day hour minute second".

# mktime --- convert a date into seconds, # compensate for time zone function mktime(str, res1, res2, a, b, i, j, t, diff) { i = split(str, a, " ") # don't rely on FS if (i != 6) return -1 # force numeric for (j in a) a[j] += 0 # validate if (a[1] < 1970 || a[2] < 1 || a[2] > 12 || a[3] < 1 || a[3] > 31 || a[4] < 0 || a[4] > 23 || a[5] < 0 || a[5] > 59 || a[6] < 0 || a[6] > 60 ) return -1 res1 = _tm_addup(a) t = strftime("%Y %m %d %H %M %S", res1) if (_tm_debug) printf("(%s) -> (%s)\n", str, t) > "/dev/stderr" split(t, b, " ") res2 = _tm_addup(b) diff = res1 - res2 if (_tm_debug) printf("diff = %d seconds\n", diff) > "/dev/stderr" res1 += diff return res1 }

The function first splits the string into an array, using spaces and tabs as separators. If there are not six elements in the array, it returns an error, signaled as the value -1. Next, it forces each element of the array to be numeric, by adding zero to it. The following `if' statement then makes sure that each element is within an allowable range. (This checking could be extended further, e.g., to make sure that the day of the month is within the correct range for the particular month supplied.) All of this is essentially preliminary set-up and error checking.

Recall that _tm_addup generated a value in seconds since Midnight, January 1, 1970. This value is not directly usable as the result we want, since the calculation does not account for the local timezone. In other words, the value represents the count in seconds since the Epoch, but only for UTC (Universal Coordinated Time). If the local timezone is east or west of UTC, then some number of hours should be either added to, or subtracted from the resulting timestamp.

For example, 6:23 p.m. in Atlanta, Georgia (USA), is normally five hours west of (behind) UTC. It is only four hours behind UTC if daylight savings time is in effect. If you are calling mktime in Atlanta, with the argument "1993 5 23 18 23 12", the result from _tm_addup will be for 6:23 p.m. UTC, which is only 2:23 p.m. in Atlanta. It is necessary to add another four hours worth of seconds to the result.

How can mktime determine how far away it is from UTC? This is surprisingly easy. The returned timestamp represents the time passed to mktime as UTC. This timestamp can be fed back to strftime, which will format it as a local time; i.e. as if it already had the UTC difference added in to it. This is done by giving "%Y %m %d %H %M %S" to strftime as the format argument. It returns the computed timestamp in the original string format. The result represents a time that accounts for the UTC difference. When the new time is converted back to a timestamp, the difference between the two timestamps is the difference (in seconds) between the local timezone and UTC. This difference is then added back to the original result. An example demonstrating this is presented below.

Finally, there is a "main" program for testing the function.

BEGIN { if (_tm_test) { printf "Enter date as yyyy mm dd hh mm ss: " getline _tm_test_date t = mktime(_tm_test_date) r = strftime("%Y %m %d %H %M %S", t) printf "Got back (%s)\n", r } }

The entire program uses two variables that can be set on the command line to control debugging output and to enable the test in the final BEGIN rule. Here is the result of a test run. (Note that debugging output is to standard error, and test output is to standard output.)

$ gawk -f mktime.awk -v _tm_test=1 -v _tm_debug=1 -| Enter date as yyyy mm dd hh mm ss: 1993 5 23 15 35 10 error--> (1993 5 23 15 35 10) -> (1993 05 23 11 35 10) error--> diff = 14400 seconds -| Got back (1993 05 23 15 35 10)

The time entered was 3:35 p.m. (15:35 on a 24-hour clock), on May 23, 1993. The first line of debugging output shows the resulting time as UTC--four hours ahead of the local time zone. The second line shows that the difference is 14400 seconds, which is four hours. (The difference is only four hours, since daylight savings time is in effect during May.) The final line of test output shows that the timezone compensation algorithm works; the returned time is the same as the entered time.

This program does not solve the general problem of turning an arbitrary date representation into a timestamp. That problem is very involved. However, the mktime function provides a foundation upon which to build. Other software can convert month names into numeric months, and AM/PM times into 24-hour clocks, to generate the "canonical" format that mktime requires.

15.8 Managing the Time of Day

The systime and strftime functions described in Functions for Dealing with Time Stamps, provide the minimum functionality necessary for dealing with the time of day in human readable form. While strftime is extensive, the control formats are not necessarily easy to remember or intuitively obvious when reading a program.

The following function, gettimeofday, populates a user-supplied array with pre-formatted time information. It returns a string with the current time formatted in the same way as the date utility.

# gettimeofday --- get the time of day in a usable format # Arnold Robbins, arnold@gnu.org, Public Domain, May 1993 # # Returns a string in the format of output of date(1) # Populates the array argument time with individual values: # time["second"] -- seconds (0 - 59) # time["minute"] -- minutes (0 - 59) # time["hour"] -- hours (0 - 23) # time["althour"] -- hours (0 - 12) # time["monthday"] -- day of month (1 - 31) # time["month"] -- month of year (1 - 12) # time["monthname"] -- name of the month # time["shortmonth"] -- short name of the month # time["year"] -- year within century (0 - 99) # time["fullyear"] -- year with century (19xx or 20xx) # time["weekday"] -- day of week (Sunday = 0) # time["altweekday"] -- day of week (Monday = 0) # time["weeknum"] -- week number, Sunday first day # time["altweeknum"] -- week number, Monday first day # time["dayname"] -- name of weekday # time["shortdayname"] -- short name of weekday # time["yearday"] -- day of year (0 - 365) # time["timezone"] -- abbreviation of timezone name # time["ampm"] -- AM or PM designation function gettimeofday(time, ret, now, i) { # get time once, avoids unnecessary system calls now = systime() # return date(1)-style output ret = strftime("%a %b %d %H:%M:%S %Z %Y", now) # clear out target array for (i in time) delete time[i] # fill in values, force numeric values to be # numeric by adding 0 time["second"] = strftime("%S", now) + 0 time["minute"] = strftime("%M", now) + 0 time["hour"] = strftime("%H", now) + 0 time["althour"] = strftime("%I", now) + 0 time["monthday"] = strftime("%d", now) + 0 time["month"] = strftime("%m", now) + 0 time["monthname"] = strftime("%B", now) time["shortmonth"] = strftime("%b", now) time["year"] = strftime("%y", now) + 0 time["fullyear"] = strftime("%Y", now) + 0 time["weekday"] = strftime("%w", now) + 0 time["altweekday"] = strftime("%u", now) + 0 time["dayname"] = strftime("%A", now) time["shortdayname"] = strftime("%a", now) time["yearday"] = strftime("%j", now) + 0 time["timezone"] = strftime("%Z", now) time["ampm"] = strftime("%p", now) time["weeknum"] = strftime("%U", now) + 0 time["altweeknum"] = strftime("%W", now) + 0 return ret }

The string indices are easier to use and read than the various formats required by strftime. The alarm program presented in An Alarm Clock Program, uses this function.

The gettimeofday function is presented above as it was written. A more general design for this function would have allowed the user to supply an optional timestamp value that would have been used instead of the current time.

15.9 Noting Data File Boundaries

The BEGIN and END rules are each executed exactly once, at the beginning and end respectively of your awk program (see section The BEGIN and END Special Patterns). We (the gawk authors) once had a user who mistakenly thought that the BEGIN rule was executed at the beginning of each data file and the END rule was executed at the end of each data file. When informed that this was not the case, the user requested that we add new special patterns to gawk, named BEGIN_FILE and END_FILE, that would have the desired behavior. He even supplied us the code to do so.

However, after a little thought, I came up with the following library program. It arranges to call two user-supplied functions, beginfile and endfile, at the beginning and end of each data file. Besides solving the problem in only nine(!) lines of code, it does so portably; this will work with any implementation of awk.

# transfile.awk # # Give the user a hook for filename transitions # # The user must supply functions beginfile() and endfile() # that each take the name of the file being started or # finished, respectively. # # Arnold Robbins, arnold@gnu.org, January 1992 # Public Domain FILENAME != _oldfilename \ { if (_oldfilename != "") endfile(_oldfilename) _oldfilename = FILENAME beginfile(FILENAME) } END { endfile(FILENAME) }

This file must be loaded before the user's "main" program, so that the rule it supplies will be executed first.

This rule relies on awk's FILENAME variable that automatically changes for each new data file. The current file name is saved in a private variable, _oldfilename. If FILENAME does not equal _oldfilename, then a new data file is being processed, and it is necessary to call endfile for the old file. Since endfile should only be called if a file has been processed, the program first checks to make sure that _oldfilename is not the null string. The program then assigns the current file name to _oldfilename, and calls beginfile for the file. Since, like all awk variables, _oldfilename will be initialized to the null string, this rule executes correctly even for the first data file.

The program also supplies an END rule, to do the final processing for the last file. Since this END rule comes before any END rules supplied in the "main" program, endfile will be called first. Once again the value of multiple BEGIN and END rules should be clear.

This version has same problem as the first version of nextfile (see section Implementing nextfile as a Function). If the same data file occurs twice in a row on command line, then endfile and beginfile will not be executed at the end of the first pass and at the beginning of the second pass. This version solves the problem.

# ftrans.awk --- handle data file transitions # # user supplies beginfile() and endfile() functions # # Arnold Robbins, arnold@gnu.org, November 1992 # Public Domain FNR == 1 { if (_filename_ != "") endfile(_filename_) _filename_ = FILENAME beginfile(FILENAME) } END { endfile(_filename_) }

In Counting Things, you will see how this library function can be used, and how it simplifies writing the main program.

15.10 Processing Command Line Options

Most utilities on POSIX compatible systems take options or "switches" on the command line that can be used to change the way a program behaves. awk is an example of such a program (see section Command Line Options). Often, options take arguments, data that the program needs to correctly obey the command line option. For example, awk's `-F' option requires a string to use as the field separator. The first occurrence on the command line of either `--' or a string that does not begin with `-' ends the options.

Most Unix systems provide a C function named getopt for processing command line arguments. The programmer provides a string describing the one letter options. If an option requires an argument, it is followed in the string with a colon. getopt is also passed the count and values of the command line arguments, and is called in a loop. getopt processes the command line arguments for option letters. Each time around the loop, it returns a single character representing the next option letter that it found, or `?' if it found an invalid option. When it returns -1, there are no options left on the command line.

When using getopt, options that do not take arguments can be grouped together. Furthermore, options that take arguments require that the argument be present. The argument can immediately follow the option letter, or it can be a separate command line argument.

Given a hypothetical program that takes three command line options, `-a', `-b', and `-c', and `-b' requires an argument, all of the following are valid ways of invoking the program:

prog -a -b foo -c data1 data2 data3 prog -ac -bfoo -- data1 data2 data3 prog -acbfoo data1 data2 data3

Notice that when the argument is grouped with its option, the rest of the command line argument is considered to be the option's argument. In the above example, `-acbfoo' indicates that all of the `-a', `-b', and `-c' options were supplied, and that `foo' is the argument to the `-b' option.

getopt provides four external variables that the programmer can use.

optind

The index in the argument value array (argv) where the first non-option command line argument can be found.

optarg

The string value of the argument to an option.

opterr

Usually getopt prints an error message when it finds an invalid option. Setting opterr to zero disables this feature. (An application might wish to print its own error message.)

optopt

The letter representing the command line option. While not usually documented, most versions supply this variable.

The following C fragment shows how getopt might process command line arguments for awk.

int main(int argc, char *argv[]) { ... /* print our own message */ opterr = 0; while ((c = getopt(argc, argv, "v:f:F:W:")) != -1) { switch (c) { case 'f': /* file */ ... break; case 'F': /* field separator */ ... break; case 'v': /* variable assignment */ ... break; case 'W': /* extension */ ... break; case '?': default: usage(); break; } } ... }

As a side point, gawk actually uses the GNU getopt_long function to process both normal and GNU-style long options (see section Command Line Options).

The abstraction provided by getopt is very useful, and would be quite handy in awk programs as well. Here is an awk version of getopt. This function highlights one of the greatest weaknesses in awk, which is that it is very poor at manipulating single characters. Repeated calls to substr are necessary for accessing individual characters (see section Built-in Functions for String Manipulation).

The discussion walks through the code a bit at a time.

# getopt --- do C library getopt(3) function in awk # # arnold@gnu.org # Public domain # # Initial version: March, 1991 # Revised: May, 1993 # External variables: # Optind -- index of ARGV for first non-option argument # Optarg -- string value of argument to current option # Opterr -- if non-zero, print our own diagnostic # Optopt -- current option letter # Returns # -1 at end of options # ? for unrecognized option # <c> a character representing the current option # Private Data # _opti index in multi-flag option, e.g., -abc

The function starts out with some documentation: who wrote the code, and when it was revised, followed by a list of the global variables it uses, what the return values are and what they mean, and any global variables that are "private" to this library function. Such documentation is essential for any program, and particularly for library functions.

function getopt(argc, argv, options, optl, thisopt, i) { optl = length(options) if (optl == 0) # no options given return -1 if (argv[Optind] == "--") { # all done Optind++ _opti = 0 return -1 } else if (argv[Optind] !~ /^-[^: \t\n\f\r\v\b]/) { _opti = 0 return -1 }

The function first checks that it was indeed called with a string of options (the options parameter). If options has a zero length, getopt immediately returns -1.

The next thing to check for is the end of the options. A `--' ends the command line options, as does any command line argument that does not begin with a `-'. Optind is used to step through the array of command line arguments; it retains its value across calls to getopt, since it is a global variable.

The regexp used, /^-[^: \t\n\f\r\v\b]/, is perhaps a bit of overkill; it checks for a `-' followed by anything that is not whitespace and not a colon. If the current command line argument does not match this pattern, it is not an option, and it ends option processing.

if (_opti == 0) _opti = 2 thisopt = substr(argv[Optind], _opti, 1) Optopt = thisopt i = index(options, thisopt) if (i == 0) { if (Opterr) printf("%c -- invalid option\n", thisopt) > "/dev/stderr" if (_opti >= length(argv[Optind])) { Optind++ _opti = 0 } else _opti++ return "?" }

The _opti variable tracks the position in the current command line argument (argv[Optind]). In the case that multiple options were grouped together with one `-' (e.g., `-abx'), it is necessary to return them to the user one at a time.

If _opti is equal to zero, it is set to two, the index in the string of the next character to look at (we skip the `-', which is at position one). The variable thisopt holds the character, obtained with substr. It is saved in Optopt for the main program to use.

If thisopt is not in the options string, then it is an invalid option. If Opterr is non-zero, getopt prints an error message on the standard error that is similar to the message from the C version of getopt.

Since the option is invalid, it is necessary to skip it and move on to the next option character. If _opti is greater than or equal to the length of the current command line argument, then it is necessary to move on to the next one, so Optind is incremented and _opti is reset to zero. Otherwise, Optind is left alone and _opti is merely incremented.

In any case, since the option was invalid, getopt returns `?'. The main program can examine Optopt if it needs to know what the invalid option letter actually was.

if (substr(options, i + 1, 1) == ":") { # get option argument if (length(substr(argv[Optind], _opti + 1)) > 0) Optarg = substr(argv[Optind], _opti + 1) else Optarg = argv[++Optind] _opti = 0 } else Optarg = ""

If the option requires an argument, the option letter is followed by a colon in the options string. If there are remaining characters in the current command line argument (argv[Optind]), then the rest of that string is assigned to Optarg. Otherwise, the next command line argument is used (`-xFOO' vs. `-x FOO'). In either case, _opti is reset to zero, since there are no more characters left to examine in the current command line argument.

if (_opti == 0 || _opti >= length(argv[Optind])) { Optind++ _opti = 0 } else _opti++ return thisopt }

Finally, if _opti is either zero or greater than the length of the current command line argument, it means this element in argv is through being processed, so Optind is incremented to point to the next element in argv. If neither condition is true, then only _opti is incremented, so that the next option letter can be processed on the next call to getopt.

BEGIN { Opterr = 1 # default is to diagnose Optind = 1 # skip ARGV[0] # test program if (_getopt_test) { while ((_go_c = getopt(ARGC, ARGV, "ab:cd")) != -1) printf("c = <%c>, optarg = <%s>\n", _go_c, Optarg) printf("non-option arguments:\n") for (; Optind < ARGC; Optind++) printf("\tARGV[%d] = <%s>\n", Optind, ARGV[Optind]) } }

The BEGIN rule initializes both Opterr and Optind to one. Opterr is set to one, since the default behavior is for getopt to print a diagnostic message upon seeing an invalid option. Optind is set to one, since there's no reason to look at the program name, which is in ARGV[0].

The rest of the BEGIN rule is a simple test program. Here is the result of two sample runs of the test program.

$ awk -f getopt.awk -v _getopt_test=1 -- -a -cbARG bax -x -| c = <a>, optarg = <> -| c = <c>, optarg = <> -| c = <b>, optarg = <ARG> -| non-option arguments: -| ARGV[3] = <bax> -| ARGV[4] = <-x> $ awk -f getopt.awk -v _getopt_test=1 -- -a -x -- xyz abc -| c = <a>, optarg = <> error--> x -- invalid option -| c = <?>, optarg = <> -| non-option arguments: -| ARGV[4] = <xyz> -| ARGV[5] = <abc>

The first `--' terminates the arguments to awk, so that it does not try to interpret the `-a' etc. as its own options.

Several of the sample programs presented in Practical awk Programs, use getopt to process their arguments.

15.11 Reading the User Database

The `/dev/user' special file (see section Special File Names in gawk) provides access to the current user's real and effective user and group id numbers, and if available, the user's supplementary group set. However, since these are numbers, they do not provide very useful information to the average user. There needs to be some way to find the user information associated with the user and group numbers. This section presents a suite of functions for retrieving information from the user database. See section Reading the Group Database, for a similar suite that retrieves information from the group database.

The POSIX standard does not define the file where user information is kept. Instead, it provides the <pwd.h> header file and several C language subroutines for obtaining user information. The primary function is getpwent, for "get password entry." The "password" comes from the original user database file, `/etc/passwd', which kept user information, along with the encrypted passwords (hence the name).

While an awk program could simply read `/etc/passwd' directly (the format is well known), because of the way password files are handled on networked systems, this file may not contain complete information about the system's set of users.

To be sure of being able to produce a readable, complete version of the user database, it is necessary to write a small C program that calls getpwent. getpwent is defined to return a pointer to a struct passwd. Each time it is called, it returns the next entry in the database. When there are no more entries, it returns NULL, the null pointer. When this happens, the C program should call endpwent to close the database. Here is pwcat, a C program that "cats" the password database.

/* * pwcat.c * * Generate a printable version of the password database * * Arnold Robbins * arnold@gnu.org * May 1993 * Public Domain */ #include <stdio.h> #include <pwd.h> int main(argc, argv) int argc; char **argv; { struct passwd *p; while ((p = getpwent()) != NULL) printf("%s:%s:%d:%d:%s:%s:%s\n", p->pw_name, p->pw_passwd, p->pw_uid, p->pw_gid, p->pw_gecos, p->pw_dir, p->pw_shell); endpwent(); exit(0); }

If you don't understand C, don't worry about it. The output from pwcat is the user database, in the traditional `/etc/passwd' format of colon-separated fields. The fields are:

Login name

The user's login name.

Encrypted password

The user's encrypted password. This may not be available on some systems.

User-ID

The user's numeric user-id number.

Group-ID

The user's numeric group-id number.

Full name

The user's full name, and perhaps other information associated with the user.

Home directory

The user's login, or "home" directory (familiar to shell programmers as $HOME).

Login shell

The program that will be run when the user logs in. This is usually a shell, such as Bash (the Gnu Bourne-Again shell).

Here are a few lines representative of pwcat's output.

$ pwcat -| root:3Ov02d5VaUPB6:0:1:Operator:/:/bin/sh -| nobody:*:65534:65534::/: -| daemon:*:1:1::/: -| sys:*:2:2::/:/bin/csh -| bin:*:3:3::/bin: -| arnold:xyzzy:2076:10:Arnold Robbins:/home/arnold:/bin/sh -| miriam:yxaay:112:10:Miriam Robbins:/home/miriam:/bin/sh -| andy:abcca2:113:10:Andy Jacobs:/home/andy:/bin/sh ...

With that introduction, here is a group of functions for getting user information. There are several functions here, corresponding to the C functions of the same name.

# passwd.awk --- access password file information # Arnold Robbins, arnold@gnu.org, Public Domain # May 1993 BEGIN { # tailor this to suit your system _pw_awklib = "/usr/local/libexec/awk/" } function _pw_init( oldfs, oldrs, olddol0, pwcat) { if (_pw_inited) return oldfs = FS oldrs = RS olddol0 = $0 FS = ":" RS = "\n" pwcat = _pw_awklib "pwcat" while ((pwcat | getline) > 0) { _pw_byname[$1] = $0 _pw_byuid[$3] = $0 _pw_bycount[++_pw_total] = $0 } close(pwcat) _pw_count = 0 _pw_inited = 1 FS = oldfs RS = oldrs $0 = olddol0 }

The BEGIN rule sets a private variable to the directory where pwcat is stored. Since it is used to help out an awk library routine, we have chosen to put it in `/usr/local/libexec/awk'. You might want it to be in a different directory on your system.

The function _pw_init keeps three copies of the user information in three associative arrays. The arrays are indexed by user name (_pw_byname), by user-id number (_pw_byuid), and by order of occurrence (_pw_bycount).

The variable _pw_inited is used for efficiency; _pw_init only needs to be called once.

Since this function uses getline to read information from pwcat, it first saves the values of FS, RS, and $0. Doing so is necessary, since these functions could be called from anywhere within a user's program, and the user may have his or her own values for FS and RS.

The main part of the function uses a loop to read database lines, split the line into fields, and then store the line into each array as necessary. When the loop is done, _pw_init cleans up by closing the pipeline, setting _pw_inited to one, and restoring FS, RS, and $0. The use of _pw_count will be explained below.

function getpwnam(name) { _pw_init() if (name in _pw_byname) return _pw_byname[name] return "" }

The getpwnam function takes a user name as a string argument. If that user is in the database, it returns the appropriate line. Otherwise it returns the null string.

function getpwuid(uid) { _pw_init() if (uid in _pw_byuid) return _pw_byuid[uid] return "" }

Similarly, the getpwuid function takes a user-id number argument. If that user number is in the database, it returns the appropriate line. Otherwise it returns the null string.

function getpwent() { _pw_init() if (_pw_count < _pw_total) return _pw_bycount[++_pw_count] return "" }

The getpwent function simply steps through the database, one entry at a time. It uses _pw_count to track its current position in the _pw_bycount array.

function endpwent() { _pw_count = 0 }

The endpwent function resets _pw_count to zero, so that subsequent calls to getpwent will start over again.

A conscious design decision in this suite is that each subroutine calls _pw_init to initialize the database arrays. The overhead of running a separate process to generate the user database, and the I/O to scan it, will only be incurred if the user's main program actually calls one of these functions. If this library file is loaded along with a user's program, but none of the routines are ever called, then there is no extra run-time overhead. (The alternative would be to move the body of _pw_init into a BEGIN rule, which would always run pwcat. This simplifies the code but runs an extra process that may never be needed.)

In turn, calling _pw_init is not too expensive, since the _pw_inited variable keeps the program from reading the data more than once. If you are worried about squeezing every last cycle out of your awk program, the check of _pw_inited could be moved out of _pw_init and duplicated in all the other functions. In practice, this is not necessary, since most awk programs are I/O bound, and it would clutter up the code.

The id program in Printing Out User Information, uses these functions.

15.12 Reading the Group Database

Much of the discussion presented in Reading the User Database, applies to the group database as well. Although there has traditionally been a well known file, `/etc/group', in a well known format, the POSIX standard only provides a set of C library routines (<grp.h> and getgrent) for accessing the information. Even though this file may exist, it likely does not have complete information. Therefore, as with the user database, it is necessary to have a small C program that generates the group database as its output.

Here is grcat, a C program that "cats" the group database.

/* * grcat.c * * Generate a printable version of the group database * * Arnold Robbins, arnold@gnu.org * May 1993 * Public Domain */ #include <stdio.h> #include <grp.h> int main(argc, argv) int argc; char **argv; { struct group *g; int i; while ((g = getgrent()) != NULL) { printf("%s:%s:%d:", g->gr_name, g->gr_passwd, g->gr_gid); for (i = 0; g->gr_mem[i] != NULL; i++) { printf("%s", g->gr_mem[i]); if (g->gr_mem[i+1] != NULL) putchar(','); } putchar('\n'); } endgrent(); exit(0); }

Each line in the group database represent one group. The fields are separated with colons, and represent the following information.

Group Name

The name of the group.

Group Password

The encrypted group password. In practice, this field is never used. It is usually empty, or set to `*'.

Group ID Number

The numeric group-id number. This number should be unique within the file.

Group Member List

A comma-separated list of user names. These users are members of the group. Most Unix systems allow users to be members of several groups simultaneously. If your system does, then reading `/dev/user' will return those group-id numbers in $5 through $NF. (Note that `/dev/user' is a gawk extension; see section Special File Names in gawk.)

Here is what running grcat might produce:

$ grcat -| wheel:*:0:arnold -| nogroup:*:65534: -| daemon:*:1: -| kmem:*:2: -| staff:*:10:arnold,miriam,andy -| other:*:20: ...

Here are the functions for obtaining information from the group database. There are several, modeled after the C library functions of the same names.

# group.awk --- functions for dealing with the group file # Arnold Robbins, arnold@gnu.org, Public Domain # May 1993 BEGIN \ { # Change to suit your system _gr_awklib = "/usr/local/libexec/awk/" } function _gr_init( oldfs, oldrs, olddol0, grcat, n, a, i) { if (_gr_inited) return oldfs = FS oldrs = RS olddol0 = $0 FS = ":" RS = "\n" grcat = _gr_awklib "grcat" while ((grcat | getline) > 0) { if ($1 in _gr_byname) _gr_byname[$1] = _gr_byname[$1] "," $4 else _gr_byname[$1] = $0 if ($3 in _gr_bygid) _gr_bygid[$3] = _gr_bygid[$3] "," $4 else _gr_bygid[$3] = $0 n = split($4, a, "[ \t]*,[ \t]*") for (i = 1; i <= n; i++) if (a[i] in _gr_groupsbyuser) _gr_groupsbyuser[a[i]] = \ _gr_groupsbyuser[a[i]] " " $1 else _gr_groupsbyuser[a[i]] = $1 _gr_bycount[++_gr_count] = $0 } close(grcat) _gr_count = 0 _gr_inited++ FS = oldfs RS = oldrs $0 = olddol0 }

The BEGIN rule sets a private variable to the directory where grcat is stored. Since it is used to help out an awk library routine, we have chosen to put it in `/usr/local/libexec/awk'. You might want it to be in a different directory on your system.

These routines follow the same general outline as the user database routines (see section Reading the User Database). The _gr_inited variable is used to ensure that the database is scanned no more than once. The _gr_init function first saves FS, RS, and $0, and then sets FS and RS to the correct values for scanning the group information.

The group information is stored is several associative arrays. The arrays are indexed by group name (_gr_byname), by group-id number (_gr_bygid), and by position in the database (_gr_bycount). There is an additional array indexed by user name (_gr_groupsbyuser), that is a space separated list of groups that each user belongs to.

Unlike the user database, it is possible to have multiple records in the database for the same group. This is common when a group has a large number of members. Such a pair of entries might look like:

tvpeople:*:101:johny,jay,arsenio tvpeople:*:101:david,conan,tom,joan

For this reason, _gr_init looks to see if a group name or group-id number has already been seen. If it has, then the user names are simply concatenated onto the previous list of users. (There is actually a subtle problem with the code presented above. Suppose that the first time there were no names. This code adds the names with a leading comma. It also doesn't check that there is a $4.)

Finally, _gr_init closes the pipeline to grcat, restores FS, RS, and $0, initializes _gr_count to zero (it is used later), and makes _gr_inited non-zero.

function getgrnam(group) { _gr_init() if (group in _gr_byname) return _gr_byname[group] return "" }

The getgrnam function takes a group name as its argument, and if that group exists, it is returned. Otherwise, getgrnam returns the null string.

function getgrgid(gid) { _gr_init() if (gid in _gr_bygid) return _gr_bygid[gid] return "" }

The getgrgid function is similar, it takes a numeric group-id, and looks up the information associated with that group-id.

function getgruser(user) { _gr_init() if (user in _gr_groupsbyuser) return _gr_groupsbyuser[user] return "" }

The getgruser function does not have a C counterpart. It takes a user name, and returns the list of groups that have the user as a member.

function getgrent() { _gr_init() if (++_gr_count in _gr_bycount) return _gr_bycount[_gr_count] return "" }

The getgrent function steps through the database one entry at a time. It uses _gr_count to track its position in the list.

function endgrent() { _gr_count = 0 }

endgrent resets _gr_count to zero so that getgrent can start over again.

As with the user database routines, each function calls _gr_init to initialize the arrays. Doing so only incurs the extra overhead of running grcat if these functions are used (as opposed to moving the body of _gr_init into a BEGIN rule).

Most of the work is in scanning the database and building the various associative arrays. The functions that the user calls are themselves very simple, relying on awk's associative arrays to do work.

The id program in Printing Out User Information, uses these functions.

15.13 Naming Library Function Global Variables

Due to the way the awk language evolved, variables are either global (usable by the entire program), or local (usable just by a specific function). There is no intermediate state analogous to static variables in C.

Library functions often need to have global variables that they can use to preserve state information between calls to the function. For example, getopt's variable _opti (see section Processing Command Line Options), and the _tm_months array used by mktime (see section Turning Dates Into Timestamps). Such variables are called private, since the only functions that need to use them are the ones in the library.

When writing a library function, you should try to choose names for your private variables so that they will not conflict with any variables used by either another library function or a user's main program. For example, a name like `i' or `j' is not a good choice, since user programs often use variable names like these for their own purposes.

The example programs shown in this chapter all start the names of their private variables with an underscore (`_'). Users generally don't use leading underscores in their variable names, so this convention immediately decreases the chances that the variable name will be accidentally shared with the user's program.

In addition, several of the library functions use a prefix that helps indicate what function or set of functions uses the variables. For example, _tm_months in mktime (see section Turning Dates Into Timestamps), and _pw_byname in the user data base routines (see section Reading the User Database). This convention is recommended, since it even further decreases the chance of inadvertent conflict among variable names. Note that this convention can be used equally well both for variable names and for private function names too.

While I could have re-written all the library routines to use this convention, I did not do so, in order to show how my own awk programming style has evolved, and to provide some basis for this discussion.

As a final note on variable naming, if a function makes global variables available for use by a main program, it is a good convention to start that variable's name with a capital letter. For example, getopt's Opterr and Optind variables (see section Processing Command Line Options). The leading capital letter indicates that it is global, while the fact that the variable name is not all capital letters indicates that the variable is not one of awk's built-in variables, like FS.

It is also important that all variables in library functions that do not need to save state are in fact declared local. If this is not done, the variable could accidentally be used in the user's program, leading to bugs that are very difficult to track down.

function lib_func(x, y, l1, l2) { ... use variable some_var # some_var could be local ... # but is not by oversight }

A different convention, common in the Tcl community, is to use a single associative array to hold the values needed by the library function(s), or "package." This significantly decreases the number of actual global names in use. For example, the functions described in Reading the User Database, might have used PW_data["inited"], PW_data["total"], PW_data["count"] and PW_data["awklib"], instead of _pw_inited, _pw_awklib, _pw_total, and _pw_count.

The conventions presented in this section are exactly that, conventions. You are not required to write your programs this way, we merely recommend that you do so.