"Linux Gazette...making Linux just a little more fun!"

An Introduction to awk

By Jose Nazario

Abstract

The awk programming language often gets overlooked in the face of Perl, which is a more capable language. However, awk is found even more ubiquitously than Perl, has a less steep learning curve than Perl, and can be used just about everywhere in system monitoring scripts where efficiency is key. This brief tutorial is designed to help you get started in awk programming.

The Basics

The awk language is a small, C style language which was designed for the processing of regularly formatted text. This usually includes database dumps and system log files. It's built around regular expressions and pattern handling, much like Perl is. In fact, Perl is considered to be a grandchild of the awk language.

The funny name of the awk language is due to the names of its original authors, who were Alfred V. Aho, Brian W. Kernighan, and Peter J. Weinberger. Most of you will recognize the name of Kernighan, one of the fathers of the C programming language and a major force in the UNIX world.

Using awk in a One Liner

This is how I began using awk, to print specific fields in output. This would work surprisingly well, but the efficiency went through the floor when I was writing large scripts that took minutes to complete.

But, here you go, this can be useful sometimes:

ls -l /tmp/foobar | awk '{print $1"\t"$9}'

What this will do is take some input like this:


-rw-rw-rw-   1 root     root            1 Jul 14  1997 tmpmsg

and will generate some output like this:


-rw-rw-rw-      tmpmsg

Quite intuitive what it just did, it printed only the first and ninth fields. Now you can see why it's so popular for one line data extraction. But, let's move on to a full fledged awk program.

An awk Program Structure

One of my favorite things about awk is the amazing readability of it, despite it's power compared to Perl or Python. Every awk program has three parts: a BEGIN block, which is executed once before any input is read; a main loop which is executed for every line of input; and an END block, which is executed after all of the input is read. Quite intuitive! Yes, I'll keep saying that about awk, because i find it to be very true.

This is a very simple awk program highlighting some of the features of the language. See if you can pick out what is happening before we dissect it:


#!/usr/bin/awk -f
#
# check the sulog for failures..
# copyright 2001 (c) jose nazario
#
# works for Solaris, IRIX and HPUX 10.20
BEGIN {
  print "--- checking sulog"
  failed=0
  }
{
  if ($4 == "-") {
    print "failed su:\t"$6"\tat\t"$2"\t"$3
    failed=failed+1
    }
}
END {
  print "---------------------------------------"
  printf("\ttotal number of records:\t%d\n", NR)
  printf("\ttotal number of failed su's:\t%d\n",failed)
}

Have you figured it out yet? Would it help to know the format of a typical line of the input file (sulog, from, say, IRIX)? Here's a typical pair of lines:


        SU 01/30 13:15 - ttyq1 jose-root
        SU 01/30 13:15 + ttyq1 jose-root

OK, read up and see if you can figure out the script. The BEGIN block sets everything up, printing out a header and initializing our one variable (in this case failed) to zero. The main loop then reads each line of input (which is the sulog file, a log of su attempts) and compares field four against the minu sign. If they match, it is because the attempt failed, so we increment out counter by one and note which attempt failed and when. At the end final tallies are presented, showing the total number of lines of input as the number of records (NR, an internal awk variable) and the number of failed su attempts we noted. Output looks like this:


failed su:      jose-root       at      01/30   13:15
        ---------------------------------------
        total number of records:        272
        total number of failed su's:    73

You should also be able to see how printf works, almost exactly like the printf does in C. In short, awk is a rather intuitive language.

By default the field separator is whitespace, but you can tweak that. In password files I set it to be a colon. This small script looks for users with an ID of 0 (root equivilent) and no passwords:


#!/usr/bin/awk -f
BEGIN { FS=":" }
{
  if ($3 == 0) print $1
  if ($2 == "") print $1
}

Other internals from awk to know and use are RS for record separator (defaults to a newline, or \n), OFS for output field separator (defaults to nothing, I think) and ORS (defaults to a newline), for output record separator. These can all be set within the script, of course.

Regular Expressions

The awk language matches normal regular expressions that you have come to know and love, and does so better than grep. For instance, I use the following awk search pattern to look for the presence of a likely exploit on Intel Linux systems:


#!/usr/bin/awk -f
{ if ($0 ~ /\x90/) print "exploit at line " NR }

You can't look for hex value 0x90 in grep, but 0x90 is popular in Intel exploits -- its the NOP call, which is used as padding in shellcode portions.

You can look for hex values using \xdd, where dd is the hex number to look for; you can look for decimal (ie ASCII) values by looking for \ddd, using the decimal value, and regular expressions based on text will, of course, work.

Random awk bits

Random numbers in awk are readily generated, but there is an interesting caveat. The rand() function does exactly what you would expect it to, it returns a random number, in this case between 0 and 1. You can scale it, of course, to get larger values. Here's some example code to show you this, as well as an interesting bit of behavior:

#!/usr/bin/awk -f
{
  for(i=1;i<=10;i++) 
  print rand(); exit
}

Run that a couple of times and you will see a problem: the random numbers are hardly random, they repeat every time you run it!

So what's the problem? Well we didn't seed the random number generator. Normally, we're used to our random number generator pulling entropy from a good source, like (in Linux) /dev/random. However, awk doesn't do this. To really get random numbers, we should seed our random number generator. This improved code will do this:

#!/usr/bin/awk -f
BEGIN {
  srand()
}
{
  for(i=1;i<=10;i++)
  print rand(); exit
}

The seeding of the random number generator in the BEGIN block is what does the trick. The function srand() can take an argument, and in the absence of one the current date and time is used to seed the generator. Note that the same seed will always produce the same 'random' sequence.

Conclusion

This isn't the most detailed intro to awk you will find, but I hope that it is more clear to you how to use awk in a program setting. Myself, I'm quite happy programming in awk, and I've got a lot more to learn.

We haven't even touched upon arrays, self built functions or other complex language features, but suffice it to say awk is hardly Perl's little brother.

Go forth and awk!

Resources

Kernighan's homepage contains a list of good awk books as well as the source for the 'one true awk', aka "nawk". It also contains a host of other interesting links and information from Kernighan.

http://cm.bell-labs.com/who/bwk/

The standard awk implementation, nawk (for "new awk", as opposed to the "old awk, sometimes found as 'oawk' for compatability), is based on the POSIX awk definitions, and contains a few functions that were introduced by two other awk implementations, gawk and mawk. I usually keep this one around as 'nawk' and use it to test the portability of my awk scripts. This one is usually found on my commercial UNIX machines, where I often don't have gawk installed.

Source for nawk: http://cm.bell-labs.com/who/bwk/awk.tar.gz

The GNU project's awk, gawk, is also based on the POSIX awk standard, but adds a significant number of useful features, as well. These include command line features like 'lint' checking and reversion to struct POSIX mode. My favorite feature in gawk is the line breaks, using '\', and the extended regular expressions. The gawk documentation has a complete discussion of GNU extensions to the awk language. This is also the standard awk on Linux and BSD systems.

Source for gawk: ftp://gnudist.gnu.org/gnu/gawk/gawk-3.0.6.tar.gz (the GNU Project's version of awk)

This is perhaps the most popular book on these two small programs, and is highly regarded. It contains, among other things, a discussion of popular awk implementations (ie gawk, nawk, mawk), a great selection of functions and the usual O'Reilly readability. The awk homepage lists several other books on the awk programming language, though this one remains my favorite.

The sed & awk book: http://www.oreilly.com/catalog/sed2

Jose Nazario

José is a Ph.D. student in the department of biochemistry at Case Western Reserve University in Cleveland, OH. He has been using UNIX for nearly ten years, and Linux since kernels 1.2.