...making Linux just a little more fun!

Setting up the mail subsystem in Linux By Ben Okopnik

The mail system is - or can be - one of the more complex parts of the Linux jigsaw puzzle. True, for a lot of folks, it's not complex at all: they install Netscape, enter their POP/SMTP server names, username, and password, and off they go... unless, of course, they want to use anything else that utilizes the mail system - such as writing a script that will mail them a report when the file system is almost full, or deciding that they'd like a different Usenet news reader, or even try to mail in a bug report using the "bug" or "bashbug" utilities. Ooops...

In Unix, mail is closely integrated with the OS itself, and not having it working properly is like driving a car with a deflated tire. Things work kinda OK, as long as you don't get up above 5mph, or shift your weight to the wrong side - or even let your girlfriend get in for a ride. As soon as you do, problems crop up by the dozen. A working mail system - like a net connection - is one of the basic assumptions in any Unix-flavored OS. What I'd like to do here is show you at least one example of a working mail system, which you can then adjust or interpolate to your own setup; the important part is being aware of the pieces that need to be operational in order for this to happen.
 

THE PIECES THAT MAKE UP THE WHOLE

The mail system consists of three somewhat loosely defined pieces: the MUA (Mail User Agent), which is the software you use to read and compose your mail, the MTA (Mail Transfer Agent), usually an SMTP server, although some directly-invoked programs are also in use, and a retrieval program (Some SMTP servers also contain POP functionality, but a stand-alone program. is more common.) The MUA can be pretty much anything you want: it's only a front end, meaning that you can use whatever you prefer once the other two pieces are working. You can even stick with Netscape if you like! For the other two in this example, I'll use Exim - a well-known MTA, and Eric S. Raymond's "fetchmail", probably the most-commonly used retrieval utility in the world.
 

GETTING YOUR STUFF

There's not much complexity in setting up "fetchmail". Pretty much all that's required is creating a file called ".fetchmailrc" in your home directory and specifying your POP-related information. As an example, here's what mine looks like:

# I want to log all retrievals to "/var/log/mail.*"
set syslog

# Set stuff that's the same for everybody
defaults       protocol pop3,
               timeout 300,
               nokeep,
               fetchall,
               mda "procmail -f-"

# Get mail from my ISP
poll "pop.happybruin.com",
    user "fuzzybear"
    password "wouldnt_you_like_to_know";

# Grab it from my other account
poll "pop3.bearsden.com", 
    user "ben-fuzzybear",
    password "shhh_its_a_secret";

Just a quick overview of the above - it's very well covered in the "fetchmail" man page: I'm retrieving mail from two different accounts. Since I have a somewhat flaky Net connection (a wireless modem), I've set "fetchmail" to time out any given connection after 5 minutes (300 seconds). I've also told it to delete all the mail on the server once it is retrieved ("nokeep"), to ignore the "already read" flag and get all the mail that's waiting ("fetchall"), and to use "procmail" to do some header processing for me ("mda ..."). The last is not needed for everyone, but some broken SMTP servers "forget" to include a so-called "Envelope-from" header, and this fixes it. Other than that, I think everything is pretty self-explanatory.

There are generally two ways in which fetchmail is launched. It can be started as one of the "init" scripts (this is useful if you have an always-on connection), or from your "/etc/ppp/ip-up.d" script (more common for dial-up connections.) Usually, you get to choose this during "fetchmail" setup. Each user can also start it manually, as a one-time run (simply by typing "fetchmail" at the command line) or as a daemon that will poll the mailboxes at a set interval (I like to do it this way, with a "fetchmail -d 600" which polls at 10 minute intervals. This can also be defined in ".fetchmailrc".)

"fetchmail" is far more flexible and powerful than this simple situation shows. Suffice it to say that it can do almost any kind of mail retrieval, with any valid mail protocol; unless you have some truly complicated lash-up - and if you did, you'd know about it - it will work for you. Of course, if you have your own preferred retrieval agent, that's fine too.
 

LOOKING AT THE BIG PICTURE

Setting up your SMTP server doesn't necessarily have to be much more complex than the above - but it definitely should take a lot more thought. The main thing to consider is, where do you fit into the Net? For those of you who have never had to think of yourself on that large of a scale, that's yet another piece of the puzzle: the reality is that most of the Net is built up of little pieces - such as the computer that you're sitting in front of right now. Your ISP is just another node of the Net; true, you're connecting through their routers, but once you have connected, you're just as much a part of the Net as they are - and consequently, responsible for making sure that your little piece works in harmony with the rest.

(One of the security-related RFCs I read recently - I don't recall exactly which one - mentions that possibly 50%+ of the mail servers connected to the Net are misconfigured to some degree. Pretty scary statistic... but also quite a testament to the reliability and flexibility of the Net mail system. All of this points up the need for all of us to contribute to the Good Side of the Force - by doing our part.)

For a lot of us, the situation is very simple: a desktop machine, a single ISP, and no need to do our own SMTP - at least any more than is necessary to forward all our mail to the ISP's SMTP server. In this situation, pretty much any MTA will do - and there's next to no tweaking necessary, except for address rewriting. Just answer the questions that you're asked at setup time, and - bingo, you're off and running. However, this part of the system is a little more "touchy" when it comes to changes: if you use more than one ISP, or want to do anything else even slightly different from the basics, it's going to take a little configuration... and this is where most folks run afoul of the mail beast.

  "sendmail"'s configuration file looks like someone's been banging their
  head on the keyboard. And after looking at it... I can see why!
   -- Anonymous

"sendmail.cf" has been responsible for more than one sysadmin being dragged away while tied down to a stretcher and foaming at the mouth. It's an ugly creature... and the configuration file that it's created from isn't any prettier. I've detailed a bit of its workings back in LG#58 (Configuring Sendmail in RedHat 6.2, or My Adventure in the Heart of the Jungle); at this point, I have the twitching mostly under control, and the doctors tell me that I can stop taking these little pills in another year or so...

Seriously, this is a decision point. If your system's network connection is going to change in major ways (ISP, host name, from a dial-up to a full-on Internet host) more than once or twice, you should consider doing your own SMTP. As an example, I do my own because I travel for a living, and use lots of different ISPs (dial-up, wireless, cable modems in hotel rooms, etc.) in many different system configurations. Doing it this way means never having to worry about what anyone else's mail setup is like, or having to configure anything when I move from one system to another - a great convenience. In other words, doing your own is not a big deal to implement, but it is a critical decision that should be made based on your own needs. I find the "do-it-yourself" approach to be far more flexible, powerful, and hassle-free in all cases where the environment is anything other than static.
 

SMTP SETUP OPTIONS

So, at this point, we've defined two typical SMTP setups:

1) Delegate everything except address rewriting (that has to be done locally.) The ISPs SMTP server (the "smarthost", from our perspective) takes care of all the routing. This is a good way to go when you have a static setup that's not likely to change, especially through a major ISP with a good reliability record (well, we can dream, can't we?)

2) Do everything ourselves. This has a number of benefits, including bypassing unreliable ISP mail services and the ability to instantly see if your mail has actually been delivered to the host on the other end (a few years ago, my ISP held some of my emails for over a week, and discarded a batch of them without notifying me. That was what initially started me doing this...)

Generally, this is a decision that's made during the installation of the MTA (Mail Transfer Agent). There's not much to it; in the case of Exim, you're given five choices, of which only the first two really apply here (the "eximconfig" program runs during the installation, or may be re-run manually at any time):

You must choose one of the options below:

 (1) Internet site; mail is sent and received directly using SMTP. If your
     needs don't fit neatly into any category, you probably want to start
     with this one and then edit the config file by hand.

 (2) Internet site using smarthost: You receive Internet mail on this 
     machine, either directly by SMTP or by running a utility such as 
     fetchmail. Outgoing mail is sent using a smarthost. optionally with
     addresses rewritten. This is probably what you want for a dialup
     system.
  
         ...

Note that these two choices fit the above two options: the "do everything ourselves" approach dovetails into #1, and the "smarthost" version is #2. "eximconfig" then walks you through a few more questions, one of which is

...

Which user account(s) should system administrator mail go to?
Enter one or more usernames separated by spaces or commas.  Enter
`none' if you want to leave this mail in `root's mailbox - NB this
is strongly discouraged.  Also, note that usernames should be lowercase!

THE BELLY OF THE BEAST

Once that's done - and we'll get to what else we need to do in the two different cases - we need to set up address rewriting. After all, your email address as seen by the system is "username@host", and unless you have your own domain, that isn't going to be an Internet-valid address. Fortunately, with Exim, it's not difficult.

First, we'll edit "/etc/exim/exim.conf", and add the following to the 6th section ("REWRITE CONFIGURATION"):

*@localhost       ${lookup{$1}lsearch{/etc/email-addresses}\
                                                {$value}fail} Ffsr

# Root shouldn't be emailing anyone outside, but just in case...
root: happybear@bruins.com
ben: happybear@bruins.com
rivka: sweetie@here.com
linda: babe@westcoast.org
jen: saucy@wench.net

THE DIFFERENT APPROACHES

If you're going with the "smarthost" option, at this point you're done. Skip ahead to the "TESTING" section. If you're a do-it-yourselfer like me, though, there's just a tiny bit more stuff to write: since we're now responsible for getting the mail to where it's going, we also have to deal with the situation when the delivery fails (i.e., the receiving host or an intermediate router is down, we lose the network connection for a moment, etc.) Most of that behavior is well-defined already, as it is in any decent MTA, but I've found one thing that reduces "trouble emails" from Exim (which it will send to you as the administrator) to nearly zero: in the first section of "/etc/exim/exim.conf", you should add the following:

auto_thaw = 5m

Oh, by the way. Now that you're a Big-Time Mail Administrator... :) what is it, exactly, that you're supposed to do? Not that much, actually. Decide what to do with problem messages (if Exim notifies you that something is stuck in the queue, run "mailq" to see what it is and look at its log file with "exim -Mvl <message_id>"), add new users to "/etc/email-addresses", and respond to any problem or spam notifications by other folks. Read the "exim" man page, just to get familiar with this beast. That's pretty much it. Experienced large-system mail administrators may shrink in horror and make warding signs in my direction, but for a single-machine or a small LAN, the above is pretty much all that's required. Once properly set up, a mail system is a remarkably trouble-free and mostly self-correcting sort of creature.
 

TESTING

Exim has a series of built-in testing modes, one of which is about to come in very handy. The main thing that we need to test is whether our rewriting rules work - and that's simple:

Baldur:~$ exim -brw ben
  sender: happybear@bruins.com
    from: happybear@bruins.com
      to: ben@localhost
      cc: ben@localhost
     bcc: ben@localhost
reply-to: happybear@bruins.com
env-from: happybear@bruins.com
  env-to: ben@localhost

Once all of the above works right, your mail system should be at least reasonably configured (the folks who set up the various distros do a pretty good job of the basics, in every case I've seen so far.) Test it out by sending yourself some mail, and look at the headers; the "From:" and the "Reply-to:" (if one is defined) should match your Net-valid address, not just your plain user name. Here's an example (the actual addresses/IPs have been changed, as in the rest of this article, to foil spambots. Eat fake address, spammer-slime!):
In the Mutt composition menu:

    From: "Benjamin A. Okopnik" <ben@localhost>
      To: Benjamin Okopnik <happybear@bruins.com>
      Cc:                                            
     Bcc:
 Subject: Rewrite test
Reply-To:
     Fcc: =Sentmail
     Mix: <no chain defined>
     PGP: Clear

mail -s "Rewrite test" happybear@bruins.com

Either way - now, we send it off, and when we get it back - presto!

Date: Tue, 30 Apr 2002 03:47:19 -0400
From: "Benjamin A. Okopnik" <happybear@bruins.com>
To: Benjamin Okopnik <happybear@bruins.com>
Subject: Rewrite test

WARNING: Deep Magic in progress.

Ben Okopnik
-=-=-=-=-=-

If we look at the actual headers (in Mutt, press the "h" key), we'll see the following:

From ben Tue Apr 30 03:48:15 2002
Return-Path: <happybear@bruins.com>
Received: from Baldur (pzw-199-999-99-999.sunbridge.com [199.999.99.999]))
        by bruins.com (9.10.3/9.10.3) with ESMTP id g3U7lR45008674
        for <happybear@bruins.com> Tue, 30 Apr 2002 00:47:32 -0700 (PDT)
Received: from ben by Baldur with local (Exim 3.35 #1 (Debian))
        id 172SM7-0004nd-00                                    
        for <happybear@bruins.com> Tue, 30 Apr 2002 03:47:23 -0400
Date: Tue, 30 Apr 2002 03:47:19 -0400
From: "Benjamin A. Okopnik" <happybear@bruins.com>
To: Benjamin Okopnik <happybear@bruins.com>
Subject: Rewrite test
Message-ID: <20020430074718.GA18398@Baldur>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.3.28i
Status: U  
X-UIDL: 27862

WARNING: Deep Magic in progress.

Ben Okopnik
-=-=-=-=-=-

WRAP-UP

If you've followed along and made it this far... congratulations. You're now that much more of a participating Netizen, one of the folks who's contributed a bit of time and effort to make the Net run a little more smoothly - and I'm glad to share the IP-space with the likes of you.

Be well, and happy Linuxing!

Ben Okopnik
-=-=-=-=-=-

Ben is a Contributing Editor for Linux Gazette and a member of The Answer Gang.

Ben was born in Moscow, Russia in 1962. He became interested in electricity at age six--promptly demonstrating it by sticking a fork into a socket and starting a fire--and has been falling down technological mineshafts ever since. He has been working with computers since the Elder Days, when they had to be built by soldering parts onto printed circuit boards and programs had to fit into 4k of memory. He would gladly pay good money to any psychologist who can cure him of the resulting nightmares.

Ben's subsequent experiences include creating software in nearly a dozen languages, network and database maintenance during the approach of a hurricane, and writing articles for publications ranging from sailing magazines to technological journals. Having recently completed a seven-year Atlantic/Caribbean cruise under sail, he is currently docked in Baltimore, MD, where he works as a technical instructor for Sun Microsystems.

Ben has been working with Linux since 1997, and credits it with his complete loss of interest in waging nuclear warfare on parts of the Pacific Northwest.