"Linux Gazette...making Linux just a little more fun!"

Experiments with SMTP

(or: Mail for a Home Network, continued..)

By Jan Stumpel

1. Introduction

I got a lot of reactions to 'Setting Up Mail for a Home Network Using Exim' in LG 43. Most of them said two things:

your article was most welcome. I have been struggling with the same problem myself.
but I did what you said, and it does not work!

So I went 'back to the drawing board' and had a good look at what happens when e-mail is sent. As a result, this article explains what you must do to make the setup in the LG 43 article really work (I think) .. and also why.

2. SMTP

SMTP, as you probably know, stands for 'Simple Mail Transfer Protocol'. It is the method by means of which mail is exchanged between computers on the Internet. Basic communication between computers (exchange of single packets and of streams of information) is provided for by TCP/IP. SMTP is a protocol 'on top of' TCP/IP for exchanging messages between computers. Let's do a few experiments to see how SMTP works.

A basic tool for making TCP/IP connections is telnet. If you type

telnet host port

your computer makes a connection to a computer named host on port port. It is like making a telephone call to the office of a company called 'host' and asking to speak to Mr. Port. Only if Mr. Port is in and willing to talk to you, will the call succeed. Similarly, a program ('daemon') must be active on the other computer, 'listening' for connections on the specified port, otherwise you will get the message 'connection refused'.

port is a (16-bit) number. Certain port numbers have been pre-assigned to certain services. Electronic mail (SMTP) uses port 25, and the daemon listening to port 25 is the MTA (the Mail Transport Agent: sendmail, exim, qmail, etc.). If your Linux box is called heaven, you call its SMTP service by typing

telnet heaven 25

You can do this from another computer through a network (LAN or Internet), but you don't need a network: you can test it also by running telnet from the same computer that the MTA runs on. You can even type

telnet heaven smtp

because telnet finds out what the port number of SMTP is by looking it up in /etc/services. The result will be something like:

Trying 192.168.1.1...
Connected to heaven.home.
Escape character is '^]'.
220 heaven.home ESMTP Exim 3.03 #1 Sun, 8 Aug 1999 12:47:24 +0200

This shows that I am running exim 3.03 (I recently upgraded from 2.05 for a good reason, see section 5 below). If I telnet in the same way to the mail server of my ISP, I see that they run Sendmail 8.8.8/1.19.

After the line beginning with 220 you see no prompt or anything; the MTA awaits your instructions. What to do next? Try typing help. The reaction is:

help
214-Commands supported:
214- HELO EHLO MAIL RCPT DATA
214 NOOP QUIT RSET HELP

These are the commands of the SMTP command language, or 'protocol', that are supported by your site. Not a lot of commands! SMTP is really a 'simple' protocol. The commands are described in the Internet standard RFC821. Some 'extended' commands were added later, in other RFC's, for instance RFC1869. Systems which recognize the extended commands are said to support 'Extended SMTP', or ESMTP. Such systems announce this in their 'welcoming line', as Exim 3.03 did above. The differences between SMTP and ESMTP are not great.

To break the SMTP connection, send the QUIT command.

3. Exchanging greetings: the HELO/EHLO command

After the welcoming line (beginning with 220) from the remote system, you are supposed to send commands. The first command should be HELO, or, if you are dealing with an ESMTP system, EHLO, the more modern version. The command should have your domain name as argument:

EHLO yourdomainname

If you have a home system without an official domain name, what name do you use? In fact anything is OK, including your own, self-chosen domain name, such as heaven.home. Let's try it with our ISP's SMTP server by typing telnet smtp.isp.com 25 or whatever. After the welcome message type

EHLO heaven.home

We get a more or less elaborate 'greeting' message, like:

250-smtp.isp.com Hello customer123.dialin.isp.com [xxx.yyy.zzz.123], pleased to meet you
250-EXPN
250-VERB
250-8BITMIME
250-SIZE

The greeting begins with '250'; this is the SMTP 'OK' code. In this case we are also greeted with our temporary domain name (customer123.dialin.isp.com) and temporary IP address (xxx.yyy.zzz.123) that were dynamically assigned to us when we opened the ppp connection. This information is available to the other system from the underlying Internet transport layer (TCP/IP). In the case of an EHLO command, the other system also sends a few '250' lines announcing which extra SMTP or ESMTP commands it understands, apart from the minimum set required by RFC821.

Mail servers generally don't look at the argument of the EHLO or HELO command at all ('heaven.home'). That means that in practice the EHLO/HELO transaction always succeeds. If the other system doesn't want to do business with you, it has already refused the telnet host smtp connection.

4. Sending the mail

By just a telnet host smtp connection to a mail server you can send electronic mail 'by hand', without even using an MTA or a mail user program like pine. Let's try this (for safety's sake) within our own network at first; in this case of course, we must have a mail server (MTA) running. User joe sends a message to user emi. This involves three steps. First the MAIL FROM: command (SMTP commands are not case sensitive, so you could also type mail from:).

MAIL FROM: joe@home
250 <joe@home> is syntactically correct

We get a '250' line as answer, so this is OK. Now the second step: the RCPT TO: command, specifying who will get the message.

RCPT TO: emi@home
250 <emi@home> is syntactically correct

So this is also 250, OK. The third step: we enter the message itself, using the DATA command:

DATA
354 Enter message, ending with "." on a line by itself

The '354' reply invites us to type the message data. This is not only the text (or 'body') of the message! The 'message data' also include the message headers, such as Subject:, To:, Cc:, and From:. The structure of a message is specified in another Internet standard, RFC822. Strictly speaking that is no longer SMTP's business. SMTP is only concerned with the envelope of the message, that is, the information in the MAIL FROM: and RCPT TO: commands. So, the To: header inside the message and the RCPT TO: address on the envelope of the message are in principle two different things. You can actually make them different (experiment only with local messages please!). So, for instance, after the '354' reply we can type a message with 'fake headers':

To: My Daughter
From: Your Dad
<--(a blank line separates the headers from the body of the message)
Happy birthday!
. <--(a period at the beginning of a line ends the message)

This will be promptly delivered to user emi. If she opens the message using pine, she will see the To: My Daughter and From: Your Dad addresses.

Problems arise when she tries to reply to the message. Replies by users do not go to the 'envelope from' address (SMTP's MAIL FROM:), but to the From: address which is part of the message data. For starters, pine will think that 'Your' and 'Dad' are two different addresses, and complain that they should be separated by a comma, not a space! And of course there are no mail accounts 'Your' or 'Dad' registered on this machine, or anywhere else. Manipulating or omitting the various addresses is a fertile field for pranksters and spammers.

5. Mail setup for your home network

Now we get to what was wrong with my article in LG 43. Apart from the 'pine bug' that I wrote a note about in LG 44, the two major problems encountered by readers were:

The MTA may not be active at all
I said that the mail client (e.g. pine) on the Linux side can be used 'out of the box'. This is true, but the problem is that with many users the mail client does not come out of the box, but is already installed and being used. Often this means that in its configuration there will be a setting for 'SMTP server', set to the address of the ISP's mail server. In such a case the mail client itself will do the SMTP transaction (most of them, apart from mail, can do this), and your MTA will not be used at all. Therefore the transport filter will not be used, and the From: address inside the message will not be changed. Remedy: set 'SMTP server' in the mail client to the name of your Linux box, so the mail client will hand over messages to the MTA. If asked for 'your e-mail address', use your local address; exim's transport filter will change it for outgoing mail.

The 'envelope from' is not changed
This was the really big problem. As I said, the exchange of e-mail with an SMTP host involves three steps:

MAIL FROM:
RCPT TO:
DATA

If everything goes well, the remote system will answer 'OK' (i.e., '250') to the first two steps. But sometimes it won't! Many (probably most) mail servers, including the one at my own ISP, do not check the MAIL FROM: address. They always say 'OK' . But some verify that the domain part of the MAIL FROM: address really exists. If not, the mail is refused. If your own ISP checks the MAIL FROM:, the mail setup of my previous article will simply not work for outgoing mail. If your ISP doesn't check, but you send mail to a destination which does, your messages will not arrive and you will get no warning of this.

This means that we definitely should fix not only the From: address inside the message (using the program outfilt described in my previous article) but also the MAIL FROM: (or 'envelope from'). There are two ways in which you can do this.

If you have exim 2.05 or earlier the only way is to add a line in the last section, REWRITE CONFIGURATION, of exim.conf:

*@home joe.bloggs@isp.com F

This changes all local MAIL FROM: addresses to the address at the ISP. Unfortunately, with this method not only the outgoing mail is changed, but the local mail also. You can live with this, because users never see the 'envelope from' when they open a mail message. Replies will go to the correct (message From:) address. However, if someone makes a mistake typing a local address, so that mail is addressed to a non-existent local user, an error message will be sent to the address at the ISP instead of locally.

If you have exim 2.10 or later you can instead add an option to the remote_smtp subsection of the TRANSPORTS CONFIGURATION section of exim.conf. These newer exims allow the 'envelope from' to be changed for outgoing mail only. After the line driver = smtp you insert a line

return_path = "joe.bloggs@isp.com"

As this is a better method, I advise you to upgrade if you have exim 2.05. In the case of Debian, the latest version of exim (at the time of writing, version 3.03) can be found at www.debian.org among the 'unstable' packages (it is quite stable, don't worry). Upgrading is pretty painless.

Both methods described above produce a fixed 'envelope from' address, just as the program outfilt in my previous article produced a fixed 'message From:' address. I am describing a situation with only one e-mail account. If your home users are known to the outside world by different e-mail addresses, the setup becomes a little bit more complicated, but still possible. It would take a little bit too long to describe the various possibilities here; you might look at 'string expansion' and possibly 'file lookup' in the exim doc's.

-o-o-o-o-o-