[afnog] Collection Engine

[Brian Candler]

On Fri, Sep 30, 2005 at 11:17:32AM +0200, Alain Ilunga wrote:
> I have Mdaemon server using a private IP address,
> collecting client's mail through a single account on a
> main exim server (POP connection).

I've never heard of Mdaemon. Do you have a URL for it? It doesn't seem to be
in the FreeBSD ports collection. Is it commercial software? For Windows?

> I plain to replace Mdaemon server by Exim, I need to
> know if it is possible to configure a FreeBSD box, in
> order to connect to my main server, collect all my
> client's mail forwarded on a single account and then
> redistribute those mails in each client's box.

You can, but Exim by itself won't do that. Exim is a Mail Transfer Agent
(MTA), and its job is to accept mail received via SMTP or submitted by local
users, and to deliver it.

* To _collect_ mail from a remote POP3 server you will need a tool such as
'fetchmail'. This makes a POP3 connection, pulls down the mail, and then
injects it into your local MTA using SMTP.

WARNING: Using a 'multidrop' mailbox does not work well, and this is not a
problem with fetchmail: it's a fundamental problem with what you're trying
to do.

Once a message has been delivered into a POP3 mailbox, it loses the
"envelope": the list of people it was delivered to. You can attempt to
reconstruct it from the headers of the message, but this often doesn't work.

Problem 1: The 'To:' header is often not related to the envelope.

e.g. incoming mail for the afnog mailing list may arrive via SMTP as

   MAIL FROM:<afnog-owner at afnog.org>
   RCPT TO:<alilunga at yahoo.fr>
   DATA
   ...

but when you look at the headers of the delivered message, you will see

   From: b.candler at pobox.com
   To: afnog at afnog.org

>From that it's impossible to work out which local recipient should receive
this message.

Sometimes, the MTA puts the envelope information in another header (e.g.
Envelope-To, X-Envelope-To, Delivered-To). This is non-standard and varies
from server to server. Without this, the best you can hope to do is to parse
all the Received: headers looking for a reference to the recipient. This is
very unreliable.

Problem 2: A message may arrive with multiple recipients

Suppose two people at yahoo.fr are subscribers to the afnog mailing list.
A single copy may be delivered via SMTP to yahoo.fr's mail server:

    MAIL FROM:<afnog-owner at afnog.org>
    RPCT TO:<alilunga at yahoo.fr>
    RCPT TO:<someone-else at yahoo.fr>
    DATA
    ...

If these two people have separate mailboxes it's not a problem, because the
MTA will create a separate copy of the message in each mailbox. But if
you've routed all the mail for a domain into a single mailbox, then only one
copy will be stored.

In this case the problem is the same as (1) but is worse, because even if
you are able to extract the envelope for the single-recipient case (e.g. the
MTA adds X-Envelope-To or you can parse the Received: headers) often this
doesn't work with multiple recipients. For example, MTAs often don't put any
recipient in the Received: headers if there are more than one.

So: I strongly recommend you don't do what you are proposing.

Better options are:

a. Have a separate mailbox on the main server for each user. fetchmail can
then open a separate POP3 connection for each user, collect their mail, and
dump it into the correct mailbox on the local mailserver.

This is easy to set up - fetchmail's config file can list many connections -
but it won't scale beyond a few tens or hundreds of accounts.

b. Use a different way of collecting mail which preserves the envelope.
Options include:

- UUCP over TCP/IP (software exists and works well, but the expertise for
setting it up is often not available)

- SMTP with ATRN (RFC 2645), also called On-Demand Mail Relay. A nice idea
(an SMTP connection which 'turns around') but not widely implemented either
on the client or server side

c. You say the client mail server has a private IP address. If it's
permanently connected and its private IP doesn't change, and the public
mailserver also has a private IP, you can still deliver mail directly using
SMTP.

                           |
                           | 1.2.3.4 (public IP)
                      central mailserver
                           | 192.168.1.1
                           |
       ---+----------------+-----
          |
          | 192.168.1.200
     client mailserver
    
You can point MX records for the client's domain at the central mailserver,
and then configure that mailserver to relay mail using SMTP to the client's
mailserver, over the private network. This is in many ways the best option,
because there's no polling and mail is delivered almost instantaneously, but
it does depend on having the right connectivity (central mailserver has a
private IP which can reach the client mailserver) and the client mailserver
being permanently connected.

If the central mailserver has only one network interface, you can always
move it behind a NAT firewall, so it has only a private IP. You would then
configure the firewall to forward inbound connections on port 25 (and 110,
143...) to the mail server, so that incoming mail and POP3 connections from
the outside world would still work.


* To allow users to read mail from your newly-built client mail server,
you'll need to install some interface to allow this: e.g. a POP3 server, an
IMAP server, a webmail server, a local Mail User Agent which they can use at
a Unix shell prompt, or some combination of these. Again, Exim doesn't
include these. Once it has delivered the message into a local mailbox file
or directory, its job is done.

Options I've used include courier-imap and sqwebmail. These work well but
(a) they require you to deliver in Maildir format (a good idea but it means
some MTA tweaking), and (b) the documentation isn't brilliant.

There are notes on all this in the materials from previous Afnog workshops.
http://www.ws.afnog.org/

If you want a free software mailserver which works "out of the box", and are
prepared to accept the limitations of a preconfigured bundle, you could have
a look at a project I've been working on sporadically:
http://www.linnet.org/

Linnet provides you with exim + pop3/imap/webmail software ready to run.
However it doesn't have any features for 'pulling' mail from a remote mail
server. You'd either have to install a utility like fetchmail yourself, or
set up the MX routing described above.

HTH,

Brian.