[afnog] design redundant mail servers

[Brian Candler]

On Sun, Aug 29, 2010 at 11:44:24AM +0200, Issa NKUSI wrote:
>    My existing platform is below, however, i'm open for better
>    suggestions.
> 
>    -          CentOS 5
> 
>    -          MTA: Postfix + Mail Scanner (ClamAV, spam assassin) +
>    Squiremail + Virtualmin + webmin + Dovecot
> 
>    -          SASL Authd

That's a good start. Presumably by "redundant" you mean "able to continue to
work in the event of a loss of a major component".

Here are some options to consider.

(1) You can buy an NFS server with multiple head-ends for redundancy, and
store your mail on that in Maildir format.  Then you can have multiple
front-end boxes all talking to the same mail directories, and a
load-balancer in front of them.  This scales well horizontonally, as you can
easily add more frontend boxes for load handling, and more NFS servers by
putting different customers' mail on different mountpoints (e.g.  /mail/0,
/mail/1 etc)

I have good experience with Network Applicance NFS servers in this kind of
role (www.netapp.com) - but they are definitely not cheap.

(2) You could consider building your own redundant NFS server. This might
consist of a shelf of SCSI disks, connected to two PCs each with their own
SCSI cards. Then you'd use some clustering software to decide which of the
two front-ends mounts the disks and shares them.

This sort of cluster is hard to get right. You need to arrange that if one
machine decides to take over the array, it forcibly disconnects or powers
down the other one ("STONITH").  If you have two machines mounting the same
volume, major filesystem corruption is guaranteed (unless you use a
distributed filesystem like GFS or OCFS2, of which I have no experience). 
Or you could elect to do the switchover manually in the event of a problem.

Also, your SCSI shelf is still a single point of failure.

(3) Another way to build redundant storage is with drbd (www.drbd.org), the
kernel modules for which are included in CentOS 5 I believe.  This is
basically RAID1 disk mirroring between two PCs, across a LAN.  One side is
the master (read-write), and the other slave (read only).  To get live-live
operation you could set up two drbd volumes, so that PC 1 is master for vol1
and slave for vol2, and vice versa.

Again, some sort of clustering or heartbeat solution is needed to switch the
master across to the second PC if the first one fails, with the same
difficulty in getting it right.

(4) A very interesting option, which I know is in use at least one large UK
ISP, is "Gluster" (www.gluster.org).  This lets you build a huge filesystem
spread across a pool of PCs.  As long as you have two or more PCs, your
files are replicated across different machines.  It uses elastic hashing to
distribute your files transparently between the servers, so it's essentially
self-managing, and can serve files using its own protocol or NFS or CIFS.

http://www.gluster.com/community/documentation/index.php/Gluster_Storage_Platform#Volume_Manager

So if you want a fully open-source solution using only commodity hardware,
this is definitely worth checking out.  Using a two-node Gluster fileserver
with two mail server nodes and a pair of load-balancers should give you a
very good level of redundancy and a good expansion route for growth.

Regards,

Brian.