RE: RE : Site Mirroir

["William Stucke" <William at zanet.co.za>]

Hi Paul,

You are actually describing two SEPARATE issues: -

1	Maintaining two identical copies of a site - called "mirroring".
2	Directing visitors to the "best" server for the situation - called
"dynamic DNS", amongst other things.

In order to make (1) - Mirroring - work, you need the following: -

a	SERV1 must operate an web site (don't forget the obvious bits!)
b	SERV1 must operate an FTP site, with the content of the web site
accessible via FTP
c	SERV2 must have the appropriate rights to allow it access to this FTP site
on SERV1
d	SERV2 must run suitable mirroring software (more on that later)
e	Optionally, if certain software is chosen, such a rsync, SERV1 must ALSO
run appropriate software
f	SERV2 must operate a web site - the content is identical to that on SERV1,
because it's copied from there
g	SERV2 must have a scheduler (cron works fine, if you are in the UNIX
world) to cause the mirror software to run at the desired intervals

[You mention hourly updates. Good grief! What are you putting on the site
that requires such frequent updates? If you want "up to the minute" changes
to be reflected on both sites, then there are alternative methods, such as
synchronising a SQL DB]

What mirror software to use?

For UNIX: -

mirror is most common. It's a Perl script, so you must have Perl installed
on your system to be able to use it. It's a relatively simple FTP folder
directory comparison tool, that will synchronise the local site to the
remote site every time it is run. It does this by: -

*	Comparing directories on both sites (various packages do either a depth
first or a breadth first comparison. A few allow you to choose which you
want to use)
*	Copying all new files from the remote site to the local site
*	Copying all changed files (i.e. ones that have a more recent modified
date) from the remote site to the local site
*	Deleting all files on the local site that no longer exist on the remote
site. CAREFUL! A "broken" run on almost any of these packages can result in
your entire local site being deleted, if you aren't careful how you set it
up!

There are several other packages that operate in a similar manner, some of
which were mentioned on this list.

rsync is different. It works on a "push" basis, rather than the "pull" basis
of the compare-directories-and-FTP-the-differences packages, like mirror.
Essentially, the "Server" (SERV1 in your case), sends the "Client" (SERV2) a
"digest" of the directory contents that have changed since the last update.
The "Server" therefore needs to: -

*	Keep track of when the Client lost connected, and what files it downloaded
*	Scan its own log files in order to generate the digest for the Client

This means that rsync requires MUCH more processing on the part of the
Server, but the Client needs to transfer less data and do less work, since
it doesn't have to read and compare every directory.

By way of comparison, if you are mirroring something like TUCOWS, the
directories alone represent several tens of MB of data, and up to 30 minutes
of downloading and comparing, before actual file transfer takes place,
depending on CPU & WAN speed.

For Windows: -

NetLoad is one of the best, but many packages will do it, such as Cute FTP
and even GetRight. They work in the same way as that described for mirror
above.


Now to the second issue - redirection.

The way this is USUALLY done is with a "Server Farm", i.e. several servers
behind a single "switch". The switch directs requests to whichever server is
(a) least heavily loaded and (b) is currently online. This is often
implemented for either load-sharing or redundancy purposes.

You can't do that, if your two servers are on different networks. In that
case, your best bet is probably "dynamic DNS". Essentially, this means that
a name server is responsible for directing requests to the "best" server,
depending on load, speed, and availability.

The problem with this scenario is DNS caching. Most name servers will cache
(remember) a recent name lookup, so if your dynamic DNS changes which server
it wants to send requests to, it will be ignored by anyone who has been
there recently. Of course, NEW visitors will be sent to the right server.

I'm by no means an expert on the subject. Best to ask someone who knows much
more than I do, like Alan Barrett.

BTW, Windows 2000 implements dynamic DNS natively, and relatively
painlessly.

Hope that helps. Bon chance!

Regards,

William Stucke
ZAnet Internet Services (Pty) Ltd
mailto:William at zanet.co.za
+27 11 465-0700


-----Original Message-----
From: owner-afnog at afnog.org [mailto:owner-afnog at afnog.org]On Behalf Of
Paul Dégbé KPOGNON
Sent: 2002/05/25 16:28
To: Noah K Sematimba
Cc: afnog at afnog.org
Subject: Re: RE : Site Mirroir


No, my problem is this.
Let me give clearly an example.

I have a web site on my webserver called SERV1. The URL to reach the site
is www.web1.com.
Now i want to have a mirror of that web site on another server named SERV2
on a remote ISP network. The transfert will be done automatically each hour.
If somebody want to visite my web site, he only know one address :
www.web1.com.
And that address can call SERV1 or SERV2 depending of the following
situations :  SERV1 is down and SERV2 is up (so SERV2 will respond), SERV2
is more quick than SERV1 (so SERV2 will respond), unless SERV1 will respond.

Now my problem is :

1 - What do I need to have like packages for that?
2 - Which mirror configuration each of the both servers SERV1 and SERV2
will have?

I think I more and more clear.

Thanks you for your help

Paul

At 05:12 PM 5/24/2002 +0300, Noah K Sematimba wrote:

> > Now like I have said, I would like to get experience from those who have
> > installed yet MIRROR.
> > What is the procedure and witch packages is needed and where to find
those
> > packages?
>
>If you're talking about the package called mirror. You simply need the
>package mirror itself and also to run an ftp server. Problem with this is
>the usual fact that ftp is unencrypted thus all ytour stuff is cleartext.
>
>You could use sitecopy and if the web server has DAV support that is an
>option. Since in that case you could use ssl.
>
>Noah.
>
>
>-----
>This is the afnog mailing list, managed by Majordomo 1.94.5
>
>To send a message to this list, e-mail afnog at afnog.org
>To send a request to majordomo, e-mail majordomo at afnog.org and put
>your request in the body of the message (i.e use "help" for help)
>
>This list is maintained by owner-afnog at afnog.org


-----
This is the afnog mailing list, managed by Majordomo 1.94.5

To send a message to this list, e-mail afnog at afnog.org
To send a request to majordomo, e-mail majordomo at afnog.org and put
your request in the body of the message (i.e use "help" for help)

This list is maintained by owner-afnog at afnog.org


---

Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.363 / Virus Database: 201 - Release Date: 21/05/2002


-----
This is the afnog mailing list, managed by Majordomo 1.94.5

To send a message to this list, e-mail afnog at afnog.org
To send a request to majordomo, e-mail majordomo at afnog.org and put
your request in the body of the message (i.e use "help" for help)

This list is maintained by owner-afnog at afnog.org