Re: BGP over satellite link - update!!

[Joe Abley <jabley at isc.org>]

On Wednesday, Apr 30, 2003, at 07:59 Canada/Eastern, Randy Bush wrote:

>> Well, this could work, if be the case, however, the box that would 
>> sit on
>> the client side of the network is only providing a one-way SCPC 
>> satellite
>> connection to the remote provider. All return traffic coming from the
>> remote provider to the client would be over a DVB circuit [another
>> interesting technology yet experienced in Africa].
>
> geoff huston did a lot of measurement and analyses in this kind of
> environment some years back when australia was using similar non-
> symmetric links.  google may be able to dig up some of his stuff.

I connected a large ISP in New Zealand to the US using a combination of 
simplex satellite and leased under-sea cable in the mid-to-late 1990s. 
I still have occasional nightmares about it :-)

Some of the things that I decided after that exercise were:

1. It can help a lot to have routers that you administer on both sides 
of the satellite circuit (i.e. don't just buy a satellite internet 
service; buy a small amount of colo at the earthstation on the sending 
side and uplink directly from there).

2. If possible, try and land your back-channel (whether it's narrowband 
terrestrial capacity, or a carrier on a different satellite) in the 
same place in the remote location. If you can do that, you can build a 
tunnel which presents both send and receive channels as a single 
interface, which can be a good thing.

3. If your solution involves tunnels, think long and hard before you 
run production internet traffic over a link that has an MTU less than 
1500 bytes. There are a lot of broken firewalls in the world, and 
you'll find lots of web and mail breakage due to stalled TCP sessions 
and broken path MTU discovery. [I tried this; it was not fun].

Our satellite circuit was presented through SDM2020 [de]modulators as 
HSSI to our routers. We couldn't find HSSI cables that were suitable to 
connect to unidirectional circuits, so we wound up buying a handful of 
those horribly expensive cisco HSSI cables and breaking them open, 
looping various tx/rx/whatever lines back to other connectors to make 
the interface appear up to the router. We had to try that a couple of 
times since our HSSI interfaces were kind of sensitive to badly made 
cables (e.g. cables made by me :-)

I spent quite a bit of time worrying about how I could get traffic to 
re-route in a sane way in the event of a satellite channel failure, or 
at least get dropped in a way that could cause alerts. There were 
definitely occasional failures where the sending side would merrily 
send frames towards the modulator and the uplink, and nothing would be 
received at the downlink; apart from the absense of traffic on the 
receiving side, there was no "interface down" type event to tell us 
that anything was wrong.

Constructing a tunnel and running OSPF over it worked, since a dead 
satellite would lead to dropped OSPF hellos. This also had the side 
effect of lowering the MTU on the NZ->US path, though, since there was 
ethernet involved in it. Eventually we decided that the lower MTU was 
causing more problems than the occasional satellite outage, and we 
ripped all the tunnel stuff out.


Joe


-----
This is the afnog mailing list, managed by Majordomo 1.94.5

To send a message to this list, e-mail afnog at afnog.org
To send a request to majordomo, e-mail majordomo at afnog.org and put
your request in the body of the message (i.e use "help" for help)

This list is maintained by owner-afnog at afnog.org