[afnog] Routing asymmetry problems

Fri Oct 15 17:51:58 UTC 2010

Hi list,

I have a problem with multi-homing that has just appeared with the addition
of a new upstream. I currently have 3 different upstreams - one terrestrial,
and two satellite. In this setup, I can send all my outbound traffic via the
terrestrial link and get the inbound between the 3 without a problem. This
is accomplished through playing around with advertising different prefix
lengths and AS-PATH prepends.

Now I am trying to add another terrestrial link. However, when I do this, if
I maintain sending all traffic out through the first terrestrial link and
receive through the second, it is not working. The round trip is not
complete.

An example below shows a traceroute to Cisco's website, first with using
terrestrial 1 as outbound and terrestrial 2 as inbound, and the other using
terrestrial 1 as outbound and satellite as inbound.

Tracing route to origin-www.cisco.com [72.163.4.161]
over a maximum of 30 hops:

  1     1 ms     1 ms    <1 ms  192.168.1.1
  2     9 ms     9 ms    10 ms  1.108.adsl.mtlonline.mw [41.221.108.1]
  3     8 ms     6 ms     6 ms  g1_0_0.ig01.std.bt.mtlonline.mw[41.221.96.34]
  4    32 ms    28 ms    26 ms  gi-5-1.cta-isp-bdr-2.tdm.mz [41.220.160.165]
  5   213 ms   213 ms   213 ms  pos0-2-1.djibouti1.dji.seabone.net[213.144.175.
5]
  6   297 ms   297 ms   297 ms  ge0-0.newark3.new.seabone.net[195.22.216.199]
  7   283 ms   283 ms   283 ms  GigabitEthernet0-0-0.GW4.EWR6.ALTER.NET[157.130
.67.177]
  8   283 ms   282 ms   283 ms  0.so-3-0-0.XL4.EWR6.ALTER.NET [152.63.2.210]
  9   330 ms   330 ms   330 ms  0.ge-4-3-0.XT4.DFW9.ALTER.NET [152.63.101.2]
 10   327 ms   330 ms   328 ms  GigabitEthernet7-0-0.GW14.DFW9.ALTER.NET[152.63
.96.69]
 11   329 ms   328 ms   329 ms  cisco-gw.customer.alter.net[157.130.134.190]
 12   356 ms   340 ms   375 ms
rcdn9-cd2-dmzbb-gw2-ten2-1.cisco.com[72.163.0.25]
 13   394 ms   409 ms   409 ms
rcdn9-cd2-dmzdcc-gw2-por-2.cisco.com[72.163.0.190]
 14   350 ms   331 ms   330 ms
rcdn9-16a-dcz05n-gw2-ten5-5.cisco.com[72.163.0.242]
 15     *        *        *     Request timed out.
 16     *        *        *     Request timed out.
 17     *        *        *     Request timed out.
 18     *     ^C

And:

Tracing route to origin-www.cisco.com [72.163.4.161]
over a maximum of 30 hops:

  1     1 ms     1 ms    <1 ms  192.168.1.1
  2    16 ms    19 ms    19 ms  1.108.adsl.mtlonline.mw [41.221.108.1]
  3     7 ms     7 ms     6 ms  g1_0_0.ig01.std.bt.mtlonline.mw[41.221.96.34]
  4    26 ms    27 ms    29 ms  gi-5-1.cta-isp-bdr-2.tdm.mz [41.220.160.165]
  5   386 ms   498 ms   385 ms  pos0-2-1.djibouti1.dji.seabone.net[213.144.175.
5]
  6   492 ms   494 ms   499 ms  ge0-0.newark3.new.seabone.net[195.22.216.199]
  7   485 ms   481 ms   496 ms  GigabitEthernet0-0-0.GW4.EWR6.ALTER.NET[157.130
.67.177]
  8   473 ms   479 ms   473 ms  0.so-3-0-0.XL4.EWR6.ALTER.NET [152.63.2.210]
  9   536 ms   542 ms   535 ms  0.ge-4-3-0.XT4.DFW9.ALTER.NET [152.63.101.2]
 10   532 ms   529 ms   519 ms  GigabitEthernet7-0-0.GW14.DFW9.ALTER.NET[152.63
.96.69]
 11   518 ms   520 ms   516 ms  cisco-gw.customer.alter.net[157.130.134.190]
 12   530 ms   532 ms   532 ms  rcdn9-cd2-dmzbb-gw2-ten2-1.cisco.com[72.163.0.2
5]
 13   524 ms   528 ms   528 ms  rcdn9-cd2-dmzdcc-gw2-por-2.cisco.com[72.163.0.1
90]
 14   523 ms   516 ms   521 ms  rcdn9-16a-dcz05n-gw2-ten5-5.cisco.com[72.163.0.
242]
 15   537 ms   529 ms   552 ms  www1.cisco.com [72.163.4.161]

Trace complete.

You will see that having the satellite as inbound works. I have a suspicion
that it could be something to do with uRPF, but I am not sure where it would
have been applied in this case to cause this disparity. I asked my
terrestrial 2 provider if they or their upstream uses uRPF, but they asked
me what that is, so I take it they do not use it.

By the way, if I set both outbound and inbound to terrestrial 2, it works.

I need some pointers on how I can sort out this problem.

Thanks in advance to your usual assistance.

Regards,

Kondie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://afnog.org/pipermail/afnog/attachments/20101015/d58881a6/attachment.html>