[afnog] High CPU Utilization Problem on 2-Stacked Cisco Switches.

[Mark Tinka]

On Tuesday, October 26, 2010 07:11:25 pm Michael Komla Nfodzo wrote:

> Hi All,

Hello Michael.

> Can someone, please, look into this problem for me and provide me with
> the assistance to resolve it.

It's funny, I didn't receive a copy of your response 
as you indicate below (although I see it in the mailing 
list archives - odd).

Anyway...

> -----Original Message-----
> From: Michael Komla Nfodzo
> Sent: Tuesday, October 19, 2010 11:12 AM

> Hi Mark,
> 
> Thanks so much for your prompt response. Please find below the output of
> the commands as requested.

> What SDM template are you running?------------------------Not running
> SDM

The 3750 and that breed of desktop switches all support 
SDM templates. If you haven't configured any, you're 
likely running the 'default-desktop' template.

You can tell with the following command:

	#show sdm prefer

Different SDM templates carve out the TCAM memory differently. 
TCAM memory is where forwarding information is held, i.e., 
IPv4/IPv6 routes, MAC addresses, e.t.c.

In legacy hardware-based platforms from Cisco, TCAM overflow 
results in traffic being punted to the CPU, which can lead 
to high CPU utilization in a busy network.

> sh platform tcam utilization
> _____________________________
> 
> 
> CAM Utilization for ASIC# 0                      Max            Used
>                                              Masks/Values
> Masks/values
> 
>  Unicast mac addresses:                        784/6272        757/5986

I'm concerned about this. 

You're border line on the values that the platform 
supports re: unicast MAC addresses.

>  IPv4 unicast directly-connected routes:       784/6272        757/5986

Also concerned about this.

You're very borderline on the v4 unicast routes that
are directly-connected. Just from looking at this, those
values correspond to your unicast MAC address count.

Does this look right to you?

> sh ip arp sum
> _________________
> 
> 5 IP ARP entries, with 0 of them incomplete

This looks okay, only 5 ARP entries.

> sh mac address-table count | i Space
> _____________________________________
> 
> Total Mac Address Space Available: 40

This is worrying, only 40 free TCAM slots in a
system that supports nearly 6,000 entries.

I think you're running low on TCAM slots and there could be
something else going on in your network that could be 
causing thrashing, exacerbating the problem and causing the
switch to forward packets via the software path instead of
the hardware path. Software forwarding would lead to high
CPU utilization.

Find out why you have that many directly connected routes
to begin with, and why each of them corresponds to a unique
MAC address.

Rebooting seems to help because the TCAM tables are probably
taking a while to fill up again, at which point the problem
reappears.

Solve that, and you should bring your CPU issue under control.

Hope this helps.

Cheers,

Mark.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://afnog.org/pipermail/afnog/attachments/20101028/abf1adbc/attachment.pgp>