[afnog] High CPU Utilization Problem on 2-Stacked Cisco Switches.

Michael Komla Nfodzo Michael.Nfodzo at vodafone.com
Tue Oct 26 11:11:25 UTC 2010


Hi All,

Can someone, please, look into this problem for me and provide me with
the assistance to resolve it.

Regards
Michael


-----Original Message-----
From: Michael Komla Nfodzo 
Sent: Tuesday, October 19, 2010 11:12 AM
To: 'mtinka at globaltransit.net'
Cc: afnog at afnog.org; Eric Ayivor; Kwame Asomaning-Sakyi
Subject: RE: [afnog] High CPU Utilization Problem on 2-Stacked Cisco
Switches.

Hi Mark,

Thanks so much for your prompt response. Please find below the output of
the commands as requested.


What SDM template are you running?------------------------Not running
SDM

What do the following say:

sh platform tcam utilization
_____________________________


CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values
Masks/values

 Unicast mac addresses:                        784/6272        757/5986
 IPv4 IGMP groups + multicast routes:          152/1216          6/26
 IPv4 unicast directly-connected routes:       784/6272        757/5986
 IPv4 unicast indirectly-connected routes:     272/2176          8/44
 IPv4 policy based routing aces:                 0/0             0/0
 IPv4 qos aces:                                768/768         260/260
 IPv4 security aces:                          1024/1024         33/33

Note: Allocation of TCAM entries per feature uses a complex algorithm.
The above information is meant to provide an abstract view of the
current TCAM utilization




sh ip arp sum
_________________

5 IP ARP entries, with 0 of them incomplete



sh mac address-table count | i Space
_____________________________________

Total Mac Address Space Available: 40



sh ver
________________________________________

HS-SWITCH-STACK#sh ver
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version
12.2(53)SE, RELEASE SOFTWARE (fc2) Technical Support:
http://www.cisco.com/techsupport Copyright (c) 1986-2009 by Cisco
Systems, Inc.
Compiled Sun 13-Dec-09 16:25 by prod_rel_team Image text-base:
0x01000000, data-base: 0x02C00000

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE
SOFTWARE (fc1)

HS-SWITCH-STACK uptime is 1 day, 4 hours, 57 minutes System returned to
ROM by power-on System image file is
"flash:c3750-ipbasek9-mz.122-53.SE.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply third-party
authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be
found
at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export at cisco.com.

cisco WS-C3750G-12S (PowerPC405) processor (revision N0) with 131072K
bytes of memory.
Processor board ID FCZ1326213Y
Last reset from power-on
3 Virtual Ethernet interfaces
40 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 00:26:0B:3A:3C:80
Motherboard assembly number     : 73-9678-11
Power supply part number        : 341-0048-03
Motherboard serial number       : FDO13230CEG
Power supply serial number      : LIT13050JWB
Model revision number           : N0
Motherboard revision number     : A0
Model number                    : WS-C3750G-12S-E
System serial number            : FCZ1326213Y
Top Assembly Part Number        : 800-26634-08
Top Assembly Revision Number    : A0
Version ID                      : V10
CLEI Code Number                : COMDN10BRA
Hardware Board Revision Number  : 0x06


Switch Ports Model              SW Version            SW Image
------ ----- -----              ----------            ----------
*    1 12    WS-C3750G-12S      12.2(53)SE            C3750-IPBASEK9-M
     2 28    WS-C3750G-24TS-1U  12.2(53)SE            C3750-IPBASEK9-M


Switch 02
---------
Switch Uptime                   : 1 day, 4 hours, 57 minutes
Base ethernet MAC Address       : 00:18:73:A3:67:00
Motherboard assembly number     : 73-9637-08
Power supply part number        : 341-0098-02
Motherboard serial number       : FOC10242ZLC
Power supply serial number      : AZS102300EZ
Model revision number           : C0
Motherboard revision number     : A0
Model number                    : WS-C3750G-24TS-S1U
System serial number            : FOC1025Y0P4
SFP Module assembly part number : 73-7757-03
SFP Module revision number      : A0
SFP Module serial number        : CAT10180BPM
Top assembly part number        : 800-25730-02
Top assembly revision number    : B0
Version ID                      : V02
CLEI Code Number                : CNMWS00ARB

Configuration register is 0xF


Regards
Michael



-----Original Message-----
From: Mark Tinka [mailto:mtinka at globaltransit.net] 
Sent: Monday, October 18, 2010 11:26 PM
To: Michael Komla Nfodzo
Cc: afnog at afnog.org; Eric Ayivor; Kwame Asomaning-Sakyi
Subject: Re: [afnog] High CPU Utilization Problem on 2-Stacked Cisco
Switches.

On Tuesday, October 19, 2010 12:58:59 am Michael Komla Nfodzo wrote:

> Hi Folks,

> The process(HLFM address ret)  is causing High CPU utilization problem

> on my 2-stacked switches and is highlighted in yellow below.  As you 
> can see, the interrupt (traffic load) is 1%

> SWITCH#sh processes cpu sorted
> 
> CPU utilization for five seconds: 73%/1%; one minute:
> 75%; five minutes: 75%
> 
>  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min  
> 5Min TTY Process
> 
>   79   108148177  18204966       5940 40.50% 39.73%
> 40.06%   0 HLFM address ret
> 
>   77    49170833  18188441       2703 11.96% 12.04%
> 11.80%   0 HLFM address lea
> 
>  210    91092894  92627097        983 11.32% 12.28%
> 12.16%   0 Spanning Tree

You seem to be spinning major CPU cycles on the top 3 as well.

What SDM template are you running?

What do the following say:

	sh platform tcam utilization
	sh ip arp sum
	sh mac address-table count | i Space
	sh ver

Cheers,

Mark.



More information about the afnog mailing list