Managing finite resources on cisco Catalyst switches

Something that has been meandered into (somewhat blindly, unfortunately) in recent weeks is the finite resources that most people don't exhaust when they use everyday equipment in everyday situations.

A particular deployment I manage uses a cisco Catalyst 3750E stack, with 4x WS-C3750E-24TD-E in the network access layer. Some stats, showing that we work our switches pretty hard:

• 250+ VLANs, each with a /27;
• Enforces security on VLAN egress;
• Runs OSPF and exchanges routes with other similarly-configured deployments;
• Currently hosts 22 physical servers, each with 4 NICs in two Port-channels.

No prizes for guessing what those hosts are doing.

A new VLAN was added recently, ACL added to the config (which is well over 100k!) and servers powered on. The provisioning guys rang me back and said they were seeing weird intermittent networking issues. After a bit of investigation (thanks JF) we came up with something along the lines of this:

%QATM-4-TCAM_LOW: TCAM resource running low for table [chars], resource type [chars], on TCAM number [dec]

It seems that we'd selected the second most appropriate SDM template for our deployment of these stacks. This stack was set up using the command 'sdm prefer routing' and not 'sdm prefer access' - these commands change the way that the Ternary Content Addressable Memory is allocated between the various roles. cisco list the various ways in which TCAM can be allocated here. From that, as you can see, the access SDM template gives a lot more space to the appropriate areas - Security ACEs.

The easy way to keep tabs on this is with the command:

This is the allocation for SDM's 'routing' template.

Behold, there are ways to manage these finite resources! An exceptionally useful way of reducing the space allocated is to re-use the same ACL on multiple interfaces to combine common rules, use of the rule 'permit tcp any any established' to reduce the need for bi-directional rules in stateless firewalls.

Another gotcha we face - we have to be mindful of the limit of 48 Port-channels per switch stack. When you can stack up to 9x48(+4) ports of gigabit goodness into one switch, you could conceivably run into that limit rather quickly.