Same L3 switch, new problem. Some vital stats for reference:

• cisco Catalyst 3750E
• 250+ VLAN interfaces
• 100+ Gig/10GE interfaces
• OSPF
• PBR is used to specify a different next-hop
• The same route-map is applied to each interface so that the route-map only needs one corresponding access-list for the match clause.

We got alerted by our monitoring to some consistently high CPU load. We followed the check list for troubleshooting 3750 CPU load here and didn't see anything that particularly caught our eye. Checking revision control (you have configuration revision control, don't you?) a change commit was found that correlated perfectly with the increase in CPU utilisation. It turns out that cisco's check list was relevant and this particular section explained it:

When configuring match criteria in a route map, follow these guidelines:

• Do not match ACLs with deny ACEs. Packets that match a deny ACE are sent to the CPU, which could cause high CPU utilization.

The ACL in use for the route-map had four deny lines at the start with a combined match count in the hundred of millions.

Lesson learned: when you think you've read all of the best practice guides, there'll be another one you haven't had to read until something goes wrong.

Simply put, by default each vCPU you configure a guest with is presented to the Guest OS as a socket. Various benchmarks suggest there's no performance difference between (for instance) a four single-core socket VM vs a single four-core socket VM, which is fine and dandy.

Unless you use fancy software that does CPU licensing.

This post has a great explanation, with anecdotal evidence that you can fix this, albeit expreimentally, in ESX 3.5u2 and beyond.

Also, something I found today that is the (now second-) most useful thing I'll find all day:

Access to Sysinternals Tools without installing them!

Something that has been meandered into (somewhat blindly, unfortunately) in recent weeks is the finite resources that most people don't exhaust when they use everyday equipment in everyday situations.

A particular deployment I manage uses a cisco Catalyst 3750E stack, with 4x WS-C3750E-24TD-E in the network access layer. Some stats, showing that we work our switches pretty hard:

• 250+ VLANs, each with a /27;
• Enforces security on VLAN egress;
• Runs OSPF and exchanges routes with other similarly-configured deployments;
• Currently hosts 22 physical servers, each with 4 NICs in two Port-channels.

No prizes for guessing what those hosts are doing.

A new VLAN was added recently, ACL added to the config (which is well over 100k!) and servers powered on. The provisioning guys rang me back and said they were seeing weird intermittent networking issues. After a bit of investigation (thanks JF) we came up with something along the lines of this:

%QATM-4-TCAM_LOW: TCAM resource running low for table [chars], resource type [chars], on TCAM number [dec]

It seems that we'd selected the second most appropriate SDM template for our deployment of these stacks. This stack was set up using the command 'sdm prefer routing' and not 'sdm prefer access' - these commands change the way that the Ternary Content Addressable Memory is allocated between the various roles. cisco list the various ways in which TCAM can be allocated here. From that, as you can see, the access SDM template gives a lot more space to the appropriate areas - Security ACEs.

The easy way to keep tabs on this is with the command:

This is the allocation for SDM's 'routing' template.

Behold, there are ways to manage these finite resources! An exceptionally useful way of reducing the space allocated is to re-use the same ACL on multiple interfaces to combine common rules, use of the rule 'permit tcp any any established' to reduce the need for bi-directional rules in stateless firewalls.

Another gotcha we face - we have to be mindful of the limit of 48 Port-channels per switch stack. When you can stack up to 9x48(+4) ports of gigabit goodness into one switch, you could conceivably run into that limit rather quickly.