Much of my work at my current job as of late has been in a consulting capacity more than anything. Instead of driving security projects or being the primary workhorse behind a given project, my role has been to provide security advice, help the given business unit prioritize a risk in the grand security scheme, and do some shepherding along its path to completion.
Not too long ago we had a fairly visible outage in one of our environments, and some groups were left scrambing in an effort to try and explain what happened and how it could be prevented in the future. Unfortunately, there was minimal logging going on, no correlation, no alerting, and little if any monitoring in place. I was asked to address these issues, and within a few hours had replicated the SEC log analysis and correlation setup we have in other environments.
In my haste I appear to have partially botched the configuration, and I was ignoring large swaths of FWSM syslog messages. Fortunately, none of those syslog message types have occured since my initial screw-up, so my bacon was saved. When I discovered this earlier today, I figured now was a good time to do some more detailed filtering on the FWSM messages we get alerted on.
After many difficult google searches and trudging through Cisco's site, I found this link. "Great!", I thought, learning that FWSM (and PIX) messages are prefixed by their severity. I was this close to implementing something that would immediately alert on severity 1 and 2 messages, and pile up larger amounts of 3 and 4 for bulk alerting, and can the rest. Well, Cisco apparently thought better than to make my life that easy, and tossed the notion of "severity" right out the window.
As an example of some of Cisco's severity ratings, behold the following gems:
- Failing over (%FWSM-1-104002) is just as severe as receiving a spoofed packet (%FWSM-1-106022)
- Dropping a UDP packet (%FWSM-2-106006) is just as bad as if your device horks itself so bad (%FWSM-2-215001), you need a TAC case..
- Dropping ICMP packets (%FWSM-3-106014) is on-par with your Cisco device failing to malloc() memory ("RAM System Memory"), aka %FWSM-3-211001
The good news here is that I was able to warp my mind in such a way that I too could understand the logic here, and accurate alerting is now in place.