Why does LEM nDepth only show 20 events, console show 80k and Cisco show 31k
For the past day I've been struggling with why the events leaving my Cisco switches haven't all shown on LEM. At first I thought it was the Cisco devices not sending the data correctly, here is that config
Logging trap debug
logging fac local2
logging host myserverip trans tcp port 514
debug spanning all (Just to generate events)
Trap logging: level debugging, 31009 message lines logged
Logging to myserverip (tcp port 514, audit disabled,
25601 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
Logging Source-Interface: VRF Name:
But then I discovered the SEM console and the "Checklogs" command. Here is that output
: Syslog Log (260K)
: SNMP Trap Log (Empty)
: Snort Alert Log (Empty)
: Auth Log (Empty)
: Daemon Log (Empty)
: User Log (Empty)
: Rawsearch Log (Empty)
: Database Log (Empty)
: Manager Configuration Log (176K)
: Kernel Log (Empty)
: Migration log (Empty)
: Syslog local0 Log (Empty)
: Syslog local1 Log (Empty)
: Syslog local2 Log (80K)
: Syslog local3 Log (Empty)
: Syslog local4 Log (Empty)
: Syslog local5 Log (Empty)
: Syslog local6 Log (Empty)
: Syslog local7 Log (Empty)
: Cron Log (Empty)
: FTP Log (Empty)
: Printer Log (Empty)
: Mail Log (Empty)
: News Log (Empty)
: Unix-to-Unix Copy Log (Empty)
I can imagine the difference between Cisco and LEM because I have recreated this trap several times trying to get it to work, so 31k to 80k, yeah I can see that but 80,000 to 20? something isn't right
Going to ops center then opening my Cisco node, changing to the last week I only see where users log in or out but none of the STP messages I had generated with the "debug spanning all". What am I missing?
Solved! Go to Solution.
At the end of the day those connector tools are just a big list of regex rules that parse the incoming event and decide what type of event they are and break out all the parts of the message for normalization. I've felt hackish before and exported mine out and imported back in a modified version as a new connector profile, but obviously that kind of thing is unsupported.
I'm assuming you've got the appropriate Cisco connector set up and pointing at local2 on the SEM?
If that's the case, one thought is that the spanning logs may not be normalized or they may be dropped. It's not super common, but there are instances where clearly junk messages would be dropped. Are you able to trigger something else that you would want to see to confirm it comes through? If you've got an unused port, can you up/down the port to see if you find those logs in the console?
I assume I have the correct connector setup and I have it setup as raw and normalized. Keep in mind I do see some events, but not everything. I'll try your idea about the ports here in a bit when I can get to the NOC
Looks good to me.
I follow your description, but with some things I have a field of probabilities and don't want to make too few/many assumptions.
If you're getting some data, but not what you'd fully expect then I'd entertain the event normalization piece (data not being normalized for "reasons"), but it's a pretty small edge case so hopefully you'll be able to see real data for expected events.
If we were looking at it side by side we could probably figure it out in short order, so hopefully you see the events from your test which I'd say would make my theory be plausible, otherwise if you need it done in a crunch Support should be able to help demystify it pretty quickly.
Happy to keep discussing, just sometimes time is the more finite resource.
So I ran the test and I see the status changes on LEM. I also got back UserLogonFailure: Logging to host (mymanagerIP) port 514 failed. I'll jump back into the switch and see if the UDP port is still configured (my firewall blocks UDP) but I am getting the status changes which tells me it is communicating (Originally I had setup the port as UDP but found firewall blocks so I moved to TCP so I might have both UDP and TCP configured)
Will report back
Fixed, re-ran test and I do see the events... just not everything. You maybe right, something is filtering out what is displayed, I just don't know how to change it. I'll call support in a bit, take notes and report back
Here is where I am right now. According to support "SEM is a security and compliance tool" and that I should consider other device monitoring tools however they were willing to send it up to see if they can enable ALL events. Hopefully they can because I purchased several tools to cover our needs
We purchased Orion\SAM to monitor our hardware and major software applications, SEM (was LEM when we purchased) to gather all our events and create actions and finally ARM to take care of our security concerns
I'll post back here once I get more information
From my experience with LEM/SEM, unless you have a small organization, Solarwinds is right, LEM/SEM is not the tool for what you want to do. Once you are getting above about 1.2 million events every 10min, SEM begins to have performance issues, and it's even worse if you start building filters with the all events. In most medium size businesses, if you're logging handshakes, connection builds and teardowns, etc. plus all of the other logs that are normally sent or retrieved from other tools, you will easily go over a million events every 10 minutes. We work with our engineers to strictly enforce sending security/audit logging to SEM to try to keep events under 1.5 million events every 10 min. and performance doing searches, and even just running flash is pretty sketchy at best. Your milage may differ, but that's been my experience.
SolarWinds solutions are rooted in our deep connection to our user base in the THWACK® online community. More than 150,000 members are here to solve problems, share technology and best practices, and directly contribute to our product development process. Learn more today by joining now.