Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

SEM\LEM not showing all events

Why does LEM nDepth only show 20 events, console show 80k and Cisco show 31k

For the past day I've been struggling with why the events leaving my Cisco switches haven't all shown on LEM. At first I thought it was the Cisco devices not sending the data correctly, here is that config

Logging trap debug

logging fac local2

logging host myserverip trans tcp port 514

debug spanning all (Just to generate events)

Show logging

Trap logging: level debugging, 31009 message lines logged

Logging to myserverip (tcp port 514, audit disabled,

link up),

25601 message lines logged,

0 message lines rate-limited,

0 message lines dropped-by-MD,

xml disabled, sequence number disabled

filtering disabled

Logging Source-Interface: VRF Name:

But then I discovered the SEM console and the "Checklogs" command. Here is that output

[1]: Syslog Log (260K)

[2]: SNMP Trap Log (Empty)

[3]: Snort Alert Log (Empty)

[4]: Auth Log (Empty)

[5]: Daemon Log (Empty)

[6]: User Log (Empty)

[7]: Rawsearch Log (Empty)

[8]: Database Log (Empty)

[9]: Manager Configuration Log (176K)

[10]: Kernel Log (Empty)

[11]: Migration log (Empty)

[12]: Syslog local0 Log (Empty)

[13]: Syslog local1 Log (Empty)

[14]: Syslog local2 Log (80K)

[15]: Syslog local3 Log (Empty)

[16]: Syslog local4 Log (Empty)

[17]: Syslog local5 Log (Empty)

[18]: Syslog local6 Log (Empty)

[19]: Syslog local7 Log (Empty)

[20]: Cron Log (Empty)

[21]: FTP Log (Empty)

[22]: Printer Log (Empty)

[23]: Mail Log (Empty)

[24]: News Log (Empty)

[25]: Unix-to-Unix Copy Log (Empty)

I can imagine the difference between Cisco and LEM because I have recreated this trap several times trying to get it to work, so 31k to 80k, yeah I can see that but 80,000 to 20? something isn't right

Going to ops center then opening my Cisco node, changing to the last week I only see where users log in or out but none of the STP messages I had generated with the "debug spanning all". What am I missing?

Find more posts tagged with

Accepted answers

mesverrum

At the end of the day those connector tools are just a big list of regex rules that parse the incoming event and decide what type of event they are and break out all the parts of the message for normalization. I've felt hackish before and exported mine out and imported back in a modified version as a new connector profile, but obviously that kind of thing is unsupported.

All comments

jrouviere

I'm assuming you've got the appropriate Cisco connector set up and pointing at local2 on the SEM?

If that's the case, one thought is that the spanning logs may not be normalized or they may be dropped. It's not super common, but there are instances where clearly junk messages would be dropped. Are you able to trigger something else that you would want to see to confirm it comes through? If you've got an unused port, can you up/down the port to see if you find those logs in the console?

wdecatur

I assume I have the correct connector setup and I have it setup as raw and normalized. Keep in mind I do see some events, but not everything. I'll try your idea about the ports here in a bit when I can get to the NOC

jrouviere

Looks good to me.

I follow your description, but with some things I have a field of probabilities and don't want to make too few/many assumptions.

If you're getting some data, but not what you'd fully expect then I'd entertain the event normalization piece (data not being normalized for "reasons"), but it's a pretty small edge case so hopefully you'll be able to see real data for expected events.

If we were looking at it side by side we could probably figure it out in short order, so hopefully you see the events from your test which I'd say would make my theory be plausible, otherwise if you need it done in a crunch Support should be able to help demystify it pretty quickly.

Happy to keep discussing, just sometimes time is the more finite resource.

wdecatur

So I ran the test and I see the status changes on LEM. I also got back UserLogonFailure: Logging to host (mymanagerIP) port 514 failed. I'll jump back into the switch and see if the UDP port is still configured (my firewall blocks UDP) but I am getting the status changes which tells me it is communicating (Originally I had setup the port as UDP but found firewall blocks so I moved to TCP so I might have both UDP and TCP configured)

Will report back

wdecatur

I haven't forgotten about this. I got pulled into another project

wdecatur

Here is where I am right now. According to support "SEM is a security and compliance tool" and that I should consider other device monitoring tools however they were willing to send it up to see if they can enable ALL events. Hopefully they can because I purchased several tools to cover our needs

We purchased Orion\SAM to monitor our hardware and major software applications, SEM (was LEM when we purchased) to gather all our events and create actions and finally ARM to take care of our security concerns

I'll post back here once I get more information

wdecatur

Last I heard their dev team was still working on this. I'm going to send them another message to see if they have anything new

choerichs

From my experience with LEM/SEM, unless you have a small organization, Solarwinds is right, LEM/SEM is not the tool for what you want to do. Once you are getting above about 1.2 million events every 10min, SEM begins to have performance issues, and it's even worse if you start building filters with the all events. In most medium size businesses, if you're logging handshakes, connection builds and teardowns, etc. plus all of the other logs that are normally sent or retrieved from other tools, you will easily go over a million events every 10 minutes. We work with our engineers to strictly enforce sending security/audit logging to SEM to try to keep events under 1.5 million events every 10 min. and performance doing searches, and even just running flash is pretty sketchy at best. Your milage may differ, but that's been my experience.

wdecatur

That's my understanding as well, sadly however that's not what the account manager told my boss when she was on the phone with him. We were looking for something that would hold all our log files and allow us to sort through them to track down a variety of problems. One of our biggest issues is trying to do a root cause analyst when issues occur. I often bring up STP on our network devices as an example because that was one such issue that this system should have helped me resolve. I had installed a new switch, configured it properly but never checked to make sure it wasn't going to be the root. That new switch became the root and ended up disabling the ether channel I had on another causing the now only 1GB link to overload bringing the network to its knees on several occasions. Sadly it took me almost a month to track down as it was random and by the time I got into the network to figure it out the problem was resolved. When we were originally looking for an application we had considered several different applications, Splunk being one of them. However considering the price and the idea that LEM would do exactly what we wanted we chose LEM.

I understand this could potentially crash the system if I gather too many alerts but I'm confident I can do exactly what you do, only send to LEM what I want to gather. I honestly would much rather be in control over this than hard limitations. With that said I did just get an email from support telling me that the connector is finished with instructions on how to install.

But first I need to contact Support to figure out why the VM keeps crashing. Once I get all this done I'll write up something pretty and post here for any future person having the same issue