Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 9

SEM\LEM not showing all events

Jump to solution

Why does LEM nDepth only show 20 events, console show 80k and Cisco show 31k

For the past day I've been struggling with why the events leaving my Cisco switches haven't all shown on LEM. At first I thought it was the Cisco devices not sending the data correctly, here is that config

Logging trap debug

logging fac local2

logging host myserverip trans tcp port 514

debug spanning all (Just to generate events)

Show logging

Trap logging: level debugging, 31009 message lines logged

        Logging to myserverip  (tcp port 514, audit disabled,

              link up),

              25601 message lines logged,

              0 message lines rate-limited,

              0 message lines dropped-by-MD,

              xml disabled, sequence number disabled

              filtering disabled

        Logging Source-Interface:       VRF Name:

But then I discovered the SEM console and the "Checklogs" command. Here is that output

        [1]: Syslog Log (260K)

        [2]: SNMP Trap Log (Empty)

        [3]: Snort Alert Log (Empty)

        [4]: Auth Log (Empty)

        [5]: Daemon Log (Empty)

        [6]: User Log (Empty)

        [7]: Rawsearch Log (Empty)

        [8]: Database Log (Empty)

        [9]: Manager Configuration Log (176K)

        [10]: Kernel Log (Empty)

        [11]: Migration log (Empty)

        [12]: Syslog local0 Log (Empty)

        [13]: Syslog local1 Log (Empty)

        [14]: Syslog local2 Log (80K)

        [15]: Syslog local3 Log (Empty)

        [16]: Syslog local4 Log (Empty)

        [17]: Syslog local5 Log (Empty)

        [18]: Syslog local6 Log (Empty)

        [19]: Syslog local7 Log (Empty)

        [20]: Cron Log (Empty)

        [21]: FTP Log (Empty)

        [22]: Printer Log (Empty)

        [23]: Mail Log (Empty)

        [24]: News Log (Empty)

        [25]: Unix-to-Unix Copy Log (Empty)

I can imagine the difference between Cisco and LEM because I have recreated this trap several times trying to get it to work, so 31k to 80k, yeah I can see that but 80,000 to 20? something isn't right

Going to ops center then opening my Cisco node, changing to the last week I only see where users log in or out but none of the STP messages I had generated with the "debug spanning all". What am I missing?

0 Kudos
1 Solution

At the end of the day those connector tools are just a big list of regex rules that parse the incoming event and decide what type of event they are and break out all the parts of the message for normalization.  I've felt hackish before and exported mine out and imported back in a modified version as a new connector profile, but obviously that kind of thing is unsupported.

- Marc Netterfield, Github

View solution in original post

14 Replies
Level 9

I haven't forgotten about this. I got pulled into another project

0 Kudos

Here is where I am right now. According to support "SEM is a security and compliance tool" and that I should consider other device monitoring tools however they were willing to send it up to see if they can enable ALL events. Hopefully they can because I purchased several tools to cover our needs

We purchased Orion\SAM to monitor our hardware and major software applications, SEM (was LEM when we purchased) to gather all our events and create actions and finally ARM to take care of our security concerns

I'll post back here once I get more information

0 Kudos

From my experience with LEM/SEM, unless you have a small organization, Solarwinds is right, LEM/SEM is not the tool for what you want to do.  Once you are getting above about 1.2 million events every 10min, SEM begins to have performance issues, and it's even worse if you start building filters with the all events.  In most medium size businesses, if you're logging handshakes, connection builds and teardowns, etc. plus all of the other logs that are normally sent or retrieved from other tools, you will easily go over a million events every 10 minutes.  We work with our engineers to strictly enforce sending security/audit logging to SEM to try to keep events under 1.5 million events every 10 min. and performance doing searches, and even just running flash is pretty sketchy at best.  Your milage may differ, but that's been my experience.

0 Kudos

That's my understanding as well, sadly however that's not what the account manager told my boss when she was on the phone with him. We were looking for something that would hold all our log files and allow us to sort through them to track down a variety of problems. One of our biggest issues is trying to do a root cause analyst when issues occur. I often bring up STP on our network devices as an example because that was one such issue that this system should have helped me resolve. I had installed a new switch, configured it properly but never checked to make sure it wasn't going to be the root. That new switch became the root and ended up disabling the ether channel I had on another causing the now only 1GB link to overload bringing the network to its knees on several occasions. Sadly it took me almost a month to track down as it was random and by the time I got into the network to figure it out the problem was resolved.  When we were originally looking for an application we had considered several different applications, Splunk being one of them. However considering the price and the idea that LEM would do exactly what we wanted we chose LEM.

I understand this could potentially crash the system if I gather too many alerts but I'm confident I can do exactly what you do, only send to LEM what I want to gather. I honestly would much rather be in control over this than hard limitations. With that said I did just get an email from support telling me that the connector is finished with instructions on how to install.

But first I need to contact Support to figure out why the VM keeps crashing. Once I get all this done I'll write up something pretty and post here for any future person having the same issue

0 Kudos

At the end of the day those connector tools are just a big list of regex rules that parse the incoming event and decide what type of event they are and break out all the parts of the message for normalization.  I've felt hackish before and exported mine out and imported back in a modified version as a new connector profile, but obviously that kind of thing is unsupported.

- Marc Netterfield, Github

View solution in original post

I'd be curious to know what you've done like this mesverrum​ sounds interesting... have you tried to make one from scratch before or figured out the syntax pretty well now?


0 Kudos

I was modifying an existing template as a one-off, but the syntax I was messing with was basically just regex with capture groups.  Building the logic wasn't rocket science but to build one completely from scratch would be a pretty significant amount of work to create all the rules.

- Marc Netterfield, Github

I wish I had a better understanding of doing just that. This program feel vastly different than the other SolarWinds products I've used.

But on a plus side I did manage to get my problem resolved and it was just about what you have done, except I didn't do it. The Dev team over at SolarWinds managed to write me up a new connector, gave me instructions on how to upload it and … with a tech support call managed to get the switches using it. I'm still confused as to how this all works but what your saying makes sense, all that connector does is filter out information. This new connector doesn't do any of that, just posts the raw data. With that said I'm guessing I've increased my events from say 10 a week per switch to around 10k, well below what my system can handle but we shall see as I start increasing what is sent. That procedure, well that's an entirely different subject

For those having the same problem, I wish I could give you a step by step on how to resolve this but it required getting the connector from SolarWinds

0 Kudos

Last I heard their dev team was still working on this. I'm going to send them another message to see if they have anything new

0 Kudos
Level 14

I'm assuming you've got the appropriate Cisco connector set up and pointing at local2 on the SEM?

If that's the case, one thought is that the spanning logs may not be normalized or they may be dropped. It's not super common, but there are instances where clearly junk messages would be dropped. Are you able to trigger something else that you would want to see to confirm it comes through? If you've got an unused port, can you up/down the port to see if you find those logs in the console?

0 Kudos

pastedImage_0.pngI assume I have the correct connector setup and I have it setup as raw and normalized. Keep in mind I do see some events, but not everything. I'll try your idea about the ports here in a bit when I can get to the NOC

0 Kudos

Looks good to me.

I follow your description, but with some things I have a field of probabilities and don't want to make too few/many assumptions.

If you're getting some data, but not what you'd fully expect then I'd entertain the event normalization piece (data not being normalized for "reasons"), but it's a pretty small edge case so hopefully you'll be able to see real data for expected events.

If we were looking at it side by side we could probably figure it out in short order, so hopefully you see the events from your test which I'd say would make my theory be plausible, otherwise if you need it done in a crunch Support should be able to help demystify it pretty quickly.

Happy to keep discussing, just sometimes time is the more finite resource.

0 Kudos

So I ran the test and I see the status changes on LEM. I also got back UserLogonFailure: Logging to host (mymanagerIP) port 514 failed. I'll jump back into the switch and see if the UDP port is still configured (my firewall blocks UDP) but I am getting the status changes which tells me it is communicating (Originally I had setup the port as UDP but found firewall blocks so I moved to TCP so I might have both UDP and TCP configured)

Will report back

Fixed, re-ran test and I do see the events... just not everything. You maybe right, something is filtering out what is displayed, I just don't know how to change it. I'll call support in a bit, take notes and report back