cancel
Showing results for 
Search instead for 
Did you mean: 
wdecatur
Level 9

SEM\LEM not showing all events

Jump to solution

Why does LEM nDepth only show 20 events, console show 80k and Cisco show 31k

For the past day I've been struggling with why the events leaving my Cisco switches haven't all shown on LEM. At first I thought it was the Cisco devices not sending the data correctly, here is that config

Logging trap debug

logging fac local2

logging host myserverip trans tcp port 514

debug spanning all (Just to generate events)

Show logging

Trap logging: level debugging, 31009 message lines logged

        Logging to myserverip  (tcp port 514, audit disabled,

              link up),

              25601 message lines logged,

              0 message lines rate-limited,

              0 message lines dropped-by-MD,

              xml disabled, sequence number disabled

              filtering disabled

        Logging Source-Interface:       VRF Name:

But then I discovered the SEM console and the "Checklogs" command. Here is that output

        [1]: Syslog Log (260K)

        [2]: SNMP Trap Log (Empty)

        [3]: Snort Alert Log (Empty)

        [4]: Auth Log (Empty)

        [5]: Daemon Log (Empty)

        [6]: User Log (Empty)

        [7]: Rawsearch Log (Empty)

        [8]: Database Log (Empty)

        [9]: Manager Configuration Log (176K)

        [10]: Kernel Log (Empty)

        [11]: Migration log (Empty)

        [12]: Syslog local0 Log (Empty)

        [13]: Syslog local1 Log (Empty)

        [14]: Syslog local2 Log (80K)

        [15]: Syslog local3 Log (Empty)

        [16]: Syslog local4 Log (Empty)

        [17]: Syslog local5 Log (Empty)

        [18]: Syslog local6 Log (Empty)

        [19]: Syslog local7 Log (Empty)

        [20]: Cron Log (Empty)

        [21]: FTP Log (Empty)

        [22]: Printer Log (Empty)

        [23]: Mail Log (Empty)

        [24]: News Log (Empty)

        [25]: Unix-to-Unix Copy Log (Empty)

I can imagine the difference between Cisco and LEM because I have recreated this trap several times trying to get it to work, so 31k to 80k, yeah I can see that but 80,000 to 20? something isn't right

Going to ops center then opening my Cisco node, changing to the last week I only see where users log in or out but none of the STP messages I had generated with the "debug spanning all". What am I missing?

0 Kudos
14 Replies
jrouviere
Level 14

Re: SEM\LEM not showing all events

Jump to solution

I'm assuming you've got the appropriate Cisco connector set up and pointing at local2 on the SEM?

If that's the case, one thought is that the spanning logs may not be normalized or they may be dropped. It's not super common, but there are instances where clearly junk messages would be dropped. Are you able to trigger something else that you would want to see to confirm it comes through? If you've got an unused port, can you up/down the port to see if you find those logs in the console?

0 Kudos
wdecatur
Level 9

Re: SEM\LEM not showing all events

Jump to solution

pastedImage_0.pngI assume I have the correct connector setup and I have it setup as raw and normalized. Keep in mind I do see some events, but not everything. I'll try your idea about the ports here in a bit when I can get to the NOC

0 Kudos
jrouviere
Level 14

Re: SEM\LEM not showing all events

Jump to solution

Looks good to me.

I follow your description, but with some things I have a field of probabilities and don't want to make too few/many assumptions.

If you're getting some data, but not what you'd fully expect then I'd entertain the event normalization piece (data not being normalized for "reasons"), but it's a pretty small edge case so hopefully you'll be able to see real data for expected events.

If we were looking at it side by side we could probably figure it out in short order, so hopefully you see the events from your test which I'd say would make my theory be plausible, otherwise if you need it done in a crunch Support should be able to help demystify it pretty quickly.

Happy to keep discussing, just sometimes time is the more finite resource.

0 Kudos
wdecatur
Level 9

Re: SEM\LEM not showing all events

Jump to solution

So I ran the test and I see the status changes on LEM. I also got back UserLogonFailure: Logging to host (mymanagerIP) port 514 failed. I'll jump back into the switch and see if the UDP port is still configured (my firewall blocks UDP) but I am getting the status changes which tells me it is communicating (Originally I had setup the port as UDP but found firewall blocks so I moved to TCP so I might have both UDP and TCP configured)

Will report back


wdecatur
Level 9

Re: SEM\LEM not showing all events

Jump to solution

Fixed, re-ran test and I do see the events... just not everything. You maybe right, something is filtering out what is displayed, I just don't know how to change it. I'll call support in a bit, take notes and report back

wdecatur
Level 9

Re: SEM\LEM not showing all events

Jump to solution

I haven't forgotten about this. I got pulled into another project

0 Kudos
wdecatur
Level 9

Re: SEM\LEM not showing all events

Jump to solution

Here is where I am right now. According to support "SEM is a security and compliance tool" and that I should consider other device monitoring tools however they were willing to send it up to see if they can enable ALL events. Hopefully they can because I purchased several tools to cover our needs

We purchased Orion\SAM to monitor our hardware and major software applications, SEM (was LEM when we purchased) to gather all our events and create actions and finally ARM to take care of our security concerns

I'll post back here once I get more information

0 Kudos
wdecatur
Level 9

Re: SEM\LEM not showing all events

Jump to solution

Last I heard their dev team was still working on this. I'm going to send them another message to see if they have anything new

0 Kudos
Highlighted
choerichs
Level 8

Re: SEM\LEM not showing all events

Jump to solution

From my experience with LEM/SEM, unless you have a small organization, Solarwinds is right, LEM/SEM is not the tool for what you want to do.  Once you are getting above about 1.2 million events every 10min, SEM begins to have performance issues, and it's even worse if you start building filters with the all events.  In most medium size businesses, if you're logging handshakes, connection builds and teardowns, etc. plus all of the other logs that are normally sent or retrieved from other tools, you will easily go over a million events every 10 minutes.  We work with our engineers to strictly enforce sending security/audit logging to SEM to try to keep events under 1.5 million events every 10 min. and performance doing searches, and even just running flash is pretty sketchy at best.  Your milage may differ, but that's been my experience.

0 Kudos