6 Replies Latest reply on Oct 29, 2019 9:59 AM by wdecatur

    SEM\LEM not showing all events


      Why does LEM nDepth only show 20 events, console show 80k and Cisco show 31k

      For the past day I've been struggling with why the events leaving my Cisco switches haven't all shown on LEM. At first I thought it was the Cisco devices not sending the data correctly, here is that config


      Logging trap debug

      logging fac local2

      logging host myserverip trans tcp port 514

      debug spanning all (Just to generate events)


      Show logging

      Trap logging: level debugging, 31009 message lines logged

              Logging to myserverip  (tcp port 514, audit disabled,

                    link up),

                    25601 message lines logged,

                    0 message lines rate-limited,

                    0 message lines dropped-by-MD,

                    xml disabled, sequence number disabled

                    filtering disabled

              Logging Source-Interface:       VRF Name:


      But then I discovered the SEM console and the "Checklogs" command. Here is that output

              [1]: Syslog Log (260K)

              [2]: SNMP Trap Log (Empty)

              [3]: Snort Alert Log (Empty)

              [4]: Auth Log (Empty)

              [5]: Daemon Log (Empty)

              [6]: User Log (Empty)

              [7]: Rawsearch Log (Empty)

              [8]: Database Log (Empty)

              [9]: Manager Configuration Log (176K)

              [10]: Kernel Log (Empty)

              [11]: Migration log (Empty)

              [12]: Syslog local0 Log (Empty)

              [13]: Syslog local1 Log (Empty)

              [14]: Syslog local2 Log (80K)

              [15]: Syslog local3 Log (Empty)

              [16]: Syslog local4 Log (Empty)

              [17]: Syslog local5 Log (Empty)

              [18]: Syslog local6 Log (Empty)

              [19]: Syslog local7 Log (Empty)

              [20]: Cron Log (Empty)

              [21]: FTP Log (Empty)

              [22]: Printer Log (Empty)

              [23]: Mail Log (Empty)

              [24]: News Log (Empty)

              [25]: Unix-to-Unix Copy Log (Empty)


      I can imagine the difference between Cisco and LEM because I have recreated this trap several times trying to get it to work, so 31k to 80k, yeah I can see that but 80,000 to 20? something isn't right


      Going to ops center then opening my Cisco node, changing to the last week I only see where users log in or out but none of the STP messages I had generated with the "debug spanning all". What am I missing?

        • Re: SEM\LEM not showing all events

          I'm assuming you've got the appropriate Cisco connector set up and pointing at local2 on the SEM?


          If that's the case, one thought is that the spanning logs may not be normalized or they may be dropped. It's not super common, but there are instances where clearly junk messages would be dropped. Are you able to trigger something else that you would want to see to confirm it comes through? If you've got an unused port, can you up/down the port to see if you find those logs in the console?

            • Re: SEM\LEM not showing all events

              I assume I have the correct connector setup and I have it setup as raw and normalized. Keep in mind I do see some events, but not everything. I'll try your idea about the ports here in a bit when I can get to the NOC

                • Re: SEM\LEM not showing all events

                  Looks good to me.


                  I follow your description, but with some things I have a field of probabilities and don't want to make too few/many assumptions.


                  If you're getting some data, but not what you'd fully expect then I'd entertain the event normalization piece (data not being normalized for "reasons"), but it's a pretty small edge case so hopefully you'll be able to see real data for expected events.


                  If we were looking at it side by side we could probably figure it out in short order, so hopefully you see the events from your test which I'd say would make my theory be plausible, otherwise if you need it done in a crunch Support should be able to help demystify it pretty quickly.


                  Happy to keep discussing, just sometimes time is the more finite resource.

                    • Re: SEM\LEM not showing all events

                      So I ran the test and I see the status changes on LEM. I also got back UserLogonFailure: Logging to host (mymanagerIP) port 514 failed. I'll jump back into the switch and see if the UDP port is still configured (my firewall blocks UDP) but I am getting the status changes which tells me it is communicating (Originally I had setup the port as UDP but found firewall blocks so I moved to TCP so I might have both UDP and TCP configured)

                      Will report back

                • Re: SEM\LEM not showing all events

                  I haven't forgotten about this. I got pulled into another project