11 Replies Latest reply on Aug 22, 2012 4:50 PM by nicole pauls

    SNMP Trap Action: rule does not fire when # of alerts is > 1

    mermel

      has anybody yet successfully created a rule with a SNMPTrap Action that fires when a correlation is configured based on the following (e.g.):

          Correlation Time

          2 Alerts within 40 seconds

       

      when I reduce the number of alerts to 1, the rule fires, if the number is raised, it fires no more...

      The basic alert is a "UserLogonFailure" and it is created based on a syslog message from a Cisco IOS Switch. The syslog message is created each time a user trys to logon to the Cisco switch with wrong credentials and each time the alert is triggered;nmp

        • Re: SNMP Trap Action: rule does not fire when # of alerts is > 1
          nicole pauls

          First, double check and make sure your "Response Window" (below correlation time) is also wide enough - it should be at least 40 seconds also. The default is 5 minutes, so you should be fine. This accounts for things like clock drift.

           

          Otherwise, everything sounds right. When you get 2 Cisco IOS Logon Failures in 40 seconds it doesn't fire, but when you get a single 1 and change the rule to 1 it does? It might come down to what you have in the correlations box. You'd want something like:

           

          UserLogonFailure.ProviderSID = "%blah-blah-blah"

          2 in 40 seconds

           

          You could also use an advanced threshold if you want it to be the same account (DestinationAccount SAME) or source IP (SourceMachine SAME).

           

          You can check the "SolarWinds Alerts" filter in Monitor to see rules firing, or look for InternalRuleFired alerts, which generally let you know what rules are being triggered and when. If the rule is being triggered and the action isn't being fired, it's something with the action, but it sounds like it's that the rule isn't being fired at all.

          1 of 1 people found this helpful
            • Re: SNMP Trap Action: rule does not fire when # of alerts is > 1
              mermel

              I let the response window at  5mins and the device and LEM (virtual appliance) are using the same NTP server. - How can I check that LEM is in sync with the NTP server? - I couldn't find a cli command or a menu in the GUI to verify this..

              The correlation is very simple, I choose

              UserLogonFailure.DetectionIP = 172.20.154.*

              2 in 40 secs

               

              no advancde threshold is configured. thats all. Changing this same rule to be

              1 in 40 secs

              the rule fires...

               

              As I use ssh to logon to the Cisco device I can see a pre-bulit rule for 3 successive failed ssh logons which I can force to fire also. I copied this one and added the SNMP Trap Action at the end - but I do not receive a trap..

                • Re: SNMP Trap Action: rule does not fire when # of alerts is > 1
                  nicole pauls

                  When you set the rule to fire on 1 in 40 seconds, does it fire once, or does it fire twice back to back?

                   

                  If it only fires once, it sounds like only one event is being generated. You could confirm that with a filter in Monitor for the exact same criteria (UserLogonFailure.DetectionIP=172.20.154.*).

                    • Re: SNMP Trap Action: rule does not fire when # of alerts is > 1
                      mermel

                      that is a good point but  with that definition the rule fires each and every time the syslog message comes in ...

                      so that is not the problem...

                      and I see an UserLogonFailure Alert for each failed logon

                        • Re: SNMP Trap Action: rule does not fire when # of alerts is > 1
                          nicole pauls

                          If you use a different action, like "Send Email", do you get one or two?

                            • Re: SNMP Trap Action: rule does not fire when # of alerts is > 1
                              mermel

                              i tried it with "Send PopUp" and the rule fires successfully; what I observerd is, having "1 in 40 sec" and the successfull "snmp action", I see "null [], null[] " as the value in the "ExtraneaousInfo" field, whereas for the "send popup" there is detailed information about

                                   the receiver IP,

                                   the popup text,

                                   the receiver ID

                                • Re: SNMP Trap Action: rule does not fire when # of alerts is > 1
                                  nicole pauls

                                  Thanks for the info. In the end, I was able to reproduce both things you reported:

                                  1. When an SNMP Trap Alert is sent, the InternalCommands message shows "null []" where it should show information about the SNMP Trap Alert that's being sent.
                                  2. When you use an SNMP Trap Alert as the action in a correlation with a threshold, a trap won't get generated. If it is the ONLY action the rule entirely does not fire. If there is a secondary action, that action fires, but the SNMP Trap Alert does not fire. If you remove the threshold (i.e. set it to 1) the rule does fire and the SNMP Trap Alert is sent.

                                   

                                  So, at this point, the workarounds are:

                                  1. Fire the SNMP Alert each time the failure happens (set threshold to 1). Kind of a bummer since you get multiples that you don't want.
                                  2. Build a rule that infers another alert based on the threshold of 2, then use that alert to trigger ANOTHER rule based on a threshold of 1. Works, but requires an extra hoop for you to jump through. For example:
                                    1. Rule 1: UserLogonFailure.DetectionIP=<yourIP>, 2 in 40 seconds, Infer Alert "FailedAuthentication" (make sure to fill out the fields by dragging them from your UserLogonFailure alert into the FailedAuthentication Infer Alert action)
                                    2. Rule 2: FailedAuthentication.DetectionIP=<yourIP>, 1 in X seconds, Send SNMP Trap Alert
                                      1. If you need additional criteria, you could also use FailedAuthentication.InferenceRule="Rule 1's Name Here" to make sure it's ONLY those events that fire the SNMP Trap.

                                   

                                  If you want to see an example of #2, let me know and I'll generate them and export them as examples that you can import.

                                   

                                  Can you submit a support case so that we can track and notify you when this issue is resolved? I will file your issue as well since I was able to reproduce consistently. I will also try to follow up in this thread when the issue is resolved.

                                   

                                  Edit: Before the community team beats me to mentioning: if you do file a support case, post the # here so we can refer back to it. Thanks!!

                                  1 of 1 people found this helpful
                                    • Re: SNMP Trap Action: rule does not fire when # of alerts is > 1
                                      mermel

                                      Thanks Nicole for reproducing the issue.

                                      In fact I have a support case open, yet; the Support Case # is 367887.

                                      I posted the question here because I wanted to know if I am really the only one who is seeing this issue - and indeed, it seems that I am the only one who is using this feature... ;-).

                                       

                                      Workarounds:

                                      1) is not an option, because I started with this very simple one, but in the end the logic should be more advanced than just "count it"  :-)

                                      2) I yet thought about this possibility and did some quick tests but perhaps I have to do some more advanced because what I saw was that much of the original information gets lost in the SNMP Trap because it fires on the Infer Alert.

                                       

                                      Hopefully this gets resolved very quickly, because we bought this software exactly because of this feature.... - I thought it is a standard feature and I didn't test it  :-(

                                      - ok, the feature is there - but it is not working ;-)

                                       

                                       

                                      Now, that you are looking at this I will take the oportunity to point you on my feature request regarding the "Snmp Trap Action" I posted here:

                                           SNMP Trap Action: new free text field combined with variables

                                       

                                      It would be great if there would be (at least) one field where you could use the standard variables (e.g. UserLogonFailure.DetectionIP, UserLogonFailure.DetectionTime, etc.) and add them one behind the ohter (seperated by or added with any free text). Currently, if I drag and drop these vars from the left pain to the fields in the right pane, one var is substituted with the new one and not added (- which is clear for the existing fields!). Having this new field - being the same for all: Alerts (or Infer Alerts) and Actions, I myself can define which Information I want to keep and carry over to new Alerts or Actions.

                                      In the end, its just one new field that I want (or a couple of these :-) ), like Info_1; Info_2; Info_3; ....

                                        • Re: SNMP Trap Action: rule does not fire when # of alerts is > 1
                                          nicole pauls

                                          For Option #2, if you select the FailedAuthentication alert as your inferred alert, the fields should be nearly identical to the UserLogonFailure alert so you don't lose any field data (until you get to the SNMP step, which relates to your second point ).

                                           

                                          Rule #1: 2 UserLogonFailures in 40 seconds, action Infer Alert, use FailedAuthentication (Generic > Security > Suspicious > AuthSuspicious > FailedAuthentication)

                                          Rule #2: FailedAuthentication exists (with criteria), action Send SNMP Trap

                                           

                                          Thanks for the feedback on the Send SNMP/Infer actions. Something like what we do in Email Templates where you can concatenate free text with field data would definitely be useful.

                                           

                                          I'll make sure your support case gets attached to the bug report.