Faulty Mute Nodes over API-Call

Hey everybody,

currently we are implementing some automation in our enviroment. Our first idea is to mute and unmute our server in SolarWinds, when we have patches by our customers.

The automation is build to schedule a mute from now to a week further.

The Audit Events looking great.

Now our Problem:

On two patchdays the mute on the node disapperared without any warning and an alert "Node down" was triggered.

In the Audit Events no "Unmute" is visible.

Have anyone a clue in which log-files i can look to get more information why the mute disapperared or any idea why this can happen?

Regards

Andreas

Find more posts tagged with

Mute

mute node

API Call

API

orion sdk

Accepted answers

FormerMember

So after a bit more testing and searching (and losing many nerves), i think i finally found the mistake.

@KMSigma.SWI gave me the hintch to look for a differenc between a mute setting over the dashboard an a mute creating over the automation.

On the webdashboard it looks the same, but in the orion database was a difference.

Comparing the two mutes i saw the "Entity Uri" wasn't the same. So in our automation we used a wrong swis "url".

After changing the swis to the one, which is used over the dashboard the mute works fine and i'm also able to deactivate the mute which was set via automation.

The only question thats remain is, why the mute was working for round about 1-2 hours and why this was at all possible

Thanks to all who helped.

Greetings

Andreas

All comments

Seashore

You should be able to see it in the audit log. If you just search for all audit logs on that node, do you see anything then?

bobmarley

You can try running this:

SELECT TOP 1000 TimeLoggedUtc, AuditEventMessage
FROM Orion.AuditingEvents
WHERE AuditEventMessage LIKE '%mute%' AND TimeLoggedUTC < ADDDAY(-30, GETDATE())
Order by TimeLoggedUTC DESC

adam.beedell

You can pull some data out, it's fiddley though and I hope the guy's query below works fine.

Calling it now though, you'll have an issue around either a) there only being one concurrent schedule. If someone/your script applies a different mute you'll have the original disappear - Including one later in the future than the original

OR b) you'll have a time zone issue.

Anything else I'd be real surprised

FormerMember

In the audit-events i can't see another user/my script to schedule a new mute. There is also no unmute message visible.

I reviewed my code and changed some points within the time zone.

Follwing thins i have change:

1. Version

start_time: "{{ ansible_date_time.date }}T{{ ansible_date_time.time }}{{ ansible_date_time.tz_offset }}"
end_time: "{{ '%Y-%m-%dT%H:%M:%S%z' | strftime( ( ansible_date_time.epoch | int ) + ( 86400 * 7 ) ) }}"

2. Version

start_time: "{{ ansible_date_time.date }}T{{ ansible_date_time.time }}"
end_time: "{{ '%Y-%m-%dT%H:%M:%S' | strftime( ( ansible_date_time.epoch | int ) + ( 86400 * 7 ) ) }}"

AND

1. Version

"suppressFrom": "{{ start_time }}",
"suppressUntil": "{{ end_time }}"

2.Version

"suppressFrom": "{{ start_time }}.000Z",
"suppressUntil": "{{ end_time }}.000Z"

I will update this thread if this changes are working correct.

FormerMember

So i do some more researching/testing and found out some more interest things.

To avoid an user-related problem i used my admin account for this. The mute worked correctly, but i was not able to unmute the node over the webinterface. Only by running our automation to unmute the node.

Also the changes with the timezone didn't work. The scheduled mute for a whole week disappered earlier. Our next step was to set the mute without an end. And yeah... it disapperaed without any user action (nothing to see in the Audit-Events)

Our code looks like:

# Mute Host in SolarWinds, if NodeID found
- name: Mute a host in SolarWinds
  uri:
    url: "https://solarwinds-url.com:17778/SolarWinds/InformationService/v3/Json/Invoke/Orion.AlertSuppression/SuppressAlerts"
    method: POST
    #Zertifikatsprüfung nach aktuellem Stand bei SolarWinds nicht möglich. Grund: Self-Signed Certificate von SolarWinds 
    validate_certs : no
    user: "{{ solar_user }}"
    password: "{{ solar_password }}"
    body: {
        "entityUris": '[ "swis://solarwinds-url.com/Orion/Orion.Nodes/NodeID={{ node_id }}" ]'
      }
    headers:
      Accept: application/json
      Content-Type: application/json
    status_code: 200
    body_format: json
  register: call
  ignore_errors: yes
  when:  not failed
- debug:
    var: call

Does anyone have any idea what that might be? I'm running out of ideas.

Regards

Andreas

FormerMember

Dind't see anything in the audit logs. No other user changed anything on the node...

Martin_Crothers

Is that YAML? I don't see a suppressFrom or suppressUntil time specified in there. Are you doing that in the code that executes the definition supplied? Aside from that, nothing in your code jumps out at me. I'd consider performing it via SWQL studio, to eliminate the code being a potential variable in the problem?

Any chance you have either Unmanage Scheduling Utility jobs (though I think those usually leave an audit log), or something being done directly at the database level?

FormerMember

Yeah, thats YAML.

We have the suppressFrom and suppressUntil in our first version of the code, till we realize that the mute isn't working. So we removed it to mute the node without an endtime. Looking into the audit events didn't show any errors.

Code looked like:

# Muten des Nodes in SolarWinds, falls NodeID vorhanden
- name: Mute a host in SolarWinds
  uri:
    url: "https://solarwinds-url.com:17778/SolarWinds/InformationService/v3/Json/Invoke/Orion.AlertSuppression/SuppressAlerts"
    method: POST
    #Zertifikatsprüfung nach aktuellem Stand bei SolarWinds nicht möglich. Grund: Self-Signed Certificate von SolarWinds 
    validate_certs : no
    user: "{{ solar_user }}"
    password: "{{ solar_password }}"
    body: {
        "entityUris": '[ "swis://solarwinds-url.com/Orion/Orion.Nodes/NodeID={{ node_id }}" ]',
        "suppressFrom": "{{ start_time }}",
        "suppressUntil": "{{ end_time }}"
      }
    headers:
      Accept: application/json
      Content-Type: application/json
    status_code: 200
    body_format: json
  register: call
  ignore_errors: yes
  when:  not failed
- debug:
    var: call

Start- and Endtime where defined above in variables:

start_time: "{{ ansible_date_time.date }}T{{ ansible_date_time.time }}{{ ansible_date_time.tz_offset }}"
end_time: "{{ '%Y-%m-%dT%H:%M:%S%z' | strftime( ( ansible_date_time.epoch | int ) + ( 86400 * 7 ) ) }}"

Using this old cold the audit events where also fine with the correct start and end time

This mute disappeared within the next 24 hours.

I will look into performing this automation with SWQL Studio, but i don't have any experience with it.

KMSigma.SWI

If you do one of these by hand (in the web interface) do the date/time audit logs appear the same way? (Dates as dd.MM.yyyy?)

FormerMember

So after a bit more testing and searching (and losing many nerves), i think i finally found the mistake.

@KMSigma.SWI gave me the hintch to look for a differenc between a mute setting over the dashboard an a mute creating over the automation.

On the webdashboard it looks the same, but in the orion database was a difference.

Comparing the two mutes i saw the "Entity Uri" wasn't the same. So in our automation we used a wrong swis "url".

After changing the swis to the one, which is used over the dashboard the mute works fine and i'm also able to deactivate the mute which was set via automation.

The only question thats remain is, why the mute was working for round about 1-2 hours and why this was at all possible

Thanks to all who helped.

Greetings

Andreas