How to find who is editing custom properties?

Question

I've a PowerShell script that correctly auto-sets custom properties, but someone else somewhere is running their own script that is overwriting the custom properties and I don't know from where they are getting changed. I've gone to the Message Center looking at all audit events, but all it shows is a Message about 'User admin changed custom property.....' but not from where it was changed.

I've looked in Scheduled Task on all of our pollers and not found anything (other than my script on one server) and I've looked in actions to see if any alert was created with an action of 'Change Custom Property' and I've not found anything. I've also looked in SAM's component monitor looking for anything running a PoweShell script, but still unable to find anything.

I was hoping to find a log file or something that would provide more log info showing the IP of the server from which the custom properties are getting changed, but I've no clue where to even try looking.

Any suggestions/ideas would be appreciated.

I

tigger2 · Answer

not knowing how many scripts you have running all the time.... your environment, etc ... and just thinking out loud...

- Obvious "solution" create more accounts with admin privs, then hand them out to specific people/machines.  Eventually lock out the admin account and see who complains 
.  But probably looks good on paper only and not going to happen in the real world...doesn't really track where actual SWIS traffic comes from though so if someone managed to get hold of another account (opening scripts they don't own) its not going to work anyway.

- Since "their" script is running all the time...Could you, during say a normal maintenance window, add some "extra" downtime and shut down the Orion system (or just parts of it, like the agents, etc.) then do the wireshark trace (or at least a netstat) on whatever server no connections should be coming in on and see where the regularly occurring API connections are coming from?

- If support isn't much help, or if you haven't looked into this, there is an "Orion Log Adjuster" app included in the Orion installation that will let you put logs (specifically the SWIS related ones) into various debugging modes.  It doesn't tell you what log it's tuning (you may be able to figure it out), but maybe something will pop out linking things to incoming IP addresses to SWIS requests, esp, if you can perform a temporary maint. "shutdown" and watch the logs.  I found a post (Re: SWIS REST/JSON API )  that indicate it may be in "C:\ProgramData\SolarWinds\InformationService\v3.0\Orion.InformationService.log."

- You might be able to turn on/up IIS logging for the SolarWinds website and catch something relating to the query/URL their SWIS script is using and the source IP address.  If you know they're changing custom node properties then the URL may be very specific and how to catch the line with the IP address (like POSTS to  .../SolarWinds/InformationService/v3/... ) I'm not sure what the forms the actual URL can be, just looking at the posts about the SWIS SDK/API.  There are tools on the net to parse the logs that might help.  Maybe the incoming browser headers ("Mozilla/5.0+(Windows+NT+......") of SWIS identified POSTS will show something unique that will indicate it's a script vs normal traffic from an agent/server.

- Make a "needed change" to change the admin password because of this specific issue  .  Tell everyone in advance, follow the official change procedures, etc.  Make note of who you have to give the new password out to.  Change the password once everyone is prepared.  This may allow you determine all the people who are using/maintaining scripts who know they're interacting with Orion and you can talk to them if they are consciously doing the changes. Maybe the offending script was owned by someone who no longer works at your company, or has moved to another group so when the password changes their script may not by updated if it's been forgotten. Maybe someone will remember it and speak up and go "oh, sorry" and turn that script off for you.

- Maybe a failing SWIS update will generate an error in some of the areas/logs above with more information that you normally get from normal traffic?  If so, find a node that the offending script always updates, and then change the admin account to allow it to see everything *but* that node (I'm thinking via the "account limitations" area on the account page.  If they're not running a query to get nodes first and then updating the nodes from the query, it might generate an error as they try to update a "nonexisting" node the admin account cannot see.

This problem is actually very interesting to me so if you find a solution please post it here 
.   It really make me think about how they need to make this type of information loggable/traceable from the SWIS API side as you should be able to track down (or disable) SWIS access from specific IP's or *something* if someone is abusing their SWIS access, even unintentionally.

tait.cyrus · Answer

I did submit a support case and the result is there is no way to answer my question. There is not enough information in any log file or DB audit table that contains the information on who (as in server/IP) changes a custom property. The API user used to run SWIS is put into the audit logs, but since all of our scripts use the same user/acct, there is no way to determine where the script is being run from.

tigger2 · Answer

hmmm... There are *lots* of logs in Orion and ways of adjusting what level of information gets logged in some (all?) of the logs, but it's not so easy to know what specific log is going to be written to, and even if making the logging more verbose (for example) will even show you what you want to know.

If no one chimes in here in a day or so I'd say this is probably something that needs to go through support.    Good luck!