5 Replies Latest reply on Sep 19, 2018 11:33 AM by tigger2

    How to find who is editing custom properties?

    tait.cyrus

      I've a PowerShell script that correctly auto-sets custom properties, but someone else somewhere is running their own script that is overwriting the custom properties and I don't know from where they are getting changed. I've gone to the Message Center looking at all audit events, but all it shows is a Message about 'User admin changed custom property.....' but not from where it was changed.

       

      I've looked in Scheduled Task on all of our pollers and not found anything (other than my script on one server) and I've looked in actions to see if any alert was created with an action of 'Change Custom Property' and I've not found anything. I've also looked in SAM's component monitor looking for anything running a PoweShell script, but still unable to find anything.

       

      I was hoping to find a log file or something that would provide more log info showing the IP of the server from which the custom properties are getting changed, but I've no clue where to even try looking.

       

      Any suggestions/ideas would be appreciated.

       

        • Re: How to find who is editing custom properties?
          tigger2

          I'm using NPM version 12.1.  (Maybe an older version doesn't  work this way)

           

          It's possible that this setting may need to be set: Admin -> Settings -> Web Console Settings -> check "ENABLE AUDIT TRAILS"?  I don't know if this turns on extended auditing or if it's literally a "all auditing ON/OFF" switch. I have this set in my environment though.

           

          When I go into Message Center and only select "show last 7 days" and "show audit events" and from the drop down "All action types" (I'm not filtering by user or type, but you might need to) I see entries that indicate when users login and their incoming IP address with a message like:  "User DOMAIN\someuser logged in from XXX.XXX.XXX.XXX."  From what I can see, this catches when people log in via the web UI OR when they use scripts using the SWIS API.  It also appears to catch users logging in with individual Orion managed accounts (like the default "admin") , and if they are in a "group" account.   If someone is using a script, it would make sense that they would login and start making changes almost immediately so you could probably correlate a login to the changes occurring right after a login.

           

          If you are seeing "user admin changed property.." audit events in the UI this should rule out a script/person connecting directly to the database (not using published API's) and making changes.  These messages you see would make me believe they are going through the published API's like your script is doing.  As a test, check that you can see your own script logging in (maybe create a special local Orion user just for this if you are also using "admin"?)

           

          Another option: Someone is manually logging into the web UI as "admin" and making mass updates via the "manage nodes" page -> (selecting some nodes) -> "Edit Properties". This should still leave an audit event of them logging in as "admin" though, but possibly causing  the "login" and "property change" audit events to be far apart, especially if they login and leave the web UI up for a long time, then make the changes later.

            • Re: How to find who is editing custom properties?
              tait.cyrus

              Thank you. Some good suggestions and in my case 'Enable audit trails' was already enabled so I won't be able to get any additional info. In my case, it appears to be an automated script running every 5 minutes that is changing the custom property. Clearly the script has an 'admin' user/pw and thus why 'admin' is showing up as performing the edits in the message center, but since it is via a script, there is no user logging in to track.

               

              I was hoping for a .log file I could look at that showed the srchost. Or maybe even a wireshark search, but it appears that the API is using port 17778 which all agents also use when connecting to the main server (which I did try hoping to see something unusual but there was too much traffic to make any determination).

               

                • Re: How to find who is editing custom properties?
                  tigger2

                  hmmm... There are *lots* of logs in Orion and ways of adjusting what level of information gets logged in some (all?) of the logs, but it's not so easy to know what specific log is going to be written to, and even if making the logging more verbose (for example) will even show you what you want to know.

                   

                  If no one chimes in here in a day or so I'd say this is probably something that needs to go through support.    Good luck!

              • Re: How to find who is editing custom properties?
                tait.cyrus

                I did submit a support case and the result is there is no way to answer my question. There is not enough information in any log file or DB audit table that contains the information on who (as in server/IP) changes a custom property. The API user used to run SWIS is put into the audit logs, but since all of our scripts use the same user/acct, there is no way to determine where the script is being run from.

                 

                  • Re: How to find who is editing custom properties?
                    tigger2

                    not knowing how many scripts you have running all the time.... your environment, etc ... and just thinking out loud...

                     

                    - Obvious "solution" create more accounts with admin privs, then hand them out to specific people/machines.  Eventually lock out the admin account and see who complains .  But probably looks good on paper only and not going to happen in the real world...doesn't really track where actual SWIS traffic comes from though so if someone managed to get hold of another account (opening scripts they don't own) its not going to work anyway.

                     

                    - Since "their" script is running all the time...Could you, during say a normal maintenance window, add some "extra" downtime and shut down the Orion system (or just parts of it, like the agents, etc.) then do the wireshark trace (or at least a netstat) on whatever server no connections should be coming in on and see where the regularly occurring API connections are coming from?

                     

                    - If support isn't much help, or if you haven't looked into this, there is an "Orion Log Adjuster" app included in the Orion installation that will let you put logs (specifically the SWIS related ones) into various debugging modes.  It doesn't tell you what log it's tuning (you may be able to figure it out), but maybe something will pop out linking things to incoming IP addresses to SWIS requests, esp, if you can perform a temporary maint. "shutdown" and watch the logs.  I found a post (Re: SWIS REST/JSON API )  that indicate it may be in "C:\ProgramData\SolarWinds\InformationService\v3.0\Orion.InformationService.log."

                     

                    - You might be able to turn on/up IIS logging for the SolarWinds website and catch something relating to the query/URL their SWIS script is using and the source IP address.  If you know they're changing custom node properties then the URL may be very specific and how to catch the line with the IP address (like POSTS to .../SolarWinds/InformationService/v3/... ) I'm not sure what the forms the actual URL can be, just looking at the posts about the SWIS SDK/API.  There are tools on the net to parse the logs that might help.  Maybe the incoming browser headers ("Mozilla/5.0+(Windows+NT+......") of SWIS identified POSTS will show something unique that will indicate it's a script vs normal traffic from an agent/server.

                     

                    - Make a "needed change" to change the admin password because of this specific issue .  Tell everyone in advance, follow the official change procedures, etc.  Make note of who you have to give the new password out to.  Change the password once everyone is prepared.  This may allow you determine all the people who are using/maintaining scripts who know they're interacting with Orion and you can talk to them if they are consciously doing the changes. Maybe the offending script was owned by someone who no longer works at your company, or has moved to another group so when the password changes their script may not by updated if it's been forgotten. Maybe someone will remember it and speak up and go "oh, sorry" and turn that script off for you.

                     

                    - Maybe a failing SWIS update will generate an error in some of the areas/logs above with more information that you normally get from normal traffic?  If so, find a node that the offending script always updates, and then change the admin account to allow it to see everything *but* that node (I'm thinking via the "account limitations" area on the account page.  If they're not running a query to get nodes first and then updating the nodes from the query, it might generate an error as they try to update a "nonexisting" node the admin account cannot see.

                     

                    This problem is actually very interesting to me so if you find a solution please post it here .   It really make me think about how they need to make this type of information loggable/traceable from the SWIS API side as you should be able to track down (or disable) SWIS access from specific IP's or *something* if someone is abusing their SWIS access, even unintentionally.