Slow "Linux/UNIX Script Monitor" Save Times

I am curious if anyone else is/has experienced the issue I am experiencing in SAM. In short, I have to click the "SUBMIT" button multiple times (up to 30 or 40) before changes are saved and I am returned to the Application Monitor Template list. If this sounds familiar or you are interested, please read on. Could really use some help with this.

In my environment I a script for collecting various information about a filesystem and then cause events based on information returned and threshold settings. I use the "Linux/UNIX Script Monitor" type Component Monitors to run these tests and reuse the same script with different, filesystem specific, script arguments for each particular filesystem. I use Application Monitors to categorize the filesystems based on things like OS related, related to a certain application we market or related to another vendors application. In some cases, a single category may have as many as 50 different filesystems and therefore 50 different Component Monitors in a single Application Monitor.

Over a year ago I started to see, as these Application Monitors were created and expanded, that "SUBMIT" processing times after changes were taking longer and longer. I opened a case with SolarWinds (636115), they investigated and are still working on a fix based on what they have learned from my system.

To explain more about what I am experiencing, the most common change I make to these monitors is related to the Perl monitor script I use to collect the information. When that single script changes, which it did again recently, I have to replace the script in hundreds of Component Monitors within our environment. [This might sound familiar to some of you as I also have a Feature Request titled ‌ (please vote at that link.)]

So back to the topic at hand... As I make this change to Application Monitors I have come to only change 10 or so Component Monitors between fully successful "SUBMIT" presses. However I many have to press "SUBMIT" 1 or 2 or up to 30 or 40 times before the changes save for all changed Component Monitors and I am returned to the Template list. Analysis I have done shows that sometimes one or two of the changed Component Monitors will update with a single "SUBMIT" press while other times no changes will save. Using the "SAVE AND CONTINUE WORKING" button results in about the same result of course always remaining on the edit page. Eventually, after continuously pressing any combination of these buttons, all the changes will save and I will be returned to the Template list but considering the number of Component Monitors that require the update and the number of "SUBMIT" button presses required to work through them all, this becomes very time consuming and painful.

Something I have not mentioned is that it seems the more Component Monitors there are in a single Application Monitor, the longer it takes for each "SUBMIT" press to complete its processing. I also figured out this week that there may be some correlation between the number of nodes the Application Monitor is assigned to affecting the length of the "SUBMIT" processing times and possibly number of "SUBMIT" presses. For Application Monitors not assigned to nodes, even with one having about 50 Component Monitors, the "SUBMIT" seems to succeed on the first press.

So is there anyone else out there that has dealt with this and if so, what have you figured out or what fixed it in your environment? I cannot continue to spend this much time on simple script updates and am hoping someone out there has the answer. Thanks in advance.

Mike

Find more posts tagged with

unix_perl_script_file

sam

solarwinds orion sam

unix_script

unixlinux_script_monitor

Accepted answers

All comments

aLTeReGo

Since it sounds like you're using the same script in different components (likely with different command line switches specified for each component) have you considered copying the script to the server being monitored and calling it from the command line arguments instead of copying the script body into each component?

Mike_Lomax

Yes that was something I thought about early on but decided it was something I really would rather avoid. We are a software vendor who monitors some of our customers environments for them. So having the script be server based would mean we would need to log out to each customer site, log onto each server at that customer site and replace the script there. Then repeat for other customers. Currently the overall node count in the customer environments is around 160 but that has fluctuated up to more than 300 so that could also involve a lot of work.

To me Orion is there to bring monitoring of environments to a centralized location. Therefore I would prefer to manage and maintain all of the scripts within the Orion environment if at all possible.

Let me add however that I am also currently updating Display Names for all sub-components which also results in the slow "Submit" times as well. I even hit this slowness when I am "Submit"ting a single threshold change in these larger Application Monitors. My guess is that if the script was moved to the node, slowness for these other changes would go away, since I don't seem to have the slowness on smaller Application Monitors, however it also seems like it would be more beneficial to resolve the underlying problem so that we can use Orion the way it was intended.

I am guessing that the current SAM design has the script for a single Component Monitor saved in multiple places like once for the template and once each for each assigned node. Unless the script is for some reason different for a particular assigned node, why not just point back to the template copy, or if and once developed, to a library copy of the script.