LEM service stoppable

I can think of a couple ways to make this work.

First, I built a crazy rule:

This rule looks for a user running the Microsoft Management Console followed by the Agent going off-line.  It's not 100% that they ran the Services snap-in or that they used it to stop the Agent, but it seems like the odds of the MMC getting started and then the agent going away would correlate pretty well.

Since the ProcessStart event has a SourceAccount, I can make an event or an e-mail that puts it all together:

Alternatively, you could make a rule that looks for anyone being added to the Local Administrators account:

And have an alert fire off that, since stopping the LEM Agent is really one of the least damaging things a person with Local Admin access can do.

That crazy rule is pretty sweet, I like how you made a work around. I was also going to look into using the Auditing policy in Windows and see if that could give me a solution. I am surprised that solarwinds doesn't have some kind of direct solution for this. If the agent's service is killed it renders auditing useless for that node, which in my eyes, seems like a pretty big downside to the design of the agent. I feel like a few lines of code in the agent's structure to write to the Windows event log with the logged in user information would at least give you a little bit of a crumb trail. 

I think the problem with the Event Log is that the "Agent Offline" event isn't generated by the Agent.  It's generated by the LEM manager, because the manager is suddenly like, "Oh, hey, there was a node there and now there's not.  I better note that."  The manager doesn't know why the node went away: did the network go down?  Is the computer off?  Did someone stop the service?

That's why my crazy rule is looking for two events: mmc starting and the Agent going off-line.  The crazy rule is based on the Agent Off-line with Timeout template.  Windows probably does write some of those events to the log (we do get ProcessStart and ProcessStop events if you're auditing them) but the Agent isn't on-line to send it's own termination notice to the Manager with that information!

Haha yea, I really like what you did with that rule I might have to duplicate that and give it a test. I feel it would be nice if LEM had a rule/action that triggered when the agent enters a "offline" state and simply just attempted at a restart of the agent's service (kind of like the MSSQL service restart rule). This would obviously require some extra functionality as the agent's service is the "connector" to LEM. I am actually quite shocked that SolarWinds hasn't found a way to alleviate this risk.

But where does that madness end?

"We wanted to know when the agent stopped and restart it.  So we made the Agent Supervisor.  Then people wanted to know if the Supervisor had stopped, so we made the Agent Supervisor Supervisor.  Then people wanted to know if..."

Pretty soon, the Agent and the Agent Supervisor Supervisor Supervisor Supervisor Supervisor Supervisor Supervisor Supervisor Supervisor Supervisor Supervisor Supervisor Supervisor Supervisor Supervisor Supervisor Supervisors are using 128 GB of memory, and Windows has crashed because it can't do anything but run this spiral of watch applications.  Better still, make sure that users don't have rights to escalate unless they're really, really dedicated...and then log that and smack them for it.