Some of our favorite moments with LEM have been the stuff that people had no idea was happening or to look for that they uncovered for the first time now that all their data was consolidated. Did you have one of these a-ha moments when you started looking at log data? What did you find? A couple of of my faves: Hospital/clinic type environment had just installed LEM and was starting to monitor workstations/endpoint activity through the hospital/clinics (since their big issue is HIPAA). They also had their web proxy hooked up to LEM. Lo and behold, after a few minutes, someone started surfing inappropriate content - we're not talking borderline "oops, I got an ad", we're talking all out intentional stuff. Their policy dictated they had to confirm it, so they shadowed the session and sure enough... it was what they thought. So, they sent the guy a popup message using LEM's active response, and they literally watched him dismiss it. Sent a couple more about HR policies, watched him dismiss those, too. Sent one that said "IT is on their way"... suddenly he logged off and walked away. Semi-regulated environment (no public regs, only internal) had an uptime/SLA policy that required a designated service account could be accessed at any time - one of a few people with a key/card to access the datacenter could log on with these shared credentials to fix certain business-critical issues. This account was only supposed to be used in those scenarios. What did they find? Yeah, not that. It was being used to log in to workstations for admin privileges, install software on other servers... oops. If you missed it, the first SolarWinds Lab episode featured an example that came from some of our experiences as well - a customer's firewall kept going down, network performance was at rock bottom, logs going nuts... it was a virus. Took them a bit but they were able to use the logs to identify new infections and confirm that systems were cleaned. Doesn't have to be LEM specific, tell us what you've got

This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.

You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

LEM Thoughts of the Week: Tell Your Favorite "Found in the Logs" Story

FormerMember over 10 years ago

Some of our favorite moments with LEM have been the stuff that people had no idea was happening or to look for that they uncovered for the first time now that all their data was consolidated. Did you have one of these a-ha moments when you started looking at log data? What did you find?

A couple of of my faves:

Hospital/clinic type environment had just installed LEM and was starting to monitor workstations/endpoint activity through the hospital/clinics (since their big issue is HIPAA). They also had their web proxy hooked up to LEM. Lo and behold, after a few minutes, someone started surfing inappropriate content - we're not talking borderline "oops, I got an ad", we're talking all out intentional stuff. Their policy dictated they had to confirm it, so they shadowed the session and sure enough... it was what they thought. So, they sent the guy a popup message using LEM's active response, and they literally watched him dismiss it. Sent a couple more about HR policies, watched him dismiss those, too. Sent one that said "IT is on their way"... suddenly he logged off and walked away.
Semi-regulated environment (no public regs, only internal) had an uptime/SLA policy that required a designated service account could be accessed at any time - one of a few people with a key/card to access the datacenter could log on with these shared credentials to fix certain business-critical issues. This account was only supposed to be used in those scenarios. What did they find? Yeah, not that. It was being used to log in to workstations for admin privileges, install software on other servers... oops.

If you missed it, the first SolarWinds Lab episode featured an example that came from some of our experiences as well - a customer's firewall kept going down, network performance was at rock bottom, logs going nuts... it was a virus. Took them a bit but they were able to use the logs to identify new infections and confirm that systems were cleaned.

Doesn't have to be LEM specific, tell us what you've got

Top Replies

FormerMember over 10 years ago in reply to zackm +1

Awesome. We have had a lot of customers surprised by people trying to bypass their proxy servers/content filters and try to install crap like IM that was blocked by policy. When you get to detecting that…

0 LGarvin over 10 years ago

Sent one that said "IT is on their way"... suddenly he logged off and walked away.

<VBG>.. I like these stories.
Makes ya wonder if the "offender" felt that he (I'm assuming here.. safe assumption, eh?) was above rules and regulations, or never read the dialogs to begin with. Of course... "we're comin' to get you"... almost always has the desired effect. Maybe the first dialog should read: "We know you you are. Escape is futile."

It was being used to log in to workstations for admin privileges, install software on other servers... oops.

Not particularly surprised there. Restrictions without any form of monitoring or enforcement will almost always be violated for the convenience of those who can. Sad state of affairs, that it is though.
Cancel
Vote Up 0 Vote Down

Cancel
0 FormerMember over 10 years ago in reply to LGarvin

Makes ya wonder if the "offender" felt that he (I'm assuming here.. safe assumption, eh?) was above rules and regulations, or never read the dialogs to begin with.

From what I've heard from folks in the healthcare industry, there is a certain class of employee in that environment that has that exact opinion of themselves. He did actually have to click on the dialog to dismiss, but it very well could have read like "blah blah blah, IT blah blah, yeah, whatever." There has to be "that guy" in every environment, I suppose.
Cancel
Vote Up 0 Vote Down

Cancel
0 byrona over 10 years ago

I just found that my fellow admin's system was broadcasting NetBios traffic like crazy... to the tune of 1200 events per minute. It turns out the cause was multiple things not the least of which all the network discovery functions were turned on. He is the only one running Windows 8 on this desktop so this may be a Win8 thing. Either way, LEM gave us the visibility to see this when it would have otherwise gone unnoticed.
Cancel
Vote Up 0 Vote Down

Cancel
0 byrona over 10 years ago

Hrm, just thought of another good one too...
A few months back I just happened to be watching LEM, I guess because I am weird that way and I saw a ton of failed logins for one of our engineers Active Directory user accounts. It was clear that it was some sort of service account that started failing due to him changing his password. Now, in sysadmin 101 we all learn that you use service accounts for that stuff, not your own personal account so I had to go have a talk with him about misuse of his account and then also let him know that it was failing. He was a bit ashamed of himself for getting caught on such a bush-league mistake.
Cancel
Vote Up 0 Vote Down

Cancel
0 FormerMember over 10 years ago in reply to byrona

Aw, man. The talk of shame.
Cancel
Vote Up 0 Vote Down

Cancel
0 zackm over 10 years ago

Not LEM, but could easily have been used in that way:
A few jobs ago, I was a contractor for a large federal agency where we installed SolarWinds for one of their departments. They had a pretty standard setup that included, among other things, UDT and NTA.
Being a federal agency, there were certain rules for usage and no expectation of privacy for the users. We were also plagued with a small pipe for traffic... Lo and behold, we quickly realized that we could pinpoint the youtubers and netflixers and etc etc. Those weekly reports at the all-managers meetings were fun to create.
Cancel
Vote Up 0 Vote Down

Cancel
0 FormerMember over 10 years ago in reply to zackm

Awesome. We have had a lot of customers surprised by people trying to bypass their proxy servers/content filters and try to install crap like IM that was blocked by policy.
When you get to detecting that stuff even REMOTELY reliably I swear there's a set of users who think you have eyes in the back of your head and start confessing to stuff before they even do anything remotely wrong, and a set who think it's a personal challenge to them to try to thwart your next level of defenses/detection
Cancel
Vote Up +1 Vote Down

Cancel