Showing results for 
Search instead for 
Did you mean: 
Create Post

Breaking Bad? How Security Guys Will Pwn You.

Level 9

I attended the Austin ISSA-sponsored Advanced Splunk Training session on March 6.  As always, the ISSA chapter delivered meaty technical training, and it was free!  The event was co-sponsored by BSides and Splunk.


While all kinds of interesting Splunk technical info was presented, for me, the most interesting part was hearing from Michael Gough and some other security practitioners at the event about what people really monitor.  As a technology provider, we are not always privy to what people are really doing with our tools, so it was an eye-opener for me.

Splunk training.JPG

Here are some of the things security guys monitor.  Of course they monitor other stuff too, but this is what we can share in mixed company

  • Administrators / Root logins, successful or failed.  "Power corrupts, total power corrupts totally” - even IT administrators.
  • Login attempts to disabled accounts.  Makes sense - there's usually a pretty good reason they're disabled.
  • Successful logins for certain accounts, such as those with elevated privileges, or accounts given to partner personnel
  • https accesses, especially to weirdly long URLs, which can be SQL injections
  • FTP from servers and workstations
  • Group membership changes and elevation of privilege
  • Database alerts
  • Suspicious files being executed
  • VPN logins
  • Outlook Web App (OWA)  and Remote Desktop Protocol (RDP) logins – looking for suspicious remote access
  • Servers downloading .exes from the internet.  They look for admins surfing for open source tools to keep an eye that malware hasn’t been downloaded
  • Share drive accesses at workstations and at servers; access to particular, sensitive shares.  They watch for shares being seen and crawled inappropriately.
  • Net.exe use to map and unmap network drives in Windows
  • Cscript.exe use. Cscript.exe lets you run scripts via command line and can be used in exploits
  • Services being installed from servers; noisy workstations

And if you are a Security Guy, please check out our SIEM, SolarWinds Log & Event Manager.  It's an understated, affordable, full-function SIEM that can help you pwn the bad guys.


Is there a way we can hear about other thing monitored that didn't make the 'mixed-company' list? I'm genuinely curious and would like to know more.

Product Manager
Product Manager

I didn't attend the session with but there's definitely digging into lower level activity (whether it's file, process, or network) that can indicate there's issues - either with people on the inside, or attack activity from the outside. Protecting data/access to data, preventing outages/downtime due to security issues, and finding naughty folks are at the core.

My list (partially borrowed from the presentations we did a while back) includes:

  • User auth activity - ways for people to authenticate to the network, because after that they have passed into one of your first concentric circles of trust...
    • New users being added - track both local users and domain, if you're using central administration.
    • New machines being added, in the centralized model. These machines will be able to browse other activity.
    • Users added/removed from groups - especially admins, extended privilege, visibility of more secure data.
  • Device activity - network-based ways for people to get on to the internal, DMZ, or other networks that also move them toward your creamy center...
    • Successful and failed logons to network devices - device management attempts
    • Actual changes on devices (commands being ran, changes to ACLs, policy save/restore) - could expose more access
    • With Wireless APs, DHCP servers, etc, you can pick up new devices appearing on the network
  • Suspicious activity hiding in normal activity - stuff trying to fly under the radar
    • Usage of accounts with domain admin or privileged access
    • Remote access and physical logons directly to servers with sensitive data
    • Interactive logons to service accounts, or using regular accounts (especially admins) to run services/tasks
    • Repeated failed logons to one or more accounts (brute force, virus, or stupid device)
    • Logons coming from external IP addresses
    • Stuff hanging out in web proxy data - people downloading executable files, accessing suspicious sites, acting as promiscuous internet users, servers accessing unexpected URLs,...
  • System-level OS activity
    • Reboots/shutdowns, device/firmware updates, software installation/removal, log full/cleared, logging policy changes
    • Processes being launched and files being accessed (volume could be an issue with these, so you'll want to look for anomalies - like the cscript example above on a bank teller machine, that sort of thing)
  • Network Stuff
    • There's a lot of time that could be spent digging in network traffic without even looking at the packet level - and with just device logs, not even NetFlow, though flow can also help in some areas.
    • Low & slow attacks - same source repeatedly accessing over a long period of time
    • Early warning of DoS attacks, worms, and the like based on increased traffic volume over normal (or basic numbers)
    • Access to unusual ports or known malware ports (dshield has a lookup tool with info about unusual ports, and tracks activity that might be rising/falling to different ports, or be newly suspicious)
    • Outgoing access from servers on unusual ports (you should have a pretty good idea of expected behavior on core devices, especially to the internet)

We've heard stories of people catching IT admins accessing CEO-level files, hospital workers surfing work-inappropriate content (no wonder they kept getting viruses) despite repeated warnings, IT contractors that brought in malware unknowingly and had to have proof that it was them (and these are people who are supposed to help?), viruses that brought networks to their knees and other similar really crappy triage situations, admins that added that "one last ACL" that effectively disabled their deny policies, open ports to the internet for RDP/pcAnywhere with default passwords, shared service accounts being used for workstation logon and unlimited unmonitored file activity, remote access to accounts that were supposed to be disabled ("oh, but we found out we still need to access their email so we re-enabled them")... and that's just from memory.

Thank you!

Good memory...

About the Author
After working all aspects of IT from lab/helpdesk support to complete IT responsibility over the span of 10 years, Nicole turned Product Manager to help bring accessible IT management software to the masses. She joined SolarWinds with the acquisition of Log & Event Manager in 2011 to find it felt just like home.