cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

I Beat Them to Firing Me: A Story of Corporate Sabotage

Product Manager
Product Manager

thwack.jpg

I BEAT THEM TO FIRING ME! (Part Two) Fight Back

Why network configuration, change and compliance management (NCCCM) is a must

Inspired by former Citibank employee sentencing

(Part Two)

We've all heard horror stories about the disgruntled employee who pillages the office supply closet and leaves the building waving an obscene gesture, security badge skittering across the parking lot in his wake. Rage-quit is a thing, folks, and it's perfectly reasonable to be afraid that someone with high-level access, someone who could make changes to a network, might do so if they get mad enough. This happens more often than anyone would like to think about, and it's something that needs to be addressed in every organization. I felt like we should talk about this and discuss ways to help control and slow the damage of said employees and their bad will. Bottom line: we need to be aware of these situations and have a plan for recovery when things like this happen.

The gist of the story is simple: there was an employee who wiped out critical network configurations to about 90% of his former company's infrastructure.  Monday he was sentenced on charges of criminal vandalism. So, I realize the article above is technically in the past, but it brings up a great starter conversation about how IT organizations can stop criminal vandalism by actually using NCCCM products to protect ourselves and others from any type of disastrous events. Sometimes you need that brief pause or slight inconvenience to help you think straight and not go over the edge. This post can also help keep your butt out of, well, jail .

Today, we are going to talk about some of the risks of not having NCCCM software:

  1. Real-time change notification not enabled.
    • There is no tracking, idea, or reference to when changes are being made via maintenance plans, change requests, or malicious intent.
      • Being able to see network changes and know the timing helps you to be proactive, and gives you immediate remediation action for your network.
    • Who's on first base, and did someone slide in to home base?
      • When you have more than a couple of network engineers, documentation can be lacking and, well, you're busy, right? Being able to track when changes happen and who made them allows you to find and discover who, when, and what was changed, even when it's a week later.
      • Being able to compare the change that was made to existing is key to correlating issues after a change was made. All of a sudden, traffic is not flowing, or it's restricted, and you find out it was an error in the config change.
    • Someone is on your network changing your critical devices and wiping them clean.
      • Receive alerts so you don't find this type of information out when it's too late. Be able to log in, and after receiving the alert, restore to previous config.
  2. Approval process not in use.
    • No change auditing.
      • Being able to make changes without approval or a process sets you up for human error or worse: attacks.
      • Implementing an approval process allows you to have an auditing system that shows that more than one person approved a change.
      • Use this with real-time change notification to see if anyone outside your team is making changes. Either allow them into your NCCCM, or delete or lock out their login info to the devices.
    • No one can verify that you are making the change, or even what that change was.
      • When you have a larger team, you delegate changes or areas of functionality. Having an approval process verifies that the correct changes are being made. That gives you an extra set of eyes on the changes that are being made, which adds another level of detection to human error.
    • One person has complete access to your devices at a control level.
      • When you give people straight access to network devices there is a single point of failure. Taking an extra step creates a safe zone of recognition, training, and the ability to track changes and implementations on your network.
  3. Advanced change alert not enabled.
    • Not having an escalation alert set up can leave you with no configurations on your devices when you come into work the next day.
      • Set up escalation alerts based on more than one action.
        • Create a mass change alert if X amount of syslog changes happen within five minutes: Alert Manager NOW.
        • Mute these when implementing maintenance plans. more info by adatole
  4. Backups you are saving to your desktop or network drive (when you remember).
    • If a crisis happens, the great news is that network devices just need to be told what to do. But if you are like me and don't remember every line of code for hundreds of devices, then you better implement a backup system NOW.
      • If you have backups being stored, recovery is a click away with an NCCCM.
      • Compare starting to running to make sure a reboot won't cancel your changes.
      • Verify you have backups in secure locations so downtime is minimized and quickly averted.
        • I generally implement server side and network share drive backups. Make your server accessible with security verification lockdown in case someone tries to delete the backups (this happens because they don't want you to recover).
  5. Recovery procedures not in place.
    • Can your team recover from an emergency without you being on site?
      • Have a plan and practice with your team. You have to have a plan to be able to recover from maintenance plans gone wrong all the way to disaster recovery.  This takes practice, and should be something the whole team discusses so that you are better engaged. It helps to have an open mind to see how others may offer solutions to each potential problem suggested.
    • Setup an automatic password change template to be easily used in case of a potential issue within or outside your organization.
    • Use your NCCCM to monitor your configurations for potential issues or open back doors within your network.
      • Sometimes people will start allowing access within your network watching your configurations with a compliance reporting service allows you to detect and remediate quickly to stop these types of security breaches in their tracks.

If your curious on setup check this out:More info Security and SolarWinds NCM

Stay tuned for part two, I'll showcase how each one of these can be used in response to security!

Now that is a few things you should be able to use within any NCCCM software package.  This should also be something you revisit consistently to reevaluate and assess your situation and how to better protect yourself.

Let's dive into the mindset and standard methodologies around the security aspect:

This isn't just for technology this is in general things to be aware of and to implement on your own.  The ability to look at these with a non-judging eye and see them as just ways to hold off malicious attacks or ill will.

  1. There needs to be a clear exit strategy for anyone that is going to be fired or removed from a position with potential harm.
    • But he is such a nice guy?  Nice guys can turn bad.
    • When this information is being circulated you need to do what's best for your career as well as the company you work for and go on the defense.
      • Bring in specialized help organizations that can come in assess and prevent issues before they are terminated or moved
      • Make sure you verify all traffic and location they were involved in
        • Any passwords etc that were globally known NEEDS CHANGED NOW not LATER
        • Check all management software and pull rights to view only in the remainder days then delete access immediately after termination
        • Verify all company technology is accounted for (Accounting and inventory within your NCCCM is vital to maintain diligence on awareness of property and access to your network)
  2. Monitoring of team
    • Some may not be happy with a decision to terminate an employee and feel betrayed
    • Monitor their access and increase awareness to their actions
      • If you see them logging in to more routers and switches than ever before might setup a meeting...
      • See them going outside of their side and digging into things they should not, meeting time
      • Awareness is key and an approval process and change detection is key to preventing damage
  3. Security policies
    • You're only as good as the policy in place
      • Dig into your policies and make sure they are current and relevant
      • If you seriously have things like "If they call from desk phone reset password over the phone" type of security measures please REVISIT these.
        • Re-read that last statement
    • Make sure your team is signing acknowledgement of what they can and cannot do
      • Easier to prosecute when they have signed and agreed
    • Verify your security policies to your network devices
      • NCCCM compliance reporting setup for your needs is a great way to stay ahead of these items
      • You can find back doors on your network that people have setup to go around security policies this way. 

     I cannot obviously solve every issue, but at least help to point you into some good directions and processes.  If any of you want to jump in and add to this, please do I'm always interested in other people's methods of security.  The main point is to be aware of these situations, have a plan and recover when things like this happen.

Thank you,

~Dez~

Follow me on Twitter:

@Dez_Sayz

27 Comments
MVP
MVP

Dez​, that is more than a nuggets worth of great points to not only ponder but look to see what you have in place that is comparable and refine to have a more solid plan in place.

Thank you for sharing this !

MVP
MVP

I'm also a fan of having accounting turned on for commands issued to a device via TACACS.    While RTCD is nice and will tell you what configuration changes someone did, it doesn't tell you what commands they issued on the device which accounting will.  That way you can potentially go back and see what they did, or check for suspicious behavior, or maybe simply figure out that what they did was "normal" and OK.   Takes very little space overall and works well...

Product Manager
Product Manager

This is where a a NCCCM software will come in handy by allowing you to compare config's also to verify what was changed.  being able to also trigger events via Syslog or Traps is a way to setup additional alerting.

Level 20

LOLOL that picture is pretty funny too!

Level 12

Nice article, Dez​. We are a small(ish) shop where everyone is a really nice guy, so we have some lapses. This is definitely an article I will pass along.

TACACS has revealed some excellent configuration typos issued by me and my peers, and also definitively answered the question: "Who made that change?"

It's also shown where more devices or interfaces than expected were adjusted, through human error.  TACACS was a complex project to implement uniformly across all our equipment, but now that it's in place, life feels a lot less unmanageable.

Level 14

Dez​ outstanding points. Too often we concentrate on the external threat and ignore potential internal ones.

ScottRich​ we are a small shop as well.. but I am passing along as I type...

Great Poitns Dez

Our biggest problem is we are a large shop and so many of these responsibilities are silo'ed and and the silo's go all the way to the top before there is any crossover.

Level 8

Very good article Dez​, excellent points, that I will pass along.

Level 9

What is in those slugs?  Is it birdshot?

MVP
MVP

Nuggets

Level 15

Nice article. congratulations.

Product Manager
Product Manager

This is separated between silos and then managed by the above management or engineer.  If there is not a set overall policy these can trickle up to and down from then that is definitely a place to start implementing security protocols within the teams.

Thank you,

Dez

Level 20

Pumpkin balls!  Chicks don't carry birdshot they carry slugs!  Better stopping power.

MVP
MVP

but a nugget is a chicken slug , well a slug of chicken

Level 20

That would be like shooting pieces of his own kind out his shotgun o.O!

MVP
MVP

either way it is fowl...

Level 14

Dez​, always have to keep the insider threat at the forefront of our minds.  Great write up.

pastedImage_0.png

MVP
MVP

Very good rschroeder​ !

Level 13

TACACS is also helpful for answering exactly what happened, in what sequence, by who.

I've had situations where we know something was changed as evidenced by symptoms (i.e. issues) and the config differences.  Comparing the before and after configs is good for seeing end-states but it doesn't really tell you what was being done to your network between those points.

There's a reason malicious attackers try to disable or erase logs.

Product Manager
Product Manager

This is a must to be alerting on these events as well.  As you are able to see when people are shutting things down and making changes to your network.  This allows you to quickly identify issues and begin your recover plan.

~Dez~

MVP
MVP

Great article. Lots to think about now.

Level 21

I think it's definitely important to have all of those technologies in place: change detection, backups, change approval process, etc. even without the disgruntled employee in the picture.

My takeaway from this article is while technology can't fix stupid, it can certainly help with early detection and recovery from it. 

Level 17

Very Nice Dez!

MVP
MVP

It always amazes me the amount of mistrust of the IT team - between micro managers, time clocks and other oversights some businesses treat their employees as a threat (yes, I know there is that side with security and all, I'm speaking primarily of the IT staff that manages everything) And yet, they will balk as the cost of automating things like change management, log management and other things that keep the honest guys honest.

Most of us are professionals that take our jobs seriously and would rather take a poke in the eye with a fork than to do something illegal or immoral when it comes to our employment. But these tools help keep us honest, help prevent finger pointing, and adds accountability without the effect of a big stick.

On the other end of the spectrum is what Dez illustrates when the employee is pushed too far (whatever that means and regardless of the cause) and goes off and does things that they normally wouldn't. Then the tools can be bacon savers.

But then these tools are like many other forms of insurance - you tend to purchase it right after an event.

Level 13

time to beef up my security again...

About the Author
I started in networking and security around 2002 by taking Cisco Certified Network Associate and Security+ courses from Central Vo-tech. This is where I fell in love with technology in general. From there I venture out to internships and started using the Engineers Toolset from SolarWinds which made me wonder about software. The company I was with purchased Cirrus which is now Network Configuration Manager (NCM) and I was officially hooked. I searched out for SolarWinds and well you guessed it I started working for them and believe it or not in sales. That was the only position open but I knew I wanted to be here. So I quickly worked my way in to the support side and became the first Sales Engineer and then the first Applications Engineer. Since I am a very curious person I have since in my 9 years of being at SolarWinds decided to pursue more education. Security is always a fascination to me so I started taking classes on INFOSEC Assessment Methodology (IAM) and INFOSEC Evaluation Methodology (IEM) of the NSA. Then I went and took the CIW Masters for web development and ventured to databases. MCITP SQL Server and Development certifications that led me to a database development degree in college. I’m pretty much a jack of all trades and LOVE IT! This all applied to my work with SolarWinds as I wanted to be able to help customers solve their issues or needs. So knowing more information allowed me to do this successfully. I also dabbled in Cisco UCS management and currently taking classes to venture toward a CCIE (crossing fingers). NCM is a product that I have worked with since its beginning. I even had the opportunity to fly to the NSA to create templates for some of their devices. I used to be the sole MIB database controller so I’m definitely your huckleberry on MIBs and OIDs. As an Applications Engineer I focused on Network Performance Monitor, Network Configuration Manager, Web Performance Monitor, Enterprise Operations Console, Patch Manager, User Device Tracker, and the Engineers Toolset. See why I like to constantly learn new things I had a lot to be on top of! SolarWinds is a passion of mine still to this very day. My new role as a Product Manager for NCM is home to me. Funny how I circled around back to my favorite product that got me here in the first place. :) My goal is to educate and work with customers to leverage our products to their fullest degree!