I Beat Them to Firing Me: A Story of Corporate Sabotage

I BEAT THEM TO FIRING ME! (Part Two) Fight Back

Why network configuration, change and compliance management (NCCCM) is a must

Inspired by former Citibank employee sentencing

(Part Two)

We've all heard horror stories about the disgruntled employee who pillages the office supply closet and leaves the building waving an obscene gesture, security badge skittering across the parking lot in his wake. Rage-quit is a thing, folks, and it's perfectly reasonable to be afraid that someone with high-level access, someone who could make changes to a network, might do so if they get mad enough. This happens more often than anyone would like to think about, and it's something that needs to be addressed in every organization. I felt like we should talk about this and discuss ways to help control and slow the damage of said employees and their bad will. Bottom line: we need to be aware of these situations and have a plan for recovery when things like this happen.

The gist of the story is simple: there was an employee who wiped out critical network configurations to about 90% of his former company's infrastructure.  Monday he was sentenced on charges of criminal vandalism. So, I realize the article above is technically in the past, but it brings up a great starter conversation about how IT organizations can stop criminal vandalism by actually using NCCCM products to protect ourselves and others from any type of disastrous events. Sometimes you need that brief pause or slight inconvenience to help you think straight and not go over the edge. This post can also help keep your butt out of, well, jail .

Today, we are going to talk about some of the risks of not having NCCCM software:

1. Real-time change notification not enabled.
   • There is no tracking, idea, or reference to when changes are being made via maintenance plans, change requests, or malicious intent.
     • Being able to see network changes and know the timing helps you to be proactive, and gives you immediate remediation action for your network.
   • Who's on first base, and did someone slide in to home base?
     • When you have more than a couple of network engineers, documentation can be lacking and, well, you're busy, right? Being able to track when changes happen and who made them allows you to find and discover who, when, and what was changed, even when it's a week later.
     • Being able to compare the change that was made to existing is key to correlating issues after a change was made. All of a sudden, traffic is not flowing, or it's restricted, and you find out it was an error in the config change.
   • Someone is on your network changing your critical devices and wiping them clean.
     • Receive alerts so you don't find this type of information out when it's too late. Be able to log in, and after receiving the alert, restore to previous config.
2. Approval process not in use.
   • No change auditing.
     • Being able to make changes without approval or a process sets you up for human error or worse: attacks.
     • Implementing an approval process allows you to have an auditing system that shows that more than one person approved a change.
     • Use this with real-time change notification to see if anyone outside your team is making changes. Either allow them into your NCCCM, or delete or lock out their login info to the devices.
   • No one can verify that you are making the change, or even what that change was.
     • When you have a larger team, you delegate changes or areas of functionality. Having an approval process verifies that the correct changes are being made. That gives you an extra set of eyes on the changes that are being made, which adds another level of detection to human error.
   • One person has complete access to your devices at a control level.
     • When you give people straight access to network devices there is a single point of failure. Taking an extra step creates a safe zone of recognition, training, and the ability to track changes and implementations on your network.
3. Advanced change alert not enabled.
   • Not having an escalation alert set up can leave you with no configurations on your devices when you come into work the next day.
     • Set up escalation alerts based on more than one action.
       • Create a mass change alert if X amount of syslog changes happen within five minutes: Alert Manager NOW.
       • Mute these when implementing maintenance plans. more info by adatole​
4. Backups you are saving to your desktop or network drive (when you remember).
   • If a crisis happens, the great news is that network devices just need to be told what to do. But if you are like me and don't remember every line of code for hundreds of devices, then you better implement a backup system NOW.
     • If you have backups being stored, recovery is a click away with an NCCCM.
     • Compare starting to running to make sure a reboot won't cancel your changes.
     • Verify you have backups in secure locations so downtime is minimized and quickly averted.
       • I generally implement server side and network share drive backups. Make your server accessible with security verification lockdown in case someone tries to delete the backups (this happens because they don't want you to recover).
5. Recovery procedures not in place.
   • Can your team recover from an emergency without you being on site?
     • Have a plan and practice with your team. You have to have a plan to be able to recover from maintenance plans gone wrong all the way to disaster recovery.  This takes practice, and should be something the whole team discusses so that you are better engaged. It helps to have an open mind to see how others may offer solutions to each potential problem suggested.
   • Setup an automatic password change template to be easily used in case of a potential issue within or outside your organization.
   • Use your NCCCM to monitor your configurations for potential issues or open back doors within your network.
     • Sometimes people will start allowing access within your network watching your configurations with a compliance reporting service allows you to detect and remediate quickly to stop these types of security breaches in their tracks.

If your curious on setup check this out:More info Security and SolarWinds NCM

Stay tuned for part two, I'll showcase how each one of these can be used in response to security!

Now that is a few things you should be able to use within any NCCCM software package.  This should also be something you revisit consistently to reevaluate and assess your situation and how to better protect yourself.

Let's dive into the mindset and standard methodologies around the security aspect:

This isn't just for technology this is in general things to be aware of and to implement on your own.  The ability to look at these with a non-judging eye and see them as just ways to hold off malicious attacks or ill will.

1. There needs to be a clear exit strategy for anyone that is going to be fired or removed from a position with potential harm.
   • But he is such a nice guy?  Nice guys can turn bad.
   • When this information is being circulated you need to do what's best for your career as well as the company you work for and go on the defense.
     • Bring in specialized help organizations that can come in assess and prevent issues before they are terminated or moved
     • Make sure you verify all traffic and location they were involved in
       • Any passwords etc that were globally known NEEDS CHANGED NOW not LATER
       • Check all management software and pull rights to view only in the remainder days then delete access immediately after termination
       • Verify all company technology is accounted for (Accounting and inventory within your NCCCM is vital to maintain diligence on awareness of property and access to your network)
2. Monitoring of team
   • Some may not be happy with a decision to terminate an employee and feel betrayed
   • Monitor their access and increase awareness to their actions
     • If you see them logging in to more routers and switches than ever before might setup a meeting...
     • See them going outside of their side and digging into things they should not, meeting time
     • Awareness is key and an approval process and change detection is key to preventing damage
3. Security policies
   • You're only as good as the policy in place
     • Dig into your policies and make sure they are current and relevant
     • If you seriously have things like "If they call from desk phone reset password over the phone" type of security measures please REVISIT these.
       • Re-read that last statement
   • Make sure your team is signing acknowledgement of what they can and cannot do
     • Easier to prosecute when they have signed and agreed
   • Verify your security policies to your network devices
     • NCCCM compliance reporting setup for your needs is a great way to stay ahead of these items
     • You can find back doors on your network that people have setup to go around security policies this way. 

     I cannot obviously solve every issue, but at least help to point you into some good directions and processes.  If any of you want to jump in and add to this, please do I'm always interested in other people's methods of security.  The main point is to be aware of these situations, have a plan and recover when things like this happen.

Thank you,

~Dez~

Follow me on Twitter:

@Dez_Sayz