1 2 Previous Next

Geek Speak

20 Posts authored by: Dez Employee

itsnormmal.jpg

Normalcy is boring, or is it?

          Something that I have been working on is helping to come up with a baseline security plan for an IT team and their infrastructure.  What I have ran into is that having a basic template and starting point really helps.  Fantastic right?  Well, when I start off by giving them credit for monitoring they look peculiar at me as in why would monitoring be a starting point?  To be fair and accurate a few high five me as they are like SAWEETNESS (meant to be spelled wrong as that literally is how I speak, ok back to the blog ) check that off the list of things to come!  Today, I'm going to go over this one portion of the plan and show why "knowing normal" is actually a starting point for a great security best practices and policies.

 

     First things first,my favorite quote "If you don't know what's normal how the heck do you know when something's wrong?".  Baseline and accurate monitoring history will show you whats normal.  This also will show you how your infrastructure handles new applications and loads when you are monitoring so its not just for up down that is just a side perk honestly.

 

Ok, now once you know what normal is the following will help you to see issues easier and be aware.  So remember the below is once you have monitored and understand your normalcy of your devices your monitoring.

 

Monitoring security features

  • Node -  up/down
    • This will show you if there is a DoS happening or a configuration error with no ability to ping a device. 
    • Will show you areas within your monitoring that are being possibly attacked.
    • Allows you to have a clear audit of the event that are taking place so you can use for management and your team for assessments.
  • Node - CPU/Memory/Volume
    • CPU will show you if there is an increase spike as that will help to show where to look for what increased or caused this spike that never went away.
    • Memory allows you to know if there is a spike obviously something is holding it hostage and you need to address this and prevent or resolve. 
    • Volume if you see a drive increase its capacity OR decrease quickly and are alerted to this you may be able to stop things like ransom ware quickly.  The trick is to be monitoring AND have alerts setup to make you aware of drastic changes.
  • Interface - utilization
    • Utilization will show you if a sudden increase of data is transferring into or out of an interface.
  • Log File monitoring
    • Know when AD attempts are failing.
      • This is something I see a lot of times and the person monitoring just states "yes, but its just an old app making the request no biggy".  Ok, to me I'm like fix the old application so this is no longer NOISE and when you have these coming in from outside this app you are more inclined to investigate and stop the whole.
    • Encryption know if files are being encrypted on your volumes
    • Directory changes if directory/file changes are happening you need to beware period
  • Configuration monitoring
    • Real-time change notification that compares to the baseline config is vital to make sure no one is changing configurations outside of your team.  Period end OF STORY.  (I preach this a lot I know.  #SorryNotSorry)
  • Port monitoring
    • rogue devices plugging into your network needs to be known when and who immediately

 

          This is obviously not all the reasons you can use against normalcy but its once again a start.  Understanding normal is vital to set up accurate alerts, reports, and monitoring features.  As you hone in your skills on assessing what you are monitoring and alerting you'll see things drop off while others will increase within your environment.

 

          Don't be shy to ask questions like, why is this important?  I seen this article on an attack, how can we be alerted in the future if this happens to us?  Some of the best monitoring I've seen is due to looking through THWACK and reading articles on what's going on in mainstream.  Bring this knowledge to your monitoring environment and begin crafting an awesome arsenal against, well, the WORLD.

 

HTH

~Dez~

  

MEME.jpg

Blog based on my "knee jerk" response to an article on an NSA breach

 

                So when you first read this article, you will notice that there are groups of hackers that are auctioning off exploits of devices.  May seem like no big deal but think about this. You have a group of people that are preying on your first line of defense and profiting on making these exploits available.  Irritation set to the highest level for one simple reason. NOT EVERYONE HAS A SECURITY TEAM. Ok, now that I feel better to commence the discussion on how they did this and why you may be concerned.

 

                Exploiting firewalls, you are now placing into the world factory defaults and settings that people may overlook or not think about when protecting your network.  Creating a gateway for script kitties and ill-willed individuals to try now and do harm just because the day ends in “Y”.  An example of why I constantly preach about compliance reports and their ability to help you protect your network and not forget the little things.

 

Some of the vulnerabilities listed were things like:

Buffer overflow in OpenLDAP

SNMP exploits on devices

Scripting advisement to gain more havoc

And much more…

 

So how do we guard against these untimely and devastating breaches?  One answer, stop ignoring security needs.  There are several free resources that help you protect yourself.  I realize a lot of people may or may not know these so I thought I would put together a few.

 

Common Vulnerabilities and Exposures

https://cve.mitre.org/

National Vulnerability Database

https://web.nvd.nist.gov

 

                If you read any of my NCM blogs, you would know that it has a firmware vulnerability data. Checking the NIST and advises you of security holes on your Cisco devices. Not a “catch-all” by any means but helps you to be aware and proactively having security checks every day by default.  Then as always there are compliance reports with even federal compliance reports right out of the box. Allowing you to lean on what others have created to ensure that you are crossing your T’s and dotting your I’s within your security needs.

 

                These are all ways we can try to use products to help us every day and have a direction to head in instead of ignoring or just simply put don’t make the time to address.  Monitoring and management software needs to be an everyday defensive tool.  To help offer guidance with your security needs and allowing you to work on security today and tomorrow.  Security teams can lean on monitoring\management solutions.  It’s not just for people that are lacking the funding for a security team it’s for everyone to stand together and help stand up to people exploiting for hire.

 

                Circling back to my last opinion on this article.  For hire, exploits are just as bad as hackers with ransomware.  These were merely saying “hey, pay me and I’ll tell you how you can do some damage” where ransomware is more “Hey, I encrypted or stole your data give me $$$ to (maybe) get it back.”  Is there a difference in the level of punishment if ever caught? I think there is not and we need to have better ways to prosecute and track down these criminals.  What’s your thoughts?  I’m always open to opinions and love hearing all of your comments!

 

~Dez~

Follow me on Twitter @dez_sayz

thwack2.jpeg

 

How to use network configuration, change, and compliance management (NCCCM) and other monitoring software in response to an actual security breach.

 

If you have not read part one, I would suggest that you give that an overview, so you can understand fully how and why this comes into play. For those that are ready for part two, welcome back!  I'll attempt to share some assessments of an internal sabotage and how to use things like monitoring and management software to see and recovery.  The best way to respond is by thinking ahead, having clear steps to prevent, and halt further damage.

 

Today, we are going to dive into a couple of scenarios, and directly assess ways to be alerted to and address situations that may be taking place within your organization.  Now, should we all live like we have a monkey on our back/shoulder?  No, but it doesn't hurt to have a little healthy "skepticism" about unusual things that are happening around you.  Being aware of your surroundings allows you to fight back and take back control of hiccups along the way.

 

 

Internal Planning Possible Sabotage:

Things to look for visually as well as with monitoring and management software.

 

  • Unusual behavior (after a confrontation or write-up has happened) - thank you sparda963 I forgot to place when to look for this
    • This can be obviously aggressive, but the one often overlooked is "overly" nice and helpful.
      • Yes, this sound condescending and I understand that concern but think of this as out of character.  They now want to help higher levels with mission critical information or configurations.  They want to "watch" you command line interface to a device.  They are "contributing" to get to know where key points are.  These are things that are outside of their scope.
    • Aggressive well the writing is on the wall at that point and if secretive comes into play then watch out and plan accordingly.
    • Use Real-time change notifications, approval systems, and compliance to help you see changes made, and users added to devices of monitoring management software.
      • Make sure that you have a script to remove access to devices ahead of time.  One that you can fill in the blank for the user ID and take permissions away quickly.
      • Verify you have alerts set up to notify you with quick access to the devices through a management software so you can cancel access levels and revert changes quickly.
  • Logon's found in unusual servers by said person
    • Use a Log Event Monitor to help you be alerted with strange behavior to login attempts and places.
    • Know your monitoring software and have quick pages to deny access to accounts quickly
  • New Users
    • Use a Log Event Monitor to alert you to new account creations.  You need to know when these were created and had a trail on these to remove.
  • Job creation for mass configuration changes
    • Verify through an approval system all changes on your network.  An excellent way to do this is with an NCCCM product and enable the approval system to be fully active.  You will want at least a 2 level approval system to help prevent issues and possible changes.
    • Real-time change notification with segmented emails for critical devices. 
    • Backups to be quickly accessible and found in multiple locations to ensure access during a breach.

 

Internal Execution of Sabotage:

Things to do if you find yourself under attack

(Network Side)

  • First things first
    • Log Event Monitoring - should be alerting you to access violations, additions of accounts, or deleting of accounts
    • TACACS - should be enabled and in full use for auditing within your monitoring and management software choices
    • Real-Time change notifications should be sending emails immediately to the correct people with an escalation of higher up network engineers on your team.
  • Now to fight back!
    • If they are opening firewalls to gain access you need to shut these down and stop traffic immediately.  You will need to have a plan on a script for a shut all or use something like Firewall Security Manager or Network Configuration Manager to implement commands from a stored location.
      • Allows time to figure out the user and what is going on while you can have the floodgate closed.
      • Addressed in a security protocol to enable you to have this authority.  Saving you and your company a lot of money when you are trying to prevent a massive break-in.
    • If they are deleting router configs
      • Real-time change notification (RTN) alerts should be sent out to you to bring you up to speed.
        • Use a script to deny access to the user that made the change shown in the RTN email.
        • Revert configurations from within your NCCCM software and get these back online
      • Verify users that have access
        • Use a compliance report to check access levels and remove where needed.
        • CONTINUE to monitor these reports
      • Check you Approval system
        • Verify who has access
        • Change passwords to all monitoring and management software logins.
          • I have had a customer that would set these up to one password for all that he would create if in crisis.  Allowing a quick shutdown of software usage to gain control when an attack was ensuing.
    • Verify critical application status
      • Log event monitor - check logs to see if access has been happening outside of usual
      • NetPath or something similar for pathways to check accessibility or changes
      • NCCCM - Verify all changes that have occurred within the past seven days minimum as this could only be the first wave of intrusion.
      • Network performance monitor to verify any malware or trojans that could be lingering and sending data on your network.
        • Volumes filling up and being alerted to this
        • Interface utilization skyrocketing
        • NetFlow monitor showcasing high amounts of unusual traffic or NO traffic history is essential here.

 

Security gut check:

Things to go over with yourself and team to make sure your security and plans for recovery are current.

 

Pre-Assessment

  • understand and know what is critical information within your organization
  • Where are your system boundaries
  • Pinpoint your security documentation

Assessment

  • Setup a meeting with your team over the above pre-assessment
  • Review your security information
  • Practice scenarios that "could" happen within your networks
  • Setup session controls
  • Verify maintenance plans
  • Ensure mapping of your critical networking connections with critical applications
  • Ensure your policies are relevant today as they were when first created
  • Verify entry points of concerns
    • Internal/External
  • System and Network Exposures

 

Team Analysis

  • Where are your vulnerabilities?
  • What are your Countermeasures?
  • What is the impact if breached?
  • Who can segment and take on sections of security recommendations?

 

Final

  • Implement new security plans as defined and found above.
  • Set up a meeting review for at least three months later to make sure all vulnerabilities are known and addressed.
  • Verify that the plan is accessible for your team to review so they are aware of actions to take.
  • Sign an agreement within your team to follow these protocols.

 

 

Well, that is a lot to cover, whef!  Once again everyone's networks and infrastructures are different.  You and I understand that.  The main point is how to use tools to help you stay ahead and be able to fight back with minimal damage.  Having a recovery plan and consistently updating these to new vulnerabilities is vital to stay ahead.  You can shift these and use for outside attacks as well.  Security is a fluid dance and ever changing so don't be stuck sitting on the outside looking in. 

 

 

Thank you,

~Dez~

thwack.jpg

I BEAT THEM TO FIRING ME! (Part Two) Fight Back

Why network configuration, change and compliance management (NCCCM) is a must

Inspired by former Citibank employee sentencing

(Part Two)

 

We've all heard horror stories about the disgruntled employee who pillages the office supply closet and leaves the building waving an obscene gesture, security badge skittering across the parking lot in his wake. Rage-quit is a thing, folks, and it's perfectly reasonable to be afraid that someone with high-level access, someone who could make changes to a network, might do so if they get mad enough. This happens more often than anyone would like to think about, and it's something that needs to be addressed in every organization. I felt like we should talk about this and discuss ways to help control and slow the damage of said employees and their bad will. Bottom line: we need to be aware of these situations and have a plan for recovery when things like this happen.

 

 

The gist of the story is simple: there was an employee who wiped out critical network configurations to about 90% of his former company's infrastructure.  Monday he was sentenced on charges of criminal vandalism. So, I realize the article above is technically in the past, but it brings up a great starter conversation about how IT organizations can stop criminal vandalism by actually using NCCCM products to protect ourselves and others from any type of disastrous events. Sometimes you need that brief pause or slight inconvenience to help you think straight and not go over the edge. This post can also help keep your butt out of, well, jail .

 

Today, we are going to talk about some of the risks of not having NCCCM software:

 

 

  1. Real-time change notification not enabled.
    • There is no tracking, idea, or reference to when changes are being made via maintenance plans, change requests, or malicious intent.
      • Being able to see network changes and know the timing helps you to be proactive, and gives you immediate remediation action for your network.
    • Who's on first base, and did someone slide in to home base?
      • When you have more than a couple of network engineers, documentation can be lacking and, well, you're busy, right? Being able to track when changes happen and who made them allows you to find and discover who, when, and what was changed, even when it's a week later.
      • Being able to compare the change that was made to existing is key to correlating issues after a change was made. All of a sudden, traffic is not flowing, or it's restricted, and you find out it was an error in the config change.
    • Someone is on your network changing your critical devices and wiping them clean.
      • Receive alerts so you don't find this type of information out when it's too late. Be able to log in, and after receiving the alert, restore to previous config.
  2. Approval process not in use.
    • No change auditing.
      • Being able to make changes without approval or a process sets you up for human error or worse: attacks.
      • Implementing an approval process allows you to have an auditing system that shows that more than one person approved a change.
      • Use this with real-time change notification to see if anyone outside your team is making changes. Either allow them into your NCCCM, or delete or lock out their login info to the devices.
    • No one can verify that you are making the change, or even what that change was.
      • When you have a larger team, you delegate changes or areas of functionality. Having an approval process verifies that the correct changes are being made. That gives you an extra set of eyes on the changes that are being made, which adds another level of detection to human error.
    • One person has complete access to your devices at a control level.
      • When you give people straight access to network devices there is a single point of failure. Taking an extra step creates a safe zone of recognition, training, and the ability to track changes and implementations on your network.
  3. Advanced change alert not enabled.
    • Not having an escalation alert set up can leave you with no configurations on your devices when you come into work the next day.
      • Set up escalation alerts based on more than one action.
        • Create a mass change alert if X amount of syslog changes happen within five minutes: Alert Manager NOW.
        • Mute these when implementing maintenance plans. more info by adatole
  4. Backups you are saving to your desktop or network drive (when you remember).
    • If a crisis happens, the great news is that network devices just need to be told what to do. But if you are like me and don't remember every line of code for hundreds of devices, then you better implement a backup system NOW.
      • If you have backups being stored, recovery is a click away with an NCCCM.
      • Compare starting to running to make sure a reboot won't cancel your changes.
      • Verify you have backups in secure locations so downtime is minimized and quickly averted.
        • I generally implement server side and network share drive backups. Make your server accessible with security verification lockdown in case someone tries to delete the backups (this happens because they don't want you to recover).
  5. Recovery procedures not in place.
    • Can your team recover from an emergency without you being on site?
      • Have a plan and practice with your team. You have to have a plan to be able to recover from maintenance plans gone wrong all the way to disaster recovery.  This takes practice, and should be something the whole team discusses so that you are better engaged. It helps to have an open mind to see how others may offer solutions to each potential problem suggested.
    • Setup an automatic password change template to be easily used in case of a potential issue within or outside your organization.
    • Use your NCCCM to monitor your configurations for potential issues or open back doors within your network.
      • Sometimes people will start allowing access within your network watching your configurations with a compliance reporting service allows you to detect and remediate quickly to stop these types of security breaches in their tracks.

 

If your curious on setup check this out:More info Security and SolarWinds NCM

 

Stay tuned for part two, I'll showcase how each one of these can be used in response to security!

 

Now that is a few things you should be able to use within any NCCCM software package.  This should also be something you revisit consistently to reevaluate and assess your situation and how to better protect yourself.

Let's dive into the mindset and standard methodologies around the security aspect:

 

This isn't just for technology this is in general things to be aware of and to implement on your own.  The ability to look at these with a non-judging eye and see them as just ways to hold off malicious attacks or ill will.

 

  1. There needs to be a clear exit strategy for anyone that is going to be fired or removed from a position with potential harm.
    • But he is such a nice guy?  Nice guys can turn bad.
    • When this information is being circulated you need to do what's best for your career as well as the company you work for and go on the defense.
      • Bring in specialized help organizations that can come in assess and prevent issues before they are terminated or moved
      • Make sure you verify all traffic and location they were involved in
        • Any passwords etc that were globally known NEEDS CHANGED NOW not LATER
        • Check all management software and pull rights to view only in the remainder days then delete access immediately after termination
        • Verify all company technology is accounted for (Accounting and inventory within your NCCCM is vital to maintain diligence on awareness of property and access to your network)
  2. Monitoring of team
    • Some may not be happy with a decision to terminate an employee and feel betrayed
    • Monitor their access and increase awareness to their actions
      • If you see them logging in to more routers and switches than ever before might setup a meeting...
      • See them going outside of their side and digging into things they should not, meeting time
      • Awareness is key and an approval process and change detection is key to preventing damage
  3. Security policies
    • You're only as good as the policy in place
      • Dig into your policies and make sure they are current and relevant
      • If you seriously have things like "If they call from desk phone reset password over the phone" type of security measures please REVISIT these.
        • Re-read that last statement
    • Make sure your team is signing acknowledgement of what they can and cannot do
      • Easier to prosecute when they have signed and agreed
    • Verify your security policies to your network devices
      • NCCCM compliance reporting setup for your needs is a great way to stay ahead of these items
      • You can find back doors on your network that people have setup to go around security policies this way. 

 

     I cannot obviously solve every issue, but at least help to point you into some good directions and processes.  If any of you want to jump in and add to this, please do I'm always interested in other people's methods of security.  The main point is to be aware of these situations, have a plan and recover when things like this happen.

 

Thank you,

 

~Dez~

 

Follow me on Twitter:

@Dez_Sayz

If you’re in management, you may not understand the effects of changes on your network.  However, if you’re the network engineer you know exactly the effects and ramifications that come with a change on your network.  The slightest change can literally cause an outage.

 

So what’s the big deal with software companies that want you to buy Network Configuration Change Management (NCCM) software?  Well I know personally that a few of you have been in this exact position and on both sides of this ball.  As a manager you want to have a seamless network and keep down costs.  As the network engineer you want to be able to have a smooth running network and a happy manager.

 

What is the happy medium here?  When are too many software tools or too many diagrams on walls and an over-abundance of saved test files enough to know software is required to actually manage all of this?

 

SolarWinds offers a Network Configuration-Change Management package. Does this mean it’s the best?  No, as that is in the eye of the beholder and user. Does this mean that it is manageable and can save me time and my manager money?  You’re darn rightit can do both very easily!

 

Yes, there are other software tools that do all about the same thing with little differences along the way.  Just like I like thin pancakes and you may like fluffy thick pancakesin the end they are still pancakes.

 

Now to know what a good NCCM is regardless of the name across it, let’s go over the top 6 reasons to have such software.

 

  1. Making changes because you were told to…
    1. You want to be able to know if someone is in fact making changes immediately and have a way to revert changes if needed.  NCCM software allows you to do this and consistently backs up your devices in case such changes are incorrect and provides a complete barebones backup if needed for a new device.
  2. Scheduled device changes
    1. Planning IOS upgrades, change in ACL lists, SNMP passwords, or many other items on your daily tasks.  Having a program that will allow you to monitor and roll out these changes saves time and show results quickly.
  3. A second pair of eyes
    1. It’s good to have an approval system in place so that scripting and changes receive a second look before deployment.  This helps prevent outages and mistakes, and definitely is valuable when your network has service level agreements and high availability needs.
  4. BACK UPS…BACK UPS!
    1. I cannot say this enough…if you do not have regular backups of your system that are easily retrievable, you do not have a fully reliable network.  PERIOD. Backing up to your local machine is not acceptable…You know who you are
  5. Automation of the tasks you might rather forget...
    1. Being able to detect issues within your configuration through compliance reporting, real-time change detection, scheduled IOS upgrades, inventory, and many more automated tasks. This allows you to focus on the integrity and availability of your network.
  6. Security
    1. If you have certain required security measures within your configurations, then you need compliance reporting.  With NCCM software, you can schedule a report or run it manually and print out that your ‘state of compliance’ within seconds instead of per device.

 

Well there are a few valuable reasons to at least consider this type of software.  If you have any other thoughts, feel free to drop me a line! Add to my list or take away, I’m a pretty open mined individual.

 

If you’re looking for more information this has a solid outlook on NCCM and businesses.

Filter Blog

By date: By tag:

SolarWinds uses cookies on its websites to make your online experience easier and better. By using our website, you consent to our use of cookies. For more information on cookies, see our cookie policy.