Skip navigation

Geek Speak

14 Posts authored by: Dez Employee

I know, I'm a day late and quite possibly 37 cents short for my coffee this morning, so let's jump in, shall we?

 

Let's start with the Equifax breach. This came up in the Shields Down Conversation Number Two, so, I thought I would invite some of my friends from our security products to join me to discuss the breach from a few different angles.

 

My take will be from a business strategy (or lack of) standpoint. Roughly 143 million people had their personal data exposed because Equifax did not properly execute a simple patching plan. Seriously?

 

Is this blog series live and viewable? I am not the only person who implements patching, monitoring, log and event management in my environments. This is common knowledge. What I don't get is the why. Why, for the love of everything holy, do businesses not follow these basic practices?

 

CIxO or CXOs do not implement these practices. However, it is their duty (to their company and their core values) to put the right people in place who will ensure that security measures are being carried out.

 

 

Think about that for a moment and then know that there was a patch produced for the vulnerability that Equifax failed to remediate in March. This breach happened, as we all know, in mid-May. Where is the validation? Where was the plan? Where is the ticketing system tracking the maintenance that should've been completed on their systems? There are so many questions, especially since this happened in an enterprise organization, not some small shop somewhere.

 

Now, let's take this another step further. Equifax dropped another juicy nugget of information of another breach in March. Don't worry, though. It was an entirely different attack. However, the incredible part is that some of the upper-level folks were able to sell their stock. That makes my heart happy, you know, to know that they had the time to sell their stock before they released information on being breached. Hat's off to them for that, right?

 

Then, another company decided they needed to market and sell credit monitoring (for a reduced fee, that just so happens to use EQUIFAX SERVICES) to the individuals who were now at a high(er) risk of identity theft and credit fraud. I'm still blown away by this.

 

Okay. Deep breath. Whooooo.

 

I was recently informed that when you have third-party software, patching is limited and that organization's SLAs for application uptime don't allow patching on some of their servers. I hear you! I am a big believer that some patching servers can cause software to stop working or result in downtime. However, this is where you have to implement a lab and test patching. You should check your patching regardless to make sure you are not causing issues with your environment in the first place. 

 

I will implement patching on test servers usually on a Friday, and then I will verify the status of my applications on the server.

I will also go through my security checks to validate that no new holes or revert have happened before I implement in production within two weeks. 

 

Now let's bring this back to the strategy at hand. When you are an enterprise corporation with large amounts of personal data belonging to your trusting customers (who are the very reason you are as large as you are), you better DARN WELL have a security plan that is overseen by more than one individual! Come on! This is not a small shop or even a business that could argue, "Who would want our customer data?" We're talking about Equifax, a company that holds data about plenty of consumers who happen to have great credit. Equifax is figuratively a lavish buffet for hackers.

 

The C-level of this company should have kept a close eye on the security measures being taken by the organization, including patching, SQL monitoring, log, events, and traffic monitoring. They should have known there were unpatched servers. The only thing I think they could have argued was the common refrain, "We cannot afford downtime for patching." But still. 

 

Your CxO or CIxO has to be your IT champion! They have to go nose to nose with their peers to make sure their properly and thoroughly designed security plans get implemented 100%. They hire the people to carry out such plans, and it is their responsibility to ensure that it gets done and isn't blocked at any level.

 

Enough venting, for the moment. Now I'd like to bring in some of my friends for their take on this Equifax nightmare that is STILL unfolding! Welcome joshberman, just one of my awesome friends here at SolarWinds, who always offers up great security ideas and thoughts.

 

Dez summed up things nicely in her comments above, but let's go back to the origins of this breach and explore the timeline of events to illustrate a few points.

 

  • March 6th: the exploited vulnerability, CVE-2017-5638, became public
  • March 7th: Security analysts began seeing attacks propagate that were designed to exploit this flaw
  • Mid-May: Equifax tracked the date of compromise back to this window of time
  • July 29th: the date Equifax discovered a breach had occurred

 

Had a proper patch management strategy been set in place and backed by the right patch management software to enable the patching of third-party applications, it is likely that Equifax might not have succumbed to such a devastating attack. This applies even if testing had been factored into the timelines, just as Dez recommends. "Patch early, patch often" certainly applies in this scenario, given the voracious speed of hackers to leverage newly discovered vulnerabilities as a means to their end. Once all is said and done, if there is one takeaway here it is that patching as a baseline IT security practice, is and will forever be a must. Beyond the obvious chink in Equifax's armor, there is a multitude of other means by which they could have thwarted this attack, or at least minimized its impact.

 

That's fantastic information, Josh. I appreciate your thoughts. 

 

I also asked mandevil (Robert) for his thoughts on the topic. He was on vacation, but he returned early to knock out some pertinent thoughts for me! Much appreciated, Robert!

 

Thanks, Dez. "We've had a breach and data has been obtained by entities outside of this company."

Imagine being the one responsible for maintaining a good security posture, and the sinking feeling you had when these words were spoken. If this is you, or even if you are tangentially involved in security, I hope this portion of this post helps you understand the importance of securing data at rest as it pertains to databases.

 

Securing data in your database

 

The only place data can't be encrypted is when it is in cache (memory). While data is at rest (on disk) or in flight (on the wire), it can and should be encrypted if it is deemed sensitive. This section will focus on encrypting data at rest. There are a couple different ways to encrypt data at rest when it is contained within a database. Many major database vendors like Microsoft (SQL Server) and Oracle provide a method of encrypting called Transparent Data Encryption (TDE). This allows you to encrypt the data in the files at the database, table space, or column level depending on the vendor. Encryption is implemented using certificates, keys, and strong algorithms and ciphers.

 

Links for more detail on vendor TDE description and implementation:

 

SQL Server TDE

Oracle TDE

 

Data encryption can also be implemented using an appliance. This would be a solution if you would want to encrypt data but the database vendor doesn't offer a solution or licensing structures change with the usage of their encryption. You may also have data outside of a database that you'd want to encrypt that would make this option more attractive (think of log files that may contain sensitive data). I won't go into details about different offers out there, but I have researched several of these appliances and many appear to be highly securitized (strong algorithms and ciphers). Your storage array vendor(s) may also have solutions available.

 

What does this mean and how does it help?

 

Specifically, in the case of Equifax, storage level hacks do not appear to have been employed, but there are many occurrences where storage was the target. By securing your data at rest on your storage tier, it can prevent any storage level hacks from obtaining any useful data. Keep in mind that even large database vendors have vulnerabilities that can be exploited by capturing data in cache. Encrypting data at the storage level will not help mitigate this.

 

What you should know

 

Does implementing TDE impact performance? There is overhead associated with encrypting data at rest because the data needs to be decrypted when read from disk into cache. That will take additional CPU cycles and a bit more time. However, unless you are CPU-constrained, the impact should not be noticeable to end-users. It should be noted that index usage is not affected by TDE. Bottom line is if the data is sensitive enough that the statement at the top of this section gets you thinking along the lines of a resume-generating event, the negligible overhead impact of implementing encryption should not be a deterrent from its use. However, don't encrypt more than is needed. Understand any compliance policies that govern your business (PCI, HIPAA, SOX, etc.).

 

Now to wrap this all up.

 

When we think of breaches, especially those involving highly sensitive data or data that falls under the scope of regulatory compliance, SIEM solutions certainly come to mind. This software performs a series of critical functions to support defense-in-depth strategies. In the case of Equifax, their most notable influence appears to be their attempt to minimize the time of detection with either the compromise or the breach itself. On one hand, they support the monitoring and alerting of anomalies on the network that could indicate a compromise. On the other, they can signal the exfiltration of data – the actual event of the breach – by monitoring traffic on endpoints and bringing to the foreground spikes in outbound traffic, which, depending on the details, may otherwise go unnoticed. I'm not prepared to make the assumption that Equifax was lacking such a solution, but given this timeline of events and their lag in response, it begs the question.

 

As always, thank you all for reading and keep up these excellent conversations.


THWACK members, I'm 100% loving the comments in this series! You all are giving me a much-needed boost in security thoughts and ideas. Thank you so much!

 

Conversation Number One led me to realize that I need to jot down the resources I use as my "go-to's." These are links to several places that help me to be cyber-aware if you will.  I would love for all of you to share your resources as well so we can help create a thread of wholesome greatness! tomiannelli, your comment, from Conversation One, that provided a link for more information (18 U.S. Code § 1030 - Fraud and related activity in connection with computers) was really thoughtful. I truly appreciate the sharing of knowledge.

 

Now, let's dive in, shall we?

 

Security Conferences

 

InfoSec

Conferences - O'Reilly Media

ShmooCon

SANS Events

 

Knowledge Links

 

Department of Homeland Security

I spend hours on this site trying to see which direction the government is leaning toward. I also like going there to view their education suggestions and which cyber security fields they are hiring in.

 

National Vulnerability Database

Checklists, data feeds, vulnerability metrics, and more resource links provided within. This is a bookmarked staple.

 

SANS Institute InfoSec

This is a white paper that I find myself reflecting on a lot. Especially when I'm focusing on new security plans with companies that have never really had one in place. The concepts and case studies within help to ground me for some reason.

 

Ciscohttps://learn-umbrella.cisco.com/ebooks Umbrella

Okay, if you click on this one it will want you to fill out information before you download any of their books. I'm a huge Cisco user and when it comes to security and concepts, well, I'm just like my best friend, Kate Asaff, when Apple has a release. Let's just say that I'm interested in the new capabilities and features.

 

There is SO much more, but these are my top picks that I consistently go back to. Now, DEF CON is not on any of my previous lists, and this is merely because I would assume it's expected. 

 

The challenge now (drum roll, please), is to prompt EVERYONE reading this to share your favorite security sites. On your mark, get set, GO!

"The network is down!" screams an unhappy user via VOIP. Ugh! How are we able to stay on top of applications, databases, networks, and services as network engineers? Metrics are something we can all understand. So, why not combine these into one view for easier troubleshooting and helping to assess situations quickly and accurately?

 

Join me and Senior Product Managers Steven Hunt and Chris O’Brien for our THWACKcamp 2017 session, "Monitoring Like a SysAdmin When You're a Network Engineer" to learn how you can apply system monitors to cover your business-critical applications and be proactive about keeping network issues to a minimum. We will also cover how to verify the performance of systems/applications after network upgrades or features have been applied, and discuss how to break down silos and engage with your systems teams to better monitor your network. You'll learn how to share dashboards that allow you to prove your network before, during, and after the fallout.

 

We are continuing our expanded-session, two-day, two-track format for THWACKcamp 2017. SolarWinds product managers and technical experts will guide attendees through how-to sessions designed to shed light on new challenges, while Head Geeks and IT thought leaders will discuss, debate, and provide context for a range of industry topics.

 

In our 100% free, virtual, multi-track IT learning event, thousands of attendees will have the opportunity to hear from industry experts and SolarWinds Head Geeks and technical staff. Registrants also get to interact with each other to discuss topics related to emerging IT challenges, including automation, hybrid IT, DevOps, and more.

 

With over 16 hours of training, educational content, and collaboration, you won’t want to miss this!

 

Check out our promo video and register now for THWACKcamp 2017! And don't forget to catch my session!

Security concerns are getting lots of media coverage these days, given the massive breaches of data that are becoming more common all the time. Businesses want to have a security plan, but sometimes don't have the resources to create or implement one. Protect your infrastructure with the simple features that a SIEM application provides. Simple, step-by-step implementation allows you to lock in a solid security plan today.

 

In my THWACKcamp 2017 session, "Protecting the Business: Creating a Security Maturity Model with SIEM," Jamie Hynds, SolarWinds Product Manager, and I will present a hands-on, end-to-end, how-to configure and use Log & Event Manager, including configuring file integrity monitoring, understating the effects of normalization, and creating event correlation rules.

 

In our 100% free, virtual, multi-track IT learning event, thousands of attendees will have the opportunity to hear from industry experts and SolarWinds Head Geeks -- such as Leon and me -- and technical staff. Registrants also get to interact with each other to discuss topics related to emerging IT challenges, including automation, hybrid IT, DevOps, and more.

 

We are bringing our expanded-session, two-day, two-track format from THWACKcamp 2016 to THWACKcamp 2017. SolarWinds product managers and technical experts will guide attendees through how-to sessions designed to shed light on new challenges, while Head Geeks and IT thought leaders will discuss, debate, and provide context for a range of industry topics.

 

Check out our promo video and register now for THWACKcamp 2017! And don't forget to catch my session!

I have wanted to start an ongoing conversation about security on Geek Speak for a long time. And now I have! Consider this the beginning of a security conversation that I encourage everyone to join. This bi-monthly blog will cover security in a way that combines the discussions we hear going on around us with the ones we have with colleagues and friends. I’d love for you to share your thoughts, ask questions, and ENGAGE! Your input will make this series that much richer and more interesting.

 

You can bring up any topic or share any ideas that you would like for me to talk about. Please join me in creating some entertaining reading with a security vibe. Let’s start…NOW!

 

Let me dive into something that I feel is going to impact hacking behaviors. Microsoft is attempting to find clever, more intense ways to go after hackers. This may not sound surprising, but think about this: They are filing legal suits over trademarks. What? That’s right. They are suing known hacker groups for trademarks. Although you can’t drag hackers to court, you can observe and disrupt their end game.

 

Okay, so they went after the group that was allegedly involved with the United States voting process. So far, Microsoft has taken over at least 70 different Fancy Bear, or FB, domains!

 

Why does this matter? Why should we care? Because FB literally became the man in the middle, legally speaking. By using Microsoft’s products and services, they opened themselves up to be taken over by... that’s right: Microsoft!

 

Since 2016, Microsoft has mapped out and observed FB’s server networks, which means they can indirectly cause their own mayhem. Okay, so they aren’t doing THAT, but they are observing and disrupting foreign intelligence operations. Cheeky, Microsoft. Cheeky!

 

Now, for me, I’m more interested in when they decide they can flip it over into their hands to eavesdrop and scan out networks. The United States’ Computer Fraud and Abuse Act gives Microsoft quite a blanket to keep warm under. But we can go into that later, as it is currently in use at Def Con...

 

Now, I started the conversation. It’s your turn to keep it going. Share your thoughts about Microsoft, security, hackers, etc. below.

Dez

Firewall Logs - Part Two

Posted by Dez Employee Jun 1, 2017

In Part One of this series, I dove into the issue of security and compliance. In case you don't remember, I'm reviewing this wonderful webcast series

to stress the importance of the information presented in each. This week, I'm focusing on the firewall logs webcast.

 

I chose the Firewall Logs webcast for this week because it is a known and very useful way to prevent attacks. Now, my takeaway from this session is that SIEMs are fantastic ways to normalize your logs from a firewall and also your infrastructure. You guys don't need me to preach on that, I know. However, I feel like when you use health performance and network configuration management tools, you really have a better solution all the way around.

 

Everyone (I think) knows that I'm not one to tell you to buy or purchase just SolarWinds products! So please do NOT take this that way. I will preach about having some type of SIEM, network performance monitor (NPM), patch manager (PaM), and a solid network configuration change management (NCM) within your environment. Let me give you some information to go along with this webcast on how I would personally tie these together. 

 

  1. Knowing the health of your infrastructure allows you to see anomalies. When this session was discussing the mean time to detection I couldn't help but think about a performance monitor. You have to know what normal is and have a clear baseline before an attack.
  2. Think about the ACLs along with your VLANs and allowed traffic on your network devices. NCM allows you to use a real-time change notification to help you track if any outside changes are being made and shows you what was changed.  Also, using this with the approval system allows you to verify outside access and stop it in its tracks as they are not approved network config changes. This is a huge win for security.  When you also add in the compliance reports and scheduled email send-outs you are able to verify your ACLs and access based on patterns you customize to your company's needs. This is vital for documentation and also if you have any type of a change request ticketing to validate.
  3. We all know we need to be more compliant and patch our stuff! Not only to be aware of vulnerabilities but also to protect our vested interests in our environment.

 

Okay, so the stage is laid out and I hope you see why you need more than just a great SIEM like LEM to back, plan, and implement any type of security policies you may need. This webcast brings up great points to think about on how to secure and think about those firewalls. IMHO, if you have LEM, Jamie's demo should help you guys strengthen your installation.  Also, the way he presents this helps you to strengthen or validate any SIEM you may have in place currently.

 

I hope you guys are enjoying this series as much as I am. I think we should all at least listen to security ideas to help us strengthen our knowledge and skill sets. Trust me, I'm no expert or I would abolish these attacks, lol! What I am is a passionate security IT person who wants to engage different IT silos to have a simple conversation about security.

 

Thanks for your valuable time! Let me know what you think by posting a comment below, and remember to follow me @Dez_Sayz!

Today, I want to bring your attention to a great series of webcasts that are available here: Security Kung Fu Webcast Series

 

I will stress the importance of each one of these over the next few weeks as I review and reflect on what I learned from these webcasts.

 

That's right. I'm reviewing the webcast as a critic in this series because I deeply believe in security, and I want to make sure you guys are aware of the content provided in each webcast. Please follow me on this security adventure and dive into the importance of the information they covered. Also, I'll be mixing them up, so the reviews won't be presented in order. 

 

Takeaways

 

1. There is a difference in being secure versus compliant.

  • I can comply with regulations, but does that cover everything within my infrastructure?
  • I can secure my environment, but does that mean I am meeting my overall compliance needs?

 

These are questions that I like to ask whenever I'm involved with any security plan. This helps to make sure that my environment is fluid and being assessed by both sides of the argument.

 

2. Too many rules to follow! I just want to do my job!

  • News flash: Security is a business issue. It's NOT just for IT!
  • This webcast talks about the rules and compliance needs for different types of businesses. However, all levels of users need to focus on security. This means engaging with and training them at every opportunity.

 

The biggest issue that I see is a lack of a solid security planning that is integral to an organization's overarching business strategy. This webcast offers insight on ways to use tools to help you complete security plans faster and strengthen your proactive and reactive security needs.

 

Summary

 

The Security vs Compliance webcast will help guide you toward implementing a solid security plan. I joined this webcast and offered some of my opinions on being secure vs compliant, so please feel free to let me know if you have more to add!

 

Remember, "Security is a very fluid dance. The music may change, but you have to keep dancing."

 

If there is something specific you guys want me to bring up, please let me know! I love talking security and how to use what you have to support any security plan. Leave me a security comment and I'll see if I can get this ramped up and answer in a future Geek Speak blog!

The latest attack seemingly took the world by surprise. However, most of the affected users were using unpatched and unlicensed versions of Windows. How do we take a stand against ransomware and avoid being sidelined by these attacks? Here are a few things that I do and am happy to share in an effort to help strengthen your resistance against these attacks.

****

Update:  Assuming is never a good idea! Of course, your need for data backups is critical in ransomware attacks. But, it's not enough to have backups. You must also validate that they are usable and that the process works through testing.

 

****

  1. File Integrity Monitoring
    1. Monitoring your files for things like changing file extensions, moving of files, and authorization. Log & Event Manager (LEM) is vital in this to help protect your businesses information.
  2. Group Policies for Windows
    1. Cryptolocker prevention kits that do not allow ransomware to install in their most common locations.
    2. Make sure the Users group does not have full access to folders. I see this a lot, where a user group has full access to numerous folders.
    3. Make sure that users do not have rights to the registry!
  3. Static Block List
    1. Block known Tor IP addresses example: 146.185.220.0/23
  4. Limit network share access
    1. If they are able to penetrate and get to a server, you do not want to freely allow the ransomware full access to network shares. You also do not want a general user to have access to network shares that hold mission critical data. Think about this. Make sure you are applying policies and not giving users access to things they shouldn't. Allowing such gives attackers the same level of access.
  5. Update patching on servers
    1. If you are not patching your servers, you are not up to date on the malicious vulnerabilities that are already known. Stop being low hanging fruit and start being the insect spray to keep these attacks to a minimum.  Patch Manager will help you schedule and push these out so you are not worrying about being up to date. 
    2. The lab environment is key to making sure your third-party software is easily able to receive a patch. We all know that when a software or application is released, it is not aware of what's coming in the future. That is why installing a lab environment to test patches is a great way to help you patch and not be worried about breaking an application in the process.
  6. Spam
    1. For the love of everything great, update your spam filters. This is key to helping you keep malware from getting to people that are not aware of these attacks, which results in them being blamed. Preventing these emails of destruction helps keep your teams aware. You can even use them as user education.
  7. Test your plan
    1. Test out a fake ransomware email with your business. See who reacts and within what departments. This will help you to train people within their areas to not react to these type of emails.
    2. You may be surprised at how many people will click and simply give away their passwords. This is an opportunity for you to shine as an IT organization by using this information to help get funds and user training for the business.
  8. Web filter
    1. Control the sites that users can access. Use egress or outbound traffic filtering to block connections to malicious hosts.
  9. Protect your servers and yourselves
    1. Have a companywide anti-virus/malware program that is updated and verified. Patch Manager will help you determine who is up to date and who is not!
  10. Web settings
    1. Verify that your web settings do not allow for forced downloads.

 

 

There are lots of ways to protect ourselves at work and at home. The main reason why I focus on the home in my user education is because we can prevent these from work -- to a point. However, when the user goes home, they are an open door. So including user education to go over ways of protecting home environments is as much of a responsibility for the IT team as it is for the users themselves. Once home, the ransomware could decipher that blocked call and take over your machine.

 

We can try to protect ourselves with things like LEM, which alerts you when users come online, and see if their files have changed or are being changed.  However, NOT clicking the "click bait" email is what will ultimately help end-users be stronger links in the equation.

 

I hope this prompts you to raise questions about your security policies and begin having conversations about setting in place a fluid and active security plan. You never know what today or tomorrow will bring in bitcoin asks...

So, I’m sure you're all aware of the Google phishing scam. It, conveniently, presents a few key items that I would like to discuss.

 

What we know, as in what Google will tell us, is that the expedition did not represent an access of information. Rather, it merely gathered contacts and re-sent the phishing email for fake Google docs. Clearly, we need to discuss the key identifiers of how to protect yourself from similar attacks. The phishing emails were sent from (supposedly) hhhhhhhhhhhhhhhh@mailinator.com. Now if that doesn't look fishy, I don’t know what does. Regardless, people obviously opened it.

 

Another critical element is that the link the Google docs directed you to led to nothing more than a long chain of craziness, instead of a normal Google doc location. However, like most phishing, it appears to be from someone you know. So how can we protect ourselves?

 

Google installed several fixes within an hour. This shows great business practices for security on their side. We have to know that there is no one-size-fits-all for security, period. New breaches are happening every second, and we don’t always know the location, intent, or result of these attacks. What we can do is be mindful that we are no longer free-range users, and we have a personal responsibility to be aware of attacks, both at home and at work.

 

So, I'd like to help you learn the basics of looking for and recognizing phishing emails. First, and always, begin with being suspicious. Here are some ideas to help strengthen your Spidey senses:

 

  • Report phishing emails to your IT team or personal email account providers. If they don’t know, they can't fix the issue. They may eventually find out, but think of this as your friendly Internet Watch program.
  • Avoid attacks. NEVER give personal information unless you know why you are being asked for it, and are100% able to verify the email address. Make sure the email address actually matches the sender.
  • Hover over links and verify if they are going to the correct location.
  • Update your browser security settings. Google released a fix for this and pushed it out within hours.
  • Patch your devices -- including MOBILE! Android had an updated phishing release from Google within hours.
  • Stop thinking of patches for your phone as a feature request.

 

We can be our own cyber security eye in the sky! All it takes is motivation and time to be hacked, breached, or attacked, so we must be diligent and not let down our guards. Being vigilant is critical, as is proactively protecting ourselves at home and work by practicing a few simple practices.

 

And another thing: Let's stop sending out our SSIDs at home like a bat signal. There are little things we can do everywhere. Go big and implement MAC address filtering that will determine if anyone is trying to access your Wi-Fi big time. (Take it from someone who has four teenage daughters.)

 

 

~Dez~

itsnormmal.jpg

Normalcy is boring, or is it?

          Something that I have been working on is helping to come up with a baseline security plan for an IT team and their infrastructure.  What I have ran into is that having a basic template and starting point really helps.  Fantastic right?  Well, when I start off by giving them credit for monitoring they look peculiar at me as in why would monitoring be a starting point?  To be fair and accurate a few high five me as they are like SAWEETNESS (meant to be spelled wrong as that literally is how I speak, ok back to the blog ) check that off the list of things to come!  Today, I'm going to go over this one portion of the plan and show why "knowing normal" is actually a starting point for a great security best practices and policies.

 

     First things first,my favorite quote "If you don't know what's normal how the heck do you know when something's wrong?".  Baseline and accurate monitoring history will show you whats normal.  This also will show you how your infrastructure handles new applications and loads when you are monitoring so its not just for up down that is just a side perk honestly.

 

Ok, now once you know what normal is the following will help you to see issues easier and be aware.  So remember the below is once you have monitored and understand your normalcy of your devices your monitoring.

 

Monitoring security features

  • Node -  up/down
    • This will show you if there is a DoS happening or a configuration error with no ability to ping a device. 
    • Will show you areas within your monitoring that are being possibly attacked.
    • Allows you to have a clear audit of the event that are taking place so you can use for management and your team for assessments.
  • Node - CPU/Memory/Volume
    • CPU will show you if there is an increase spike as that will help to show where to look for what increased or caused this spike that never went away.
    • Memory allows you to know if there is a spike obviously something is holding it hostage and you need to address this and prevent or resolve. 
    • Volume if you see a drive increase its capacity OR decrease quickly and are alerted to this you may be able to stop things like ransom ware quickly.  The trick is to be monitoring AND have alerts setup to make you aware of drastic changes.
  • Interface - utilization
    • Utilization will show you if a sudden increase of data is transferring into or out of an interface.
  • Log File monitoring
    • Know when AD attempts are failing.
      • This is something I see a lot of times and the person monitoring just states "yes, but its just an old app making the request no biggy".  Ok, to me I'm like fix the old application so this is no longer NOISE and when you have these coming in from outside this app you are more inclined to investigate and stop the whole.
    • Encryption know if files are being encrypted on your volumes
    • Directory changes if directory/file changes are happening you need to beware period
  • Configuration monitoring
    • Real-time change notification that compares to the baseline config is vital to make sure no one is changing configurations outside of your team.  Period end OF STORY.  (I preach this a lot I know.  #SorryNotSorry)
  • Port monitoring
    • rogue devices plugging into your network needs to be known when and who immediately

 

          This is obviously not all the reasons you can use against normalcy but its once again a start.  Understanding normal is vital to set up accurate alerts, reports, and monitoring features.  As you hone in your skills on assessing what you are monitoring and alerting you'll see things drop off while others will increase within your environment.

 

          Don't be shy to ask questions like, why is this important?  I seen this article on an attack, how can we be alerted in the future if this happens to us?  Some of the best monitoring I've seen is due to looking through THWACK and reading articles on what's going on in mainstream.  Bring this knowledge to your monitoring environment and begin crafting an awesome arsenal against, well, the WORLD.

 

HTH

~Dez~

  

MEME.jpg

Blog based on my "knee jerk" response to an article on an NSA breach

 

                So when you first read this article, you will notice that there are groups of hackers that are auctioning off exploits of devices.  May seem like no big deal but think about this. You have a group of people that are preying on your first line of defense and profiting on making these exploits available.  Irritation set to the highest level for one simple reason. NOT EVERYONE HAS A SECURITY TEAM. Ok, now that I feel better to commence the discussion on how they did this and why you may be concerned.

 

                Exploiting firewalls, you are now placing into the world factory defaults and settings that people may overlook or not think about when protecting your network.  Creating a gateway for script kitties and ill-willed individuals to try now and do harm just because the day ends in “Y”.  An example of why I constantly preach about compliance reports and their ability to help you protect your network and not forget the little things.

 

Some of the vulnerabilities listed were things like:

Buffer overflow in OpenLDAP

SNMP exploits on devices

Scripting advisement to gain more havoc

And much more…

 

So how do we guard against these untimely and devastating breaches?  One answer, stop ignoring security needs.  There are several free resources that help you protect yourself.  I realize a lot of people may or may not know these so I thought I would put together a few.

 

Common Vulnerabilities and Exposures

https://cve.mitre.org/

National Vulnerability Database

https://web.nvd.nist.gov

 

                If you read any of my NCM blogs, you would know that it has a firmware vulnerability data. Checking the NIST and advises you of security holes on your Cisco devices. Not a “catch-all” by any means but helps you to be aware and proactively having security checks every day by default.  Then as always there are compliance reports with even federal compliance reports right out of the box. Allowing you to lean on what others have created to ensure that you are crossing your T’s and dotting your I’s within your security needs.

 

                These are all ways we can try to use products to help us every day and have a direction to head in instead of ignoring or just simply put don’t make the time to address.  Monitoring and management software needs to be an everyday defensive tool.  To help offer guidance with your security needs and allowing you to work on security today and tomorrow.  Security teams can lean on monitoring\management solutions.  It’s not just for people that are lacking the funding for a security team it’s for everyone to stand together and help stand up to people exploiting for hire.

 

                Circling back to my last opinion on this article.  For hire, exploits are just as bad as hackers with ransomware.  These were merely saying “hey, pay me and I’ll tell you how you can do some damage” where ransomware is more “Hey, I encrypted or stole your data give me $$$ to (maybe) get it back.”  Is there a difference in the level of punishment if ever caught? I think there is not and we need to have better ways to prosecute and track down these criminals.  What’s your thoughts?  I’m always open to opinions and love hearing all of your comments!

 

~Dez~

Follow me on Twitter @dez_sayz

thwack2.jpeg

 

How to use network configuration, change, and compliance management (NCCCM) and other monitoring software in response to an actual security breach.

 

If you have not read part one, I would suggest that you give that an overview, so you can understand fully how and why this comes into play. For those that are ready for part two, welcome back!  I'll attempt to share some assessments of an internal sabotage and how to use things like monitoring and management software to see and recovery.  The best way to respond is by thinking ahead, having clear steps to prevent, and halt further damage.

 

Today, we are going to dive into a couple of scenarios, and directly assess ways to be alerted to and address situations that may be taking place within your organization.  Now, should we all live like we have a monkey on our back/shoulder?  No, but it doesn't hurt to have a little healthy "skepticism" about unusual things that are happening around you.  Being aware of your surroundings allows you to fight back and take back control of hiccups along the way.

 

 

Internal Planning Possible Sabotage:

Things to look for visually as well as with monitoring and management software.

 

  • Unusual behavior (after a confrontation or write-up has happened) - thank you sparda963 I forgot to place when to look for this
    • This can be obviously aggressive, but the one often overlooked is "overly" nice and helpful.
      • Yes, this sound condescending and I understand that concern but think of this as out of character.  They now want to help higher levels with mission critical information or configurations.  They want to "watch" you command line interface to a device.  They are "contributing" to get to know where key points are.  These are things that are outside of their scope.
    • Aggressive well the writing is on the wall at that point and if secretive comes into play then watch out and plan accordingly.
    • Use Real-time change notifications, approval systems, and compliance to help you see changes made, and users added to devices of monitoring management software.
      • Make sure that you have a script to remove access to devices ahead of time.  One that you can fill in the blank for the user ID and take permissions away quickly.
      • Verify you have alerts set up to notify you with quick access to the devices through a management software so you can cancel access levels and revert changes quickly.
  • Logon's found in unusual servers by said person
    • Use a Log Event Monitor to help you be alerted with strange behavior to login attempts and places.
    • Know your monitoring software and have quick pages to deny access to accounts quickly
  • New Users
    • Use a Log Event Monitor to alert you to new account creations.  You need to know when these were created and had a trail on these to remove.
  • Job creation for mass configuration changes
    • Verify through an approval system all changes on your network.  An excellent way to do this is with an NCCCM product and enable the approval system to be fully active.  You will want at least a 2 level approval system to help prevent issues and possible changes.
    • Real-time change notification with segmented emails for critical devices. 
    • Backups to be quickly accessible and found in multiple locations to ensure access during a breach.

 

Internal Execution of Sabotage:

Things to do if you find yourself under attack

(Network Side)

  • First things first
    • Log Event Monitoring - should be alerting you to access violations, additions of accounts, or deleting of accounts
    • TACACS - should be enabled and in full use for auditing within your monitoring and management software choices
    • Real-Time change notifications should be sending emails immediately to the correct people with an escalation of higher up network engineers on your team.
  • Now to fight back!
    • If they are opening firewalls to gain access you need to shut these down and stop traffic immediately.  You will need to have a plan on a script for a shut all or use something like Firewall Security Manager or Network Configuration Manager to implement commands from a stored location.
      • Allows time to figure out the user and what is going on while you can have the floodgate closed.
      • Addressed in a security protocol to enable you to have this authority.  Saving you and your company a lot of money when you are trying to prevent a massive break-in.
    • If they are deleting router configs
      • Real-time change notification (RTN) alerts should be sent out to you to bring you up to speed.
        • Use a script to deny access to the user that made the change shown in the RTN email.
        • Revert configurations from within your NCCCM software and get these back online
      • Verify users that have access
        • Use a compliance report to check access levels and remove where needed.
        • CONTINUE to monitor these reports
      • Check you Approval system
        • Verify who has access
        • Change passwords to all monitoring and management software logins.
          • I have had a customer that would set these up to one password for all that he would create if in crisis.  Allowing a quick shutdown of software usage to gain control when an attack was ensuing.
    • Verify critical application status
      • Log event monitor - check logs to see if access has been happening outside of usual
      • NetPath or something similar for pathways to check accessibility or changes
      • NCCCM - Verify all changes that have occurred within the past seven days minimum as this could only be the first wave of intrusion.
      • Network performance monitor to verify any malware or trojans that could be lingering and sending data on your network.
        • Volumes filling up and being alerted to this
        • Interface utilization skyrocketing
        • NetFlow monitor showcasing high amounts of unusual traffic or NO traffic history is essential here.

 

Security gut check:

Things to go over with yourself and team to make sure your security and plans for recovery are current.

 

Pre-Assessment

  • understand and know what is critical information within your organization
  • Where are your system boundaries
  • Pinpoint your security documentation

Assessment

  • Setup a meeting with your team over the above pre-assessment
  • Review your security information
  • Practice scenarios that "could" happen within your networks
  • Setup session controls
  • Verify maintenance plans
  • Ensure mapping of your critical networking connections with critical applications
  • Ensure your policies are relevant today as they were when first created
  • Verify entry points of concerns
    • Internal/External
  • System and Network Exposures

 

Team Analysis

  • Where are your vulnerabilities?
  • What are your Countermeasures?
  • What is the impact if breached?
  • Who can segment and take on sections of security recommendations?

 

Final

  • Implement new security plans as defined and found above.
  • Set up a meeting review for at least three months later to make sure all vulnerabilities are known and addressed.
  • Verify that the plan is accessible for your team to review so they are aware of actions to take.
  • Sign an agreement within your team to follow these protocols.

 

 

Well, that is a lot to cover, whef!  Once again everyone's networks and infrastructures are different.  You and I understand that.  The main point is how to use tools to help you stay ahead and be able to fight back with minimal damage.  Having a recovery plan and consistently updating these to new vulnerabilities is vital to stay ahead.  You can shift these and use for outside attacks as well.  Security is a fluid dance and ever changing so don't be stuck sitting on the outside looking in. 

 

 

Thank you,

~Dez~

thwack.jpg

I BEAT THEM TO FIRING ME! (Part Two) Fight Back

Why network configuration, change and compliance management (NCCCM) is a must

Inspired by former Citibank employee sentencing

(Part Two)

 

We've all heard horror stories about the disgruntled employee who pillages the office supply closet and leaves the building waving an obscene gesture, security badge skittering across the parking lot in his wake. Rage-quit is a thing, folks, and it's perfectly reasonable to be afraid that someone with high-level access, someone who could make changes to a network, might do so if they get mad enough. This happens more often than anyone would like to think about, and it's something that needs to be addressed in every organization. I felt like we should talk about this and discuss ways to help control and slow the damage of said employees and their bad will. Bottom line: we need to be aware of these situations and have a plan for recovery when things like this happen.

 

 

The gist of the story is simple: there was an employee who wiped out critical network configurations to about 90% of his former company's infrastructure.  Monday he was sentenced on charges of criminal vandalism. So, I realize the article above is technically in the past, but it brings up a great starter conversation about how IT organizations can stop criminal vandalism by actually using NCCCM products to protect ourselves and others from any type of disastrous events. Sometimes you need that brief pause or slight inconvenience to help you think straight and not go over the edge. This post can also help keep your butt out of, well, jail .

 

Today, we are going to talk about some of the risks of not having NCCCM software:

 

 

  1. Real-time change notification not enabled.
    • There is no tracking, idea, or reference to when changes are being made via maintenance plans, change requests, or malicious intent.
      • Being able to see network changes and know the timing helps you to be proactive, and gives you immediate remediation action for your network.
    • Who's on first base, and did someone slide in to home base?
      • When you have more than a couple of network engineers, documentation can be lacking and, well, you're busy, right? Being able to track when changes happen and who made them allows you to find and discover who, when, and what was changed, even when it's a week later.
      • Being able to compare the change that was made to existing is key to correlating issues after a change was made. All of a sudden, traffic is not flowing, or it's restricted, and you find out it was an error in the config change.
    • Someone is on your network changing your critical devices and wiping them clean.
      • Receive alerts so you don't find this type of information out when it's too late. Be able to log in, and after receiving the alert, restore to previous config.
  2. Approval process not in use.
    • No change auditing.
      • Being able to make changes without approval or a process sets you up for human error or worse: attacks.
      • Implementing an approval process allows you to have an auditing system that shows that more than one person approved a change.
      • Use this with real-time change notification to see if anyone outside your team is making changes. Either allow them into your NCCCM, or delete or lock out their login info to the devices.
    • No one can verify that you are making the change, or even what that change was.
      • When you have a larger team, you delegate changes or areas of functionality. Having an approval process verifies that the correct changes are being made. That gives you an extra set of eyes on the changes that are being made, which adds another level of detection to human error.
    • One person has complete access to your devices at a control level.
      • When you give people straight access to network devices there is a single point of failure. Taking an extra step creates a safe zone of recognition, training, and the ability to track changes and implementations on your network.
  3. Advanced change alert not enabled.
    • Not having an escalation alert set up can leave you with no configurations on your devices when you come into work the next day.
      • Set up escalation alerts based on more than one action.
        • Create a mass change alert if X amount of syslog changes happen within five minutes: Alert Manager NOW.
        • Mute these when implementing maintenance plans. more info by adatole
  4. Backups you are saving to your desktop or network drive (when you remember).
    • If a crisis happens, the great news is that network devices just need to be told what to do. But if you are like me and don't remember every line of code for hundreds of devices, then you better implement a backup system NOW.
      • If you have backups being stored, recovery is a click away with an NCCCM.
      • Compare starting to running to make sure a reboot won't cancel your changes.
      • Verify you have backups in secure locations so downtime is minimized and quickly averted.
        • I generally implement server side and network share drive backups. Make your server accessible with security verification lockdown in case someone tries to delete the backups (this happens because they don't want you to recover).
  5. Recovery procedures not in place.
    • Can your team recover from an emergency without you being on site?
      • Have a plan and practice with your team. You have to have a plan to be able to recover from maintenance plans gone wrong all the way to disaster recovery.  This takes practice, and should be something the whole team discusses so that you are better engaged. It helps to have an open mind to see how others may offer solutions to each potential problem suggested.
    • Setup an automatic password change template to be easily used in case of a potential issue within or outside your organization.
    • Use your NCCCM to monitor your configurations for potential issues or open back doors within your network.
      • Sometimes people will start allowing access within your network watching your configurations with a compliance reporting service allows you to detect and remediate quickly to stop these types of security breaches in their tracks.

 

If your curious on setup check this out:More info Security and SolarWinds NCM

 

Stay tuned for part two, I'll showcase how each one of these can be used in response to security!

 

Now that is a few things you should be able to use within any NCCCM software package.  This should also be something you revisit consistently to reevaluate and assess your situation and how to better protect yourself.

Let's dive into the mindset and standard methodologies around the security aspect:

 

This isn't just for technology this is in general things to be aware of and to implement on your own.  The ability to look at these with a non-judging eye and see them as just ways to hold off malicious attacks or ill will.

 

  1. There needs to be a clear exit strategy for anyone that is going to be fired or removed from a position with potential harm.
    • But he is such a nice guy?  Nice guys can turn bad.
    • When this information is being circulated you need to do what's best for your career as well as the company you work for and go on the defense.
      • Bring in specialized help organizations that can come in assess and prevent issues before they are terminated or moved
      • Make sure you verify all traffic and location they were involved in
        • Any passwords etc that were globally known NEEDS CHANGED NOW not LATER
        • Check all management software and pull rights to view only in the remainder days then delete access immediately after termination
        • Verify all company technology is accounted for (Accounting and inventory within your NCCCM is vital to maintain diligence on awareness of property and access to your network)
  2. Monitoring of team
    • Some may not be happy with a decision to terminate an employee and feel betrayed
    • Monitor their access and increase awareness to their actions
      • If you see them logging in to more routers and switches than ever before might setup a meeting...
      • See them going outside of their side and digging into things they should not, meeting time
      • Awareness is key and an approval process and change detection is key to preventing damage
  3. Security policies
    • You're only as good as the policy in place
      • Dig into your policies and make sure they are current and relevant
      • If you seriously have things like "If they call from desk phone reset password over the phone" type of security measures please REVISIT these.
        • Re-read that last statement
    • Make sure your team is signing acknowledgement of what they can and cannot do
      • Easier to prosecute when they have signed and agreed
    • Verify your security policies to your network devices
      • NCCCM compliance reporting setup for your needs is a great way to stay ahead of these items
      • You can find back doors on your network that people have setup to go around security policies this way. 

 

     I cannot obviously solve every issue, but at least help to point you into some good directions and processes.  If any of you want to jump in and add to this, please do I'm always interested in other people's methods of security.  The main point is to be aware of these situations, have a plan and recover when things like this happen.

 

Thank you,

 

~Dez~

 

Follow me on Twitter:

@Dez_Sayz

If you’re in management, you may not understand the effects of changes on your network.  However, if you’re the network engineer you know exactly the effects and ramifications that come with a change on your network.  The slightest change can literally cause an outage.

 

So what’s the big deal with software companies that want you to buy Network Configuration Change Management (NCCM) software?  Well I know personally that a few of you have been in this exact position and on both sides of this ball.  As a manager you want to have a seamless network and keep down costs.  As the network engineer you want to be able to have a smooth running network and a happy manager.

 

What is the happy medium here?  When are too many software tools or too many diagrams on walls and an over-abundance of saved test files enough to know software is required to actually manage all of this?

 

SolarWinds offers a Network Configuration-Change Management package. Does this mean it’s the best?  No, as that is in the eye of the beholder and user. Does this mean that it is manageable and can save me time and my manager money?  You’re darn rightit can do both very easily!

 

Yes, there are other software tools that do all about the same thing with little differences along the way.  Just like I like thin pancakes and you may like fluffy thick pancakesin the end they are still pancakes.

 

Now to know what a good NCCM is regardless of the name across it, let’s go over the top 6 reasons to have such software.

 

  1. Making changes because you were told to…
    1. You want to be able to know if someone is in fact making changes immediately and have a way to revert changes if needed.  NCCM software allows you to do this and consistently backs up your devices in case such changes are incorrect and provides a complete barebones backup if needed for a new device.
  2. Scheduled device changes
    1. Planning IOS upgrades, change in ACL lists, SNMP passwords, or many other items on your daily tasks.  Having a program that will allow you to monitor and roll out these changes saves time and show results quickly.
  3. A second pair of eyes
    1. It’s good to have an approval system in place so that scripting and changes receive a second look before deployment.  This helps prevent outages and mistakes, and definitely is valuable when your network has service level agreements and high availability needs.
  4. BACK UPS…BACK UPS!
    1. I cannot say this enough…if you do not have regular backups of your system that are easily retrievable, you do not have a fully reliable network.  PERIOD. Backing up to your local machine is not acceptable…You know who you are
  5. Automation of the tasks you might rather forget...
    1. Being able to detect issues within your configuration through compliance reporting, real-time change detection, scheduled IOS upgrades, inventory, and many more automated tasks. This allows you to focus on the integrity and availability of your network.
  6. Security
    1. If you have certain required security measures within your configurations, then you need compliance reporting.  With NCCM software, you can schedule a report or run it manually and print out that your ‘state of compliance’ within seconds instead of per device.

 

Well there are a few valuable reasons to at least consider this type of software.  If you have any other thoughts, feel free to drop me a line! Add to my list or take away, I’m a pretty open mined individual.

 

If you’re looking for more information this has a solid outlook on NCCM and businesses.

Filter Blog

By date: By tag: