cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Shields Down Conversation Number Three

Product Manager
Product Manager

I know, I'm a day late and quite possibly 37 cents short for my coffee this morning, so let's jump in, shall we?

Let's start with the Equifax breach. This came up in the Shields Down Conversation Number Two, so, I thought I would invite some of my friends from our security products to join me to discuss the breach from a few different angles.

My take will be from a business strategy (or lack of) standpoint. Roughly 143 million people had their personal data exposed because Equifax did not properly execute a simple patching plan. Seriously?

Is this blog series live and viewable? I am not the only person who implements patching, monitoring, log and event management in my environments. This is common knowledge. What I don't get is the why. Why, for the love of everything holy, do businesses not follow these basic practices?

CIxO or CXOs do not implement these practices. However, it is their duty (to their company and their core values) to put the right people in place who will ensure that security measures are being carried out.

Think about that for a moment and then know that there was a patch produced for the vulnerability that Equifax failed to remediate in March. This breach happened, as we all know, in mid-May. Where is the validation? Where was the plan? Where is the ticketing system tracking the maintenance that should've been completed on their systems? There are so many questions, especially since this happened in an enterprise organization, not some small shop somewhere.

Now, let's take this another step further. Equifax dropped another juicy nugget of information of another breach in March. Don't worry, though. It was an entirely different attack. However, the incredible part is that some of the upper-level folks were able to sell their stock. That makes my heart happy, you know, to know that they had the time to sell their stock before they released information on being breached. Hat's off to them for that, right?

Then, another company decided they needed to market and sell credit monitoring (for a reduced fee, that just so happens to use EQUIFAX SERVICES) to the individuals who were now at a high(er) risk of identity theft and credit fraud. I'm still blown away by this.

Okay. Deep breath. Whooooo.

I was recently informed that when you have third-party software, patching is limited and that organization's SLAs for application uptime don't allow patching on some of their servers. I hear you! I am a big believer that some patching servers can cause software to stop working or result in downtime. However, this is where you have to implement a lab and test patching. You should check your patching regardless to make sure you are not causing issues with your environment in the first place. 

I will implement patching on test servers usually on a Friday, and then I will verify the status of my applications on the server.

I will also go through my security checks to validate that no new holes or revert have happened before I implement in production within two weeks. 

Now let's bring this back to the strategy at hand. When you are an enterprise corporation with large amounts of personal data belonging to your trusting customers (who are the very reason you are as large as you are), you better DARN WELL have a security plan that is overseen by more than one individual! Come on! This is not a small shop or even a business that could argue, "Who would want our customer data?" We're talking about Equifax, a company that holds data about plenty of consumers who happen to have great credit. Equifax is figuratively a lavish buffet for hackers.

The C-level of this company should have kept a close eye on the security measures being taken by the organization, including patching, SQL monitoring, log, events, and traffic monitoring. They should have known there were unpatched servers. The only thing I think they could have argued was the common refrain, "We cannot afford downtime for patching." But still. 

Your CxO or CIxO has to be your IT champion! They have to go nose to nose with their peers to make sure their properly and thoroughly designed security plans get implemented 100%. They hire the people to carry out such plans, and it is their responsibility to ensure that it gets done and isn't blocked at any level.

Enough venting, for the moment. Now I'd like to bring in some of my friends for their take on this Equifax nightmare that is STILL unfolding! Welcome joshberman, just one of my awesome friends here at SolarWinds, who always offers up great security ideas and thoughts.

Dez summed up things nicely in her comments above, but let's go back to the origins of this breach and explore the timeline of events to illustrate a few points.

  • March 6th: the exploited vulnerability, CVE-2017-5638, became public
  • March 7th: Security analysts began seeing attacks propagate that were designed to exploit this flaw
  • Mid-May: Equifax tracked the date of compromise back to this window of time
  • July 29th: the date Equifax discovered a breach had occurred

Had a proper patch management strategy been set in place and backed by the right patch management software to enable the patching of third-party applications, it is likely that Equifax might not have succumbed to such a devastating attack. This applies even if testing had been factored into the timelines, just as Dez recommends. "Patch early, patch often" certainly applies in this scenario, given the voracious speed of hackers to leverage newly discovered vulnerabilities as a means to their end. Once all is said and done, if there is one takeaway here it is that patching as a baseline IT security practice, is and will forever be a must. Beyond the obvious chink in Equifax's armor, there is a multitude of other means by which they could have thwarted this attack, or at least minimized its impact.

That's fantastic information, Josh. I appreciate your thoughts. 

I also asked mandevil (Robert) for his thoughts on the topic. He was on vacation, but he returned early to knock out some pertinent thoughts for me! Much appreciated, Robert!

Thanks, Dez. "We've had a breach and data has been obtained by entities outside of this company."

Imagine being the one responsible for maintaining a good security posture, and the sinking feeling you had when these words were spoken. If this is you, or even if you are tangentially involved in security, I hope this portion of this post helps you understand the importance of securing data at rest as it pertains to databases.

Securing data in your database

The only place data can't be encrypted is when it is in cache (memory). While data is at rest (on disk) or in flight (on the wire), it can and should be encrypted if it is deemed sensitive. This section will focus on encrypting data at rest. There are a couple different ways to encrypt data at rest when it is contained within a database. Many major database vendors like Microsoft (SQL Server) and Oracle provide a method of encrypting called Transparent Data Encryption (TDE). This allows you to encrypt the data in the files at the database, table space, or column level depending on the vendor. Encryption is implemented using certificates, keys, and strong algorithms and ciphers.

Links for more detail on vendor TDE description and implementation:

SQL Server TDE

Oracle TDE

Data encryption can also be implemented using an appliance. This would be a solution if you would want to encrypt data but the database vendor doesn't offer a solution or licensing structures change with the usage of their encryption. You may also have data outside of a database that you'd want to encrypt that would make this option more attractive (think of log files that may contain sensitive data). I won't go into details about different offers out there, but I have researched several of these appliances and many appear to be highly securitized (strong algorithms and ciphers). Your storage array vendor(s) may also have solutions available.

What does this mean and how does it help?

Specifically, in the case of Equifax, storage level hacks do not appear to have been employed, but there are many occurrences where storage was the target. By securing your data at rest on your storage tier, it can prevent any storage level hacks from obtaining any useful data. Keep in mind that even large database vendors have vulnerabilities that can be exploited by capturing data in cache. Encrypting data at the storage level will not help mitigate this.

What you should know

Does implementing TDE impact performance? There is overhead associated with encrypting data at rest because the data needs to be decrypted when read from disk into cache. That will take additional CPU cycles and a bit more time. However, unless you are CPU-constrained, the impact should not be noticeable to end-users. It should be noted that index usage is not affected by TDE. Bottom line is if the data is sensitive enough that the statement at the top of this section gets you thinking along the lines of a resume-generating event, the negligible overhead impact of implementing encryption should not be a deterrent from its use. However, don't encrypt more than is needed. Understand any compliance policies that govern your business (PCI, HIPAA, SOX, etc.).

Now to wrap this all up.

When we think of breaches, especially those involving highly sensitive data or data that falls under the scope of regulatory compliance, SIEM solutions certainly come to mind. This software performs a series of critical functions to support defense-in-depth strategies. In the case of Equifax, their most notable influence appears to be their attempt to minimize the time of detection with either the compromise or the breach itself. On one hand, they support the monitoring and alerting of anomalies on the network that could indicate a compromise. On the other, they can signal the exfiltration of data – the actual event of the breach – by monitoring traffic on endpoints and bringing to the foreground spikes in outbound traffic, which, depending on the details, may otherwise go unnoticed. I'm not prepared to make the assumption that Equifax was lacking such a solution, but given this timeline of events and their lag in response, it begs the question.

As always, thank you all for reading and keep up these excellent conversations.

73 Comments
Level 10

Wow, wish more people thought like you do in regards to Gartner and their reports. Here where I am there are a couple of people that look at it as if it were the bible and that just sucks. Mainly because they can't see anything else or for that matter think for them selves. Just voicing some frustrations, but I do agree with you wholly.

Yes, my organization used to believe Gartner's Magic Quadrant listed the only products to be evaluated.  I was surprised to get the impression Gartner may not do any testing at all, and that everything in their Magic Quadrant simply reflects glowing marks someone input into their survey.  With no testing of the product, no checking up on the veracity or validity of the remarks . . .  what conclusion is one to make?

It's too bad, because our industry REALLY needs the equivalent of an independent and trustworthy third-party that tests and evaluates products, much like Consumer Reports.  Which is what I'd assumed Gartner was.

Level 13

Yea, During Command and General Staff College we studied Starship Trooper for leadership examples that went wrong.

MVP
MVP

I have always questioned Gartners evaluations...  They never seem to pick up on the peculiarities of a product or it's weaknesses.  They seem to scrape the glossies and build their vision of a product segment from that instead of people in that industry doing decent evaluations and scoring...

Things that require many multiple manual actions are never brought up.  Different mechanisms of action, the fact that an automatic action fired off as a result of an event only returns a return code to the tool that it fired off the action instead of the return code of the action that was fired.

Those are things that matter and make a difference.

Level 21

Jfrazier​ I would agree with this completely!

SIEM tools are a great example; Splunk always ends up on the top of their list.  While I don't think Splunk is a bad tool, they don't seem to take any penalty in the ratings for the fact that it is super expensive or the fact that it pretty much doesn't do anything out of the box, you have to spend hundreds of hours configuring every little thing yo want it to do.  I even had an auditor tell me that Splunk is the SIEM that they most often find configured incorrectly when they do audits.  This is an example of where it feels like the Gartner system may be a bit skewed and is likely ignoring important details and maybe not putting enough emphasis on others.

What makes this even more concerning is the fact that a lot of folks out there put a lot of weight on what Gartner says and will make significant purchasing decisions based on the Gartner data.

I'd no idea that book/movie was a resource.

Of course, being fiction, it's not as relevant as using real-world failures as examples.  But it's probably a LOT easier to analyze a book/movie than a disaster in which many political/military types may have been trying to use CYA processes to cover up or hide their bad decisions.

Level 9

Unbelievable

MVP
MVP

Nice article comments/replies

Why do businesses not follow these basic practices you ask? Well...

In my experiences they generally do. But there is usually this one app, or appliance, or piece of code, that is either: old, obsolete, the vendor is out of business, the SME's are no longer employed, it's orphaned, etc. and no one wants to touch it for fear of it breaking beyond the point or remediation. So everything else around is updated/upgraded, patched, replaced, etc.

What does it take to get rid of that albatross? A Champion! Not a C-level or a VP. Usually a manager or below who will rally troops. Who will kick budds, knock down doors, and point fingers followed up with sentences that start off with, "You will..." Because if none of this done all of IT will play the "Not It!" game muttering, "Something should be done about that." and letting that albatross sit until something really bad happens.

I've seen it happen dozens of times. Be a Champion!

Yes.  This!

I've seen all those excuses far too many times.

MVP
MVP

Well put Dez​  !

It will be interesting to see that becomes of the C-level and below management that failed to ensure things were in place.

It will also be interesting to see what the SEC says and does about the stock sales...everything was way to suspect.

It will also be interesting in what this does for the credit industry as a whole...not to mention IRS and taxes since the new big thing is fraudulent tax refunds using other peoples identities now that much of the required info is now available.

Oh, I'm so jaded.  C-Level and higher level management will either keep their jobs and get raises, or leave under golden parachutes, only to take over at positions of higher responsibility at other companies.

No one will see jail time, few will even see court time.

And the customers and investors will lose their security and their investments, and be victims of identify theft and lawsuits that drag the company's value down to zilch.

The company will go under, and clients will move to new companies that will do the same thing to them.

MVP
MVP

This is what I've been trying to tell people... splunk is great if you know regex like the back of your hand and have the time to create all the queries... if you aren't a regex expert it's a non trivial thing setting it all up.

MVP
MVP

the funny part is that the  people who may not face court time and such are possibly just as exposed...

Level 11

Do we continually have to accept companies failing to secure our data.  I haven't seen these companies held accountable, other than a years free credit monitoring.  What a crock.  It is about time companies are held to accountable by financially having to pay each individual who has had their data compromised.

I'm no splunk expert, but I have access to one one staff.  I've continued to ask his advice, and I record his queries for learning and future re-use.

In today's world of GUI-Everything, I imagine a Splunk interface that was about ten times more intuitive for our generic Windows users, with drop-downs and selections via radio buttons to select queries.

Level 12

I just read another story on cnet about some of the things that were said during equifaxes ex-ceo beating on capital hill today. Equifax ex-CEO blames breach on one person and a bad scanner - CNET

Couple of things that popped out at me immediately in the story:

"The company, which staffs 9,900 employees, only had one person in charge of its patching process, Smith said."

That means they had at least 9,900 computers, not to mention the servers on the backend to support all the programs and apps, and one single person in charge of patch management. You can do a lot with automation, but you can't do that much on your own, it's impossible. Basically they didn't take patch management seriously at all. Also this person I feel bad for them because you know they will be made a scapegoat for this, if they haven't been already.

"Rep. Walden bluntly noted that it would be difficult to stop cyberattacks from human errors like the one Equifax suffered. 'I don't think we can pass a law that fixes stupid,' Walden said."

Someone finally had the balls to say what most of us are probably thinking in the IT world.

"Both human deployment, and the scanning did not work. But the protocol was followed"

"Smith blamed a faulty scanner for not flagging the vulnerability on March 15, and a single Equifax staffer responsible for mishandling patches on March 9."

Basically this is what happens if you trust automation without checking it for accuracy. You wont know it is not working properly if you don't actually look at it. Also I guess the protocol was followed, so you know, this thingy over here didn't blink red so we were in compliance.

Skeptical Cat wonders if they're just throwing one rider to the wolves, or whether the problem is actually more systemic and part of the nature of their IT deployment and management style.

It's easy to point at just one poor soul and blame them for everything, fire them, ruin their reputation, and then proceed with business as usual until the next time something makes it into the headlines.  And then another ceremonial lamb will be sacrificed for the media, and so on, until the company's C-Level are required to pay the damages from their own pockets, and spend time in prison.

It's one thing to point a finger for the public and media to follow, and to say "It's THAT person's fault--They screwed up!"

It's another to ask why that person didn't have the training and the staff and the tools and the follow-up to ensure their tasks would always be successful.  That's what corporate's for.   Corporate is the source that fell short of the mark.

And it's who should take the punishment.

And why not?  Since our government has officially given corporations legal status equal to a person, send the members of that corporation to jail.  Or take away their corporate "personhood" and just focus on the leaders who ensured failure was inevitable by not funding training and staffing and confirming tasks succeeded.

MVP
MVP

Likely they found a scapegoat so they can keep their golden parachute and stock profits....

Again, any automated process requires checking up on it to be sure it is doing what it should.

If only one person was doing patching for a company that size, then they didn't have enough boots on the ground...

Level 12

I think the telling line in there was that procedure was followed. Meaning everyone did what they were supposed to, or in this case the one person. I feel they are likely trying to make him a scapegoat, but it would be hard to blame this on a single person like that. Then again it happens all the time sadly.

MVP
MVP

very sad and unfortunately very true.

Level 10

I just saw this and can't believe it. This is the governments response to the Equifax Breach.

"Senators expressed bewilderment Wednesday that credit reporting company Equifax, under siege after a data breach affecting more than 145 million people, has received a $7.25 million contract with the IRS to provide taxpayer and personal identity verification services."

"Why in the world should you get a no-bid contract right now?" Sen. Ben Sasse, R-Neb., asked former Equifax CEO Richard Smith at a Senate Banking, Housing and Urban Affairs Committee hearing.

Sasse's indignation was soon topped by Sen. John Kennedy, R-La., who said, "You realize to many Americans right now, that looks like we're giving Lindsay Lohan the keys to the mini-bar."

"I understand your point," Smith said in response to Kennedy's observation and reference to the actress who has struggled with drugs and alcohol.

"Why?"

  • Pork barrel spending
  • Cronyism
  • Ignorance of the problems Equifax has (hard to imagine no one's heard about this yet)
  • Politics
  • Process over intelligence (which I also like to refer to as "The Death of Common Sense").
  • The contract is needed, a process and clock are in play, there's no way for the GAO and / or the pertinent government office(s) to get a replacement product/contract in the necessary time

pastedImage_12.png

I agree, it's throwing good money away after bad . . .