Skip navigation

I’ve discussed the idea that performance troubleshooting is a multi-dimensional problem and that virtualization adds more dimensions. Much of the time it is sufficient to look at the layers independently. The cause of a performance problem may be obvious in an undersized VM or an overloaded vSphere cluster. But sometimes you need to correlate the performance metrics across multiple layers. Worst of all is when the problem is intermittent. Apparently random application slowdowns are the worst to troubleshoot. The few times that I have needed to do this correlation I have always had a sinking feeling. I know that I am going to end up gathering a lot of performance logs from different tools. Then I am going to need to identify the metrics that are important and usually graph these together. There is a sinking feeling when I know I need to get the data from Windows Perfmon, the vSphere client, the SAN, and maybe a network monitor into a single set of graphs.

 

My go-to tool for consolidating all this data is still Microsoft Excel, mostly because I have a heap of CSV files and want a set of graphs. Consolidating this data has a few challenges. The first is getting a consistent start and finish times for the sample data. The CSV files are generated from separate tools and the time stamps may even be in different time zones. Usually looking at one or two simple graphs identifies the problem time window. Once we know when to look at we can trim each CSV file for the time range we want. Then there are challenges with getting consistent intervals for the graphs. Some tools log every minutes and others every 20 minutes. On occasion, I have had to re-sample the data to the lowest time resolution just to get everything on one graph. That graph also needs to have sensible scales, meaning applying scaling to the CSV values before we graph them. I’m reminded how much I hate having to do this kind of work and how much it seems like something that should be automated.

 

Usually, when I’m doing this I am an external consultant brought in to deal with a high visibility issue. Senior management is watching closely and demanding answers. Usually, I know the answer early and spend hours putting together the graph that proves the answer. If the client had a good set of data center monitoring tools and well-trained staff, they would not need me. It troubles me how few organizations spend the time and effort in getting value out of monitoring tools.

 

I have been building this picture of the nightmare of complex performance troubleshooting for a reason. Some of you have guessed the reason, PerfStack will be a great tool to avoid exactly this problem. Seeing an early demo of PerfStack triggered memories. Not good memories.

Ensure your IoT devices are secure, or face the consequences. That’s the message being sent to some hardware manufacturers by the Federal Trade Commission. In the aftermath of the ever-increasing number of attacks perpetrated by compromised IoT devices like routers and cameras, the Federal Trade Commission’s Bureau of Consumer Protection has targeted companies such as TrendNet, Asus, and more recently, D-Link.

 

TRENDnet

Back in 2013, the FTC settled its very first action against a manufacturer of IP-enabled consumer products, TRENDnet. TRENDnet’s SecurView cameras were widely used by consumers for a wide range of purposes including home security and baby monitors. By their product name alone, these products were seemingly marketed as “secure." The FTC accused TRENDnet of a number of issues, including:

 

  • Failing to use reasonable security to design and test its software
  • Failing to secure camera passwords
  • Transmitting user login credentials in the clear
  • Storing consumers’ login information in clear, readable text on their mobile devices

 

In January of 2012, a hacker exposed these flaws and made them public, resulting in almost 700 live feeds being posted and freely available on the internet. These included babies sleeping in their cribs.

 

ASUS

Once again the FTC fired a shot across the bow at manufacturers of consumer IoT devices when they leveled a complaint against ASUSTek Computer, Inc. This time, the security of their routers was questioned. ASUS had marketed their consumer line of routers with claims they would “protect computers from any unauthorized access, hacking, and virus attacks” and “protect the local network against attacks from hackers.” However, the FTC found several flaws in the ASUS products, including:

 

  • Easily exploited security bugs in the router’s web-based control panel
  • Allowing consumers to set and retain the default login credentials on every router (admin/admin)
  • Vulnerable cloud storage options AiCloud and AiDisk that exposed consumers’ data and personal information to the internet.

 

In 2014, hackers used these and other vulnerabilities in ASUS routers to gain access to over 12,900 consumers’ storage devices.

 

D-Link

Now, in 2017, the FTC has targeted D-Link Corporation, a well-known manufacturer of consumer and SMB/SOHO networking products. This complaint alleges that D-Link has “failed to take reasonable steps to secure its routers and Internet Protocol (IP) cameras, potentially compromising sensitive consumer information including live video and audio feeds from D-Link IP cameras.”

The FTC complaint goes on to outline how D-Link has promoted the security of its devices with marketing and advertising citing “easy to secure” and “advanced network security," but outlines several issues:

 

  • Hard-coded login credentials (guest/guest) in D-Link camera software
  • Software vulnerable to command injection that could enable remote control of consumer devices
  • Mishandling of a private code signing key, which was openly available on a public website for six months
  • User login credentials store in clear, readable text on mobile device

 

The severity of an exposed and vulnerable router is amplified by the fact that these are a home networks’ primary means of defense. Once compromised, everything behind that router is then potentially exposed to the hacker, and the FTC emphasizes, could result in computers, smartphones, IP cameras, and IP-enabled appliances to be attacked as a result.

 

The DDoS Landscape

According to Akamai’s quarterly State of the Internet report, DDoS attacks continue to flourish and evolve as a primary means to attack both consumers and businesses. In Q4 2016, there was a 140% increase in attacks greater than 100Gbps, a 22 percent increase in reflection-based attacks, and a 6 percent increase in Layer 3 and 4 attacks. At the application layer, a 44% increase in SQLi attacks was observed over the same period. These examples are more evidence that these types of attacks are moving ever upwards in the stack.

 

Not surprisingly, the United States continues to be the largest source of these attacks, accounting for approximately 28 percent of the global web application attacks in Q4 2016. As IoT devices continue to proliferate at exponential rates, and companies like TREN,Dnet, ASUS and D-Link fail to secure them, these numbers may only increase.

 

There is hope however, that organizations like the FTC can send a strong message to device manufacturers in the upcoming months as they continue to identify and hold accountable the companies that fail to protect consumers, and the rest of us, from exposed and vulnerable devices.

 

Do you feel the FTC and FCC (or other government organizations) should be more or less involved in the enforcement of IoT security?

Devops, the process of creating code internally in an effort to streamline the functions of the administrative processes within the framework of the functions of the sysadmin, are still emerging within IT departments across the globe. These tasks have traditionally revolved around the mechanical functions the sysadmin has under their purview. However, another whole category of administration is now becoming far more vital in the role of the sysadmin, and that’s the deployment of applications and their components within the environment.

 

Application Development is taking on a big change. The utilization of methods like MicroServices, and containers, a relatively new paradigm about which I’ve spoken before, makes the deployment of these applications very different. Now, a SysAdmin must be more responsive to the needs of the business to get these bits of code and/or containers into production far more rapidly. As a result, the SysAdmin needs to have tools in place so as to to respond as precisely, actively, and consistently as possibly. The code for the applications is now being delivered so dynamically, that it now must be deployed, or repealed just as rapidly.

 

When I worked at VMware, I was part of a SDDC group who had the main goal of assisting the rollouts of massive deployments (SAP, JDEdwards, etc.) to an anywhere/anytime type model. This was DevOps in the extreme. Our expert code jockeys were tasked with writing custom code at each deployment. While this is vital to the goals of many organizations, but today the tools exist to do these tasks on a more elegant manner.

 

So, what tools would an administrator require to push out or repeal these applications in a potentially seamless manner? There are tools that will roll out applications, say from your VMware vCenter to whichever VMware infrastructure you have in your server farms, but also, there are ways to leverage that same VMware infrastructure to deploy outbound to AWS, Azure, or to hybrid, but non-VMware infrastructures. A great example is the wonderful Platform9, which exists as a separate panel within vCenter, and allows the admin to push out full deployments to wherever the management platform is deployed.

 

There are other tools, like Mezos which help to orchestrate Docker types of container deployments. This is the hope of administrators for Docker administration.

 

But, as of yet, the micro-services type of puzzle piece has yet to be solved. As a result, the sysadmin is currently under the gun for the same type of automation toolset. For today, and for the foreseeable future, DevOps holds the key. We need to not only deploy these parts and pieces to the appropriate places, but we also have to ensure that they get pushed to the correct location, that they’re tracked and that they can be pulled back should that be required. So, what key components are critical? Version tracking, lifecycle, permissions, and specific locations must be maintained.

I imagine that what we’ll be seeing in the near future are standardized toolsets that leverage orchestration elements for newer application paradigms. For the moment, we will need to rely on our own code to assist us in the management of the new ways in which applications are being built.

By Joe Kim, SolarWinds Chief Technology Officer

 

While it’s essential to have a website that is user-friendly, it’s equally important to make sure that the backend technologies that drive that site are working to deliver a fast and fluid performance. In short, good digital citizen engagement combines reliability, performance, and usability to help connect governments and citizens.

 

Assuming you’ve already developed a streamlined user interface (UI), it’s time to start centering your attention on the behind-the-scenes processes that will help you build and maintain that connection. Here are three strategies to help you achieve this ultimate ideal of form and function.

 

Closely monitor application performance

 

Slow or unresponsive applications can undermine federal, state, and local government’s efforts to use online solutions to connect with citizens. What’s the incentive for constituents to use their government’s digital platform if a site is slow and doesn’t easily or quickly lead to information that answers their questions? They might as well pick up the phone or (shudder) pay a visit to their local office.

 

Monitoring application performance is imperative to helping ensure that your digital platforms remain a go-to resource for citizens. Use application monitoring solutions to track and analyze performance and troubleshoot potential issues. The data you collect can help you identify the causes of problems, allowing you to address them quickly and, ideally, with minimal impact to site performance.

 

Log and manage events

 

The feds need to take care to plug any potential holes, and part of that effort entails logging and managing events. Events can range from a new user signing up to receive emails about local information, to potential intrusions or malware designed to collect personal information or compromise website performance.

 

Put steps in place to monitor any types of website events and requests, and this will help you identify, track, and respond to potential incidents before they do lasting damage. You can monitor and manage unauthorized users, failed login attempts, and other events, and even mitigate internal threats by changing user privileges.

 

Test site performance

 

Once application monitoring and log and event management processes are in place, continue to test your website’s performance to ensure that it delivers an optimal and user-friendly experience.

 

The goal is to identify any slow web services that may be impacting that experience. Use web performance monitoring solutions to identify infrastructure issues and various site elements that could be causing latency issues.

 

Your site should be tested from multiple devices, locations, and browsers to provide your users with a fast and reliable experience. These tests should be done proactively and on a regular basis to help ensure a consistently optimal performance that delivers on the promise of true citizen engagement.

 

Remember that as you strive to achieve that promise, it’s important to invest in the appropriate backend solutions and processes to power your efforts. They’re just as important as the surface details. Without them, you run the risk of having your site be just another pretty face that citizens may find wanting.

 

Find the full article on GovLoop.

Leon Adato

Trite Business Lies

Posted by Leon Adato Expert Feb 27, 2017

pantsonfire.jpg

When someone brings up the topic of lies at work, most of the time people think of the big stuff, like embezzlement, falsifying data, or fudging fantasy football stats. But there's a whole other category of lying that happens all the time. Not only are these lies tolerated, but through some combination of repetition, vehemence, and mind control, employees of all stripes have come to believe them as true.

 

Part of the reason for this is that the lies are small (and therefore insidious). They often pass themselves off as sage advice or commonly-understood truisms. At worst, they may appear to be clichés or business tropes. Many are hard to argue with unless you've thought it through.

 

Nevertheless, they are lies, and especially in this era of #AlternateFacts, deserve to be exposed as such.

 

So take a look below and see which of these phrases match up with your personal experience or reality. If you have stories, reactions, or your own additions, let us know about them in the comments below!

 

“Perception is reality”

No, it’s not. I perceive myself to be an AMAZING dancer. But 48 years of evidence (as well as testimony from my wife and children. ESPECIALLY my children!) indicate otherwise.

 

The truer statement is that perception is powerful, and can override facts. Knowing this is so does NOT mean we must kowtow to a requestor’s built-in perceptions or preconceived notions, but rather that we need to understand those perceptions and build compelling stories that bring the person to the actual truth.

 

“The customer is always right”

The truth is that customers can be, and often are, wrong. This can be due to the fact that they’ve been misled, or un- (or under-) informed, or one of a hundred other reasons why they are asking for the wrong thing, looking for the wrong solution. And then there’s the rare (but not rare enough) case when they are simply bat-guano crazy and want something stupid.

 

However, the customer IS always the customer. They are always the one who wants to buy, and we are always the one who wants to sell. Understanding this relationship doesn’t mean we have to sell our soul, ethics, or values to sell our product. Rather, it gives us the freedom to find the right customer for our product, and come to terms with the fact that not EVERYONE is a customer (or at least not a customer right now.)

 

“Work smarter, not harder”

What am I supposed to think here? That I've been working stupider up until now? That I've willfully withheld a portion of my intellectual capacity? Or that I'm just phoning it in? Regardless of which inference you make, it's not a kind reflection on me, nor on what you think of my work product and work habits.

 

If you think I stink, or that I'm slacking off, or that I've gotten into a behavioral rut, then please just say it. If you think I'm overlooking something obvious, then say THAT. And if you don't think EITHER of those things, don't say this to fill the dead air.

 

“Think outside the box”

Like "work smarter, not harder," this one has the one-two punch of an insult AND the implication that I can’t solve a particular problem.

 

I list this as a lie because there IS no box. I may have fallen into a rut or a set of sub-optimal habits. I might be hyperfocused on a particular outcome or trajectory; I could just be lazy and unwilling to put in the extra effort that thinking about something differently might require.

 

Whatever the reason for my inability to find a novel solution is, it's not "a box." Calling it that doesn't get me any closer to changing my behavior OR solving the challenge.

 

“Lean and Mean”

Back in 2004, Bob Lewis translated this as "emaciated and unpleasant" (http://www.weblog.keepthejointrunning.com/?p=1886) and that has stuck with me. Lean is all well and good, as long as we mean "healthy" rather than "underfed.”

 

But "mean" (unless you mean "average,”which you probably don't) is a trait I would not find advantageous in the workplace. When would it be considered organizationally good to be rude, dismissive, short-tempered, or (to use Lewis' term) unpleasant?

 

Rather, we should strive for our teams and organizations to be healthy, focused, and determined. You can even throw in “hungry” if you want, as long as it’s not due to a lack of sustenance (i.e., resources).

 

"I just need you to hold out until..."

In my experience, this is a statement that comes in three month cycles. Just keep working on-call for a little longer. Just keep putting in the extra hours. Just deal with this (last) fire drill.

 

Just put up with this sub-optimal situation.

 

After two years of ongoing struggle where the answer was routinely, "This is only going to be until the end of quarter,” I finally confronted a manager about how the situation hadn't gotten better. The situation may have changed, but the workload, sense of crisis, etc. had not improved. His response was surprisingly transparent, "Oh, I didn't mean it would get better. I just meant that it would change."

 

A company, department, or team that has gotten into the habit of trying to wait out bad situations is one that has given up on solving problems. Just remember that.

 

If you find yourself on the receiving end of this particular lie, one response is to ask for specifics. Such as:

  1. “Then what,”  as in, what is the situation going to be at the end of the <time period>?
  2. How are we going to get there between now and then?
  3. What are your expectations of me during that period?
  4. What will we do if we miss our target (date or end-state)?

 

By using these questions, and then the logical follow-up responses along with copious documentation, you are asking management to commit to a set of goals and outcomes (this is a technique also known as “managing up”). It also gives you a fairly accurate barometer as to how hard you should be looking for a better situation.

 

“This company is like family”

No, it isn't. Most companies are too large to have the group dynamics of a family. A company lacks the long-term history, shared genealogy, etc. that create lasting bonds.

 

That's not to say that people working at a company can't feel a close friendship and camaraderie. But it's still not family.

 

Those who make this type of comment are typically trying to instill in you a sense of loyalty to them right before they ask you to do something that goes against your personal interests.

 

“Human Resources”

I know, I know, this is the name of a department. But the name of the group is a misnomer bordering on a trite falsehood. Almost all of the frustration I've ever had (or heard coworkers have) with HR stems from misunderstanding their core mission.

 

In my experience, HR exists for two primary reasons:

  1. To keep the company out of lawsuits arising from employee interactions
  2. To shield upper management from the messier aspects of employees, including salary negotiations, grievances, etc.

 

They are NOT there to help you grow as an employee. They are NOT there to provide a sounding board. They are NOT there to help create a positive work environment.

 

They are NOT a department designed to help the employee in any way, unless "help" intersects with one of the two areas above. Use them in the same way you would use the legal department. Because that's pretty much what they are an extension of.

 

“Treat Your Users Like Your Customers”

I have a few issues with this. First, when I run my own company I can choose which customers I want to deal with. I do that by deciding how and where I market, which jobs I accept, and which I'm too busy to take on right now, and by setting a price for each job that reflects the level of effort and aggravation I expect to have while doing the work.

 

Equally, my customers can choose whether to hire me or the person down the street.

 

Within a company, NONE of those things are true. I can't say no to the accounting department, and they can't find someone ELSE in the company to provide the same set of services.

 

In addition, customers and vendors come and go. But Sarah in the mail room today will become Sarah the head of accounting tomorrow. So how I treat her today matters for the duration of our time at this company.

 

“The squeaky wheel gets the grease”

I learned long ago that this is SOMETIMES true. But more often, the more accurate phrase is "The squeaky wheel gets REPLACED!"

 

When, how, and to whom one squeaks is a lesson many of us learn by experience (meaning: doing it wrong) over time.

 

So, anyone who blithely tells you this probably wants one of the following outcomes, none of which will be good for YOU:

  1. They don't realize the truth of the situation either
  2. They have the same complaint and know better than to open their mouths. but they are perfectly willing for you to lead the charge and take all the heat.
  3. They want to watch you make a spectacle of yourself for their entertainment

 

“The elevator pitch”

Brevity is wonderful when it pushes us to create simple elegance. But often it just causes us to stress more, talk faster, widen the margins, shrink the font, and try to jam more into less.

 

Effective storytelling is partially about knowing when your forum and format fit the story you want to tell. Otherwise, you end up babbling like a fool and ruining your chances of making a case later on.

 

Some issues, requests, and explanations are complicated and can't be reduced to a 30-second overview. If you run into someone who demands that all of your interactions, requests, etc. be put in that format, then they're not really listening anyway.

 

“That's not part of our corporate culture/DNA”

There's a famous story about five gorillas (you can read it here).

 

Phrases like, "that's now how we do things,” or "That's just how it's done here,” or "It's not part of our DNA,”  are all lies, and all fall under the heading of Not Invented Here (NIH). It is often code for, "I don't want to," or worse, "The thought of doing things that way scares me."

 

As stated earlier, companies are not family. They also aren't organisms and thus don't have DNA. If they have a culture, it is because the people who make up the company actively choose to propagate a set of habits or a particular perspective when doing business. And culture or no culture, a good idea is a good idea. People want to do well, want to succeed, want to get ahead.

 

How you respond to this lie depends largely on your stake in doing something differently, and your role in the company.

 

“If you hire good people, they won't require supervision”

Stephen Covey famously said (http://www.azquotes.com/quote/855055),

"If you can hire people whose passion intersects with the job, they won't require any supervision at all. They will manage themselves better than anyone could ever manage them. Their fire comes from within, not from without. Their motivation is internal, not external."

 

That's great. Let's see if they file their expense forms correctly and on time.

 

People at all levels of the organizational chart require supervision. What they may not need is meddling middle managers.

 

The best supervisors are part janitor, part secretary, and part cheerleader. They keep things clean (meaning they ensure an unobstructed path for their staff to pursue work); they attend higher level meetings and report back the information honestly and transparently so that staff can take the actions that support the business goals; and they publicly recognize successes so that their team feels validated in their work.

 

Also, this is insulting in the same way that "work smarter" and "think outside the box" are. It's a form of managerial "negging" (https://en.wikipedia.org/wiki/Negging). It implies that if you DO need supervision, you are obviously not passionate enough about your work and may be the wrong person for the job.

 

“It's easier to ask forgiveness than permission”

Yes. And all planes are equipped to make a water landing. Well, at least once. Whether they can take off after that is just a minor detail, right?

 

In an unhealthy environment, people do things on the sly and hope they don't get caught. If they DO get caught,  it seems many believe throwing themselves on the mercy of the court is as good a strategy as any.

 

But in mature, professional, adult environments, asking for permission is always preferable and always easier for everyone.

 

“Failure is not an option”

Nope, in some situations (often ones where this lie is uttered), it's practically a sure thing!

 

There is no guaranteed success. There is no outcome that is 100% predictable. Some failures, once they are in motion, are unavoidable no matter how much planning was done beforehand, or how much staff are on hand during the failure to try to save it.

 

Not only is this statement a lie, it's also not particularly helpful advice.

 

“So what's the point?”

My point is certainly not that everything, or even most things, said at work are a lie. What I AM saying is that some of these trite and overused clichés have reached the end of their useful period.

 

Or as Sam Goldwyn said, "Let's have some new clichés."

 

Ain't that the truth.

Karen Lego Figures (c) Karen Lopez InfoAdvisors.com

As promised in my previous post on Better IT, in this series I will be talking about collaboration. Today I'm sharing with you anti-patterns in collaboration.

Anti-pattern - Things you shouldn't be doing because they get in the way of success in your work, or your organization's efforts.  Antonym of "pattern."

In my troubled project turnaround work, when I start to talk about collaboration, I usually get many eye rolls. People think we're going to start doing team-building exercises, install an arcade game, and initiate hourly group hugs. (Not that these would be so bad.)  But most collaboration missteps I see are the result of anti-patterns that show up in how teams work. So in this post, let's look at the not-so-great-things that will get your team and your organization into trouble.

 

IT admins who don't know who is responsible for what, or can't find them

This is often the case in geo-diverse teams, spread over several time zones, and teams with a high staff turnover. Their processes (their "pattern") is to go on a "responsibility safari" to find the person and their contact information for a resource. On one project, it took me almost a month to find the person, who lived on another continent, who was responsible for the new networks we were going to deploy to our retail locations. By the time I found him, he was planning on moving to another company within a week. Having to hunt down people first, then their tools, then their data, is both costly and time-consuming, which delays one's ability to resolve issues. Having to find people before you find data is not the right way to manage.

 

IT admins who collect duplicate data about resources and their metrics, often in difficult to integrate formats and units of measure

This is almost always the result of using a hodgepodge of tools across teams, many of which are duplicate tools because one person has a preference of toolsets. This duplication of tools leads to duplication of data.  And many of these tools keep their data locked in, with no way to share that data with other tools. This duplication of data and effort is a huge waste of time and money for everyone. The cost of incompatible tool sets producing data in incompatible formats and levels of granularity is large and often not measured. It slows down access to data and the sharing of data across resource types.

 

IT pros who want to keep their data "private" 

This dysfunction is one my friend Len Silverston calls "data mine-ing," keeping data to yourself for personal use only. This is derived from the fact that data is indeed power. Keeping information about the status of the resources you manage gives you control of the messaging about those systems. This is a terrible thing for collaboration.

 

Data mine-ing - Acting in a manner that says, "This data is mine."

- Len Silverston

Agile blocking is horrible

A famous Agilista wants people to report false statuses, pretend to do work, tell teams that "all is good" so he can keep doing what he is doing without interruption. He also advocates for sharing incorrect data and data that makes it look like other teams are to blame. I refuse to link to this practice, but if you have decent search skills, you can find it. Teams that practice blocking are usually in the worst shape possible, and also build systems that are literally designed to fail and send their CEO to jail.  It's that bad. Of all these anti-patterns, this is the most dangerous and selfish.

 

IT admins who use a person-focused process

We should ensure that all of our work is personable. And collaborative. But "person-focused" here means "sharing only via personal intervention." When I ask them how they solve a problem, they often answer with, "I just walk over to the guy who does it and ask them to fix it." This is seen as Agile, because it's reactionary, and needs no documentation or planning. It does not scale on real-life projects. It is the exact opposite of efficiency. "Just walking over" is an interruption to someone else who may not even manage one of the actual resources related to the issue. Also, she might not even work in the same building or country.  Finally, these types of data-less visits increases the us-versus-them mentality that negatively impacts the collaboration success. Sharing data about an instance is just that: data. It's the status of a set a resources. We can blame a dead router without having to blame a person. Being able to focus on the facts allows us to depersonalize the blame game.

 

Data will save you

These anti-patterns don't just increase costs, decrease team function, increase risk, and decrease organizational confidence, they also lead to employee dissatisfaction and morale. That leads to higher turnover (see above) and more pressure on good employees. Having the right data, at the right time, in the right format, will allow you to get to the root cause of issues, and better collaborate with others faster, cheaper, and easier.  Also, it will let you enjoy your 3:00 ams better.

 

Are there other anti-patterns related to collaboration that you've seen when you've tried to motivate cross-team collaboration?  Share one in the comments if you do.

A few years back, I had the privilege of attending InfoSec World in Orlando. I served on a security panel for Tech Field Day Live, where we discussed some game changers in security. It was a strange feeling being there. I don’t think anyone knew who we were or what Tech Field Day was. I was a security guy, at least from my point of view I was. I was sitting next to the former program manager of the CCIE Security, Natalie Timms, Edward Haletky, and Jack Daniel. I had been on Twitter for years, written a few books for Cisco Press, and I still felt out of place.

 

As I sat at dinner with Stephen Foskett, I came to the conclusion that these security folk were not very social. I think things have loosened up a bit, but it’s still not an easy world to break into. Why is this important? For some of you, it’s not. For me, time and again being on social networking sites has saved my bacon when I found myself in a jam. What about now?

 

Well, I can’t say that social networking has improved a lot for security professionals. It may have, but I don’t run in those circles. What I can tell you is that security vendors see the value in, and advertise and respond to, direct inquiries. Sometimes their response to social networking questions is days ahead of an email request submitted through a web form.

 

Still, following a few security accounts on Twitter can’t hurt. Here are a few that I think are beneficial. Trust me when I say that I have followed several, and unfollowed them just as fast. There’s a lot of repetition and noise out there.

 

Illumio

Illumio is a security vendor that I learned about while attending a networking field day. They are based out of Sunnyvale, California, and most of their posts are informative and focus on adaptive security for the data center. They have an interesting solution that I should have taken the time to break down a bit more on my personal blog. You can look them up on the Tech Field Day website. The video is worth a watch, as is their Twitter feed.

 

InfoSecurityMag

I’ve followed @infosecuritymag for some time. They also have informative news that tends to be focused on general information security. You’re not going to learn earth-shattering techniques here, but for someone breaking into the world of security, it will give you a variety of topics to discuss with your peers.

 

Krebs on Security

Brian Krebs is a journalist who frequently covers profit-seeking cybercriminals. He uses his platform to inform on various security topics. He’s become quite a name in the space and can be found on Twitter; he also hosts his own podcast. In late 2016, he was the target of one of the largest DDoS attacks on record. Yep. He ticked off that certain person that made it a point to shut his site, Krebs on Security, down for quite some time.

 

Cisco Security

If you work with Cisco Security products, you may be interested in following @ciscosecurity on Twitter. The bulk of what you see here is going to be product announcements and write-ups on whatever direction Cisco thinks is important these days. It’s hard to ignore Cisco, even though many will argue the quality of their security portfolio. But when you consider that they house the Talos group, and have tons of customer-provided data that they can analyze, which keeps them up-to-date on emerging threats, it’s worth a little bit of marketing just to get to the good stuff.

 

Others I’ve Followed

I’ve also followed Bruce Schneier, but after a while his posts don’t really interest me. But you have to admit that the guy is brilliant. Aside from Bruce, there have been others that are more Cisco-specific, but I wouldn’t really recommend them for someone starting out in security.

 

Final Thoughts

If you are coming from a data networking background and transitioning into a security role, the news is a solid start. Trying to engage individual security professionals as you would networking and virtualization guys is a tough nut to crack. As time goes by, you’ll likely become familiar with peers you meet at conferences and such. Those are the ones to interact with online, if possible. When you find yourself in a bind, these are the folks you want to ask. However you should be aware that the conversation you have may not be in the public eye. Many security professionals are a little but more hush-hush publicly. They understand that certain things that are said could lead people to the wrong conclusions about the organizations they represent. You can’t blame them. The lawyers are going to force their organizations' representatives to be very tight-lipped about possible breeches and such. For example, if you ask a question about some ransomware that you’ve just encountered, and a buddy of yours that happens to work for Wells Fargo tells you how to fix the issue, people may naturally assume that Wells Fargo has experienced a similar breech. Yep, that’s bad for press.

 

So be patient, interact with those you know personally, and use social networking to keep up on industry trends. Doing so may guide your learning path as you progress as a network security professional.

In my last blog post, I began laying out the case that troubleshooting can be a challenging endeavor, especially when dealing with multiple disciplines (silos) in a larger IT organization. More often than not, it becomes a race to innocence between the groups, with each trying to prove that they are not the problem. The trouble is, nobody is trying to figure out what the problem really is. They just want to know it's not them. As I pointed out, that can largely be attributed to a lack of good tools, though that's less and less of an issue in today's world.

 

While many organizations have already taken steps to reduce the effect of silos on IT operations, challenges still abound; chief among them is the disparate collection of tools each group uses. Because different tools all represent data in myriad ways, even as cooperation is fostered, roadblocks still remain. Luckily, someone has done something about all of this, and is providing a tool that cuts across multiple disciplines.

 

Solarwinds' new PerfStack product, a part of the Orion Platform of network monitoring applications, aggregates multiple data points from across the range of products in that suite. As the Orion suite is very widely adopted, centralizing the collected data is a no-brainer. Solarwinds is already deployed in many NOCs, and having the ability to perform cross-discipline troubleshooting from within the same tool should prove invaluable.

 

While Solarwinds has always been on point with their solutions, and has had systems monitoring dashboards for some time, those dashboards have not necessarily been dynamic. That's not a bad thing, it's just a function of their purpose; NOC monitoring does not typically require quick changes of information displays (dashboards) on a regular basis. Frequently, the dynamic troubleshooting has been handled by various point tools. Perfstack fixes these issues, and more.

 

PerfStack can now create dashboards on the fly, filled with all of the pertinent pieces of data needed to remediate a problem. More than that, however, they can give another user that same dashboard, who can then add their own bits and bobs. You are effectively building up a grouping of monitoring inputs consisting of cross-platform data points, making troubleshooting across silos seamless in a way that it has never been before.

 

By way of example, the networking team typically fields initial trouble tickets that involve nebulous symptoms of "can't get there from here," or "something, something, really slow." That team would use whatever point tools at their disposal to look for potential issues, beginning at layer-1 with cabling, optics, etc., and progressing, as needed, up through the stack until a problem is found. At some point, assuming the problem is elsewhere (c'mon, it's never really the network) the trouble ticket gets passed off to security, perhaps, or applications, storage, wherever. But at that point all historical information in the point system is unavailable to the new team who must start from scratch. That is inefficient at best. PerfStack flips that modality on its head, and allows teams to share data from the initial investigation through to an eventual resolution.

 

With everything in IT connected at such a fundamental level so as to be inseparable on any practical level, the era of point solutions really should be relegated to the IT trash bin, an anachronistic relic of our past. PerfStack is one of the first serious products, built upon a proven technology platform, to usher in this new paradigm in root-cause analysis and remediation. Solarwinds was uniquely positioned to create this product, and while it would disingenuous to say that they've invented this new space, they have certainly capitalized on a need, executed a flawless solution, and dramatically accelerated the time-to-value over what I can now safely call legacy approaches.

In my previous two posts, DDoS and The Broken Internet and The Internet of Hacked Things, we discussed how there are some critical flaws in key services and internet infrastructure that easily allow attackers to cripple large portions of the internet, as well as highlighting how IoT is really the Internet of Vulnerable Things as manufacturers and consumers fail to properly secure these devices. These devices are then easily gathered into massive botnets, which can then be used to exploit the cracks in our infrastructure’s armor.

 

For all we know, Mirai, or some other massive botnet, could suddenly become self-aware, and then we’re on the path to Skynet and Judgement Day, right?

 

Now we know what the problem is. So, how do we fix it before we are forced to bow to our IoT overlords?

 

Who is ultimately responsible?

 

Some of you wondered if this was an engineering versus marketing problem. The classic battle between engineers and/or designers who want their product to be polished and ready for market, and marketing execs who just want the revenue from getting the product on the shelves as quickly as possible, a few flaws be damned. Nobody can deny the rapid and unfettered uptake of these devices, which are now in the wild numbering several million.

 

Others blamed the consumer, the person purchasing and installing the IoT devices without “reading the fantastic manual” and properly securing the device from the treacheries of the internet. A problem borne of ignorance. How could we expect the average non-techie consumer to be aware of SSL vulnerabilities?

 

Security of the IoT, and key internet services as a whole, has to be handled with a layered approach. The mandate of everyone involved, from the designer, manufacturer, consumer, and network operator, should be to ensure that they each are doing their part to secure their piece of this puzzle from continuing to allow massive DDoS attacks to destabilize large sections of the internet.

 

Design

 

Designers have to balance ease of use and convenience with security. While it might be convenient to have Telnet and HTTP services on a device, so users can easily configure or manage these IP-enabled devices, it’s irresponsible to have these types of unsecured services exposed to the internet, often with well-documented default passwords. At a minimum, encrypted services like SSH and HTTPS should be used, and the use of default passwords should require a forced change. Hard-coded passwords that cannot be changed are a major foul. Ensuring the use of the most recent and patched versions of these protocols should also be required.

 

Government

 

What about a governing body, like the FCC in the United States, or the CRTC in Canada, getting involved to qualify these devices? We’ve all seen the FCC/CRTC stickers on wireless devices, showing the devices meet standards set forth by these agencies, certifying them for use under proper regulations. Why can’t we apply this model to IoT devices, and require they are put through a proper vetting and testing process before being allowed to market? This would keep designers and marketing execs accountable for the products they are putting on shelves, and give the average consumer some comfort in knowing they aren’t inadvertently adding to the problem when purchasing one of these devices.

 

Service Providers

 

Service Providers of all types (Internet, DNS, Hosting) need to ensure their infrastructure is not only secure, but resilient, and able to mitigate DDoS attacks before they ever impact their customers. This means investment in their infrastructure and careful monitoring of traffic flow across and through their networks. Top tier providers really have no excuse, they know they are high priority targets, and they should do everything in their power to prevent the negative impact of these attacks.

 

Enterprise Networks

 

Enterprise network operators rely heavily on their Service Provider partners to perform their due diligence and clean up a lot of the internet trash thrown their way before it even hits their network edge. That doesn’t remove their responsibility, however, and enterprise networks should be designed and built with as much resiliency as possible, so that if one of their service providers is attacked, or any attack traffic gets through directly to them as a target, that there are backup services in place to allow uninterrupted business traffic flow.

 

Don’t be part of the problem! Monitoring your own network for compromised endpoints is also critical. Tools such as Solarwinds’ Long & Event Manager can assist with identifying traffic aimed at botnet CnC servers, and identify compromised devices that you can then patch or remove.

 

A microcosmic layered approach is beneficial as well. A first line of defense that includes cloud services that scrub and pre-filter traffic, followed by deeper levels and layers of defense, such as load balancers, firewalls, IPS, and application filters/firewalls, are all pieces of the protection puzzle that should be integrated.

 

Everyone

 

It’s unlikely that the folks behind massive DDoS attacks like the ones aimed at Brian Krebs, Dyn, or Akamai are going to get bored and stop on their own. It’s equally unlikely that we’ll ever find a single magic bullet that fixes all of the issues with our underlying infrastructure. So, we need to plan for these flaws and ensure that everyone involved has some measure of protection, and that the majority of the malicious traffic can get dumped or filtered out before it has any major impact to the end-user or customer. This can only be accomplished with a multi-faceted approach at all levels of the internet.

 

We’ve seen attacks over 600Gbps, and current projections of the Mirai botnet speculate that attacks exceeding 1Tbps are presently possible. It seems to be an arms race, and the only way to stop the attackers from continuing to gain strength (literally) in numbers, is to secure IoT as much as possible and limit the attack surface of our own networks.

 

Are there any other individuals and/or groups that need to be held accountable for the security of IoT?

Back from Germany and on vacation this week. But I'm not going to let a few days off get in the way of browsing the internet for things to share with you. Enjoy!

 

Mobile Ransomware Jumps 50% in a Year

The article focuses on Android devices, which have always had quality control concerns for apps in their store. Say what you will about Apple, but control over their store leads to lower risk for consumers. And Microsoft doesn't have this issue because they have no apps.

 

Trump and His Android Phone Putting National Security at Risk

I have no idea why we don't simply take away his toys. It's almost as if Trump has no idea about the risks involved.

 

German parents urged to destroy data-collecting toy doll

As if having my refrigerator and television spying on me wasn't enough to worry about. Now I have to worry about dolls.

 

Why buying used cars could put your safety at risk

"The car is smart, but not smart enough to know who it's owner is." Classic design fail here, as the developers never thought about the idea that a car may have more than one owner.

 

Poor Access Management Leads to $5.5M HIPAA Penalty

Show this to the next person that wants to know why you care so much about data security. Well, that is, if you do. And you should.

 

Amazon Takes on Skype For Business With Chime But It Won’t Come Cheap

Watching Microsoft and Amazon compete head-to-head on each and every product possible tells me one thing: Amazon is the new Apple, but without the overpriced hardware and underwhelming software. The next 30 years should be fun to watch as Microsoft and Amazon compete for our dollars.

 

Our workshop was full last week, with 50 people grouping together to do hands-on lab exercises with SQL Server 2016 running on Azure VMs:

IMG_6245.JPG

I wrote last week about how performance troubleshooting is a problem with multiple dimensions. From the client, across the network, and into the application server, there are many places where application performance can be impacted. One of the key parts of performance troubleshooting is sorting through all those dimensions, then finding the one that is limiting application performance. Unfortunately, there are many more dimensions when your applications are inside virtual machines. Understanding and considering these dimensions is critical to performance troubleshooting in a virtual environment.

 

The first place we add more dimensions is between the VM and the hypervisor. VM-sizing as well as virtual HBA and NIC selection all play a part in application performance. Then there is the hypervisor and its configuration. A lot of the same issues that affect operating systems also effect hypervisors. Are updates applied? Is the storage configuration correct? How about NIC teaming? Even simple things like storage vendor recommended optimization can make a big difference to the performance of applications in the VMs.

 

The next dimension is that multiple VMs share a single physical server. So your application’s performance may be dependent on the behavior of other VMs. If another VM, or group of VMs, uses most of the physical CPU, then your application may run slow. If your VM resides on the same storage as a bunch of VDI desktops, then storage performance will fluctuate. We see this noisy neighbor problem most often in cloud environments. But noisy neighbors absolutely can happen in on-premises virtualization. A particular challenge is the invisibility of hypervisor issues to the VM and its applications. The operating system inside a VM will report the clock speed of the underlying physical CPU, even if it is only getting a small fraction of the CPU cycles. A VM that uses 100% of its CPU is not necessarily getting 100% of a CPU, it is just using 100% if what it gets from the hypervisor.

 

There is a time dimension here, too; one related to the portability and mobility of VMs. Your application server may move from one virtualization host to another over time. So can other VMs. The noisy neighbor VM that caused a performance issue half an hour ago may be nowhere near your VM right now. On most hypervisors, VMs can also move from one piece of storage to another. That performance issue yesterday might have triggered an automated movement of your VM to a better storage location. The migration made the performance problem disappear. This VM mobility can lead to performance issues being intermittent, they only manifest when a specific other VM is on the same physical host.

 

Another virtualization dimension is when the user’s PC is really a VM. VDI adds more dimension to performance. There is a whole network between the user and their PC, as well as another device right in front of the user that may bring its own performance issues. Users seldom have the right words to differentiate a slow VDI session from a slow application. All of the noisy neighbor issues we see with server VMs are multiplied with desktops.

 

Virtualization has enabled many awesome changes in the data center. But virtualization has added serious complexity to performance troubleshooting. There are so many dimensions to understand to find and resolve the restricting dimension.

By Joe Kim, SolarWinds Chief Technology Officer

 

In World War I, the U.S. Army used lumbering GMC® trucks for the first time in combat, which was revolutionary for its time. Today, these vehicles would be considered slow, cumbersome, and archaic in comparison to today's fast, powerful and, most of all, constantly connected war-fighting machines.

 

In fact, thanks to the Internet of Things (IoT), just about everything that can be connected—from tanks to smartwatches—is connected. The Defense Department’s entire work force depends on thousands of devices that work off of disparate operating systems. The net result is a security risk nightmare for those who must secure government IT networks.

 

IoT has made establishing a strong security posture much more challenging for beleaguered IT administrators. Still, practicing a few simple strategies can help these administrators beef up security in the face of the IoT onslaught.

 

Step One: Build security in from the beginning

 

Last year, the Federal CIO Council’s Mobile Technology Tiger Team released standardized security protocols for agencies that build their own mobile apps. The protocols outline the need to vet applications by building security and functionality together throughout the app development process.

 

For federal IT administrators, security must be interwoven into the fabric of agency networks. This starts with strategic planning—considering every possible breach scenario, identifying potential threats before they occur, and responding to emergencies. Attention must continue with the deployment of automated tools that scan for and alert users to threats as they occur. Solutions that offer automated, round-the-clock monitoring and real-time notifications help administrators react more quickly to potential threats and mitigate damage.

 

Step Two: Assess security risks associated with every app

 

Protection against external applications that constantly track and collect data over secure networks require administrators to be particularly vigilant in the types of apps and devices they allow over these networks.

 

Managers can create “white lists” of approved apps, and use monitoring tools to alert whenever an unauthorized app requests network access. They can also track those applications back to individual users if necessary.

 

Step Three: Do the same for devices

 

IoT takes us well beyond smartphones and tablets into a new realm of connected tools that might not yet be accepted, and administrators must closely monitor the devices accessing the networks. They might permit smartphones and tablets on the network, provided they meet security standards, while eschewing untested or non-essential devices. Simultaneously, they should set up a system to track devices by MAC and IP address, and monitor the ports and switches that those devices use.

 

Mobile technology has brought us some great things: the ability for fighters to easily communicate and access information from anywhere in the field, opportunities for greater productivity, and collaboration across agencies. But it’s also brought a great deal of headaches. With the IoT here, those headaches could potentially turn into continuous migraines. It is helpful to view the aforementioned strategies as the IT equivalent of an Excedrin tablet: a way to ward off the pain and secure the network before it gets out of control.

 

Find the full article on Signal.

When a SysAdmin implements a DevOps approach to an environment, there are many things that need to be considered. Not least of which is developing a strong line of communication between the developer and the SysAdmin. Occasionally, and more frequently as time goes on, I’m finding that these people are often one and the same, as coders or scripters are spreading their talents into the administrator’s roles. But these are discreet skills, and the value of the scripter will augment the skills and value of the system administrator.

 

Process is a huge component of the SysAdmin's job. The ordering process, billing process, build process, delivery process. How many of those tasks could be automated? How easily could these functions be translated to other tasks that are conscripted, and could therefore be scripted?

 

The wonderful book, The Phoenix Project, is a great example of this. It outlines how an IT manager leverages the powers of organization, and develops a process that helps facilitate the recovery of a formerly financially strapped industry leader.

In the book, obvious communications issues initially bred the perception that IT was falling behind in its requisite tasks. Systems, trouble tickets, and other tasks were not being delivered to the requestor in any sort of accurate time frame. As the main character analyzed both the perceptions and realities of these things falling short, he began questioning various stakeholders in the company. People who were directly affected, and those outside his realm, were solicited for advice. He also evaluated processes within the various groups of the organization, found out what worked and why, then used his successes and failures to help him to establish assets worth leveraging. In some cases, those assets were skills, individuals, and personalities that helped him streamline his functions within the organization. Gradually, he was able to skillfully handle the problems that arose, and fix problems that had been established long before.

 

A number of key takeaways from the character's experiences in this series of actions illustrate what I was trying to outline. For example, the willingness to question yourself and your allies and adversaries in a given situation, with the goal of making things better. Sometimes it takes being critical in the workplace, which includes scrutinizing your successes and failures, may be the best way to fix issues and achieve overall success. 

 

What does your process look like for any given set of tasks? How logical are they? Can they be eased, smoothed, or facilitated faster? Evaluating the number of hours it takes to perform certain tasks, and then critically evaluating them, could potentially improve metrics and processes. As I see it, the scriptability of some of these tasks allows us to shave off precious hours, improve customer perception, and gain credibility again. Being willing to embrace this new paradigm offers countless benefits. 

 

It all begins with communication. Don’t be afraid to allow yourself to be seen as not knowing the answer, but always be willing to seek one. We can learn from anyone, and by including others in our evaluation, we can also be perceived as humble, collaborative, and productive.

cl-berlin-2016-night-460x230.png

Last year I wrote about my experiences at CiscoLive Berlin (or #CLEUR, as it is hashtagged on Twitter).

 

This year, as I'm sitting on the plane to meet up with patrick.hubbard, ding, stevenwhunt, and the rest of the gang, I thought I would take a few minutes before catching some fitful transatlantic naps and share some of the things I'm looking forward to seeing.

 

Sharing the PerfStack story

Without a doubt, the thing I'm MOST excited to share with the attendees is PerfStack and the way it allows network engineers to combine data from across the infrastructure and compare it with application insight. This isn't just your usual "MTTI / It's never the network" gag. This is about enabling every IT professional to become a true "full-stack" engineer, and even a data scientist, snapping together data from across the enterprise the way Master Builders snap together "interlocking plastic building pieces" in the Lego Movie.

 

Arduino

A close second to PerfStack is my excitement to see what the Arduino booth has in store this year. As I mentioned in my summary from 2016, this activity captured my imagination like nothing that I've seen at a convention in a long time. I'm hopeful they up their game, but even if they just do a repeat of last year's side-quest, I'll be happy to play along. It was the ultimate cross-vendor ice-breaker.

 

IoT

So much has happened in the world of IoT this year (which frequently finds its way onto the pages of Geek Speak), that I'm keen to see how Cisco, the speakers, and even the vendors on the floor are responding. To be certain, we can't keep doing the same old things. But HOW everyone is choosing to respond will be as interesting as anything else.

 

Clouds and Containers

Finally, the inexorable march continues from pure on-premises through the haze of hybrid IT, to the eventual nirvana of cloud-based everything. As much as this has been a "virtualization, server, and application" type story, network (and Cisco) have had their roles to play and will continue to drive the conversation. I haven't paid as much attention to this sector of the industry as I should, leaning heavily on the experience and insight of fellow Head Geeks kong.yang and sqlrockstar, but it's time for me to start building my knowledge in this area.

 

That's what I'm looking forward to. Is there something YOU are particularly keen to hear about? Let me know in the comments and I'll try to do some of the legwork for you and report back next week!

Nobody is safe.

 

From the highest tier service provider to a small business network in rural Iowa, every network is susceptible to a massive scale DDoS attack. Whether as a direct target of the attack, or indirectly as critical services are affected elsewhere on the internet.

 

In my previous article, DDoS and the Broken Internet, I outlined how the recent attack on Dyn highlighted the interconnected services that we have all come to rely on as network operators, and how their service interruptions affect all of us.

 

This highlights a major flaw in these networks, and largely in the internet as a whole. And yet, we keep adding to it, adding devices that previously had no need for internet access, yet the Internet of Things (IoT) is here, and it’s exacerbating the flaws in our infrastructure.

 

On Tuesday, September 20th 2016, Akamai (the world’s largest content delivery network) defended against a DDoS attack that was in excess of 620Gbps. This attack was perpetrated largely by the Mirai botnet, malware that has amassed an army of IP-enabled devices with common open ports on the internet – Telnet, SSH, HTTP, SMTP, etc. These are not malware-infected home computers, mind you. These are vulnerable connected devices like cameras and DVRs. Devices that are either vulnerable to SSH exploits, or don’t use SSH at all, and often use default credentials.

 

Akamai identified the top protocols used in the attack as SSH and Telnet, both very common protocols. The top usernames used were root, admin, and shell. All common and well-documented defaults. How could so many of these devices be so easily compromised? Many of these devices are purchased and used by consumers who are often unaware of the risks of leaving these settings at their defaults, and, as a result, they are left vulnerable and are easily assimilated into these massive botnets.

 

To make matters worse, some of the identified IoT devices had hard-coded credentials that could not be changed!

 

SSH vulnerabilities have been around for a long time, and as new exploits are detailed, patches are released and updates are required. Many of these IoT devices don’t get updates. They are installed by the user and then simply left. Who has time to do a firmware update on the home security camera system they bought at Costco, right?

 

The attack on DNS provider Dyn in October was also at least partially the responsibility of Mirai and the hacked IoT devices. This attack had far-reaching effects, causing a large number of high profile websites to be inaccessible, including Twitter, Spotify, and Reddit, among others.

 

The number of IoT devices is increasing exponentially, and with it, the attack potential of massive botnets like Mirai.

 

How can we secure the Internet of Things, or protect our infrastructure from them?

There is a reason why resolving application performance issues is so hard. Actually, I think there are a lot of reasons, but I’m going to spend a few posts looking at one set of reasons.

 

A lot of human endeavors are linear: move in one direction until you reach your goal. Troubleshooting application performance is non-linear. It has a lot of different, yet interrelated dimensions. The only way to resolve application performance issues is to address the dimension that is limiting performance. But with the interrelationships between the dimensions, the symptom may be far removed from the constraining dimension. The cause of a performance issue can be a long way from its visible effects.

 

One dimension is the performance of the user interface. Whether it is a native application or a web browser, every application has a User Interface (UI). Since the UI is close to the user, it is what they perceive as the application. Something as simple as a bug in Adobe Flash or a laptop that hasn’t been rebooted in months can lead to a poor user experience. Of course, the application that is slow for the user may not be the cause. Another application taking up 99% of their laptop’s CPU can be the bully, the application that they are trying to use may be a victim. Just inside the user’s computer, there are multiple performance dimensions.

 

The next dimension is the network between the user’s device and the application servers. From our desks outside the data center, the network can be very fast. But if your users are on the end of a busy WAN circuit, or over a VPN, or half way around the world,j then they will have a very different experience. What about load balancers, firewalls, and even simple things like name resolution? Having an out of date primary DNS server can add a minute to the application’s initial load time. The network adds still more dimensions to the troubleshooting problem.

 

Then there are dimensions on the application server. Does it have enough RAM and CPU? Are its disks up to the load of the application? What about the optimization of the operating system? Have the drivers for the storage and network adapters been updated to the latest? Or have they been updated to a faulty release? Is the cause of the performance problem a periodic scan by the anti-virus software? The application server adds more dimensions to any performance problems.

 

Maybe there is a time dimension? End-users in another country may access the application server at an unusual time. Are there maintenance tasks happening that might make the application slow at exactly the time that your European users need to get their end of day processing done? With all these possible dimensions to be investigated, it is no surprise that performance troubleshooting can take some time.

 

Performance troubleshooting has many dimensions. Until you identify the limiting dimension you will never fully resolve the performance problem. The number of dimensions increases substantially when you add virtualization to the mix. Next week I will look at some additional dimensions that you need to consider when virtualization comes into play.

There are several security certifications that one can choose from. While the list is long, we're primarily going to touch on five of them here. But for good measure and simply to prove our point, here's a more extensive mound of security certifications that sit before you.

 

CompTIA Security+

The CompTIA Security+ certification has been around for a long time and is a well-recognized and respected certification in the field. In fact, it meets the ISO 17024 standard and is approved by U.S. Department of Defense to fulfill Directive 8570.01-M requirements. That being said, this certification is more entry level than anything else. You can find the details on the CopmTIA web site. This certification is going to provide you with understanding in the following areas:

  • Threat management
  • Cryptography
  • Identity management
  • Security systems
  • Security risk identification and mitigation
  • Network access control
  • Security infrastructure

All of these areas are powerful in terms of what would be useful in a production environment. You'll probably want to have the Network+ certification first, or at least hold that level of knowledge before this material can fully sink in.

 

Would the CompTIA Security+ Certification Benefit Me?

If you're in a government job and need to meet certain standards, this certification may prove to be useful.  If you're a newbie to security, this certification will likely offer you a good introduction to security, but many hiring managers understand that this is an introductory certification. This is probably not the kind of certification that's going to dress your resume up enough to demand the big bucks, but it can't hurt to have it. Time learning is usually not time wasted.

 

GSEC: SANS GIAC Security Essentials

This is another entry-level security course, but it's designed a bit differently. This course is designed to demonstrate hands-on capability in security administration. The certification is good for four years before you need to renew it, and it is much more expensive compared to the Security+. Whereas the Security+ certification will cost you $320.00 USD, the SANS GIAC Security Essentials exam will run you just over $1200.00 USD.

You can find the details on the giac.org Web site.

Topics covered by this certification include:

  • Identifying and preventing common attacks
  • Identifying and preventing wireless attacks
  • Access controls
  • Authentication and password management
  • DNS Security
  • Cryptography fundamentals
  • ICMP Security
  • IPv6 Security
  • Public key infrastructure
  • Linux security
  • Network mapping

 

Would the GSEC: SANS GIAC Security Essentials Certification Benefit Me?

For a lot of people, hands-on is the way to go. In fact, the CCIE Certification Program offered by Cisco has been seen as one of the most credible certifications to hold. Much of that has to do with the fact that it's a hands-on certification, which has the benefit of credibility. If you've passed one of these exams, you must know how to do whatever you were tested on. So if you want to break in at the entry level with a bit more than a sheet of paper, this is the cert for you.

 

Certified Ethical Hacker (CEH)

The CEH certification is a common certification that is considered intermediate-level. It's not uncommon for organizations to request network security assessments. The CEH certification is a key certification that companies engaged in this type of offering look for.  This certification teaches you the same techniques that hackers use.  Armed with this knowledge you would then be better positioned to identify threats as they come across the network.

Some areas touched on in this certification include:

  • Reconnaissance
  • Scanning networks
  • Enumeration
  • Trojans, worms and viruses
  • Sniffers
  • Denial-of-Service attacks
  • Session hijacking
  • Hacking web servers, wireless networks, and web applications
  • SQL injection
  • Cryptography
  • Penetration testing
  • Evading IDS, firewalls, and honeypots

As you can see, the list is a bit more extensive than the Security+ certification. You'll need to have that general security knowledge before you take on a certification like this. This is another intermediate certification.

 

Would the CEH Benefit Me?

If you want to be an ethical hacker, this certification is a must. If you want to be a Cyber Security Analyst working in a Security Operations Center, this certification is also valuable because it lets you identify potentially malicious activity much easier than if you didn't have this underlying knowledge.  At the end of the day, I see a lot of people get this for the fun of it rather than to advance their career, but employers still recognize the certification. In specialized environments, they look for it.

 

Certified Information Systems Security Professional (CISSP)

The CISSP is an advanced-level certification. It's vendor neutral and is one of the certs that's been around the longest. It's been on the "Certifications Most-wanted" list within organizations for many years. Those that hold the CISSP are usually Senior Security Personnel and thus make a bit more cash. Some of the topics you'd be tested on include:

  • Risk management
  • Access control
  • Application security
  • Cryptography
  • Security architecture and design
  • Investigation and ethics

 

Would the CISSP Benefit Me?

If you have a minimum of 5 years experience in two of what the (ISC)2 called a Common Body of Knowledge domain, or 4 years experience and a college degree, this is your cert.  That's because these are the requirements to obtain this certification. But what are the domains you ask? They are Security and Risk Management, Asset Security, Security Engineering, Communications and Network Security, Identity and Access Management, Security Assessment and Testing, Security Operations, and Software Development Security.

 

Certified Information Security Manager (CISM)

The CISM certification is designed for anyone that's going to be managing, developing, and overseeing information security systems. This is a newer certification on the scene, but what sets it apart is that its geared toward maintaining the highest quality standards when it comes to audit, control, and security of an organization's security systems. It's not an entry-level certification either. This certification is designed for one with experience. The requirements for this certification include:

  • Agree to ISACA's code of professional ethics
  • Pass an exam
  • Have 5 years experience
  • Comply with a continuing education policy
  • Submit a written application

As you can tell, there's a bit of work included in just obtaining the certification, and that's not counting the actual security knowledge you need.

 

Would the CISM Benefit Me?

The CISM is a bit more expensive compared to other certifications. If you have the money, have the time, and can meet the requirements, then holding this certification is extremely beneficial.  Hiring managers recognize the certification, and when you combine it with experience, the Infosec Institute ranges the pay from $52,402 to $243,610.  Yes that's a very wide range, but you have to factor experience into the mix. An entry-level position isn't going to pay top dollar, no matter what certification you hold.

 

Final Thoughts

At the end of the day it's up to you. How much time to you want to commit to certifications vs hands-on experience?  Are you even looking for a job? I knew a guy that had about 40 different certifications and the only reason he got them is because he was bored at work. He had no intention of leaving his high-paying job that was paying for him to become certified. Especially when he didn't have much to do when he did have to work.

 

Still, one should recognize that employers try to filter through potential candidates, and having a security certification can help shuffle your resume to the top. If you get that far you'll have to prove that you know your stuff in an interview, and that's a whole other conversation.

A few years ago I was working on a project as a project manager and architect when a developer came up to me and said, "You need to denormalize these tables…" and he handed me a list of about 10 tables that he wanted collapsed into one big table. When I asked him why, he explained that his query was taking four minutes to run because the database was "overnormalized." Our database was small: our largest table had only 40,000 rows. His query was pulling from a lot of tables, but it was only pulling back data on one transaction.  I couldn't even think of a way to write a query to do that and force it to take four minutes. I still can't.

 

I asked him to show me the data he had to show me the duration of his query against the database. He explained that he didn't have data, he had just timed his application from button push to results showing up on the screen. He believed that because there could be nothing wrong with his code, then it just *had* to be the database that was causing his problem.

 

I ran his query against the database, and the results set came back in just a few milliseconds. No change to the database was going to make his four-minute query run faster. I told him to go find the cause that was happening between the database and the application. It wasn't my problem.


He eventually discovered that the issue was a complex one involving duplicate IP addresses and other network configuration issues in the development lab.

 

Looking back on that interaction, I realize that this is how most of us in IT work: someone brings us a problem, ("the system is slow"), we look into our tools and our data and make a yes-or-no answer about whether we caused it. If we can't find a problem, we close the ticket or send the problem over to another IT group. If we are in the database group, we send it over to the network or storage guys. If they get the report, they send it over to us. These sort of silo-based responses take longer to resolve, often lead to a lot of chasing down and re-blaming. It costs time and money because we aren't responding as a team, just a loose collection of groups.


Why does this happen?


perfstacksingle.pngThe main reason we do this is because typically we don't have insights into anyone else's systems' data and metrics. And even if we did, we wouldn't understand it. Then we throw in the fact that most teams have their own set of specialized tools and that we don't have access to. I had no access to network monitoring tools nor permissions to run any.  It wasn't my job.

 

We are typically measured and rewarded based on working within our own groups, be it systems, storage, or networks, not on troubleshooting issues with other parts of infrastructure.  It's like we build giant walls around our "stuff" and hope that someone else knows how to navigate around them. This "not my problem' response to complex systems issues doesn't help anyone.

 

 

What if it didn't have to be that way?

 

Another contributing factor is the intense complexity of the architecture of modern application systems. There are more options, more metadata, more metrics, more interfaces, more layers, more options than ever before. In the past, we attempted to build one giant tool to manage them all. What if we could still use specialty tools to monitor and manage all our components *and* pull the graph of resources and their data in one place so that we could analyze and diagnose issues using a common and sharable way?

 

True collaboration requires data that is:

  • Integrated
  • Visualized
  • Correlated
  • Traceable across teams and groups
  • Understandable

 

That's exactly what SolarWinds' PerfStack does. PerfStack builds upon the Orion Platform to help IT pros troubleshoot problems in one place, using a common interface, to help cross-platform teams figure out where a bottleneck is, what is causing it and get on to fixing it.

PerfstackScreen.png

 

From <https://thwack.solarwinds.com/community/solarwinds-community/product-blog/blog>

 

PerfStack combines metrics you choose from across tools like Network Performance Monitor Release Candidate @network  and Server & Applications Monitor Release Candidate from the Orion Platform into one easy-to-consume data visualization, matching them up by time. You can see in the figure above how it's easy to spot a correlated data point that is likely the cause of less-than-spectacular performance your work normally delivers. PerfStack allows you to highlight exactly the data you want to see, ignore the parts that aren't relevant, and get right to the outliers.

 

As a data professional, I'm biased, but I believe that data is the key to successful collaboration in managing complex systems. We can't manage by "feelings," and we can't manage by looking at silo-ed data. With PerfStack, we have an analytics system, with data visualizations, to help us get to the cause faster, with less pain-and-blame. This makes us all look better to the business. They become more confident in us because, as one CEO told me, "you all look like you know what you are doing." That helped when we went to ask for more resources

 

Do you have a story?

 

Later in this series, I'll be writing about the nature of collaboration and how you can benefit from shared data and analytics in delivering better and more confidence-instilling results to your organization. Meanwhile, do you have any stories of being sent on a chase to find the cause of a problem?  Do you have any great stories of bizarre causes you've found to a systems issue?

As a longtime IT professional, responding to problems in systems of one type or another has become old hat. It comes with the job description, and one tends to develop habits and methodologies early on in your career. That doesn't mean that I, or anyone else, have developed the best habits, however, and often our methods are quite ineffective indeed.

 

Standard practice in the industry is for a network operations center (NOC) to monitor some portion of the network for immediate or impending troubles. Companies spend millions of dollars on entire rooms filled with beautiful monitors mounted on walls, desks and workstations built to look as futuristic as possible, low lights at just the right hue, and comprehensive monitoring suites to keep track of it all. The trouble is, this often makes people aware of a problem, but offers nothing in the way of a troubleshooting methodology or tool to actually fix the problem.

 

As often as not, when some sort of event worth responding to grabs the attention of a NOC engineer, they either call someone or start a trouble ticket, or both. The lucky recipient of the aforementioned prodding then digs into the problem or passes it onto the next person in the chain, with each successive person having to start over in their own domain (compute, network, security, etc.) with new tools and limited information.

 

This entire approach may seem logical and even expedient, though I suspect that's largely due to a little bit of Stockholm Syndrome and the ever popular "but this is the way we've always done it" argument. I'm not saying that this is a bad approach--or at least that it hasn't always been a bad approach--given the historical dearth of cross-silo troubleshooting tools available on the market. Most of us instinctively knew that this was inefficient, but didn't have a good sense of what we could do about it.

 

Various tools and paradigms were suggested, developed, sold, and subsequently put on shelves that attempted to fix the full-stack troubleshooting void. Comprehensive network tools are one of the favorites, offering a truly staggering array of dashboards, widgets, alerts, and beautiful graphics in a noble attempt to present the most information possible to the engineers tasked with fixing the relevant problems. Many tools also exist for doing the same thing inside of virtual environments, or on storage arrays, or the cloud, etc., and many are very good at what they do. But they don't do what we need, which is to collapse the silos between IT disciplines into one unified system - until now.

 

Solarwinds NPM, part of the Orion Platform suite of products, has long been the darling of NOCs everywhere, and with good reason. It is a comprehensive and well-thought-out approach to network and systems monitoring. Collapsing the silos in IT, however, requires more than just a great tool for the NOC, or even a great tool for the network and systems teams. It

requires a tool that is not only useful for all of these teams, but preserves the chain of data (of troubleshooting) as it moves between specialties. In other words, if I'm the systems guy, I want to see the data that the network team is seeing, and the steps they've taken to resolve the problem. The thing is, I want to see it in the system, not a hastily or poorly-crafted email, which is the equivalent of tossing a flaming bag of excrement over the wall on our way out.

 

NPM 12.1 has taken a stab--a good stab--at solving these problems with the inclusion of a tool called PerfStack. I'll be exploring what this tool can do, and where in the troubleshooting process it fits, in a series of blog posts over the coming weeks. I'll likely also toss in some of my own personal horror stories of troubleshooting problems, as I've had more than my fair share in my past, and confession is cathartic. In the meantime, I'd encourage everyone to check out this already fantastic series of posts on the new tool:

 

https://thwack.solarwinds.com/community/solarwinds-community/product-blog/blog

In a recent post, @gerardodada asked, “What you would do with PerfStack?” The community provided some great ways to use this new feature. For those who don’t know what PerfStack is, I’d like to take explain further. PerfStack is  a new performance analysis visualization component of the Orion Platform that enables users to build custom dashboards of time series data. This allows end-users to troubleshoot issues quickly by pinpointing root cause. By building on Orion’s modular architecture (shown below), this new component efficiently filters through a multitude of collected metrics. It automatically understands entity dependencies from application to infrastructure, seamlessly adheres to access control rules, and presents data in a way that’s both intelligent and easy to use.

Orion’s Modular Architecture

The Orion Platform provides a set of common services, including security, UX framework, alerting, reporting, data acquisition, and data persistence. SolarWinds products NPM, NCM, NTA, SAM, VMAN, SRM, and others plug into this framework and leverage these common platform services. On top of the platform services, there are components like PerfStack and AppStack that automatically inherit other components in the Orion Platform.


PerfStack gets much of its ease of use from SWIS and the AppStack component, as discussed in the blog, A Lap Around AppStack. PerfStack extends AppStack by quickly gathering, correlating, and presenting the thousands of metrics related to this chain of entities in a logical, intuitive, and searchable fashion. 

For the visualization, PerfStack relies heavily on the UI framework, specifically two recent additions: Apollo, which is an Angular JS framework, and D3, a JS graphing library. The Apollo Angular framework renders the Metric Palette, enabling succinct navigation and coupling of statistics to graphs. D3 provides the incredible array of high resolution graphs and charts. D3 is the same technology behind the awe-inspiring topology graphs you see in NetPath.

 

PerfStack automatically inherits account limitations from the security service that restricts access only to the entities the end-user has permission to access. Custom-configured Performance Analysis dashboards can be saved via a data persistence service that allows for easy loading of common, recurring troubleshooting scenarios.

SolarWinds continues to invest heavily in Orion’s modular platform architecture, which adds functionality to existing services and new components, increasing overall functionality and scalability.  What that means for customers is that products like NPM, SAM, and others – including PerfStack – will add new features faster, with more functionality and scalability.

For a long time there has been a bit of isolation between the person referred to as a network engineer and the security guy. The security guy is nice to have around because when all else fails, you can blame the firewall. That’s right. The network is fine, but you have to talk to the security guy. The security guy won’t tell you much. Why? Because in security you’re on a need-to-know basis, and you don’t need to know.

 

Alas, the point of this blog post is not to discuss the network and security silos and the way we point fingers at different departments. This is, in part, because there’s a shift that's been happening for some time now. Network engineers have been turning into security engineers. I think there are a few reasons for this shift. Let me explain.

 

Networking People Might Be Worried About Their Jobs

For the past several years, the fundamentals of networking as a career have shifted. In times past, the network engineer role dealt with cabling up equipment, connecting cables, drawing complex diagrams of the network, configuring various features and protocols via the command line, and testing and monitoring the network. With the advent of Software Defined Networking (SDN) and Network Function Virtualization, these things are quickly disappearing. There’s not much to physically connect now (at least not as much as before), and the network is building itself dynamically through the use of software tools, controllers, and so on.

 

Security Has Some Exciting Aspects that Challenge Network Engineers

It’s true that security elements like a firewall and an IPS can be set up using a controller, and that there’s a lot that software can do. However, there’s an aspect of forensics that adds an exciting aspect to network security that may be appealing to network engineers. As network automation takes more of a hold, network engineers are taking up coding and using languages like Python and such. Learning these languages actually helps in the transition to security. Decoding the activity of a would-be attacker and figuring out what their script does and how to put a stop to it is a mind exercise that’s appealing to network engineers.

 

Security Certifications Look Good on a Resume

Another fact that lends itself to the appeal is the fact that employers are looking for cybersecurity professionals, and there’s money in that work. Forbes said that there were 1 million cybersecurity jobs available in 2016. Information Week says that intrusion detection, secure software development, risk mitigation, cloud security, network monitoring and access management, security analysis, and data security skills are all in high demand. Putting these certifications on a resume increases your chances of landing a high-paying job in the security space.

 

So How Do I Keep Up?

It is true that it's hard to keep up these days. In this series of articles, I’ll tackle some areas that I think are helpful in breaking into the cybersecurity space as a network engineer. In the next article, I'll look at the mound of security certifications that are available, and discuss which ones are of most value. After that we will dive into the world of security in the social space. Believe it or not, there’s a large segment of security professionals that don’t engage in social communities like Twitter and Facebook. We’ll talk about why, and highlight some of the more vocal security folks out there. Then in our fourth post, I'll cover Cisco’s annual security report and why you should care to read it. Finally, we’ll wrap up the segment by discussing the transition into a security role and where to go from there.

 

I look forward to the comments and perhaps even questions that I can address in these future articles. Until then, happy labbing!

This week's Actuator is coming to you from Darmstadt, Germany, where I am attending the SQL Konferenz. I had one session yesterday and will be delivering another one tomorrow. So, there's still time to make your way over to Darmstadt and talk data with me and datachick

 

As always, here's a bunch of links I found on the Intertubz that you may find interesting. Enjoy!

 

How to Spot Visualization Lies

Because people, and statistics, can sometimes stretch the truth, articles like this are a good reminder for everyone to ask questions about their data.

 

Zuckerberg shows off Oculus gloves for typing in VR

Including this because the real point is buried at the bottom: "Oculus will close 200 of its 500 demo stations inside Best Buy stores, after some stations went days without use." Maybe Oculus should focus on why people have lost interest in their product. I'm guessing it wasn't because they needed gloves.

 

Visual Studio 2017 Arrives On March 7

A big milestone for Microsoft is right around the corner. I can recall when I first opened up Visual Studio for the first time in order to create some ASP.NET applications. Can't believe it's been 20 years.

 

A rash of invisible, fileless malware is infecting banks around the globe

Oh, joy. Hey, maybe we'll get lucky and it will add money to our accounts.

 

Who's watching who? Vizio caught spying, FTC says

I am someone that doesn't mind the collecting of anonymous data in an effort to make products and services better for customers. But what Vizio did here was awful. Not only was this done without consent, the data was not anonymous, and Vizio made money by selling this data to other parties. And as I sit here typing this all I can think about is how Vizio is just a company that got caught, there are likely many more examples out there that we don't know about.

 

Oracle Settlement Puts Focus on Cloud Revenue Claims

I wish I could say I'm shocked that Oracle would resort to such tactics, but then I remembered they once told everyone how their database was "unbreakable," and called SQL Server In-Memory OLTP "vaporware." So yeah, the details of this case aren't that surprising.

 

WordPress Bug Allows Hackers to Alter Website Content

Including this as a warning to anyone out there running WordPress: you should update to the latest build before you get hacked. I noticed a handful of blogs last week had been defaced.

 

One thing I enjoy about visiting Germany is seeing how they solve problems in unique ways:

 

LJJX4885.jpeg

You may be a Network Administrator for a small law office, or a quiet small-town school district, or even a midsize enterprise with three or four offices scattered throughout a relatively small geographical area. Have you ever stopped and wondered if a cyber-attack was something you had to be concerned with? You’re not a high value target, right? Not a major financial institution, an ISP, a high profile cloud service provider, or any kind of organization that someone would want to target with an attack, so it can’t (or won’t) happen to you, right?

 

Perhaps. You might go your entire career without having experienced the crippling impact of a DDoS attack on your infrastructure, or you might learn the hard way that even the most inconspicuous network can be prone to the ripple effect some of these attacks can generate.

 

"The Internet is a series of tubes..." - the famous quote by former United States Senator Ted Stevens, gave an interesting (if incorrect) laypersons perspective on what the Internet was. What it didn't do was highlight the complexity of the interconnections, and how, despite how enormous it is, everything within it is closely related. Six degrees of the Internet, maybe?

 

The recent attack on Dyn is the perfect example of this. While your network may not have been the actual target of that attack, if you were a Dyn customer, you certainly felt its effects. External services, websites, email, etc. that relied on Dyn for DNS, all intermittently reachable. Seemingly, random websites reported as down or unreachable. No indications of a problem on your own infrastructure, links aren’t saturated, and no packet loss or latency found.

 

Then the news finally reaches you, maybe a Tweet, or someone in a Slack channel posts a link to a report of the problem. The news spreads of an ongoing attack on a major DNS provider...your DNS provider. Now it all makes sense, and now, you are officially he victim, albeit indirectly, of a massive DDoS attack.

 

Don't feel too bad. Other victims included Twitter, Spotify, and Reddit, among thousands of others.

 

While you may not be a high-value target, some of the critical services you rely on are. Especially as these attacks continue to exploit and target simple services, lower down in the stack. Things like DNS, FTP, and NTP. Services almost all networks rely on to a certain degree, and are common enough to be able to cripple almost anywhere, and anytime, with far-reaching impact.

 

Nobody is safe. (Queue dramatic music)

 

That is a huge flaw in something so intrinsic to our daily lives, both personally and professionally. We rely on our networks and the Internet, and when something so simple can interrupt service, it highlights some major problems with the foundation of what we have built.

 

So, the Internet is broken. Who (or what) is going to fix it?

 

Can it be fixed?

By Joe Kim, SolarWinds Chief Technology Officer

 

In the past, cybersecurity threats were thought to come solely from malicious activity outside an organization, so agencies focused on protecting sensitive information from foreign governments, hackers, and more. Today, however, careless or untrained employees are just as dangerous to network security as malicious threats.

 

In fact, according to the results of SolarWinds' third annual federal Cybersecurity Survey, 48 percent of federal IT pros cited careless or untrained insiders as one of the greatest sources of IT security threats to their agency -- the third consecutive year (2014-2016) insider threats topped the list.  Most recently those insiders tied foreign governments as the greatest security threat for federal agencies. Many security breaches were also reported to have been caused by human error, phishing, and malware.

 

Sources of security threats

 

 

2014

2015

2016

Careless/untrained insiders

42%

53%

48%

Foreign governments

34%

38%

48%

General hacking community

47%

46%

46%

 

For federal security pros, this means that protecting the network has become much harder. Not only must agencies continue to mitigate threats from foreign governments and hacktivists, but they must also protect the network from agency personnel, which can be a far more unpredictable challenge.

 

Expecting the unexpected

 

User error is nothing new. Federal IT pros have been dealing with this since the first bits passed over the first pulled wires. The challenge is that careless users are not getting any more careful, yet the information and data they can access has become much more personal and, in some cases, critical to the agency mission.

 

What’s the solution? While there is no one single answer, federal IT pros have found that a combination of products presents a formidable security barrier. In fact, most respondents to the aforementioned survey said they use an average of five different tools in conjunction to get the job done. Among the most valuable solutions cited were:

 

  • Smart card/common access card
  • Identity and access management
  • Patch management
  • Configuration management
  • Security information and event management (SIEM)
  • Web application management

 

Of these tools, users reported the following three as being particularly effective:

 

  1. Patch management software decreased the time to detect and respond to IT security incidents. Agencies using patch management software are far more likely to detect -- within minutes -- rogue devices, denial of service attacks and unauthorized configuration changes.
  2. Configuration management software also cut response time for security incidents.
  3. SIEM tools helped agencies detect phishing attacks within minutes as well as almost all threats presented within the survey.

 

At the end of the day, federal IT pros understand that users will not change, and threats will continue to escalate. The solution is to evolve agency IT security practices to expect the unexpected and implement the most effective combination of tools to create the strongest security posture possible.

 

Find the full article on Government Computer News.

You shouldn't be running unpatched versions of SQL 2000. That's what you need to know.

 

First reported back in 2002, the SQL Slammer virus caught fire in January of 2003, and spread worldwide. It wasn't much more than a nuisance—it merely propagated itself and brought networks to a crawl. The worm could have done much more damage, considering that many of those same instances had blank 'sa' passwords (by default!).

 

But all of that is in the past. Here's what you need to know about SQL Slammer today.

 

First, this worm infects unpatched SQL 2000 and MSDE instances only. About a week ago, I would have thought that the number of such installs would be quite small. But the recent uptick in Slammer tells me that there are enough of these systems to make Slammer one of the top malware detected at the end of 2016. And a quick search at Shodan shows thousands of public-facing database servers available. And if you want to have some real fun at Shodan®, Ian Trump (phat hobbit ) has a suggestion for you.

 

Second, old worms never die; they get saved and re-used every now and then by attackers. Some software vendors get a bit lazy, maybe even have some hubris in thinking they don't need to protect themselves from old attack vectors. Attackers know that vendors are maybe lazy, so they will routinely try old exploits just to see what they can find. Right now, they are finding a bunch of unsecured instances of SQL 2000. This does beg the question: "why?" Perhaps they are just poking around in an effort to distract us from something else. Or maybe they are delivering a modified payload we don't know about yet. With so many legacy systems out there, it's hard to tell what is the real target.

 

Third, it's quite possible we are simply seeing IoT devices (or vending machines, maybe POS devices, or remote kiosks in industries like healthcare), which are running older versions of MSDE. Perhaps a vendor thought "hey, I can use this old version of MSDE for free" and shipped a few thousand units in the past year, and now suddenly we see the Slammer attack uptick. This may not be a targeted attack... yet. But it almost certainly has been noticed as a possible attack vector by now.

 

Here's what you can do right now.

 

  1. Stop using old software. Yes, I know that the software works. And it's cheap. It's also less secure than you might think. Slammer is just one of the known holes. Imagine the number of holes we haven't learned about yet.
  2. Patch the software you do have. I know companies like to hold off on patching systems for a period of time. Microsoft® issued a patch for Slammer a full six months before the attacks really started. There's no excuse for any company to have waited so long to apply the patch.

  3. Read this security bulletin. It's old, but it provides details on the Slammer worm, how it works, and what it can do to your systems.
  4. Review your use of firewalls, ACLs, and default ports. Do everything you can do to limit your exposure. Even if you can't upgrade to newer versions of SQL, or patch the one you have, you can use other methods to minimize your risk of a breach.

 

Lastly, I will leave you with this thought from Ian in an email exchange we had regarding this post:

 

"It’s no coincidence this attack is taking place, as there recently was exploit code for a Windows® SMB attack; perhaps the return of “The Slammer” is a great way to identify at-risk, legacy systems, which a myriad of unpatched Windows® exploits exist for. It’s unlikely you are running SQL 2000 on Windows® Server 2016—great way to get a nice list of targets if the spread of “The Slammer” has a command and control element identifying who’s been infected."

 

There's a lot of bad actors out there. Don't think your public-facing database server isn't at risk. Any piece of information you provide to a hacker allows them to learn ways to possible attack someone else.

 

We are all in this together.

What’s an IT seagull? An IT seagull is a person who swoops into projects, takes a dump on everything, then flies off, leaving the rest of the team to clean up the mess. We all know an IT seagull. They tend to be in management positions. Heck, we might even be guilty of being IT seagulls ourselves every once in a while. So how does one prevent IT seagulls from wreaking havoc on mission-critical projects?

 

First, stick to the data – specifically, time series data that can be correlated to root cause issues. The key to troubleshooting is to quickly surface the single point of truth to eliminate the IT blame game. This effectively nullifies IT seagulls with data that clearly shows and correlates cause and effect.

 

Second, collaboration tends to deter IT seagulls. Being able to share an expert’s point of view with another subject matter expert to give them specific insights to the problem is powerful when you are trying to quickly remediate issues across multiple stacks because it allows decisive action to take place.

 

Third, by focusing on the connected context provided by time series data that cuts across the layers of the entire stack, teams can eliminate the IT seagull’s negative potential, even as they are busy dropping gifts that keep on giving from the skies.

 

Do you know any IT seagulls? Share your stories and how you overcame them in the comments below.

In part one, I outlined the differentiations between a traditional architecture and a development oriented one. We’ve seen that these foundational and fundamental differences have specific design approaches that really can significantly impact the goals of a team, and the approaches of the sysadmin to support them. But, what we haven’t addressed are some of the prerequisites and design considerations necessary to facilitate that.

 

Let me note that DevOps as a concept began with the operations team creating tools within newer scripting and programming languages all centered around the assisting of the facilities and sysadmin groups to support the needs of the new IT. Essentially, the question was: “What kinds of applications do we in the infrastructural teams need to better support the it needs as a whole. Many years ago, I had the great pleasure to work on a team with the great Nick Weaver ( @LynxBat ). Nick had an incredible facility to imagine tools that would do just this. For example, if you’ve a requirement to replicate an object from one data center to another, which could be a simply Push/Get routine, Nick would tell you that you’re doing it wrong. He’d create a little application which would be customized to accomplish these tasks, verify completion, and probably give you a piece of animation while it was taking place to make you feel like you’re actually doing something. Meanwhile, it was Nick’s application, his scripting, that was doing all the heavy lifting.

 

I’ll never underestimate, nor appreciate the role of the developer more than the elegance with which Nick was able to achieve these things. I’ve never pushed myself to develop code. Probably, this is a major shortfall in my infrastructural career. But boy howdy, do I appreciate the elegance of well written code. As a result, I’ve always leaned on the abilities of my peers who have the skills in coding to create the applications that I’ve needed to do my job better.

 

When a sysadmin performs all of their tasks manually, no matter how strong the attention to detail, often, mistakes get made. But when the tools are tested, and validated, running them should be the same across the board. If some new piece of the infrastructure element must be included, then of course, the code must know about it, but still, the concept remains. 

 

So, in the modern, Cloud Based, or even Cloud Native worlds, we need these tools to keep ourselves on-top of all the tasks that truly do need to be accomplished. The more that we can automate, the more efficient we can be. This means everything from deploying new virtual machines, provisioning storage, loading applications, orchestration of the deployment of applications toward cloud based infrastructures (Either hybrid, on premises, or even public cloud based). In many cases, these framework types of orchestration applications didn’t exist. To be able to actively create the tools that would ensure successful deployments has become mission critical.

 

To be sure, entirely new pieces of software are often created that are now solving these problems as they arise, and many of these are simple to use, but whether you buy something off the shelf, or write it yourself, the goals are the same. Help the poor sysadmin to do the job they want to do as seamlessly, and efficiently as possible.

Getting ready to head to Germany and SQL Konferenz next week. This will be the fourth consecutive year for me at SQL Konferenz. I enjoy the event and visiting Darmstadt, home of the European Space Agency. (Canada is a member of the ESA despite not being in Europe, and they say Americans are geographically challenged, but whatevs). This year I will have two sessions. Together with Karen López datachick, we will be offering a full training day session on February 14th titled “Advanced Accidental Database Design for SQL Server." The other session I have is on February 16th and is titled “Upgrading to SQL Server 2016." If you do attend, please stop by and say hello and ask me about the Super Bowl.

 

As always, here's a bunch of links I found on the Intertubz that you may find interesting. Enjoy!

 

Hunting for evidence, Secret Service unlocks phone data with force or finesse

Never mind getting Apple to help unlock a phone; the government can always just take it apart if needed.

 

After a decade of silence, this computer worm is back and researchers don't know why

Ah, memories. I recall when and where I was when I found out Slammer had made its way inside our systems. The classics always find their way back.

 

Just One in Four Banks Confident of Breach Detection

The real number is probably closer to zero than one. And I don't want my bank to just detect a breach, I want them to prevent a breach.

 

The Midlife IT Career Crisis

I found this interesting enough to share with anyone that has 15+ years in IT already and is noticing all the new shiny things being used by those darn kids coming out of college.

 

Compliance as code

This is one of those things that sounds great in theory, but not so great in practice. Humans will always find a way around the rules. Always.

 

Ransomware Freezes Eight Years of Police Evidence

This ransomware business will get worse before it gets better.

 

I'm not saying that the only reason I go to Germany is for the schweinshaxe, but it's near the top of the list:

 

IMG_1986.JPG

By Joe Kim, SolarWinds Chief Technology Officer

 

There is hardly a government IT pro who has not seen sluggish applications create unhappy users.

 

Because the database is at the heart of every application, when there’s a performance issue, there’s a good chance the database is somehow involved. With database optimization methods—such as identifying database performance issues that impact end-user response times, isolating root cause, showing historical performance trends, and correlating metrics with response time and performance—IT managers can speed application performance for their users.

 

Start with these four database optimization tips:

 

Tip #1: Get visibility into the entire application stack

The days of discrete monitoring tools are over. Today’s government IT pros must have visibility across the entire application stack, or the application delivery chain comprising the application and all the backend IT that supports it (software, middleware, extended infrastructure and especially the database). Visibility across the application stack will help identify performance bottlenecks and improve the end-user experience.

 

Tip #2: See beyond traditional infrastructure dashboards

Many traditional monitoring tools provide a dashboard focused on health and status, typically featuring many hard-to-interpret charts and data. In addition, many of these tools don’t provide enough information to easily diagnose a problem, particularly a performance problem.

 

Tools with wait-time analysis capabilities can help IT pros eliminate guesswork. They help identify how to execute an application request step-by-step and will show which processes and resources the application is waiting on. This type of tool provides a far more actionable view into performance than traditional infrastructure dashboards.

 

Tip #3: Reference historical baselines

Database performance is dynamic. It is critical to be able to compare abnormal performance with expected performance. By establishing historic baselines of application and database performance that look at how applications performed at the same time on the same day last week, and the week before that, etc. , it is easier to identify a slight variation before it becomes a larger problem. And, if a variation is identified, it’s much easier to track the code, resource or configuration change that could be the root cause and solve the problem quickly.

 

Tip #4: Align the team

An entire stack of technologies supports today’s complex applications. Despite this, most IT operations teams are organized in silos, with each person or group supporting a different part of the stack. Unfortunately, technology-centric silos encourage finger pointing.

 

A far more effective approach shares a unified view of application performance with the entire team. In fact, a unified view based on wait-time analysis will help ensure that everyone can focus on solving application problems quickly.

 

Remember, every department, group, or function within an agency relies on a database in some way or another. Optimizing database performance will help make users happier across the board.

 

Find the full article on Government Computer News.

In traditional IT, the SysAdmin’s role has been established as supporting the infrastructure in its current dynamic. Being able to consistently stand up architecture in support of new and growing applications, build test/dev environments, and ensure that these are delivered quickly, consistently, and with a level of reliability such that these applications work as designed is our measure of success. Most SysAdmins with whom I’ve interacted have performed their tasks in silos. Network does their job, servers do theirs, and of course storage has their unique responsibility. In many cases, the silos worked at cross purposes, or at minimum, with differing agenda. The rhythms of each group caused for, in many cases, the inability to deliver the agility of infrastructure that the customer required. However, in many cases, our systems did work as we’d hoped, and all the gears meshed properly to ensure our organization’s goals were accomplished.

 

In an agile, cloud native, devops world, none of these variables can be tolerated. We need to be able to provide the infrastructure to deliver the same agility to our developer community that they deliver to the applications.

 

How are the applications different than the more traditional monolithic agencies for which we’ve been providing services for so long? The key here is the concepts of containers and micro-services. I’ve spoken of these before, but in short, a container environment involves the packaging of either the entire stack or discrete portions of the app stack which is not reliant necessarily on the operating system or platform on which they sit. In this case, the x86 service is already in place, or can be delivered in generic modes on demand. The application or portions of it can be deployed as the developer creates it, and alternately, removed just as simply. Because there is so much less reliance on the virtually deployed physical infrastructure, the compute layer can be located pretty much anywhere. Your prod or dev environment on premises, your hybrid cloud provider, or even the public cloud infrastructure like Amazon. As long as the security and segmented environment has been configured properly, the functional compute layer and its location are actually somewhat irrelevant.

 

A container based environment, not exclusively different than a MicroServices based one, delivers an application as an entity, but again, rather than relying on a physical or virtual platform and its unique properties, can sit on any presented compute layer. These technologies, such as Kubernetis, Docker, Photon, Mesosphere and the like are maturing, with orchestration layers and delivery methods far more friendly to the administrator than they’ve been in the past.

 

In these cases, however, the application platform being delivered is much different than traditional large corporate apps. An old Lotus Notes implementation, for example, required many layers of infrastructure, and in many cases, these types of applications simply don’t lend themselves to a modern architecture. They’re not “Cloud Native.” This is not to disparage how Notes became relevant to many organizations. But the value of a modern architecture, the mobility of the applications, and data locales simply does not support the kinds of infrastructure that an SAP, JDEdwards, etc. type of monolithic architecture required. Of course, these particular applications are solving the cloud issues in different ways, and are still as vital to their organizations as they’d been in the past.

 

In the following 4 blog postings, I’ll address the architectural/design/implementation issues facing the SysAdmin within the new paradigm that Cloud Native brings to the table. I hope to address those questions you may have, and hope for as much interaction, clarification, and challenging conversation as I can.

At Tech Field Day 13, we gave everyone a sneak peek at some of the cool features we’ll be launching in March. If you missed the sneak peak, check out the footage below:

 

 

Our PerfStack Product Manager has also provided a detailed blog post on what you can expect from this cool feature. In the near future, look for the other Product Managers to give you more exclusive sneak peeks right here on THWACK.

 

Join the conversation

We are curious to hear more about what you think of PerfStack. After all, we used your requests to build this feature. With that said, I’ve seen several requests in THWACK expressing what you need from an Orion dashboard. I would love to hear from those of you directly in the IT trenches, specifically some ideas on how you would manipulate all this Orion data with PerfStack.

 

Personally, I would use PerfStack to visually correlate the page load times in synthetic transactions as observed by WPM with web server performance data in WPM and network latency from NetPath, and maybe storage performance from SRM to get a better understanding of what drives web server performance and what is likely to become a bottleneck that may impact end user experience if load goes up. But we want to hear from you.

 

If you had access to PerfStack right now, what would you do with it?

What interesting problems could you solve if you could take any monitoring object from any node that you are monitoring with Orion? What problems would you troubleshoot? what interesting use cases can you imagine? What problems you are facing today would it help you solve?

 

Let us know in the comments!

 

In Geek Speak(TM), THWACK(R) MVP Eric Hodeen raised the concern that networks are increasingly becoming vulnerable to outdated network devices like routers and firewalls, and even IoT devices now making their way into the workplace. (See Are Vulnerable Routers and IoT Devices the Achilles Heel of Your Network?) The reason, he writes, is that these devices are powered by lightweight and highly specialized operating systems that are not hardened and don’t always get patched when vulnerabilities are discovered. Eric then goes on to give several recommendations to defend against the risks these devices pose. In closing his article, Eric reminds us that security defenses rest on a combination of technical and procedural controls and that there is a direct correlation between configuration management and security management. He then makes the assertion that “perhaps one of the best security tools in your toolbox is your Network Configuration and Change Management software.” We couldn’t agree more. 

 

It’s for this very reason that Network Configuration Manager (NCM) continues to receive so many industry security awards. In late 2016, SolarWinds(R) NCM received two noteworthy awards for security management.

In November, SC Magazine recognized NCM as a 5-star “Best Buy” for Risk and Policy Management. In recognizing NCM, SC Magazine said:

 

“Simple to deploy, very clean to use, and SolarWinds’ decades of experience with internet-working devices—especially Cisco(R) —really shows.” 

 

  You can read the full article here

 

In December GSN recognized SolarWinds NCM as the Best Compliance/Vulnerability Assessment Solution as part of its 2016 Homeland Security Award. The GSN Awards are designed to recognize the most innovative, important technologies and strategies by U.S. and International IT and Cybersecurity companies, Physical Security companies, and Federal, State, County and Municipal Government Agencies. 

 

You can review all GSN award winners here

 

Network security and compliance is a never-ending job. Bad actors devise new hacks, new software mean new vulnerabilities, and network changes bring new security controls. Manually doing everything needed to make your network secure and compliant no longer works. This is where NCM can help you to: 

 

  • Standardize configurations based on policy
  • Immediately identify configuration changes
  • Create detailed vulnerability and compliance reports
  • Remediate vulnerabilities and policy violations

 

Using NCM is like having an experienced security engineer on staff.  So don’t just do the work of network security and compliance—automate it using NCM to get more done.  

To learn more or to try NCM for 30 days, please visit the NCM product page.

 

Do you use NCM to improve your network security and compliance?  Share your story below.

Recent news headlines report alarming intrusions into otherwise strong, well-defended networks. How did these incursions happen? Did perpetrators compromise executive laptops or critical servers? No, these highly visible endpoints are too well defended. Instead, hackers targeted low-profile, low-value network components like overlooked network routers, switches, and Internet of Things (IoT) devices.  

 

Why are network and IoT devices becoming the target of choice for skilled hackers? Two reasons. First, vendors do not engineer these devices to rigorously repel intruders. Unlike servers and desktops, the network/IoT device OS is highly specialized, which ironically may make it more difficult to secure. However, vendors do not make the effort to harden these platforms. Second, these devices are out of sight and out of mind. As a result, many of them may be approaching technical obsolescence and are no longer supported.

Many of us remember recently how the Mirai IoT botnet compromised millions of Internet-enabled DVRs, IP cameras, and other consumer devices to launch a massive distributed denial-of-service (DDoS) attack against major DNS providers to “brown out” vast regions of the internet. For many, this attack was simply an inconvenience. However, what if IoT devices or weakly defended routers and switches were compromised in a way that impacted our offices, warehouses, and storefronts? We can easily see how weak devices are targeted and compromised to disrupt commercial operations. Many companies use outdated routers, switches, and weakly secured IoT devices. So how do we protect ourselves?

 

One solution is to forbid any outside electronics into the workplace, which is my vote, though I know this is increasingly unrealistic. But “where there is a will there is a waiver” is a common response I hear. A second solution is to retire old hardware and upgrade firmware containing verified vulnerabilities. Another approach would be to design, build, and implement a separate network access scheme to accommodate IoT devices so they do not interfere with corporate network productivity. Once this network is operational, then it is the job of the corporate technology engineers and security to ensure they are used in an appropriate manner. To complement these strategies, it’s helpful to have mature change management processes, a network configuration, and a change management (NCCM) solution with EOL, vulnerability assessment, and configuration management capabilities. 

 

Fortunately, these solutions are straightforward. By using a combination of technical and procedural controls, you can alleviate much of the risk. There is a direct correlation between configuration management and security management. So in reality, one of the best security tools in your toolbox is your network configuration and change management software. Using the right tools and taking deliberate and sensible steps can go a long way to keep your company out of the headlines.

 

About the author

Eric Hodeen is a Solarwinds expert and THWACK MVP with over 20 years’ experience in network engineering with expertise in network management and operations and STIG, PCI and NIST compliance.  Eric has designed, implemented and managed networks for the Department of Defense across the US and Europe.  He earned his MS Management of Technology with a specialization in security from the University of Texas at San Antonio and holds numerous certifications including Cisco CCNA R&S, CCDA, CCNA Security, CCNP Security, Juniper JNCIA/JNCIS, ITIL V2, Security+CE and COMPTIA CASP.

By the time you read this, I will already be in Austin for Tech Field Day #13 hosted at SolarWinds. I am looking forward to attending my first ever TFD, after having virtually attended some of the previous events. I enjoy learning from the best industry experts and TFD allows for that to happen.

 

As always, here's a bunch of links I found on the Intertubz that you may find interesting. Enjoy!

 

Trump aides' use of encrypted messaging may violate records law

Leave it to our government to decide that encrypting messages somehow means they can't be recorded, despite the fact that industries such as financial services have been tracking encrypted messages for years due to SEC rules.

 

A Quarter of Firms Don’t Know if They’ve Been Breached

That number seems low. I think it's closer to 100%, because even firms that know they have been breached likely have no idea about how many breaches they have suffered.

 

Why security is best in layers

And here's the reason companies have no idea if they have been breached: they don't have the right talent in-house to discover such things. Identifying a breach takes more than just one person—it requires analysis across teams.

 

Microsoft Almost Doubled Azure Cloud Revenue Last Quarter

Articles like this remind me about the so-called "experts" out there a few years back who were dismissing the cloud as anything worthwhile. And maybe, for some workloads, it isn't. But there is one thing the cloud is right now, and that's a cash cow.

 

Monday Vision, Daily Outcomes, Friday Reflection for Remote Team Management

A must read for anyone working remotely. I've done a similar weekly report in the past, listing three things I did, three things I want to do next week, and three roadblocks.

 

Why your voice makes you cringe

Listening to me speak is awful. I don't know how anyone does it, TBH, let alone watch me.

 

Should the cloud close the front door to the database?

Why is this even a question? As a host, they MUST protect everyone, including the ill-informed customer. If the host left security up to the customer, the cloud would collapse in days.

 

Booted up in 1993, this server still runs—but not for much longer

If you replace 20% of the hardware components for a server, is it the same server as when it started? Never mind the philosophical question; this is quite a feat. And yes, I did think of Battlestar Galactica, and how the lack of upgrades proved to be the thing that saved it from destruction. I'm hoping they never turn this thing off.

 

A quick image from a flight last week—somewhere over Hoth, apparently:

hoth.jpg

Filter Blog

By date: By tag: