Skip navigation
1 2 3 Previous Next

Geek Speak

2,254 posts

The SolarWinds crew including sqlrockstar, chrispaap, and I just returned stateside after a successful jaunt across the Atlantic at VMworld Europe in Barcelona, Spain. Thank you to all of the attendees who joined us at Tom’s speaking sessions and at our booth. Thank you to Barcelona for your hospitality!

 

Below are a few pictures of the SolarWinds team as we walked the walk and talked the talk of monitoring with discipline.

 

The SolarWinds Family at VMworld Europe 2017 in BarcelonaThe SolarWinds Family Team Dinner

 

 

Our journey doesn’t stop with the end of the VMworld two-continent tour. We are about to ignite a full course of monitoring with discipline in Orlando. At Microsoft Ignite, visit us in Booth #1913 for the most 1337 swag as well as fantastic demos on monitoring hybrid IT with discipline.

Let us know in the comments if you'll be joining us in Orlando for Microsoft Ignite.

I know, I'm a day late and quite possibly 37 cents short for my coffee this morning, so let's jump in, shall we?

 

Let's start with the Equifax breach. This came up in the Shields Down Conversation Number Two, so, I thought I would invite some of my friends from our security products to join me to discuss the breach from a few different angles.

 

My take will be from a business strategy (or lack of) standpoint. Roughly 143 million people had their personal data exposed because Equifax did not properly execute a simple patching plan. Seriously?

 

Is this blog series live and viewable? I am not the only person who implements patching, monitoring, log and event management in my environments. This is common knowledge. What I don't get is the why. Why, for the love of everything holy, do businesses not follow these basic practices?

 

CIxO or CXOs do not implement these practices. However, it is their duty (to their company and their core values) to put the right people in place who will ensure that security measures are being carried out.

 

 

Think about that for a moment and then know that there was a patch produced for the vulnerability that Equifax failed to remediate in March. This breach happened, as we all know, in mid-May. Where is the validation? Where was the plan? Where is the ticketing system tracking the maintenance that should've been completed on their systems? There are so many questions, especially since this happened in an enterprise organization, not some small shop somewhere.

 

Now, let's take this another step further. Equifax dropped another juicy nugget of information of another breach in March. Don't worry, though. It was an entirely different attack. However, the incredible part is that some of the upper-level folks were able to sell their stock. That makes my heart happy, you know, to know that they had the time to sell their stock before they released information on being breached. Hat's off to them for that, right?

 

Then, another company decided they needed to market and sell credit monitoring (for a reduced fee, that just so happens to use EQUIFAX SERVICES) to the individuals who were now at a high(er) risk of identity theft and credit fraud. I'm still blown away by this.

 

Okay. Deep breath. Whooooo.

 

I was recently informed that when you have third-party software, patching is limited and that organization's SLAs for application uptime don't allow patching on some of their servers. I hear you! I am a big believer that some patching servers can cause software to stop working or result in downtime. However, this is where you have to implement a lab and test patching. You should check your patching regardless to make sure you are not causing issues with your environment in the first place. 

 

I will implement patching on test servers usually on a Friday, and then I will verify the status of my applications on the server.

I will also go through my security checks to validate that no new holes or revert have happened before I implement in production within two weeks. 

 

Now let's bring this back to the strategy at hand. When you are an enterprise corporation with large amounts of personal data belonging to your trusting customers (who are the very reason you are as large as you are), you better DARN WELL have a security plan that is overseen by more than one individual! Come on! This is not a small shop or even a business that could argue, "Who would want our customer data?" We're talking about Equifax, a company that holds data about plenty of consumers who happen to have great credit. Equifax is figuratively a lavish buffet for hackers.

 

The C-level of this company should have kept a close eye on the security measures being taken by the organization, including patching, SQL monitoring, log, events, and traffic monitoring. They should have known there were unpatched servers. The only thing I think they could have argued was the common refrain, "We cannot afford downtime for patching." But still. 

 

Your CxO or CIxO has to be your IT champion! They have to go nose to nose with their peers to make sure their properly and thoroughly designed security plans get implemented 100%. They hire the people to carry out such plans, and it is their responsibility to ensure that it gets done and isn't blocked at any level.

 

Enough venting, for the moment. Now I'd like to bring in some of my friends for their take on this Equifax nightmare that is STILL unfolding! Welcome joshberman, just one of my awesome friends here at SolarWinds, who always offers up great security ideas and thoughts.

 

Dez summed up things nicely in her comments above, but let's go back to the origins of this breach and explore the timeline of events to illustrate a few points.

 

  • March 6th: the exploited vulnerability, CVE-2017-5638, became public
  • March 7th: Security analysts began seeing attacks propagate that were designed to exploit this flaw
  • Mid-May: Equifax tracked the date of compromise back to this window of time
  • July 29th: the date Equifax discovered a breach had occurred

 

Had a proper patch management strategy been set in place and backed by the right patch management software to enable the patching of third-party applications, it is likely that Equifax might not have succumbed to such a devastating attack. This applies even if testing had been factored into the timelines, just as Dez recommends. "Patch early, patch often" certainly applies in this scenario, given the voracious speed of hackers to leverage newly discovered vulnerabilities as a means to their end. Once all is said and done, if there is one takeaway here it is that patching as a baseline IT security practice, is and will forever be a must. Beyond the obvious chink in Equifax's armor, there is a multitude of other means by which they could have thwarted this attack, or at least minimized its impact.

 

That's fantastic information, Josh. I appreciate your thoughts. 

 

I also asked mandevil (Robert) for his thoughts on the topic. He was on vacation, but he returned early to knock out some pertinent thoughts for me! Much appreciated, Robert!

 

Thanks, Dez. "We've had a breach and data has been obtained by entities outside of this company."

Imagine being the one responsible for maintaining a good security posture, and the sinking feeling you had when these words were spoken. If this is you, or even if you are tangentially involved in security, I hope this portion of this post helps you understand the importance of securing data at rest as it pertains to databases.

 

Securing data in your database

 

The only place data can't be encrypted is when it is in cache (memory). While data is at rest (on disk) or in flight (on the wire), it can and should be encrypted if it is deemed sensitive. This section will focus on encrypting data at rest. There are a couple different ways to encrypt data at rest when it is contained within a database. Many major database vendors like Microsoft (SQL Server) and Oracle provide a method of encrypting called Transparent Data Encryption (TDE). This allows you to encrypt the data in the files at the database, table space, or column level depending on the vendor. Encryption is implemented using certificates, keys, and strong algorithms and ciphers.

 

Links for more detail on vendor TDE description and implementation:

 

SQL Server TDE

Oracle TDE

 

Data encryption can also be implemented using an appliance. This would be a solution if you would want to encrypt data but the database vendor doesn't offer a solution or licensing structures change with the usage of their encryption. You may also have data outside of a database that you'd want to encrypt that would make this option more attractive (think of log files that may contain sensitive data). I won't go into details about different offers out there, but I have researched several of these appliances and many appear to be highly securitized (strong algorithms and ciphers). Your storage array vendor(s) may also have solutions available.

 

What does this mean and how does it help?

 

Specifically, in the case of Equifax, storage level hacks do not appear to have been employed, but there are many occurrences where storage was the target. By securing your data at rest on your storage tier, it can prevent any storage level hacks from obtaining any useful data. Keep in mind that even large database vendors have vulnerabilities that can be exploited by capturing data in cache. Encrypting data at the storage level will not help mitigate this.

 

What you should know

 

Does implementing TDE impact performance? There is overhead associated with encrypting data at rest because the data needs to be decrypted when read from disk into cache. That will take additional CPU cycles and a bit more time. However, unless you are CPU-constrained, the impact should not be noticeable to end-users. It should be noted that index usage is not affected by TDE. Bottom line is if the data is sensitive enough that the statement at the top of this section gets you thinking along the lines of a resume-generating event, the negligible overhead impact of implementing encryption should not be a deterrent from its use. However, don't encrypt more than is needed. Understand any compliance policies that govern your business (PCI, HIPAA, SOX, etc.).

 

Now to wrap this all up.

 

When we think of breaches, especially those involving highly sensitive data or data that falls under the scope of regulatory compliance, SIEM solutions certainly come to mind. This software performs a series of critical functions to support defense-in-depth strategies. In the case of Equifax, their most notable influence appears to be their attempt to minimize the time of detection with either the compromise or the breach itself. On one hand, they support the monitoring and alerting of anomalies on the network that could indicate a compromise. On the other, they can signal the exfiltration of data – the actual event of the breach – by monitoring traffic on endpoints and bringing to the foreground spikes in outbound traffic, which, depending on the details, may otherwise go unnoticed. I'm not prepared to make the assumption that Equifax was lacking such a solution, but given this timeline of events and their lag in response, it begs the question.

 

As always, thank you all for reading and keep up these excellent conversations.

Back from Barcelona and VMworld Europe. In the past three weeks, I've logged 14k miles, on 10 flights, and delivered three sessions, and had two sessions listed in the Top Ten for each event. And in a week, I get to do it all over again at Microsoft Ignite. If you are heading to Ignite, stop by the booth and say hello. We've got plenty of stickers and buttons and maybe a few pairs of socks, too.

 

As always, here is a bunch of links I hope you will find interesting!

 

Atlanta Tests Self-Driving Vehicle In Heart Of The City

It's been a while since I've posted about autonomous vehicles, so I decided to fix that.

 

Self-driving trucks enter the fast lane using deep learning

And then I decided to double down by sharing info about a self-driving truck. I love living in the future! Next we'll have self-tuning databases!

 

Oracle preps autonomous database at OpenWorld, aims to cut labor, admin time

And there it is, the other shoe dropping. Microsoft, AWS, and now Oracle are all on the AI train and offering self-tuning systems. If you are an operational DBA, it's time to think about a pivot.

 

Azure Confidential Computing will keep data secret, even from Microsoft

Microsoft continues to make progress in the area of data security, because they know that data is the most critical asset any company (or person) owns.

 

Understanding the prevalence of web traffic interception

And this is why the Microsoft announcement matters more than any AWS product announcement. Faster storage and apps don't mean a thing if your data is breached.

 

Google Parent Alphabet To Consider $1 Billion Investment In Lyft

Goodbye Uber, I won't miss you, your questionable business practices, or your toxic work culture.

 

Identity Theft, Credit Reports, and You

Some advice for everyone in the wake of the Equifax breach: be nice until it's time to not be nice.

 

It's been a fun two weeks, but you are looking at two very tired friki cabezas atop the Arenas de Barcelona:

Next week I'm flying down to Orlando, Florida to spend a week filling my brain with all things MS: Ignite. I'm especially excited about this trip because it is one of the first times as a Head Geek that I'm traveling to a server- and application-centric show.

 

To be sure, we get our fair share of questions on SAM, DPA, SRM, WPM, VMAN, and the rest of the systems-side of the house at shows like Cisco Live, but I'm expecting (and looking forward to) a whole different class of questions from a convention that boasts 15,000+ folks who care deeply about operating systems, desktops, servers, cloud, and applications.

 

There are a few other things that will make next week special:

          • There. Will. Be. SOCKS. It was a hit at Cisco Live. It was even more of a hit at VMWorld. And now the THWACK socks are ready to take MS: Ignite (and your feet) by storm. We can't wait to see the reactions to our toe-hugging goodness.
          • SWUGLife on the beach: For the second year in a row, Ignite will play host to the most incredible group of users ever assembled: the illustrious, inimitable SolarWinds User Group (or SWUG for short).
          • Geek Boys, Assemble!:  For the first time ever, Patrick, Tom, Kong, and myself will all be at the same show at the same time. It obviously a crime that Destiny couldn't join in the fun, but somehow I think she'll find a way to be with us in spirit. And of course we can just consider this a prelude to the all out Geeksplosion next month at THWACKcamp.
          • THERE. WILL. BE. EBOOKS.: For several weeks, I've been busy crafting the second installment in the SolarWinds Dummies series: Systems Monitoring for Dummies. While you can download it now from this link, we'll also have handouts at the booth to let all of the Ignite attendees know about it, marking the book's first live appearance on the show floor.


But that's more or less the view from show floor. There are things that I'm eager to experience beyond the booth border (#1913, for those who will be in the neighborhood).

 

Tom will be giving two different talks, both of which I personally want to hear about: "Upgrading to SQL Server 2016" is going to be packed full of information and one of those sessions where you'll either want a keyboard or a recorder to get all the details. But "When bad things happen to good applications" promises to be classic SQLRockStar in action. For that one, I plan to bring a lighter for the encore number at the end.

 

I also am very eager to get out and see what Microsoft is all about these days. Sure, I use it on the desktop, and I read the news, and I'm friends with an ever-growing number of folks who work in Redmond. But shows like these are where you get to see the aspirational side of a company and it's technology. Ignite is where I will get to see who Microsoft WANTS to be, at least in the coming year.

 

That aspirational quality will be on display nowhere as much as the keynote talk by Satya Nadella on Monday. Look for me to be live-tweeting at that event, at the very least.

 

Stay tuned for my follow-up log in two weeks, which I expect will be full of unexpected discoveries, a few food-related pictures, and hopefully a few shots of the SolarWinds friends we met up with while we were there.

By Joe Kim, SolarWinds EVP, Engineering and Global CTO

 

My colleague Patrick Hubbard made some interesting predictions about 2017 for government IT professionals, and how DevOps culture could change the landscape. I’d like to share them with you now, as we approach Q4, to see how his predictions have played out so far.

 

If there is one thing government organizations are used to, it’s change. Budgets, technology, and policies are constantly changing, and there’s no surprise that the IT professional’s role is constantly evolving.

 

Not only do government IT professionals have to deal with the usual difficulties of trying to keep up with new technology, such as cloud, containers, microservices, and the Internet of Things (IoT), they also need to deal with budget cuts, restrictive policies, and a lack of resources. It is now more important than ever to scrap the traditional siloed IT roles, such as network, storage, and systems administrators.

 

A general, holistic approach to government IT

Having generalists is particularly important within government IT, where resources and budgets may be stretched. The ability to have a holistic understanding of the IT infrastructure and make quick and informed decisions is crucial over the next year and beyond.

 

2017 is likely to bring new machine-based technologies and the continued adoption of DevOps, which encourages collaboration between siloed IT departments. Government IT professionals need to expand their viewpoints to focus on tools and methodologies they may not be immediately familiar with to prepare for and manage next-generation data centers.

 

Leave the automation to machines

As predicted, new machine-based technologies are going to become better and more sophisticated over time. Before technology, such as bots and artificial intelligence, is leveraged, new management and monitoring processes will need to be introduced to government organizations.

 

DevOps culture is coming

DevOps describes the culture and collaboration of the development and operations teams that is geared toward software development. The transition to DevOps is certainly not without its challenge, however. By leveraging these principles, government organizations can be well on their way to reap the benefits of an integrated DevOps mentality.

 

DevOps is a positive organizational movement that will help government organizations empower IT departments to innovate. It also has the potential to improve agility, deliver innovation faster, provide higher quality software, better align work and value, and give the ability to respond to problems or changes.

 

The role of the government IT professional is constantly evolving. Since the good old days, when IT pros did little more than assist when emails stopped working, they now have much more power to shape the wider business strategy due to the reliance on technology for everyday tasks. By staying relevant and maintaining a general knowledge across the entire IT infrastructure, embracing a collaborative DevOps culture, and being open-minded to the integration of machines, government IT professionals will find themselves prepared for the changes that are coming their way.
 

Find the full article on Adjacent Open Access.

I have, on the occasional gray and challenging day, suggested that the state of IT contains a measure of plight. That it’s beset on all sides with the hounds of increasing complexity, reduced budgets, impossible business demands, and unhappy users. Fortunately for us all, however, being an IT professional is, as it has always been, pretty freaking awesome. Would I really make a major career shift away from casting spells on hardware, writing a little code to automate drudgery, and generally making the bezels blink? Nope. I’m not saying I’d do it for free, but after all these years I still have moments where it’s hard to believe I get paid to do what I love. And again this year we get to celebrate IT pro day.

 

There’s a certain special comradery among IT professionals that runs deeper than merely sharing a foxhole in the server room. It’s celebrating the power-on of a new data center. It's the subdued antithesis of the sales dude-style slap-on-the-back compliment when we finally identify a bedeviling, intermittent root cause. It's mostly cultural. Our ilk actually cares about users, and for many it’s what brought them to technology in the first place. We solve problems they didn’t know they had in ways they wouldn’t understand, avoiding the adulation usually reserved for such heroics. Aww, shucks. We're just here to do what we can. #humblebrag

 

This year, share the fun and recognition of IT Pro Day with a friend. Take an extra-long lunch, and remember you’re the one that keeps the bezel lights on. We don’t just keep our businesses working, we affect the human experiences and the feelings of our users. With a little time and the right questions, we truly make the world a better place.

 

So have a little fun, send some eCards, and maybe appear in next year’s video here: http://www.itproday.org

 

Happy IT Pro Day!

Data is the new gold to be mined, analyzed, controlled, and wielded to create disruption. The value of data-driven decisions is guiding the next generation of services. Data can be utilized to frame any story, but you should avoid being framed by your data.

 

Join me, fellow Head Geek Kong Yang, and industry experts Stephen Foskett and Karen Lopez for "Optimizing the Data Lifecycle" a discussion of the challenges presented by the data-driven era. In this panel discussion, we will also share best practices to optimize the consumption of that data.

 

THWACKcamp is the premier virtual IT learning event connecting skilled IT professionals with industry experts and SolarWinds technical staff. Every year, thousands of attendees interact with each other on topics like network and systems monitoring. This year, THWACKcamp further expands its reach to consider emerging IT challenges like automation, hybrid IT, cloud-native APM, DevOps, security, and more. For 2017, we’re also including MSP operators for the first time.

THWACKcamp is 100% free and travel-free, and we'll be online with tips and tricks on how to your use SolarWinds products better, as well as best practices and expert recommendations on how to make IT more effective regardless of whose products you use. THWACKcamp comes to you so it’s easy for everyone on your team to attend. With over 16 hours of training, educational content, and collaboration, you won’t want to miss this!

 

Check out our promo video and register now for THWACKcamp 2017! And don't forget to catch our session!

IT organizations are embracing hybrid IT because these services and technologies are critical to enabling the full potential of an application’s disruptive innovation. Although change is coming fast, the CIO’s mission remains the same: keep the app healthy and running smoothly. It’s time for application performance management to extend its strategy and practice to handle the modern application’s needs.

 

In the "Extend Your Modern APM Strategy" session, I will be joined by a panel of SolarWinds product experts, including Jerry Schwartz, director of product marketing, Robert Mandeville, product marketing manager, product managers Steven Hunt, Chris Paap, and Chris O'Brien, and Dan Kuebrich, director of engineering. We will explore the five elements of a modern approach to APM, including product demonstrations. The session will cover concepts from WPM to response-time analysis. After attending this session, you will have a better understanding of what an APM approach entails and the technologies that are available to support each of the five fundamental aspects of APM.

 

After attending this session, you will have a better understanding of what a comprehensive APM strategy entails and what technologies are available to support each of the five elements.

 

THWACKcamp is the premier virtual IT learning event connecting skilled IT professionals with industry experts and SolarWinds technical staff. Every year, thousands of attendees interact with each other on topics like network and systems monitoring. This year, THWACKcamp further expands its reach to consider emerging IT challenges like automation, hybrid IT, cloud-native APM, DevOps, security, and more. For 2017, we’re also including MSP operators for the first time.

THWACKcamp is 100% free and travel-free, and we'll be online with tips and tricks on how to your use SolarWinds products better, as well as best practices and expert recommendations on how to make IT more effective regardless of whose products you use. THWACKcamp comes to you so it’s easy for everyone on your team to attend. With over 16 hours of training, educational content, and collaboration, you won’t want to miss this!

 

Check out our promo video and register now for THWACKcamp 2017! And don't forget to catch our session!

>register now

sqlrockstar

Thank You, IT Pros

Posted by sqlrockstar Employee Sep 14, 2017

They work in mystery, toiling away at all hours. Nobody ever sees them working, but many are happy with the results. And if anyone tries to reproduce their work, they end up disappointed. No, I’m not talking about the Keebler® Elves, although I suppose there are some similarities between these two groups of workers. Both are overworked, underpaid, and no one understands how they do their job so well.

 

I am talking about IT professionals, the unsung heroes of modern corporate enterprises around the globe. Except they are no longer unsung because back in 2015, SolarWinds created IT Pro Day! Created by IT professionals for IT professionals, IT Pro Day happens on the third Tuesday of September each year. IT Pro Day serves as a great reminder about all the work that goes on behind the scenes.

 

Here’s some data for you to think about this IT Pro Day:

 

  • IT pros spend 65% of their time managing IT and IT-related services
  • Nearly half of that time (47%) is dedicated to resolving technology issues for senior executives/chief officers

 

Let that sink in for a minute. Most of our time is spent catering to executives. You would think that the executives would appreciate all this effort, right? Maybe not:

 

  • 61% of IT pros are concerned about job security, with almost half (42%) suggesting the key reason is that company leadership does not understand the importance of IT

 

Okay, so maybe the executives appreciate the effort, but IT pros don’t believe that the executives understand the importance of IT. Which only seems odd when you find out everyone else does understand the importance:

 

  • 63% of end-users agree that IT has a greater impact on their daily work lives than the executives in the corner office

 

I’ve always thought most executives started out as regular employees. I guess I was wrong, because if that were true, then the above numbers would be different. And so would this one:

 

  • 91% of IT pros surveyed work overtime hours, and of those, 57% do so with no compensation for working overtime

 

Lots of overtime, for people who don’t understand the importance of quality IT staff. Overworked. Underpaid. And no one can explain what it is they do for work. But we are dedicated to making things better for others:

 

  • 25% of IT professionals agree that half of the time, end-users who try to solve their own IT problems ultimately make things worse

 

Okay, so making things better for others also makes things better for us. But IT pros aren’t just looking out for the people (or themselves). They’re also looking out for the business:

 

  • 89% of IT professionals most fear a security breach

 

Somehow, all this data makes sense to me. I understand each data point because I have lived each data point. I am an IT pro, and damn proud to say that to anyone who cares to listen. Oh, who am I kidding? I’ll say it even to people who don’t care to listen.

 

IT pros don’t do this for money. We aren’t interested in that. (But it’s nice, don’t get me wrong, and here’s hoping someone in the corner office on the fourth floor sends me bacon for Christmas this year.) We truly love what we do for work:

 

  • 94% of IT pros say they love what they do

 

Here’s to you, IT pro. Enjoy your day. Walk with your head held high. Smile a few seconds longer when an executive asks you to fix their kid’s iPad®.

 

You’ve earned this day, and many more.

 

Thank you.

 

 

Starting out in IT there are many things that I wish I had known about, but one of them is the value of the soft skills required.  Organizations want people who are willing to learn with the proper drive, but the ability to communicate, support, empathize, and the ability to help other people in the business will go a long way for your success within any enterprise.

 

Finding a Job

 

Over the years I have spent in the field I have been on both the interviewing side and the interviewee side of the table.  I have found that it always starts with how you related to others and whether or not you can have a real conversation with the person you are talking to. I have met people during the interviewing process that have been proud to be the guy/gal from Office Space with the red stapler: hiding out without any social skills.  I have not ever once seen them be hired into the organizations I have worked in.  So, what are the key skills that a person must have to succeed in IT?  Let’s break it down here.

 

  • Communication – The ability to have a conversation with a person will go a long way for your IT career.  In most IT roles staff interacts with the business daily.  From the ability to just have a conversation to the ability to listen, and then assist by articulating clearly is necessary. I read something somewhere that said you should be able to explain complex technology in a simple form, so simple that even a child can understand. That is not always an easy task, but I compare it to when I go to the doctor. They have a lot of complex terms like we do, but at the end of the day, they need to remove that from the conversation and explain what they are doing so that a non-medical professional can understand. That is the same level of communication required to be successful in your IT career.

 

  • Negotiation – The art of negotiation is so important to anyone in life as a whole, but here is how it applies to your IT career. As you are looking at third-party products to support your organization, are you going to pay retail price? No way! Negotiation is necessary. How about when you are talking to a future employer about salary. Do you ever take the first offer? No way! Lastly, we even get to negotiate with our users/management/team in IT.  They may ask for the most complex and crazy technology to do their jobs. You may be inclined to say no, but this is not how it works. Figure out what it takes, price it out, and odds are they won’t do it. This is the art of negotiation.

 

  • Empathy – Always empathize with the frustrated user. They are likely consuming the technology you implement. While the issue may not even be your fault, it is important to share that you understand they are having a hard day. More importantly, you will do what you can to resolve their issue as quickly as possible.

 

Soft skills go further than even the key ones that I have highlighted, but my hope is that this did get you thinking. IT is no longer full of people that don’t communicate well with others. That is a stereotype that needs to go away.

 

Long-term success

 

The only way to be successful in IT is to communicate well and play nice with others.  Use those soft skills that you have.  Any other approach, no matter how well you know your job, will find you looking for a new one sooner rather than later.

In my previous blog, I discussed the somewhat unique expectations of high availability as they exist within a healthcare IT environment. It was no surprise to hear the budget approval challenges that my peers in the industry are facing regarding technology solutions. It also came as no surprise to hear that I’m not alone in working with businesses that demand extreme levels of availability of services. I intentionally asked some loaded questions, and made some loaded statements to inspire some creative dialogue, and I’m thrilled with the results!

 

In this post, I’m going to talk about another area in healthcare IT that I think is going to hit home for a lot of people involved in this industry: continuity of operations. Call it what you want. Disaster recovery, backup and recovery, business continuity, it all revolves around the key concept of getting the business back up and running after something unexpected happens, and then sustaining it into the future. Hurricane Irma just ripped through Florida, and you can bet the folks supporting healthcare IT (and IT and business, in general) in those areas are implementing certain courses of action right now. Let’s hope they’ve planned and are ready to execute.

 

If your experiences with continuity of operations planning are anything like mine, they evolved in a sequence. In my organization (healthcare on the insurance side of the house), the first thing we thought about was disaster recovery. We made plans to rebuild from the ashes in the event of a catastrophic business impact. We mainly focused on getting back and running. We spent time looking at solutions like tape backup and offline file storage. We spent most of our time talking about factors such as recovery-point objective (to what point in time are you going to recover), and recovery-time objective (how quickly can you recover back to this pre-determined state). We wrote processes to rebuild business systems, and we drilled and practiced every couple of months to make sure we were prepared to execute the plan successfully. It worked. We learned a lot about our business systems in the process, and ultimately developed skills to bring them back online in a fairly short period of time. In the end, while this approach might work for some IT organizations, we came to realize pretty quickly that this approach isn’t going to cut it long term as the business continued to scale. So, we decided to pivot.

 

Next we started talking about the next evolution in our IT operational plan: business continuity. So, what’s the difference, you ask? Well, in short, everything. With business continuity planning, we’re not so much focused on how to get back to some point in time within a given window, but instead we’re focused on keeping the systems running at all costs, through any event. It’s going to cost a whole lot more to have a business continuity strategy, but it can be done. Rather than spending our time learning how to reinstall and reconfigure software applications, we spent our time analyzing single points of failure in our systems. Those included software applications, processes, and the infrastructure itself. As those single points of failure were identified, we started to design around them. We figured out how to travel a second path in the event the first path failed, to the extreme of even building a completely redundant secondary data center a few states away so that localized events would never impact both sites at once. We looked at leveraging telecommuting to put certain staff offsite, so that in the event a site became inhabitable, we had people who could keep the business running. To that end, we largely stopped having to do our drills because we were no longer restoring systems. We just kept the business survivable.

 

While some of what we did in that situation was somewhat specific to our environment, many of these concepts can be applied to the greater IT community. I’d love to hear what disaster recovery or business continuity conversations are taking place within your organization. Are you building systems when they fail, or are you building the business to survive (there is certainly a place for both, I think)?

 

What other approaches have you taken to address the topic of continuity of operations that I haven’t mentioned here? I can’t wait to see the commentary and dialogue in the forum!

Anyone who is having issues with performance or considering expanding their deployment has had to wrestle with the question of how, exactly, to get the performance they need. This session will focus on maximizing performance, whether tuning equipment to optimize capabilities, tuning polling intervals to capture the data you need, or adding additional pollers for load balancing and better network visibility.

 

In the "Orion at Scale: Best Practices for the Big League" session, Kevin Sparenberg, product manager, SolarWinds, and Head Geek Patrick Hubbard will teach you best practices for scaling your monitoring environment and ways to confidently plan monitoring expansion. They will focus on maximizing the performance of your expanded deployment, and more!

 

THWACKcamp 2017 is a two-day, live, virtual learning event with eighteen sessions split into two tracks. This year, THWACKcamp has expanded to include topics from the breadth of the SolarWinds portfolio: there will be deep-dive presentations, thought leadership discussions, and panels that cover more than best practices, insider tips, and recommendations from the community about the SolarWinds Orion suite. This year we also introduce SolarWinds Monitoring Cloud product how-tos for cloud-native developers, as well as a peek into managed service providers’ approaches to assuring reliable service delivery to their subscribers.

 

Check out our promo video and register now for THWACKcamp 2017! And don't forget to catch our session!

Training is a topic I hold near and dear to my heart. Here are some of my thoughts about how a company will succeed or fail based on the training (and thereby the competence) of their technical staff.

 

 

My team members decide what they need to learn to better support our needs, then set aside a couple of hours each week, during work hours, to do training. This is informal, undirected time that benefits the company a lot!

 

 

Companies miss out when they don't allocate formal time and funds to help ensure that their employees have professional training. It doesn't matter whether those needs involve learning internal safety procedures, corporate IT security policies, basic or advanced switching/routing/firewalling, setting up V-Motion or VoIP or Storage LUNs, or just learning to smile while talking to customers on the phone.

 

Companies that don't budget time and money to train their staff risk not having the right staff to

  • Answer questions quickly
  • Do great designing
  • Provide excellent implementations
  • Troubleshoot problems efficiently and effectively.

 

It may surprise or dismay you, but training is more effective when it's done off site. Being at a training facility in person--not remotely or via eLearning--gets you more bang for your training dollars. It may look more expensive and inconvenient than participating in recorded or online/remote training sessions, but that perception is deceiving.

 

Relying solely on distance learning has unique costs and drawbacks:

  • Technical problems
    • Hearing audio
    • Sharing screens
    • Losing training time while waiting for the instructor to troubleshoot others' technical problems
  • Missing out on the pre-class, lunchtime, and post-class conversations and meetings. I've learned a lot from sharing information with students during these "off class" times. I've made some personal connections that have helped me get a lot more out of the training, long after the sessions are over. Those opportunities are lost when a class is attended online.
  • Remote eLearning sessions conducted onsite are ineffective due to work interruptions. Work doesn't stop when you are attending training sessions in your cube. The help desk calls our desk phones when we are needed, and our cell phones when we're not at our desks. Work doesn't stop when you are attending training sessions at your desk. People stop by for help without notice (we call these "drive-bys"), expecting us to interrupt our online training session to deal with their issues whenever they stop by our cubes. Hours or days of training are lost this way.
  • Remote or recorded training sessions are often dry and time-consuming.   We don't need to sit through introductions and explanations of training settings, yet that's what some companies include in their online training offerings. These sessions end up becoming cut-rate solutions for people or companies who can't afford to do training the right way. Actual hands-on, real-time, face-time experiences are richer in training fulfillment. They are critical to getting the most out of every training dollar.  Plus, getting out of the office helps encourage active participation during training, and results in a refreshed employee coming back to work. Training is no vacation (especially when taking a regimen of 12 to 14-hour classes for four or five days straight), but a change of environment is a welcome pick-me-up.

 

Relying on people to seek their own training using their own time and money is often a mistake

You can end up with people who either can't serve your company's needs or are burned out and frustrated. They'll look for a company that properly supports them with in-house training, and you'll potentially lose whatever expense you budgeted to train them, as well as losing the time wasted during their learning curve when they were a new employee.

 

To avoid this, establish a corporate policy that protects your investment.

  1. If a person leaves within twelve months of receiving training at the company's expense, they must reimburse the company for travel costs, meals, hotel, and tuition.
  2. If a person leaves between twelve months and twenty-four months after receiving training at the company's expense, they must only reimburse the company the cost of the tuition, not the travel, hotel, or meals.
  3. Once a person has been with the company for some arbitrary longer length of time (7-8 years or so), they don't have to reimburse any training costs when they leave, no matter how soon after training they take off. Your human resources team should be able to provide statistics about the likelihood of a person staying with the company after X years. Use their figures, or you can omit this option.

 

 

 

If you don't fund enough training for your people, you won't have the needed tools for the job when you need them. Your company will not prosper as well as it should. Those underappreciated employees will either inadvertently slow down your progress, or they'll take their services to a company that appreciates them. They'll see their value when the new company reinvests in those employees by sending them to great training programs.

 

How much does training cost?

 

The real question is, "How much does it cost to have untrained people on staff?"

If your people can't do the job because they haven't been trained, they'll make mistakes and provide poor recommendations. You won't be able to trust them.  You'll have to contract out for advanced services that bring in a hired gun to solve one issue one time. Once the expert leaves, you still have needs that your staff can't fill. Worse, you don't have impartial internal experts to advise you about the directions and solutions you should implement.

 

You can find many different vendor-certified training solutions at varying price points, but we can talk about some general budget items for a week of off-site training.

  • Tuition:  ~$3,500 - $6,000  (or more!) for a one-week class at the trainer's facility
  • Travel:
    • Flight ~$750 (depending on source and destination)
    • Car rental ~$300 (again, depends on distance, corporate discounts, etc.)
    • Hotel ~$150 per night (roughly)
    • Meals ~$125 per day (this is pretty high, but we're just looking at ballpark figures here)

 

You could spend up to $7,500 for one week of training one person.

 

Consider discounts and special offers.  You may be able to reduce your company's training costs to almost zero, especially if your employees live in the same city that is hosting the training.

  • Cisco Learning Credits can pay for all of the Cisco training if you have a good arrangement with your Cisco reseller if you choose a training company that accepts Learning Credits. If you don't have Cisco hardware, approach the vendor or your VAR for free or discounted training.
  • Some training centers offer big discounts or two-for-one training (or better) opportunities. It never hurts to ask for incentives and discounts to use their services.
  • Some training companies cover all hotel costs when training at their sites!
  • Some training programs include breakfast and lunch as part of the overall cost, leaving you to expense only dinners.
  • Car rental may not be required if you select a hotel adjacent to the training facility. Walk between them, rely on the hotel's airport shuttle, or use a taxi.

 

Do not rely solely on Cisco Learning Credits (CLC's)

 

A CLC is typically worth about $100, and if a class costs $3,500, you need 35 Learning Credits for an employee to have "free" training. Of course, those learning credits are NOT free. Your company either buys them (at a discount) or earns them as an incentive for their business. Perhaps you can sign an agreement with Cisco or your VAR that guarantees you'll spend X dollars on new hardware or services annually, and in return receive some percentage of X to use as learning credits. I've worked with two VARs who do this, and it's much appreciated.

 

CLCs are never enough to cover all of our training needs.  For one thing, they're only good for Cisco training.  If you have F5's, CLC's are of no value for their training.  Many training companies offer 2-for-1 discounts, or buy-one-get-a-second-at-50%-off, or better.  And you can make those dollars go further if you follow a great "Train The Trainer" program.  In this, you select a person who has great communication and understanding skills to receive the training.  When they return to the company, they train their peers.  They're fresh, they have contacts from their class that can be queried for answers to questions, and they may save you the cost of sending people to training.

 

Relying solely on CLCs means you've either got to spend a lot of capital dollars up front (to build up a bank of CLS's to use in the next twelve month), or you need more budget to cover the training gap.  Allocate sufficient funds to ensure your people have the exposure, training, and knowledge to correctly guide your company to a better IT future.  I can't emphasize this enough!

 

Discover your training needs. I have found that each analyst typically needs two weeks of off-site training annually, perhaps more for the first few years, until everyone is up to speed.

 

Why so much training?  Training is necessary for your team to:

  • Keep up with versions, bug fixes, better ways of doing things, security vulnerabilities and their solutions.
  • Do the highly technical and specialized things that make your network, servers, and applications run the best they can.
  • Maintain their skill sets and ensure they're aware of the right options and security solutions to apply to your organization.
  • Ensure they can properly design and implement and support every new technology that your company adopts.
  • Trust them to provide the right advice to decision makers.

 

 

You COULD hire outside contractors to be your occasional technical staff . . .    But then you'd be left with unthinking, non-advancing worker drones on your staff, who'll drag you down or leave you in the lurch when they find employers who will believe and invest in them.

 

Harsh? You bet! But when you understand the risks of having untrained people on staff, you see all the benefits that result from training.

 

If you have staff who sacrifice their personal expenses and family time (evenings, weekends, and holidays) to train themselves for the company's benefit, cherish them--they're unusual, and won't stay with you long.  They're on the fast path to leave you behind.  Give them raises and promotions to encourage them to stay, and compensate their training expenses. If you don't, they'll leave for the competition, who'll jump another step ahead of you.

 

Succeed by reinvesting in your staff, showing them they're appreciated by sending them to training, and they will help your company succeed.

Greetings from Barcelona! I’m here for VMworld and you can find me at the SolarWinds booth, at my session on Wednesday, or in the VM Village hanging out with other vExperts. If you are at the event, please stop by and say hello. I’d love to talk data with you.

 

As always, here are some links from the intertubz that I hope will hold your interest. Enjoy!

 

Equifax Says Cyberattack May Have Affected 143 Million Customers

There are only 126 million adults in the United States. So, yeah. You are affected by this.

 

Are you an Equifax breach victim? You could give up right to sue to find out 

As if suffering a breach isn’t bad enough, Equifax is tricking people into waiving their right to sue. Like I said, you are affected. Don’t bother checking. Equifax needs to notify you in a timely manner.

 

Three Equifax executives sold $2 million worth of shares days after cyberattack

If true, that they sold knowing about the breach, then my thought is this: Equifax can’t go out of business fast enough for me.

 

Surprising nobody, lawyers line up to sue the crap out of Equifax

Oh, good. That should solve everything because lawyers can go back in time to prevent the theft of our identities, right?

 

Windows 10 security: Microsoft offers free trial of latest Defender ATP features

Security should be free for everyone. Here's hoping Microsoft does the right thing and tries to protect everyone, always, for no additional cost. Too bad they didn’t help Equifax.

 

Hackers Gain Direct Access To US Power Grid Controls

If the Equifax stories didn’t rattle you enough, here’s one about hackers controlling your electricity.

 

A Simple Design Flaw Makes It Astoundingly Easy To Hack Siri And Alexa

EVERYTHING IS AWFUL

 

To Understand Rising Inequality, Consider the Janitors at Two Top Companies, Then and Now

Long, but worth the read. It’s a fascinating comparison between the American workforce 35 years ago and today.

 

The view from my hotel room at VMworld, overlooking Plaza Espanya at sunset:

scuff

SaaS and the SysAdmin

Posted by scuff Sep 12, 2017

In the SMB market, SaaS vendors are quick to promote that you can turn off your on-premise servers and ditch your IT guy/gal (I kid you not). In the Enterprise, it’s unlikely that all of your workloads will move to SaaS, so the IT Pros may still be safe. But let’s pick on one technology for a moment as an example – Microsoft Exchange. Assuming you ditch your Exchange boxes for Exchange Online, what’s an Exchange Administrator to do? How does their role change in a SaaS world?

 

What stays the same?
Administration: There’s still a need for general administration of Exchange Online, especially Identity & Access Management. People will still join, leave, change their names and move teams. Departments will still want distribution groups and shared mailboxes. The mechanics to do this are different and tasks will likely be done by someone who’s administering the other Office 365 services at a tenancy level, but that’s not too different to Enterprises that have a separate “data security” team anyway for IAM.

 

Hello, PowerShell: Speaking of changes in how you achieve things, being proficient in PowerShell is the best new skill to have, thought PowerShell is not limited to Exchange Online/Office 365. If you’re already using PowerShell to administer on-premises Exchange servers, you’re more than halfway there.

 

Compliance: It’s rare to find an organization that leaves all the settings at their defaults. Exchange Online may still need tweaking to ensure it locks down things and applies rules that you’re using in-house to achieve and maintain policy or regulatory compliance. That can be as simple as the blocked/allowed domains or more complex like Exchange transport rules and Data Loss Prevention settings.

 

Integration: We’ve been using SMTP to handle information flow and systems alerts for a very very long time now. It’s possible that you’ll need to replicate these connections from and to other systems with your Exchange Online instance. There’s a gotcha in there for ageing multi-function fax machines that don’t support TLS (don’t laugh), but this connectivity doesn’t just go away because you’ve moved to the Cloud.

 

End user support: Sorry, the Cloud won’t make all the support calls go away. Brace yourselves for calls that Outlook isn’t doing what it’s supposed to, and it’s only impacting one user. Then again, maybe that’s an Outlook problem and not an Exchange server problem (usually). A quick “do you see the same problem in Outlook Web Access” is your first troubleshooting step.

 

What changes?
Bye bye, eseutil: Sorry not sorrry, the Exchange database is no longer your problem. I will miss using eseutil to check and repair it.

 

No more upgrades: Patches, service packs and major version upgrades be gone, when the Microsoft team are managing the application. Ditto for the same on the underlying server operating system.

 

Monitoring: We’re still interested in knowing if the service is down before the users have to tell us, but we’re no longer able to directly monitor the running Microsoft Exchange services. In addition, we’re monitoring the Office 365 status information and message center.

 

Server provisioning and consolidation: Shutting down a big project and making people redundant? Expanding the business with a merger or acquisition? No more building servers or decommissioning them – just add more licenses or close accounts.

 

Your new role
The more things change, the more they stay the same. Though technology changes how we do our jobs, the things that need to be done don’t change. Yes, in this case Microsoft has the responsibility and the power for some parts that you would have taken care of with your own server. But I’m not seeing that the shift is enough to cut your hours in half just yet.

 

Join the conversation – let me know how adopting a SaaS solution has changed what you do in your role or how you do it.

Filter Blog

By date: By tag: