Skip navigation
1 2 3 Previous Next

Geek Speak

2,159 posts

User, Help Thyself

Posted by scuff Jun 23, 2017

I’m going to switch gears on the automation topic now. It’s natural to think of scripts, packages, images, tools, triggers, and actions when you think of automating IT tasks. We automate our technical things with more tech. But what if we removed ourselves from the equation some other way? Are there tasks that we are holding onto that we could use to empower other, non-technical humans to do instead?


Don’t rush off to outsource your monitoring checks to someone on Fiverr. Instead, we’re going to talk about what we can get our users to do for themselves, without breaking anything.


Self-service password reset - In a cloud SaaS world, we’re used to a lovely little “Forgot your password?” link that will send a reset link to our recorded email address. For better security, you’d want 2FA or some secret questions as well. Microsoft’s Azure Active Directory lets you enable this for your users and, for a change, it’s not on by default. If you have directory synchronization and password write-back turned on, presto! Your users have just reset their own on-premises AD password, too. If your AD isn’t connect to the cloud, a ton of third-party vendors jumped on this need to create paid tools of their own to achieve this for you. Might be worth a look if you have high help desk stats for password resets.


AD automation – While we’re on the subject of AD, how manual is your process for creating new user accounts? Have you played with CSVDE or PowerShell as a scripted input method? Could you take that to the next level and wrap a workflow around it that gets HR to enter the correct data (first name, last name, role, department, etc.) that could then feed into your script and run an automation user creation process? There are third- party tools that handle this, including the workflow and an approval step.


Azure Active Directory Premium also offers dynamic group membership. You can set attributes on a user object (such as, Department) and have a group that queries AAD and automatically adds/removes members based on that attribute, for access to resources. Now, if you could automate HR submitting a web form that changes the Department value in AD, you are now hands-off. Sounds good in theory, but is anyone using it?


Chatbots as help desks – We've previously talked about chatbots saving the world (or not). They are good at providing answers to FAQ-style queries, for sure. Facebook for Work and Microsoft Teams certainly think it’s important to support bots in their collaboration tools. Has anyone replaced their help desk with a bot, yet? Are users helping themselves with this modern day Clippy replacement? Or is this tech gone mad?


Aside from the bots, are we seeing collaboration tools enabling users to help each other with questions they may otherwise call the help desk for? Are we using those tools to communicate current IT known issues, to reduce incoming call volumes? Is it working?


Let me know if you’ve managed to automate yourself out of a process by enabling someone else to do it, instead.


IT Right Equals Might?

Posted by kong.yang Employee Jun 23, 2017

If I learned anything from Tetris, it’s that errors pile up and accomplishments disappear.

– Andrew Clay Shafer (@littleidea on Twitter).


In IT, we make our money and maintain our job by being right. And we have to be right more often than not because the one time we are wrong might cost us our job. This kind of pressure can lead to a defensive, siloed mentality. If might equals right, then look for an IT working environment that is conducive to hostilities and sniping.


I’ve witnessed firsthand the destructive nature of a dysfunctional IT organization. Instead of working as a cohesive team, that team was one in which team members would swoop in to fix issues only after a colleague made a mistake. It was the ultimate representation of trying to rise to the top over the corpses of colleagues. Where did it all go wrong? Unfortunately, that IT organization incentivized team members to outdo one another for the sake of excellent performance reviews and to get ahead in the organization. It was a form of constant hazing. There were no mentors to help guide young IT professionals to differentiate between right and wrong.


Ultimately, it starts and ends with leadership and leaders. If leaders allow it, bad behaviors will remain pervasive in the organization’s culture. Likewise, leaders can nip such troubling behavior in the bud if they are fair, firm, and consistent. That IT team’s individual contributors were eventually re-organized and re-assigned once their leaders were dismissed and replaced.


Rewards and recognition come and go. Sometimes it’s well-deserved and other times we don’t get the credit that’s due. Errors, failures, and mistakes do happen. Don’t dwell on them. Continue to move forward. A career in IT is a journey and a long one at that. Mentees do have fond memories of mentors taking the time to help them become a professional. Lastly, remember that kindness is not weakness, but rather an unparalleled kind of strength.


I’ve attempted to locate a manager or company willing to commit to the pretense of corporate pushback against a hybrid mentality. I’ve had many conversations with customers who’ve had this struggle within their organizations, but few willing to go on record.


As a result, I’m going to relate a couple of personal experiences, but I’m not going to be able to commit customer references to this.


Let’s start with an organization I’ve worked with a lot lately. They have a lot of data of an unstructured type, and our goal was to arrive at an inexpensive “SMB 3.0+” storage format that would satisfy this need. We recommended a number of cloud providers, both hybrid and public, to help them out. The pushback came from their security team, who’d decided that compliance issues were a barrier to going hybrid. Obviously, most compliance issues have been addressed. In the case of this company, we, as a consultative organization, were able to make a good case for both the storage of the data, the costs, and an object-based model for access from their disparate domains. As it turned out, this particular customer chose a solution that placed a compliant piece of storage on-premises that could satisfy its needs, but as a result of the research we’d submitted for them, their security team agreed to be more open in the future to these types of solutions.


Another customer had a desire to launch a new MRP application and was evaluating hosting the application in a hybrid mode. In this case, the customer had a particular issue with relying on the application being hosted entirely offsite. As a result, we architected a solution wherein the off-prem components were designed to augment the physical/virtual architecture built for them onsite. This was a robust solution that ensured a guarantee of uptime for the environment with a highly available approach toward application uptime and failover. In this case, just what the customer had requested. The pushback in this solution wasn’t one of compliance because the hosted portion of the application would lean on our highly available and compliant data center for hosting. They objected to the cost, which seemed to us to be a reversal of their original mandate. We’d provided a solution based on their requests, but they changed that request drastically. In their ultimate approach, they chose to host the entire application suite in a hosted environment. Their choice to move toward a cloudy environment for the application, in this case, was an objection to the costs of their original desired approach.


Most of these objections were real-world, and decisions that our customers had sought. They’d faced issues they had not been entirely sure were achievable. In these cases, pushback came in the form of either security or cost concerns. I hoped we had delivered solutions that met their objections and helped the customers achieve their goals.


It’s clear that the pushback we’d received was due to known or unknown real-world issues facing their business. In the case of the first customer, we’d been engaged to solve their issues regardless of objections, and we found them a storage solution that gave them the best on-premises solution for them. But in the latter, by promoting a solution that was geared toward satisfying all they’d requested, we were bypassed in favor of a lesser solution provided by the application vendor. You truly win some and lose some.


Have you experienced pushback in similar situations? I'd love to hear all about it.


The Actuator - June 21st

Posted by sqlrockstar Employee Jun 21, 2017

Hope everyone enjoyed Father's Day this past weekend, and that your day was filled with good food and good times with family. This week's Actuator is timed with the summer solstice, the longest day of the year. But as any SysAdmin knows, the longest day of the year is any day you are working with XML.


As always, here are some links from the Intertubz that I hope will hold your interest. Enjoy!


Amazon to buy Whole Foods for $13.7 billion, wielding online might in brick-and-mortar world

In the biggest news story last week, Amazon agreed to purchase Whole Foods. I am cautiously optimistic for what this could mean with regards to world hunger. By purchasing Whole Foods, Amazon gets a brand name AND a distribution channel they need not build themselves. Combined with drone delivery, Amazon could find a way to provide food to remote locations. Heck, if Amazon partners with a real estate company such as McDonald's (who already feeds 1% of the world each day), Amazon could be feeding 5% of the global population within ten years.


Divide and Conquer: How Microsoft Researchers Used AI to Master Ms. Pac-Man

Good news for SkyNet fans, we've now created AI smart enough to defeat video games. It won't be long now before the AI decides that the best way to win is to not play and instead eliminate the game creators.


Complete list of wifi routers from WikiLeaks' Cherry Blossom release detailing CIA hacking tools

If your home router is on this list, you might want to make sure you've protected yourself against the exploits that have been publicly released.


Forget Autonomous Cars—Autonomous Ships Are Almost Here

And now I have something else to write about other than just autonomous cars. Autonomous ships!


Marissa Mayer Bids Adieu to Yahoo

Only in America can someone be given the opportunity to run an already failing corporation into the ground and then walk away with a quarter of a billion dollars.


Block Untrusted Apps Using AppLocker

For anyone looking to add an extra layer of protection against malware. As much as I know users are a large security surface area to control, I also know that a lot of SysAdmins take and run scripts they find from internet help forums. Running random scripts you find on blogs are also a risk. Be careful out there, folks.


20 Percent of Users Still Don’t Know about Phishing or Ransomware, Reveals Survey

That 20% seems like a low estimate, IMO.


For all the fathers out there:


Screen Shot 2017-06-20 at 12.47.16 AM.png


IT professionals are a hardworking group. We carry a lot of weight on our shoulders, a testament to our past and future successes. Yet, sometimes we have to distribute that weight evenly across the backs of others. No, this is not because we don’t want to do something. I’m sure that any of you, while capable of performing a task, would never ask another person to do something you wouldn’t willingly do yourself. No. Delegating activities to someone else is actually something we all struggle with.


Trust is a huge part of delegating. You're not only passing the baton of what needs to be done to someone else, but you’re also trusting that they’ll do it as well as you would, as quickly as you would, and -- this is the hard part -- that they'll actually do it.


As the world continues to evolve, transition, and hybridize, we are faced with this challenge more often. I’ve found there are some cases where delegation works REALLY well, and other cases where I’ve found myself banging my head against the wall, desk, spiked mace, etc. You know the drill.


One particular success story that comes to mind involves the adoption of Office 365. Wow! My internal support staff jumped for joy the day that was adopted. They went from having to deal with weird, awkward, and ridiculous Exchange or Windows server problems on a regular basis to... crickets. Sure, there were and still are some things that have to be dealt with, but it went from daily activity to monthly activity. Obviously, any full-time Exchange admin doesn't want to be replaced by Robot365, but if it's just a small portion of your administrative burden that regularly overwhelms, it's a good bet that delegating is a good idea. In this particular use-case, trust and delegation led to great success.


On the other hand, I’ve seen catastrophes rivaled only by the setting of a forest fire just for the experience of putting it out. I won’t name names, but I've had rather lengthy conversations with executives from several cloud service providers we all know and (possibly) love. Because I’m discussing trust and delegation, let’s briefly talk about what we end up trusting and delegating in clouds.


  • I trust that you won’t deprecate the binaries, libraries, and capabilities that you offer me
  • I trust that you won’t just up and change the features that I use and my business depends on
  • I trust that when I call and open a support case, you’ll delegate activities responsibly and provide me with regular updates, especially if the ticket is a P1


This is where delegating responsibility and trusting someone to act in your best interest versus the interests of themselves or some greater need beyond you can be eye-opening.


I’m not saying that all cloud service providers are actively seeking to ruin our lives, but if you talk to some of the folks I do and hear their stories, THEY might be the one to say that. This frightful tale is less about the fear and doubt of what providers will offer you, and more about being aware and educated about the things that could possibly happen, especially if you aren’t fully aware of the bad things that happen on the regular.


In terms of trust and delegation, cloud services should provide you with the following guarantees:

  • Trust that they will do EXACTLY what they say they will do, and nothing less. Make sure you are hearing contractual language around that guarantee versus marketing speak. Marketing messages can change, but contracts last until they expire.
  • Trust that things DO and WILL change, so be aware of any depreciation schedules, downtime activities, impacts, overlaps of changes, and dependencies that may lie within your business.
  • Delegate to cloud services only those tasks and support that may not matter to your production business applications. You want to gauge how well they can perform and conform to an SLA. It’s better to be disappointed early on when things don’t matter than to be in a fire-fight and go looking for support that may never come to fruition.


This shouldn't be read as an attack or assault on cloud services. Instead, view this as being more about enlightenment. If we don’t help make them better support organizations, they won’t know to and will not improve. They currently function on a build-it-and-they-will-come support model, and if we don’t demand quality support, they have no incentive to give it to us.


Wow! I went from an OMG Happy365 scenario to cloudy downer!


But what about you? What kinds of experiences with trust and delegation have you had? Successes? Failures? I’ll offer up some more of my failures in the comments if you’re interested. I would love to hear your stories, especially if you've had contrary experiences with cloud service providers. Have they gone to bat for you, or left you longing for more?


IT Is Everywhere

Posted by shidoshi1000 Employee Jun 20, 2017

By Joe Kim, SolarWinds Chief Technology Officer


This is evident from two surveys we conducted last year. First, we asked more than 800 employed, non-IT adult end-users in North America a series of questions about how they use technology at work, and the types of technologies being used within their organizations. We also asked more than 200 IT professionals to give their impressions on these end-users’ expectations. Here’s a sample of what we found:


Users are taking IT everywhere. Forty-seven percent of end-user respondents said they connect more electronic devices, whether personally or company-owned, to their employers’ networks than they did 10 years ago. In fact, they connect an average of three more devices than they did a decade ago, two of which they own themselves.


The cloud has taken IT outside the agency. Most organizations allow some form of cloud-based applications, such as Google® Drive or Dropbox®, and 53 percent of respondents said they use these applications at work. Forty-nine percent said they regularly use work-related applications outside the office, on either personally or company-owned devices. Our survey also found that end-users will occasionally use non-IT-sanctioned cloud applications, such as iTunes® or something similar, while at work.


IT professionals must manage technology that may be outside their comfort zones. They must be versed in cloud-driven applications, mobile devices, open source software, and, increasingly, hybrid IT environments that incorporate aspects of on-premises and outsourced components. They must also continually be aware of and monitor the security risks that these solutions – and the actions of end- users – can present, adding one more layer of complexity to an already intricate set of concerns.


Eighty-seven percent of end-user respondents said they expect their organizations’ IT professionals to help ensure the performance of the cloud-based applications they use at work. Further, 68 percent blamed their IT professionals if these applications did not work correctly (“Dropbox isn’t working! Someone call IT!”).


According to the IT is Everywhere survey, 62 percent of IT professional respondents felt that the expectation to support users’ personally-owned devices on their networks is significantly greater than it was 10 years ago. Meanwhile, 64 percent of IT professionals said that end-users expect the same time to resolution for issues with both cloud-based and local applications. The inference is that users do not draw a distinction between cloud and on-premises infrastructures, despite the many differences between the two, and the fact that hybrid IT operations can be exceedingly complex and difficult to manage.


All of this is to say that IT is indeed everywhere. It’s in our offices and homes. It’s on our desktops and smartphones. It’s onsite and in the cloud.


IT professionals are constantly on deck to help ensure always-on availability and optimal performance, regardless of device, platform, application, or infrastructure. The end-users don’t care, as long as things are working.


Find the full article on GovLoop.

As technology professionals, we live in an interruption-driven world; responding to incidents is part of the job. All of our other job duties go out the window when a new issue hits the desk. Having the right information and understanding the part it plays in the organization is key to handling these incidents with speed and accuracy. This is why it's critical to have the ability to compare apples-to-apples when it comes to the all-important troubleshooting process.


What is our job as IT professionals?

Simply put, our job is to deliver services to end-users. It doesn't matter if those end-users are employees, customers, local, remote, or some combination of these. This may encompass things as simple as making sure a network link is running without errors, a server is online and responding, a website is handling requests, or a database is processing transactions. Of course, for most of us, it's not a single thing, it's a combination of them. And considering the fact that 95 percent of organizations report having migrated critical applications and IT infrastructure to the cloud over the past year, according to the SolarWinds IT Trends Report 2017, visibility into our infrastructure is getting increasingly murky.


So, why does this matter? Isn't it the responsibility of each application owner to make sure their portion of the environment is healthy? Yes and no. Ultimately, everyone is responsible for making sure that the services necessary for organizational success are met. Getting mean time to resolution (MTTR) down requires cooperation, not hostility. Blaming any one individual or team will invariably lead to a room full of people pointing fingers. This is counterproductive and must be avoided. There is a better way: prevention via comprehensive IT monitoring.


Solution silos

Monitoring solutions come in all shapes and sizes. Furthermore, they come with all manner of targets. We can use solutions specific to vendors or specific to infrastructure layers. A storage administrator may use one solution while a virtualization and server administrator may use another and the team handling website performance a third solution. And, of course, none of these tools may be applicable to the database administrators.

At best, monitoring infrastructure with disparate systems can be confusing, at worst, it can be downright dangerous. Consider the simple example of a network monitoring solution seeing traffic moving to a server at 50 megs/second, but the server monitoring solution sees incoming traffic at 400 megs/second. Which one is right? Maybe both of them, depending on if they mean 50 MBps and 400 Mbps. This is just the start of the confusion. What happens if your virtualization monitoring tool reports in Kb/sec and your storage solution reports in MB/sec? Also, when talking about kilos, does it mean 1,000 or 1,024?


You can see how the complexity of analyzing disparate metrics can very quickly grow out of hand. In the age of hybrid IT, this gets even more complex since cloud monitoring is inherently different than monitoring on-premises resources. You shouldn't have to massage the monitoring data you receive when troubleshooting a problem. That only serves to lengthen MTTR.


Data normalization

In the past, I've worked in environments with multiple monitoring solutions in place. During multi-team troubleshooting sessions, we've had to handle the above calculations on the fly. Was it successful?  Yes, we were able to get the issue remedied. Was it as quick as it should have been? No, because we were moving data into spreadsheets, trying to align timestamps, and calculating differences in scale (MB, Mb, KB, Kb, etc.). This is what I mean by data normalization: making sure everyone is on the same page with regard to time and scale.


Single pane of glass

Having everything you need in one place with the timestamps lined up and everything reporting with the same scale — a single pane of glass through which you see your entire environment — is critical to effective troubleshooting. Remember, our job is to provide services to our end-users and resolve issues as quickly as possible. If we spend the first half of our troubleshooting time trying to line up data, are we really addressing the problem?


About the Post

This is a cross-post from my personal blog @ [Link].



At least once a week I read or hear the familiar refrain, “SQL Server® is a memory hog,” or “SQL Server uses all the memory.” If you, or anyone you know, are saying these things, I am here today to tell you something.




Just no.


Stop. Saying. This.


It’s like hearing fingernails on a chalkboard when people say such things. It’s time to put an end to this myth.


And, as always, you’re welcome.


SQL Server Is a Software Program

That’s right. SQL Server is a piece of software. And software programs are good at doing what they have been programmed to do. Typically, software programs are programmed and configured by humans.


That’s where you come in, my fellow humans.


SQL Server will, by design, read data pages from disk into memory. SQL Server will store as many pages as you tell it to store, and will only evict them from memory as needed. My conclusion, for which I’ve done no research, is that 95% of the people complaining about SQL Server using all the memory on a server are 100% responsible for not configuring SQL Server memory properly.


And there’s the crux of the problem. The people that don’t understand how SQL Server uses memory also don’t understand that it is up to them to decide how much memory SQL Server will use.


That’s the #hardtruth folks. It’s been you all along.


(Editor’s note: I think Oracle®/UNIX® folks don’t have these complaints about memory because Windows® makes it easier to see memory consumption in Task Manager. Perhaps this myth would have died a long time ago if it weren’t for giving RDP access to people who don’t understand how SQL Server works, or that Task Manager is a dirty, filthy liar. But I digress.)


95 < 100

For those Generation Next-ers out there (people who install software by clicking Next-Next-Finish), you should know that SQL Server will not try to use all the available memory for data pages. The default setting allows for SQL Server to dynamically manage the memory consumption and it will not allocate more than 95% of the total physical memory.


For those of us old experienced enough to remember database servers with 8GB of RAM, that 95% is close enough to “all” for appearance's sake. And SQL Server has other memory needs than just database pages. Over the years, we have seen different data objects share the buffer cache with data pages. These days we can query the sys.dm_os_memory_clerks dynamic management view to find out how much of our memory is assigned to the various memory clerks.


The bottom line is that 95% is not 100%. SQL Server will not try to use all the memory by default. And setting the minimum memory will not cause SQL Server to start allocating memory, either. SQL Server will not allocate pages without being asked to do so.


By a human, most likely.


Deciding the Max Memory Setting

Assuming you have gotten this far, you understand that you are responsible for how much memory SQL Server will use. The next logical question becomes: What should the max memory be set to by default?


I have no idea. And neither does anyone else. If someone tells you they know exactly how much memory your SQL Server needs they are either (1) lying, (2) trying to sell you something, or (3) both.


There is no shortage of formulas out there for prognosticating the initial amount of memory to set as a max value for SQL Server. I’ve even seen suggestions that you use the size of a database to guess at your max memory setting. That’s absurd. It’s not the size of the database that determines the amount of memory needed, it’s the workload that matters.


Here is the formula I offer to clients and customers that ask for help with finding a max memory setting. This formula assumes you are trying to right-size the memory for a dedicated database server (engine only, no SSAS, SSRS, SSIS, etc., or any other significant applications), and this is for physical servers (but holds mostly true for virtualized servers, too):


• Take physical memory (say, 128 GB RAM)

• Subtract memory for the O/S itself (1 GB for every 8 GB of RAM; 16 GB in this example)

• Subtract memory needed for thread stack size (the number of worker threads multiplied by thread size; typical example for a x64, 4 CPU system would be 512 * 2 = 1024 MB, or 1 GB in our example here)


That would give us a max memory setting of about 111 GB in this example. Again, this doesn’t consider any other applications that might be running. The formula also does not consider the use of features such as Columnstore indexes or In-Memory OLTP. These features will require you to adjust your settings further.


Once you arrive at your number, you set your max memory and then monitor memory consumption, adjusting the settings as necessary. The 111 is not an absolute, it is meant as a decent starting point in the absence of any other information regarding the specific workload for the server.


Do You Need More Memory?

This is a common question that comes up in SQL Server and memory discussions. How do you know if you need more?


The first thing you need to know is if memory is the resource constraint you are facing. If so, then yeah, maybe you need more memory. One way to know is if your instance is using all the memory you assigned following the formula above. Another is if you are seeing memory errors in the SQL error logs. Still another way to know is if you are seeing a lot of disk activity (because SQL Server is not able to keep pages in memory). Any one of those items could mean that you need to allocate more memory to your instance.


However, it could be the case that by adding more memory you end up hurting performance. For example, in a virtualized environment it could be possible that the additional memory is spread over physical NUMA cores, resulting in slower performance than if the entire memory could fit inside one NUMA core.


I advocate that you monitor memory consumption over time, noting if you are trending upwards. Measuring and monitoring memory consumption is the best way to understand if your database server needs more memory.


Anything else is just a wild guess.



It’s not SQL Server, it’s you.


You have been in control all along.


It’s about time you understand, and accept, that you are responsible for what SQL Server is doing.


Blaming SQL Server for using all the memory that you have allowed it to access is like blaming a coffee maker for using all the water you placed inside.


The Actuator - June 14th

Posted by sqlrockstar Employee Jun 14, 2017

Home again after a trip to Austin where I was filming sessions for THWACKcamp 2017. Excited yet? You should be because you can register now by going here. It happens in October, and although it's four months away, it feels like it will be here in a week. I can't wait for you to see all the buttery goodness we have in store for you this year!


As always, here are some links from the Intertubz that I hope will hold your interest. Enjoy!


Ex-Admin Deletes All Customer Data and Wipes Servers of Dutch Hosting Provider

Remember this the next time someone asks you for elevated permissions. Insider threats are a real thing, folks. (HT to Radioteacher for pointing me to this story over the weekend.)


Microsoft realigns its cloud, AI, data organizations

If only there was a sign telling us that traditional data storage technologies were shifting toward the cloud and integrating with new technologies, like machine learning and artificial intelligence. Then we might be able to better prepare ourselves for this shift.


It's so windy in Britain the electricity price went negative

Because every now and then I see someone comment about how renewable energy sources aren't able to produce enough energy to meet demand. I believe they can produce enough if we are willing to invest enough.


UK cops arrest man picked out by automatic facial recognition software

We are just one step away from arresting people because we *think* they are about to commit a crime.


Microsoft buys security-automation vendor Hexadite

Interesting acquisition in the wake of WannaCry, although I am certain the wheels were in motion for many months prior. I believe this is yet another example of how Microsoft is taking data security seriously, and being as proactive as possible to minimize risks as Azure continues to gobble up data.


Why You Shouldn’t Use SMS for Two-Factor Authentication (and What to Use Instead)

I was somewhat aware of the risks with using SMS, and I liked how this article was able to explain the issue and possible workarounds.


Gamestop hacked. Financial data of online shoppers accessed by crooks

Yep. We got a letter last week about this matter. The letter, however, didn't specify that it was for online purchases only, as this article indicates.


As soon as this Evil-Clown-as-a-Service (ECaaS) becomes available in the US, I know what I'm getting some folks for their birthday:




Want to know a secret?


I'm going to start at the end.


If your environment collects syslog and trap messages, no matter what vendor solution you are using, create a filtration layer that will take all those messages, process them, and forward just the useful ones along.


Now, moving from the end back to the beginning, here's what you want to do: Get some copies of Kiwi Syslog Server, set up a load balancer like an F5 to do UDP round robin between all those servers, and set rules on the first server to filter out everything but the alerts you want to keep. For the messages you want to keep, set up rules to transparently forward them to the system(s) that will process and act on them. Export that rule set and import it to the other servers sitting behind the load balancer. Finally, update all of the devices in your enterprise to send their trap and syslog messages to the VIP presented by the load balancer.


That's the secret! Now that I've explained it, the trick, the bottom line, are you curious to know WHY I am telling you all this?


This is why: I've seen the following scenario a half-dozen times. I'm brought in to consult on a monitoring project and someone announces, "My monitoring sucks! It's dog slow and just doesn't work. Find me something else!" So, I poke around and realize that all of their traps and syslog messages are going to a single system, which also happens to be the monitoring system. In Solarwinds terms, that's the primary poller.


In my experience, network devices generate a metric buttload (yes, that's a scientifically accurate measurement) of messages per hour. In more boring terms, we're talking about roughly 4,000 messages per hour per machine.


If you have a server that is trying to manage pinging a set of devices (and collecting and storing those metrics) along with pulling SNMP or WMI data from that same set of devices (again, and storing that data), along with presenting that information in the form of views and reports, and checking the database for exceeded thresholds to create alerts, and analyzing that data to provide baselines, and... Well, you get the point. Polling engines have a lot of work to do. And one of the ways they stay on top of that work is having a finely tuned scheduler that manages all those polling cycles.


If you then start throwing a few million spontaneous messages, which must be processed in real-time, what you have is a VERY unhappy system. What you have is monitoring solution that "sucks" through no fault of its own.


Once I am able to point this out to clients, the next question is, "Should we turn off syslog or traps?" Of course not. That is a rich and vital source of information. What you need is to put something in front of those messages to filter them out.


Which brings me back to the "filtration system."


BUT... there's a catch! The catch is that most syslog and trap receivers expect to also process those messages themselves - to create alerts, to store the data, etc. What is needed in my example is to be able to ignore the messages that are unimportant, but then FORWARD the ones that matter to another system that is able to act upon them. The challenge here is to forward them without changing the source machine.


Many trap and syslog handlers can forward messages, but they replace the original machine with itself as the source. That's not helpful when you want to correlate a syslog message with data collected another way, say SNMP polling, for example. To do that, you need to perform what is called "transparent" forwarding, which keeps the original source machine information intact.


Kiwi Syslog has done this for years. But not so with SNMP traps. For a variety of reasons, which I won't get into now, that capability hasn't existed until 9.6, the latest version.


Now that this essential function within your monitoring infrastructure is available (not to mention really, REALLY affordable) you can impact the performance of your monitoring system in a great big, positive way.


So, take a minute and check out the new version. Forwarding traps transparently isn't the only new feature, by the way. There's also IPv6 support, SNMP v3 support, use of VarBinds in output, logging to Papertrail, and more! Try it and let me know what you think in the comments below.


By Joe Kim, SolarWinds Chief Technology Officer


With container adoption on the rise, I wanted to share a blog written in 2016 by my SolarWinds colleague, Kong Yang.


While the initial inroads are primarily still in the education phase, container technology has started making its way into federal IT networks, and the appeal is clear. Container-based technology provides value specifically in the areas of efficiency, optimization, and security, particularly as networks grow. This combination is uniquely suited to meet government IT needs.


Before an agency dips its toes into container technology, it’s vital for federal IT pros to gain an understanding of exactly what containers are, and what benefits they can bring.


What are Containers?


Container technology is far less complex than it sounds. Containers wrap a piece of software in a complete file system that contains everything the software needs to run, including code, run time, system tools, and libraries. Containers guarantee that the software will always run the same, regardless of the compute environment.


Let’s say you’re building an application that handles online transactions. The user experience consists of logging in, clicking on an item to add it to the cart, walking through the checkout process, and finally submitting to complete the transaction. With containers, you can isolate these services into loosely coupled services, aka microservices, across multiple containers. The advantage of doing it this way is that if the microservices fail, they will not take down the application.


In fact, a failure of a container or a system running containers will result in those services spinning up on other systems to get the work done. With non-container technology, there’s a good chance a tiered application is running on one or multiple systems to take care of that entire transaction. A failure that occurs on a tier or in a system will result in a degraded application or potential downtime as that tier restarts or fails over.


With container technology, however, each piece is separated out into its own tiny package. The login, for example, may be one container. Adding something to your cart may be another container, and so on. It’s like a distributed assembly line. Each container is responsible for its own small, unique task, which it does expertly, as opposed to one large monolithic application tier that’s responsible for many, often vastly different tasks, and carries much overhead.


How Can I Get Started Using Containers?


As with any new technology, the first thing to do is become familiar with that technology by learning about it. Because containers are typically open source, there is a wealth of publically available information and source materials that can be used for education and replication. and are great places to start.


The next step is to ramp up on skill sets. IT teams should dedicate some resources and time and start building experience around containers and microservices. Spend time testing to understand where these services might be implemented throughout the agency to increase efficiency. Again, Docker® provides installable platforms, such as Docker for Mac®, Linux®, and Windows® that you can leverage to level up your container experience.


Once there is a baseline understanding of containers, how they work, and how they can be used, apply that to your own environment and start mapping out a strategy for implementation.


Find the full article on Federal Technology Insider.

There’s no question that trends in IT change on a dime and have done so for as long as technology has been around. The hallmark of a truly talented IT professional is the ability to adapt to those ever-present changes and remain relevant, regardless of the direction that the winds of hype are pushing us this week. It’s challenging and daunting at times, but adaptation is just part of the gig in IT engineering.


Where are we headed?


Cloud (Public) - Organizations are adopting public cloud services in greater numbers than ever. Whether it be Platform, Software, or Infrastructure as a Service, the operational requirements within enterprises are being reduced by relying on third parties to run critical components of the infrastructure. To realize cost savings in this model, operational (aka employee) and capital (aka equipment) costs must be reduced for on-premises services.


Cloud (Private) - Due to the popularity of public cloud options, and the normalization of the dynamic/flexible infrastructure that they provide, organizations are demanding that their on-premises infrastructure operate in a similar fashion. Or in the case of hybrid cloud, operate in a coordinated fashion with public cloud resources. This means automation and orchestration are playing much larger roles in enterprise architectures. This also means that the traditional organizational structures of highly segmented skill specialties (systems, database, networking, etc.) are being consumed by engineers who have experience in multiple disciplines.


Commoditization - When I reference commoditization here, it isn’t about the ubiquity and standardization of hardware platforms. Instead, I’m talking about the way that enterprise C-level leadership is looking at technology within the organization. Fewer organizations are investing in true engineering/architecture resources, and instead are bringing those services in either via utilization of cloud infrastructure, or bringing this skill set on through consultation. The days of working your way from a help desk position up to a network architecture position within one organization are slowly fading away.


So what does all of this mean for you?

It’s time to skill up. Focusing on one specialty and mastering only that isn’t going to be as viable a career path as it once was. Breadth of knowledge across disciplines is going to help you stand out because organizations are starting to look for people who can help them manage their cloud initiatives. Take some time to learn how the large public cloud providers like AWS, Azure, and Google Compute operate and how to integrate organizations into them. Spend some time learning how hyperconverged platforms work and integrate into legacy infrastructures. Finally, learn how to script in an interpreted (non-compiled) programming language. Don’t take that as advice to change career paths and become a programmer.  That line of thinking is a bit overhyped in my opinion. However, you should be able to do simple automation tasks on your own, and modify other people’s code to do what you need. All of these skills are going to be highly sought after as enterprises move into more cloud-centric infrastructures.


Don’t forget a specialty. While a broad level of knowledge is going to be prerequisite as we go forward, I still believe having a specialty in one or two specifics areas will help from a career standpoint. We still need experts, we just need those experts to know more than just their one little area of the infrastructure. Pick something you are good at and enjoy, and then learn it as deeply as you possibly can, all while keeping up with the infrastructure that touches/uses your specialty. Sounds easy, right?


Consider what your role will look like in 5-10 years. This speaks to the commoditization component of the trends listed above. If your aspiration is to work your way into an engineering or architecture-style role, the enterprise may not be the best place to do that as we move forward. My prediction is that we are going to see many of those types of roles move to cloud infrastructure companies, web scale organizations, resellers/consultants, and the technology vendors themselves. It’s going to get harder to find organizations that want to custom-design their infrastructure to match and enhance their business objectives, instead opting to keep administrative-level technicians on staff and leave the really fun work to outside entities. Keep this in mind when plotting your career trajectory.


Do nothing. This is bad advice, and not at all what I would recommend, but it is an equally viable path. Organizations don’t turn on a dime (even though our tech likes to), so you probably have 5 to 10 years of coasting ahead. You might be able to eek out 15 if you can find an organization that is really change averse and stubbornly attached to their own hardware. It won’t last forever, though, and if you aren’t retiring before the end of that coasting period, you’re likely going to find yourself in a very bad spot.


Final thoughts


I believe the general trend of enterprises viewing technology as a commodity, rather than a potential competitive advantage, is foolish and shortsighted. Technology has the ability to streamline, augment, and enhance the business processes that directly face a business’ customers. That being said, ignoring business trends is a good way to find yourself behind the curve, and recognizing reality doesn’t necessarily indicate that you agree with the direction. Be cognizant of the way that businesses are employing technology and craft a personal growth strategy that allows you to remain relevant, regardless of what those future decisions may be. Cloud skills are king in the new technology economy, so don’t be left without them. Focusing on automation and orchestration will help you stay relevant in the future, as well. Whatever it is that you choose to do, continue learning and challenging yourself and you should do just fine.

Patrick Hubbard

THWACKcamp 2017

Posted by Patrick Hubbard Administrator Jun 9, 2017

When SolarWinds hosted its first virtual event, THWACKcamp in 2012, about 250 very active THWACK® community members attended, along with technology managers from a few large customers. There were a handful of sessions, with topics concentrated largely around network monitoring best practices, with a nod to IT systems management. THWACKcamp returns this October 18-19, and will mark the sixth year in what has grown to become a live, multi-track event for thousands of skilled IT professionals. It now spans expert advice in everything from networking, to automation, to hybrid IT, to cloud-native APM, DevOps, security, and even MSP operations. And, again this year, IT professionals will be at THWACKcamp’s core, collaborating (and occasionally commiserating), but learning and sharing ideas that make IT more reliable, innovative, and perhaps even fun.


Moon Landing


Being voluntold that you’re supporting a physical tradeshow booth can be nerve-wracking. First, the whole endeavor is, at its heart, a marketing thing. You must specify and configure demo gear that must somehow be squeezed into impossibly designed sets without overheating. You also become a Cord Master, asked to improvise never-before-seen cabling and connectivity, like HDMI to ½” pipe-thread. On top of this, add Layer 8 configurations, live code that attendees can actually see and touch that’s also interesting. Finally, throw the whole mess into crates months before the event, aware that forgetting even something small might mean five days of blank screens. Event tech is not IT’s comfort zone. I know I certainly prefer to have the safety of a hardware lab and dev team nearby.


While THWACKcamp has the advantage of being a virtual event, (more than a few admins have said that attending in shorts and a T-shirt working from home is the way to go), it is nonetheless a live event. And this year, as more technologies and topics are included than ever before, the Q&A and open chat conversations will be wider-ranging and more technical than ever. It’s not limited to what we can fit into a few crates. It’s an opportunity to interact with IT of all types, including very small businesses that rely on Managed Service Providers, midsized businesses managing the complexity of hybrid IT on a budget, to the largest enterprises with hundreds of IT professionals. It’s an open congress of some of the sharpest admins in IT, just as eager to attend and engage as the presenters are to share and learn something new.


Over-provisioned Geek Prize Closet


IT professionals attend technical conferences to learn, talk, and network, but they also certainly enjoy swag. Awesome geek giveaways return in 2017, along with THWACK community status points and bragging rights for those attending live. And for 2017, THWACKcamp attendees may earn up to 20,000 THWACK points for participating in activities, mini-missions, and, of course, attending sessions.

So, whether you’ve never missed a session of THWACKcamp, or you’ve never even been to a technical learning event, be sure to check out the registration page when it goes live in August. Maybe even set a reminder to register, because you can’t attend, chat with others, win prizes, or earn THWACK points if you don’t register. We look forward to seeing you live at THWACKcamp 2017, October 18 and 19!


Automating the Cloud

Posted by scuff Jun 8, 2017

Let’s stick our heads in the cloud for a moment. With your very first test account to play with a SaaS product or an Infrastructure as a Service environment, it’s natural to set up users and servers manually. That’s how we learn. That’s not sustainable on an ongoing basis for a production environment unless you want to screenshot every box you ticked and you know that the next tech will follow that documentation to the letter.


Decisions, decisions
Server builds and user account creation are two SysAdmin processes that are perfect for automating, even when they’re in the cloud. Your biggest challenge will be deciding what tool to use. Do you have a single vendor approach, so a native tool from that vendor will suffice? Are you splitting your risk between AWS and Azure, and looking for one tool that supports both environments? Are you running a hybrid model where there’s still a requirement for internal user accounts that you want to integrate with cloud SaaS products?


The single vendor approach
I’m going to pick on Azure and AWS because they are the two I’m most familiar with and I also have a word count to (roughly) stick to. If you’re a Rackspace or Google Cloud fan, or prefer some other IaaS flavor, add your thoughts in the comments.


Azure: It will be no surprise that Azure’s own automation service is based on PowerShell. PowerShell scripts and workflows (known as runbooks) to be exact. Learn more about Azure Automation here:


AWS: AWS Cloud Formation uses JSON or YAML text files. You can choose from a library of templates or you the designer to create your own.


The multi-vendor approach
I’ve briefly mentioned before the powerhouses of Chef and Ansible. Both have tools that integrate with both Azure and AWS.


Chef and Azure:


Chef and AWS:


Ansible and Azure:


Ansible and AWS:


DevOps also caught my eye, but it integrates with AWS, Digital Ocean, and Linode:


Usage and Billing
The "pay as you use" subscription model for SaaS products can lead to some large, unexpected bills. If the business loads a ton of new content (data) or places a significant amount of new traffic on one particular cloud server, you won’t see it until you get the monthly invoice. There are a few vendors jumping on board to help solve this problem.


Cloud Ctrl shows usage trends, compares spending between business units and allows you to set usage thresholds and alerts. It is compatible with Azure, AWS, Google Cloud, Soft Layer, and Office 365.


Startup Meta SaaS has just come out of stealth mode after a seed investment of around $1.5 million. Their product helps you analyze your spend and usage of SaaS products, including alerting on renewal dates. It will also tell you when accounts are being left dormant, which is handy if people have left your organization and their SaaS accounts haven’t been canceled. Meta SaaS currently supports 224 SaaS vendors and is adding new integrations at a rate of 20 per week.


Over to you!

I've offered just a taste of what you can automate in the cloud. We haven’t covered the automation of account provisioning when you run a hybrid environment (with tools like Azure AD Connect in the Microsoft world), but see my previous comment regarding word count.

Would a move to the cloud make you more open to investigating automation tools? Are they a necessity in the cloud world, or just another thing that will sit on your to-do list? Do you find it easy or hard to wrap your head around things like JSON scripts, to move to a world of cloud infrastructure as code?  Let me know what you think.

Root Cause.png


I remember the largest outage of my career. Late in the evening on a Friday night, I received a call from my incident center saying that the entire development side of my VMware environment was down and that there seemed to be a potential for a rolling outage including, quite possibly, my production environment.


What followed was a weekend of finger pointing and root cause analysis between my team, the virtual data center group, and the storage group. Our org had hired IBM as the first line of defense on these Sev-1 calls. IBM included EMC and VMware in the problem resolution process as issues went higher up the call chain, and still the finger pointing continued. By 7 am on Monday, we’d gotten the environment back up and running for our user community, and we’d been able to isolate the root cause and ensure that this issue would never come again. Others, certainly, but this one was not to recur.


Have you experienced similar circumstances like this at work? I imagine that most of you have.


So, what do you do? What may seem obvious to one may not be obvious to others. Of course, you can troubleshoot the way I do. Occam’s Razor or Parsimony are my courses of action. Try to apply logic, and force yourself to choose the easiest and least painful solutions first. Once you’ve exhausted those, you move on to the more illogical, and less obvious.


Early in my career, I was asked what I’d do as my first troubleshooting maneuver for a Windows workstation having difficulty connecting to the network. My response was to save the work that was open on the machine locally, then reboot. If that didn’t solve the connectivity issue, I’d check the cabling on the desktop, then the cross-connect before even looking at driver issues.


Simple parsimony, aka economy in the use of means to an end, is often the ideal approach.


Today’s data centers have complex architectures. Often, they’ve grown up over long periods of time, with many hands in the architectural mix. As a result, the logic as to why things have been done the way that they have has been lost. As a result, the troubleshooting toward application or infrastructural issues can be just as complex.


Understanding recent changes, patching, etc., can be an excellent way to focus your efforts. For example, patching Windows servers has been known to break applications. A firewall rule implementation can certainly break the ways in which application stacks can interact. Again, these are important things to know when you approach troubleshooting issues.


But, what do you do if there is no guidance on these changes? There are a great number of monitoring software applications out there that can track key changes in the environment and can point the troubleshooter toward potential issues. I am an advocate for the integration of change management software into help desk software and would like to add to that some feed toward this operations element with some SIEM collection element. The issue here has to do with the number of these components already in place at an organization, and with that in mind, would the company desire changing these tools in favor of an all-in-one type solution, or try to cobble pieces together. Of course, it is hard to discover, due to the nature of enterprise architectural choices, a single overall component that incorporates all of the choices made throughout the history of an organization.


Again, this is a caveat emptor situation. Do the research and find out a solution that best solves your issues, determines an appropriate course of action, and helps to provide the closest to an overall solution to the problem at hand.

Filter Blog

By date:
By tag: