Skip navigation
1 2 3 Previous Next

Geek Speak

2,131 posts



We’ve all seen dashboards for given systems. A dashboard is essentially a quick view into a given system. We are seeing these more and more often in the monitoring of a given system. Your network monitoring software may present a dashboard of all switches, routers, and even down to the ports, or all the way up to all ports in given WAN connections. For a large organization, this can be a quite cumbersome view to digest in a quick dashboard. Network is a great example of fully fleshed out click-down views. Should any “Red” appear on that dashboard, a simple click into it, and then deeper and deeper into it, should help to discover the source of the problem wherever it may be.


Other dashboards are now being created, such that the useful information presented within the given environment may be not so dynamic, and harder to discern in terms of useful information.


The most important thing to understand from within a dashboard environment is that the important information should be so easily presented that the person glancing at it should not have to know exactly how to fix whatever issue is, but that that information be understood by whoever may be viewing it. If a given system is presenting an error of some sort, the viewer should have the base level of understanding necessary to understand the relevant information that is important to them.


Should that dashboard be fluid or static? The fluidity is necessary for those doing the the deep dive into the information at the time, but a static dashboard can be truly satisfactory should that individual be assigning the resolution to another, more of a managerial or administrative view.


I believe that those dashboards of true significance have the ability to present either of these perspectives. The usability should only be limited by the viewer’s needs.


I’ve seen some truly spectacular dynamic dashboard presentations. A few that spring to mind are Splunk, the analytics engine for well more than just a SIEM, Plexxi, a networking company with outstanding deep dive capabilities into their dashboard with outstanding animations, and of course, some of the wonderfully intuitive dashboards from SolarWinds. This is not to say that these are the limits of what a dashboard can present, but only a representation of many that are stellar.


The difficulty with any fluid dashboard is how difficult is it for a manager of the environment to create the functional dashboard necessary to the viewer? If my goal were to fashion a dashboard intended for the purpose of seeing for example Network or storage bottlenecks, I would want to see, at least initially, a Green/Yellow/Red gauge indicating if there were “HotSpots” or areas of concern, then, if all I needed was that, I’d, as management assign someone to look into that, but if I were administration, I’d want to be more interactive to that dashboard, and be able to dig down to see exactly where the issue existed, and/or how to fix it.


I’m a firm believer in the philosophy that a dashboard should provide useful information, but only what the viewer requires. Something with some fluidity always is preferable.



Hey, everybody!  Welcome to this week’s quandary of Root Cause, Correlation Analysis, and having to collaborate across cross-functional teams where you have all the hands but none of the fingers!


If that sounds confusing to you, it’s because frankly, it is! I’d like to share a tale of woe and heartbreak driven by frustration in functional and equally dysfunctional IT team dynamics!


The story is set in a fairly cross-functional organization. You're probably familiar with the type. While there are clearly defined teams with responsibilities, there are also hard lines in the sand of who does what, where, when, how and why. Honestly, this story rings so true that I’ve seen this story blur with other ones. If that isn’t excitement, I don’t know what is!


As the story goes, our team had deployed a series of tools enabling a cross-stack data correlation engine allowing us to identify and truly correlate events as they happen to allow troubleshooting to be better, easier.   The problem was the true burden of responsibility this team had ALL the responsibility of identifying problems, but none of the authority to actually resolve those problems, let alone the authorization to work on them!   What makes this particularly fun is that we were chartered with and burdened by the responsibility of being held accountable for the issues until they were resolved.   If that sounds like some kind of decision made in a government sector… I wouldn’t tell you you’re wrong! J


This is where simple technical skills while essential were not good enough.  And frankly, all of the project management skills in the world wouldn’t matter here, because it’s not like a “problem” is a “project” per se.   No, we had to get everyone on board, every stakeholder at the table where egos were strong and stubborn.   Just like we discussed recently in Better Together - Working Together in Silo Organizations and When Being an Expert Isn’t Good Enough: Master of All Trades, Jack of None merely knowing the answer or the cause of the problem wasn’t good enough here.   All parties would reject the issue being theirs, even in light of evidence proving otherwise and would instead resort to finger pointing. Fortunately how we started to navigate these waters was through education of the tools we were using and how it would provide insight into their systems, access to our tools so we weren’t just the messenger they were trying to shoot but a helpful informant in things, and we also offered our guidance as IT Professionals to help them navigate the errors or problems so they could resolve them better.


It sounds so simple, it’s something fairly straight-forward but the timing it took and would continue to take whenever new members would join a team, or new problems would surface would take months or longer to reach a sense of team parity.


It’s been an interesting element of Systems Operations in the face of having intelligence, and knowledge not meaning much of anything unless you had all parties engaged, and even then that was no guarantee that people would agree, let alone do anything about it.


Have you faced a similar issue as well, where you identify a problem which isn’t your problem and the challenges faced in trying to resolve it?  Or perhaps even just having accountability for something which isn’t your responsibility and the woes of trying to get parties to take responsibility?


Or really any other story of problem correlation and root cause and how you were able to better or faster resolve it than what we faced!

You've decided it's time to learn how to code, so the next step is to find some resources and start programming your first masterpiece. Hopefully, you've decided that my advice on which language to choose was useful, and you're going to start with either Python, Go or PowerShell. There are a number of ways to learn, and a number of approaches to take. In this post, I'll share my thoughts on different ways to achieve success, and I'll link to some learning resources that I feel are pretty good.


How I Began Coding


When I was a young lad, my first foray into programming was using Sinclair BASIC on a Sinclair ZX81 (which in the United States was sold as the Timex Sinclair 1000). BASIC was the only language available on that particular powerhouse of computing excellence, so my options were limited. I continued by using BBC BASIC on the Acorn BBC Micro Model B, where I learned to use functions and procedures to avoid repetition of code. On the PC I got interested in what could be accomplished by scripting in MS-DOS. On Macintosh, I rediscovered a little bit of C (via MPW). When I was finally introduced to NetBSD, things got interesting.


I wanted to automate activities that manipulated text files, and UNIX is just an amazing platform for that. I learned to edit text in vi (aka vim, these days) because it was one tool that I could pretty much guarantee was installed on every installation I got my hands on. I began writing shell scripts which looped around calling various instantiations of text processing utilities like grep, sed, awk, sort, uniq, fmt and more, just to get the results I wanted. I found that often, awk was the only tool with the power to extract and process the data I needed, so I ended up writing more and more little awk scripts to fill in. To be honest, some of the pipelines I was creating for my poor old text files were tricky at best. Finally, somebody with more experience than me looked at it and said, Have you considered doing this in Perl instead?


Challenge accepted! At that point, my mission became to create the same functionality in Perl as I had created from my shell scripts. Once I did so, I never looked back. Those and other scripts that I wrote at the time are still running. Periodically, I may go back and refactor some code, or extract it into a module so I can use the same code in multiple related scripts, but I have fully converted to using a proper scripting language, leaving shell scripts to history.


How I Learned Perl


With my extensive experience with BASIC and my shallow knowledge of C, I was not prepared to take on Perl. I knew what strings and arrays were, but what was a hash? I'd heard of references but didn't really understand them. In the end—and try not to laugh because this was in the very early days of the internet—I bought a book (Learn Perl in 21 Days), and started reading. As I learned something, I'd try it in a script, I'd play with it, and I'd keep using it until I found a problem it didn't solve. Then back to the book, and I'd continue. I used the book as more as a reference than I did as a true training guide (I don't think I read much beyond about Day 10 in a single stretch; after that was on an as-needed basis).


The point is, I did not learn Perl by working through a series of 100 exercises on a website. Nor did I learn Perl by reading through the 21 Days book, and then the ubiquitous Camel book. I can't learn by reading theory and then applying it. And in any case, I didn't necessarily want to learn Perl as such; what I really wanted was to solve my text processing problems at that time. And then as new problems arose, I would use Perl to solve those, and if I found something I didn't now how to do, I'd go back to the books as a reference to find out what the language could do for me. As a result, I did not always do things the most efficient way, and I look back at my early code and think, Oh, yuck. If I did that now I'd take a completely different approach. But that's okay, because learning means getting better over time and —  this is the real kicker — my scripts worked. This might matter more if I were writing code to be used in a high-performance environment where every millisecond counts, but for my purposes, "It works" was more than enough for me to feel that I had met my goals.


In my research, I stumbled across a great video which put all of that more succinctly than I did:


Link: How to Learn to Code - YouTube


In the video, (spoiler alert!) CheersKevin states that you don't want to learn a language; you want to solve problems, and that's exactly it. My attitude is that I need to learn enough about a language to be dangerous, and over time I will hone that skill so that I'm dangerous in the right direction, but my focus has always been on producing an end product that satisfies me in some way. To that end, I simply cannot sit through 30 progressive exercises teaching me to program a poker game simulator bit by bit. I don't want to play poker; I don't have any motivation to engage with the problem.


A Few Basics


Having said that you don't want to learn a language, it is nonetheless important to understand the ways in which data can be stored and some basic code structure. Here are a few things I believe it's important to understand as you start programming, regardless of which language you choose to learn:


scalar variablea way to store a single value, e.g. a string (letters/numbers/symbols), a number, a pointer to a memory location, and so on.
array / list / collectiona way to store an (ordered) list of values, e.g. a list of colors ("red", "blue", "green") or (1,1,2,3,5,8).
hash / dictionary / lookup table / associative arraya way to store data by associating a unique key to a value, e.g. the key might be "red", and the value might be the html hex value for that color, "#ff0000". Many key/value pairs can be stored in the same object, e.g. colors=("red"=>"#ff0000", "blue"=>"#00ff00", "green"=>"#0000ff")
zero-based numberingthe number (or index) of the first element in a list (array) is zero;  the second element is 1, and so on. Each element in a list is typically accessed by putting the index (the position in the list) in square brackets after the name. In our previously defined array colors=("red", "blue", "green") the elements in the list are colors[0] = "red", colors[1]="blue", and colors[2]="green".
function / procedure / subroutinea way to group a set of commands together so that the whole block can be called with a single command. This avoids repetition within the code.
objects, properties and methodsan object can have properties (which are information about, or characteristics of, the object), and methods (which are actually properties which execute a function when called). The properties and methods are usually accessed using dot notation. For example, I might have an object mycircle which has a property called radius; this would be accessed as mycircle.radius. I could then have a method called area which will calculate the area of the circle (πr²) based on the current value of mycircle.radius; the result would access as mycircle.area() where parentheses are conventionally used to indicate that this is a method rather than a property.


All three languages here (and indeed most other modern languages) use data types and structures like the above to store and access information. It's, therefore, important to have just a basic understanding before diving in too far. This is in some ways the same logic as gaining an understanding of IP before trying to configure a router; each router may have a different configuration syntax for routing protocols and IP addresses, but they're all fundamentally configuring IP ... so it's important to understand IP!


Some Training Resources


This section is really the impossible part, because we all learn things in different ways, at different speeds, and have different tolerances. However, I will share some resource which either I have personally found useful, or that others have recommended as being among the best:





The last course is a great example of learning in order to accomplish a goal, although perhaps only useful to network engineers as the title suggests. Kirk is the author of the NetMiko Python Library and uses it in his course to allow new programmers to jump straight into connecting to network devices, extracting information and executing commands.




Go is not, as I think I indicated previously, a good language for a total beginner. However, if you have some experience of programming, these resources will get you going fairly quickly:



As a relatively new, and still changing, language, Go does not have a wealth of training resources available. However, there is a strong community supporting it, and the online documentation is a good resource even though it's more a statement of fact than a learning experience.





Parting Thoughts


Satisfaction with learning resources is so subjective, it's hard to be sure if I'm offering a helpful list or not, but I've tried to recommend courses which have a reputation for being good for complete beginners. Whether these resources appeal may depend on your learning style and your tolerance for repetition. Additionally, if you have previous programming experience you may find that they move too slowly or are too low level; that's okay because there are other resources out there aimed at people with more experience. There are many resources I haven't mentioned which you may think are amazing, and if so I would encourage you to share those in the comments because if it worked for you, it will almost certainly work for somebody else where other resources will fail.


Coincidentally a few days ago I was listening to Scott Lowe's Full Stack Journey podcast (now part of the Packet Pushers network), and as he interviewed Brent Salisbury in Episode 4, Brent talked about those lucky people who can simply read a book about a technology (or in this case a programming language) and understand it, but his own learning style requires a lot of hands-on, and the repetition is what drills home his learning. Those two categories of people are going to succeed in quite different ways.


Since it's fresh in my mind, I'd also like to recommend listening to Episode 8 with Ivan Pepelnjak. As I listened, I realized that Ivan had stolen many of the things I wanted to say, and said them to Scott late in 2016. In the spirit that everything old is new again, I'll leave you with some of the axioms from RFC1925 (The Twelve Networking Truths) (one of Ivan's favorites) seem oddly relevant to this post, and to the art of of programming too:


         (6a)  (corollary). It is always possible to add another
               level of indirection.    
     (8)  It is more complicated than you think.
     (9)  For all resources, whatever it is, you need more.
    (10)  One size never fits all.
    (11)  Every old idea will be proposed again with a different
          name and a different presentation, regardless of whether
          it works.
         (11a)  (corollary). See rule 6a.  

The Actuator - May 24th

Posted by sqlrockstar Employee May 24, 2017

I made it to Antwerp and Techorama this week. I am delivering two sessions and looking forward to my first ever presentation on a movie theater screen. I am now wishing I had embedded movie clips into my slides, just for fun.


As always, here are some links from the Intertubz that I hope will hold your interest. Enjoy!


CS4G Netsim

An interesting network simulator game, useful for folks like me who still need to get up to speed on all things networking.


SQL Server Command Line Tools for MacOS Released

If you were ever wondering about signs that Hell had frozen over, SQL Server on Linux is as good as any other. So would be the release of SQL Server tools for MacOS.


Extending the Airplane Laptop Ban

This makes no sense to me, and I am hopeful it never becomes real. I cannot imagine being asked to travel to speak and not have my laptop in my possession at all times.


North Korea's Unit 180, the cyber warfare cell that worries the West

Some interesting views into how North Korea may be able to not only hack, but get away with hacking by using the networks inside other countries.


Keylogger Found in Audio Driver of HP Laptops

This is why we can't have nice, secure things.


U.S. top court tightens patent suit rules in blow to 'patent trolls'

While I like that this was done, something tells me that this will only be a temporary shift. The real winners here, as always, are the lawyers.


No photo editing required. Cars in Belgium are this tiny:


By Joe Kim, SolarWinds Chief Technology Officer


The Data Center Optimization Initiative was introduced last year, superseding the Federal Data Center Consolidation Initiative (FDCCI).


While there have been some major wins, including billions of dollars saved and thousands of data centers shuttered, those wins do not change the fact that there are still major cybersecurity concerns surrounding the consolidation effort. According to a SolarWinds and Market Connections cybersecurity survey from last year, these concerns mainly stem from incomplete transitions during consolidation and modernization projects, overly complex enterprise management tools, and a lack of familiarity with new systems.


The fact that these concerns are still top of mind several years into the FDCCI is not surprising, considering the rapid evolution of the threat landscape.


Let’s take a look at four strategies federal network administrators can adopt to help circumvent this challenge and make their data consolidation efforts a little more secure.


1. Create a clearly defined organizational structure


Ultimately, everyone in an agency has a hand in data center operations—not just IT administrators, but also developers, managers and executives. Each responsible party should be assigned unique responsibilities and remain in contact with each other. That way, if a breach or outage occurs, the team will be able to work together to address the issue.


2. Follow up with lightweight and flexible procedures


One of the goals behind the federal government’s modernization effort is to become more agile and flexible, but this should not be confined to hardware and software. Once the organizational structure is defined and it’s time to put processes and procedures in place, agencies should help ensure that they are highly flexible and can adapt to changing conditions.


3. Encrypt and segment data at rest and in flight


All data, whether at rest or in flight, must be encrypted, especially as agencies continue their data center transitions. There are simply too many risks involved in the transition process itself -- too many places where data is vulnerable and too many opportunities for increasingly savvy hackers or insiders to access information left in the open.


Data segmentation is also critical, as it can limit the attack damage to a subset of data. Segmenting can reduce the potential for cascading -- and often catastrophic -- network failures.


4. Automate security and gain complete control


Administrators must implement solutions that can monitor applications and network activity and deliver patches and updates as necessary. These goals can be achieved with modern performance monitoring software that gives data center managers a complete view of the health of every aspect of their data centers, including compute, storage, network, and applications.


Administrators willing to lay the security groundwork now will find their road toward data center consolidation easier to travel. Their efforts will also provide a solid foundation for managing what promises to be a tricky post-consolidation world, where the amount of data continues to grow even as the number of data centers has shrunk.



Find the full article on Government Computer News.

In the contemporary American cinema classic, “Back to the Future,” Marty McFly takes a DeLorean-turned-time-machine into the future, into the past, and subsequently into the future again. If only your network and system infrastructure had a similar means of interdimensional travel to reveal the catalyst to events and incidents. Unfortunately, there is no flux capacitor for your network. You cannot get your firewall up to 88MPH, lock horns with a one-billion-volt bolt of lightning, and go back in time to determine the underlying cause of historical incidents on your network. Instead, we stay vigilant, watching and monitoring our networks for issues and trends across a historical period of time. However, in most environments, monitoring and observing all devices for ANY event is a foreboding task.


Luckily for us, most devices log events and have the ability to forward their log files to a centralized syslog server for collection, aggregation, review, and action. These log entries can range from configuration change notifications and port flapping on network devices, to services stopping on a system, or an intrusion. These log messages are paramount to your historical monitoring, and in some cases, compliance to legal and/or regulatory standards and audits. However, a log can only give you the information you need if you read it. This presents a challenge when many devices, such as firewalls, can produce millions of log messages per minute, many of which you might not need to read at all. With Kiwi Syslog® Server, you no longer have to hunt through log files on each individual device. Instead, they are all at your fingertips, allowing you to collect, filter, parse, and alert on log messages based on your criteria.


Ever vigilant, Kiwi Syslog Server becomes your eyes and ears, watching and listening for unusual log entries so you don’t have to. It is like a DeLorean for your network.


Kiwi Syslog Key Benefits

  • Deploy quickly. Accepts Syslog, SNMP, and Event Log data from your existing deployment.
  • Monitor real-time logs. Display logs locally or anywhere through the secure web access module.
  • React to messages. Send email, run programs, or forward data when selected messages arrive.
  • Troubleshoot problems. Centralize logs from systems and network devices to quickly pinpoint issues.
  • Comply with regulations. Implement log retention requirements of SOX, FISMA, PCI-DSS, and more.


Kiwi Syslog Key Features

  • NO LIMIT on maximum number of sources
  • Built and tested to handle MILLIONS of messages an hour
  • Run as a service (or foreground application) on most Windows operating systems
  • Collect log data from Syslog messages (both UDP and TCP), SNMP traps, and Windows® Event Logs (through the included Windows Event Forwarder)
  • Display real-time logs in multiple windows in a local viewing console, or from anywhere through secure web access
  • Split written logs by device, IP, hostname, date, or other message or time variables
  • Manage log archives with scheduled compress, encrypt, rename, move, and delete rules
  • Forward logs to other syslog servers, SNMP servers, or databases
  • Send email alerts, run programs, play sounds, and perform other actions when messages arrive
  • Act as a syslog proxy (forwarding messages with original IP information)
  • Ship syslog information securely across insecure networks with included Kiwi Secure Tunnel
  • View trend analysis graphs and send email with traffic statistics


Of course, it would be much more fun and adventurous to traverse the space-time continuum. Who wouldn’t want to leap into another dimension to get a glimpse of what’s to come, or a head’s up on things before they happen? However, for those of us without a time machine, there’s always Kiwi Syslog Server. Download it today and start your journey to better understand your network.


McCrory's Law: Data Gravity

Posted by kong.yang Employee May 19, 2017

I was fortunate enough to be in the audience for my friend, Dave McCrory's presentation at Interop during the Future of Data Summit. Dave is currently the CTO of Basho, and he famously coined the term "data gravity" in 2010. Data gravity, or as friends have come to call it, McCrory's Law, simply states that data is attracted to data. Data now has such critical mass that processing is moving to it versus data moving to processing.


Furthermore, Dave introduced this notion of data agglomeration, where data will migrate to and stick with services that provide the best advantages. Examples of this concept include car dealerships and furniture stores being in the same vicinity, just as major cities of the world tend to be close to large bodies of water. In terms of cloud services, this is the reason why companies that incorporate weather readings are leveraging IBM Watson. IBM bought The Weather Company and all their IoT sensors, which has produced and continues to produce massive amounts of data.


I can't do enough justice to the quality of Dave's content and its context in our current hybrid IT world. His presentation was definitely worth the price of admission to Interop. Do you think data has gravity? Do you think data agglomeration will lead to multi-cloud service providers within an organization that is seeking competitive advantages? Share your thoughts in the comment section below.



I spend a lot of my time with data. Too much time, really. An almost unhealthy amount of time.


I read about data. I write about data. I tweet about data.


I'm all about the data, no trouble.


And because I spend so much time immersed in data I see a lot of mistakes out there. A lot of issues that are avoidable. Data breaches caused by a lack of adequate data security. Disasters made worse by not having backups. Data centers offline but nobody being alerted because the alerting is not configured correctly.


Avoiding these issues is simple: Hire someone with a working knowledge of data, databases, and data administration. Of course, good help is hard to find. In the absence of finding good help, you need some guidance about what you can be doing better.


So that’s what I want to do for you today. Here are seven things that you can do to love your data, starting today. Not all the items listed here can be done in a day. Some will take longer than others. Review the list, build a plan, and get started on making your data the best it can be.


Automation – As a data professional you need to spend time thinking of ways to automate yourself out of a job. I would even suggest that you must be thinking about ways to “cloud-proof” your job. GUIs, and people, don’t scale. Code does.


Security – Data is the most critical asset your company owns. Without data, your company would not exist. All that hardware your company owns? Yeah, that’s there to move bits of data back and forth. When you consider the value and criticality of your data, you will understand that it is necessary to deploy data security and privacy tools.


Maintenance – In an unscientific study I did last year, I found that the bulk of database performance problems was related to the lack of proper maintenance. In some cases, there is no maintenance being done at all, or it is the wrong maintenance (such as VM snapshots instead of database dumps).


Alerting – Many folks lump alerting together with monitoring, but they are different. You use monitoring to decide what you want to alert upon. Alerts require action; everything else is just information you can collect and use later.


Monitoring – Here’s the most important thing you need to know about monitoring: If you create inbox rules for your monitoring system, then you’ve already lost. Monitor and measure what you need, but only send emails when action is needed (see above).


Analytics – While we are talking about alerting and monitoring, here’s your reminder to learn some basic data analysis and use it against the monitoring data. Learn to spot outliers. Stop guessing about what happened and start learning about what will happen.


Backups – It’s 2017. Do you know where your database backups are? But it’s not just backups. Last month the cloud went down. Be prepared. Have a proper business continuity plan in place and test the process twice a year.


Successful data professionals use these seven items to exercise proper control over their enterprise. Further, many of the issues I read about all fall into one of these seven buckets.


The difference between being prepared and being unprepared comes down to your willingness to make sure you have each of these seven items covered.


The Actuator - May 17th

Posted by sqlrockstar Employee May 17, 2017

The big news since last week is the WannyCry attack. I've got a lot I want to say on the subject of data security and will put together my thoughts in a different post. But for now I just want to remind everyone that security is a shared responsibility. With each attack, there seems to be more finger pointing and fewer solutions being offered.


As always, here are some links from the Intertubz that I hope will hold your interest. Enjoy!


Everything you need to know about the WannaCry / Wcry / WannaCrypt ransomware

Nice summary from Troy Hunt, helping to make sense of what happened last week. And while it is easy to say "just patch everything", the reality is that some systems aren't able to be patched. The truth is, the current software business model is broken.


Why “Just Patch It!” Isn’t as Easy as You Think

Having worked in an industry that doesn't like to touch systems that are working, I can relate to how some systems might be patched infrequently, if ever.


Don’t Blame Microsoft For WannaCrypt Vulnerability Exploitation

There are some people out there that believe Microsoft should be providing security updates for every OS they have ever built since 1983. What scares me most is that these same people are allowed to vote and drive cars.


Logs and Metrics

What's the difference between logs and metrics? Seems like an easy question to answer. But for some, there is little difference between the two.


Understanding the Kubernetes ecosystem

A quick Q&A about Kubernetes that reinforces the concept that there is no silver bullet when it comes to technology.


Microsoft debuts Azure Cosmos DB, a superset of its DocumentDB service

Microsoft is taking the first steps towards creating a truly global database for any type of data. Relational, NoSQL, NewSQL, it all just data, and Microsoft wants to make it easy to store your data with them.


Cybercrime on the high seas: the new threat facing billionaire superyacht owners

The struggle is real.


I've found the company responsible for every butt-dial phone call ever placed:


There is a dramatic shift underway in federal IT on a scale that is rarely seen more than once every decade or so. Computing environments are evolving from traditional on-premises-only to hybrid strategies that migrate some infrastructure to the cloud, while keeping some critical systems onsite.


According to SolarWinds’ IT Trends Report 2016: The Hybrid IT Evolution, which includes a survey of government IT professionals, hybrid IT will continue to be the norm for the foreseeable future.


What does this mean for federal IT pros? Should we anticipate a re-invention of sorts?


At a conceptual level, it means understanding a new normal and how best to operate within this hybrid environment. On a practical level, it means learning how to align current skill sets with new requirements and, just as important, integrating new expertise, such as hybrid IT monitoring and management, data analytics, automation, and cloud application migration with existing skill sets.


Understanding the New Normal


A hybrid IT environment—the new normal—can be complex. Unlike legacy environments that were relatively homogeneous, hybrid IT environments are inherently heterogeneous.


In a hybrid IT environment, different systems and applications exist in different locations. For example, according to the report, 70 percent of respondents say they have migrated applications to the cloud. In addition, 55 percent say they have migrated storage and 36 percent have migrated databases to the cloud.


Here’s where the challenge lies for today’s federal IT pros. What is the best way to manage hybrid IT?


The key to successful management is maximum visibility. Having a single point of truth across platforms—on-premises and cloud—is essential. Specifically, consider implementing a centralized dashboard to remediate, troubleshoot, and optimize all of your environments.


Skill Sets Required for the New IT Prototype


Of course, understanding the hybrid IT environment—and having a conceptual management strategy—is only half the equation. Managing a hybrid IT infrastructure requires new skills in addition to those needed to manage on-premises infrastructures.


Your team will need to learn service-oriented architectures, automation, vendor management, application migration, distributed architectures, API and hybrid IT monitoring, and more. In fact, the team’s skill set will be the driving force behind the success of your implementation.


What kinds of skills will bring the most value to a hybrid environment? According to the report, these are the required skills:



Also, hybrid IT often involves working with multiple service providers in different geographic locations. IT pros will need to become accustomed to working with various providers handling different tasks. In fact, it may become necessary to know how to negotiate contracts, understand budget management and service level agreements, manage workflows and deadlines, and dissect contract terms and conditions.




Hybrid IT environments may take federal IT pros outside of their comfort zones. The key to success is commitment: commit to embracing the new environment and commit to learning and honing new skills. Understanding the environment and bringing in the right skill sets and tools will be the key to maximizing the benefits of this new computing reality.


Find the full article on our partner DLT’s website.

The latest attack seemingly took the world by surprise. However, most of the affected users were using unpatched and unlicensed versions of Windows. How do we take a stand against ransomware and avoid being sidelined by these attacks? Here are a few things that I do and am happy to share in an effort to help strengthen your resistance against these attacks.


Update:  Assuming is never a good idea! Of course, your need for data backups is critical in ransomware attacks. But, it's not enough to have backups. You must also validate that they are usable and that the process works through testing.



  1. File Integrity Monitoring
    1. Monitoring your files for things like changing file extensions, moving of files, and authorization. Log & Event Manager (LEM) is vital in this to help protect your businesses information.
  2. Group Policies for Windows
    1. Cryptolocker prevention kits that do not allow ransomware to install in their most common locations.
    2. Make sure the Users group does not have full access to folders. I see this a lot, where a user group has full access to numerous folders.
    3. Make sure that users do not have rights to the registry!
  3. Static Block List
    1. Block known Tor IP addresses example:
  4. Limit network share access
    1. If they are able to penetrate and get to a server, you do not want to freely allow the ransomware full access to network shares. You also do not want a general user to have access to network shares that hold mission critical data. Think about this. Make sure you are applying policies and not giving users access to things they shouldn't. Allowing such gives attackers the same level of access.
  5. Update patching on servers
    1. If you are not patching your servers, you are not up to date on the malicious vulnerabilities that are already known. Stop being low hanging fruit and start being the insect spray to keep these attacks to a minimum.  Patch Manager will help you schedule and push these out so you are not worrying about being up to date. 
    2. The lab environment is key to making sure your third-party software is easily able to receive a patch. We all know that when a software or application is released, it is not aware of what's coming in the future. That is why installing a lab environment to test patches is a great way to help you patch and not be worried about breaking an application in the process.
  6. Spam
    1. For the love of everything great, update your spam filters. This is key to helping you keep malware from getting to people that are not aware of these attacks, which results in them being blamed. Preventing these emails of destruction helps keep your teams aware. You can even use them as user education.
  7. Test your plan
    1. Test out a fake ransomware email with your business. See who reacts and within what departments. This will help you to train people within their areas to not react to these type of emails.
    2. You may be surprised at how many people will click and simply give away their passwords. This is an opportunity for you to shine as an IT organization by using this information to help get funds and user training for the business.
  8. Web filter
    1. Control the sites that users can access. Use egress or outbound traffic filtering to block connections to malicious hosts.
  9. Protect your servers and yourselves
    1. Have a companywide anti-virus/malware program that is updated and verified. Patch Manager will help you determine who is up to date and who is not!
  10. Web settings
    1. Verify that your web settings do not allow for forced downloads.



There are lots of ways to protect ourselves at work and at home. The main reason why I focus on the home in my user education is because we can prevent these from work -- to a point. However, when the user goes home, they are an open door. So including user education to go over ways of protecting home environments is as much of a responsibility for the IT team as it is for the users themselves. Once home, the ransomware could decipher that blocked call and take over your machine.


We can try to protect ourselves with things like LEM, which alerts you when users come online, and see if their files have changed or are being changed.  However, NOT clicking the "click bait" email is what will ultimately help end-users be stronger links in the equation.


I hope this prompts you to raise questions about your security policies and begin having conversations about setting in place a fluid and active security plan. You never know what today or tomorrow will bring in bitcoin asks...

As I type this, I find myself somewhat unexpectedly winging my way back to Israel. For those who recall, I was here last December for DevOpsDays Tel Aviv (described here and here) with the specific goals to:


  • Continue my conversations about the intersection between DevOps and traditional monitoring
  • Increase my knowledge of cloud technologies and the causes behind the push to cloud
  • Eat my body weight in kosher shawarma


Very recently, an opportunity to speak at Bynet Expo fell into my lap, where we had a chance to articulate SolarWinds' approach to cloud and hybrid IT monitoring and management. The shawarma was calling me back, so I had to go.


This is where I'll be for the next week, soaking up some bright Israeli sunshine, meeting with over 3,000 IT pros in the booth, and offering my thoughts on how Sympathetic Innovation is the key to managing hybrid IT.


I can't wait to share the experience with everyone when I get back.


Oh, and there may be some food pictures. And even a video or two.


Mitigating Ransomware

Posted by jordan.martin May 11, 2017

Malware is an issue that has been around since shortly after the start of computing and isn't something that is going to go away anytime soon. Over the years, the motivations, sophistication, and appearance have changed, but the core tenants remain the same. The most recent iteration of malware is called ransomware. Ransomware is software that takes control of the files on your computer, encrypts them with a password known only to the attacker, and then demands money (ransom) in order to unlock the files and return the system to normal.


Why is malware so successful? It’s all about trust. Users needs to be trusted to some degree so that they can complete the work that they need to do. Unfortunately, the more we entrust to the end-user, the more ability a bad piece of software has to inflict damage to the local system and all the systems it’s attached to. Limiting how much of your systems/files/network can be modified by the end-user can help mitigate this risk, but it has the side effect of inhibiting productivity and the ability to complete assigned work. Often it’s a catch-22 for businesses to determine how much security is enough, and malicious actors have been taking advantage of this balancing act to successfully implement their attacks. Now that these attacks have been systematically monetized, we're unlikely to see them diminish anytime soon.


So what can you do to move the balance back to your favor?


There are some well-established best practices that you should consider implementing in your systems if you haven't done so already. These practices are not foolproof, but if implemented well should mitigate all but the most determined of attackers and limit the scope of impact for those that do get through.


End-user Training: This has been recommended for ages and hasn't been the most effective tool in mitigating computer security risks. That being said, it still needs to be done. The safest way to mitigate the threat of malware is to avoid it altogether. Regularly training users to identify risky computing situations and how to avoid them is critical in minimizing risk to your systems.


Implement Thorough Filtering: This references both centralized and distributed filtering tools that are put in place to automatically identify threats and stop users from making a mistake before they can cause any damage. Examples of centralized filtering would be systems like web proxies, email spam/malware filtering, DNS filters, intrusion detection systems, and firewalls. Examples of local filtering include regularly updated anti-virus and anti-malware software. These filtering systems are only as good as the signatures they have though so regular definition updates are critical. Unfortunately, signatures can only be developed for known threats, so this too is not foolproof, but it’s a good tool to help ensure older/known versions/variants aren't making it through to end-users to be clicked on and run.


The Principle of Least Privilege: This is exactly what it sounds like. It is easy to say and hard to implement and is the balance between security and usability. If a user has administrative access to anything, they should never be logged in for day-to-day activities with that account and should be using the higher privileged account only when necessary. Users should only be granted write access to files and shares that they need write access to. Malware can't do anything with files it can only read. Implementing software that either whitelists only specific applications, or blacklists applications from being run from non-standard locations (temporary internet files, downloads folder, etc…) can go a long way in mitigating the threats that signature-based tools miss.


Patch Your Systems: This is another very basic concept, but something that is often neglected. Many pieces of malware make use of vulnerabilities that are already patched by the vendor. Yes, patches sometimes break things. Yes, distributing patches on a large network can be cumbersome and time consuming. You simply don't have an option, though. It needs to be done.


Have Backups: If you do get infected with ransomware, and it is successful in encrypting local or networked files, backups are going to come to the rescue. You are doing backups regularly, right? You are testing restores of those backups, right? It sounds simple, but so many find out that their backup system isn't working when they need it the most. Don't make that mistake.


Store Backups Offline: Backups that are stored online are at the same risk as the files they are backing up. Backups need to be stored on a removable media and then that media needs to be removed from the network and stored off-site. The more advanced ransomware variants look specifically to infect backup locations, as a functioning backup guarantees the attackers don't get paid. Don't let your last recourse become useless because you weren't diligent enough to move them off-line and off-site.


Final Thoughts


For those of you who have been in this industry for any time (yes, I'm talking to you graybeards of the bunch), you'll recognize the above list of action items as a simple set of good practices for a secure environment.  However, I would be willing to bet you've worked in environments (yes, plural) that haven't followed one or more of these recommendations due to a lack of discipline or a lack of proper risk assessment skills. Regardless, these tried and true strategies still work because the problem hasn't changed. It still comes down to the blast radius of a malware attack being directly correlated with the amount of privilege you grant the end-users in the organizations you manage. Help your management understand this tradeoff and the tools you have in your arsenal to manage it, and you can find the sweet spot between usability and security.

Last week we discussed whether automation is expected in today’s IT environment. Many of you agreed that it’s definitely not a new thing, with SysAdmins scripting things years ago (hello, Kix32!). We could even argue that it was much easier to script things in the old days when DOS and Windows 3.1 relied on .ini files for configuration, and there was no pesky registry.


Like other technology buzzwords, automation may prompt a different mental picture for you than it does for me. To online entrepreneurs, automation is about lead magnets feeding into sales funnels, auto responders and content delivery systems – that’s the magic behind the “make money while you sleep” crowd (or so they say). To a manufacturer, automation might mean robotics. To a customer services manager, automation can be customer support self-service or chatbots.


So let’s discuss what SysAdmins are actually automating.

Builds – Gone are the days of inserting 6 x 3.5” disks to install an operating system, or being able to copy it from one machine to another (totally showing my age here). After that, you were lucky if someone had taken screenshots or written down what settings to select as you stepped through the setup screens of NT4.0. We’ve always wanted our server and desktop builds to be consistent, but the need to document got in the way for some of us. Microsoft Small Business Server gave us wizards, and image cloning technology and Sysprep became normal for desktops. Ignoring cloud services for now, how are your current server and desktop builds automated? Have you gone fully "Infrastructure as Code" with Puppet, Chef, or Ansible?


Standard desktop settings – After a while in the small business space your memory fades of locked down Standard Operating Environments, but the reasons for them remain strong. In a Microsoft world, it seems Group Policy settings still rule here. Am I right?


Server reboots and service restarts – While I’d love to see us progress further with autonomous computing (computer, heal thyself), we’re not quite there yet. In reality, you can have a few workhorse servers that just need a scheduled reboot every now and again. My other favorite remediation step is automatically starting services that shouldn’t have stopped. What automation do you have in place to prevent things from dying or for trying to revive them?


User provisioning – When used for the occasional staff member, the GUI isn’t that bad a place to create new accounts, as long as you remember to add the required group memberships, mailboxes etc. Even then, it’s probably quicker to update their details on a spreadsheet and run a PowerShell command. Has PowerShell become our new favorite thing for user adds, changes, and deletes?


This is a very brief overview, so what have I missed? Do you live on GitHub, searching for new scripts?


Jump into the comments and share what SysAdmin tasks you’ve automated and how. Maybe together we can automate us out of some work!



When focusing on traditional mode IT, what can a Legacy Pro expect?




This is a follow-up to the last posting I wrote that covered the quest for training, either from within or outside your organization. Today I'll discuss the other side of the coin because I have worked in spaces and with various individuals where training just wasn't important.


These individuals were excellent at the jobs they'd been hired to do, and they were highly satisfied with those jobs. They had no desire for more training or even advancement. I don't have an issue with that. Certainly, I’d rather interact with a fantastic storage admin or route/switch engineer with no desire for career mobility than the engineer who’d been in the role for two months and had their sights set on the next role already. I’d be likely to get solid advice and correct addressing of issues by the engineer who’d been doing that job for years.


But, and this is important, what would happen if the organization changed direction. The Route/Switch guy who knows Cisco inside and out may be left in the dust if he refused to learn Arista, (for example) when the infrastructure changes hands, and the organization changes platform. Some of these decisions are made with no regard to existing talent. Or, if as an enterprise, they moved from expansion to their on-premises VMware environment to a cloud-first mentality? Those who refuse to learn will be left by the wayside.


Just like a shark dying if it doesn't move forward, so will a legacy IT pro lose their status if they don’t move forward.


I’ve been in environments where people were siloed to the extent that they never needed to do anything outside their scope. Say, for example, a mainframe coder. And yet, for the life of the mainframe in that environment, they were going to stay valuable to the organization. These skills are not consistent with the growth in the industry. Who really is developing their mainframe skills today? But, that doesn’t mean that they intrinsically have no impetus to move forward. They actually do, and should. Because, while it’s hard to do away with a mainframe, it’s been known to happen.


Obviously, my advice is to grow your skills, by hook or by crook. To learn outside your standard scope is beneficial in all ways. Even if you don’t use the new tech that you’re learning, you may be able to benefit the older tech on which you currently work by leveraging some of your newly gained knowledge.


As usual, I’m an advocate for taking whatever training interests you. I’d go beyond that to say that there are many ways to leverage free training portals, and programs to benefit you and your organization beyond those that have been sanctioned specifically by the organization. Spread your wings, seek out ways to better yourself, and in this, as in life, I’d pass on the following advice: Always try to do something beneficial every day. At least one thing that will place you on the moving forward path, and not let you die like a shark rendered stationary.

Filter Blog

By date:
By tag: