1 2 3 4 Previous Next

Geek Speak

2,609 posts

So far in this series we have reviewed a few popular and emerging models and frameworks. These tools are meant to help you make sense of where you are and how to get where you’re going when it comes to information security or cybersecurity. We’ve also started the process of defining a new, more practical, more technology-focused map of the cybersecurity landscape. At this point you are familiar with the concept of four critical domains, and six key technology categories within each. Today we’ll dive into the second domain: Endpoint and Application.

 

I must admit that not everyone agrees with me about lumping servers and applications in with laptops and mobile phones as a security domain. I admit that the choice was a risk, but I believe it makes the most sense. So many of the tools and techniques are the same for both groups of devices. Especially now, as we move our endpoints out onto networks that we don’t fully control (or control at all in some cases). Let’s explore it together - and then let me know what you think!

 

Domain: Endpoint & Application

If we stick with the castle analogy from part 2, endpoints and applications are the people living inside the walls. Endpoints are the devices your people use to work: desktops, laptops, tablets, phones, etc. Applications are made up of the servers and software your employees, customers, and partners rely upon. These are the things that are affected if an attack penetrates your perimeter, and as such, they need their own defenses.

 

The categories in the endpoint and application domain are endpoint protection, detection, and response (EPP / EDR), patch and vulnerability management, encryption, secure application delivery, mobile device management (MDM), and cloud governance.

 

Category: EPP / EDR

The oldest forms of IT security are firewalls and host antivirus. Both have matured a lot in the past 30+ years. Endpoint protection (EPP) is the evolution of host based anti-malware tools, combining many features into products with great success rates. Nothing is perfect, however, and there are advanced persistent threats (APT) that can get into your devices and do damage over time. Endpoint detection and response (EDR) tools are the answer to APT. We're combining these two concepts into a single category because you need both – and luckily for us, many manufacturers now combine them as features of their endpoint security solution.

 

Category: Patch and Vulnerability Management

While catching and stopping malware and other attacks is great, what if you didn’t have to? Tracking potential vulnerabilities across your systems and automatically applying patches as needed should reduce the exploit capabilities of an attacker and help you sleep better at night. While you can address patch management without vulnerability management, I recommend that you take a comprehensive and automated approach, which is why they are both covered in this category.

 

Category: Encryption

When properly applied, encryption is the most effective way to protect your data from unwanted disclosure. Of course, encrypted data is only useful if you can decrypt it when needed – be sure to have a plan (and the proper tools) for extraction! Encryption/decryption utilities can protect data at rest (stored files), data in use (an open file), and data in motion (sending/receiving a file).

 

Category: Secure Application Delivery

Load balancers used to be all you needed to round-robin requests to your various application servers. Today application delivery controllers (ADC) are much more than that. You always want to put security first, so I recommend an ADC that includes web application firewall (WAF) and other security features for secure application delivery.

 

Category: Mobile Device Management

EPP and EDR may be enough for devices that stay on-prem, under the protection of your perimeter security tools, but what about mobile devices? When people are bringing their own devices into your network, and taking your devices onto other networks, a more comprehensive security-focused solution is needed. These solutions fall under the umbrella of mobile device management (MDM). 

 

Category: Cloud (XaaS) Governance

Cloud Governance is a fairly emergent realm and in many ways is still being defined. What’s more is that to an even higher degree than the other categories here, governance must always include people, processes, and technology. Since this reference model is focused on technology and practical tools, this category includes technologies that enable and enforce governance.  As your organization becomes more and more dependent on more and more cloud platforms, you need visibility and policy control over that emerging multi-cloud environment. A solid cloud governance tool provides that.

What's Next?

We are now three parts into this six-part series. Are you starting to feel like you know where you are? How about where you need to be going? Don’t worry, we still have two more domains to cover, and then a final word on how to make this model practical for you and your organization. Keep an eye out for part 4, where we’ll dive into identity and access - an area that many of you are probably neglecting, despite its extreme importance. Talk to you then!

What Should We Be Monitoring?

 

To effectively begin getting a grasp on your applications performance, you must begin mapping out all the components in the path of your application. It might be wise to begin with the server or servers where your application lives. Again, this could be multiple different scenarios depending on the architecture housing your application. Or it could easily be a mixture of different architectures.

 

Let's say your application is a typical three-tier app that runs on several virtual servers that run on top of some hypervisor. You would want to start collecting logs and metrics from the hypervisor platform such as CPU, memory, network, and storage. You would also want to collect these metrics from the virtual server. Obviously, in addition to the metrics being collected from the virtual server, your application logs and metrics would be crucial to the overall picture as well. If, for some reason, your application does not provide these capabilities, you will need to either develop them or rely on some other existing tooling that you could easily integrate into your application. These metrics are crucial in identifying potential bottlenecks, failures, and overall health of your application. In addition to what we have already identified, we also need to collect metrics from any load balancers, databases, and caching layers. Obtaining all these metrics and aggregating them for deep analysis and correlation gives us the overall view into how our application is stitched together and assists us in pinpointing where a potential issue might arise.

How Should We Be Monitoring?

 

We have now identified a few of the things we should be monitoring. What we need to figure out next is how will we begin monitoring these things and ensure that they are accurate as well as providing us with some valuable telemetry data. Telemetry data comes from logs, metrics, and events.

Logging (Syslog)

 

First, let us begin with logging. We should have a centralized logging solution that can not only receive our log messages, but also have the ability to aggregate and correlate events. In addition, our logging solution should provide us with the ability to view graphs and customized dashboards, and also provide us with some level of alerting capabilities. If we have these initial requirements available to us from our logging solution, we are already off to a good beginning.

 

Now we need to begin configuring all our hypervisors, servers, load balancers, firewalls, caching layers, application servers, database servers, etc. There are many, many systems to ensure we are collecting logs from. But we need to make sure we get everything that is directly in the path of our application configured for logging. Also, remember that your application logs are important to be collected as well. With all these different components configured to send logging data, we should begin seeing events over time, painting a picture of what we might determine as normal. But remember, this might be what we termed after-the-fact application monitoring.

Metrics

 

There are numerous different methods of obtaining metrics. We should be clear about one thing when we begin discussing these methods, and that would be not using SNMP polling data. Now don't get me wrong--SNMP polling data is better than nothing at all. However, there are much better sources of metric data.

 

Our performance metrics should be time series-based. Time series-based metrics are streamed to our centralized metrics collection solution. With time series-based metrics we can drill into a performance graph at a very fine level of detail.

 

Most time series-based metrics require an agent of some sort on the device that we would like to collect metrics from that is responsible for providing the metrics that we are interested in. The metrics we are interested in include those that were mentioned before: CPU, memory, network, disk, etc. However, we are interested in obtaining metrics for our application stack as well. These metrics would include application-related metrics such as response latency, searches, database queries, user patterns, etc. We need all of these metrics to visually see what the environment looks like, including the health and performance.

 

With metrics and logging in place, we can begin correlating application performance with the events/logs to start understanding what the underlying issue might be when our application performance is degraded.

Welcome to the Halloween Edition of the Actuator! It’s the Halloween Edition because of the date, not because the stories are scary. Although you may be concerned that the data is coming from inside the house.

 

As always, here are some links from the Intertubz that I hope will hold your interest. Enjoy!

 

Red Hat Cloud Prowess Drives $33 Billion IBM Deal

You want scary? Here’s IBM rising from the grave to buy the largest open source company in the world. Red Hat was set to make $3 billion this year selling software that runs on top of Linux, which is free. But hey, as long as they aren’t Microsoft, it’s OK for someone to profit, right?

 

Feds Say Hacking DRM to Fix Your Electronics Is Legal

In other news, our federal government made a decision that favored consumers. Yes, I’m scared, too.

 

"Smart home" companies refuse to say whether law enforcement is using your gadgets to spy on you

[NARRATOR VOICE]: They are totally spying on you.

 

Tim Cook calls for strong US privacy law, rips “data-industrial complex”

"Profits over privacy," well stated and I agree. We need better laws here. Sadly, our elected officials lack the necessary experience to make this happen.

 

Hubble Telescope’s Broken Gyroscope Seemingly Fixed After Engineers Try Turning It Off and On Again

Houston, have you tried turning it off and back on again?

 

GM’s data mining is just the beginning of the in-car advertising blitz

If you thought the ads at the gas pump, or inside taxis in Vegas and NYC were horrible, just wait until you see them in your own car.

 

Minds, the blockchain-based social network, grabs a $6M Series A

Yet another waste of time and money on an idea with Blockchain in the title. If this company is worth $6 million in funding, my Bacon Blockchain idea is worth at least $600 million in the first round.

 

As I was saying:

 

In my previous posts, I talked about the basics of AI in relation to network and systems management as well as why I love it. This post isn’t going to be as optimistic about the current state of affairs when it comes to AI. It isn’t the tech that bothers me. So what is my big problem with AI? The answer is simple and comes down to basically two things: marketing and money.

 

There are a lot of products touting the latest and greatest in artificial intelligence. They all claim to be the only thing that can save your infrastructure from the impending doom of manual work. The literature, webinars, and sales calls all start off with the horrors of running a network without AI and how inefficient it is. There is no shortage of FUD (fear, uncertainty, doubt) on the topic, all spewed in the name of selling you the latest and greatest cloud-based widget that’s smarter and faster than you are.

 

“How can you possibly, responsibly, safely run a network without our algorithms looking over your shoulder?”

 

They show off demos of their system finding impossible correlations in a veritable ocean of data and maybe even show the system reacting automatically while telling nice stories of administrators and engineers sleeping soundly while the mighty AI saves the day. The biggest question to ask there is “Where did the data come from?” Was it created specifically for the demo? Was it taken from a customer’s live network? Is that network identical to yours? The training that goes into building a reasonably sophisticated AI is intense. Especially when you’re talking about monitoring dozens, if not hundreds, of applications across a massive network. Does it understand your needs? Does it adapt? Does it dream of electric sheep?

 

Marketing aside, the topic of money is usually saved until the target (you) has decided they can’t live another minute without their product. Let’s assume for a minute that you’ve found a company that is building an AI suited perfectly for your needs. It ingests massive amounts of data from all over your network into a box (or boxes) that crunch all that info actionable results for you. It spits out pretty reports and integrates with all your systems. Where does that box (or boxes) live? On-prem? That going to take up a lot of rack space, power, and cooling. Plus those machines aren’t going to be your run-of-the-mill 1U servers. They will be power hungry, face-melting, GPU-packed behemoths. Cloud-based? Now you’ve just moved those same beasts to someone else's data center. Granted, the cost will decrease a bit because the vendor is likely counting on scale and flexibility to process multiple customers on the same hardware just as effectively. But add to that the bandwidth requirements of sending all your application logs, all your network logs, and anything else you can through at it up to the cloud, constantly. There are a ton of costs that go into running a system that can think like a person, and licensing is only part of it.

 

The marketing is strong when it comes to these products and it can be a bit misleading. AI is not going to save you from something that you couldn’t do yourself. AI isn’t going to magically fix all your user issues. AI is definitely not going to put your network and systems on autopilot. AI is going to be expensive and time-consuming to set up properly.

 

However, AI can save you money in the long run by becoming a force multiplier for your IT staff. AI can add in efficiencies that you weren’t able to even come close to before, and maybe even start making some money for your company (depending on your industry) by aligning your data better and creating new avenues to generate income from the systems already in place.

By Paul Parker, SolarWinds Federal & National Government Chief Technologist

 

Here is an article from Federal Technology Insider on securing medical device networks, in which I was interviewed about how customers are using our tools to help with this issue.

 

Connected devices have been an integral part of the healthcare world for years. Networked devices that form the basis of connected healthcare for monitoring, record-keeping, and drug delivery are commonplace in hospitals, clinics, and research facilities operated by federal agencies, including the Department of Veterans Affairs.

 

“Many of these devices, frankly, present a huge security threat to government networks as well as to the personal data of patients,” cautioned Paul Parker, Chief Technologist, Federal and National Government at SolarWinds, which develops IT management and cybersecurity technology.

 

The issue, Parker explained, stems from the sheer variety of wired and wireless devices from multiple manufacturers that can connect to the network. “Wireless medical devices can be moved around the building, or even taken to other locations. Yet, this ‘Internet of Health’ isn’t always treated with the same security mindset as a laptop or cell phone,” he said.

 

With a significant number of networked medical devices in use, the Department of Veterans Affairs has made aggressive moves toward solving this issue. Under the Medical Device Protection Program, launched in 2009, a new set of protocols was introduced to isolate connected medical devices on the network.

 

However, several security challenges remain. As Parker explained, “Many medical devices can store personally identifiable information, and most have restrictions on software patches and updates, which can make them vulnerable to attack. Not only can someone steal or inadvertently release the data that’s stored on each device, they are potential entry points to the entire network.”

 

In addition, medical devices may not be supported by enterprise management and cybersecurity tools, and some older devices simply can’t be updated to meet new security protocols. “These devices aren’t always visible to the network management tools, and, ironically, they can’t be scanned for infections,” Parker pointed out.

 

The solution, he said, stems from viewing these devices as entry points to the network. “Just like any other mobile device or a connected appliance, like a thermostat or lighting system, medical devices provide benefits and risks.”

 

Parker emphasized that compliance with security standards and regulations, including the Federal Information Security Management Act (FISMA) and the Risk Management Framework, are absolutely essential, adding, “Another must-have is centralized access control and configuration capabilities for every asset on your network.”

 

In a healthcare environment, where no breaches are acceptable, Parker suggested that an effective approach requires a comprehensive access control list (ACL) strategy that incorporates standardizing ACL configurations across the network while isolating and protecting medical devices. He also stressed the need to review and implement authorized network configuration changes while detecting and fixing unauthorized or suspicious changes, all in real time.

 

“Just like everything else in security, it comes down to having a strategy, enforcing policies, training the users and network staff, and using the right technologies for the task,” Parker said. “With the Internet of Health, the risks are huge, but those risks can be contained. For any agency dealing with medical technologies, there is no choice: these devices and the data they collect must be protected.”

 

Find the full article on Government Technology Insider.

 

The SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates.  All other trademarks are the property of their respective owners.

Leon Adato

Directions

Posted by Leon Adato Expert Oct 30, 2018

My wife called me for the third time, and I could hear that she was working hard to remain calm but was undeniably at the end of her rope. She had missed the freeway exit. For the third time. And was going to be late for our lunch date. Could I PLEASE tell her JUST ONE MORE TIME what the exit name was?

 

We were in Switzerland. The company I worked for had moved us there just a week earlier, and my wife was meeting me so we could have a quick lunch and then go house hunting. I told her, again, that the exit was "sheb." Since this was our third time on the phone, I was beginning to doubt myself. Did I have the exit name wrong?

 

And that's when it hit me. My wife's second language is Spanish. I, on the other hand, learned French growing up. For those unfamiliar with linguistic differences, Spanish is a delightfully phonetic language. It is almost impossible to misspell a word in Spanish, presuming you know how to say it out loud. French? Not so much. I had been telling her to get off at the exit named "sheb," because my French-speaking brain never gave it a second thought.

 

And how do you spell "sheb" in the French-speaking part of Switzerland? (Answer: "chexbres")

 

I learned something that day about how I process and communicate directions, regardless of the language. Those lessons continued for the duration of our stay. Of course, distances and speeds were measured in kilometres. But it turns out the Swiss don't hold much stock in street signs. Roads operate as a network of roundabouts pointing to various villages. Getting from place to place means knowing you are going from Lausanne to Crissier to Pully to Renens. It's a far cry from "turn north at Elm and Wadsworth."

 

Directions, it turns out, are an incredible way to find out how someone thinks, and how they might work (both as an individual and within a team). Not just in terms of geography, but in other areas as well.

 

From time to time during my IT career, I've been on the other side of the desk, evaluating people we wanted to hire.

 

I discovered a few truths early on.

             

  • Everyone's background and path to IT is as unique as are their personalities, so you can never expect to understand someone's skills or level of accomplishment just by looking at how they got here.
  • Asking cookie-cutter technical questions rarely tells you anything except whether the individual on the other side of the table is good at answering cookie-cutter technical questions.
  • Questions like "tell me your biggest shortcoming" rarely elicit an honest answer (let alone foster a sense of trust or open-ness).
  • Questions that begin with "Tell me about a time when ...." are really an invitation to see if the candidate could improvise a work of fiction on the spot.
  • Asking deep technical questions usually just proves whether the candidate knows the same weird trivia about a certain technology that I know well, rather than whether they have meaningful skills to bring to the job.

 

After a bunch of really bad interviews, I was struggling with this issue yet again when I thought back to that day with my wife on the phone in Switzerland, and it all clicked. The next time I had a chance to interview a candidate, I threw out all the other frou-frou and tested my theory:

 

"Tell me how to get to your favorite restaurant."

 

The beauty of this question is that it's immediately obvious there's no wrong answer, and equally obvious that there's no way to "game" the system. You can't fake your way through it to give the answer the interviewer wants. You can't study a list of really good answers or crib off someone else. For the interviewer, this question also cancels out interviewer bias. Directions aren't dogmatic, and even if a candidate gives a different route to a location I know, that's not the point of the question anyway.

 

It's the way in which the candidate answers which reveals so much.

 

Do they ask clarifying questions? Things like “From here, or from your house?” or “Are you walking, biking, or driving?” or my favorite, “Are you a north-south person, a left-right person, or a ‘There's a K-mart on the corner’ person?”

 

Do they validate that I'm understanding their instructions? Anything from "Does that make sense?" to "Do you want a minute to write this down?"

 

Do they ensure that I'm even interested in going to that location? "Hey, my favorite restaurant is this weird little Thai place. Do you like Thai food?"

 

Do they skip all the niceties and just give me their set of directions, without preamble?

 

When I ask for clarification or even change the rules ("Oh, I forgot to tell you, I love public transportation. Can you get a bus to this place?") are they able to adapt?

 

And still, the point is that there's no right answer. I may be interviewing for a position where I need the employee to get right down to business, to avoid chit chat, to execute instructions as documented. Or I might be looking for a someone who can put themselves in the user's place, and therefore ask a lot of clarifying questions.

 

In the world of IT, there's an almost continuous focus on understanding where we've been, by collecting and analyzing baseline data; where we are, in terms of real time system statistics and performance metrics; and of where we're going, in terms of predictive analysis and data-based recommendations.

 

And maybe because of this, we can lose sight of two other data sets that are incredibly important: how we came to be here, and how we want to get to the step of our destination.

Ask a good server engineer where their server configuration is defined and the answer will likely be something similar to In my Puppet manifests. Ask a network administrator the same thing about the network devices and they'll probably look at you in confusion. Likely responses may include:

 

  • Uh, the device configuration is on the device, of course.
  • We take configuration backups every day!

 

Why is it that the server team seems to have gotten their act together while the network team is still working the same way they were twenty years ago?

 

The Device As The Master Configuration Source

 

To clarify the issue described, for many companies, the instantiation of the company's network policy is the configuration currently active on the network devices. To understand a full security policy, it's necessary to look at the configuration on a firewall. To review load balancer VIP configurations, one would log into the load balancer and view the VIPs. There's nothing wrong with that, as such, except that by viewing the configuration on a running device, we see what the configuration is, not what it was intended it to be.

"We see what the configuration IS, not what it was intended to be"

Think about that for a moment: taking daily backups of a device configuration tells us absolutely nothing about what we had intended for the policy to be; rather, it's just a series of snapshots of the current implemented configuration. Unless an additional step is taken to compare each configuration snapshot against some record of the intended policy, errors (and malicious changes) will simply be perpetuated as the new latest configuration for a device.

 

Contrast this to a Linux server managed by, for example, Puppet. The server team can define a policy saying that the server should run Perl v5.10.1, and code that into a Puppet manifest. A user with appropriate permissions may decide that for some code they are writing, they need to have Perl v5.16.1, so they install the new version, overwriting the old one. In the network world, a daily backup of the server configuration would now include Perl 5.16.1 and from then on that would implicitly be the version of Perl running on that device, even though that wasn't the owning team's intent. Puppet, on the other hand, runs periodically and checks the policy (as represented by the manifest) against what's running on the the device itself. When the Perl version is checked, the discrepancy will be identified, and Puppet will automatically restore v5.10.1 because that's the version specified in the policy. If the server itself dies, all a replacement server really needs is to load the OS with a basic configuration and a Puppet agent, and all the policies defined in the manifest can be instantiated on the new server just as they were on the old server. The main takeaways are that the running configuration is just an instantiation of the policy, and the running configuration is checked regularly to ensure that it is still an accurate representation of that policy.

"The running configuration is just an instantiation of policy"

Let's Run The Network On Puppet!

 

Ok, nice idea, but let's not get too far ahead of ourselves here. Puppet requires an agent to run on the device. This is easy to do on a server operating system, but many network devices run a proprietary OS, or limit access to the system sufficiently that it wouldn't be possible to install an agent (there are some notable exceptions to this). Even if a device offers a Puppet agent, creating the configuration manifests may not be straightforward, and will certainly require network engineers learning a new skillset.

 

Picking on Junos OS as an example, the standard Puppet library supports the configuration of physical interfaces, VLANs, LAGs, and layer 2 switching, and, well, that's it. Of course, there's something deeper here worth considering: the same manifest configuration works on an EX and an MX, despite the fact that the implemented configurations will look different, and that's quite a benefit. For example, consider this snippet of a manifest:

 

Puppet manifest snippet

 

On a Juniper EX switch, this would result in configuration similar to this;

 

Juniper EX configuration sample

 

On a Juniper MX router, the configuration created by the manifest is quite different:

 

Juniper MX configuration sample

 

The trade-off for learning the syntax for the Puppet manifest is that the one syntax can be applied to any platform supporting VLANs, without needing to worry about whether the device uses VLANs or bridge-domains. Now if this could be supported on every Juniper device and OS version and the general manifest configuration could be made to apply to multiple vendors as well, that would be very helpful.

 

Programmability

 

A manifest in this instance is a text file. Text files are easy for a script to create and edit, which makes automating the changes to these files relatively straightforward. Certainly compared to managing the process of logging into a device and issuing commands directly, creating a text file containing an updated manifest seems fairly trivial, and this may open the door to more automated configuration than might otherwise be possible.

 

Centralized Configuration Policy

 

Puppet has been used as an example above, but that does not imply that Puppet is the (only) solution to this problem; it's just one way to push out a policy manifest and ensure that the instantiated configuration matches what's defined by the policy. The main point is that as network engineers, we need to be looking at how we can migrate our configurations from a manual, vendor- (and even platform-) specific system to one which allows the key elements to be defined centrally, deployed (instantiated) to the target device, and for that configuration to be regularly validated against the master policy.

 

It's extremely difficult and, I suspect, risky, to jump in and attempt to deploy an entire configuration this way. Instead, maybe it's possible to pick something simple, like interface configurations or VLAN definitions, and seeing if those elements can be moved to a centralized location while the rest of the configuration is on-device. Over time, as confidence increases, additional parts of the configuration can be pulled into the policy manifest (or repository).

 

Roadblocks and Traffic Jams

 

There's one big issue with moving entire configurations into a centralized repo, which is that each vendor offers different ways to remotely configure the devices, some methods do not offer full coverage of the configuration syntax available via the CLI (I'm squinting at you, Cisco), and some operating systems are much more amenable to receiving and seamlessly (i.e., without disruption) applying configuration patches than others. Network device vendors are notoriously slow to make progress when it comes to network management, at least where it doesn't allow them to charge for their own solution to a problem, and developing a single configuration mechanism which could be applied to devices from all vendors is a non-trivial challenge (cf: OpenConfig). Nonetheless, we owe it to ourselves to keep nagging our vendors to make serious progress in this area and keep it high on the radar. When I look at trying to implement this kind of centralized configuration across my own company's range and age of hardware models and vendors, my head spins. We have to have a consistent way to configure our network devices, and given that most companies keep network devices for a least a few years, even if that was implemented today, it would still be 3-4 years before every device in a network supported that configuration mechanism.

"We owe it to ourselves to keep nagging our vendors"

 

On a more positive note, however, I will raise a glass to Juniper for being perhaps the most netdev friendly network device vendor for a number of years now, and  I will nod respectfully in the direction of Cumulus Networks who have kept their configurations as Unix standard as possible within the underlying Linux OS, thus opening them up to configuration via existing server configuration tools.

 

What Do You Do?

 

How do you manage the expectations that devices are implementing the policies they were intended to, and do not become an ever-changing source of truth for the intended policy? How do you push configurations to your devices, or does that idea scare you or seem impossible to do? If automation means swapping the CLI for a GUI, are on on board?

What do you do?

Please let me know; I hope to see a light at the end of the tunnel (and I hope it's not an oncoming train).

Sascha Giese

VMworld EMEA 2018

Posted by Sascha Giese Employee Oct 25, 2018

My life with SolarWinds is so interesting.

I’m sitting in an aircraft returning from Dubai where we visited the GITEX show, yet I’m writing a few lines to prepare for the next show we’ll be attending.

 

 

VMworld® EMEA in Barcelona, running between 5th – 9th November, doesn’t need an introduction to THWACK®!

Both sqlrockstar and I will be attending as well as a group of handpicked experts from our offices in EMEA and the U.S.

 

We will show the features released in VMAN 8.3 a few weeks ago, and we’re looking forward to hearing your opinions in regards to what we are working on for the next releases!

As usual, this will also be an opportunity for you to just drop by and have a chat with us to see how we can help.

There is no need to register upfront to speak with us, just drop by at booth B529 when you have a chance.

 

And yes, there will be swag. As we don’t intend to take anything back home with us, we need your help to pick it up!

Ryan Adzima

My Affection for AI

Posted by Ryan Adzima Oct 25, 2018

In my last post I mentioned one of the basic use cases for Artificial Intelligence (AI) in a modern network. AI can be utilized to not only bridge the informational gaps between functional teams, but can crunch that data down and provide predictions of growth or even failures for proactive management. Digesting large historical data sets and spitting out basic correlations is only scratching the surface of what AI can bring to your operations teams. AI can be applied to to bring more efficiency and better user experience to your network and applications. Performance and security monitoring are two of my many my favorites.

 

When it comes to running a large network, you can’t be everywhere and experience what the user sees at all times. Tracking down transient or isolated issues is hard enough, but when those issues could be a cascade of small failures, it could be nearly impossible to find. For example, in the case of wireless, a minor delay from the RADIUS server adds a second or two and then a hiccup in DHCP adds a couple more, and finally the captive portal adds yet another 3-5 seconds… eventually causing a timeout failure on the client. You may be able to review the logs on a single service and see somewhat normal behavior, but to track it all down takes a lot more insight into the network.

 

Now if we have that insight, it may still be difficult to find, but handing that data over to AI allows it to see anomalous events on the network. Most AI systems take feeds in from as many points within the network infrastructure and applications as you’ll give them; ingesting server logs, packets captures, and so much more. They build all the relationships and baselines automatically and they understand what it healthy on your network and what isn’t. An AI-monitored network may be able to warn you beforehand that this wireless issue is going to occur. By seeing into all the parts of a network and actually understanding “normal,” AI can start alerting before your users even have an idea there may be a problem. In the same way systems have monitored applications in the past, from web server to database to client, we can have AI trained to see the entire transaction in context and learn about it.

 

Securing networks and applications is a tough job. There are constant new threats coming out. Some are persistent threats from all areas of the world that could cause utter chaos for any company. The complexity and vectors are so advanced that a simple firewall and anti-virus application from the good ole days simply won’t cut it. Layer 7 deep-packet inspection and application filtering aren’t even good enough anymore. These days it takes an intelligent system to truly protect a company’s assets; ideally, a system with human-like intelligence that can look at the information in context and make the same decision you would, just faster. Taking the same approach as the performance example, AI can learn your network and the behavior of all the components: servers, clients, and applications. With the right products, over time it will build out “normal” behaviors and start alerting to things outside the defined norm. Hopefully you can train it up to the point to act automatically and quarantine or completely block a threat. We’ve heard of self-healing and self-defending networks. But now it’s not just marketing--we’re finally seeing it.

 

These are just a couple of high-level examples I use when talking about why I love AI for IT infrastructure. A good product can take a powerful tool like we’ve used for years and turn it into a highly customized system meant to monitor your infrastructure knowing all the ins and outs. All you have to do is add a little data.

 

P.S. You may or may not have noticed I’ve used the word “context” a few times. That’s because it’s an important part of running a company’s infrastructure; knowing when and when not to react based on corporate policies, politics, or even culture. How do you teach an AI system that sort of advanced context? That’s coming in a later post…

Is our contribution measured by what we're doing or how we're doing it? Are we providing value or are we just getting caught up in what's exciting? How do we ensure that we're seen as contributing despite appearing to be less busy? Do our efforts to automate sometimes take away from our value rather than adding?

 

Automate All of the Things!

 

Automation is based on the principle that our expertise should not be wasted on the manual execution of simple and repetitive tasks. Spending some extra time to offload these tasks and free ourselves up for more exciting undertakings just makes sense, right? Well, that depends on a few things.

 

The Human Factor (Working Hard, or Hardly Working?)

 

Perception is an easily-overlooked consideration. It's no secret that the work of IT professionals is sometimes seen as a bit arcane by our management and peers. There's a general understanding of what we do, but the details of how we get it done are often another story.

 

Are we being evaluated based on the value of the work that we do, or based on how busy we appear to be when we do it? If we minimize the daily grind, are we creating the impression that we're less valuable because we don't appear to be as busy? The answer to these questions is going to vary depending on our work environment, but it's important that we manage perception effectively from the beginning.

 

The Technical Factor (What's the Value Proposition?)

 

As IT professionals, we tend to be passionate about the work that we do, so it's really easy to get excited about coding our way out of the daily grind. When this happens, we sometimes let our excitement get the better of us and don't give due consideration to the real value of what it is that we're doing.

 

If we're spending a week to automate a task that previously wasted days each month, we have a reasonable return on our time investment. Yes, it might take a few months before we really see the benefits, but we will see them. On the other hand, if we're spending the same amount of time to address a task that only took a few minutes out of each month, we really have to start thinking about the efficient use of our time. We can be absolutely certain that our management and peers will be thinking about it if we're not.

 

The Whisper in the Wires

 

Are we approaching automation in the traditional way where we just develop a collection of scripts as we need them, or are we embracing a larger framework? At this point, I'm guessing that we have more of the former than the latter, but with a lot of push to take a more strategic approach. This is a good direction, but not if we're out of a job before we can get there because we didn't manage the expectations properly.

 

If we want to address concerns of perception and value proposition, we have to do more than script our pain away when we can. It's very difficult to manage either of these if we're addressing everything piecemeal. We need a consistent policy framework incorporated into a well-documented strategy, and we need to communicate that effectively to our peers.

 

A documented strategy addresses perception problems by providing a view into the process and setting expectations. A consistent policy framework keeps us from getting so caught up in our own custom script development that we fail to show value.

Back from Austin and the best THWACKcamp ever! Thanks to everyone that helped make this year's THWACKcamp so full of awesome. Each year we get a little bigger, a little better, and a little more bacon.

 

As always, here are some links from the Intertubz that I hope will hold your interest. Enjoy!

 

Ticketmaster buys a blockchain company to guard against ticket fraud

I'm willing to bet money that this is not going to solve the issue of fraudulent tickets. Nor, will it help people hate Ticketmaster any less.

 

Comcast complains it will make less money under Calif. net neutrality law

Interesting how Comcast is complaining about lost revenues, as opposed to complaining about end-user experience.

 

Up to 9.5 million net neutrality comments were made with stolen identities

Even if true, this won’t be enough for the FCC to reverse course on net neutrality. It will be up to the states, like California, to enact their own laws.

 

Microsoft sports director allegedly tried to embezzle $1.5 million and stole employees’ Super Bowl tickets

Nice reminder that the biggest corporate threats are often from within.

 

Major Facebook Shareholders Join Call to Boot Mark Zuckerberg as Chairman

This would be a good first step for Facebook to win back some trust. It would also be the good first step for Facebook to remove Zuckerberg as CEO, too. These items might be related.

 

Crypto is the Mother of All Scams and (Now Busted) Bubbles While Blockchain Is The Most Over-Hyped Technology Ever, No Better than a Spreadsheet/Database

Grab some popcorn and enjoy.

 

Fishcoin

Fishcoin, proving that blockchain hasn’t hit the lowest level of stupidity yet.

 

For those that were wondering, here's the Bacon Pie given to me during the close of THWACKcamp this year. Thanks to thegreateebzies and DanielleH for making this happen:

 

By Paul Parker, SolarWinds Federal & National Government Chief Technologist

 

Blockchain is already one of the top five most important technologies in the IT strategy of 12% of surveyed public sector employees, according to the recent IT Trends Report from SolarWinds. The U.K. government is heavily encouraging blockchain-based technologies, with a £19 million investment in innovative product or service delivery projects.

 

The promise of blockchain lies in how it can help accelerate the verification processes using many connected computers to store blocks of information. Blockchain is transparent by design, allowing data to be shared more easily between buyers and sellers.

 

Blockchain has the potential to revolutionize the way government agencies acquire services and solutions, but, as the financial world has discovered, network monitoring and management strategies play a critical role in blockchain’s success within public sector organizations.

 

Distributed network monitoring and visibility

 

The success of blockchain in procurement is dependent on a high throughput of transactions and low latency. Unfortunately, those goals can be difficult to achieve over a disparate network. In addition, according to the SolarWinds IT Trends Report, 58% of public sector IT professionals surveyed felt their network was not working at optimum levels.

 

On-prem and hybrid network infrastructures are highly distributed. Teams need to be able to monitor data as it passes between all of these services to help ensure that their networks are operating efficiently and dependable. The best way to get this insight is by monitoring strategies that are designed to provide access and visibility into the entirety of the network, wherever it may exist.

 

Resilient, but not impervious

 

Blockchain technology has been suggested as potentially more secure than alternatives, if

used correctly. This is due to its decentralized nature, which can make it a harder target for hackers to hit. Agencies must still make sure that they are maintaining the same high level of security practices they would do otherwise.

 

It is also important to remember that blockchain is a relatively new technology. As such, there may be vulnerabilities that have not yet been exposed. At this very moment, it is likely that many hackers are attempting to identify and exploit blockchain vulnerabilities. Maintaining a sound security position can help agencies fortify themselves against those efforts while taking strides to improve their procurement processes.

 

Innovation beyond the procurement process

 

Blockchain has considerable potential for the public sector in the U.K. It has been shown to be innovative and powerful in other industries and could very possibly revolutionize government procurement processes in the near future. However, this is only the start of the potential blockchain revolution. The same technology could work to track government loans and spending, protect critical infrastructure, or even help to deliver on the government’s foreign aid commitments in a

more secure and transparent way.

 

Success with blockchain, though, is contingent on supporting the technology with comprehensive network management. Clear visibility across all nodes and management of performance levels will be integral to helping maintain security and preventing blockages in the network. Only then can blockchain and distributed ledger technology successfully transform government digital services.

 

Find the full article on Open Access Government.

 

The SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates.  All other trademarks are the property of their respective owners.

There is a traditional market research technique called voice of the customer (VoC). Many people are familiar with this process, which involves surveys, interviews, and even watching customers interact with your products. Some companies use customer advisory boards (CAB) to collect feedback, too.

 

There’s no shortage of ways to get feedback from customers. The #hardtruth is that many of the ways are cold and unfeeling. It can take years for feedback to work its way into a product. I find that many companies talk about valuing customer feedback but fall short of connecting with their customers in a meaningful way.

 

When I joined Confio in 2010, my perception of the VoC process changed. It was there that I understood how it should work. Shortly after coming on board, I provided feedback regarding the monitoring of databases running inside of VMware. Have a look at this screen:

 

 

That stacked view, showing metrics inside the engine, the guest, the host, and storage? That’s what I told our dev team a DBA needed to see. Seven years later and that view still stands out when giving demos. The annotations were also something I requested. I wanted DBAs to know if there was an event on the host for that VM.

 

I still remember the feeling I had when I saw my feedback was used to make our product better. Imagine if a customer had the opportunity to do the same.

 

Well, imagine no more! The SolarWinds Usabili-buddy program is your opportunity! Go here to sign up for the program.

 

Usabili-buddy Example

Here’s an example of how the program has had a positive impact for everyone.

 

While working on the upgrade to alerting, the UX team was working with customers and gathering feedback. While there was already a roadmap and screens of how the UI for alerting would look, these customers noticed a gap in the feature. The customers recalled experiences in the past where they had misconfigured an alert and accidentally triggered an alert storm.

 

These customers wanted a way to avoid spamming end users due to a misconfigured alert. As a group, they came up with the feature below, on their own

 

 

I know, you can't see that, here is a closer look:

 

 

This was NOT on the original roadmap, but product management loved this idea. It was included in the very next release. 

 

Other companies talk about listening to their customers. We don’t just talk the talk. You can see the impact that the Usabili-buddy program and UX team has had over the years.

 

At SolarWinds, we’re listening. We know that we’re all in this together.

 

We don’t treat customers as revenue. We build relationships with them.

 

When you become a customer, you are a member of a community, not just a number in Salesforce.

 

Help Us, Help You, Help Them

 

We have 55 products now.

 

Managing many products is a challenging task. But we are fortunate that our user community makes it easier. The quality of customers we have allows for better feedback, and better feedback leads to better products. Thank you for giving us your time, helping to make the products better for the next user.

 

But they aren’t our products. They are yours. We just maintain the source code.

 

Usability improvements are part of every release, and the UX team meets with product development on a daily basis. In each product cycle, UX plays an important role. It's a cycle of continuous improvement, and a project that is never done.

 

At the end of the day we all want the same thing: happiness. We want happy customers, enjoying long weekends, without worry.

 

If you have an idea, or just want to help, join the program and share.

 

Help us, help you, help them.

At what point does the frequency and volume of “it will only take a second to change” become too much to bear and force us to adopt a network automation strategy? Where is the greatest resistance to change? Is it in the technical investment required, or is it the habit of falling back to the old way of doing things because it's "easier" than the new way?

 

The Little Things

 

We all have those little tasks that we can accomplish in a heartbeat. They're the things we've done so many times that the commands required have almost become muscle memory, taking little to no thought to enter. They're the easy part of our jobs, right? Perhaps, but they can also be the most time consuming. There's a reason those commands have become so ingrained. We perform them far more than we should, but haven't necessarily figured that out yet... well, not until now anyway.

 

The solution? Network automation! Let's get all of those mind-numbingly simple day-to-day tasks taken care of by an automation framework so that we can free ourselves up for work that's actually challenging and rewarding. It's that easy! Or is it?

 

The Huge Amount of Work Required to Avoid a Huge Amount of Work

 

Automation, even with the best of tools, is a lot of work. That process itself is something that we will wish could be automated before we're done. There's the needs analysis; the evaluation and selection of an automation framework; training of staff to use it; and the building, documentation, and maintenance of the policies themselves.

 

When a significant portion of the drive for automation comes from overload, the additional technical workload of building an automation framework in parallel to the current way of doing things can be daunting. Yes, it's definitely a case of working smarter rather than harder, but that's still hard to swallow when we're buried in the middle of both.

 

The Cultural Shift

 

People are creatures of habit, especially when those habits are deeply ingrained. When all of those little manual network changes have reached the point that they can be done without real thought, we can be absolutely sure that they're deeply seated and aren't going to be easy to give up. The actual technical work to transition to network automation was only half of the challenge. Now we have to deal with changing people's thinking on the matter.

 

Here's a place where we really shouldn't serve two masters. If there isn't full commitment to the new process, the investment in it yields diminishing returns. The automation framework can make network operations far more efficient, but not if everyone is resistant to it and is continuing to do things the old way. There needs to either be incentive to adopt the new framework 100% or discouragement from falling into habitual behaviour. This could even represent a longer process than the technical side of things.

 

What Must Be Done

 

Neither the technical hurdles nor the human ones remove the ultimate need to automate. The long-term consequences of repeatedly wasting time on simple tasks, both to individuals' technical skills and job satisfaction and the efficiency of the organization, makes a traditional approach to networking unsustainable. This is especially true at any kind of scale. Growth and additional workload only serve to make the problem more apparent and the solution more difficult to implement. Still, there's no question that it needs to be done. The real questions revolve around how best to handle the transition.

 

The Whisper in the Wires

 

It's difficult to say what's harder, the technical transition to network automation itself, or ensuring that it becomes the new normal. By the time we reach a point where it becomes necessary, we may have painted ourselves into a corner with a piled-up workload that should have been automated in the first place. It also represents a radical change in how things are done, which is going to produced mixed reactions that have to be factored in.

 

For those of you who have automated your networks, whether in large installations or small, at what point did you realize that doing things the old was no longer a viable option? What did you do to ensure a successful transition both technically and culturally?

In part 1 of this series, we covered some of the most prevalent and most promising cybersecurity models and frameworks available today. These are all tools that can help you determine the size and shape of the current information security landscape, and where you and your organization are within it. We also realized that even with all of this, you still can’t answer some fundamental questions about the specific technology you need to protect your digital infrastructure. As promised, I’m going to spend the next four posts covering the four critical domains of IT infrastructure security and the categories they each contain. Let’s start today with the perimeter.

 

Domain: Perimeter

The perimeter domain can be seen as the walls of a castle. These technologies are meant to keep information in and attackers out.  In many cases, a Demilitarized Zone (DMZ) and other public network services are exposed to the routable internet via systems within the perimeter domain. Additionally, an organization may have multiple perimeters, similar to an outer wall and an inner wall protecting a castle.

 

The categories in the perimeter domain are network security, email security, web security, DDoS protection, data loss prevention (DLP), and ecosystem risk management.

 

Category: Network Security

Network security is typically the primary line of defense for traffic entering or leaving an organization’s network, providing a first-look analysis of traffic inbound and a last-look at traffic leaving your network’s span of control. The primary products in this category are firewalls, network intrusion detection/prevention systems (IDS/IPS), deep packet inspection (DPI), and other security gateways. Today, we rely on so-called next generation firewalls (NGFW) to package the functionality of what used to be many devices into a single appliance or virtual machine. More and more we are facing the challenges of deperimeterization as BYOD and cloud services stretch and blur the previously hard lines that defined our networks' boundaries. This is leading to the rise of software defined perimeter (SDP) tools that push security to the very edge of your new multi-cloud network.

 

Category: Email Security

Email has become a nearly universal communication medium for individuals and businesses alike, which also makes it a prime attack vector. Spam (Unsolicited Commercial Email - UCE) has been a nuisance for many years, and now phishing, click-bait, and malware attachments create real organizational threats. These attacks are so prolific that it often makes sense to layer email-specific security measures on top of network and endpoint solutions. Included within this category are email security products that offer antivirus, anti-spam, anti-phishing, and anti-malware features. Additional tie-ins to DLP and encryption are also available.

 

Category: Web Security

Much of our online activity centers around the web. This is increasingly true in our more and more SaaS-focused world. Web security seeks specifically to protect your users from visiting malicious websites. URL filtering (whitelist/blacklist) and other DNS tools fit into this category. Today, known and emerging threats are addressed within this category using Advanced Threat Protection (ATP) capabilities to analyze, diagnose, and dynamically implement rules governing web access in real-time.  This capability is typically provided using a subscription service to a threat database that has an influence on data exchange or name resolution traffic traversing a network.

 

Category: DDoS Protection

Pundits and others spend a lot of time talking about “going digital.” What this likely means to you is that internet access is crucial to your business. Your employees need to reach the information and services they need, and your customers need to reach your website and other applications. Denial of Service (DoS) and Distributed Denial of Service (DDoS) attacks generate malformed/malicious packets or an excessive amount of inbound traffic to flood systems responsible for responding to valid queries.  Under such an attack, systems are unable to keep up with responses. D/DoS protection services recognize these attack techniques and implement methods to block the attempts or clean the inbound data streams so that only the valid traffic remains.

 

Category: Data Loss Prevention

Data is the new gold. Your intellectual property is now made up of ones and zeros, so you can’t lock it in a file cabinet or a safe. You can still protect it though – probably better than you could when it was on paper. Data loss prevention (DLP) tools classify, analyze, and react to data at rest, in use, or in motion. DLP ensures that your data remains available to those who need it, and out of the hands of would-be attackers.

 

Category: Ecosystem Risk Management

Your cybersecurity is only as strong as the weakest link in your ecosystem. A vulnerability anywhere in the supply chain escalates organizational risk and jeopardizes productivity, profitability, and reputation. Partner, supplier, and vendor security risk is a major area that cannot be ignored as a business issue any longer. You need to be able to continuously identify, monitor, and manage risk to improve the cyberhealth of your vendor ecosystem.

 

Up Next

Obviously, the castle walls are only one part of a well-crafted defense. In the next three posts of this 6-part series, we’ll cover the remaining domains of endpoint & application, identity & access, and visibility & control. In the final post, we’ll look at the full model that these four domains create, how it fits into the broader cybersecurity landscape, and provide some advice on how to put it all into practice. Stay tuned!

Filter Blog

By date: By tag:

SolarWinds uses cookies on its websites to make your online experience easier and better. By using our website, you consent to our use of cookies. For more information on cookies, see our cookie policy.