How do you test firewall changes?

How do you test firewall changes? Firewalls help us keep intruders our of our networks and precious data in them, and changes to those safeguards can be precarious to say the least. To minimize the likelihood that a change will cause some sort of problem -- such as creating a hole in our defenses or blocking a critical service -- it's important to have a firewall management process in place that includes a procedure for testing and implementing changes in a way that minimizes the chance for any negative impact.

When I considered the myriad approaches organizations could take to this end, I came up with four general categories:

  • Implementing changes on the fly and then testing them
  • Implementing changes during maintenance periods and then testing them
  • Testing changes in a lab and then implementing them
  • Testing changes with change modeling software and then implementing them

Changing Firewalls on the Fly

You have to be pretty brave to implement firewall changes in this fashion. Doing so successfully assumes you have a deep knowledge of your network infrastructure and how firewalls and routing work. If you don't have this requisite knowledge, chances are you're using this approach because you don't know any better. The risks inherent in this approach include implementing a change that breaks something you won't know is broken until something catastrophic happens, or testing a change while some vulnerability is exploited before you have the opportunity to find it (and address it) for yourself. Of the four approaches I listed, this is by far the least desirable.

Changing Firewalls and Testing During Maintenance Periods

This is similar to the first approach, but it's considerably more conservative. This assumes your organization has a specific schedule in place that defines both production windows and maintenance windows. Such a schedule usually dictates that nothing but critical changes are made to any business-critical network systems within a production window, reserving all routine changes for the maintenance windows. The maintenance windows usually fall during off-peak times, which significantly reduces the risk of a service being blocked by a faulty change when someone needs it. However, the risk of exposure still exists with this approach, so it's only moderately more desirable than the on-the-fly option.

Testing Firewall Changes in a Lab First

This approach is at the top of the list for DIY testing options. It assumes you have a non-production lab environment in which to test your firewall changes before you implement them. In a perfect world, the infrastructure in your lab environment would emulate your production environment exactly, so you can be absolutely sure the changes you test in the lab will behave just as reliably (or unreliably) in production. Presuming your testing goes 100% according to plan, you can use the same change scripts in your production environment that you used to implement the changes in the lab. The main drawback here is scalability.

Testing Firewall Changes with a Change Modeling Tool

As your network grows, you'll feel increasing pressure to upgrade your testing effort to something more automated. A comprehensive firewall management tool not only offers a central repository for your firewall configs, it also offers change modeling functionality. Change modeling allows you to test your changes without having to create and maintain a test environment, or manually test every single scenario. Ideally, the tool would also allow change modeling to be collaborative so you can distribute the testing load between security specialists, application engineers, and firewall administrators. For organizations of any size, automation is the key to efficiency and accuracy.

A General Best Practice

Regardless of the method you use to test and roll out your firewall changes, a general best practice is to expand your testing effort beyond the change at hand. An important (and, sadly, often-missed) aspect of firewall configuration change management is testing to ensure changes to accommodate or block one service don't have some other negative effect on the network landscape at large. Document and test all business-critical services with every change, and make sure no known vulnerabilities are exposed when you implement any sort of change. Again, automation helps make this a lot easier than it would be otherwise; but if you're married to a more manual process, documentation and due diligence will work just as well (though probably more slowly).

So how do you roll (out firewall changes)?  If I've missed anything in this post, let me know in the comments section. Or, if you'd just like to share your process or experiences, feel free chime in.

Thwack - Symbolize TM, R, and C