Showing results for 
Search instead for 
Did you mean: 
Create Post

Where Is Your Configuration Source Of Truth?

Level 13

Ask a good server engineer where their server configuration is defined and the answer will likely be something similar to In my Puppet manifests. Ask a network administrator the same thing about the network devices and they'll probably look at you in confusion. Likely responses may include:

  • Uh, the device configuration is on the device, of course.
  • We take configuration backups every day!

Why is it that the server team seems to have gotten their act together while the network team is still working the same way they were twenty years ago?

The Device As The Master Configuration Source

To clarify the issue described, for many companies, the instantiation of the company's network policy is the configuration currently active on the network devices. To understand a full security policy, it's necessary to look at the configuration on a firewall. To review load balancer VIP configurations, one would log into the load balancer and view the VIPs. There's nothing wrong with that, as such, except that by viewing the configuration on a running device, we see what the configuration is, not what it was intended it to be.


Think about that for a moment: taking daily backups of a device configuration tells us absolutely nothing about what we had intended for the policy to be; rather, it's just a series of snapshots of the current implemented configuration. Unless an additional step is taken to compare each configuration snapshot against some record of the intended policy, errors (and malicious changes) will simply be perpetuated as the new latest configuration for a device.

Contrast this to a Linux server managed by, for example, Puppet. The server team can define a policy saying that the server should run Perl v5.10.1, and code that into a Puppet manifest. A user with appropriate permissions may decide that for some code they are writing, they need to have Perl v5.16.1, so they install the new version, overwriting the old one. In the network world, a daily backup of the server configuration would now include Perl 5.16.1 and from then on that would implicitly be the version of Perl running on that device, even though that wasn't the owning team's intent. Puppet, on the other hand, runs periodically and checks the policy (as represented by the manifest) against what's running on the the device itself. When the Perl version is checked, the discrepancy will be identified, and Puppet will automatically restore v5.10.1 because that's the version specified in the policy. If the server itself dies, all a replacement server really needs is to load the OS with a basic configuration and a Puppet agent, and all the policies defined in the manifest can be instantiated on the new server just as they were on the old server. The main takeaways are that the running configuration is just an instantiation of the policy, and the running configuration is checked regularly to ensure that it is still an accurate representation of that policy.


Let's Run The Network On Puppet!

Ok, nice idea, but let's not get too far ahead of ourselves here. Puppet requires an agent to run on the device. This is easy to do on a server operating system, but many network devices run a proprietary OS, or limit access to the system sufficiently that it wouldn't be possible to install an agent (there are some notable exceptions to this). Even if a device offers a Puppet agent, creating the configuration manifests may not be straightforward, and will certainly require network engineers learning a new skillset.

Picking on Junos OS as an example, the standard Puppet library supports the configuration of physical interfaces, VLANs, LAGs, and layer 2 switching, and, well, that's it. Of course, there's something deeper here worth considering: the same manifest configuration works on an EX and an MX, despite the fact that the implemented configurations will look different, and that's quite a benefit. For example, consider this snippet of a manifest:


On a Juniper EX switch, this would result in configuration similar to this;


On a Juniper MX router, the configuration created by the manifest is quite different:


The trade-off for learning the syntax for the Puppet manifest is that the one syntax can be applied to any platform supporting VLANs, without needing to worry about whether the device uses VLANs or bridge-domains. Now if this could be supported on every Juniper device and OS version and the general manifest configuration could be made to apply to multiple vendors as well, that would be very helpful.


A manifest in this instance is a text file. Text files are easy for a script to create and edit, which makes automating the changes to these files relatively straightforward. Certainly compared to managing the process of logging into a device and issuing commands directly, creating a text file containing an updated manifest seems fairly trivial, and this may open the door to more automated configuration than might otherwise be possible.

Centralized Configuration Policy

Puppet has been used as an example above, but that does not imply that Puppet is the (only) solution to this problem; it's just one way to push out a policy manifest and ensure that the instantiated configuration matches what's defined by the policy. The main point is that as network engineers, we need to be looking at how we can migrate our configurations from a manual, vendor- (and even platform-) specific system to one which allows the key elements to be defined centrally, deployed (instantiated) to the target device, and for that configuration to be regularly validated against the master policy.

It's extremely difficult and, I suspect, risky, to jump in and attempt to deploy an entire configuration this way. Instead, maybe it's possible to pick something simple, like interface configurations or VLAN definitions, and seeing if those elements can be moved to a centralized location while the rest of the configuration is on-device. Over time, as confidence increases, additional parts of the configuration can be pulled into the policy manifest (or repository).

Roadblocks and Traffic Jams

There's one big issue with moving entire configurations into a centralized repo, which is that each vendor offers different ways to remotely configure the devices, some methods do not offer full coverage of the configuration syntax available via the CLI (I'm squinting at you, Cisco), and some operating systems are much more amenable to receiving and seamlessly (i.e., without disruption) applying configuration patches than others. Network device vendors are notoriously slow to make progress when it comes to network management, at least where it doesn't allow them to charge for their own solution to a problem, and developing a single configuration mechanism which could be applied to devices from all vendors is a non-trivial challenge (cf: OpenConfig). Nonetheless, we owe it to ourselves to keep nagging our vendors to make serious progress in this area and keep it high on the radar. When I look at trying to implement this kind of centralized configuration across my own company's range and age of hardware models and vendors, my head spins. We have to have a consistent way to configure our network devices, and given that most companies keep network devices for a least a few years, even if that was implemented today, it would still be 3-4 years before every device in a network supported that configuration mechanism.


On a more positive note, however, I will raise a glass to Juniper for being perhaps the most netdev friendly network device vendor for a number of years now, and  I will nod respectfully in the direction of Cumulus Networks who have kept their configurations as Unix standard as possible within the underlying Linux OS, thus opening them up to configuration via existing server configuration tools.

What Do You Do?

How do you manage the expectations that devices are implementing the policies they were intended to, and do not become an ever-changing source of truth for the intended policy? How do you push configurations to your devices, or does that idea scare you or seem impossible to do? If automation means swapping the CLI for a GUI, are on on board?


Please let me know; I hope to see a light at the end of the tunnel (and I hope it's not an oncoming train).

Level 13

It'll be someone with a torch bringing you more work


Good write up

Level 14

At my work, it's not that simple.  It's almost like each department is it's own company.  As the "network team" Orion person, my focus stays on my inventory, which we use NCM for configuration management.  However, we have had a really hard time trying to deploy the firmware repository (upgrade) feature in NCM.  Hopefully one day it will be fixed. 

Level 13

There may be something in what you say... 🙂

Level 13

Thanks, Vinay.

Level 13

Watch this space; I have a post coming up which talks about just such a problem (although the solutions are rarely technical I'm afraid).

A configuration is reality, not policy.  

It is our responsibility, as administrators who are required to interpret the intent of policy, to implement that intent in the configuration.

Thus we change policy and intent into reality:  a configuration.

(You won't find THIS out in a meme search)


Level 13

Nicely put 🙂

Now, why do I have a headache?

You must accept a headache is your new reality as you interpret policies.


You are falling into a deep OSPF loop, where each new path and stub reveals a higher cost than the previous one.  Your electrons are slowing, cooling down.  Soon they will stop, and you will exhale steadily until the headache is gone, the policy complete.

Level 20

It is hard to do this... for us it's not just a good configuration but also one that will survive the SCAP scanner and score a high percentage.

Level 8

Good read! And to answer your question:

We are wading in to using configuration templates with Ansible for network automation for deployments, which centralizes the configuration and breaks it in to modular parts. This helps because we simply need to update the templates in order to implement new policies in to the configuration of network devices.

Then we can work towards applying those templates to in-production network equipment that may not have the correct configuration, ensuring compliance.

Level 13

"There is nothing wrong with your television set."

Level 13

Ansible seems to be the most common solution I'm hearing about right now. I don't know if it's the best, but certainly it's common Thank you for sharing!

Level 9


Level 8

And there are community supported (or not ) dynamic inventory plugings for Ansible already:

- GitHub - alexkross/swipam-script: Ansible dynamic inventory script for SolarWinds IPAM product

- GitHub - cbabs/solarwinds-ansible-inv: Dynamic Inventory for Solar Winds hosts in Ansible

First for IPAM. Second for NPM.

Level 7

We use SolarWinds Config Mgr COMPLIANCE reports to review saved device configs, generating reports where deployed configs don't match our basic security and config settings.  We set up rules with regular expressions that match or an exception is flagged.  Granted, it's not as fancy as Ansible or some other such tool, but it's built into our NPM.  One thing, it would be useful if the scripting capabilities of those rules was a little more full-featured and easier to write.