As a network engineer, I don't think I've ever had the pleasure of having every device configured consistently in a network. But what does that even mean? What is consistency when we're potentially talking about multiple vendors and models of equipment?
There Can Only Be One (Operating System)
Claim: For any given model of hardware there should be one approved version of code deployed on that hardware everywhere across an organization.
Response: And if that version has a bug, then all your devices have that bug. This is the same basic security paradigm that leads us to have multiple firewall tiers comprising different vendors for extra protection against bugs in one vendor's code. I get it, but it just isn't practical. The reality is that it's hard enough upgrading device software to keep up with critical security patches, let alone doing so while maintaining multiple versions of code.
Why do we care? Because different versions of code can behave differently. Default command options can change between versions; previously unavailable options and features are added in new versions. Basically, having a consistent revision of code running means that you have a consistent platform on which to make changes. In most cases, that is probably worth the relatively rare occasions on which a serious enough bug forces an emergency code upgrade.
Corollary: The approved code version should be changing over time, as necessitated by feature requirements, stability improvements, and critical bugs. To that end, developing a repeatable method by which to upgrade code is kind of important.
Consistency in Device Management
Claim: Every device type should have a baseline template that implements a consistent management and administration configuration, with specific localized changes as necessary. For example, a template might include:
- NTP / time zone
- SNMP configuration
- Management interface ACLs
- Control plane policing
- AAA (authentication, authorization, and accounting) configuration
- Local account if AAA authentication server fails*
(*) There are those who would argue, quite successfully, that such a local account should have a password unique to each device. The password would be extracted from a secure location (a
break glass type of repository) on demand when needed and changed immediately afterward to prevent reuse of the local account. The argument is that if the password is compromised, it will leave all devices susceptible to accessibility. I agree, and I tip my hat to anybody who successfully implements this.
Response: Local accounts are for emergency access only because we all use a centralized authentication service, right? If not, why not? Local accounts for users are a terrible idea, and have a habit of being left in place for years after a user has left the organization.
NTP is a must for all devices so that syslog/SNMP timestamps are synced up. Choose one timezone (I suggest UTC) and implement it on your devices worldwide. Using a local time zone is a guaranteed way to mess up log analysis the first time a problem spans time zones; whatever time zone makes the most sense, use it, and use it everywhere. The same time zone should be configured in all network management and alerting software.
Other elements of the template are there to make sure that the same access is available to every device. Why wouldn't you want to do that?
Corollary: Each device and software version could have its own limitations, so multiple templates will be needed, adapted to the capabilities of each device.
Claim: Pick a device naming standard and stick with it. If it's necessary to change it, go back and change all the existing devices as well.
Response: I feel my hat tipping again, but in principle this is a really good idea. I did work for one company where all servers were given six-letter dictionary words as their names, a policy driven by the security group who worried that any kind of semantically meaningful naming policy would reveal too much to an attacker. Fair play, but having to remember that the syslog servers are called WINDOW, BELFRY, CUPPED, and ORANGE is not exactly friendly. Particularly in office space, it can really help to be able to identify which floor or closet a device is in. I personally lean toward naming devices by role (e.g. leaf, access, core, etc.) and never by device model. How many places have switches called
Chicago-6500-01 or similar? And when you upgrade that switch, what happens? And is that 6500 a core, distribution, access, or maybe a service-module switch?
Corollary: Think the naming standard through carefully, including giving thought to future changes.
Why Do This?
There are more areas that could and should be consistent. Maybe consider things like:
- an interface naming standard
- standard login banners
- routing protocol process numbers
- vlan assignments
- BFD parameters
- MTU (oh my goodness, yes, MTU)
But why bother? Consistency brings a number of obvious operational benefits.
- Configuring a new device using a standard template means a security baseline is built into the deployment process
- Consistent administrative configuration reduces the number of devices which, at a critical moment in troubleshooting, turn out to be inaccessible
- Logs and events are consistently and accurately timestamped
- Things work, in general, the same way everywhere
- Every device looks familiar when connecting
- Devices are accessible, so configurations can be backed up into a configuration management tool, and changes can be pushed out, too
- Configuration audit becomes easier
The only way to know if the configurations are consistent is to define a standard and then audit against it. If things are set up well, such an audit could even be automated. After a software upgrade, run the audit tool again to help ensure that nothing was lost or altered during the process.
What does your network look like? Is it consistent, or is it, shall we say, a product of organic growth? What are the upsides -- or downsides -- to consistency like this?