I read an interesting thread the other day about a network engineer that tried to use an automated tool to schedule a rolling switch upgrade. He realized after it completed and the switches restarted that he had the wrong image for the device and they weren't coming back up. It was about fifty switches in total, which resulted in a major outage for his organization.
What struck me about the discussion thread was first that he wondered why the tool didn't stop him from doing that thing. The second was that the commenters responded that it wasn't the tool's job to sanity check his inputs. The end result was likely a severe discipline discussion on the part of the original engineer.
Tools are critical in network infrastructure today. Things have become far too complicated for us to manage on our own. SolarWinds makes a large number of tools to help us keep our sanity. But it is the fault of the tool when it is used incorrectly?
Tools are only as good as their users. If you smash your fingers over and over again with a hammer, does that mean the hammer is defective? Or is the fact that you're holding it wrong in play? Tools do their jobs whether you're using the correctly or incorrectly.
2016 is the year when we should all stop blaming the tools for our problems. We need to reassess our policies and procedures and find out how we can use tools effectively. We need to stop pushing all of the strange coincidences and problems off onto software that only does what it's told to do. Software tools should be treated like the tools we use to build a house. They are only useful if used properly.
Best practices should include sanity checking of things before letting the tool do the job. A test run of an upgrade on one device before proceeding to do twenty more. A last minute write up of the procedure before implementing it. Checking the inputs for a monitoring tool before swamping the system with data. Tapping the nail a couple of times with the hammer to make sure you can pull your fingers away before you start striking.
It's a new year full of new resolutions. Let's make sure that we stop blaming our tools and use them correctly. What do you think? Will you resolve to stop blaming the tools? Leave me a comment with some of your biggest tool mishaps!