cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

New Coder: STOP WHAT YOU'RE DOING!

Level 13

You may be wondering why, after creating four blog posts encouraging non-coders to give it a shot, select a language and break down a problem into manageable pieces, I would now say to stop. The answer is simple, really: not everything is worth automating (unless, perhaps, you are operating at a similar scale to somebody Amazon).

The 80-20 Rule

Here's my guideline: figure out what tasks take up the majority (i.e. 80%) of your time in a given time period (in a typical week perhaps). Those are the tasks where making the time investment to develop an automated solution is most likely to see a payback. The other 20% are usually much worse candidates for automation where the cost of automating it likely outweighs the time savings.

As a side note, the tasks that take up the time may not necessarily be related to a specific work request type. For example, I may spend 40% of my week processing firewall requests, and another 20% processing routing requests, and another 20% troubleshooting connectivity issues. In all of these activities, I spend time identifying what device, firewall zone, or VRF various IP addresses are in, so that I can write the correct firewall rule, or add routing in the right places, or track next-hops in a traceroute where DNS is missing. In this case, I would gain the most immediate benefits if I could automate IP address research.

I don't want to be misunderstood; there is value in creating process and automation around how a firewall request comes into the queue, for example, but the value overall is lower than for a tool that can tell me lots of information about an IP address.

That Seems Obvious

You'd think that it was intuitive that we would do the right thing, but sometimes things don't go according to plan:

Feeping Creatures!

Once you write a helpful tool or an automation, somebody will come back and say, Ah, what if I need to know X information too? I need that once a month when I do the Y report. As a helpful person, it's tempting to immediately try and adapt the code to cover every conceivable corner case and usage example, but having been down that path, I counsel against doing so. It typically makes the code unmanageably complex due to all the conditions being evaluated and worse, it goes firmly against the 80-20 rule above. Feeping Creatures is a Spoonerism referring to Creeping Features, i.e. an always expanded feature list for a product.

A Desire to Automate Everything

There's a great story in What Do You Care What Other People Think (Richard Feynman) that talks about Mr. Frankel, who had developed a system using a suite of IBM machines to run the calculations for the atomic bomb that was being developed at Los Alamos.

"Well, Mr. Frankel, who started this program, began to suffer from the computer disease that anybody who works with computers now knows about. [...] Frankel wasn't paying any attention; he wasn't supervising anybody. [...] (H)e was sitting in a room figuring out how to make one tabulator automatically print arctangent X, and then it would start and it would print columns and then bitsi, bitsi, bitsi, and calculate the arc-tangent automatically by integrating as it went along and make a whole table in one operation.

Absolutely useless. We had tables of arc-tangents. But if you've ever worked with computers, you understand the disease -- the delight in being able to see how much you can do. But he got the disease for the first time, the poor fellow who invented the thing."

It's exciting to automate things or to take a task that previously took minutes, and turn it into a task that takes seconds. It's amazing to watch the 80% shrink down and down and see productivity go up. It's addictive. And so, inevitably, once one task is automated, we begin looking for the next task we can feel good about, or we start thinking of ways we could make what we already did even better. Sometimes the coder is the source of creeping features.

It's very easy to lose touch with the larger picture and stay focused on tasks that will generate measurable gains. I've fallen foul of this myself in the past, and have been delighted, for example, with a script I spent four days writing, which pulled apart log entries from a firewall and ran all kinds of analyses on it, allowing you to slice the data any which way and generate statistics. Truly amazing! The problem is, I didn't have a use for most of the stats I was able to produce, and actually I could have fairly easily worked out the most useful ones in Excel in about 30 minutes. I got caught up in being able to do something, rather than actually needing to do it.

And So...

Solve A Real Problem

Despite my cautions above, I maintain that the best way to learn to code is to find a real problem that you want to solve and try to write code to do it. Okay, there are some cautions to add here, not the least of which is to run tests and confirm the output. More than once, I've written code that seemed great when I ran it on a couple of lines of test data, but then when I ran it on thousands of lines of actual data, I discovered oddities in the input data, or in the loop that processes all the data reusing variables carelessly or similar. Just like I tell my kids with their math homework, sanity check the output. If a script claims that a 10Gbps link was running at 30Gbps, maybe there's a problem with how that figure is being calculated.

Don't Be Afraid to Start Small

Writing a Hello World! script may feel like one of the most pointless activities you may ever undertake, but for a total beginner, it means something was achieved and, if nothing else, you learned how to output text to the screen. The phrase, "Don't try to boil the ocean," speaks to this concept quite nicely, too.

Be Safe!

If your ultimate aim is to automate production device configurations or orchestrate various APIs to dance to your will, that's great, but don't start off by testing your scripts in production. Use device VMs where possible to develop interactions with different pieces of software. I also recommend starting by working with read commands before jumping right in to the potentially destructive stuff. After all, after writing a change to a device, it's important to know how to verify that the change was successful. Developing those skills first will prove useful later on.

Learn how to test for, detect, and safely handle errors that arise along the way, particularly the responses from the devices you are trying to control. Sanitize your inputs! If your script expects an IPv4 address as an input, validate that what you were given is actually a valid IPv4 address. Add your own business rules to that validation if required (e.g. a script might only work with 10.x.x.x addresses, and all other IPs require human input). The phrase Garbage in, garbage out, is all too true when humans provide the garbage.

Scale Out Carefully

To paraphrase a common saying, automation allows you to make mistakes on hundreds of devices much faster that you could possibly do it by hand. Start small with a proof of concept, and demonstrate that the code is solid. Once there's confidence that the code is reliable, it's more likely to be accepted for use on a wider scale. That leads neatly into the last point:

Good Luck Convincing People

It seems to me that everybody loves scripting and automation right up to the point where it needs to be allowed to run autonomously. Think of it like the Google autonomous car: for sure, the engineering team was pretty confident that the code was fairly solid, but they wouldn't let that car out on the highway without human supervision. And so it is with automation; when the results of some kind of process automation can be reviewed by a human before deployment, that appears to be an acceptable risk from a management team's perspective. Now suggest that the human intervention is no longer required, and that the software can be trusted, and see what response you get.

A coder I respect quite a bit used to talk about blast radius, or what's the impact of a change beyond the box on which the change is taking place? Or what's the potential impact of this change as a whole? We do this all the time when evaluating change risk categories (is it low, medium, or high?) by considering what happens if a change goes wrong. Scripts are no different. A change that adds an SNMP string to every device in the network, for example, is probably fairly harmless. A change that creates a new SSH access-list, on the other hand, could end up locking everybody out of every device if it is implemented incorrectly. What impact would that have on device management and operations?

However...

I really recommend giving programming a shot. It isn't necessary to be a hotshot coder to have success (trust me, I am not a hotshot coder), but having an understanding of coding will, I believe, will positively impact other areas of your work. Sometimes a programming mindset can reveal ways to approach problems that didn't show themselves before. And while you're learning to code, if you don't already know how to work in a UNIX (Linux, BSD, MacOS, etc.) shell, that would be a great stretch goal to add to your list!

I hope that this mini-series of posts has been useful. If you do decide to start coding, I would love to hear back from you on how you got on, what challenges you faced and, ultimately, if you were able to code something (no matter how small) that helped you with your job!

31 Comments

It's not inappropriate to approach this with a skeptical view.  But if tests and implementations show promise, moving forward can be beneficial.

Just keep watch on re-inventing what has already been built (and what is working satisfactorily), and watch the time required to create something new that works properly.  Compare it to the benefits received (or perceived!) to determine the value of the project and your team's time.

Don't waste time on something that isn't going to bring results worth the cost of development.  DO spend time where there's a huge need.  And don't be afraid to require what it takes to reimburse you for the work.

Where would we be if Bill Gates and his peers had been able to look into the future and see the harm and expense caused by their computing products?  Would those innovators and business people have given up?  Or would they have plowed ahead, profit in mind, and tried to build something safer and more secure, with a view towards preventing hacks and vulnerabilities?

MVP
MVP

Good article

Level 21

Wow, this article totally describes a guy I used to work with.  Every Month or so he would come to me and have me look at this new "thing" that he had created.  He was always very proud of these creations that he felt were really going to somehow be helpful; however, in reality they almost never got used because they were not needed.  He was the best creator of solutions for which there were no problems that I ever met.

MVP
MVP

Well written, I would add that the training that you get from the building itself has a benefit that needs to be considered in the value proposition.

MVP
MVP

Whenever automation is discussed, I always think of these two XKCD comics:

xkcd: Automation

xkcd: Is It Worth the Time?

I'll admit that I've sometimes been guilty for creating solutions to non-existent problems!

Level 15

A lot of valid points and items that my first year comp sci professors used to convey daily to us.  Some times I find myself spending days looking for examples where someone wrote a library or a function that I needed rather than just sitting down and coding out the details all in the ever quest to automate a task.

Keep up the great and thought-provoking articles.

Level 14

Nicely done.... one of my mantras is "take the best... leave the rest".... I've spent time cleaning up after "re-inventors" .... not my favorite thing to do.

byrona​ I knew one of those guys' too!!! (A forest and trees situation if ever there was one!)

MVP
MVP

Nicely written, totally agree with what you've said.

Really, really good article, XKCD popped in to my head too.

Level 12

jgherbert​ nice article. hmmn.

Level 13

Good points and reminds me of someone.

MVP
MVP

The 80/20 rule is a good thing...

Find what you do all the time that is repetitive and see if you can code a solution to make the work easier..low hanging fruit.

Once that is done, and well documented (within the code and in the groups documentation set in a 3 AM friendly format) then you go to the next one.

Some thing cannot be fixed this way, but you may be able to use smaller bits of code to help..

Level 12

OK, I'll admit to coming out on the wrong side of 80/20 many times in my 40 year career. A lot of times, I was simply fighting the language or tool, and had to abandon an approach. I would add another factor to the 80/20 triage - pain. How difficult is it to do the task? Is it 1 click and wait for a result while I do something else (ok, i'm a multitask-aholic), or is it 1 login, drill 3 levels, scroll, scroll or ctrl-F, enter parameters, click go, copy, paste into new window, click go again, and so on. Count the clicks and keystrokes. 1 point for a click or a keystroke, 2 for scrolling, 5 for copy to new screen, 5 for a login.

Over those decades, I learned a few things I'd like to share:

  • Not everything belongs in a database. A flat file and knowledge of command line tools and built in commands is often much more flexible and portable.
  • Logging things to dated files often solves operational problems because it provides history of when things changed. Don't worry about data redundancy. Disk space is cheap, Text compresses well, and many servers and PCs are idle most of the time. All ove my logs going back 5 years or more is about 70GB, but I only search the most recent single dir or a subtree.
  • CSVs are so much better than XLS, XLSX, and PDFs - they're searchable and processable from command lines, or can be loaded to Excel for sorting and further manipulation. A corollary to this is "avoid using commas in description fields". It messes up Text to Columns in Excel.
  • When it's worth it, parse configs or logged output into CSVs for settings that you need to search or compare on more than one device, This is especially useful for auditing dual core configs on big iron, like Nexus.

For example, I have a couple of batch jobs that dump Windows DNS and DHCP daily. This has been really useful so many times - especially when records get deleted, devices change IPs and break ACLs, or people want to know when a device first or last showed up.

DNS doesn't always show the true history due to problems on the PC and record aging. So I also dump ARP and MAC tables to dated directories,. Products like IPAM and UDT cannot keep that much history, and have many quirks and limitations involving specific models of switches, security in AD, etc. Just remember that dumps are snapshits in time. A device can be moved to a new jack a dozen times in an hour.

A few versions back in NCM, we lost a major ability to log individual commands to separate files for all of a group of devices in a periodic scheduled job. So I've been using PLINK.EXE (part of the PuTTY suite) instead. I wish SolarWinds would put that functionality back into NCM. Maybe there's still a way to do it with NCM, but I haven't found it. Let me know, please.

I also wish that SolarWinds would provide export to CSV for all tabular windows. I find that PDF is useful only for showing something to management to convince them of something. CSV makes the window one step towards a tool, and allows more analysis. Please start with Syslog.

I know, I can use the API and PowerShell to do much of this stuff. I've done some, and am slowly converting things. The learning curve for both the API and PowerShell are steep. I'm an old dog, but I'm constantly learning. I've forgotten more languages than most people know. Plus, more and more, I want data that Orion gets via SNMP that's often impossible to do via SSH. But for now, I spend most of my time in the Windows cmd shell using PLINK on lists of devices and then findstr on the results, or findstr/s to on a subtree.

=seymour=

Level 13

As always, rschroeder​, you make good points.

I was thinking about your comments about Bill Gates, and I wondered if perhaps the answer - in part at least - is that if they had the experience we have now, I'd like to think that they'd put security higher on the list. Then I look at almost every product out there and realize that security is still almost always an afterthought. Code starts off as a proof of concept, and for those we make time-saving compromises like using http rather than https ("I'll get the certificate once I know it works") or using a plain text file for password storage ("I'll put in the crypto stuff later on") and so forth. Before you know it, the proof of concept becomes the first release of the product and all those bad habits are out there. This is also a consequence of the Silicon Valley / agile mantra of "Release early, release often" which is brilliant for getting the product shaped in the right direction earlier rather than later, and absolutely terrible for the kind of shortcuts that get taken early on in the product development lifecycle...

Level 13

That's kind of weird isn't it? I hope at least he got something out of the tools he wrote!

Level 13

Definitely. Even if you end up abandoning the tool for some reason, the work you put into it will inevitably benefit you when you start writing the next tool. I suspect that scripters/coders, like everybody else, probably look back on things they wrote even a year ago and think "Ugh... what was I ​thinking​?!"

Level 13

xkcd i ching.png

Level 13

There's a parallel to that as well. There are number of sites which provide great answers to programming questions, and while it's really handy to grab the code and shove it into a script in order to get the job done quickly, it doesn't help us learn much. I liken this to somebody who is provided with router configurations and simply cuts and pastes them into the device without ever looking to see what they are**. Sure, call yourself a network engineer if you like, but you'll never progress if you aren't curious about what's in that configuration and why it's there.

** True story: I interviewed somebody one time who had been performing CE router configurations for three years for a service provider. In all that time, they had never really looked at what the configuration did, and couldn't answer even basic questions about how the service provider configured the CE routers 😞

Level 13

"Low hanging fruit" - a phrase I wish I had thought to include in the post! That's exactly it.

Level 13

I have a fondness for text-based data files probably stemming from years of Perl scripting, and so that's my default go-to. The time spent figuring out a proper (i.e. Codd's rules) relational database schema which I can then start implementing is usually time I could have spent coding an actual solution.

Structurally, to add to your comments above, I have learned that despite the apparent complexity, it's useful even for relatively simple programs to consider very early on how to feed a request to a script in a structured format (e.g. JSON) and how to provide the output in the same format, so that one tool can very easily consume the output form another in a reliable format and, even better, the tools can be easily wrapped behind a web service effectively providing a REST API for the tools. I have written a few scripts which consume multiple other of my tools via REST API (as well as querying other commercial REST APIs) and the best part of that is once I have written the function to query a REST API, I can just keep on re-using it with the different information sources.

Spot on!  

I've tried to use that kind of example to PREVENT Proof-Of-Concept deployments for something that's not secure, not compatible, that goes against our best network practices.  Unfortunately, "wiser heads" adopt the hardware or applications, and the rest of the teams have to try to plug the gaps, document the one-offs, and compensate for vendor products that aren't ready for prime time.

One day . . .

I recovered "some" of the NCM command line logging functionality for those of my devices that support TACACS.  That full AAA solution really shows show did what, and when--and keep unauthorized folks/commands from taking place.

Now, if only EVERYTHING fully supported TACACS . . .

Level 20

It's nice to have all your devices able to use XTACACS... I love my ACS server.

My Cisco ACS server was not reliable, and I was relieved when I was able to migrate all TACACS to Cisco ISE, which has not had any problems in the last two years.  I'm happy to hear your ACS experience is better than mine.  Our ACS was hanging, preventing remote access into switches & routers, preventing remote management of the ACS primary & standby/failover units . . .

Level 20

Maybe it was just newer... I got this one last year and found it to be so big I had to order a larger enclosed rack for it.  It's 1U but it's like 30" long o.O!  The old ACS servers we had were like 22" deep.

Level 12

Good suggestions

In honor of newbies writing code, and experienced coders being able to write better code, I offer this paraphrased version of Billy Joel's song "Get it Right the First Time:"

I don't believe in frequent rewrites

So just this once I hope my code is gonna work

Well-written lines are always delights

I've got to make that syntax flow

Gonna get that afterglow!

I've gotta get it right the first time, that's the main thing

I can't afford to write a kluge

You get it right the next time that's not the same thing

I'm a coder not a stooge!

I'm not much good at Perl or Python

I never was too smooth at scripting code real smooth

If all it takes is inspiration

Then I might have just what it takes

If I don't make no bad mistakes and I

I've gotta get it right the first time, that's the main thing

I can't afford to write a kluge

You get it right the next time that's not the same thing

I'm a coder not a stooge!

I might find the language

Yeah I'll learn it from Youtube

But if my coding ain't just right I'll look just like a Rube.

I don't know, I don't know, I don't know how

To fix it from my cube

If I want to make a good impresh

My coding has to be so fresh!

So I suppose it's now or never

Before the customer walks out of my life

Just let me pull myself together

I've got to give it one good try

Gotta take my chance tonight

I've gotta get it right the first time, that's the main thing

I can't afford to write a kluge

You get it right the next time that's not the same thing

I'm a coder not a stooge!

The 80-20 Rule, aka the Pareto Principle.

And this computer disease that you reference... I submit "smart" phones as exhibit A. They (over)solved a problem that never existed and now look what we got.

MVP
MVP

Just look at the texting thing they (cell phones) started...now it seems many of the millennials can't listen or carry on a conversation as a result. They even have to have the subtitles rolling on everything they watch on TV or Netflix....ugh.

Level 20

I watch the subtitles too during movies... but never news  because it's delayed too much.

Level 15

good article.