The Luck of the Tech Pro

St. Patrick’s Day is right around the corner, so dust off your “Kiss Me, I’m Irish” shirt and celebrate! Before heading out to show off your Irish stepdancing skills, tell us about your luckiest moments as a tech pro—because leprechauns aren’t the only ones who can find a pot of gold on the other side of the rainbow.

We want to know when you’ve felt luckiest in your career—the situations that made you raise a glass to yourself, or to the “luck of the tech pro!” When did you feel like things could have gone terribly wrong but didn’t? Did something break? Get lost or deleted? You know that feeling—all hope seemed lost, but like magic, everything worked out. (At least we hope you know that feeling).

Share your luckiest stories about finding the tech pro pot o’ gold by March 12, 2019, and we’ll give you 250 THWACK points in return!

  • Many years ago I was showing my son how to make switch changes - IN Production - When he was finished I said "Now do a save start run" As soon as he hit enter I knew my mistake and we were in trouble. I connected directly to the device and was able to do a Save run start and the config ended up being what we intended. I still don't know how that worked, but it did and production was not disrupted - All I can attribute to is pure luck.

  • Reading the stories here reminded me of other occasions that worked out well, but were too random to predict--situations for which only luck / coincidence could account.

    • Your Richard Branson meme struck a chord with me--I had an amazing opportunity, could NOT fill the need immediately, but said I could.    That instance of being lucky happened while traveling with a college jazz band during a bus concert tour and hearing two members say their 4-piece dance band had lost its bass player just before the band received an offer to play the house band job at a beautiful Minnesota resort.  Their responsibilities:  play four hours a night, six nights a week, at a place with a quarter mile of sand beach and towering pines, eat the same food as the guests, work with thirty college-aged women who were waitresses, housekeepers, and bar tenders, get free use of all the resort's facilities (including unlimited access to the 45-hole golf course next door, water skiing, fishing boats, sail boats, etc.), plus get paid in cash every two weeks.  Without a bass player they were in trouble, and my envy of their wonderful opportunity prompted me to comment "That sounds like such a great time. If I had a chance at that job I'd LEARN how to play bass!"  The seed was planted, and few days later the band interviewed me and asked if I'd been serious about learning to play bass.  They liked my piano skills, and knew the kind of sax player I was from our performances together in that college jazz band, and they hoped I was telling the truth--that I could learn to play electric bass in the two short months before their summer gig began.  I said "Yes, I was serious!" and they offered me the opportunity.  I had a friend build a custom bass for me and I started playing along with Greatest Hits albums from the Eagles, America, Jim Croce, etc., and never looked back.  That was forty years ago and I'm still close friends with everyone in that band (who played together for over twenty years), and also great friends with all the fun staff who worked at that resort.  It was one of the lucky moments ever, and I'd go back to that idyllic life playing music with good performers and good friends in that vacation settings.  Today I play in similar bands a few times a month, and still enjoy the setup & take down, the new music we get every time, the solos and the singing, the lights & the sound stage.
    • Trying to "prove a negative" last week.  My team had installed a temporary 48-port trunked switch to a remove Tech office so a desktop support team could start imaging computers to use Windows 10.  The solution had worked well for six weeks, and then failed last Tuesday afternoon.  They called me to complain.  Looking at the switch's configuration, "it should be working".  Looking at the upstream port to which this switch attached, "it should be working".  Looking at logs revealed many disconnects between the switches, and a replaced patch cable eliminated that issue but did not fix the failed imaging problem.  Further log reviews showed someone had plugged two ports from this switch into two different data jacks via a 4-port mini-switch--and this created a Spanning-Tree Loop which our port configuration correctly isolated and prevented a site outage.  Removing that second connection didn't improve the situation.  We often experience "challenges" with devices not being profiled by Cisco ISE which leaves the devices pingable but unusable.  Reviewing the ISE logs didn't reveal anything of the sort.  36-hours later I walked into the Desktop Support "advanced" team's office and related what I saw and the date of the failure.  The expert there said "Well, about that time we instituted a new certificate policy, but no one else is having problems doing Windows 10 imaging."  I put him in touch with the other person who was having problems imaging the boxes and shortly a new procedure was shared that accounted for the new certificates, and imaging was suddenly 100% successful without any work from my team's side to the network switches.  "Proving a negative" (It's not the network) was slightly impeded by my team thinking that temporary training switches and imaging switches don't need remote management IP addresses, don't need to be in NPM, don't need to have their configs backed up in NCM--"Hey, it's temporary; why go through all the extra work?"  (To save us support hours by enabling all the troubleshooting capabilities of NCM and NPM and NTA, that's why.)  It was lucky I stopped in on the Desktop Support Admin and shared the problem with him--he effected the solution.  Which was appropriate, I suppose, since his team inadvertently created the problem.  It would have been luckier if I'd just gone to him immediately after using NCM to verify nothing had changed on those ports, and it would have been luckier if my teammates had added that environment to NPM and NCM.  But I'll take what I can get.
  • Had a similar ground incident many years ago.

    No names mentioned, but  - - -  big mainframe client with two very large systems. I was working as an engineer for one of the mainframe manufacturers and one of the systems was ours. They'd had intermittent hangs for months and the system restart took over ten hours. The client was right on the verge of throwing us out and replacing our system with another from the other manufacturer. It would have cost us a fortune and loss of face. Management decided that we would replace our system. Completely. I've never heard of that level of swap-out before or since. There were strings of tape drives, strings of disk drives, umpteen controller cabinets, console desks and the central complex. it was many truck loads of kit. A team of us worked non-stop to strip out the old system and replace it with the new & the client was finally very happy.

    But - as I was breaking down one of the system's console desks I discovered that its main earth connection was loose and corroded. EEEEEK. I told my boss, he went as white as a sheet & told me not to tell anyone. He had greater respect for my attention to detail after that and some years later, and after a few transfers, I found that he was one of the folk that I was managing. Sweet.

  • My luckiest moment was seating for and passing the CISSP and CISM exams the first time.  Months of preparation and it paid off.  You never know how well you will do when you take these test, but I did it.  Afterward, someone told me 85% of people taking it the first time fail.  Glad I didn't know that before taking the test.

Thwack - Symbolize TM, R, and C