cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Redefining the “Backhoe” Event

Level 13

Until today, I hadn’t heard it called such, but I’ve worked in places that experienced a “backhoe” event. For anyone like me, who doesn’t know what this is, allow me to explain: A “backhoe” event is when someone—like a gardener, for instance—is doing some work around your office, removing plants, planting new things, etc., and digs up your cabling. Of course, most of the time it’s unintentional but causes an outage for your office until it’s resolved. This could be anything from an actual backhoe cutting buried Ethernet cabling to a truck hitting the pole with the transformer powering the building. Generally, businesses will have some backup plan prepared for such an event, or should, and a response prepared for customers who encounter the inconvenience of interruption in service.

I’ve been working from home for several years now, and four or five years ago I experienced a “backhoe” event at home. The interruption of work was only to one person—me—instead of an entire building’s worth of employees, as I officed at home. My ISP at the time had sent a technician to cut the hard line to my neighbor’s house (I don’t know why) and instead, had cut the hard line to my house by mistake. As you can imagine, I first went through the troubleshooting steps of lost connectivity to ensure it wasn’t a problem on my end. The next step was to call my ISP. Now at first, they had no idea what had happened, so they scheduled a technician to come out to my house. This was going to take a couple of days. Meanwhile, I had to figure out how to do my job—I took a day off to figure it out. At first, I took my laptop to a public place to use the free Wi-Fi, but it was full of distractions and insecure, so it wouldn’t work for long. Luckily, my cousin lived nearby and agreed I could office out of his house until the situation was resolved. It took two weeks, several technician visits, and lots of phone calls to my ISP to resolve the situation.

Fast forward to today. Today, my 10-year-old was the cause of a “backhoe” event for my neighbors. Oops. He was mowing the lawn in the front of our house and didn’t notice the cable laying in the grass. The mower chewed through it, and he stopped mowing and came to tell me right away—quite upset. After I calmed him down—accidents happen—we went outside to assess the problem. Sure enough, about six inches of cable had been chewed through by the mower. OK. First, we clean up the pieces and move the wire over into my neighbor’s yard. I can only assume when they recently mowed their own lawn, they had moved the wire over to my lawn to be out of the way and forgot to move it back. I rang their doorbell and we waited for a few minutes, but they weren’t at home. I wrote them a note explaining what happened and offering to pay any fees associated with reconnection.

The event today made me think about my previous at-home event, and Leon reminded me this happened all the time at offices in the “before” times. I realized my previous plan of working from a friend or family member’s home in the event of an outage would no longer be a viable solution—at least not for the near future. So, what now? These things can happen to anyone at any time, and now we have billions of people in this WFH state. Now, as mentioned before, this takes the affected employee and affected portion of business down to a factor of one (maybe two, if you have the rare married couple or roommates working in your office). This makes it unlikely to impact customers nearly as much as an office outage would have in the past. However, we should have some plans to mitigate the problems, if possible. For power outages, I keep a UPS at home to hopefully tide me over for a short time. What would happen now if I lost internet though? To continue to mitigate risk of infection and spread of COVID-19, I won’t be able to pick up and go to my cousin’s house this time. So, this is a good time to pick up some skills/tools, if you don’t have them. Learning how to do some basic cabling and having the tools handy could correct the issue we recently gave my neighbors until the technicians can come replace the cable. Can I solve every problem myself? No. There could be events I cannot even temporarily resolve, and those times will have to include me taking a day or two off until it’s resolved.

Please chime in with your plans to mitigate “backhoe” events for your home office.

12 Comments
Level 11

Mobile hotspots on phones can provide emergency Internet backup, though they are not cheap if used for any significant period. When I moved in to my current home seven years ago, I used the hotspot from my corporate phone (since Internet hadn't been moved from my old home yet) to work while things were getting done at the new place. A couple of months later, I received a nastygram that I had used too much data on my corporate phone that month. After explaining why, I did not have further issues.

This actually reminds me a bit of my time 20 years back, when I used to deal with T1 customers that used DSL or ISDN (!?) for a backup.

I actually still have a genuine, single-copper-pair landline phone at home in addition to my mobile. Redundancy.

MVP
MVP

Being out in a rural area my options are limited as I have fixed point to point wireless for internet access.

Worse comes to worse I can use the wireless hotspot on the phone, but even with unlimited data I have a limit on hotspot data.  Then there is the signal and bandwidth issue.  Some days I get full signal strength on the phone but less than stellar data throughputs.  Normally I have 2 bars.  But the phone shows signal strength for voice and not data since that is on a different carrier (slightly different band) so keep from overloading the voice "channels".

 

If there is an outage power wise in the local area (3 mile radius) my ISPs tower may not have sufficient backup power.  

MVP
MVP

I don't want to admit how many days I worked from my phone hotspot in my consulting days.  Only a couple times was it actually due to internet failure, usually it was because I'd often mix vacations in with my remote work.  Sitting in a hotel taking advantage of the late checkout policy to do my work (hyatts let high status guests keep the room until 4pm most places 😉 ) while my wife was enjoying the beaches or mountains.   Sometimes I'd work from my laptop in the passenger seat while she drove and if I needed to take a meeting we'd pull into a town.

Backhoe fade is a fact of WAN life, and it hurts when it reaches out & touches hundreds or thousands of employees at stub sites in rural businesses.  But when it hits at home, for a home-office employee, options are great to know about.

  • If you're working for a bigger company, and are telecommuting to manage exposure to pandemic, or simply to reduce costs (both yours AND your employer's), you may be able to go back into the office temporarily.  I've worked where there are several open cubes designated "hotel space" for transient employees.  Or you might find someone is gone on vacation or for medical leave, and receive permission to temporarily use their work area until your home connectivity is restored.
  • I recently started a project to install a second driveway on my property, and I knew there were buried ISP/Phone cables in the way.  I called the toll free number to have all the buried cables marked well before I had a contractor start the excavation and bring in 150 cubic yards of pit run and Class 5 gravel.  I made sure to be present when the buried cables were being marked, and I pestered the phone company's sub-contractor to identify the different types of cables, and their locations.  When everything was marked, I got the number for my ISP/phone provider and scheduled them to come move the cables to a new location that would accommodate the driveway.  When the guy arrived with a shovel and gloves, I knew it wasn't going to be done the way I expected.  He dug up the area around the post hosting the connection, showed me the cables were 12" - 15" deep, and advised me to just have the contractor come in and do the work.  If the cables were cut, I now had the cell phone of the phone company's man responsible for splicing & moving cables.  He also automatically offered his supervisor's cell phone number, to ensure I'd have same-day repairs if the lines were cut by the dozer & the heavy dump trucks that would be passing directly on top of them.  He also gave me instructions for what to tell the County's contractors to do when they come later this summer to wide the driveway and put in a wider culvert.  In addition, he said if any of the workers cut the buried wires, he had a 50' cable cut and prepped in his truck, ready to install quickly--and he put my name and street address on it so it wouldn't be scavenged by anyone else borrowing his truck.  To my surprise, the bulldozer did NOT cut the cable.  Nor was it damaged by the many big dump trucks that came through on top of it.  But I'm keeping those two phone numbers for emergency repairs anytime they may be needed.  Most folks won't have these resources handy in case of short-notice outages.
  • Investigate alternate services now, if you're relying on buried phone lines for home Internet.
    • There's buried fiber across the street from my house, but the provider won't run connections from it into my home, nor to any homes along my highway, due to the low population density.  It isn't worth the provider's expense to add one or two homes into this fiber, which runs ten miles up the road to where the housing density is much greater.  Hmph.
    • I found a provider who'd trench fiber into my house and give me synchronous gig service--for $350/month.  Well, I'm paying $50/month now for DSL that's 10 Mb down and 765Kbps up.  Jumping to $350/month isn't in my budget until something significantly different happens in my life--but it's nice to know the option's there if needed.
    • A wireless ISP advertises coverage of my area.  I'll ask for a 30-day test service and see if their bits actually flow here.  I don't care for WISP's; I've had customers try them and be quite disappointed when the wind/weather is bad.  Snow & frost & ice & heavy fog & rain do a good job stopping 802.11 services temporarily.
    • Satellite service is available here, as well.  But the latency has been a killer in the past, and many of my coworkers who moved home for COVID-19 assumed they could just start using it and be exactly as happy as they were in the office.  Nope.  250 miles up to the satellite and 250 miles back down isn't going to get them sub-millisecond latency. Nor will it be 10 ms, or even 100 ms.  Worse, it's half duplex.  Ugh.  
    • Finally, if things were really bad and required it, there are companies that offer leased cube space on a daily/weekly/monthly basis.  Not my first choice, but if it means being able to work, it's worth checking out.  The expenses can be deducted in your taxes if your employer won't cover them, but itemized deductions have to reach a specific threshold before you get much benefit from the work of claiming them.

May your packets be swift and always ready and reliable!

 

Rick Schroeder

You have to watch out for these guys, I can remember more then one snacking on our lines. We now paid for 2 connections, with different egress paths whenever possible. 

oXJQHJz

 

Level 14

ahh yes..... Murphy pays a visit!..... I am impressed that your 10 year old mows the lawn. Nice work dad!

The closest I have come to a home version of the backhoe incident was when one of my kids drop some chips on the family room floor and proceeded to unplug the internet router to vacuum...... ( no good deed goes unpunished,,,) I am in the front of the house and online fixing problem when I went dark... I smile, finish vacuuming and thanked my daughter for cleaning up so quickly. Plugged said router back in and went back to work. 

Level 16

I worked at a company and the parking lot was being resurfaced. They had a dozer scrape off the top 4 inches of dirt and then prepped and laid down new blacktop. That's when we found a set of conduits that the main Data center power and original communications cabling were only buried about 3 inches deep. Data Center luckily flipped over to generator but the rest of the building was without power. Everyone was sent home or over to another office. The power company retrenched a new cable down three feet deep and restored power but the legacy 600 pair buried copper cable, about as thick as a mans arm was reburied again, 3 inches deep. The contractor was told the communications cable were abandoned by someone and they started to resume work. The 600 pair cable had been abandoned but was still terminated in the Data Center telco room several hundred feet within the building. I let our management know that it wasn't ok to start work until the cable and conduit had been cut outside the building otherwise if the dozer hit it, it would pull down everything in its path inside the building, a 600 pair cable won't break. Now six years later if I drive by my old employer I see there is still a few feet of parking lot that was never finished. Seems the contractor refused to dig there any more. 

MVP
MVP

Always have a backup to your backup!  I have complied with all recommendations in your article.  Thanks for the share .. makes you almost human!!! 

Some photos of damage to buried copper and fiber you may have missed:

 

rschroeder_0-1592255820927.png

rschroeder_1-1592255851768.png

 

 

Rodents chewed these communication cables up:

rschroeder_2-1592255950957.png

 

 

I'll let you use your imagination--was this woodpecker damage to a suspended overhead telephone cable, or was this damaged by a dog or other mammal because it was easy to reach from the ground?

rschroeder_3-1592256134560.png

 

I've had a site down for three days as a WAN provider worked to find the cause of the outage--it turned out to be a Pileated Woodpecker had put holes in the junction box high on a utility pole, far from a highway, in the middle of a swamp, and pulled out all the pretty colored plastic "worms" it found.

This was a new one to me:  Grizzly Bear claws damaged fiber in British Columbia at the site of a land slide that had damaged a highway.

 

https://www.researchgate.net/publication/324227112_Fibre_Bragg_grating_and_Brillouin_optical_time_do...

 

 
 

grizzlyFiberDamage.JPGgrizzlyFiberDamage2.JPG

 

Level 13

Thanks for sharing your stories! These are cracking me up - in solidarity, I assure you. What else can we do but laugh and fix it, am I right?

MVP
MVP

Redundancy, Redundancy, redundancy!  I totally forgot about the most recent event that we had about 2 months ago!  Since we have redundant service providers with redundant paths, along with a generator, my operation did not suffer... thank goodness ... its 911!

zennifer_0-1592324911185.png

 

hank goodness ... its 911!!!  A semi truck took a turn too tight....