Leveraging Network Monitoring to Quickly and Effectively Respond to Disasters (like COVID-19)

I provide 24/7 network support to a health care system comprised of a hundred hospitals, clinics, and business offices across northern Minnesota, eastern North Dakota, and northern Wisconsin.

When my hospital administrators saw the early news of COVID-19 in December and January they asked whether our network would support many more people working from home.  Was there enough bandwidth available to handle more remote access, more high-speed transfer of medical images and studies via VPN and across our WAN?  Would our hardware infrastructure handle the additional load?

A good Network Management/Monitoring System was part of the solution that gave me the confidence to provide the answers needed.

It was easy to see we'd been experiencing peak loads of around 250 AnyConnect home VPN accesses on any given day.  Knowing our ASA's were licensed and sized for up to 2,500 connections, we could support more employees working from home.

The 65 CiscoRegistered Meraki devices we’d previously deployed for Radiologists, Cardiologists, Call-Center Operators, and IT Staff to work from home had a proven track record, and we recommended ordering an additional 400 units of various sizes to facilitate high-speed/fat pipe needs for special-purpose employees.  That order went through promptly, thanks to the open minds of these hospital business administrators.  I learned we received the last 250 Meraki devices sold in the U.S. this week.

Ramping up home connectivity with Meraki devices was necessary for people moving to work from home who required advanced connectivity needs.  Some examples where a plain Citrix connection wouldn’t suffice:

  • Complex Receptionist phones with up to two sidecars, enabling them to manage and route and supervise dozens of calls simultaneously
  • Physicians who remotely analyzed very-high-resolution Radiological images 24/7 during on-call shifts in any kind of bad northern Minnesota weather
  • IT staff who had brought home their IP phones along with IP cameras and PC’s to remotely support servers, apps, databases, and the network in general.

Our tools showed our resilient Internet service had recently been upgraded from a single 1 Gb/s service to dual 2 Gb/s feeds from two different providers.  I could see we were using less than 25% of our available bandwidth prior to COVID-19.  We’d easily ramp up to cover the expected surge with more than 4 Gb/s Internet capacity based on the burstable contracts we’d already negotiated with our multiple ISP’s.

In early March we started testing Always-On VPN and it was proving superior and easier to use and operate than anything we’d tried previously. It would come in handy later, when were asked to turn in our Meraki devices for redeployment so Health Care Providers could work from home.

Our Citrix Portal was already comfortably handling 4,000 sessions a day. Recent upgrades more than doubled that in preparation for future needs. This was perfect timing for sending more of our 17,000 employees home to keep them and our patients safe from contagion. Sending them home freed up office space for us to quickly turn to tele-visit work areas for providers to remotely consult with concerned patients or be repurposed for other medical needs. In some cases, cube farms would be thinned out, allowing care providers to maintain safe social distancing while still seeing and talking to patients and peers via TeleHealth systems. E-visit patient care sessions have increased by 400 percent—somewhat miraculously without increasing internet consumption, thanks to a project where my team members coordinated connecting directly to the Minnesota Internet Cooperative Exchange (MICE).

MICE connections enable us to peer directly to Zoom, other ISPs, and cloud service providers, including Microsoft. This was a huge bit of synergy because it reduced our original Office 365 demand on our corporate internet pipes while providing faster O365 access. Today our NMS tools showed our MICE connection exceeded the bandwidth we’d previously been using for general internet services, and that’s a huge savings in our resources and contracted ISP fees.

The right NMS can enable a team to respond quickly to hard questions and to provide the needed information quickly.

It doesn’t take a medical emergency to see the benefits of baselining systems and bandwidth and database performance.  That’s just part of normal business operations, monitoring growth and predicting needs for future expansion.  But it’s certainly nice when a problem occurs and the right tool for the job is already part of the job.

In the last week we’ve seen AnyConnect sessions jump from 250 peak per day to over 1,000.  Internet utilization and Citrix sessions have reflected O365 being used through that new MICE service, while LAN and WAN bandwidth have decreased measurably—a sure sign of more people working from home instead of at the office.

My eight Network Analyst moved to work from home last week, as did many of our Apps Analysts, DBA's and System Admins.  We brought our laptops and desk cameras and multiple monitors and set up home office areas as best we could.  But the Response to COVID-19 requires being fast and flexible to meet rapidly changing needs.  In only a few days we were asked to bring all our monitors and cameras back into work so they could be redistributed to medical staff for Tele-Health visits.

Then we were asked to bring all our Meraki's back into work for redistribution to medical and transcription staff who needed their larger deskphones that have sidecars for many incoming lines.

My team has moved to Always On VPN, as did many of our other 400+ I.T. staff members across three states.   Combining AOVPN with AnyConnect VPN and our Citrix Remote Desktop Portal, our customers are saying great things about our team.  And by sharing our computer monitors and Merakis back to this organization of 17,000 health care providers & support staff, we're helping to fight back against COVID-19.

We're deploying "toy hauler" trailers that have been customized & converted to mobile screening offices attached to drive-through tents in business parking lots, to quickly test people and ensure our health care providers don't bring COVID-19 into our clinics & hospitals.  These trailers are networked with point-to-point wireless solutions that give them 100 Mb/s throughput from the adjacent clinic/hospital/support building's internal LAN, connected back to the main organization via multiple gig or 10G links.

That ability to sacrifice our personal work environment desires immediately saw results.

Over the last weekend we heard from one of our nurse practitioners who started using virtual visits with patients in assisted living facilities. She shared this with her team members:

“I was surprised at the positive response from my dementia patients. I wasn’t sure how they would respond to Tele-Health. They were super intrigued with seeing my face talking with them on the screen. They laughed and loved it. Some of them talked to me more via Tele-Health than they have ever talked to me in person. I was shocked! It was great.”

One of our Pediatrics doctors told this story during our daily COVID-19 live/Teams meeting:

A mother with a sick little girl called in for help, and she was quickly transferred to a Pediatrian.  The doctor could hear the worry in the mother's voice, and more concerning, could hear the labored breathing of her daughter.  The doctor immediately asked that scared mother to connect her smartphone to our Tele-Visit web page.  She did and pointed her phone's camera at her daughter.  What the Pediatrician saw made them very concerned; the doctor told her an ambulance and EMTs were on the way for her daughter immediately.  If she'd waited and then gone to a clinic, then been screened to be seen by an Urgent Care nurse, then waited to be transferred to a hospital . . . well, instead of that, the right people went to the little girl and saved her life.

And we're not only hearing about employee's sacrifices.  We're seeing the public sacrificing for our care workers to be able to provide better care for our community as shown here:

The administrator at our hospital in Virginia, Minnesota, shared a note about a gentleman from the community who gave a triage nurse five full boxes of brand new N95 masks, which is about 100 masks. He said that he used them while sanding his woodworking projects and commented, “You all need them more than I do.” It’s a heartwarming example of the support in our communities for our caregivers.

These are interesting times, and I’m happy that forward-thinking people have been driving these changes in advances of disasters, instead of staying static and letting red tape and budget and events come down the pike and impact us without warning.

Swift Packets to you all!

Rick Schroeder (operating out of a little house in the northern woods, supporting medical network needs remotely)

P.S.:  Trust the doctors and practice social isolation and hand washing, and you'll be part of the solution that helps save lives.

Parents
  • Nice write-up, .

    On a more personal note, thanks for helping medical care in your area. While I don't have much family there now, my mom's uncle still lives in Fargo -- the last of nine siblings who grew up there before spreading out across the midwest (mostly) and beyond who still lives in the area.

Comment
  • Nice write-up, .

    On a more personal note, thanks for helping medical care in your area. While I don't have much family there now, my mom's uncle still lives in Fargo -- the last of nine siblings who grew up there before spreading out across the midwest (mostly) and beyond who still lives in the area.

Children
No Data
Thwack - Symbolize TM, R, and C