cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Importance of Dashboards for Remote Workers

Last week we could see the writing on the wall. Work from home was coming. IT was tasked to determine where we might have problems delivering the same level of service. It was assumed that 3 services would drastically increase utilization.

Internet Connections

First our connections to the internet. They often get busy, but the circuits themselves can burst and scale up. We need to watch it, but there was a secondary concern, the firewall devices behind them are old and not scaled to the current demand. They were scheduled to be replaced next month.

jm_sysadmin_6-1584478476400.png

 

We started with the interfaces that connect to the internet, and the nodes they attach to. They and one layer deeper (next hop I the datacenter) got added to and Orion map, and saved. We wanted to see that we weren’t maxing out the circuit, and we aren’t, but we added PerfStacks (one per connection) to display the peak transmit and receive bandwidth. To wrap up the connections, we added a PerfStack to show errors and discards on the circuit connection. We didn’t expect to see an increase, and thankfully its not there. What we do see is the pain between the edge device and the next layer in we expected. The errors and drops there are a tiny fraction of the traffic, but still way higher than a normal week. I could reset the baseline, but we decided to leave it for now.

Virtual Private Network Connections

Next, VPN sessions. Here our hardware was recently refreshed, and we know that emergency burst licenses can be allocated. We wanted to make sure that those were rock solid, so we are watching performance stats, using the license and concurrent user counts to judge scale.

jm_sysadmin_7-1584478476405.png

Our VPN is a cluster that spans two data centers (which is one reason we don’t have direct layer 2 access, just the layer 3 connections in the Orion map.  Again, I added those devices, and their relevant interfaces, and one hop back. I get less good stuff since it’s a layer 3 connection, but its nice to know the big picture. (Same devices as the previous dashboard).

Conference/Collaboration Software

The final concern was Skype for Business. We are still very much on premise and Skype is our primary conference/collaboration software. Staff also use Teams, WebEx, and Zoom, and Slack, but we assume that if we can keep data following through the internet connection, then those services will be fine. (Not always true but not something we can change today.) Skype was built and scaled out with heavy (for us, I know other places are much larger) loads in mind, but its never really been pushed hard.

jm_sysadmin_8-1584478476411.png

 

We have had a Skype dashboard for a long time, but it needed a bit of clean up and refresh if a bunch of people wanted to look at it. I changed out a Network atlas map (with a Visio bitmap background) for a much quicker loading Orion map. We also made sure the AppStack widget was configured to include the entire environment. Lastly, we added a tab with some metrics, specifically the number of conferences and users per front end server.

jm_sysadmin_9-1584478476415.png

 

Everything looks great, but we started getting tickets about people having issues calling into meetings. They got a busy signal. We ruled out Skype (hesitantly) because it looked normal despite the increase of connections. We checked the next hop out, our phone system, and that was good too. So, next we checked our phone provider, AT&T. It was also good. But our users were calling in from locations with Verizon. Yup, the entire city is having issue with Verizon, mostly when calls are handing off to other providers. Did the dash fix it? No. But it did make it easier to rule out possibilities.

We have some other IT challenges going on at the same time. We set up WIFI in tents for COVID-19 testing stations in a parking lot. We are building and deploying laptops to supplement the “normal” laptop workforce. We are redirecting fax numbers from multi-function devices to a RightFax servers. Citrix and RDP servers are getting built, VDI as well to exposes more work from home options. The list continues, but it’s just re-prioritizing our efforts. So far, we have held up well to the changes.

Comments

Great post. Demonstrates the importance of identifying how the dynamics of IT resource utilisation has changed and how important the monitoring of this change is so vital.

Anticipating to the needs of the organization, adapting the tools on hand and adding new capability will be needed for everyone for a while.

Anticipate - Adapt - Add New

Paul

This is definitely something I'm going to want to talk to my directors about.  We are taking everything each day and coming up with a business continuity plan that can address this sort of COVID-19 thing if it happens again.  These are crazy times for sure so whatever I can do as a monitoring engineer to help the company stay calm and connected is a win in my book.

@the_ben_keenWe definitely had plans, for a lot of different disasters that mean work from home. A lot of those plans were/are being used, but when things really need to take effectits different. The biggest difference this time is that even though we are taking drastic measures, everyone wants to continue like its the same with no work slow downs. I think when plans were drawn up, its assumed, well if no one can come to work, then we don't need to worry about X, Y and Z, we just need these mission critical things. When the closings started, basically the approach was to continue doing everything unless it was impossible or prevented by government mandate, and do it all remotely.

We are in the same boat as many others on the cruise of life. Increasing SAN resources to provide services to application servers. Adding additional bandwidth to internet pipes. All those things helped our boat experience smooth sailing. Sure there were bumps when too many people were in the "pool". That changed once our crew (server, apps teams, security and network) put together the needed changes to prepare for the new demand. Over all the journey continues and we are island hoping 😉 enjoying the sun and proving IT can.

Version history
Revision #:
8 of 8
Last update:
‎03-18-2020 09:39 AM
Updated by: