Last week we could see the writing on the wall. Work from home was coming. IT was tasked to determine where we might have problems delivering the same level of service. It was assumed that 3 services would drastically increase utilization.
Internet Connections
First our connections to the internet. They often get busy, but the circuits themselves can burst and scale up. We need to watch it, but there was a secondary concern, the firewall devices behind them are old and not scaled to the current demand. They were scheduled to be replaced next month.
We started with the interfaces that connect to the internet, and the nodes they attach to. They and one layer deeper (next hop I the datacenter) got added to and Orion map, and saved. We wanted to see that we weren’t maxing out the circuit, and we aren’t, but we added PerfStacks (one per connection) to display the peak transmit and receive bandwidth. To wrap up the connections, we added a PerfStack to show errors and discards on the circuit connection. We didn’t expect to see an increase, and thankfully its not there. What we do see is the pain between the edge device and the next layer in we expected. The errors and drops there are a tiny fraction of the traffic, but still way higher than a normal week. I could reset the baseline, but we decided to leave it for now.
Virtual Private Network Connections
Next, VPN sessions. Here our hardware was recently refreshed, and we know that emergency burst licenses can be allocated. We wanted to make sure that those were rock solid, so we are watching performance stats, using the license and concurrent user counts to judge scale.
Our VPN is a cluster that spans two data centers (which is one reason we don’t have direct layer 2 access, just the layer 3 connections in the Orion map. Again, I added those devices, and their relevant interfaces, and one hop back. I get less good stuff since it’s a layer 3 connection, but its nice to know the big picture. (Same devices as the previous dashboard).
Conference/Collaboration Software
The final concern was Skype for Business. We are still very much on premise and Skype is our primary conference/collaboration software. Staff also use Teams, WebEx, and Zoom, and Slack, but we assume that if we can keep data following through the internet connection, then those services will be fine. (Not always true but not something we can change today.) Skype was built and scaled out with heavy (for us, I know other places are much larger) loads in mind, but its never really been pushed hard.
We have had a Skype dashboard for a long time, but it needed a bit of clean up and refresh if a bunch of people wanted to look at it. I changed out a Network atlas map (with a Visio bitmap background) for a much quicker loading Orion map. We also made sure the AppStack widget was configured to include the entire environment. Lastly, we added a tab with some metrics, specifically the number of conferences and users per front end server.
Everything looks great, but we started getting tickets about people having issues calling into meetings. They got a busy signal. We ruled out Skype (hesitantly) because it looked normal despite the increase of connections. We checked the next hop out, our phone system, and that was good too. So, next we checked our phone provider, AT&T. It was also good. But our users were calling in from locations with Verizon. Yup, the entire city is having issue with Verizon, mostly when calls are handing off to other providers. Did the dash fix it? No. But it did make it easier to rule out possibilities.
We have some other IT challenges going on at the same time. We set up WIFI in tents for COVID-19 testing stations in a parking lot. We are building and deploying laptops to supplement the “normal” laptop workforce. We are redirecting fax numbers from multi-function devices to a RightFax servers. Citrix and RDP servers are getting built, VDI as well to exposes more work from home options. The list continues, but it’s just re-prioritizing our efforts. So far, we have held up well to the changes.
