August 4, 2022
As with any city, the region-wide 911 Computer-Aided Dispatch (CAD) application within El Paso, Texas, is a critical component of its technology infrastructure. It helps forge the connection between the city and fourty other county-wide offices during an emergency to ensure citizens receive the quickest response from the most appropriate service.
The system takes in phone calls, GPS coordinates, and citizen information—it dispatches vehicles and runs the radio dispatch system, traffic alerts, fire alerts, and so much more.
However, the district’s systems configurations limited their ability to respond as quickly as possible. “We needed to help cut down on the time it took to get emergency service to our citizens,” said Jennifer Aguilar, senior network administrator with the El Paso County 911 District.
The desire to improve response time for its 911 CAD application was only one of the district’s challenges. Others included the following:
- Network performance and capacity issues from poor visibility
- Not having the tools to troubleshoot network problems or outages
- Unclear picture of system and server health
- Lack of visibility into application or database performance
- Assessing and remediating vulnerabilities took too much time
- Networks and data center needed critical modernizations
- No central location for end-user and systems support
El Paso’s 911 IT team runs a 24/7 NOC that houses critical infrastructure. So how did the team know things needed to change?
First, its connections were slow, and applications—including the computeraided-dispatch system—lagged. The district was using two encrypted tunnels that had reached 97% utilization. The pipe was being overrun with traffic.
When Aguilar first arrived at the district, she inherited an environment that was not segmenting the 911 network from its other networks; the network running the 911 CAD application was part of its administrative network. In addition, the district had test applications running on production servers. Aguilar noted with so many applications running on one network, it was an extreme security risk.
Finally, the district’s software and patch updates were “all over the place,” according to Aguilar, which caused additional security challenges.
CONFIGURATION, SEGMENTATION, AND MORE
The district started with the goal of enhancing performance—specifically, finding out why the encrypted tunnels were being overwhelmed and why applications were so slow.
The El Paso team brought in a range of SolarWinds® applications and made many immediate, critical discoveries.
The first discovery was a configuration error not only impacting applications but having a similar impact elsewhere because the error had been replicated several times throughout the system. “When we found that configuration issue, we went from 95% utilization on that line to 35% utilization,” Aguilar said. This made a dramatic difference immediately. The best part, she added, was the ability to enhance performance to such a great degree with a simple configuration change without having to provide additional bandwidth.
Once the configuration errors were fixed, the next priority was segmentation. “Our 911 CAD application is so important; it simply must be separate,” Aguilar said, “yet we found that our 911 CAD application was sharing our administrative network.” As stated earlier, she also had security concerns. “We’re all under attack,” she said. “With everything on one network, anyone could have taken out all of our services.”
The district moved quickly to “flatten the organization” and segment the application onto its own network, which increased performance and provided a far stronger security posture for all the district’s 911 CAD applications.
Next, the El Paso 911 IT team ensured all patches and other software instances were updated—another important stanchion in the district’s 911 security framework.
The final piece was to implement a single-pane-of-glass perspective to allow the team to see everything happening within all networks from one screen.
"Being able to manage everything through a single pane of glass was huge. I can literally get any information I need with a quick click." - Jennifer Aguilar, Senior Network Administrator, El Paso County 911 District
“LEAPS AND BOUNDS” IMPROVEMENTS
With SolarWinds offerings—specifically Network Performance Monitor (NPM), Server & Application Monitor (SAM), Virtualization Manager (VMAN), Database Performance Analyzer (DPA), and more—El Paso 911 stabilized the environment, dramatically improved application performance, and enhanced database performance in particular by “leaps and bounds,” according to Aguilar.
The 911 district implemented SolarWinds solutions to monitor as many as 1,000 objects and currently uses SolarWinds tools for the following:
- Network management and configuration
- Systems, applications, servers, and storage monitoring
- Remote support and IT help desk
- Database management and performance
- IP address tracking and management
Looking specifically at database performance, before implementing SolarWinds DPA, there was an 800-second wait time for a database response—more than 13 minutes. With DPA, the wait time is now down to 180 seconds—a whopping 10 minutes less than the previous norm. The time saved with each query was significant.
Other examples of things the El Paso 911 team was able to accomplish with the addition of SolarWinds products include the following:
- Correlate network, server, and application issues to resolve performance problems
- Improve resolution rates for networking performance and capacity issues
- Improve configuration management
- Improve system and application monitoring and troubleshooting
- Provide better service levels to the organization
- Improve log tracking and management
Today, SolarWinds solutions continue to assist the 911 District with its day-to-day duties—including streamlining and visibility into upgrades—and have enabled the district to expand its team. “I put the monitor up on the big screen and watch the interfaces go up and down when we are performing hardware upgrades, and we watch live performance metrics during application upgrades,” Aguilar said. One of the best parts? El Paso realized a return on investment for its SolarWinds purchases in as little as six months.
