"SolarWinds allows us to deliver high-level technical support to integrated care board's, general practitioners, and corporate. It supports us to meet very tight time thresholds."
Martin Russell-Raymond, SCW (NHS), Observability Systems Manager
ABOUT NHS SOUTH, CENTRAL, AND WEST
NHS South, Central and West (NHS SCW) is an NHS organisation providing support and transformation services to health and care systems, helping them achieve the best possible outcomes for people, communities and populations. As partners of choice to more than 200 organisations that support millions of people, our mission is to help deliver the NHS Long Term Plan, improving the health and well-being of the nation.
The Service and Technology Management team is part of Digital, Data, and Technology (DDaT). This department is made up of change management, incident and problem management and the Technical Operations team, of which there are approximately 70 employees responsible for the infrastructure for:
- Nine integrated care boards (ICBs)
- 1,200 general practitioner (GP) primary care sites
- 11,878 sq. mile area covered
- 51,351 endpoint devices
- 2,100+ servers
- On 300+ VM hosts
- 1,900+ switches
- 800+ firewalls
- 1,300 routers
A key initiative for the team in 2022 was enabling 97% of their 563 GP practices and 30,017 GP staff in England to consult patients online.
NHS SCW offers the following services to customers: analytics, clinical support, corporate digital, digital data, and technology, people solutions, procurement and contracts, as well as finance, public health, safety, and transformation consultancy.
Products and services catalogue - NHS SCW Support and Transformation for Health and Care (scwcsu.nhs.uk)
NHS SCW delivers these services based on five core corporate values:
- Aspirational: Our people aspire to exceed expectations, deliver excellence, drive innovation and be the best in our sector for the benefit of our customers and patients
- Collaborative: We work hard to build and maintain close relationships with our customers and partners. We value the importance of working towards a shared goal and focusing on its delivery.
- Insightful: As experts in our fields, we understand the bigger picture and its complexities, we challenge convention, and we uncover opportunities.
- Patients First: Our key inspiration is to make a difference to patients, service users and communities by delivering effective services to improve patient care, treatment , and experience.
- >We treat everyone with respect and hope it will be returned. Recognising and valuing their knowledge, we involve our customers, teams, and partners in decisions.
Our values - NHS SCW Support and Transformation for Health and Care (scwcsu.nhs.uk)
CHALLENGE
The Service and Technology Management team closely aligns with NHS England’s long-term goals and plans. These initiatives include their central mission of meeting client and patient expectations today and continuing to do so in the future with the anticipated increase in patients.
Martin Russell-Raymond, Observability Systems Manager, is a member of the Technical Operations Service Delivery team, specialising in developing and managing the SolarWinds® Platform. Apart from their long-term strategy, Martin and the rest of the Service and Technology Management team are confronting several important challenges facing their customers:
Budget stewardship and financial pressures
- Increased scrutiny on taxpayer funds and NHS budgets, including technology spending
- Need for financial reporting and insights into how technology benefits patients and staff
Evolving GP and patient demands
- Greater digitisation has increased network traffic, with GPs remotely accessing medical data and using Office 365® and delivering remote online consultations/telehealth services to patients.
- NHS policies still require many staff to work from home, increasing the number of remote devices on the network and introducing demand for a monitored VPN solution to protect sensitive data.
SOLUTION
How is the Service and Technology Management team supporting new technology and sustainability initiatives?
- The team monitors customer’s infrastructure to ensure that they have systems and connectivity available during core operational times, maintaining the GPs’ daily operations, patient visits, and processes.
- It is continuing its cloud migration journey, anticipating that over time data will be centralised in the cloud, with greater use of OneDrive® and SharePoint®.
- It strives to implement a “One Domain” approach across Integrated Care Board (ICB) such as Sussex, Somerset, Bristol, North Somerset and South Gloucestershire, Wiltshire and Bath and North-East Somerset. This enables NHS staff in these ICBs to gain access to resources from any location within their ICB footprint in a secure manner.
- The team is working to support NHS England’s goal to become carbon neutral by 2040 and reach an 80% reduction in carbon emissions by 2028 to 2032 and will explore traffic optimisation to reduce environmental impacts.
Sustainability - NHS SCW Support and Transformation for Health and Care (scwcsu.nhs.uk).
Martin’s management of the SolarWinds Platform has enabled NHS SCW to not only support these ongoing initiatives but also continue supporting the region's day-to-day operations.
“I thrive to get the best out of SolarWinds on a day-to-day basis,” Martin says. “We can’t live without our dashboards. I have made the main status dashboard so clear; it shows what devices are down, which are up, which have been set to unmanaged and which devices are unreachable.”
Going forward, Martin plans to use dependencies capability within SolarWinds to further reduce the duplicate alerts raised when a remote site goes down.
Unified data with modern dashboards
The Service and Technology Management team supports clinical systems, to manage where performance impacts may be patient-impacting, and for the most urgent events, the team targets first contact within 30 minutes, and resolution within four hours. SolarWinds assists the team in accomplishing this goal.
It sees the value in the clear and customisable dashboards provided by SolarWinds, “We don't have a busy dashboard. You know, a lot of people said the main dashboard they view is so clear and crisp, and they know exactly what it's saying rather than having dots and numbers all over the place,” Martin says.
A clear dashboard enables each site to identify and resolve the root cause quickly. When a site goes down, dashboards and maps display the relationships of firewalls, switches, and virtual machines, enabling the team to quickly discover the root cause without sifting through multiple alerts.
“We alert through Microsoft Teams rather than through email purely because of email fatigue,” Martin says. “Members of The Service and Technology Management team disliked email notification. They said, ‘We don’t want an email because all we’ll do is set up a rule to delete or file into a folder with 45,000 unread messages”. Having the alerts going to Microsoft Teams® means that employees can subscribe to the device channel that they wish to be notified about.
The initiative has been so successful that the Service and Technology management team are looking to directly integrate the alerts with their service desk ticketing system to automate ticket creation and alert sites to outages before they see the effect.
Meet budget constraints by consolidating redundant tools
Though funding for NHS England has been cut by nearly £2 million in 2023, the Service and Technology team finds ways to leverage SolarWinds to save more money. Earlier this year, Martin saved the organization £250,000 by not renewing a redundant tool, and better utilising their SolarWinds Virtualization Manager (VMAN) licenses.
“I explained to the Enterprise Architecture team, that VMAN can do pretty much everything or the majority of what we want from the current solution so why not use VMAN?’” Martin says.
Consolidating existing tools, by fully utilising the functionality of SolarWinds, is delivering real value for the local health economy. “To bring in a replace solution to SolarWinds could cost the organisation an extensive amount in terms of staff time and the cost of a new solution to get us to the same point we are with the reports, dashboards and general design we have got setup with SolarWinds now,” Martin says.
Integrating new technologies and service sites
The Service and Technology Management team has supported several technology upgrades that have required quick thinking and ongoing support.
- Microsoft® Office 365, OneDrive, and SharePoint access for all GP sites. This has been challenging to monitor since the internet connection speeds and infrastructure varies significantly between sites, resulting in slower upload speeds. SolarWinds NetFlow Traffic Analyzer allows the Service and Technology Management team to quickly see whether links are underutilised or overutilised and recommend increases where needed.
- The User Device Tracking functionality enables the Service and Technology Management team to track where devices are logging on, identify unusual traffic patterns, and quickly act on reported anomalies from NHS England Digital team.
- Over time, regional consolidations and site changes resulted in several spreadsheets containing IP address information. With the capabilities provided by SolarWinds IP Address Manager, Martin can consolidate the spreadsheets into a single source of truth and allow access to download and upload changes when necessary.
- The NHS’s Internet First policy requires that GP services be available in the cloud as soon as possible. The Service and Technology Management team monitors traffic and ensures the cloud workloads are secure and reliable using SolarWinds Technology
- The systems Virtualisation Manager reduces sprawl and assists in capacity planning as the Service and Technology Management team migrate and consolidate data centres while minimising disruption and maintaining their current level of performance.
BENEFITS
The Service and Technology Management team relies heavily on the information their SolarWinds products deliver to enable proactive problem resolution. Not only does this help reduce overhead costs, but it also delivers mission-critical visibility into the systems they maintain.
“The weekly disk space report has been invaluable and has prevented serious outages for at least three practices where disk space was running dangerously low,” a platform technical consultant told Martin. At the beginning of each week, the report lists the top ten culprits for sites with the lowest disk space. This allows the team to address the problem before it becomes critical and causes an outage.
Martin has held workshops and assisted teams with optimising their SolarWinds environments, and this has helped many teams see how they can optimise their work and have greater visibility into their areas of responsibility.
“I approached other teams within DDaT asking, ‘How are you monitoring your systems at present?’’” Martin said. He provided these departments with a custom SolarWinds portal with limited view of the servers they are responsible for. The team immediately took to the dashboard. “They’re constantly asking me to add to their portal so that they can investigate issues in more detail.”
When a problem does occur, the team can isolate the issue and address it while tracking the downtime. This enables them to report problems and respond to customers accurately.
“A device goes down, and when it comes back up, rather than relying on a customer saying, ‘Oh no, it's been down all day,’ we can see it’s been down for 20 minutes,’” Martin says. “We can actually give them an accurate time of when the device goes down.”
SolarWinds monitoring allows SCWs Service and Technology Management team to identify problems even before they happen. Issues caused during routine maintenance no longer become a crisis the next day.
“It’s helped me snap up an issue on the back of the change we did this evening for the MPLS [Multiprotocol Label Switched Network],” an NHS network technical consultant reported to Martin. “For some reason unknown to anyone, it had caused the site-to-site VPN solution to stop working. I wouldn’t have known if it wasn’t for SolarWinds,” says an NHS SCW technical consultant. “That is why SolarWinds is so valuable to us; without it, we’d have been blind to the problem until we got calls in the morning.”
SolarWinds is a critical element of their monitoring on a daily basis and “provides a single pane of glass to the infrastructure the Service and Technology Management team supports.”
Thanks to the time he saves through proactive monitoring, Martin has ambitious plans to implement APIs and additional programming aspects. He and the team expect increased productivity and proactive issue detection and resolution in the future. Though they have been utilising SolarWinds for a decade, in many ways, Martin is just getting started.
SOLARWINDS PLATFORM NETWORK AND INFRASTRUCTURE MONITORING FUNCTIONALITIES IN USE
- Availability and performance monitoring
- Application monitoring, using both built-in and customized templates
- Server virtualization monitoring, management, optimization, and capacity planning
- Network configuration monitoring, management and backup
- Network firmware vulnerabilities monitoring
- Network traffic and bandwidth analysis
- User and device tracking
- Automated IP address tracking with integrated DHCP, DNS, and IP address management
- IP address alerting, troubleshooting, and reporting
- Cloud monitoring
