This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

My response to Sunburst

It is no secret that the past few weeks have been a little bit more of a struggle than most.  While we are used to things being a little crazy at the end of one year and the start of the next; having to deal with a major security event is one thing I do not think any of us really ever want to deal with.  With that being said; I would like to share how I responded to everything and the steps taken to bring SolarWinds back up and running.

The Sunday started off like any other but by 2:00 AM I knew things were not going to be the same for much longer.  Following the email sent out by SolarWinds; I sent that out to my manager, director, and Chief Information Security Officer.  Shortly after the start of the business day, we made the decision to shut down all of the SolarWinds servers.  We did this as a way to protect ourselves into further information was provided.  At the time of the attack, we were running 2019.4 HF1.  We had over 12 Additional Pollers and one Primary Poller supporting over 80,000 elements on over 14,000 nodes.  The Primary Poller was also acting as the webserver for the SolarWinds GUI.  

After shutting down the entire environment, I offered up my personal home lab instance of SolarWinds to act as a testing platform with any updates coming out as a result.  I figured it was safer to put my home lab at risk rather than trying to rush things through the approval process and put my corporate environment at any risk.  As information started to come out from SolarWinds, CISA, and other sources, we quickly decided to upgrade to the latest version of SolarWinds once it was available.  The reason for this decision was based on the fact that even though everything pointed to 2019.4 HF5 being the bad version, we didn't want to risk anything by building off any servers that ever had the 2019.4 software.  We removed those servers and built brand new VMs as we waited for the new version.  

Once the new version was available we first installed it on my home lab server along with CrowdStrike.  This was done to again, protect anything on the corporate side as well as give the stakeholders direct feedback from a system that I had control over.  We had things run for about 2 weeks and then moved to install SolarWinds on our new DEV.  And while we don't really use the DEV server to monitor anything it was our first steps toward getting SolarWinds back up and running in Production.  After another 2 weeks, I finally had all the stakeholders feeling comfortable with moving things to Production.  

It was at this point that we decided to add an additional web server and just go through all the security hardening steps one last time and as of 22 February 2021, we hae started the process of turning alerts on.  I made the decision to enable alerts in stages.  This will help me deal with any issues that might come out of having the system down for over 50 days.  

So what about you?  What steps did you take or not take?  Is your environment stronger now as a result?  How are your end-users feelings now that we are finally out of the woods with all this?

Parents
  • Thankfully our systems here were running on much older versions and not impacted by this. We examined the systems and executables and found no correlation to the added entities, as well as reviewed no odd potential C&C contact from the Solarwinds server. We deemed it safe to continue operating our current system, but I will be rebuilding it and migrating to a new system in a few months anyway as part of an OS upgrade.

    The main thing it did for us was to refresh and confirm our restrictions on that system and what it can talk to going out of our network. Healthy caution in examining sensitive systems when they were installed and trying to know exactly what something should be talking to and enforcing that has many benefits. Its not really micro-segmentation but that is the idea at least to our internet facing border.

Reply
  • Thankfully our systems here were running on much older versions and not impacted by this. We examined the systems and executables and found no correlation to the added entities, as well as reviewed no odd potential C&C contact from the Solarwinds server. We deemed it safe to continue operating our current system, but I will be rebuilding it and migrating to a new system in a few months anyway as part of an OS upgrade.

    The main thing it did for us was to refresh and confirm our restrictions on that system and what it can talk to going out of our network. Healthy caution in examining sensitive systems when they were installed and trying to know exactly what something should be talking to and enforcing that has many benefits. Its not really micro-segmentation but that is the idea at least to our internet facing border.

Children
  • Yeah I had leapfrogged over the impacted releases because we went about 18 months between upgrades that time around.  Security was able to pull detailed logs of all communications in and out of the Orion server and found nothing suspicious so we were able to carry on.  It actually turned out that somewhere in the company some contractor had installed one of the affected Orion demo products to their workstation so we had to put way more heat on investigating that guy's machine than I ended up getting on our real servers.