SUNBURST HF1 Mitigation Details?

Morning.

Like many, I'm trying to get a handle around our security posture and mitigation in response to last night's SUNBURST exploit. One of the questions I'm left with after reading the SolarWinds Security Advisory is what exactly the HF1 fix actually did.

From what I understand, the infected DLL was installed in updates through March 2020 and June 2020. HF1 came out in November, but SW was only made aware of the infection last night.

But the Security Advisory instructs users to update to HF1 "to ensure the security of their systems." This makes it sound like there's nothing to worry about if you have HF1 applied. What did HF1 actually do? How should I respond to my clients? Because the article goes on to say that HF2, due out tomorrow, is what will actually replace the infected components.

Any clarification on this would be appreciated.

Cheers,

Ted

Find more posts tagged with

Accepted answers

adam.hert

The security advisory page is being updated: https://www.solarwinds.com/securityadvisory

It now describes which versions replace the compromised component.

All comments

abby.warren

We are in a similar boat. We are on the latest release. We went ahead and powered off our Orion servers until that update comes out.

tsadler

So did we -- I'm hoping SW provides more details so we can better assess our position...

adam.hert

The security advisory page is being updated: https://www.solarwinds.com/securityadvisory

It now describes which versions replace the compromised component.

Radioteacher

Here is some background reading on what is going on with Solarwinds Orion. Reports are coming out that a supply chain attack of Solarwinds Orion was used in the breech of FireEye and US Government resources.

News
https://mobile.reuters.com/article/amp/idUSKBN28N/
https://www.zdnet.com/article/microsoft-fireeye-confirm-solarwinds-supply-chain-attack/

From Solarwinds
https://www.solarwinds.com/securityadvisory/

From FireEye
https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-ch...

IOC's from Github
https://github.com/fireeye/sunburst_countermeasures/

DHS Emergency Directive 21-01. Mitigate SolarWinds Orion Code Compromise.
https://cyber.dhs.gov/ed/21-01/

gstadter

DHS Emergency Directive 21-01 says affected versions are 2019.4 through 2020.2.1 HF1.

https://cyber.dhs.gov/ed/21-01/

Conflicts with how SolarWinds's advisory reads.

bmallon

I was told this morning by a SolarWinds support agent the HF1 doesn't actually fix the problem, it merely, "...makes it more difficult for the vulnerability to be exploited". I asked when HF2 will be available and was told "sometime tomorrow". No definitive info.

ecklerwr1

@Radioteacher once again with the best information. The best details of the incident are in:

FIREEYE:

https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html

The github link contains tools/filters to help see the APT.

Bill

rsprim

Please include the Date and time the Advisory is updated on the website. We are not seeing the changes you mentioned reflected on the webpage at this time. Thanks!

vinay.by

Hello @adam.hert , all,

I have gone through several posts regarding the same and it states SolarWinds 2019.4 through 2020.2.1 is vulnerable at the moment. I don't want to assume anything that's the reason i am asking this -> Does that mean SolarWinds Orion Platform 2018.4 HF3 is stable at the moment and isn't vulnerable to this malware (SUNBURST) ?

One of my projects is running on SolarWinds Orion Platform 2018.4 HF3 and i had plans of moving it to 2019.X this week. I would now wait for SolarWinds to provide a feasible path.

happyfunnorm

It would be nice to have an actual live chat like we have during ThwackCamp for, like... today. Is there anything like that anywhere? Can someone from Solarwinds set that up, or someone else and just share the URL?

Radioteacher

@ecklerwr1 thank you. I got about 4 hours sleep last night. I was lucky at my new employer that they have an old version that predates the issue. We have been too busy to upgrade since February.

I gathered that list to pass out to those that ask about the Microsoft named "Solarigate" issue.

RT

"...gate" issues? Really. I am really old but I was in elementary school when "Watergate" happened.

ecklerwr1

@Radioteacher Me too... gate was way before my time really. I'm lucky with one system and the other is offline right now. I'm hoping SEM has nothing to do with this. I'm pretty sure it's a totally different codebase and totally different developers. It seems this may have been done to target specific targets. The DHS lists some pretty drastic measure though. This is only the 5th time DHS has issued a directive like this.

Bill

sturdyerde

I like the idea of a live chat, although that would get noisy fast without breakout rooms for specific questions (and pinned posts)!

We were fortunate to be on an unaffected version still, but still stayed up most of the night confirming this to be true while also blocking internet access from our Orion servers.

FireEye has the most detailed post by far, but this SANs post is a great combination of detail and readability: InfoSec Handlers Diary Blog (sans.edu)

mwb

Can anyone share the log file path for Orion installers to determine exactly which patches went on the software when?

rguthier

So if my Orion platform says:

Orion Platform 2018.4 HF3,

Leave it be for now?

Thanks

mesverrum

@bmallon I will say I never trust anything a first line support rep tells me. They are often the last to know anything real but many view it as part of their job to sound like they know what's going on. *Now I am speculating here, but I'm not pretending its official from anything*

From the detailed breakdown announced so far of places where the exploit was actively used in a hack I saw there were a lot of mentions of the fact that the hackers would change files to escalate themselves, then put the files back to how they were supposed to be after they no longer needed them. Helps to remove the footprints as they went about their business. I would not be surprised if they also used that exact same pattern while they were within the SolarWinds code repo, got their DLL added to some targeted releases for a few months, and when they confirmed they were in the networks they wanted to get to they put the affected DLL file back how it was in the SW repo to reduce their risks of being detected. I doubt SW themselves fixed HF1, at least not intentionally.

fitzy141

I have updated to HF1 - but we have no confidence right now in that fix and we will be turning off our environments till HF2 has been released or more information from SW.

sturdyerde

The default log file path for installers is at C:\ProgramData\SolarWinds\Logs\Installer. @mwb

mwb

I presume "current_setup.log.csv" shows what is going on, but is there a list of corresponding version numbers to HFs?

"SAM";"Server & Application Monitor";"Product";"2019.4.1.8180";"Rtm";"1"

"CORE";"SolarWinds Orion Core";"Component";"2019.4.5220.20161";"Rtm";"0"

SeaDave

Here's what we found so far:

We updated to 2020.2.1 HF 1 on 6/28/20, going back through our logs, we saw the following DNS traffic on July 3rd through the July 6th from our Orion server.

_time	host_addr{}	query	query_type{}
2020-07-06T20:52:10.154-0800	20.140.84.127	6eivqct649pcg0g16ol4.appsync-api.eu-west-1.avsvmcloud.com	A
2020-07-05T19:42:09.525-0800	8.18.145.151	707gigk9vbc923hf27fe.appsync-api.eu-west-1.avsvmcloud.com	A
2020-07-04T18:15:08.949-0800	8.18.144.150	ea99hr2sfen95nkjlc5g.appsync-api.eu-west-1.avsvmcloud.com	A
2020-07-03T18:08:08.316-0800	8.18.145.62	hnhb3v1b37dvv09icg0edp0.appsync-api.eu-west-1.avsvmcloud.com	A
2020-07-03T17:03:30.418-0800	8.18.145.139	if9prvp9o36mhihw2hrs260g12eu1.appsync-api.eu-west-1.avsvmcloud.com	A

We use app control software so we are fairly confident that any RAT activity would have been blocked but we are still checking.

This AM, our App Control software flagged the following on the Orion Server:

File Name

Publisher or Company

User Name

Trust

Threat

Local State

Group Name

File Path

File Size

Product Version

Publisher

MD5

SHA-1

SHA-256

solarwinds.orion.core.businesslayer.dll

Solarwinds Worldwide, LLC

NT AUTHORITY\SYSTEM

Malicious

Banned

67ffb90.msi

c:\windows\temp\solarwinds\configurationwizard\assembly\dl3\fd596b68\00db1f4c_2728d601

1028072

2020.2.5300.12432

Solarwinds Worldwide, LLC

846e27a652a5e1bfbd0ddd38a16dc865

d130bd75645c2433f88ac03e73395fba172ef676

ce77d116a074dab7a22a0fd4f2c1ab475f16eec42e1ded3c0b0aa8211fe858d6

solarwinds.orion.core.businesslayer.dll

Solarwinds Worldwide, LLC

NT AUTHORITY\SYSTEM

Malicious

Banned

67ffb90.msi

c:\program files (x86)\solarwinds\orion

1028072

2020.2.5300.12432

Solarwinds Worldwide, LLC

846e27a652a5e1bfbd0ddd38a16dc865

d130bd75645c2433f88ac03e73395fba172ef676

ce77d116a074dab7a22a0fd4f2c1ab475f16eec42e1ded3c0b0aa8211fe858d6

solarwinds.orion.core.businesslayer.dll

Solarwinds Worldwide, LLC

NT AUTHORITY\SYSTEM

Malicious

Banned

67ffb90.msi

c:\program files (x86)\solarwinds\orion\ncm

1028072

2020.2.5300.12432

Solarwinds Worldwide, LLC

846e27a652a5e1bfbd0ddd38a16dc865

d130bd75645c2433f88ac03e73395fba172ef676

ce77d116a074dab7a22a0fd4f2c1ab475f16eec42e1ded3c0b0aa8211fe858d6

As previously mentioned the Fireeye page lists full details and lots of IOCs:

https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html

I'd like more info as to it relates to "HF1 makes it harder to exploit". I'd also like to know if HF2 will remove the instances of the exploited file and if our App Control system has banned this file, will it cause Orion to stop working (it doesn't appear to so far) or the update to install?

ralphriley023

Same here - alot of people in this boat.

SeaDave

Everyone should note the traffic to the C2 servers was over 4th of July weekend. Also what we don't understand is we didn't see the "compromised" file until 6/28/20 when we updated Orion, but this very same file that is now identified as "compromised" is in the HF1 update that SW is advising everyone install? Maybe because we had the HF1 DLL version already installed, the "attempt" to reach out to the C2 failed in early July and we got lucky, as SW indicates having this makes the system harder to exploit? We simply aren't seeing any other indicators of activity and we are definitely a targeted industry.

foonly

I thought 2020.2.1 came out in August.

https://documentation.solarwinds.com/en/Success_Center/orionplatform/Content/Release_Notes/Orion_Platform_2020-2-1_release_notes.htm

Orion Platform 2020.2.1 Release Notes
Release date: August 25, 2020

rpetrini

CISA released Emergency Directive 21-01

cyber.dhs.gov/.../

This may be a precaution, I would assume others have reported further persistence mechanisms. They do provide good information, however they do not give out the information that others have reported. Just FYI

"Treat all hosts monitored by the SolarWinds Orion monitoring software as compromised by threat actors and assume that further persistence mechanisms have been deployed"

EthanH

Hi,

can someone please verify which version should I be seeing for the (obviously) not affected version "2020.2.1 Hotfix 1" in "Version Info"?

I can see the following versions on my productive installation.

Thanks.

SOLARWINDS.ORION.CORE.ACTIONS.DLL	2020.2.15300.12766
SOLARWINDS.ORION.CORE.ACTIONS.STRINGS.DLL	2020.2.15300.12766
SOLARWINDS.ORION.CORE.ALERTING.DLL	2020.2.15300.12766
SOLARWINDS.ORION.CORE.ALERTING.PLUGINS.DLL	2020.2.15300.12766
SOLARWINDS.ORION.CORE.AUDITING.DLL	2020.2.15300.12766
SOLARWINDS.ORION.CORE.BUSINESSLAYER.DLL	2020.2.15300.12766
SOLARWINDS.ORION.CORE.CERTIFICATEUPDATE.DLL	2020.2.15300.12766
SOLARWINDS.ORION.CORE.COLLECTOR.DLL	2020.2.15300.12766
SOLARWINDS.ORION.CORE.COMMON.DLL	2020.2.15300.12766
SOLARWINDS.ORION.CORE.COMMON.STRINGS.DLL	2020.2.15300.12766
SOLARWINDS.ORION.CORE.DATA.DLL	2020.2.15300.12810
SOLARWINDS.ORION.CORE.DATABASE.SETUP.DLL	2020.2.15300.12766

ahamiltonXD

May I ask how you are tracking external DNS lookups...?

adam.hert

hey @vinay.by the specific versions are now on the bottom of this page: https://www.solarwinds.com/securityadvisory

mwb

How about the version numbers that correspond to the current_setup_log.csv file in C:\ProgramData\SolarWinds\Logs\Installer?

"CORE";"SolarWinds Orion Core";"Component";"2019.4.5220.20161";"Rtm";"0"

I'm trying to determine when the version went on without backtracking my notes.

adam.hert

The security advisory now has an 'updated at' timestamp at the top of the document.: https://www.solarwinds.com/securityadvisory

scottyplo

Pretty drastic is an understatement. They basically just say unplug it until the issue can be resolved.

SeaDave

We log to Splunk from our DNS hosts. You may also be able to do this if you have a UTM based firewall and logging solution (I know Fortinet and FortiAnalyzer can do this). We use the Spluk App for Stream to sniff the traffic to and from each DNS host.

https://splunkbase.splunk.com/app/1809/

mwb

It did and 2020.2.1 is not listed here at the moment:

https://www.solarwinds.com/securityadvisory

robt

I suspect I have some further reading to do, but I'm more concerned why it seems the affected product list has changed, I originally read the e-mail and previous version of the Security Advisory as versions 2019.4 HF 5 to 2020.2.1 inclusive as being affected.

It now states "SolarWinds asks customers with any of the below products for Orion Platform v2020.2 with no hotfix or 2020.2 HF 1 to upgrade to Orion Platform version 2020.2.1 HF 1"

I interpret this as 2020.2.1 not being affected as this would surely sit between 2020.2 HF1 & 2020.2.1 HF1?

I'm assuming with it's release date of HF1, that is not a fix nor does it negate the issue, as such I am better off going straight to 2020.2.1 HF2 once released?

janobi

Consider yourself compromised. Turning off the servers is unlikely to resolve anything, as they will likely have moved into another part of the network, and dropped persistence measures.

kdara

We have updated to 2020.2.1 but not with HF1 and turned off all Orion servers in our environment until HF2 released.

cabarnes

SANS Emergency Webcast: What you need to know about the SolarWinds Supply-Chain Attack

Monday, December 14, 2020 at 5:00 PM EST

https://www.sans.org/webcasts/emergency-webcast-about-solarwinds-supply-chain-attack-118015

Yesterday, SANS asked Jake Williams, an expert in both offensive techniques and incident response capabilities, to examine the data and provide detailed information on what we currently know about this incident.

Jake Williams presented on:

— The latest information regarding the Solarwinds incident and the mechanics of the supply chain attack.
— Any known detection mechanisms, including IOCs, have been released at this point.
— How the incident could impact organizations that use SolarWinds and where to begin investigations.

The archive of this webcast is also available on the SANS YouTube channel: https://youtu.be/qP3LQNsjKWw

ahbrook

This was a really good, easy to digest video, though there was a bit of dunking on ops folks.

He also had some concerns with how Solarwinds is handling this, which I have to echo. I'd absolutely love to see a lot more public information on today's patch. How it works, why it is secure, etc. Basically I need all the ammunition I need to convince my superiors that Orion is a safe, secure platform... Again.

At one point he talked about how most Orion deploys have local admin access... which is true, because I spent a lot of time trying to hunt down how to secure it more, and things like AppInsight are insanely complex in what security they need in order to function properly. This is another area I'd like to see Solarwinds focus on in the future - minimizing the attack plane by lowering what security it needs to the bare minimum, and properly documenting what those permissions are.

krc85aggie

We were lucky,

I had upgraded to HotFix1 to get the latest Orion Maps updates a month or so ago.

I am really disappointed SW has not been more informative today about the HF2 release!

I mean tell us it may be a day or two more to insure a good product or update every hour or two that it will be later in the day.... whatever.....

Communications during this time could have been much better.

mesverrum

For anyone that didn't already see it, HF2 was released a few hours ago. I just finished my upgrade, so far so good but I'm testing now to confirm the security changes didn't break anything for me.

krc85aggie

So,

They just sent an update the Hot Fix was available but it is a dead link..... Come on!

foonly

Turning off servers and disconnecting them from the switch preserves evidence for further analysis when additional info comes in down the road. People want to know how much rebuilding they may have to do due to lateral infection from the SolarWinds server.

If you must run Orion, create a new Orion machine and do a fresh install.