Rebuilding Orion from Scratch - including Database?

ahbrook

I didn't know if this would be the best part of Orion to put this, but there isn't a general Orion Core discussion point.

Has anyone else started considering completely rebuilding Orion from scratch, and ways to make that process less painful? We've seen the guidance about rebuilding objects that were managed by Orion, but our security office is looking into scenarios to basically reset things. I wanted to see what others were considering. 

I can see a few different levels:

1. Rebuild the app servers to bare metal, leave the database as-is. Reinstall agents
2. Same as above, but rebuild the database cluster as well, and then re-load the data into the database cluster.
3. Completely start from scratch with Orion - fresh database, fresh servers, etc. Re-add all our nodes and monitoring back in and rebuild our security.

We're trying to develop scenarios and timelines for all these options, but obviously starting from scratch would likely knock us out for quite a while. I am wondering if there are ways to speed up the process, perhaps involving scripts or other systems to re-implement custom monitors without having to hand craft them again.

There's also our fear of what might have burrowed its way into all of our servers...

mesverrum

The compromise has been somewhat documented at this point, it's no longer a "we have no idea what could have happened" and more a matter of reviewing the indicators of compromise and confirming if they do or do not apply to your environment. 

https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html

https://github.com/fireeye/sunburst_countermeasures

https://github.com/fireeye/sunburst_countermeasures/tree/main/indicator_release

https://www.volexity.com/blog/2020/12/14/dark-halo-leverages-solarwinds-compromise-to-breach-organizations/

https://www.splunk.com/en_us/blog/security/sunburst-backdoor-detections-in-splunk.html

https://www.devo.com/blog/how-to-detect-and-investigate-threats-using-devo/

https://krebsonsecurity.com/2020/12/malicious-domain-in-solarwinds-hack-turned-into-killswitch/

https://github.com/RedDrip7/SunBurst_DGA_Decode

Microsoft took control of the command and control IP's and lists have been published of the domains that were running beacons.  If your security team has proper logging of DNS requests and other network traffic over the last few months they can cross reference the publish lists of DNS queries to watch out for and load the various sets of scan tools that identify the documented foot prints of this hack.  If you have those logs and there is no IOC then it is at least reasonable to assume that you were not actually targeted since this was not some kind of automated blind worm type of attack and appears to have been relatively focused on specific types of organizations, they did not go out and hack every lab that happened to install the compromised releases.  With no indicator of compromise then your company can rationally assess how much of a scorched earth response makes sense.

If you don't have those kind of logs then you are in the spot of having to assume the worst because you have no evidence otherwise.  At that point "start from scratch" isn't rebuild your SW server or a database.  It's probably best at that point to contract a professional security consultancy to have them do recon on your environment, because if you didn't already have the tooling to confirm you weren't breached then honestly you aren't going to be equipped to clean out a breach.  Trying to go it alone would be purely security theater and going through the motions with no chance of actually accomplishing anything if you really have been targeted by state sponsored hackers.