I honestly fell cheated of the money we paid for this year maintenance and here 3 reason why:
- support has hit an all time low in quality and response time
- they created the premier support but even they suck
- this stupid backdoor that they had since march and the way they handled the situation.
Please feel free to add to the list 🙂
I had to chuckle a little at the idea that SW would provide the guidance on how to change your AD and device accounts. They provide a monitoring tool, not an all inclusive system administration robot. The situation is really unfortunate but ultimately we are all professionals and I would hope that it should be within our capabilities to handle those kind of basic administrative tasks like rotating accounts. These are the kinds of issues that business continuity and disaster recover plans in theory should have been made for, and if the plans weren't already in place then now is the time for your company to put their hands on the steering wheel and assert some control.
I can't think of any software vendor that would provide that kind of guidance except perhaps Microsoft, who would just be referring you back to their pre-existing AD administration documentation. I also happen to know of a few SolarWinds employees who have been trying to come up with example scripts to help people with the common pain points around a rebuild from scratch, so they already know your struggles and are trying to get ahead of it. Unfortunately it is non-trivial to just crank that out so it's going to take some time for them to put together some help.
I also want to add that, as customers, it's easy for us to put the cart before the horse. SolarWinds is in the middle of managing the remediation (which is priority #1) and also working with several agencies to assess the situation. We're two days into this... give it time, y'all. Patch your systems. The proper thing to do is to wait for an official announcement from the company that indicates their next steps before you start making demands.
I hesitate to put my two cents in but here goes, and no offense to anyone. The current issue could basically happen to any one of SW's competitors, which makes them no better or worst than others. I agree that they could have handled it better, but hindsight is always better after an incident than before. The choice is ultimately yours to stay with a company or jump ship, but I would also say that it never hurts to monitor a service that monitors your servers. It's like using multiple antivirus tools, if one doesn't catch a glitch it's likely that one of the others will so your still protected. Anyway, that's just my take on the situation.
What would be more useful is;
a. notifications - I don't think I received anything from Solarwinds by email about this - I learned about it from my colleagues who got told by various vendors they use. This is really poor, the technical contact on the account should always have emails about new updates, patches, etc.
b. test/dev keys for the software so we can try out updates in development before we update production. A separate key for each product, with minimal (but sufficient) licenses, say 15 nodes - to cover the large variety of machines some of us have, 2003, 2008, 2008 R2, 2012, 2012 R2, SQL, IIS, AD, etc. That helps us (we should be showing the upgrade has worked in a dev environment before updating the prod one), and helps Solarwinds - we can find the bugs they don't see. All our environments are unique and they aren't going to replicate all our environments in their lab testing.
c. they need to sort out support - the capabilities and response times suck. We absolutely rely on this, I've had System Down tickets logged for days before anybody has responded to them. I realise we are in a pandemic, we all realise that, we're all trying to get on with our day jobs which isn't helped by a lack of confidence in Solarwinds support. They need to be able to contact development, or an intermediary L2 for support, without it requiring a decree from congress.
d. senior management at Solarwinds need to take full responsibility - I don't want to hear blaming others - this is about compliance, ITIL, auditing your work/processes, strict change control and software development life-cycle. That is what management are there to implement, and by all accounts they didn't do their jobs properly - no bonuses for that lot this year.
I have updated - got a configuration error on the primary poller - just re-ran it from the web interface and it went through ok. Everything appears to be working ok. I'm now on 2020.2.1 HF2.
Final note - updating the software is just that, not the end. We now need to figure out how to change all the service account passwords in Solarwinds - audit AD for anything which could have been created. Solarwinds need to provide a guide to this.
Just a quick update on updates:
This page went up on Monday, and was updated at 8pm CST Tuesday night.
And this page has additional details, and will be updated regularly as information becomes available:
for "b. test/dev keys for the software so we can try out updates in development before we update production"
There was a really large thread about this that I argued as much as I could on to be able to get free test/dev licenses to do stuff with. Official keys, not "30 day trials". I even accepted a 30 day trial and said that maybe they offer "extension" or "revival" licenses so you could have the 30 day trial all set up, let it "die" and then when you needed it again you simply "revive" it with a free key. Maybe you get a few of these keys gifted to you every year via the support portal, etc. If you need more, you can beg for a few, or if its excessive, buy them at some rate.
I tried arguing and postulating all I could. I never heard anything come of it other than "if you ask nicely, your rep may give you extensions" and "if you become a Solarwinds MVP (or something) you get free installs to play with". these ideas are good and useful, but in an enterprise environment can be frowned upon, especially if you want to have an "official test installation" you use and patch etc. Of course, the argument turs into "if you're that needful for a test system, and are enterprise, surely you can pay the small license cost for a full install". And well, its hard to argue with that, as the software is not as expensive as other vendors. i counter this with "if we pay for an enterprise license (like an SLX class license, or whatever the highest tier is), then throw in a test system license/extension licenses/every year.
For "a. notifications - I don't think I received anything from Solarwinds by email about this..."
Just for anyone's info, and if anyone wants to collaborate or validate my assumptions:
I received an "official" email notification that arrived in my inbox on 12/13/2020 8:35 AM. It wasn't from my support rep, but looked legit.
My guess is that I received it because I am in the customer support site for my company, and have all the notification boxes checked (support, maintenance, orders, whatever they all are) for me. I would check there and first to make sure you're listed, and the contact info in your profile is "good".
I'm also betting (assuming) that they followed the "notification preferences" listed on the support site, and may have also only sent it out to people who are listed as having a known affected product (known at that time)? vs "everyone on the site".
Why would you want more of the same, horrible maintenance? We are about to move to another software solution ourselves; we've had enough of SolarWinds and their lack of commitment to overly priced auditing solutions. We too have noticed a severe waning in their support since around the start of COVID, and its all been downhill since.
No doubt. I feel like I have stockholm syndrome. We use most of the products and finding one alternative that performs all functions has been difficult (NPM, NCM, SAM, VNQM)
I would be interested in the alternative product you are choosing? Is it replacing just NPM or other components.
Latest information of the hack, shows that the FTP details were leaked to github in 2018!! Password was about as secure as a house with the door open. Whilst the attribution to an APT has been suggested, it would appear that SW themselves gave the keys to the castle with poor security implementation processes.
This could be the final nail in the SW coffin 😞
I don't really thing asking for free year of maintenance, or thwack points is appropriate at this time. The company as a whole is struggling because of this intrusion. It's not something that is going to be solved overnight, and no amount of freebies is going to help in reputation damage this has caused.
By offering free maintenance for the next year they are not forcing existing customers to make a quick decision if they are going to pay the maintenance or invest the money in a new software.
This gives Solarwinds 1 year to earn back customer thrust. Even if they will lose some money now in the long run it will lead to better customer retention.
IMHO, i'd say it's a little late to try and regain trust. This is still the early part of the incident and the full impact wont be known for sometime. Having a year of additional support is going to be worthless, i'd be surprised if companies are not already looking at alternative solutions, and mitigating their damage limitation to their brand.
Trust is something that takes a long time to gain, and a few seconds to destroy. Many companies will likely move away from Orion as a product, the share price of the company halved overnight. And I suspect that will continue to drop the more information that comes out about this hack.
Also SW's as a company have been extremely quiet about all this, as they likely don't know themselves the full impact of this hack. I would be surprised if they're not going to face regulatory fines for this breach, which in all honesty could result in millions in fines in all regions of the world. And other companies taking their business elsewhere, will cause the company to collapse into administration. So getting a good will gesture of a year of free maintenance isn't imo going to cut it with the majority of companies, who've placed their trust in this company, and it's products.
As a premier support customer, I would argue that the cost, while heavy is very worthwhile for the right environments. I'm a massive fan of "Don't knock it till you've tried it.
I think if you already think the response time from support is slow, putting a pressure on them to respond in a certain timescale is going to result in an even worse experience as you will simply get a mass of T1 engineers who simply log and close tickets at the first opportunity. So far in my many years of experience with SolarWinds and the support reps, I have only had a small handful of occasions where I felt the support I got was lacking, and that was nothing that a quick follow up with my account manager or post here in Thwack didn't fix.
By far the best support I have had is here on Thwack anyway, but it depends on your environment and the speed of response you require. SolarWinds are very clever not to sell any set SLA response times with the basic maintenance packages so they are not in breach if things get busy and it means the engineer can spend 20 minutes or 2 weeks working your case.
As for this weeks issues, I can honestly say this is the first time I've ever heard of SolarWinds having any sort of threat or security risk, I think everyone is allowed a bad week there certainly isn't a pattern here for me at all.
I use support 99% of the times for these 3 scenarios:
1. I need official confirmation from Solarwinds that the software really dose not work the way my managers think / where told they work by the Solarwinds sales person.
I need to upload a full diagnostic even if I only have a question and i have clearly marked the ticked as such.
If I don't I get calls from L1 that I need to upload diagnostic in order to get an answer. WTF?!!??!?!?
2. Totally new, never before seen error that doesn't have a KB or a THWACK article and its not caused by performance of infrastructure.
This never ends good for me and there is a total pain to get pass level 1 and most of the times even level 2 are total newbies that have been using Solarwinds for less then 2 years and have experience in only one or 2 modules and have never seen an installation with 3+ modules , APE, AWS and HA.
Lass time I called, I had to explain to L1 and to L2 the difference between HA and FOE as they had no clue how HA worked and they keep confusing it with FOE.
3. I need a license reset or offline activation: these guys are the only ones that actually do something and are very fast with response time. Never had a problem with them.
We renewed the week before so 😉
Smelly stuff happens, as long as we dont find out they sat on it longer than reasonable.
I suspose it hammers home that we should be a little more restrictive on what websites we let servers talk to. I suspect I wont be removing the ACL mitigation I put in place (only allowing SW pollers to talk to websites we're actively polling) unless I find a reason to do so.
SolarWinds solutions are rooted in our deep connection to our user base in the THWACK® online community. More than 150,000 members are here to solve problems, share technology and best practices, and directly contribute to our product development process.