Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Did you inherit your SolarWinds environment?--Tell your stories and ask your questions below!

The User Experience (UX) team at SolarWinds is gathering stories of how you inherited your SolarWinds environment--every hilarious, sad, interesting thing that happened on your journey to proficiency, wherever you are along that path, plus questions you had relating to your starting path on that journey. For example, one question we hear inheritors ask is "should I blow up this old instance and start afresh, or should I salvage what's here?" Whatever stories and questions we gather here may be used (completely anonymously) in future SolarWinds training and learning opportunities with other SolarWinds inheritors. Our guess is, a number of you have similar experiences that could be helpful to others to hear; or you might have similar questions that we should be addressing in future trainings. Both stories and questions can be posted below, and we may follow up with short WebEx calls, or, we may just leave things here for you all to respond to each other. We're excited to read what you post!

Find more posts tagged with

Accepted answers

All comments

superfly

In my original fulltime job, I put together the Solarwinds server over about 9 years. I got made redundant from there last year in March and since then I've had 3 contract jobs. First one used Solarwinds but it was only a 5 week job so did nothing to Solarwinds. At the next job they installed Solarwinds during my time these so I ended up helping configure it up. Now where I am, I've inherited a Solarwinds server that is still on 2015 with active maintenance! But as I recently found out, the software itself is out of support so I'm unable to get support. Whoever configured this Solarwinds server has done a pretty good job. I've since then only finetuned it more and put together a bill of materials to get a new Solarwinds environment. At this stage, I will keep the current data and just upgrade it all.

jeremymayfield

I started here in 2012 with nothing, so I built the SolarWinds environment. Then added and added and added. The issue was i had one orion server, Single poller and SQL server all in one. I have had two with a great deal of help from the support side, redo my environment in 2015. Now running on a Seperate SQL server, I still have the one Orion machines, single poller but my environment didn't require much more. I have added more items along the way but all in all, its still the same. In 2018 I built a all new Orion server on a 2016 box and have a new SQL 2017 server which helps a lot with the DB side. I am now pretty sure i will not need to make any changes for at least 12 months.

meech

Hi superfly99 Thinking about this place where you are now, were there any things that tripped you up or were hard to figure out about the state of the Orion environment? Anything about "what objects are in here and are they the right objects?" Anything about anything else? We're all super curious about how we can better support people who inherit an environment.

designerfx

In three separate instances in my career I jumped into someone else's Solarwinds environment. In one (the first) it was an improvement, in two it has been a rip and replace (fresh db and copying in the data we want). In my first job I was handed the solarwinds environment by my colleague and I was 100% fresh into solarwinds. They had a lot of customization, a lot of sql queries, and I was mostly given the administrator guide and some help when I asked for it but it was basically handed off to me. In that environment, some of the major changes I made was making use cases for additional modules (WPM), established a network atlas for a hybrid environment between on-prem and an azure PAAS environment, got SAM agents to work in the cloud, enhanced visibility to network team (it was mostly server-first/server-focused until I showed how well network was doing), and adding universal login via AD groups for all of IT as they were adding people 1 by 1 as people asked for access. I trained the noc staff in how to do mass editing of information, participated in a tough mudder, made lifelong friends, and basically grew into being an IT person. It was my first real IT job. I certainly screwed up a few times in the process as lfransen can attest to while I figured out which way was up in my path This includes the few times Orion was down completely due to my botching an install and the hilarious coincidences of a cleaning person triggering our data center's emergency shutdown.

Honestly, part of it goes to credit of having a manager willing to help but also willing to let me screw things up within reason to learn, I don't know how my career had been if that wasn't the manager I worked with for a majority of my career on that job as well as some other great people there that really wanted to take challenge on and help things grow and challenge me accordingly.

meech I can probably go all day on that job if you want an even longer story - that's just one of the three, a place I was at for 9 years going from customer service -> sales -> IT, which the last 3 years or so were IT. There were quite a few stories from that environment.

meech

designerfx thanks for this! What questions did you have to go find answers to when inheriting an environment? For example, did you know where the nodes came from/if you were monitoring the right stuff, or not? If not, how did you figure it out? Do you remember other questions you had to go research and find answers to?

wluther

My first experience with our SolarWinds environment was a bit over 10 years ago, and I only used it to see red/green dots. As our network grew, and the responsibilities of the other folks moved around, I started adding/removing nodes. At that point, management had decided to replace it with a different vendor. While we were getting the new product setup and configured, we still had our SolarWinds monitoring going. After nearly 2 years, the other vendor pretty much broken themselves out of being our provider. New management came in, noticed we did not really have anything official, and asked what we wanted. By this time, many of the NOC folks had moved onward and upward, so when we said SolarWinds, I was basically the only one with any experience. (not that experiences is really even required with SolarWinds... things just work)

We ended up reclaiming the hardware used to host the previous vendor's junk, and rebuilt SolarWinds on that. We decided to start as fresh as we could, so we basically built it out new, and manually imported each node from the old system to the new. Over the years, I have upgraded/migrated to new servers, each time being a little more easy than the previous. Personally, I usually prefer to just start fresh. The last rebuild I did was at the beginning of 2018, and I was given permission to start completely fresh. With a ton of Thwack time logged and under the belt, the build was the smoothest ever. I was able to harness the API, via PowerShell, in such a way where I imported groups of nodes from the old system into the new. It took a couple of months for everything/everyone to settle into the new environment, but it was well worth it. (It didn't take that long to move/upgrade, but just to figure out the best places for our new VMs to live in our virtual environment.)

With all of the advances now, I kinda feel like that old grandpa who tells you how bad things used to be compared to what the kids have to do today... Back in my day, when we upgraded our servers we had to do so much planning and preparing... you know, planning and preparing, when you had to think of how many days the upgrade was going to take, or when you needed to move every single file from every single product to every single server, where we had to write down all our upgrade plans/ideas on a notepad... and that notepad was an actual notepad... you know, a notepad, those little books with blank papers in them... you know, a book, those things which have paper in them, and you open and read them... you know, paper, those very thin, rectangular shaped things people write on... you know, people, those crazy things walking around outside on two legs... you know, outside, that place you go when you're not inside...

Uh, yeah, so the folks taking over your system in the future probably won't have as many fun adventures as those who upgraded/built before them.

designerfx

Absolutely, I do remember! The most crucial first steps I had to go through was understanding the nomenclature of the environment. Every single company I've ever worked for has their own way to tell you that server X is in (group a) but also (group b) and not (group c), whether that's regions/production/function/state/countries/group. That in and of itself is usually a 3 month to 1 year affair to really nail down, and that is the most crucial part of understanding what makes up an environment itself. Without that, you don't know what you're asking for because you aren't speaking their language.

At the same time, you need to learn what other people are doing and why - you need to understand their processes and logic behind adding nodes, discoveries (if they even know they exist, for that matter), custom properties.

Then, you also need to find out who the people are, the processes they go through, the avenues for improvement. These can be things like change management, or manager IT meetings, or where people go to present recommendations of modules.

So we have multiple forms of learning you get to pick up simultaneously:

(company) IT language

(company) Solarwinds language

(company) People, products and processes

There is no way to not go through those, and depending on how complex the environment is that could be 3 months or 1 year.

Keep in mind I was 100% fresh, so I didn't even have knowledge of what the modules were/what their function was/what the group wanted/what management wanted, so all of that mostly came later. The first year was a lot of doing and learning and less of thinking until I really could pin down what was a reasonably large environment. You'd think having a single datacenter on-prem was small, but in reality it was a pretty large environment with how diverse it was. In the middle of all that I had to sign up for thwack, get a a customer portal account. It really was Thwack that helped me with a lot of my early questions. Meanwhile, as all of this is going on someone SHOULD be making a configuration document of how things were set up and why, etc - that way they aren't an individual responsible for decisions and someone has a reference of what's being done.

Hope this helps meech

rschroeder

I'm probably that old Grampa that wluther talked about. I came to this organization in 2003 and, although it had 14,000 employees, it had no network management monitoring tool, no network discovery solution, no network configuration management, etc.

I'd just come from a public school system where I'd set up 33 sites with their first internet, first WAN, first firewall, etc., and originally it was all 3Com leveraging HP's OpenView Network Node Manager--with all the custom tie-ins to the 3Com network. I easily migrated it to Cisco when 3Com went toes-up and thought HP OV was a sweet product. When I moved and found no NMS present, I recommended HP OV NNM.

That was my first big mistake, and it cost me no end of money and frustration and political capital and good will and trust. HP OV NNM is NOT a product that can quickly and easily discover and monitor and manage a network. Getting out of that fiasco took many months and eating much humble pie. $20K+ in 2004 dollars for worthless training costs, plus costs for physical servers and licensing and lost time, all down the drain.

Re-reviewing the NMS's available in the end of 2003, the only one that looked promising and full-featured was Solarwinds. I bought it in 2004 and started with NPM. I quickly realized NCM was going to be mandatory for properly supporting a health care system that would grow to a hundred hospitals, clinics, and business offices, with 17,000 employees across three states. NTA came soon after, and was my proof the WAN circuits were undersized and people were using them for inappropriate personal entertainment at the expense of business and customer needs.

When I retire in some (hopefully near) future, people will ask "Why did Rick do it this way?"

The honest and simple answer is that I've done the best I knew how, but was not given training or dollars for NMS consulting and design to do it the way the top professionals might recommend. Between Thwack and Solarwinds documentation and other online fora I've put together standards and best practices, and I've changed them as I learned better ways.

Eventually we hired an awesome network architect who seriously is one of those genius people to whom Cisco turns for answers to pressing issues. He helped create a better standardized network drawing and node and site naming convention, and just that simple bit provided obviously increased ease and efficiency in dealing with sites and nodes that may have problems.

I'd love to see someone's Work-In-Progress for all the best standards & practices they recommend, learned through practical experience and trying and using (or discarding) other best practices that may be rooted in theory but disproven in practice. I hope it would parallel mine. Things like:

All snmp must be read-only and v3 (both for security reasons)
All sites will use an intuitive network name that complies with a specified naming convention that is well thought out and flexible
All nodes will use network node names that reflect their site, their network room, their rack location, their purpose, and a unique identifier. All of these will be leveraged by NPM and NCM in the future, so deciding on a good convention today will pay benefits in the future
Limit the styles of hardware to the fewest different models possible. This limits the amount of skill sets and IOS or IOS-XE syntax you must know well.
- Settle on one or two routers and keep to them, even if the smallest is overkill for certain sites.
- Settle on one switch size and model if possible, and stick to it. That gets you the ability to keep cold spares on the shelf and swap them quickly and easily without having to wait for an RMA when they fail.
- Keep Distribution Switch hardware the same between all sites. Don't use VSS 6509's at site A, Nexus 5K's at Site B, and L3 3750's or 3850's or 9348's at Site C
ALWAYS use resilience!
- Every network must be designed as hub-and-spoke, never daisy-chained, if you want to limit down time for patients and physicians
- Every access switch must use to port-channeled uplinks to a pair of Distribution Switches that are in a VSS relationship. This gives you the ability to upgrade either Distribution Switch without causing outages to anyone. And you get resilience in case fiber or a GBIC fail--traffic uses the second path and no one knows about the outage except your team, You get to fix it hitlessly!
- Use Chassis Switches with dual Supervisors everywhere, that have dual uplinks and dual power supplies with dual power sources (one UPS, one generator or unprotected building power). It's all about Five 9's, or Six 9's, of up time. This is the hardware that can provide it, since it can perform "In Service Software Upgrades" hitlessly to end users. Stackable switches can't do this.
- All data drops at users' areas must have an A and a B drop, and each A goes to the "A" switch, and each B goes to the "B" switch. It ensures no nursing station, Operating Room, Emergency Room has all of its network connectivity fail simultaneously. If I have to replace chassis switch A in a hospital, staff know that every "B" computer will be up and running.
- A and B switches are to be mounted in different racks, and A and B jacks must terminate in those same racks. Why not make it easy for technicians to get it right and patch to the nearest switch instead of dragging long patch cables across the front of network racks to the wrong switch? Make it hard to do the job wrong, and you'll end up with more right actions than wrong ones, especially when those actions are out of your sight right up until you come to replace a blade or a switch.
- I followed the Access Layer, Distribution Layer, Core Layer method with full meshed resilience; no one can say I didn't do the right work to provide the most up time for patients and staff.
- Push for at least two WAN providers for every site, with duplicate hardware and intelligent routing protocols that quickly fail over in case of problems with a router or a circuit.
- Do the same for multiple ISP's and manage your own BGP so you can control changes quickly and efficiently.
- Budget for sufficient training for your staff so any person can move on to another life without leaving you without someone to pick up the baton and keep on running. That's TRUE Resilience!

Sigh. It's Friday afternoon, and I could write a book about what I think are best practices for hospital networking, for WAN, for firewalling, for wireless design and use, for NAC, for DNS and DHCP, and for network monitoring. I'm heading home. Maybe I'll get all fired up again next week and add onto this, but I suspect you all know the best ways to serve your users in your own environments. Let's hear what things your descendants will inherit from your work.

Swift Packets!

Rick Schroeder

mesverrum

My first Solarwinds environment was a great learning opportunity. AKA, an inherited train wreck. I was a brand new CCNA working my first gig as a "Network Technician" position where it turns out that only 5% of what I did in any given week was networking at all. My first day they asked me "Hey can you jump onto this server and check something for me" and after a moment of panic I went with "How would YOU jump into a server?" because the Cisco-centric training I had only ever used Putty to access anything and all I knew was that Putty was not going to be the tool for this task.

While I was there one thing that drove me insane was the complete lack of guidance on what was out there. 3 locations, 2000 switches, and nobody could tell me how to find anything except to grab a probe and tone the wires. I set out to document the logical network and the physical infrastructure completely reverse engineering everything from scratch and learning a ton on the way. 2 months into that my manager saw the things I had been putting together in Google docs and pointed me toward their neglected Solarwinds instance, suggesting I take over responsibility for updating it and see if that was helpful for my project. We opened a ticket with support and they walked me through my first time upgrading from a 2 year old release to what was current at the time, fortunately things went pretty smoothly and then I set out to figure out how I could take this tool, bring in all the notes I had been compiling in GDocs, and turn it into something that kept me from getting called at 3AM every...single...day during my on-call rotation weeks.

I made really solid progress and I have to say, my youthful enthusiasm for just trying any crazy idea I had and rigging up my own solutions was very compatible with the way Orion works. Many ways to accomplish most tasks, and the platform is extremely flexible and willing to tough it out through whatever goofy stuff I came up with. From that point forward my career has largely been built on seeing things I wish I knew how to do, learning how to make that happen, and then showing my tricks to other people.

"I need to get info from that switch, what's SNMP?", "I wish I could do these things faster without clicking so much, what's SQL?", "I wish I could just double click and have the computer just do all my work for me, what's Powershell?"

Now that I do the consulting thing for a living I have built up a whole stack of processes, queries, and scripts that help me to dissect whatever the last admin had in place so I know what I'm starting with and how to move it forward. Things like double checking all the existing custom properties and alert logic, finding which alerts are the most noisy, figuring out what credentials are in use where, checking for any custom SAM templates, spot checking for places where people have changed polling and retention settings from their defaults. Now that I know what a standard deployment looks like the game is to find out all the places where anyone else may have deviated from those so I can ensure that we either keep, or maybe improve that logic. I have a SQL report I run that gets the name of every custom property, checks for all the alerts/view resources/reports/account limitations/group definitions that those properties are currently referenced in. That way before I change anything I can feel reasonably confident that I know what might break.

superfly

I think if you know the Orion platform, then you should be able to figure out any environment that you come across. The particular environment where I'm currently working is fairly simple so there were no things that tripped me up. I wasn't hired though for my Solarwinds skills but it was a bonus for them so now I can work on getting Solarwinds updated.

I think it would be hard for you to support someone who has inherited an environment that they can't understand. Solarwinds admins do things in a way that best suits their purpose and in a matter that makes sense to them. It may not necessarily make any sense to the next Solarwinds admin. But a good Solarwinds admin should be able to work out enough to see what's going on.

If I can see things in the environment that can be done better, I will implement them. At the same time, I'm teaching the usual Solarwinds admin on how to do things and what things can be done to make things better.

superfly

My original journey sounds pretty similar except someone had set up a basic config of Solarwinds. I then kept building on it. It too was a single box with everything on it inc NTA database. Then eventually I moved databases onto a proper SQL environment and virtualised the server. At the end it was still a single server that monitored about 40 odd sites (3 major sites) and had NPM, NCM, IPAM, NTA, UDT, VNQM & VMAN.