Out with the old...Ah crap...

Hello,

So I've used the Solarwinds Toolset in the military (it's been 2 years since I got out). I'm now working for a company who is using Orion and since I have some Solarwinds background, they want me to rebuild their Orion Server. However, this is a large difference as I used Solarwinds to monitor and manage 20 devices tops. This server manages 100's+. One of my colleagues states the server was never setup right and that's why we are getting a lot of false alerts and possibly not being alerted on things that we should be.

Does anyone have any idea where I should start? Whether I should try to find/correct issues on the current server, or scrap it like I'm being told? I'm a bit unconfident in my ability to rebuild a server of this scope without a test environment to work in first...

Tip, tricks, or shortcuts are greatly appreciated. I'm grabbing as much info as I can get from Google, but all the "setup" guides are pretty much Step 1, lets build a server...If possible, it would be great to avoid dumping god knows how much time into scrapping and rebuilding the configs on this server...

Find more posts tagged with

troubleshoot

orion

solarwinds

rebuild

server

Accepted answers

All comments

mrxinu

Hi silphen! I feel your pain, definitely. First of all, thanks so much for your service!

You've definitely come to the right place. There are thwackers all over with a goldmine of information. To get the ball rolling, can you answer the following?

Which software do you have in the new environment? (NPM and SAM are likely, but there may be others)
How many nodes, interfaces, and volumes are you monitoring? (Hint: Settings > License Details)
Are you going to be adding any additional software after the refresh?
Do you have a SQL database server already or would you need to stand one up for this? (Bonus: Do you have DBAs or are you on the hook for this part?)
How many different teams are expecting alerts? (E.g., do all the emails go to one distribution list or are does it depend on the alerting system?)
Are you processing any syslog or traps in the existing system? (Check out the Syslog & Traps links in the menu bar.)
Do you have a requirement to use SNMPv3 in your environment or will SNMP v2 suffice?
Is the current server physical or virtual? Will the next server be physical or virtual?
Do you have SAN/NAS storage for these systems or will they be local drives? If local, do you have any control over the RAID configuration?
Do you have any additional information about the devices (nodes) squirreled away in a spreadsheet somewhere that can be turned into custom properties? (E.g., site, building, contact name, location, production vs. development vs. test, etc.)

Hopefully this will get this thread moving and you can get the information you need. Alternatively, and I don't know how much runway you have to work with or how pressed you are for time, but Loop1 Systems routinely works with our clients to do these kinds of refreshes. If you decide you want a copilot for this thing, give me a shout and I'll put you in touch with the right folks.

Also, have you had a chance to check out SolarWinds YouTube channel? The content is beyond awesome and I think they have a whole series on "first steps" that do a great job of getting you acclimated.

Steven W. Klassen

Programmer Analyst @ Loop1 Systems

http://www.loop1systems.com/

http://www.linkedin.com/in/mrxinu

silphen

Had to dig up some of the info, but I have most of it.

1) We currently have NCM, NPM, Packet Analysis Sensors (Free Commercial License), IVIM, and the toolset.

2) Interfaces: 3423/unlimited Nodes: 184/Unlimited Volumes: 91/Unlimited

3) No new software is being added at this time.

4) We do have an SQL server being ran on the Orion server. No DBA's, my department handles our own DB's.
5) Currently only my team is expecting alerts. In the future, I would like to have alerts configured for the companies who are being provided service to get an alert if one of their interfaces is affected.

6) We had ti disable syslog recently due to some reporting issues. Colleagues said it was spamming the server with false alerts. I would like to setup Kiwi as a replacement, as not having a log system in place is not exactly ideal.

7) I believe we're currently running SNMPv2. We have no requirements for v3, but v3 is believed to be more secure and they would like to use that at some point.

8) Current server is physical. I intend to build a virtual one for redundancy, but our main machine will always be physical in this instance.

9) Drives will be local.

10) I do not have any of this information unfortunately. My engineers MIGHT have it, but it's unlikely.

I should have also mentioned I work for a service hosting company. So we actually provide services for other companies and, depending on which services they pay for, we monitor their connections as well.

I don't know if geting a copilot is something we can do right now. Corporate manages our budget kinda tightly, so I doubt they will pay for any external services right now. As far as the youtube channel, I didn't even know that existed! So I'll be checking that out during my downtime for sure.

designerfx

for 4): You don't want to run the SQL server on the Orion server - you could run NTA's netflow database on the server if you had a sufficient raid but that is not advised for SQL as your environment grows. With 184 nodes (lots of switches I'm assuming), you've got some time - but when you get to 1k or 2k nodes you're going to need a dedicated sql server with some power. Especially if you're going to go virtual you'll probably end up doing this anyway.

6) Syslog is known to have some issues, so you may want to check the versions of NPM/NCM/toolset you have. Are you on the latest?

7) SNMPv3 is ultimately more secure but not likely to be all that important unless you have things being exposed to the internet and SNMP polled from it.

Solarwinds lab does indeed have a "monitoring 101" and "monitoring 102" video as Steven mentions, and it is worth looking over. I'd also look over videos relevant to the tools you are in use of - which are basically NPM/NCM

Long story short, with NPM/NCM/Toolset as an addition, that means you are probably: monitoring availability of switches, maybe checking switch/router configs, and occasionally doing helpdesk support/basic troubleshooting on the side.

silphen

Your assumption on our functions is correct. I do understand SNMPv3 is more secure, which is why we would like to move to it some day, but as it stands with our environment setup, SNMPv2 hasn't been exploited and will have minimal impact if it is.

As far as application versions, I don't currently know the status of that. I'm still awaiting server access from the original system admin.

mrxinu

Hi silphen - thanks for wrangling that info. Let's see what we can do with it.

Software Load (NPM + NCM)

This is a really common pairing and an excellent start for your monitoring. designerfx's observation is spot-on - with that mix of software and node-to-interface ratio you're probably using this for your managing your network at this point which again is a great place to begin. If you decide to branch out and start monitoring servers you'll definitely want to grab an evaluation of Server & Application Monitor (SAM) to add all those other monitoring points (services, ports, user experience monitors, etc.). If you have a current version of the software then you're also sitting at about 25% of capacity so you have plenty of room to grow.

Database Location/Size

Running the database on the same server as the application is an economical choice if you can get away with it. The immediate concern is making sure that you've got fast enough storage. The SolarWinds database amounts to a hybrid OLTP/OLAP database that spends a lot of time writing data that comes from monitoring but then is also on the hook for processing everything that gets thrown up on the web console (graphs, reports, etc.). RAID 10 is your best bet for local disks if you can swing it (you can go RAID 1 for your OS drives if you're able to split it up and have multiple arrays). If you're still using Express (1 GB RAM limit, 10 GB data limit) I would highly recommend going to at least SQL Standard if it's in the budget to get out from under that RAM limitation.

Alerting Different Groups

Since you're starting out with emailing a single group then setting the default To/Cc/From addresses in the main configuration (Settings > Configure Default Send Email Action) and call it good. Just make sure that you're using the ${DefaultFrom}, ${DefaultTo} and ${DefaultCC} variables when you create your email alert actions so the definitions stay easily configurable when you need to make a change. Once you decide to break this out and start emailing different people based on what node, interface, volume, etc. in pain then you'd benefit from using a node custom property called ContactEmail, for example. They don't advertise it right next to the "To" field but you can go ahead and use variables in that field including that custom property by copying out what would go into the body of your email if you wanted to embed that value (i.e., ${N=SwisEntity;M=Node.CustomProperties.ContactEmail}). That way by setting the node's ContactEmail property to whoever cares about that particular device, they'll get whatever alerts fire against it.

Syslog/Traps

I've never heard of reporting issues from syslog being that it doesn't make any noise at all unless you specifically go into the Syslog Viewer utility on the server and tell it to start sending you emails when a particular syslog value comes through. Maybe they thought it was muddying the Message Center (it combines all the different types of messages including syslog, traps, audit event)? You can always adjust what you're seeing in that interface and omit the syslog. In any case, the reason I mentioned it above is that keeping too much syslog/traps can cause some issues with latency so the 7-day window (Settings > Polling Settings > Syslog Retention) is just about perfect. It gives you just enough syslog/traps that you can go back and see what's happened recently without inundating the database with a whole bunch of data you'll never actually pore through weeks later.

Next Steps

I say forge ahead! You've got a modestly-sized environment (not quite one full poller), no big red flags that I can see, and if you can come back to us with more information about the alerts you're getting that you don't feel like you should be we can certainly help you whittle them down. I'd just start with enabling one alert at a time on the new system until you get back up to full speed again.

Hope that helps!

Steven W. Klassen

Programmer Analyst @ Loop1 Systems

http://www.loop1systems.com/

http://www.linkedin.com/in/mrxinu

silphen

So I actually managed to finally get all my access squared away. As it stood, the server had no maintenance cycle and was basically neglected before I got here. I installed over 200 windows updates, applied the latest NPM and NCM, and viola. So far no complaints from the guys monitoring in regards to notifications anymore. As far as syslog, since the system hadn't been rebooted in god knows how long, and was severely out of date, the syslog service was failing until the updates were installed and system rebooted.

So thankfully, I didn't have to rebuild the system. It's simply the technicians before me failing to perform routine maintenance. After updating, out engineers were unable to click on the nodes under the group folders, but I found a simple database query that resolved that.

Thanks for all the info and guidance fellas!