Open for Voting
over 2 years ago

Data Warehouse for long term interface statistics, reporting, trending, and capacity planning

One of the useful aspects of Orion and NPM is the statistics gathering for performance and capacity planning...until you have to change the IP address for whatever reason or similar events.
Now all of the carefully gathered performance data is gone.

What do you use to trend the old to the new?
How do you know if you have gained any ground or not?

Similar data is gone if you re-index a switch's interfaces (hardware upgrade).

There is also the issue of being able to retain detailed performance data for long period of time...6 months,1 year, 2 years, or longer for reporting and accurate trending.

One of the items Orion is missing is a data warehouse to archive that performance data for long term reporting.

It would not be practical to keep 18 months or more months of performance data in the production database...and then risk losing 6 months of valuable perfromance data after a hardware change. 

So introduce the data warehouse.

It would be a separate out of band database optimized for reporting.  It syncs updates every hour or so from the production database through a scheduled process.
This allows for near time data for reporting.  Thus any reports are not hitting the production database unless it requires recent data within say 4 hours.

Thus the production database can be set to trim at a lower level (days or weeks instead of months) keeping tables smaller and the product more efficient.
Based on this it would be desirable to locate the data warehouse on the SAN as it not as critical for shear speed or for SolarWinds to operate.

This will help product performance by keeping the production database size down.

Now you can retain detailed historical data for longer periods of time and be able to map annual trends/growth during seasonal fluctuations.
An example would be, your company has a spike in network activity related to the Christmas season every year.  How do you trend the last three years to see
if you may be close to saturation?  Without the historical data available you can't.  How does the new switch chassis you put into production 2 mnths ago fit into those trends?

Chances are all the data for the old interfaces prior to the current switch is gone since the new interface layout changed.

By having a data warehouse you would still have the data, but  in an out of band database you can query to your hearts content and not impact production polling thus providing the ability to answer those questions.

This should also apply to any monitored statistic in SAM and WPM as well.  This allows you to trend data for applications and see what changes to the environment have done and whether they
have positively or negatively impacted response time and availability as well as other monitored metrics over time.  This is huge for software upgrades and troubleshooting.

So the biggest wins are long term data points for  capacity planning, out of band reporting reducing overhead on primary database, reduced cost of SAN based database for long term storage, retention of data through hardware and IP address changes, smaller more efficient production database for increased performance via smaller polled metric specific tables.