Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

EMPTY INTERFACE STATISTICS FOR 8 WEEK PERIOD

Hello,

I need some insight as to what actions to take. Recently, there was a major outage on my Solarwinds environment that spanned 5 days. The outage was traced to issues with the database.

The database maintenance was seen to have been failing repeatedly after taking over 5 hours to run. Also, the size of the database is seen to be very large.

Now my Solarwinds is back up and running however i noticed that i have lost data for an 8 week period.

I cant understand this as despite the size of the database, we have been unable to shrink it and before the outage there was information on nodes and interfaces being stored within this time period.

Firstly; was this caused by the failing database maintenance?

Secondly; can this be rectified ? If yes, how ?

Lastly; how do i explain that this loss of data will affect quarterly reports and thus translate to loss of revenue.

After a week of trouble shooting and several trial and errors, I'm rather confused at this point.

Find more posts tagged with

Accepted answers

mesverrum

DBA, as in a Database Administrator, a person who is trained in the care and feeding of databases. If you are really lucky you would have someone with some ETL (Extract, Transform and Load) experience as that's pretty much exactly what you are looking at doing in this case.

Since the data was definitely collected then I would restore one of the old backups from before it disappeared and also a current copy of the live database to a secondary location so they can experiment with their process there without negatively impacting production systems. Then start checking the various tables relating to interface history for the time period in question (there is supposed to be a new one created every day). They would probably want to dig around in the built in stored procedures, I believe there is already one saved in there that does the work of moving data between the various tables during maintenance, so you can take a look at that procedure and see if it gives you a model to follow for your DBA to develop their own script to move the data that exists in the older copy into the copy of "prod." Once they went through that process and hammered out the bugs they should be able to let you know what plan they come up with to repair your live system, and if they expect those repairs would require any downtime or anything like that.

If it's not obvious already, none of this would be supported by SolarWinds themselves and you'd be in relatively uncharted waters figuring this stuff out for yourselves, but I have spent enough time in the DB to be fairly certain it can be done once someone sits down and figured out all the details.

All comments

hpstech

Something affected collection that could have been caught with correct monitoring in place. I check everyday for anomalies with stats collection to avoid this.

You must deploy the Orion application SAM template and alert for jobs lost, jobs queued, job results notified errors - all three of these are critical for monitoring stats collection problems. The one that will really show a problem with Interface stats collection is job results notified errors. For a large environment, this can get into the 80,000 - 100,000 range quickly.

I feel your pain. Losing stats is terrible for manged solution providers. I keep a very close eye on this myself.

I also manually check an interface page everyday to make sure stats are collected.

wluther

evi.evru Sorry you're having some troubles there. Just to verify, have you contacted support/opened a ticket yet? If not, I'd recommend doing that as soon as you can. Additionally, listing the specs of your SolarWinds environment (physical/vm, Windows Server 2008/2012/2016/etc., SQL Server 2012/2016/etc.), as well as the Orion product versions currently running, may help us run down the correct path.

Are you missing data for interfaces (some/all), across the board for everything (nodes, volumes, interfaces, assets, cpu, etc.), only for specific vendors/types?

What was the issue with your database?

Do you have backups of the db you can compare with?

Do your audit logs/messages show any changes on/around the times the data stops and starts again? (Could someone have unmanaged/remanaged the interface(s))

I understand you want to know what to do now, but perhaps some more details can help us find the answer.

Thank you,

-Will

mesverrum

^ backups are vital, ESPECIALLY if the database contains data that is considered revenue generating for your business. It would be as though a bank didn't backup the records of money moving in and out of accounts and lost that.

With good backups and some clever DBA work you might be able to backfill the gap in the data.

As far as the database maintenance failures causing the loss, definitely possible/likely. The "maintenance" is a collection of jobs that go through the database and, among many other tasks, deletes orphaned objects, move polled data from the details tables to the hourly and weekly data tables, and deletes data the should be aged out. What I would suspect may have happened was that due to the failure to complete the db maintenance you either did not get the tables created to hold the hourly data during the gap, or the data that was supposed to get moved into the hourly interval sections did not make it over.

The other big question mark, are you certain the data was even collected and stored succesfully? If it was in there before then you are pretty much golden and a good DBA could restore a copy of the database and pull out what you need and add it back in. If the system just wasn't writing data for weeks and nobody noticed then that would warrant a serious look at your business processes around this revenue generating activity.

evi.evru

Hello, with regards to your questions, yes this data was collected and stored, as prior to the issue with database, i am a 100% sure this data was available as i was closely monitoring some specific interfaces and i had all the necessary information available on there. Furthermore, i have lost this data across all interface not some specific interfaces only.

This i am very sure of.

Can you please go into more details about this DBA and its functions ?

mesverrum