cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 13

Orion 9.1 - Database Maintenance

The release notes didn't indicate any changes in this area; however the first database maintenance run after the installation of 9.1 appears to be taking a very long time.

Typically it finishes very quickly; today it's been running 13 hours (and still going).

It's still doing database activity (currently on a step called NetperfMon.dbo.dbm_InterfaceTraffic_DeleteStale;1) - and has utilised a huge amount of CPU and I/O.Anyone else seen something like this?

Dave.

0 Kudos
22 Replies
Level 7

Hello,


 


Im new using the NPM. I´ve been monitoring interfaces for a log time. Recently appeared a message indicating that the Database was getting full. I just did a manual database maintenance but the message is still appearing.


Could anybody  tell me how I can backup the database?? Where are the database files located in the PC???


How much data the database can save???


 And if you have any documentation that may help to understand everything about NPM, please share it.


 Thanks

0 Kudos

Cinlu,

Are you running the eval?  If so, we install SQL Express, which has a 4GB limit.   Are you running the NetFlow module? 

If you're running NetFlow, you should go to Admin and  change your retention settings to just save a few days of data.  On NPM, you can go to System Manager, under File > Settings, you can change the retention interval.

 If you are running in production, you will need to install a full SQL Server that has no storage limt.

0 Kudos
Level 12

I upgraded my Dev instance of Solarwinds to 9.1 and haven't seen an issue with db maintenance yet.  Usually runs in less than 30 seconds.  Of course I have about 15 hosts being monitored on that Dev box so may not be a great test for you.

0 Kudos

Well, 32 hours later the db maintenance job ended. I have no idea why this occurred but will keep an eye on the next run.


The maint log indicated that the bulk of the time was used 'Removing summarized older data from InterfaceTraffic'


Dave.

0 Kudos

Just curious, is your database much smaller now? Another thought is the maintenance job may have re-indexed your database. Just a thought.


How big was your database before? How big is the database now? How much free space is in the database?

0 Kudos

The database size didn't appear to change but the free space has increased. Part of our maint plan is a regular reorg and index rebuild. The reorg sets freespace to 10% - currently it's almost 25%. The index rebuild normally only takes 3-4 minutes and the reorg 15 minutes - so I have a good handle on those tasks.


So the next run of our maint plan should reduce the database size.


I can only assume that for some reason our past Solarwinds maintenance wasn't actually cleaning up the required data with each run, and 9.1 resolved that (I'll find out on the next run!).


I have checked to see if we still have all the expected interface data history - seems to be all there, so no concerns on that front.


Dave.

0 Kudos

 In 9.1, we did some work to look for interfaces that belonged to nodes that were longer in the database.  When we found one, we marked that interface deleted.  DB Maintenance cleans out all the data from the database associated with those orphaned interfaces.  Now, I'm amazed that it took as long as it did, but I'm guessing your next pass will be much quicker.  If not, open a ticket with support and we can investigate what's going on.

0 Kudos

Thanks Casey - I'm sure that is exactly what happened. We have 4000+ nodes, and at least 1500 have been refreshed over the last 3-6 months. This would have left us with a large number of orphaned interfaces as you have explained.

Always nice to have a explanation to these things!

Dave.

0 Kudos

I am going on about 72 hours right now.   It is sitting at the "older data from InterfaceTraffic" step.   I know it is still chugging, because the free space is slowing going up.   Right now I have a database size of 28gb with 18gb free and growing. 


I knew something was wrong because my website was really slow.  I always know that the Database-Maint.exe is running when the website slows down.  Can't wait to see if it speeds up after maintenance is finally done.


 MarkS

0 Kudos

In the second maintenance run after the 9.1 install, everything is now normal for us (total elapsed time back to a more normal 25 minutes).


Dave.

0 Kudos

 Same problem here after upgrading to 9.1 after running great on 9.0 SP2 since the summer.  The database maintenance started taking a REALLY long time and has not been able to complete since upgrading to 9.1. When I would get to work in the morning it would still be running, it starts every night at 2:15AM.    I had to end task on the "Database-Maint.exe" in task manager in order to get the Polling Engine running again.  For some reason the Polling Engine stops during the database maintenance at some point and I end up loosing like 7 hours worth of polling data.  Essentially, every night it is getting stuck on "Removing summarized older data from VolumeUsage".  The VolumeUsage_Detail table is a 1.6GB table which I have compacted and re-indexed, but the problem remains.  This never has happened before upgrading to 9.1!!!  

 PLEASE HELP!!!!  I did open support case #65251 for this.   So far, all support has recommended I do is truncate the table the maintenance is getting stuck on and try running it again, but that would mean loosing all of my data in that table which is unnacceptable.

0 Kudos

I also have a case open for this (65777)


So far 9.1 has been a disaster. This product was not ready for release and it has too many bugs! Is there a 9.1 SP release date yet. 9.1 has made my system unusable as it stops everynight until we reboot it in the morning.

0 Kudos

9.1 has made my system unusable as it stops everynight until we reboot it in the morning.
 

I see from the ticket that Support told that it was probably database maintenance, which is a good hypothesis.  For users with lots of orphan devices in the database (from removing nodes or interfaces), the db cleanup is impacting performance.  In SP1 (which *should* be out in a few days), we handle the cleanup in more gently.

0 Kudos

 I'm also having the same problem....on my SLX installation I have over 3000 interfaces.  The DB maint is impacting disk performance so badly that we are missing polls creating hundreds of false positives.....I'm all for getting rid of the old data but this is brutal. 

What will happen if I temporarily rename the DBMaint.exe file so it can't run.....will it just error or halt the system.

0 Kudos

 SP1 should help this problem, and it will be out shortly.

0 Kudos

 Just applied SP1 and it fixed the issue!   The database maintence completes MUCH quicker now, back to normal!  What did SP1 change to fix this?

0 Kudos

Me too SP1 seems to have fixed all my troubles for now. The support were actually very good (Joe Hanly very professional indeed). Its a shame 9.1 got released in that state, but all's well that ends well!

0 Kudos

Denny,


Has anyone else reported the same disk performance issues that we had?  Renaming the DBMaint file was the only thing I could think of to stop the performance hits.  I checked the table and there are over 6 million entries in that table most of which probably belong to interface that no longer exist.


Thanks,

0 Kudos

We are seeing the same issues. I ran the DB Maint manually last week and let it finish (about 30 hours). After that, a manual run only took about 30 minutes.

However, this weekend, it started at 12:30 AM Saturday and I had to kill it this morning at 4:30 AM to get the main polling engine to resume collecting data at more than 4 hour intervals - 52 hours and it still was not done. 

0 Kudos

CTSChap,


If you rename the file....it can't run.  I tested and SW only creates an error saying that the DB Maint application is not installed.  This way it will not start automatically tonight.

0 Kudos