3 Replies Latest reply on May 29, 2018 8:32 PM by mesverrum

    Database Corruption Recovery? What did you all do?

    kbuska

      We are in the midst of trying to recover from a corrupted database. We are working a bit with Solarwinds but I wanted to throw a discussion out there and see if anyone had any thoughts on what worked for them. Maybe a KB article, or a 3rd party software or perhaps a services contractor?

       

      We have backups, the question is, when did the corruption start occurring? We thought it was 1/22 so yesterday morning we restored the whole VM to 1/21 image and then I ran a database maintenance which completed successfully. At 2:36 AM this morning we get the following message in the event log and Solarwinds stops working. Solarwinds think our local backup plan is the blame.

       

      Thank for your feedback. below is a few of the errors we are getting if you have any thoughts, I would love your feedback.

       

      Error from last night:


      During undoing of a logged operation in database 'SolarWindsOrion', an error occurred at log record ID (3898809:62572:66).


      Errors during a DB Check:


      Msg 8904, Level 16, State 1, Line 1

      Extent (1:3433248) in database ID 8 is allocated by more than one allocation object.

      Msg 8913, Level 16, State 1, Line 1

      Extent (1:3433248) is allocated to 'GAM' and at least one other object.

      Msg 8904, Level 16, State 1, Line 1

      Extent (1:3433256) in database ID 8 is allocated by more than one allocation object.

      Msg 8913, Level 16, State 1, Line 1

      Extent (1:3433256) is allocated to 'GAM' and at least one other object.

      Msg 8904, Level 16, State 1, Line 1

      Extent (1:3566144) in database ID 8 is allocated by more than one allocation object.

      Msg 8913, Level 16, State 1, Line 1

      Extent (1:3566144) is allocated to 'GAM' and at least one other object.

      Msg 8905, Level 16, State 1, Line 1

      Extent (1:5978288) in database ID 8 is marked allocated in the GAM, but no SGAM or IAM has allocated it.

      Msg 8905, Level 16, State 1, Line 1

      Extent (1:5978312) in database ID 8 is marked allocated in the GAM, but no SGAM or IAM has allocated it.

      Msg 8904, Level 16, State 1, Line 1

      Extent (1:15987336) in database ID 8 is allocated by more than one allocation object.

      Msg 8913, Level 16, State 1, Line 1

      Extent (1:15987336) is allocated to 'GAM' and at least one other object.

       

      Some errors during the repair:

       

      Msg 1505, Level 16, State 1, Line 1

      The CREATE UNIQUE INDEX statement terminated because a duplicate key was found for the object name 'dbo.ResponseTime_Detail_20160122' and the index name 'PK_ResponseTime_Detail_20160122'. The duplicate key value is (Jan 21 2016  2:48PM, 218, 0).

      The statement has been terminated.

      Msg 1505, Level 16, State 1, Line 1

      The CREATE UNIQUE INDEX statement terminated because a duplicate key was found for the object name 'dbo.DiscoveredInterfaces' and the index name 'PK_DiscoveredInterface'. The duplicate key value is (16, 91, 1, 0).

      Msg 1505, Level 16, State 1, Line 1

      The CREATE UNIQUE INDEX statement terminated because a duplicate key was found for the object name 'dbo.DiscoveredNodes' and the index name 'PK_DiscoveredNodes'. The duplicate key value is (16, 96).

      Msg 1505, Level 16, State 1, Line 1

      The CREATE UNIQUE INDEX statement terminated because a duplicate key was found for the object name 'dbo.ResponseTime_Detail_20160122' and the index name 'IX_NodeID'. The duplicate key value is (8, Jan 21 2016  2:48PM, 0).

       

      Repair: The page (1:5979048) has been deallocated from object ID 275584120, index ID 1, partition ID 72057594112704512, alloc unit ID 72057594124173312 (type In-row data).

      Repair: The page (1:5979049) has been deallocated from object ID 275584120, index ID 1, partition ID 72057594112704512, alloc unit ID 72057594124173312 (type In-row data).

      Repair: The page (1:5979050) has been deallocated from object ID 275584120, index ID 1, partition ID 72057594112704512, alloc unit ID 72057594124173312 (type In-row data).

      Repair: The page (1:5979051) has been deallocated from object ID 275584120, index ID 1, partition ID 72057594112704512, alloc unit ID 72057594124173312 (type In-row data).

       

      Table error: Object ID 325588944, index ID 1, partition ID 72057594676510720, alloc unit ID 72057594701611008 (type In-row data). Index node page (1:5982829), slot 71 refers to child page (1:5979155) and previous child (1:5979153), but they were not encountered.

              The error has been repaired.

       

      CHECKDB found 14 allocation errors and 527 consistency errors in database 'SolarWindsOrion'.

      CHECKDB fixed 14 allocation errors and 527 consistency errors in database 'SolarWindsOrion'.