Getting "Database maintenance ... has finished with errors." alert in the console (NPM, SAM), and this when clicking on it:
swdebugMaintenance.log has a bunch of these:
2023-04-13 02:15:05,111 [1] ERROR SolarWinds.Data.DatabaseMaintenance.TimeSeriesTableHandler - Retention interval settings not specified
... and also these:
2023-04-13 02:15:44,557 [9] ERROR SolarWinds.Orion.LogMgmt.Maintenance.LogMaintenancePlugin - Failed to run partition maintenance in Log , Attempt #1, ErrorNumber 596
System.Data.SqlClient.SqlException (0x80131904): Cannot continue the execution because the session is in the kill state.
A severe error occurred on the current command. The results, if any, should be discarded.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at SolarWinds.Orion.Common.SqlHelper.ExecuteNonQuery(SqlCommand command, SqlConnection connection, SqlTransaction transaction)
at SolarWinds.Orion.LogMgmt.Common.Storage.LogStorageController.ExecuteNonQuery(SqlCommand command, Action`1 connectionModifier)
at SolarWinds.Orion.LogMgmt.Maintenance.LogMaintenancePlugin.RunStoredProcedure(String procedureName, SqlParameter[] parameters)
at SolarWinds.Orion.LogMgmt.Maintenance.LogMaintenancePlugin.RunWithRetry(Action action, Int32 maxAttempts, TimeSpan retryDelay, String errorMessage, List`1 retryableSqlErrors)
ClientConnectionId:43336942-17cc-42f7-8fcc-b1c392aab36f
Error Number:596,State:1,Class:21
2023-04-13 02:15:44,604 [9] ERROR SolarWinds.Data.DatabaseMaintenance.MaintenanceEngineHelper - Action failed for plugin [SolarWinds.Orion.LogMgmt.Maintenance.LogMaintenancePlugin]
System.Data.SqlClient.SqlException (0x80131904): Cannot continue the execution because the session is in the kill state.
A severe error occurred on the current command. The results, if any, should be discarded.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at SolarWinds.Orion.Common.SqlHelper.ExecuteNonQuery(SqlCommand command, SqlConnection connection, SqlTransaction transaction)
at SolarWinds.Orion.LogMgmt.Common.Storage.LogStorageController.ExecuteNonQuery(SqlCommand command, Action`1 connectionModifier)
at SolarWinds.Orion.LogMgmt.Maintenance.LogMaintenancePlugin.RunStoredProcedure(String procedureName, SqlParameter[] parameters)
at SolarWinds.Orion.LogMgmt.Maintenance.LogMaintenancePlugin.RunWithRetry(Action action, Int32 maxAttempts, TimeSpan retryDelay, String errorMessage, List`1 retryableSqlErrors)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at SolarWinds.Orion.LogMgmt.Maintenance.LogMaintenancePlugin.RunWithRetry(Action action, Int32 maxAttempts, TimeSpan retryDelay, String errorMessage, List`1 retryableSqlErrors)
at SolarWinds.Orion.LogMgmt.Maintenance.LogMaintenancePlugin.EnforceRetentionPeriods()
at SolarWinds.Orion.LogMgmt.Maintenance.LogMaintenancePlugin.PreMaintenace(EntityMaintenanceDetails details)
at SolarWinds.Data.DatabaseMaintenance.MaintenanceEngineHelper.<>c__DisplayClass6_0.b__0(IMaintenancePlugin p)
ClientConnectionId:43336942-17cc-42f7-8fcc-b1c392aab36f
Error Number:596,State:1,Class:21
2023-04-13 02:16:15,395 [1] ERROR SolarWinds.Data.DatabaseMaintenance.Reporter - [Failure] 0.69s Plugin action - LogMaintenancePlugin->PreMaintenace for Orion.LogMgmt
Message:
--------------------------------------------
Cannot continue the execution because the session is in the kill state.
A severe error occurred on the current command. The results, if any, should be discarded.
--------------------------------------------
Anyone knows what the latter two are about? (The first one, "Retention interval settings not specified", is the infamous database maintenance bug discussed in "Database Maintenance Error"?)
Running SolarWinds Platform, NTA, NPM, SAM: 2023.1.0.
On a related note: what is the best way to monitor these logs so that I could alert on errors and warnings, and run basic analytics? (E.g. "when did this specific error start and how often do they happen?") Can SolarWinds do it as part of SAM w/o too much pain or $$$, or do I need to get something else?
Thanks!
