Attempted to upgrade our NPM/SAM instance to 2020.2.1 (from 2020.2) and it failed spectacularly: web site not accessible: timeouts, "bad password", various other errors. The server is not connected to the Internet so I had to use the offline ISO installer; the server was restarted several times. If I remember correctly the Configuration Wizard is supposed to run after an upgrade - and it did not. Attempting to run it - fails with "package deployment fails'. Attempting to rerun the installer or the hotfix package: no errors, shows the existing modules (NPM, SAM) but the "next" button greyed out, i.e. "nothing to do, as you were".
It seems that the upgrade failed to install or initialize a number of modules.
Submitted a support case to Solarwinds; thought I'd post here as well in case someone runs into a similar issue.
At one point after multiple restarts was able to login, and the UI would display some elements , charts and graphs - but not the menu: "Menu Failure" with "RunQuery failed, check fault information. Object reference not set to an instance of an object.", and the following "error details":
{
"data": {
"message": "An error has occurred.",
"exceptionMessage": "RunQuery failed, check fault information.\nObject reference not set to an instance of an object.",
"exceptionType": "System.ServiceModel.FaultException`1[[SolarWinds.InformationService.Contract2.InfoServiceFaultContract, SolarWinds.InformationService.Contract2, Version=2020.2.1.4275, Culture=neutral, PublicKeyToken=null]]",
"stackTrace": " at SolarWinds.InformationService.InformationServiceClient.InformationServiceCommand.CreateFaultException(Message message)\r\n at SolarWinds.InformationService.InformationServiceClient.InformationServiceCommand.HandleQueryResponse(Message message)\r\n at SolarWinds.InformationService.InformationServiceClient.InformationServiceCommand.ExecuteReader(CommandBehavior behavior)\r\n at SolarWinds.InformationService.InformationServiceClient.InformationServiceCommand.ExecuteDbDataReader(CommandBehavior behavior)\r\n at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior)\r\n at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)\r\n at System.Data.Common.DbDataAdapter.Fill(DataTable[] dataTables, Int32 startRecord, Int32 maxRecords, IDbCommand command, CommandBehavior behavior)\r\n at System.Data.Common.DbDataAdapter.Fill(DataTable dataTable)\r\n at SolarWinds.Orion.Core.Common.InformationService.InformationServiceProxy.<>c__DisplayClass8_0.<Query>b__0(InfoServiceProxy n)\r\n at SolarWinds.Orion.Core.Common.ChannelProxy`1.CallMethodWithPool[TResult](Func`2 fnc, Boolean retryCall)\r\n at SolarWinds.Orion.Core.Common.ChannelProxy`1.Invoke[TResult](Func`2 fnc, ChannelProxyBehavior behavior, String methodName)\r\n at SolarWinds.Orion.Core.Common.ChannelProxy`1.Invoke[TResult](Func`2 fnc, ChannelProxyBehavior behavior)\r\n at SolarWinds.Orion.Core.Common.InformationService.InformationServiceProxy.Query(String query, IDictionary`2 parameters)\r\n at SolarWinds.Orion.Core.Common.InformationService.InformationServiceProxyImpersonate.Query(String query, IDictionary`2 parameters)\r\n at SolarWinds.Orion.Web.Platform.Data.SwisRepository.Query(ISwisQuery query)\r\n at SolarWinds.Orion.Web.Platform.Data.SwisRepository.Query(SwisQuery query)\r\n at SolarWinds.Orion.Web.Services.Data.SwisQueryRepository.Query(String query)\r\n at SolarWinds.Orion.Web.Services.SwisService.Query(String query, Boolean federated)\r\n at SolarWinds.Orion.Api.Host.Controllers.SwisApi2Controller.Query(QueryParam query)\r\n at lambda_method(Closure , Object , Object[] )\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass10.<GetExecutor>b__9(Object instance, Object[] methodParameters)\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Tracing.ITraceWriterExtensions.<TraceBeginEndAsyncCore>d__18`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Tracing.ITraceWriterExtensions.<TraceBeginEndAsyncCore>d__18`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()"
},
"status": 500,
"config": {
"method": "POST",
"transformRequest": [
null
],
"transformResponse": [
null
],
"jsonpCallbackParam": "callback",
"params": {
"swAlertOnError": false,
"lang": "en-us",
"swLogOutOnAuthError": false,
"swLogOnError": [
401,
403,
500,
501,
502,
503,
504,
505,
506,
507,
508,
509,
510
],
"swToastOnError": [
401,
403,
500,
501,
502,
503,
504,
505,
506,
507,
508,
509,
510
]
},
"headers": {
"Accept": "application/json, text/plain, */*",
"Content-Type": "application/json;charset=utf-8",
"X-XSRF-TOKEN": "wKDIXl1gdl2wUzTF25MvvnYrcTa8pG8ay3WuNWu3TMs="
},
"cache": false,
"url": "https://solarwinds.local:443/api2/swis/query",
"data": {
"query": "\n SELECT\n ID\n , ParentID\n , Type\n , Name\n , DisplayName\n , Description\n , Url\n , OpenInNewWindow\n , SortOrder\n FROM Orion.Web.Menu\n ORDER BY SortOrder",
"parameters": null
}
},
"statusText": "",
"xhrStatus": "complete"
}Attempting to run "Orion Wb Console" locally on the server:
---------------------------
Orion Network Performance Monitor
---------------------------
Exception of type 'SolarWinds.ApiProxyFactory.TransientApiProxyException' was thrown.
Troubleshooting Steps
1. Confirm that the SolarWinds Orion Module Engine service is running, as shown in the following steps:
1. Click Start > Administrative Tools > Services.
2. Right-click SolarWinds Orion Module Engine, and then click Start.
2. Your antivirus may be blocking some necessary ports.
3. If the SolarWinds Orion Module Engine service is already running, contact SolarWinds Support.
4. Refresh the Orion Web Console browser.
5. If the Orion Web Console does not display, contact SolarWinds Support.
---------------------------
OK
---------------------------
"Database Maintenance" gets cancelled about 20% into it with no action on my part.
"SolarWinds Active Diagnostics" - a number of critical issues including:
- Connect to Messaging Service (Agent Infrastructure)
Pinging the messaging service has failed. The Agent Management Service is probably offline or the messaging infrastructure does not work correctly. Check the status of the Orion Module Engine service. If it is stopped, start the service. Otherwise try to restart the service. For Additional Polling Engines, check whether all prerequisites of the Agent Management Service are properly installed. - Nested groups test (Groups)
Diagnostic failed: RunQuery failed, check fault information. Entity Orion.Groups does not contain requested property ContainerID See KB MT2582. - Check if Database Maintenance works properly (Orion Database health) Database Maintenance wasn't able to complete.
Following errors were detected:
- 2020-11-28 23:31:45 : Failed to connect to broker [redacted], port 5671, vhost / -> RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> System.AggregateException: One or more errors occurred. ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it [redacted]:5671...
...
Please move Database Maintenance log files into another folder and rerun Database Maintenance. If the issue persists please investigate the new log file for details.
Please investigate these errors. Find out more on Knowledge Base. See KB MT99308. - Check SWIS to SQL custom properties integrity (Orion Database health)
- Diagnostic failed: RunQuery failed, check fault information.
Cannot resolve property EntityName See KB MT91758.
Check IP restrictions (Website Health) Orion website isn't found. Please run Config wizard and rebuild the site. See KB MT7646.
Check site bindings (Website Health) Orion website isn't found. Please run Config wizard and rebuild the site. See KB MT7646.
Check site certificates (Website Health) Orion website isn't found. Please run Config wizard and rebuild the site. See KB MT7646.
Verify the Maximum Worker Processes setting value for SolarWinds Orion Application pool is set to 1 (Website Health) Diagnostic failed: Filename: \\?\C:\Windows\system32\inetsrv\config\applicationHost.config
Error: Unrecognized configuration path 'MACHINE/WEBROOT/APPHOST/SolarWinds NetPerfMon'
See KB 1876. - .... etc...
