Polling Engine Down - Database Sync not occurring

We're experiencing a problem with an installation of NPM 11.5.3, NTA 4.1.2 and SAM 6.2.3. Following an upgrade, we've found that the primary (only) poller is down. Database updates are occurring as normal, but the Last Database Sync is increasing and the KeepAlive value in the AllEngines table has not updated since the services started.

We have looked in the Windows Event Logs and found an event which occurs each time the services start, ID 1024: "Unhandled Exception caught in Core Service Engine startup. Specified cast is not valid."

We have tried:

Checking the time between the database and application server is the same
Restarting services
Restarting the server
Running the config wizard (repairing all components)
Running permission checker
Tested granting "Everyone" access to SolarWinds directories to rule out permissions
Uninstalling and reinstalling everything
Recreating application certificates
Forcing business layer plugins to start in different processes

We've even built a new VM and tested it with a newly created database (on the same instance as the live is running). We still got the error though.

We've checked the collation, CLR and compatibility of the database. There are no error messages in the event logs for SQL.

Has anyone seen something like this before?

Find more posts tagged with

invalid

last database sync

11.5.3

npm

cast

keepalive

Accepted answers

julionavarro1414

I was told to :

1.     Stop all Orion Service via the Orion Service Manager.
     2. Go to C:\ProgramData\SolarWinds\Installers
     3. Run CoreInstaller and choose repair.
     4. Complete Configuration Wizard.

And it fixed the issue BUT I don't feel comfortable because I had to do it twice :-(

All comments

julionavarro1414

Hi!

We are getting the same error but we haven't made any changes (upgrade,etc) in the environment.
It started today.

We are running :

Orion Platform 2015.1.2, SAM 6.2.1, DPA 9.2.0, IVIM 2.1.0, QoE 2.0, NPM 11.5.2, NTA 4.1.1

I opened a ticket "system down" but no response from SW yet.

I have seen that the data is being collected but still it could be worst under the hood.

The only way to fix it is restarting the Orion Module but still 10 minutes later it come back.

Did you open a ticket with SW?

neomatrix1217

What does it say in the log file called businesslayerhost log it's located in c:\programdata\solarwinds\logs\orion go all the way to the bottom is ther anything there that keeps repeating?

julionavarro1414

Here is the log:

*** Assembly SolarWinds.Orion.Core.Common, Version=2015.1.25300.8291, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

2016-02-25 18:12:20,436 [Main] INFO SolarWinds.Orion.Core.Common.CoreLicense - Package type is "Full". Core is not licesnsed on the main server. Assume the license is valid.

*** Assembly SolarWinds.Common, Version=2015.1.25300.8197, Culture=neutral, PublicKeyToken=null, .NET version v2.0.50727 ***

*** Assembly SolarWinds.BusinessLayerHost, Version=2015.1.25300.8197, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

*** Assembly SolarWinds.Orion.Common, Version=2.0.0.0, Culture=neutral, PublicKeyToken=316e86daa515e407, .NET version v4.0.30319 ***

*** Assembly SolarWinds.InformationService.Contract2, Version=2015.1.1.6134, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***

2016-02-25 18:12:21,059 [8] INFO SolarWinds.InformationService.Contract2.InfoServiceProxy - Non-used connection was opened. Information for developers. No impact on product functionality. See verbose log for more details.

2016-02-25 18:12:21,106 [8] INFO SolarWinds.InformationService.Contract2.InfoServiceProxy - Non-used connection was opened. Information for developers. No impact on product functionality. See verbose log for more details.

2016-02-25 18:12:21,129 [8] INFO SolarWinds.InformationService.Contract2.InfoServiceProxy - Non-used connection was opened. Information for developers. No impact on product functionality. See verbose log for more details.

2016-02-25 18:12:21,197 [8] ERROR SolarWinds.BusinessLayerHost.PluginManager - Plugin "D:\Program Files (x86)\SolarWinds\Orion\SolarWinds.Orion.Core.BusinessLayer.dll - Copy" could not be loaded correctly

System.IO.FileNotFoundException: Could not load file or assembly 'file:///D:\Program Files (x86)\SolarWinds\Orion\SolarWinds.Orion.Core.BusinessLayer.dll - Copy' or one of its dependencies. The system cannot find the file specified.

File name: 'file:///D:\Program Files (x86)\SolarWinds\Orion\SolarWinds.Orion.Core.BusinessLayer.dll - Copy'

at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)

at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)

at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection, Boolean suppressSecurityChecks)

at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark)

at System.Reflection.Assembly.LoadFrom(String assemblyFile)

at SolarWinds.BusinessLayerHost.PluginManager.GetPluginTypesFromFile(String dllFile)

WRN: Assembly binding logging is turned OFF.

To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.

Note: There is some performance penalty associated with assembly bind failure logging.

To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

2016-02-25 18:12:21,875 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "Core Business Layer"

2016-02-25 18:12:21,877 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceSeparateProcess - Initialize plugin \Program Files (x86)\SolarWinds\Orion\AgentManagement\SolarWinds.AgentManagement.ServiceCore.dll.config - ProcessID:-1

2016-02-25 18:12:21,877 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceSeparateProcess - Starting process \Program Files (x86)\SolarWinds\Orion\SolarWinds.BusinessLayerHost.exe -config="D:\Program Files (x86)\SolarWinds\Orion\AgentManagement\SolarWinds.AgentManagement.ServiceCore.dll.config" -id=2863df32-4593-4e57-a8be-a42c423fa772 -parent=5868

2016-02-25 18:12:22,226 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "Agent Management Watchdog"

2016-02-25 18:12:22,832 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "APM Business Layer"

2016-02-25 18:12:22,843 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "Asset Inventory Business Layer"

2016-02-25 18:12:23,567 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "Collector Business Layer Plugin"

2016-02-25 18:12:23,579 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "Device Studio Business Layer"

2016-02-25 18:12:24,198 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "DPA Business Layer"

2016-02-25 18:12:24,793 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "DPI Business Layer"

2016-02-25 18:12:24,803 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "Interfaces Business Layer"

2016-02-25 18:12:25,400 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "NPM Business Layer"

2016-02-25 18:12:25,408 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "Orion Improvement"

2016-02-25 18:12:25,417 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "Recommendation Engine Business Layer"

2016-02-25 18:12:26,081 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "Virtualization Business Layer"

2016-02-25 18:12:26,683 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Initializing plugin: "WLHM Business Layer"

2016-02-25 18:12:26,684 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "Core Business Layer"

2016-02-25 18:12:36,551 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceSeparateProcess - Starting plugin \Program Files (x86)\SolarWinds\Orion\AgentManagement\SolarWinds.AgentManagement.ServiceCore.dll.config - ProcessID:16856

2016-02-25 18:12:36,551 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "Agent Management Watchdog"

2016-02-25 18:12:37,132 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "APM Business Layer"

2016-02-25 18:12:43,626 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "Asset Inventory Business Layer"

2016-02-25 18:12:43,691 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "Collector Business Layer Plugin"

2016-02-25 18:12:50,258 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "Device Studio Business Layer"

2016-02-25 18:12:53,195 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "DPA Business Layer"

2016-02-25 18:12:57,378 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "DPI Business Layer"

2016-02-25 18:13:02,556 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "Interfaces Business Layer"

2016-02-25 18:13:05,396 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "NPM Business Layer"

2016-02-25 18:13:08,277 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "Orion Improvement"

2016-02-25 18:13:11,186 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "Recommendation Engine Business Layer"

2016-02-25 18:13:13,809 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "Virtualization Business Layer"

2016-02-25 18:15:29,245 [8] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Starting plugin: "WLHM Business Layer"

neomatrix1217

OK everything looks good there. Your going to need to open a support ticket without diagnostics it's hard to diagnose the only other thing I can suggest is to run a repair on the following installers that are located in the c:\programdata\solarwinds\installers Jobengine, jobengine2 and collector.

Deltona

System.IO.FileNotFoundException: Could not load file or assembly 'file:///D:\Program Files (x86)\SolarWinds\Orion\SolarWinds.Orion.Core.BusinessLayer.dll - Copy' or one of its dependencies.

istefanos

did you run a repair ?