cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 16

Reccommendations engine kept stopping [Solved]

In my newly upgraded HA environment we had an issue failing backwards on some of our servers -- the Recommendations engine kept stopping with logfiles that look like this:

*** SolarWinds Recommendations Service v1.2.0.4674, .Net Runtime v4.0.30319 ***
2018-05-22 07:17:39,271 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.InformationService.Contract2.InfoServiceProxy - Non-used connection was opened. Information for developers. No impact on product functionality. See verbose log for more details.
2018-05-22 07:17:40,083 [SolarWindsRecommendations.OnStart] WARN  SolarWinds.Orion.Common.Helpers.CryptoHelper - Unable to load SolarWinds.Orion.Security.dll from execution directory.
System.IO.FileNotFoundException: The system cannot find the file specified. (Exception from HRESULT: 0x80070002)
   at System.Reflection.RuntimeAssembly.nLoadFile(String path, Evidence evidence)
   at System.Reflection.Assembly.LoadFile(String path)
   at SolarWinds.Orion.Common.Helpers.CryptoHelper..cctor()
2018-05-22 07:17:40,083 [SolarWindsRecommendations.OnStart] WARN  SolarWinds.Orion.Common.Helpers.CryptoHelper - Trying to load SolarWinds.Orion.Security.dll from Orion install directory...
2018-05-22 07:17:40,099 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.Orion.Common.ConnectionStringProvider - SqlWinAuthImpersonate=true
2018-05-22 07:17:40,427 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.InformationService.Contract2.InfoServiceProxy - Non-used connection was opened. Information for developers. No impact on product functionality. See verbose log for more details.
2018-05-22 07:17:40,911 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.Recommendations.ServiceContainer.ServiceManager - DataProviderService started successfully.
2018-05-22 07:17:41,130 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.Recommendations.ServiceContainer.ServiceManager - EngineService started successfully.
2018-05-22 07:17:41,568 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.Recommendations.Orchestration.Notifications.SubscriptionService - Starting subscription service
2018-05-22 07:17:41,614 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider - Orion.PubSub.PubSubOverMessageBusEnabled == false, so not attempting message bus connection. (Change in /orion/admin/advancedconfiguration/global.aspx)
2018-05-22 07:17:42,599 [SolarWindsRecommendations.OnStart] ERROR SolarWinds.Recommendations.Orchestration.Notifications.SubscriptionService - Adding swis subscriptions failed. Is swis running?
System.ServiceModel.FaultException`1[SolarWinds.InformationService.Contract2.InfoServiceFaultContract]: Create failed, check fault information.
Object reference not set to an instance of an object. (Fault Detail is equal to InfoServiceFaultContract [ System.NullReferenceException: Object reference not set to an instance of an object.
   at SolarWinds.InformationService.ChangeBroker.PersistentNotificationServiceForwarder.GetSubscriptionKeyFromUri(String subscriptionId)
   at SolarWinds.InformationService.ChangeBroker.PersistentNotificationServiceForwarder.Subscribe(Subscription subscription, NetworkCredential credentials, String[] tags)
   at SolarWinds.InformationService.ChangeBroker.SubscriptionProvider.SubscriptionProvider.CreateSubscription(IDictionary`2 columns)
   at SolarWinds.InformationService.Addons.DataProvider.<Create>d__61.MoveNext()
   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
   at SolarWinds.InformationService.Core.CrudProcessor.<CreateLocal>d__21.MoveNext()
   at SolarWinds.InformationService.Core.CrudProcessor.<CreateInternal>d__18.MoveNext()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at SolarWinds.InformationService.Core.CrudPro...).
2018-05-22 07:17:42,599 [SolarWindsRecommendations.OnStart] FATAL SolarWinds.Recommendations.Service.RecommendationsService - Unhandled exception caught.
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ServiceModel.FaultException`1[SolarWinds.InformationService.Contract2.InfoServiceFaultContract]: Create failed, check fault information.
Object reference not set to an instance of an object.

...

The FATAL error is like 27, Caused by the ERROR on line 15, but the hint is on line 14 as to what is going wrong. for some reason PubSubOverMessageBusEnabled was set to false on a random selection of servers.

to check this visit the advanced settings page (which is where the URL us trying to send you at ... https://your-server/Orion/Admin/AdvancedConfiguration/ServerSpecific.aspx

Check the settings and make sure they are set the same on all of your servers... I can't imagine a case when not setting these all the same would be the right thing to do

pastedImage_3.png

Hope this helps someone

2 Replies
Level 9

Hey man,

just wanted to say thank you for the suggested solution. It fixed my polling engines! I was about to open a system down ticket with support, but I found your solution and it fixed all of my issues.

0 Kudos

And this was the culprit again after migrating to Server 2016 on the polling engines.

I really can't thank you enough!!!

0 Kudos