1 Reply Latest reply on Mar 25, 2019 6:34 AM by justthwackit

    Reccommendations engine kept stopping [Solved]

    RichardLetts

      In my newly upgraded HA environment we had an issue failing backwards on some of our servers -- the Recommendations engine kept stopping with logfiles that look like this:

       

      *** SolarWinds Recommendations Service v1.2.0.4674, .Net Runtime v4.0.30319 ***
      2018-05-22 07:17:39,271 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.InformationService.Contract2.InfoServiceProxy - Non-used connection was opened. Information for developers. No impact on product functionality. See verbose log for more details.
      2018-05-22 07:17:40,083 [SolarWindsRecommendations.OnStart] WARN  SolarWinds.Orion.Common.Helpers.CryptoHelper - Unable to load SolarWinds.Orion.Security.dll from execution directory.
      System.IO.FileNotFoundException: The system cannot find the file specified. (Exception from HRESULT: 0x80070002)
         at System.Reflection.RuntimeAssembly.nLoadFile(String path, Evidence evidence)
         at System.Reflection.Assembly.LoadFile(String path)
         at SolarWinds.Orion.Common.Helpers.CryptoHelper..cctor()
      2018-05-22 07:17:40,083 [SolarWindsRecommendations.OnStart] WARN  SolarWinds.Orion.Common.Helpers.CryptoHelper - Trying to load SolarWinds.Orion.Security.dll from Orion install directory...
      2018-05-22 07:17:40,099 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.Orion.Common.ConnectionStringProvider - SqlWinAuthImpersonate=true
      2018-05-22 07:17:40,427 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.InformationService.Contract2.InfoServiceProxy - Non-used connection was opened. Information for developers. No impact on product functionality. See verbose log for more details.
      2018-05-22 07:17:40,911 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.Recommendations.ServiceContainer.ServiceManager - DataProviderService started successfully.
      2018-05-22 07:17:41,130 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.Recommendations.ServiceContainer.ServiceManager - EngineService started successfully.
      2018-05-22 07:17:41,568 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.Recommendations.Orchestration.Notifications.SubscriptionService - Starting subscription service
      2018-05-22 07:17:41,614 [SolarWindsRecommendations.OnStart] INFO  SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider - Orion.PubSub.PubSubOverMessageBusEnabled == false, so not attempting message bus connection. (Change in /orion/admin/advancedconfiguration/global.aspx)
      2018-05-22 07:17:42,599 [SolarWindsRecommendations.OnStart] ERROR SolarWinds.Recommendations.Orchestration.Notifications.SubscriptionService - Adding swis subscriptions failed. Is swis running?
      System.ServiceModel.FaultException`1[SolarWinds.InformationService.Contract2.InfoServiceFaultContract]: Create failed, check fault information.
      Object reference not set to an instance of an object. (Fault Detail is equal to InfoServiceFaultContract [ System.NullReferenceException: Object reference not set to an instance of an object.
         at SolarWinds.InformationService.ChangeBroker.PersistentNotificationServiceForwarder.GetSubscriptionKeyFromUri(String subscriptionId)
         at SolarWinds.InformationService.ChangeBroker.PersistentNotificationServiceForwarder.Subscribe(Subscription subscription, NetworkCredential credentials, String[] tags)
         at SolarWinds.InformationService.ChangeBroker.SubscriptionProvider.SubscriptionProvider.CreateSubscription(IDictionary`2 columns)
         at SolarWinds.InformationService.Addons.DataProvider.<Create>d__61.MoveNext()
         at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
         at SolarWinds.InformationService.Core.CrudProcessor.<CreateLocal>d__21.MoveNext()
         at SolarWinds.InformationService.Core.CrudProcessor.<CreateInternal>d__18.MoveNext()
         at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
         at SolarWinds.InformationService.Core.CrudPro...).
      2018-05-22 07:17:42,599 [SolarWindsRecommendations.OnStart] FATAL SolarWinds.Recommendations.Service.RecommendationsService - Unhandled exception caught.
      System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ServiceModel.FaultException`1[SolarWinds.InformationService.Contract2.InfoServiceFaultContract]: Create failed, check fault information.
      Object reference not set to an instance of an object.
      
      ...

       

      The FATAL error is like 27, Caused by the ERROR on line 15, but the hint is on line 14 as to what is going wrong. for some reason PubSubOverMessageBusEnabled was set to false on a random selection of servers.

       

      to check this visit the advanced settings page (which is where the URL us trying to send you at ... https://your-server/Orion/Admin/AdvancedConfiguration/ServerSpecific.aspx

       

      Check the settings and make sure they are set the same on all of your servers... I can't imagine a case when not setting these all the same would be the right thing to do

       

      Hope this helps someone