12 Replies Latest reply on Jan 25, 2008 8:09 AM by josh.clark

    APM Portal Error - login fails

    dedwards

       APM Beta was working fine but today it will not authenticate - I have bounced the web service and app pool but still get the same error as below.

      The only thing that has changed is some additional nodes have been added to NPM - the rest of the web site is working fine just APM 

      An error has occurred with the Orion website.

      Additional Information

      System.ServiceModel.Security.SecurityNegotiationException: The server has rejected the client credentials. ---> System.Security.Authentication.InvalidCredentialException: The server has rejected the client credentials. ---> System.ComponentModel.Win32Exception: The logon attempt failed
      --- End of inner exception stack trace ---
      at System.Net.Security.NegoState.ProcessAuthentication(LazyAsyncResult lazyResult)
      at System.Net.Security.NegotiateStream.AuthenticateAsClient(NetworkCredential credential, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel)
      at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeInitiator.OnInitiateUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)
      --- End of inner exception stack trace ---

      Server stack trace:
      at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeInitiator.OnInitiateUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)
      at System.ServiceModel.Channels.StreamSecurityUpgradeInitiatorBase.InitiateUpgrade(Stream stream)
      at System.ServiceModel.Channels.ConnectionUpgradeHelper.InitiateUpgrade(StreamUpgradeInitiator upgradeInitiator, IConnection& connection, ClientFramingDecoder decoder, IDefaultCommunicationTimeouts defaultTimeouts, TimeoutHelper& timeoutHelper)
      at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreamble(IConnection connection, ArraySegment`1 preamble, TimeoutHelper& timeoutHelper)
      at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
      at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
      at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
      at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
      at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
      at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
      at System.ServiceModel.Channels.CommunicationObject.Open()

      Exception rethrown at [0]:
      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
      at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
      at System.ServiceModel.ICommunicationObject.Open()
      at SolarWinds.APM.Common.APMBusinessLayerProxy.Connect()
      at SolarWinds.APM.Common.APMBusinessLayerProxy..ctor(String host, String port, HandleBusinessLayerException exceptionDelegate)
      at SolarWinds.APM.Web.BusinessLayerFactory.Create()
      at SolarWinds.APM.Web.ApmApplication.GetAllObjects()
      at Orion_APM_Resources_Summary_NoApplicationDefined.OnInit(EventArgs e)
      at System.Web.UI.Control.InitRecursive(Control namingContainer)
      at System.Web.UI.Control.AddedControl(Control control, Int32 index)
      at System.Web.UI.ControlCollection.Add(Control child)
      at Orion_ResourceContainer.ResourcePlaceHolder_DataBind(Object sender, EventArgs e)
      at System.Web.UI.Control.OnDataBinding(EventArgs e)
      at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
      at System.Web.UI.Control.DataBind()
      at System.Web.UI.Control.DataBindChildren()
      at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
      at System.Web.UI.Control.DataBind()
      at System.Web.UI.WebControls.Repeater.CreateItem(Int32 itemIndex, ListItemType itemType, Boolean dataBind, Object dataItem)
      at System.Web.UI.WebControls.Repeater.CreateControlHierarchy(Boolean useDataSource)
      at System.Web.UI.WebControls.Repeater.OnDataBinding(EventArgs e)
      at System.Web.UI.WebControls.Repeater.DataBind()
      at System.Web.UI.Control.DataBindChildren()
      at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
      at System.Web.UI.Control.DataBind()
      at System.Web.UI.Control.DataBindChildren()
      at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
      at System.Web.UI.Control.DataBind()
      at Orion_ResourceContainer.DataBind()
      at Orion_APM_Summary.OnInit(EventArgs e)
      at System.Web.UI.Control.InitRecursive(Control namingContainer)
      at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
        • Re: APM Portal Error - login fails
          josh.clark

          1.  Are the website (IIS) and Orion running on the same machine?

          2. This is a Windows authentication error.  Did anything change with the user IIS runs under?  Did you change which user the Orion services run under?

            • Re: APM Portal Error - login fails
              dedwards


              1.  Are the website (IIS) and Orion running on the same machine?

              2. This is a Windows authentication error.  Did anything change with the user IIS runs under?  Did you change which user the Orion services run under?

               

               

              1. Yes both running on the same server (SQL is running on additional poller) 

              2.No changes have been made to the server at all since the install of APM, NPM is the only website running on this as well.

               

              Just found this in the Event log on the server:

              Event Type:    Error
              Event Source:    SolarWinds.APM.BusinessLayer
              Event Category:    None
              Event ID:    1024
              Date:        20/12/2007
              Time:        13:45:36
              User:        N/A
              Computer:    xxxxxxxxxx
              Description:
              Unhandled Exception caught in APM Service Engine startup. Cannot listen on pipe name 'net.pipe://localhost/orion/apm/businesslayer' because another pipe endpoint is already listening on that name.

              For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

                • Re: APM Portal Error - login fails
                  josh.clark

                  Try stopping the SolarWinds Orion Module Engine service.  Wait for 20 seconds or so and then start the service again. 

                    • Re: APM Portal Error - login fails
                      dedwards

                       Tried that (in fact after that did not work I stopped all Solarwinds services then started them up again) but APM still gave the same error - I have now rebooted the server and it is working again...

                        • Re: APM Portal Error - login fails
                          dedwards

                           Just looking back through the monitors and it would appear that all of them stopped working at about 10pm last night (all graphs are blank after this) so I am now looking back through all the logs for anything around that time.

                            • Re: APM Portal Error - login fails
                              dedwards

                               OK I've not found anything in the logs yet but it has happened again - nothing has changed on the server at all and it was working last night but this morning APM gives exactly the same error - initially the site worked for 3 days without issue but has not lasted 24 hours since the reboot yesterday. I have tried bouncing services again but this has not resolved it - I have not rebooted yet as I want to try to find the actual issue. Any ideas?

                                • Re: APM Portal Error - login fails
                                  josh.clark

                                  Can you run C:\Program Files\Solarwinds\Orion\SolarwindsDiagnostics.exe and upload the zip file to ftp://devftp.solarwinds.net.

                                   I'll send the credentials in a private message.

                                    • Re: APM Portal Error - login fails
                                      josh.clark

                                      There seem to be two different problems going on.  

                                      1. Even though your website and Orion server are on the same box, we are connecting as if you are remote.  If the website determines it is local to the Orion Module Engine, it will communicate using named pipes (they're faster and more secure).  If we are not local, we will use TCP.  The error you are getting shows that we are using TCP.  This shouldn't matter since both should work
                                      2. The TCP communication uses Windows authentication to secure the communications between the website and the Module Engine.  For some reason this authentication is failing.  Under a clean install, Windows authentication will work just fine locally, so something must be different for your site.

                                       
                                       I'd like to try to determine #1 first. That might lead us to the problem with #2.


                                      First, we need to turn on debug logging for the website. 

                                      1. Open up C:\Inetpub\SolarWinds\web.config in notepad.
                                      2. Search for "<threshold value="WARN" />" (without the quotes)
                                      3. Replace "WARN" with "DEBUG"

                                      Now lets view the APM website when the error occurs and then again when the error doesn't occur (Its ok to reboot to fix the error)

                                      Can you post, the file C:\Inetpub\SolarWinds\OrionWeb.log?  It should have some details about what happened.  Also, if you could note the approximate time you saw both the success and failure, it would help me find the issue in the logs.

                                       
                                       


                                       

                                       

                                       

                                       

                                       

                                        • Re: APM Portal Error - login fails
                                          josh.clark

                                          Did this problem ever get resolved or are you still getting the error on your website? 

                                            • Re: APM Portal Error - login fails
                                              dedwards

                                               Still an issue but have noticed it is intermittent - sometimes it works sometimes not, occasionally even a reboot does not fix it. As soon as I get chance I will try and replicate again and post the debug logs this week. I have also had the issue of polling stopping and graphs not updating etc. but generally a bounce of the job engine fixes this.

                                                • Re: APM Portal Error - login fails
                                                  dedwards

                                                  OK will try and post logs when I can get this working again (re-installed and forgot to backup logs doh!) but when it works it is using pipes when it doesn't it is the tcp connection - in the debug of a failed connection I get:

                                                  2008-01-24 21:37:29,750 [1] DEBUG SolarWinds.APM.Common.APMBusinessLayerProxy - Connecting
                                                  2008-01-24 21:37:29,750 [1] DEBUG SolarWinds.APM.Common.APMBusinessLayerProxy - Channel created to net.tcp://CN-ULUY1F:17605/orion/apm/businesslayer
                                                  2008-01-24 21:37:29,906 [1] WARN  SolarWinds.APM.Common.APMBusinessLayerProxy - An error occured when connecting to the communication channel.
                                                  2008-01-24 21:37:29,906 [1] ERROR ASP.global_asax - Application_Error
                                                  System.Web.HttpUnhandledException: Exception of type 'System.Web.HttpUnhandledException' was thrown. ---> System.ServiceModel.Security.SecurityNegotiationException: The server has rejected the client credentials. ---> System.Security.Authentication.InvalidCredentialException: The server has rejected the client credentials. ---> System.ComponentModel.Win32Exception: The logon attempt failed
                                                     --- End of inner exception stack trace ---

                                                  etc. the channel is being created to the database server (also running an additional poller) though not the local machine so that would explain when it thinks it is remote that it fails I guess