8 Replies Latest reply on Oct 21, 2015 2:53 PM by james.w.troy

    Problem when using "CreateApplication" verb with SAM

    james.w.troy

      Dear Sir or Madam -

       

      We are using the "CreateApplication" verb in Powershell to add an application in SAM (version 6.0).  The application is created successfully.   The script then pauses for 7 seconds, then creates the next application, etc.  In total, 530 applications were created.  Each application has two SNMP Monitor components.

       

      The first few applications that were created worked fine, but the majority of the components were hung in the  "Initial poll in progress"  state.  Since we've encountered that problem before, we used the procedure provided by support to correct the issue (re-install CollectorInstaller.msi, JobEngine.msi, and JobEngine.v2.msi).   After following that procedure, all applications were working fine.

       

      How can we avoid this problem ?   Should we wait longer after creating each application ?   If so, how long ?

       

      Thank you.

       

      James Troy

        • Re: Problem when using "CreateApplication" verb with SAM
          derhally

          Let me take a look at this and get back to you.

          • Re: Problem when using "CreateApplication" verb with SAM
            derhally

            Any chance you can provide diagnostics when the issue occurs?

             

            If it does happen next time, you shouldn't have to re-install .    You can just restart the Collector service and Orion BusinessLayer Host.    I would try restarting the Orion BusinessLayer host first and see if that solves the issue.

              • Re: Problem when using "CreateApplication" verb with SAM
                james.w.troy

                Hello Derhally -

                 

                I'm trying to find a way to add a large number of applications to SAM and avoid the hung state (Initial poll in progress).   Rather than getting into the hung state and then trying to recover from it.

                 

                First, I ran tests in which I added just four applications and then watched their status manually to see when they would transition from  "unknown"  status to  "up".   In one case, that took about 23 minutes !   I don't understand why it takes so long for an application with one SNMP component to initialize itself.

                 

                Now, I've changed the code so that after an application is added, I run a SWIS query to get its status.  The query is repeated every minute.  When the status transitions to "up", the script will continue and add the next application.   For some reason, right after adding the application, my SWIS query returns no data at all.  It is a very simple query -

                 

                          SELECT FullyQualifiedName, Status FROM Orion.APM.Application where ApplicationID = nnnn

                 

                However, I can run SWQL and see the entry for the application in Orion.APM.Application.   So why can't my Powershell script retrieve the same data ?    Anyway, it is finally able to get the data, but it may take anywhere from 3 to 37 minutes.

                 

                Is there a way to add a large number of applications, and not have them be hung, and not have to wait such a long time to add each one ?

                 

                We've bought four polling engines so that we could handle a large number of applications and components.   But the intial setup is proving to be quite challenging.

                 

                Thank you.

                James Troy

                  • Re: Problem when using "CreateApplication" verb with SAM
                    derhally

                    Hi James,

                     

                    Just to be clear, you are saying the powershell script is not returning the same data as running the query in SWQL studio?

                     

                    Have you tried calling the PollNow verb after you create the application.  The PollNow takes the application ID (integer) of the application as an argument.

                     

                    Zeid

                      • Re: Problem when using "CreateApplication" verb with SAM
                        james.w.troy

                        Hello Zeid / Derhally -

                         

                        Right after a new application has been added, this Powershell SWQL query usually does not return any data -

                         

                                  SELECT FullyQualifiedName, Status FROM Orion.APM.Application where ApplicationID = nnnn

                         

                        If I run SWQL and have it return all rows from  "Orion.APM.Application" , I see the row for the application that was just added.   In the meantime, my script is retrying the query every minute and ultimately, it will return data from the table.   This behavior was surprising to me, but it does not prevent me from accomplishing the desired goal (i.e. waiting to confirm that the application is up).

                         

                        I have not tried  "PollNow" , because I thought that SAM automatically tries to poll newly-added components.  I can try this next week and let you know if it makes any difference.

                         

                        Right now, newly-added applications require about 6 minutes on average to reach the state  "up".   Since it is so slow, I am going to try running multiple scripts to add applicaitons in parallel.   This approach may improve the performance.   But its also seems possible that if I run two instances of the script at once, they may each take twice as long to finish.   It depends on where the bottleneck is.

                         

                        Previously, we used a different script to add CPU and memory pollers for the same nodes.  That script added the two pollers for a node, then paused for six seconds, then added the two pollers for the next node, etc.  We never encountered any problems doing that for 230 nodes at a time.  This is why I was so surprised to have a problem adding applications that only have SNMP components in them.   An SNMP component retrieving one OID is conceptually no different from a poller returning one OID (or a few OIDs).

                         

                        Thank you.

                        James Troy

                        • Re: Problem when using "CreateApplication" verb with SAM
                          james.w.troy

                          Hello Zeid / Derhally -

                           

                          Invoking the  "Pollnow"  verb after creating the application has no effect on the symptoms that I reported earlier.   It is still necessary to wait for many minutes before the application is up.

                           

                          Thank you.

                          James Troy

                    • Re: Problem when using "CreateApplication" verb with SAM
                      njoylif

                      Hello james.w.troy, did you ever get resolution to this?

                        • Re: Problem when using "CreateApplication" verb with SAM
                          james.w.troy

                          Hello njoylif -

                           

                          As you can see from this post, no alternate resolution was ever provided.   Our method of doing scripted  "bulk adds"  of Orion SAM applications always waits for the application to be  "up"  before continuing to add the next application.   As far as I can determine, this is the only way to ensure we won't have applications stuck forever in the dreaded  "Initial poll in progress"  state.

                           

                          From additional experience, I have determined that the amount of time you must wait for the application to be  "up"  depends on the polling frequency you chose.   If you are polling every 300 seconds, you may have to wait up to 300 seconds for the application to be "up".   But, if you are polling every 3600 seconds, you may have to wait up to 1 hour for the application to be  "up". 

                           

                          Its a mystery to me as to how the Orion polling engine decides it will start polling a new application.

                           

                          James Troy