How to create a API poller proxy in Python using Orion API

Hi,

The native SAM api poller handling  is very static requiring manual handling in SW for changes on the target side.

With a proxy Python script we could handle changes as well as just installing the script for other sites.

Further, the api poller is a SAM function whereas the api data is related to different network elements.

The proxy could then also resolve this mismatch.

 I have envisaged that we should be able to generate a python script which will:

  • Initialize - identify the components from the api response to be monitored and configure SW variables accordingly
  • For each polling it will detect if any configuration changes have been made on the target side and adapt the SW config accordingly
  • For each polling it will update the SW variables and SW actions should be taken if status is down etc as natively handled by SW

My starting point for changing values in SW is to use the Orion api via Python as documented at https://github.com/solarwinds/orionsdk-python

One should then be able to add nodes and set custom properties for those.

However, I do not understand how one should be able to apply a “push model” (from the proxy into SW) yielding a SW handling with alarms etc for these variables / custom properties.

The documentation is focused on poller types which is the opposite way of data handling (fetch model).

I would then be pleased to get some guidance on how use the orion api with python for our purposes.

Thanks!
Goran

Parents
  • This all depends on what you are looking to change.  If you want to assign or unassign specific Application Templates (you mentioned Server & Application Monitor (SAM)  near the beginning), that can be done with the API.  If you want to assign custom properties to either the assigned application or nodes, this can also be done.

    But most people "shortcut" this by using Groups.  Here's a (simplified) scenario:

    • You have an application template (we'll call it "Custom Application")
    • This application needs to be assigned to a set of nodes.
    • You create a custom property (for Nodes) called "RunsCustomApp" (True/False) and check the box when the server runs the application.
    • You create a Group called "Custom App Servers" and have a Dynamic Query defining membership where "RunsCustomApp" is True.
    • You can then assign the "Custom Application" template to the group.
    • When nodes move in and out of the group, the application is either assigned or unassigned from the device.
  • Thanks for the feedback!
    I am sorry, but I am not able to follow some details in what you are suggesting (as I am not an experienced user), and would be pleased if you could direct me to the associated documentation and to add some details 

    • From your reply my understanding is that my (Python) proxy would be the Custom application to be created. (I have no experience in applying custom applications)
    • I will then have to create a Group of nodes ("Custom App Servers") there the nodes have a specific custom property defined and set. I would name the new property myself e.g. "RunsCustomApp".
    • There should then be a dynamic membership as to add/remove nodes from the Group which should be controlled by the proxy/application

    Thereby the node mapping is achieved and one could then e.g. use maps to present the topology.
    However, it is not clear how the status information shall be updated / pushed into SW by the proxy/application.

  • Actually I am heading away from the API poller approach for the reasons outlined above.
    I want to handle the API polling inside a Python proxy which will create "virtual nodes" via the Orion SDK.
    A virtual node is then a regular SW node object but it does not have IP reachability etc,
    The question is then how to push the status information from the proxy into the nodes.

    To some extent this would relate to the SNMP traps handling as status info would be pushed and not polled into SW.

  • You are building something in Python to do the External API polling work, but you want the SolarWinds Solution to use the data received back to the Python program?

    There are a couple of ways.

    • You could send the SolarWinds Platform a syslog or a trap indicating "bad things." (I wouldn't send it informational or statistical information).
    • You can create a Application Script Component that goes out and reads a flat file on your Python worker every so often and stores it as Application metrics. (How I would do it)
    • You could build an API on the Python machine and then pull the data from the Python-based API. [Lots of maintenance]

    This is very much an edge case, but I get where you are going with it.  It's a cool setup and I applaud the work. Clap

  • Thanks!
    Yes, I want Solarwinds to use the data from the Python program. Essentially we have a remote network with different entities which is reflected in the rest api data. The customers are today using SolarWinds and wants to integrate the mobile network in their SW operations. A critical demand is then to be able to view the topology and locate where a specific problem is located as they would for their LANs etc. Alarms , dashboards ... shoudl then also be similar and handled within SW.

    Thus, I was targeting to use network nodes as to be able to handle the topology as this is not within the SAM context as handling "application metrics".

    Your suggested approaches would then take care of the updating problem as the Python program will adjust to the topology changes and SW "readers" would be able to digest all data exported by the Python program.

  • Hi Kevin,

    I'm hoping you can help me with the Invoke methods mentioned in your comment since you posted it recently and I can't find much documentation or discussion on it.

    Using SWQL studio, I can see the list of templates with this query: SELECT TOP 1000 ID, Guid, Name, Version, DisplayName, Description, Created, Updated, TemplateData, RequestsCount, MetricsCount, IsCustom FROM Orion.APIPoller.Templates

    My goal is to create a script to automate the creation of an API poller using one of these template ID's. The use case is that one of the teams in my organization has been quietly monitoring almost 300 API's via testing using Postman and script automation for the past several years, and I would like to plug in whatever document they give me (.txt or .csv) with all of the necessary details, into a script so that it takes all of the params and builds the API Pollers accordingly so that I don't have to do it manually.
    They are mulesoft API's hosted in the cloud, if that matters.
    My ultimate goal would be to use this functionality for DevOps to automate the monitoring of new API's with "self-service" for dev teams in mind.
    ***The issue is that I can't figure out what to pass with the Orion.APIPoller.Invoke - CreateApiPollerFromTemplate  in SWQL Studio, specifically, the "Configuration" body and the "Parameters" body***
    I've had success with similar invoke methods with SWQL, but this one is giving me a hard time.
    Please help. Thanks.
    Klobk
  • I haven't done this work before.  The verbs and read/write capabilities are there.  I just haven't done the work before.  My API pollers are VERY simple when I work with them.

  • Hi Kevin,

    Perhaps you could invite some expert on the API Pollers into this discussion as to overcome the problems with the static configuration.

    Unfortunately I do not have a SW instance up and running at the moment as to refer to the various DB tables for API Pollers.
    However, I see that one could structure the problem as:

    When making the configuration in the GUI what are the steps to create the related DB instances.
    There is then eg to define the values to be read and their translations into SW e.g. map 'Up' into SW value = 1.
    (Using the GUI for configuration one defines the URLs, token handling, parameters to be read and associated SW mapping, and could serve as a way to understand the API poller tables)

    The question is then if there is some additional SW functions that would be automatically triggered when creating the basic data in the DB and to populate some other tables.
    The critical question is then how the polling will be triggered, potentially SW will automatically check all the API pollers in the DB and populate the DB tables holding the polling data. (Alarm handling will be defined on top)

    Surprisingly the polling interval is not controlled in the API pollers but in the "hidden system configuration file".

    In addition, one has to consider how to apply changes related to data to be read etc along the road.

    Best regards
    Goredb

  • Without an instance, there's very little that can be done.  And I'm not working with the database directly, but with the API.  Without an instance, the API's are not available.

    We don't support working with the database tables directly - because they can change version to version.

  •    Kevin, can you refer to someone who has created API pollers via the SDK or some documentation with examples.

    We thought about opening a support ticket, but they push back on helping with custom scripts!

  • We don't have any documentation for it.  All the API's docs are up in the GitHub Wiki (and some samples).  I don't think we have any specific for API pollers.

  • I don't see a section for the API poller.  Can you help point to that specific section of the Github Wiki?

Reply Children
  • Here is an example adapted from some internal design documents.  I haven't tested this, so take it with a big grain of salt.  If this doesn't work, I'll ask the team that worked on this feature to comment here.

    CreateApiPollerFromTemplate allows you to create an API Poller based on a template exported either from an existing API Poller or from the out-of-the-box template library.  But how do you get that template file to begin with?  If you're exporting an existing API Poller to a file (template.xml in the example below), you could use the ExportTemplate verb:

    Import-Module SwisPowerShell
     
    $hostname = "IP"
    $username = "Username"
    $plainTextPassword = "***"
    $password = $plainTextPassword | ConvertTo-SecureString -AsPlainText -Force
     
    $cred = New-Object -typename System.Management.Automation.PSCredential -argumentlist $username, $password
    $swis = Connect-Swis -host $hostname -cred $cred
    
    $templateId = 1
    $result = Invoke-SwisVerb $swis Orion.APIPoller.Templates ExportTemplate @($templateId)
    $exportedTemplate = $result.InnerText
    $exportedTemplate | Out-File ".\template.xml"

    Once you have a template file to work with, you can import it via the CreateApiPollerFromTemplate verb:

    Import-Module SwisPowerShell
     
    $hostname = "IP"
    $username = "Username"
    $plainTextPassword = "***"
    $password = $plainTextPassword | ConvertTo-SecureString -AsPlainText -Force
     
    $cred = New-Object -typename System.Management.Automation.PSCredential -argumentlist $username, $password
    $swis = Connect-Swis -host $hostname -cred $cred
     
    $apiPollerTemplate = (Get-Content -Path ".\template.xml" | Out-String)
     
    $configuration= @{
        "CredentialType" = "BasicAuth"
        "CredentialID" = "1"
      };
    $parameters= @{ };
     
    $result = Invoke-SwisVerb $swis "Orion.ApiPoller.ApiPoller" "CreateApiPollerFromTemplate" @("Orion.Nodes", 1, $apiPollerTemplate, $configuration, $parameters)
    $apiPollerId = $result.InnerText

  •     Thank you for the response.  This documentation helps a lot.

    I was able to get it to work manually through the SWQL studio, but when trying via the PowerShell script I keep getting this message.  I can't figure out what is does not like.

    Invoke-SwisVerb : Verb Orion.APIPoller.ApiPoller.CreateApiPollerFromTemplate cannot unpackage parameter 3 of type System.Collections.Generic.Dictionary`2[System.String,System.String]

    Initially I was trying to pass in the XML string as it was passed in via SWQL studio and then I changed the $configuration and $parameters to the hash table format you suggested, but it returns the same error.

    Here is my code snippet:

    $entityType = "Orion.Nodes"
    $entityID = 3045
    $template = '<?xml version="1.0" encoding="utf-8"?><Template xmlns:xsi="">www.w3.org/.../XMLSchema-instance" xmlns:xsd="">www.w3.org/.../DisplayName><Description /><Created>2024-03-05T04:38:10.3644824Z</Created><Updated>2024-03-05T04:38:10.3644824Z</Updated><Version>1</Version><RequestDetailsCollection><RequestDetails><Url>blah.com</Url><Body /><HttpVerb>Get</HttpVerb><ValueToMonitorCollection><ValueToMonitor><DisplayName>status code</DisplayName><Path>$</Path><ThresholdRule>NotEqualTo</ThresholdRule><WarningThresholdValue xsi:nil="true" /><CriticalThresholdValue>200</CriticalThresholdValue><Type>StatusCode</Type><StringToNumberTransformationRules /><StringToNumberTransformationOtherValues>0</StringToNumberTransformationOtherValues></ValueToMonitor></ValueToMonitorCollection><RequestVariables /><RequestDetailsOrder>0</RequestDetailsOrder></RequestDetails></RequestDetailsCollection><PollingInterval>60</PollingInterval></Template>'
    
    ## Variable format option 1
    #$configuration = '<ArrayOfKeyValueOfstringstring xmlns:i="">www.w3.org/.../XMLSchema-instance" xmlns="'">schemas.microsoft.com/.../ArrayOfKeyValueOfstringstring>' | ConvertFrom-StringData
    #$parameters = '<ArrayOfKeyValueOfstringstring xmlns:i="">www.w3.org/.../XMLSchema-instance" xmlns="'">schemas.microsoft.com/.../ArrayOfKeyValueOfstringstring>' | ConvertFrom-StringData
    
    ## Variable format option 2
    $configuration = @{
    "Name" = $pollerName
    "DisplayName" = $pollerName
    "Url" = $pollerURL
    "VerifySslCertificate" = $true
    "UseProxy" = $false
    "PollingInterval" = 60
    "CredentialType" = "OAuth2Shared"
    "CredentialID" = "173"
    };
    
    $parameters = @{ };
    
    Invoke-SwisVerb $swis Orion.APIPoller.ApiPoller CreateApiPollerFromTemplate @($entityType, $entityID, $template, $configuration, $parameters)

  • Internally, the last two parameters for this SWIS verb (configuration and parameters) are of type Dictionary<string, string> in C#.  That means it's a dictionary data structure with a string key and a string value.  When serialized as XML, Microsoft uses ArrayOfKeyValueOfstringstring (see https://learn.microsoft.com/en-us/openspecs/sharepoint_protocols/ms-sadmws/5a8ba9b2-6287-44b4-a8fc-ab88b7d7c3c4).  There's a lot of useful information about working with that kind of data and some examples of what the XML looks like at https://stackoverflow.com/questions/49519253/deserializing-xml-from-soap-call-into-array-of-objects.  See Where are Orion.Credential.CreateCredentials & Orion.Credential.UpdateCredentials for an example of another place this data format is used with the Orion SDK for a different verb.  The XML there should give you a good idea of what this should look like. 

  • Did you ever get this working?

    I'm stuck on same issue.

  • Hi Daryl. We got it working with Powershell and I later converted it to Python. I'm sure you can finagle this to suit your needs. Note, that it reads a list of url's and uses regex to create the Poller Name based on the text in the URL so that you can mass create API pollers. We had some weirdness where I had to go in and re-select the 'No Auth' for each one (or select the auth, you can also set the auth in the script, but we did not) otherwise they were stuck in 'unknown' status. You can modify the regex as needed. Here is an example of the format of the API list .txt file: 'https://api.corporation.corp/sapi-thing1-events-storage/v1/healthcheckendpoint' and here is the script. Cheers!

    from orionsdk import SwisClient
    import requests
    import re
    requests.packages.urllib3.disable_warnings()




    configuration = {"VerifySslCertificate" : "False", "UseProxy" : "False"}
    parameters = {}
    nodeid = 1234
    entity = 'orion.nodes'


    def create_template_from_txt():
        file_path = 'C:\\path\\apiURLs.txt
        with open(file_path, 'r') as file:
            urls = file.readlines()

    # Extracting the API names and entire URLs
        api_info = [(re.search(r'.corp/(.*?)/v1', url).group(1), url.strip()) for url in urls]

    # Printing the results
        for api_name, full_url in api_info:
            main(api_name, full_url)
       

    def main(api_name, full_url):
        template_xml = (f'<?xml version="1.0" encoding="utf-8"?><Template xmlns:xsi="">www.w3.org/.../XMLSchema-instance" xmlns:xsd="">www.w3.org/.../Name><DisplayName>PRODUCTION{api_name}</DisplayName><Description /><Version>1</Version><RequestDetailsCollection><RequestDetails><Url>{full_url}</Url><Body /><HttpVerb>Get</HttpVerb><RequestHeaders><RequestHeader><Name>x-api-key</Name><Value>123lj21o3i3o42i5o4h5oij34ghi5ygu</Value></RequestHeader></RequestHeaders><ValueToMonitorCollection><ValueToMonitor><DisplayName>status code</DisplayName><Path>$</Path><ThresholdRule>NotEqualTo</ThresholdRule><WarningThresholdValue xsi:nil="true" /><CriticalThresholdValue>200</CriticalThresholdValue><Type>StatusCode</Type><StringToNumberTransformationRules /><StringToNumberTransformationOtherValues>0</StringToNumberTransformationOtherValues></ValueToMonitor></ValueToMonitorCollection><RequestVariables /><RequestDetailsOrder>0</RequestDetailsOrder></RequestDetails></RequestDetailsCollection></Template>')
        npm_server = 'orionserver01'
        ORION_USERNAME = '<username>'
        ORION_PASSWORD = '<password>'
        swis = SwisClient(npm_server, ORION_USERNAME, ORION_PASSWORD)
        invoke_create_poller = swis.invoke ('Orion.ApiPoller.ApiPoller', 'CreateApiPollerFromTemplate', entity , nodeid,  template_xml, configuration, parameters)
        if invoke_create_poller:
            print('Success')

        else:
            print('Failed')
       

    if __name__ == "__main__":
        create_template_from_txt()
     
  • Im actually doing this in powershell, I should be able to convert it back.

    Thanks

  • you may want to remove your KEY in the xml