Create custom resource to show "Who is On Call" via OpsGenie API

I'm setting up an OpsGenie trial right now, with SolarWinds Orion integration for on-call management. The feature set looks pretty sweet after having managed scheduling, rotation, and weekly changes manually!

Question: I'm curious if there is a way to create a custom resource on an Orion page that shows the current on-call person[s] via OpsGenie's API.

  • You can do this by adding a custom html widget in your dashboard and setting that resource up with a javascript to query the API then parse the response into something that looks readable.

    I don't have any code examples of this specifically, but here's something I snagged from google that would give you a starting point

    1. var url ="http://www.rediff.com/rss/inrss.xml";//change url and params
    2. try{
    3.   $.ajax({
    4.   url  :'http://ajax.googleapis.com/ajax/services/feed/load?v=1.0&num=10&callback=?&q='+ encodeURIComponent(url),
    5.   dataType :'json',
    6.   success  :function(data){
    7. if(data.responseData.feed && data.responseData.feed.entries){
    8.   $.each(data.responseData.feed.entries,function(i, e){
    9. var title = e.title;
    10. var link = e.link;
    11.   list = list +'<li><a href="javascript:display_post(\''+ link +'\')">';
    12.   list = list + title;
    13.   list = list +'</a></li>';
    14. });
    15. }
    16.   list = list +'</ul>';
    17.   document.getElementById("single_feed").innerHTML = list;
    18.   $.ui.loadContent("#single_feed");
    19. }
    20. });
    21. }
    22. catch(e){
    23.   alert(e);
    24. }
  • Thanks. This gives me a little to go on, although I have almost zero experience with scripting API calls. Will have to keep digging and learn as I have time unless someone else has already tried this!

  • Since I'm most familiar with PowerShell, I started by testing with that. The following code successfully pulls the "name" (email address) of the on call person. My next step will still be to figure out how to translate this query into JavaScript.

    $key = "REDACTED"
    $header = @{'Authorization' = 'GenieKey ' + $key}
    $scheduleID = "REDACTED"
    $uri = "">api.opsgenie.com/.../on-calls"
    (Invoke-RestMethod -Method GET -Uri $uri -Headers $header).Data.onCallParticipants.Name

    Open to more suggestions, and thanks!

  • Continuing this quest, I spoke with a developer colleague and he determined that api.opsgenie.com doesn't allow CORS requests (Cross-Origin Resource Sharing). In his words,

    "Basically, for security reasons the browser prevents javascript code from making requests to a domain that is different than the origin from which it was served, unless the requested resource returns the proper headers. When a CORS request is made, the browser sends a “pre-flight” check – an OPTIONS request to the server.  If the server responds with the proper access control header(s),  the “real” request is sent, otherwise it’s a no-go. I’m not sure what the simplest way around this is.  You’d either need a proxy server to return the proper CORS headers to allow the browser to make the api call or a local endpoint (ie, one that we control) that would make the call and return the results to the browser client." - Developer Colleague

    Already being beyond my pay grade, I decided to return to what I know: PowerShell and Orion application templates. The process began with creating a new node in Orion for "app.opsgenie.com" and configuring it to monitor status only (ICMP). For extra measure, I also populated the URL field with https://app.opsgenie.com farther down in the properties page.

    Node

    app.opsgenie.com Node Small.PNG

    app.opsgenie.com Node Large.PNG

    Application Template

    The next step was to create a template to poll OpsGenie. This template uses a Windows PowerShell Monitor to poll OpsGenie and pull the on-call information into Orion!

    As an aside, I also used this template to send an API heartbeat to OpsGenie. The OpsGenie service can be configured to alert the team if a heartbeat is not received. This is a good way to tell if your internal infrastructure might have a problem that is preventing alerts from getting posted to OpsGenie. After receiving the heartbeat 'PING', OpsGenie then sends a 'PONG' response back to the PowerShell script in Orion. Working both ways now, if Orion does not receive the 'PONG' response, it will alert that OpsGenie is non-functional. Anyway, the details of this will be a separate post, I think!

    The script below is what I used to pull all IT teams' on-call schedules. It can be simplified to pull specific teams if necessary, but in this case we wanted all of the IT teams presented.

    $key = "REDACTED" #OpsGenie Default Integration API Key
    $header = @{'Authorization' = 'GenieKey ' + $key} #Authentication is required with every OpsGenie API request.

    $OnCall = @{} #Create a hash table to store team names and on call names.
    $allSchedules = (Invoke-RestMethod -Method GET -Uri "">api.opsgenie.com/.../" -Headers $header).Data #Get the deta for all on call schedules.
    foreach ($schedule in $allSchedules) #Loop through each on call schedule to get the team name and the on call person's name.
    {
        $thisScheduleID = $schedule.ID
        $uriEach = "
    ">api.opsgenie.com/.../on-calls"
        $OnCall.Add( (Invoke-RestMethod -Method GET -Uri $uriEach -Headers $header).data._parent.name, `
            (Invoke-RestMethod -Method GET -Uri $uriEach -Headers $header).Data.onCallParticipants.Name )
    } #End foreach schedule

    $OnCall.GetEnumerator() | ForEach-Object {
        $StatLabel = $_.Key.ToString().Replace(' ','')+":" #Write the statistic label as the team name without spaces, followed by a colon.
        $StatOutput=$StatLabel + " " + $_.Value #Create the stat output string with the label and a numeric value.
        #Format the output for Orion. The message can be a string, but the statistic must be a number, so the hash table item indecies are passed.
        Write-Host "Message.$StatOutput"
        Write-Host "Statistic.$StatLabel" $($OnCall.Keys).IndexOf($_.Key)
      }

    The resulting preview output shows:

    • A message with the team name and that team's person on call.
    • A statistic with the index of the item (team) in the $OnCall hash table. (Because it seems Orion will only accept an integer as the returned stat.)

    Script Output.PNG

    Back in the application template, we now have 4 script output values, with one example shown here:

    Script Output Value.PNG

    It feels like we're close! Now I'm just looking for a way to capture the script output message[s] into an alert or a custom property that can then be queried on a page in a custom view. This last piece would fulfill the original goal.

  • Using this should render a nice widget with optional Lync (Skype for Business) link to direct open a chat window with them. Adding in and pulling additional services like Teams, Slack, and more should be relatively straightforward.

    This bit of script assumes emails are formatted first.last@domain and then automatically converts that to First Last with mailto link applied to it. If you want to get fancy you can even add in a subject field. 

    <div id="oncall_table" class="sw-rpt-tbl-frame">
    <script>
    
    //OpsGenie Oncall Table Generator 0.9.5 written by Chris O'Rourke April 27th, 2020. 
    
    	var swql="SELECT de.ColumnLabel, de.StringData FROM Orion.APM.DynamicEvidence AS de JOIN Orion.apm.CurrentComponentStatus AS ccs ON ccs.ComponentStatusID = de.ComponentStatusID WHERE ccs.ComponentID = 2120 AND de.StringData IS NOT NULL"
    
    	var params = JSON.stringify({
    		query: swql,
    		parameters: {
    		}
    	});
    	
    	$.ajax({
    		type: 'POST',
    		url: '/Orion/Services/Information.asmx/QueryWithParameters',
    		data: params,
    		contentType: "application/json; charset=utf-8",
    		dataType: "json",
    		success: function(response) {
    			//console.log(response.d.Rows);
    		
    			//Initial Table
    			var oncallTable = "<table class='sw-custom-query-table NeedsZebraStripes' style='table-layout: fixed; width: 100%;' cellpadding='2' cellspacing='0'>";
    			
    			//Colgroup
    			oncallTable += "<colgroup>";
    			oncallTable += "<col span='2' />";
    			oncallTable += "<col span='2' style='width: 50px' />";
    			oncallTable += "</colgroup>";
    			
    			//Table Headers
    			oncallTable += "<thead><tr class='HeaderRow'>";
    			oncallTable += "<th class='ReportHeader' style='font-weight: bold; font-size: 12px; text-align: left;'>Team</th>";
    			oncallTable += "<th class='ReportHeader' style='font-weight: bold; font-size: 12px; text-align: left;'>Oncall Resource</th>";
    			oncallTable += "<th class='ReportHeader' colspan='2' style='font-weight: bold; font-size: 12px; text-align: left;'>Contact</th>";
    			oncallTable += "</tr></thead>";
    
    			//Add Resources to Table
    			for(var i=0; i < response.d.Rows.length; i++){
    				oncallTable += "<tr>";
    				
    				for(var j=0; j < response.d.Rows[i].length; j++) {
    					if (response.d.Rows[i][j].indexOf("@") == -1) {			
    						oncallTable += "<td>" + response.d.Rows[i][j] + "</td>";
    					} else {
    						
    						//Retrieve slack userid
    						
    					
    						//Massage email into First Last name (expects single user oncall)
    						var name = '' + response.d.Rows[i][j];
    						name = name.split("@");
    						name = name[0].split(".");
    						name[1] = name[1].replace(/[0-9]/g, '');
    						//console.log(name);
    						
    						//Render mailto  
    						oncallTable += "<td style='text-transform: capitalize;'>";
    						oncallTable += "<a href='mailto:" + response.d.Rows[i][j] + "'>" + name[0] + " " + name[1] + "</a>";
    						oncallTable += "</td>";
    						
    						//Render skype
    						oncallTable += "<td style='text-transform: capitalize;'>";
    						oncallTable += "<a href='im:sip:" + response.d.Rows[i][j] + "'><img src='/Orion/yourimgs/lync-icon.png'></a>";
    						oncallTable += "</td>";
    
    					}
    				}
    				
    				oncallTable += "</tr>";
    			}
    
    
    			//Table Close
    			oncallTable += "</table>";
    			
    			//Grab Element from DOM and Write
    			var table = document.getElementById('oncall_table');
    			table.innerHTML = oncallTable;
    		}
    	})
    </script>
    </div>
  • I've updated the script to include email addresses and phone numbers. It also uses a more optimized method of creating the team/user object:

    <#
    	.NOTES
    	=====================================================
    	 Created with: 	PowerShell ISE x64
    	 Created on: 	8/5/2020 2:50 PM
    	 Created by:	sturdyerde
    	 Updated by:	chrisorourke
    	 Updated by:	jack.vaughan
    	 Versioning:	1.3
    	=====================================================
    	.DESCRIPTION
    		Get on-call users from OpsGenie teams and output them to a SolarWinds
    		SAM Windows PowerShell Monitor component
    #>
    
    ### Verify that no more than 10 teams have been entered in Script Arguments
    If ($args.Count -gt 10){ 
        Write-Error "No more than 10 message/statistic pairs may be added to one SolarWinds component." 
        exit 1
    }
    
    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
    ### OpsGenie Base API Url
    $urlBase = 'https://api.opsgenie.com/v2'
    ### OpsGenie Default Integration API Key
    $apiKey = "APIKEY"
    ### Authentication is required with every OpsGenie API request
    $apiHeader = @{'Authorization' = 'GenieKey ' + $apiKey} 
    
    ### Get the data for all on call schedules
    $allSchedules = $(Invoke-RestMethod -Method GET -Uri "https://api.opsgenie.com/v2/schedules/" -Headers $apiHeader).Data 
    
    ### Loop through each on call schedule to get the team name and the on call person's name
    $allOnCall = 
        ForEach ($schedule in $allSchedules) {
            $scheduleID = $schedule.ID
            $uriSchedule = "$urlBase/schedules/$scheduleID/on-calls"
            $scheduleResponse = Invoke-RestMethod -Method GET -Uri $uriSchedule -Headers $apiHeader
            $userID = $scheduleResponse.Data.onCallParticipants[0].id
            $uriUser = "$urlBase/users/$($userID)?expand=contact"
            $userResponse = Invoke-RestMethod -Method Get -Uri $uriUser -Headers $apiHeader
            [pscustomobject]@{
                Team = $schedule.ownerTeam.name
                User = $userResponse.data.fullName
                Email = $scheduleResponse.data.onCallParticipants[0].Name
                Phone = $($userResponse.data.userContacts | Where-Object { $_.contactMethod -eq 'voice' -and $_.enabled -eq 'True' -and $_.to -notcontains '*314587*' }).to
            }
        }
        #End foreach schedule
    
    ### Counter used to give each team a unique statistic to return to SolarWinds
    $i = 1
    
    ### If team names were supplied in the Script Arguments field, use those
    If ($args) { 
        $teamsSelection = $args
        $teamsOnCall = $allOnCall | Where-Object { $_.team -in $teamsSelection } | Sort-Object Team
    }
    ### Otherwise list all teams
    Else {
        $teamsOnCall = $allOnCall | Where-Object { $_.Email -ne $null } | Sort-Object Team
    }
    
    ### SolarWinds-friendly output
    $teamsOnCall | ForEach-Object {
        Write-Host "Message.$($_.Team -replace ' ',''): $($_.User)|$($_.Email)|$($_.Phone)"
        Write-Host "Statistic.$($_.Team -replace ' ',''): $i"
        $i++
    } 

    I've also added two queries to use to display the data:

    1. Using SWQL in a Custom Query resource on any view in SolarWinds.

    SELECT 
        de.ColumnLabel AS [Team]
        ,SUBSTRING(de.StringData, 1, CHARINDEX('|', de.StringData) - 1) AS [Name]
        ,SUBSTRING(de.StringData, CHARINDEX('|', de.StringData) + 1, CHARINDEX('|', de.StringData, CHARINDEX('|', de.StringData) + 1) - CHARINDEX('|', de.StringData) - 1) AS [Email]
        ,SUBSTRING(de.StringData, CHARINDEX('|', de.StringData, CHARINDEX('|', de.StringData) + 1) + 3, LENGTH(de.StringData)) AS [Phone]
    FROM 
        Orion.APM.DynamicEvidence AS de 
    JOIN 
        Orion.APM.CurrentComponentStatus AS ccs ON ccs.ComponentStatusID = de.ComponentStatusID 
    WHERE 
        ccs.ComponentID IN (
            SELECT ComponentID
            FROM Orion.APM.Component
            WHERE ComponentName = 'On-Call Users'
        )
        AND de.StringData IS NOT NULL
    ORDER BY
        de.ColumnLabel

    Or using SQL for anything else (in Confluence for example).

    SELECT
      de.ColumnLabel as 'Team'
      ,REVERSE(PARSENAME(REPLACE(REVERSE(de.StringData), '|', '.'), 1)) AS 'Name'
      ,CONCAT(REVERSE(PARSENAME(REPLACE(REVERSE(de.StringData), '|', '.'), 2)), '.com') AS 'Email'
      ,REVERSE(PARSENAME(REPLACE(REVERSE(de.StringData), '|', '.'), 4)) AS 'Phone'
    FROM APM_DynamicEvidence AS de
    JOIN APM_CurrentComponentStatus AS ccs ON ccs.ComponentStatusID = de.ComponentStatusID
    WHERE 
      ccs.ComponentID IN (
        SELECT ID
        FROM APM_Component
        WHERE Name = 'On-Call Users'
      )
      AND de.StringData IS NOT NULL
    ORDER BY
      de.ColumnLabel
  • Keep getting these errors and have been unable (so far) to have this pull back all teams in one query so far.
    "You are making too many requests! To avoid errors, we recommend you limit requests.

    Has anyone else hit this limit? If so, How are you getting around it?