Office 365 Service Health Status

Anyone having issues today with this Service Health Monitor?
I am getting the unknown and critical status with no Message with the below error:

Exception was caught: The remote server returned an error: (403) Forbidden.

Thanks!

Also seeing the same issue with the 403 errors since 1/3.

mat12 and nglynn, 403 errors indicate some sort of permission issue. Either the account that you are using no longer has the ability hit the endpoint, or the endpoint no longer accepts those types of requests.

Is the account you are using a registered app with the appropriate permissions?

I no longer have a Demo instance to test a lot of this stuff out with, I'm working on standing one up.

My recommendation would be to run the script in Powershell ISE from the Polling engine it's assigned to and see if it provides any additional errors that might point to the permission you might be missing. 

I suspect that the Request URL might have changed, if it was not permissions. But - I have noticed that things change unpredictably in Azure. 

Yeah I need to get around to testing via ISE still.  Noone has really noticed so far so... lol  But since other people were having the same issued starting at the same time it seemed likely something changed on that side of things and not likely specific to our tenant.

The Office 365 Service Communications API has been retired with Service Health information now available in the Microsoft Graph.

docs.microsoft.com/.../office-365-service-communications-api-reference

Just looking at what's required to update the script to call Graph instead.

Hi robertblackstock,
Thanks for the information. Same thought I have when we first encounter this issue as when i run the powershell script manually i get the same error the first time.

However, I am not really familiar on what to update with this template connecting to M365 as I can't find an updated information about this.

Even the Solarwinds built-in API Poller templates like 'Microsoft 365 Exchange Online' is not working same day this template have an issue. I hope Solarwinds team can provide an updated working template for everyone to use like this one or hoping someone can come up with a fix.

Hi christopher.t.jones123,

Yes I was able to run the powershell manually as a test and got the same error showing on the webconsole at first "Exception was caught: The remote server returned an error: (403) Forbidden.". After running the test again, now I am getting this error: "Warning: Exception was caught: Unable to connect to the remote server".

There is no changes on the credential and permission used so as per robertblackstock , Office 365 Service Communications API has been retired.

Now we are running days without this monitoring and hopefully waiting from Solarwinds Team or someone in thwack that can help fix this service health cloud monitoring that up to now, no built-in template.

mat12, what's the URL it's trying to hit? i don't have a copy of this application monitor so I cannot look myself. Additionally, I would suggest looking at this API builder for Graph, if the endpoint did indeed change this would be the way to find out the new endpoint to update to.

The scope/ request URL has most likely change from something like 

https://manage.office.com/api/v1.0/blablablah

Graph documentation feels a lot like a trail of breadcrumbs. Anyone familiar enough with Graph to comment further? IF you are using the new API poller this will stick out. Send a test request and see if you get : The remote name could not be resolved: 'manage.office.com'

I dug abit and the request URL might be this: 

https://graph.microsoft.com/v1.0/admin/serviceAnnouncement/healthOverviews/Microsoft 365 suite

Here is a reference document:
Get serviceHealth - Microsoft Graph v1.0 | Microsoft Docs: docs.microsoft.com/.../servicehealth-get

Good Luck, I am in the middle of five other things but will post when I get a chance. 

Marc

Looking through the Graph API "List healthOverviews" documentation, I've been able to rewrite the application script body to successfully get the Status and Messages. It's also actually removed the need to do the second messages API query as the Issues are included in full in the first status call.

MS Documentation: docs.microsoft.com/.../serviceannouncement-list-healthoverviews

Updated Graph API Script included here, use at your own risk. Let me know if you spot any issues!

#Pass Credentials from Solarwinds
Function SecureStringToString($value)
{
    [System.IntPtr] $bstr = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($value);
    try
    {
        [System.Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr);
    }
    finally
    {
        [System.Runtime.InteropServices.Marshal]::FreeBSTR($bstr);
    }
}
$c = Get-Credential -credential ${CREDENTIAL}
[string] $username = $c.Username
[string] $password = SecureStringToString $c.Password

# Objects
$service = $args[0]
$tenantId = $args[1]
$client_id = $username
$client_secret = $password


# Construct URI for OAuth Token
$uri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"

# Construct Body for OAuth Token
$body = @{
    client_id     = $client_id
    scope         = "https://graph.microsoft.com/.default"
    client_secret = $client_secret
    grant_type    = "client_credentials"
}

# Get OAuth 2.0 Token
$tokenRequest = try {

    Invoke-RestMethod -Method Post -Uri $uri -ContentType "application/x-www-form-urlencoded" -Body $body -ErrorAction Stop

}
catch [System.Net.WebException] {

    Write-Warning "Exception was caught: $($_.Exception.Message)"
    
}

$token = $tokenRequest.access_token

$o365status = try {

    Invoke-RestMethod -Method Get -Uri 'https://graph.microsoft.com/v1.0/admin/serviceAnnouncement/healthOverviews?$expand=issues' -ContentType "application/json" -Headers @{Authorization = "Bearer $token"} -ErrorAction Stop

}
catch [System.Net.WebException] {

    Write-Warning "Exception was caught: $($_.Exception.Message)"
    
} 

$Status = $o365status.Value | ? {$_.Id -eq $Service}
$IncidentIds = $Status.issues | Where-Object isResolved -eq $False

if ($Status.status -match "ServiceOperational") {$Stat = 99}
if ($Status.status -match "FalsePositive") {$Stat = 6}
if ($Status.status -match "ExtendedRecovery") {$Stat = 3}
if ($Status.status -match "InformationUnavailable") {$Stat = 8}
if ($Status.status -match "PIRPublished") {$Stat = 7}
if ($Status.status -match "RestoringService") {$Stat = 2}
if ($Status.status -match "ServiceDegradation") {$Stat = 1}
if ($Status.status -match "ServiceInterruption") {$Stat = 0}
if ($Status.status -match "Investigating") {$Stat = 4}
if ($Status.status -match "ServiceRestored") {$Stat = 5}

#Write-host 'Statistic.Status: '$Stat
#write-host 'Message.Status: '$Status.StatusDisplayname
Write-host 'Statistic: '$Stat

if ($Status.status -notlike "ServiceOperational") {
$Result = foreach ($issue in $IncidentIds) {
$Output = '<p>' + $issue.Status + ' - ID: ' + $issue.ID + ' - ' + $issue.ImpactDescription + '</p>'
$Output
}
write-host 'Message: '$Result
}
Else {
write-host 'Message: '$Status.StatusDisplayname
}
exit

Hi robertblackstock

Thanks for providing this information.
were you able to successfully get the correct output and running the powershell script manually?
When i try to use your script, i was getting the same error as before using our own tenant credentials:

I am getting this error: "Warning: Exception was caught: Unable to connect to the remote server".

Hi robertblackstock

Thanks for providing this information.
were you able to successfully get the correct output and running the powershell script manually?
When i try to use your script, i was getting the same error as before using our own tenant credentials:

I am getting this error: "Warning: Exception was caught: Unable to connect to the remote server".

The previous script was getting 401 Unauthorized and upon manual execution I discover a message noting the API had been replaced with the Graph API.

Suspect you have issues getting out to the internet from that host. Is there a proxy that you need to go through?

Hi robertblackstock

I already added and used the proxy we used on our previous template and i update yours but same issue occur.

So on your manual execution test can you share what is showing on the result via powershell?
Here is what i am getting when running manuall on powershell:

II also confirmed that the website for https://graph.microsoft.com/ is accessible on the machine i'm using.

Thanks in advance!

P.S: hoping anyone can share if they were able to successfully used this new template?

Bump! anyone able to use robertblackstock template successfully without issues?

I even used a machine with no proxy that can connect to the internet but same occur and got the below error:

WARNING: Exception was caught: The remote server returned an error: (403) Forbidden.

I also confirmed there is no changes on the " ServiceHealth.Read.All" for Application.

Thanks in advance!

With your manual execution, just make sure you are getting a valid access token returned. Check the contents of $token.

Once you have a valid token, the expected response from the Graph API will have @data.context and value.

I suspect there's something at play with your access token or connectivity to the Graph API. I assume with that 403 Forbidden error that the application either doesn't have the permission granted with admin consent or the token isn't being assigned.

Hi robertblackstock

Thanks for the information.

May i know if there is any changes you made to make this work or we need to update compare to the previous working M365 API monitoring? previously we only have the below API permission and we are only provided by the following by our administrator:

client ID
tenant ID
object ID
client secret

The above 4 information is the only thing we used on the SAM template.

Do we need to change any permission on the API after the Jan 4 issue or is there anything else we need to update? in here we are not allowed to have admin rights for monitoring purposes.

Can you also share the output of your working script when run manually?

Thanks in advance

Ah I think you got it. My App Registration has been given the permission ServiceHealth.Read.All under Microsoft Graph.

Hi robertblackstock,

Thanks for the update.

Can you share the same screenshot of your application permission?

We already added ServiceHealth.Read.All under Office 365Management APIs but same issue occur.

Do you mean to change the current Reports.Read.All to ServiceHealth.Read.All under Microsoft Graph when expanded on my last screenshot?

After that changes, all settings are the same and your template works already?

Thanks in advance for your help.

Correct, SerivceHealth.Read.All under Microsoft Graph.

Hi robertblackstock

Yes you are correct.
Adding the ServiceHealth.Read.All under Microsoft Graph and using the updated template you provided now works on our environment. This confirms the changes happen on the Microsoft side that we need to update in order for this template to work. Im just waiting for Solarwinds built-in API to be updated by Solarwinds as those are also seems to be affected by this changes.

Thank you very much on your time and information you have shared here in Thwack and hopefully this can also help other customers having the same issue as we have. 

Thanks again!