cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Monitoring Microsoft Office 365 with SAM

Level 10

Companies are moving their email to cloud in droves.

Let's face it, administering Microsoft Exchange is one of those jobs that when everything goes right, no one knows you exist.  And when things go wrong, everyone knows you exist. The good news is that many companies are offloading their Exchange to Microsoft through the use of Microsoft Office 365.  If you doubt that Office 365 is big, consider that in July of this year Office 365 online workplace tools brought in more revenue than the traditional version of Office that’s installed on people’s computers. When you think about it, e-mail server replacement is the perfect SaaS application.  It's well defined without huge deviation from one organization to the next, scales well across multiple servers, needs to be accessible from anywhere and often needs permanent retention of records.  All things that the cloud is good at.

Moving to the cloud means I'll never have to worry about email again, right?

It's important to remember that while moving to the cloud alleviates your responsibility for the servers that run e-mail, you still are responsible for monitoring the e-mail itself and your company's connectivity to the cloud.  Monitoring cloud-based applications is different than monitoring on-premises applications.  Where you may have been concerned with memory and disk capacity on your servers, or server-to-server communication in the past.  Those are not concerns with SaaS.  But some potential issues still exist.    Here are just a few of the metrics you may need to be concerned with in an Office 365 environment:

  • Portal Access - Rather than server availability, it's important to know portal availability. This includes the user portal, the administration portal and the billing portal.  These may each be used by different users in your company but are all important.
  • Forwarded Exchange Users -  Are these mailboxes really necessary?  Are they violating company or government policies? What if a healthcare worker is forwarding messages containing patient information to a personal account, for example? 
  • Inactive Exchange Users - While sometimes you may keep a user's mailbox for a period of time after they are gone, sometimes you just forget to delete them and are paying for unneeded accounts.
  • Groups Accepting External Email - Do you really want external entities to be able to bulk mail these groups?
  • Top Senders - This is a handy metric for telling if your accounts have been hacked and are being used by spammers.
  • Administrative Roles - Did the number of administrators change unexpectedly? 
  • License Usage - Get a handle on how quickly your license usage grows.  How many licenses are being used?  What percentage of my total?  You still need capacity planning for SaaS, just a different type of capacity.
  • Last Password Change - Number of users with a password that is 90 days old or more.  How many users have a password that never expires?
  • User Mailbox Security - How many users have access to a large number of mailboxes?  Should they? 

Earlier this year, in collaboration with Loop1 Systems, we developed a set of templates for Microsoft Office 365 to monitor these and much more.  The templates have been very popular with customers, but there are a few things you can do to improve their implementation and function. Since these templates monitor Software as a Service, they aren't exactly like other templates that we typically provide.

Microsoft Office 365 is Software as a Service and it doesn't run on any of your servers.  What node should you apply the template to?

Since these templates are PowerShell scripts that run against a Microsoft URL, the best solution is to create an external node and apply the templates to it.  You can use "outlook.office365.com" as the node.   This is the URL for the mail API requests.  Technically for the Portal, Subscription, Security Statistics and License Statistics templates the scripts use "api.admin.microsoftonline.com", but splitting the Office 365 templates between two nodes can be confusing and forces the SAM user to understand which components of the service reside on each node.

Screen Shot 2017-12-13 at 8.53.19 AM.png

You can also use an ICMP node rather than an external node

External nodes don't report status.  By using an ICMP node, you will get a rudimentary status indication on the node icon based on a ping of the URL.  External nodes give no status and always display a purple "arrow" icon without status.  However the URL "api.admin.microsoftonline.com" doesn't seem to respond to ping requests so it will always appear to be down if you point an ICMP node there.  Here is the external icon vs. the ICMP node icon.

Screen Shot 2017-12-13 at 5.42.29 PM.pngScreen Shot 2017-12-13 at 5.43.34 PM.png

Get a real picture of Office 365 availability with NetPath

Another way to determine the responsiveness of the Office 365 application is to set up a NetPath service for "outlook.office365.com".  If you have NetPath, you can use it to get a detailed view of the bottlenecks between your site and the application portal.

Screen Shot 2017-12-20 at 9.07.10 AM.png

Improving responsiveness to queries by polling less frequently

Depending on the number of mailboxes in your environment and the number of templates implemented, you can experience throttling of your API requests from the Office 365 API.  If you are throttled, the choices are to either run less component monitors or reduce your polling frequency on some templates.  Most users can actually reduce the polling frequency substantially on most or all Office 365 templates since the majority of the metrics don't change frequently.  One thing to keep in mind is that if you want to ensure enough data points to avoid gaps in history, you might want to use less than an hour for your polling frequency, so try setting the frequency to 1200 (20 minutes) rather than the default of 300 (5 minutes). If you want to know more about Microsoft API throttling, see Avoid getting throttled or blocked in SharePoint Online | Microsoft Docs for a description.  The article is about Sharepoint but the concept is the same for Office 365.

I don't like the output of the detailed data from the templates.  Can I make it more readable?

The data comes back from the API in a comma-delimited format which is great for programming but not so readable.  To make the data more readable, you can modify your own copies of the scripts as follows:

Replace:

[string]::Join( ", ", $users) 

With

[string]::Join( "< br/>", $users)

NOTE: You should be aware that this modification is injecting HTML directly into the output from the PowerShell script.  When viewed on the SAM console it will display correctly.  However, this change could create unexpected results in other areas of SAM that are not displayed on a web server, such as reports.

Comparing Exchange 2013/2016 templates with the Office 365 template.  They are both Exchange, why are they so different?

Since Office 365 is SaaS, many of the metrics in our previous Exchange templates is either not available or not meaningful.  Metrics like disk I/O and disk latency aren't available for a cloud service where the hardware is abstracted away from the user.  Similarly attempting to monitor processes and services on the hosts is not possible.  Primarily with Office 365 we monitor application data, which is available through the Office 365 API.

There was a MAPI round trip template available for Exchange.  Can I run this template against Office 365?

The MAPI round trip template was intended to check connectivity between multiple Exchange servers.  Since Office 365 is SaaS, you don't control the physical servers that are used for your accounts.  With cloud-based applications, you should check connectivity between your network and the Office 365 website.  You can get a sense of this connectivity by using the portal templates and the ICMP option discussed above.  Also as mentioned above, you can use NetPath to show the actual path your connections take to Microsoft.  Another option is to use Web Performance Monitor to record a typical mail transaction and get perspective on each part of the session.

Screen Shot 2017-12-20 at 9.20.55 AM.png

A comprehensive approach to monitoring Microsoft Office 365

Hopefully, this post has given you some ideas about why and how to monitor Office 365. SolarWinds offers many tools to help you from SAM templates to network tools to user simulation.

30 Comments

So all of this is great.  I just setup netpath to both portal.office365.com and outlook.office365.com to see how this will work.  I am a bit confused as to how to setup SAM templates in Office 365. Am I missing something above?

Level 10

It kind of describes it, but it's a little confusing.  Setup login.microsoft.com or api.admin.microsoftonline.com or outlook.office365.com or whatever either as NCMP nodes (that'll ping them and show up/down as well if they're pingable) or just as External nodes.  Then just apply monitors to it as any other node.  You can't do system nodes, obviously, but HTTPS or IMAP or Port monitors, etc, all work fine.  Also powershell works well if you just have them run on the poller (pick Local Host as Execution Mode).

Good luck!

OHHH ok yeah that makes sense now. I was a bit confused between that and the NetPath service setup.  Thanks!

Level 8

Our O365 Netpath generally looks like this, although there's no indication that we have bad connectivity:

Path Inspector.png

Any thoughts or suggestions?

Level 12

can i get a list of credentials required for each office 365 template?

pastedImage_0.png

MVP
MVP

Hey aaswi​! All of the templates were run with a user that had administrative access to Office 365. Are you not able to get it running that way?

Level 7

I'm trying to use this template and I get "100 NoDataAssumingUp" on every component in every template I try with a variety of user accounts. When I look at the details of the test results I get:

Invoke-RestMethod : Sendera:UnauthorizedUnauthorized

The template is using the credentials I assign to the component, right? Am I missing something here?

MVP
MVP

Hi mark.behrendt​! That's absolutely right - it takes the credential that's assigned to the component and runs with it. If it's throwing that error then it means that it can't get the information you're asking for with the credentials supplied. What you can do it is take the body of the one you're working with, copy it, and paste it into PowerShell ISE and try running it from there. You'll need to modify the portion that assigns the credential.


For example, you would change this:

$cred = Get-Credential -Credential ${CREDENTIAL}

To this:

$cred = Get-Credential (Get-Credential)

That'll force it to prompt you for the credentials that you're using when you run the script. If you still get the error above, reply to this thread and I'll take a closer look for you.

Level 7

I tried that and it still does the same thing. I'm sure the account I'm using has admin rights and the username/password are correct. I still get the 100 exit code and the error:

Invoke-RestMethod : Sendera:UnauthorizedUnauthorized

At line:24 char:16

+ ...  $cookie = (Invoke-RestMethod -ContentType "application/json" -Method ...

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException

    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : Sendera:BadRequestBad Request

At line:32 char:16

+ ...  $events = (Invoke-RestMethod -ContentType "application/json" -Method ...

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException

    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Statistic: 100

MVP
MVP

Are you using modern auth on your account?

Level 7

I believe so.

Mark Behrendt

Sr. Systems Engineer

Mark.Behrendt@TeletracNavman.com

MVP
MVP

That's the problem! You'll need to create a service account that doesn't have that enabled. Here's the blurb from Microsoft:

Office 365 Service Communications API Overview

pastedImage_1.png

In particular it's the 2nd bullet from the bottom under the 401 error.

Level 7

I must be missing something. The article you linked has no information about modern auth. There doesn’t appear to be a way to disable it per user, and I can’t justify disabling it for the entire organization. I know the credentials are valid, and I’m pretty sure basic auth is supported as I’m able to connect via:

$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection

I can then run:

Import-PSSession $Session -DisableNameChecking

And get

ModuleType Version Name ExportedCommands

MVP
MVP

I may be using the wrong term for the authentication hurdle we're trying to clear. I just worked through this with someone else. I'll check with them and see what adjustment they made to get us past it. It only needed to be done for the single service account and not the entire org.

Level 7

I'm trying to use this template and I get "100 NoDataAssumingUp" on every component in every template I try with a variety of user accounts. When I look at the details of the test results I get:

I was getting similar code and realized my credentials were incorrect

I was using my username instead of username@domainname  after I made the change things starting working.

Level 11

We recently added articles about SAM's Office 365 templates to the SolarWinds Success Center. You can access them via this link: Using Microsoft Office 365 templates.

All feedback is welcome!

Thank you. I am reading these over now and will work on updating my setup.

Level 12

I am getting this when I test: Microsoft Office 365 Subscription

Output: ==============================================

Statistic: 6

Message: FalsePositive

MVP
MVP

That means that the most recent event for that service/feature is of type 'FalsePositive' which has a numeric value of 6. If I remember correctly, anything higher than 5 was a "bad" status.

Level 9

At our organization, we provide access to our internet via a central egress point.  I'm thinking that we can leverage NTA port classification for applications and services and really get some useful data.

I'm thinking that a Multi-Port application may be a good example.

Has anyone attempted this?  If so, did it yield any benefits?

Level 12

Couple of questions on this and perhaps I'm just not clear on how this cloud service works.

Using "outlook.office365.com" as the node, how does SW-SAM know I'm only interested in our specific instance of Office 365? That seems like a very generic target.

According to https://support.solarwinds.com/SuccessCenter/s/article/Using-Microsoft-Office-365-templates these templates require

An Office 365 account with global administrator privileges.

The account must be a member of an Office 365 admin role.

The account should be an all-in-one, inclusive account to support the monitoring of all mailboxes.

As we have a limited number of O365 Global Administrator accounts available to us and, our Exchange/Mail Admin/Engineer does not like giving Global Administrator account privileges , is there a break down of exactly what specific rights must be granted to the service account for each of these templates to work?

MVP
MVP

Hi there,

Because you only have access to your own information, that's all you'll get back based on your login's permissions. As for using a global administrator account, that's all the documentation would tell me when I wrote the SAM templates originally but I think that a colleague was able to whittle down the permissions to something that would work but wouldn't be global admin. I'll send this thread along to him so he can chime in.

Level 12

@Steven Klassen, any luck here ?

MVP
MVP

I was actually writing a response in another window. Too funny.

MVP
MVP

Back again - sorry for the delay. So we tinkered with this a bit in our lab and got mixed results. I've worked with tdanner​ a bit too when these were first being developed and he came to the same conclusion - if Microsoft won't be more specific about the minimum requirement in their API documentation we can't do any better.

Tim, is this still the current thinking or is my knowledge out of date?

Level 12

Is there a distinction between Managing and Monitoring?

MVP
MVP

I would say monitoring, but I'm not sure what you're referring to. Can I get a screenshot?

Level 12

I wasn't clear. I'm thinking in terms of what permissions it takes to manage, like increase quota's, move boxes, etc....vs. Monitoring queue lengths or what have you.

How much of this crosses over to the O365 templates?

  1. Local administrator permissions are needed for automatic configuration, but are not needed for monitoring after configuration is complete.
  2. To provide organization-wide capability, the service account (Domain User) must be a member of the View-Only Organization Management group. Membership to this group gives the user object read-only access to the entire Exchange environment, without providing any domain or local access on the Exchange server. It also prevents unauthorized access to the account.
  3. To gather information, the user object must be assigned the Mailbox Search management role within Exchange. The account must be a member of the Local Administrators group.
  4. Note: Users without this role can access Exchange, but the additional level of permission is required modify Exchange and WinRM settings on the server, or poll performance counters.
  5. For Mailbox statistics, Hub Transport Servers need to be accessed via RPC.
MVP
MVP

Thanks for the clarification! I'm not an O365 administrator myself, but looking at the templates I'm going to take a swing at the questions.

  1. There's certainly no configuration changes being made; everything is read-only.
  2. This sounds good to me, still holding to the no-write-access-required thinking as #1.
  3. The scripts that look at mail traffic statistics would fall into this category. To get top receivers & senders I used Get-MessageTrace.
  4. This looks like part of 4 and we're still not modifying anything so I think we're good here.
  5. I'm not sure why it mentions RPC because I'm using the ExchangeOnline PowerShell endpoint via HTTPS and I'm still able to get at that information. Maybe it means on-prem?
Level 19

Sorry, I haven't worked with this since then. I don't have any additional information.