Monitoring Azure Data Factory mostly automatically with SAM

Update! The Powershell script is updated due to following reasons:

  • Microsoft changed the "AZ" module a little bit. Therefore one line of the script needed to be changed if you use that module
  • The script now searches all subscriptions the given Azure Application has access in the given Tenant
  • Creation of a top level menu was added.
    This can be disabled by putting a '#' in front of the line  
'addViewToMenu -swis $swis -menuName "Cloud" -viewID $viewID'

Many thanks to  

  • A problem with the Login to Azure is fixed.

During the implementation of Azure Data Factory the question of how to monitor them raises. After a longer discussion we started with the monitoring of Pipeline runs and Triggers. Because they will be created and removed by the business owners without the notice of the IT department I started to automate the management of the monitoring which means:

  • Automatically add Pipelines and Triggers to monitoring
  • Automatically delete them from monitoring
  • Create a new group if needed (name: "Data Factory [Factory name]") to grant the Factory owners access to the metrics
  • Add new Pipelines and Triggers to the group
  • Create a new View if needed (name: "DataFactory [Factory name]") to have an overview of the Factory
  • Add a top Menu called "Cloud" and add the views in it

The outcome was two application templates and a Powershell script. 

Each template has one component which does the monitoring. Because SWISS is not (yet) able to add components to an template or application (as far as I know) each trigger and pipeline has it's own application monitor.

What do you have to prepare in front for using the script:

The files are attached at this document. The script is the "ManageDataFactoryMonitoring.txt" (Thwack don't like the "ps1" 

The Script takes the following parameters:

  • TenantID : The ID of the Azure Tenant
  • appid : The ID of the monitoring application
  • key : The Key of the monitoring application
  • OrionServer : The name or IP of the Orion server
  • User : Username for accessing Orion. If not provided the user which runs the script will be taken (trusted access)
  • password : The password for the user. If not provided you will be asked for it

Hope this script helps others too.