Don't have time to read this article and just want to try out the example template?  You can download it here Docker Basics for Linux . 

Overview

As I discuss cloud utilization with customers, one subject that often comes up is Docker.  Many customers are using cloud servers as Docker farms, or to supplement their on-premise docker footprint.  OH!  There's that hybrid IT thing again.

Since some of our readers may be devops people without experience in SAM, I'm going to explain every step in detail.

We watch applications in SAM with templates.  An application template is simply a collection of one or more component monitors. Today we are going to create a simple Docker template as an example of what you can do to monitor docker.

To monitor Docker we are going to:

1.  Build a Docker Server

2. Use the Component Monitor Wizard to create component monitors for critical Docker processes and generate the base template.

3. Create a script Component Monitor to query Docker for critical metrics and insert that Component Monitor into the template from step 2.

Build a Docker Server (step 1 of 3)

First we need to load Docker on a target server.  I won't bore you with instructions about that.  If you need help installing, refer to the documentation from Docker themselves at Docker Documentation | Docker Documentation .

The next step is to manage the node where we installed Docker with SAM (if it wasn't already managed).  If you're using Linux, I'd recommend installing the Linux agent when you set up the node.  Again, this is well documented in the SolarWinds documentation starting here Agent deployment .

Now that we've got a Linux node with the Solarwinds agent and Docker installed, we are ready to build a template to monitor it.

Start with a Process Monitor type component monitor (step 2 of 3)

In order to monitor an application we have to decide what to look at on the target node.  I'm talking about basic availability monitoring by ensuring that the service/process is up.  We accomplish this with either a service or process component monitor.  Using a Service or Process monitor is a best practice when creating a new template.

SAM offers Windows Service monitors along with Windows or Linux Process monitors.  In this example I'm targeting an Ubuntu server running Docker, so I'll use a Linux Process Monitor.

The easiest way to create a component monitor is using the Component Monitor Wizard.  Sometimes new users miss this option because it's located at Settings->All Settings->SAM Settings->Getting Started with SAM. 

In the wizard, I will choose Linux Process Monitor and click "Next"

Now we get to the reason it was so important to have an application node set up and defined in SAM before we started.  The next screen allows us to enter the IP address of the Linux server I set up earlier.  In this case, it's a 64-bit operating system. Be sure to change the 32-bit setting to 64-bit.  Once again, click "Next"

This is where the power of the wizard becomes evident.  You will see a list of the processes on the target server with empty checkboxes next to them.  Just check the processes associated with Docker (in this case dockerd and docker-containerd), then click "Next".

(For more about why Docker split the docker engine into two pieces, take a look at Docker containerd integration - Docker Blog .)

Now our component monitors have been created.  You can optionally check the boxes to the left and test them now.  Since the wizard actually had found the processes previously on the test node, the test should pass.  Just click "Next" again.

You can either create a new application template with these component monitors or add them to an existing one.  In this case we are going to create a new template called "Docker Basics for Linux".  It's a good idea to include the OS type in the name since we might run Docker on Windows in our environment later and we will want to know that this particular template is for Linux.  Also I included the term "Basics" since I get ambitious and write a more complex template later.

Click "Next" to continue to a screen where we can assign the template to SAM nodes.  The node we used for developing the template should be checked by default.  Just click "Next" again if that's the only node we are monitoring for now.

The final step is to confirm the creation of the component monitors.  Click "OK, Create" and you're done.

If you go to the SAM Summary page, you'll see your template has been applied.  Don't worry if it shows an "unknown" state at first.  It can take a few minutes to do its initial polling.  It should turn green once discovered.

That's it.  You're now monitoring Docker availability.

If you wanted to monitor Docker on Windows, you could have used a Windows Process Monitor or a Windows Service Monitor, which both work similarly.  This would provide the most basic information at whether Docker is even running on a target system.

Monitor Docker from the inside with a Script Component Monitor (Step 3 of 3)

For the next step, let's add some visibility into Docker itself.

Go to Settings->All Settings->SAM Settings->Manage Templates

You should see a list of all templates on your SAM system.  In the search box at the upper right corner, type "docker" and click "Search".  This should filter the list to only include the Docker template we created above.

Check the checkbox to the left of our template and click "Edit"

You should see the details of the template with the 2 component monitors we created earlier.

At this point, you can add a description and tags, or change the polling settings.  You can also change component monitors in the template.

Let's add a component monitor to show basic stats about Docker.  The easiest way to get those stats is to run the "docker info" command on the Linux server.

Wow!  That's a lot of information.  But how to do we get it into SAM?

Not to worry, SAM lets us run scripts and can collect up to 10 statistic/message pairs from the script.

Just select "Add Component Monitor" on the screen above and click "Submit".

Then select "Linux/Unix Script Monitor" (you may have to advance to the 2nd page of results) and click "Add".

Now you should see a new component monitor on the list that is opened for editing.

Enter a description and choose the authentication method and credentials. In this case I chose to inherit credentials from the node.  The script is basically run using ssh so port 22 must be open in any firewall configuration.

I entered a working directory of "/tmp" for any disk I/O required by SAM.

I also will rename the component monitor to something more meaningful.

The "command line" field will be set up for a perl script by default.  We are using a shell script so we will delete the "perl" part and replace it with "bash".

Next, click "Edit Script" to enter the actual script commands.

Insert a statistic into SAM with a bash script

In order to use a script with SAM, we are going to need to format the data in a very specific way.  SAM expects the only output from a script to consist of key/value pairs, separated by a ":", that either are a statistic or a message.  The key for statistics consists of the keyword "statistic" then a variable name, then ":"

For example if we were trying to capture number of containers, we would use Statistic.container:1 and for a description we would use Message.container:Containers This would allow SAM to display a value of 1 with a heading of "Containers".  The variable "container" would tie the two together.  That's a brief explanation of getting statistics into SAM.  I could write an entire article on Script Component Monitors alone.  Oh, look!  Someone already did... SAM Script Component Monitors - Everything you need to know

This would be pretty simple if we only wanted the single "Containers" line from the "docker info" command.  We could just pipe the output of the "docker info"  through "grep" to find the line that contains "Containers" and then pipe that line through "sed" to substitute our special identifying text for the heading "Containers: "  To see how this works type this on the Linux command line:

docker info | grep "Containers: "| sed s/"Containers: "/"Message.container:Containers \nStatistic.container:"/

Which would output:

Message.container:Containers

Statistic.container:1

Show more than one statistic

However if we want to grab more than one line from "docker info", it gets a bit more complicated.

First, to avoid having to grab multiple lines, I decided to just have docker send the output in JSON format.  This can be done by using the format option:

docker info --format '{{json .}}'

which returned:

Now we have the output in a single, JSON-formatted line. 

Now to create a poor-man's JSON parser.  I simply used awk to grab the data I was interested in.  I didn't use a JSON parser like "jq" because I didn't want the burden of ensuring that this code was installed on the target system. 

Here is the script that I used:

#! /bin/bash
JSON="$(/usr/bin/docker info --format '{{json .}}')"
echo $JSON| awk -F ','  '{print $3}' | awk -F ':'  '{print "Message.running:"$1,"\nStatistic.running: "$2}'
echo $JSON| awk -F ','  '{print $4}' | awk -F ':'  '{print "Message.paused:"$1,"\nStatistic.paused: "$2}'
echo $JSON| awk -F ','  '{print $5}' | awk -F ':'  '{print "Message.stopped:"$1,"\nStatistic.stopped: "$2}'
echo $JSON| awk -F ','  '{print $6}' | awk -F ':'  '{print "Message.images:"$1,"\nStatistic.images: "$2}'

This grabs the 3rd, 4th, 5th and 6th objects in the JSON and plugs them into SAM as message and statistic respectively.  At this point I have to admit that there is some risk that Docker could change the output of this file and break my positional parsing, while not affecting someone using a real JSON parser to search for the proper key/value pair.  You can test the script from the Linux command line to be sure that it works.

Now that we've got a working script on the Linux server, we can insert it into the SAM Linux script component monitor.

Here's a screen shot of the script being put into SAM

A couple of things to notice on this screenshot.

• I couldn't run the "docker info" command as a normal user.  I had to use "sudo" to run it as super-user.  This required me to modify my "sudoers" file to add the "NOPASSWD:ALL" option for the user whose credentials I'm using.  This change allowed SAM to run a privelidged command without being prompted again for the password.  More on that here How To Monitor Linux With SSH And Sudo
• I used a fully qualified path for the docker executable.  This is to ensure that I don't have issues due to the user's PATH statement.

Here are some screenshots of the monitor as I spun up some images and containers on the target system.

.

That's it!  You're monitoring Docker.

For those of you running Docker on Windows.  Most of the Docker Info commands are similar, if not the same and a script component monitor could be written in PowerShell.