Looking for advice on Alerting for Docker Containers

Hello,

I am looking for some advice on setting up Alerts for Docker Containers.

In my current environment, we have Several Kubernetes containers running on several Linux servers.

I have been tasked to alert when a container restarts.

What I am seeing is the container in Exited status, and I alert on that status. However, it takes 7 days to change to Deleted, which will reset the alert.

In the meantime, the server spins up another running container.

So far, the best I have been able to do is provide a count of the Running state for the container with the similar name on the same node, with instructions to Acknowledge the alert as long as a running container is seen.

I am hoping somebody has some better ideas to assist me in my quest to monitor containers.

I have considered alerting on a SWQL query, but cannot seem to get the logic, and get it to fit in the the default Container Select statement.

In each container, I filter for the Name to contain a certain string.I also filter out "PODS" to eliminate the paused containers.

I then have one or more "names" (names vary by container), which I parse out to get a simple name.From the resulting list, there is 1 "running" container. When that container is no longer running, I want an alert, and when 1 is running again, the alert should clear.

Find more posts tagged with

sam

containers

docker

Alerting

kubernetes

Accepted answers

neomatrix1217

What about the uptime of the container?

All comments

neomatrix1217

Have you tried alerting off of this?

natetech

I thought that would be the direction to go, but for some reason, despite however many "exited" states I see, the container never shows a Restarts state in SolarWinds. Cortex.Orion.Cman.Container.RestartCount is 0 for every entry.

neomatrix1217

What about the uptime of the container?

natetech

I have to alert one one of the containers which is "running" disappears. The "exited takes that container's TimeCreated stamp. I am working on a query that will list all the containers that I need to alert on, and then find a container where nothing is in 'running" state.

Great ideas. I will be sure to post my success, as I am sure somebody else out there will end up in my position someday.

natetech

I think I worked out the logic, thought I would share

SELECT Nodes.Uri, Nodes.DisplayName FROM Orion.Nodes AS Nodes
JOIN (
SELECT Nodeid,Name,State,StartedAt,GetUtcDate() as date, MinuteDiff(StartedAt, GetUtcDate()) as mydiff
FROM Cortex.Orion.Cman.Container CC
Join Orion.NodesCustomProperties CP on CP.NodeId = CC.NodeID
where Name like '%ns-dc%' and Name not like '%POD%' and State='running'
and MinuteDiff(StartedAt, GetUtcDate()) < 5
Order by StartedAt desc
)AS tbl1 ON tbl1.NodeID = Nodes.NodeID

natetech