Monitoring Central

2 Posts authored by: ikongf Employee

What are some common problems that can be detected with the handy router logs on Heroku? We’ll explore them and show you how to address them easily and quickly with monitoring of Heroku from SolarWinds Papertrail.

 

One of the first cloud platforms, Heroku is a popular platform as a service (PaaS) that has been in development since June 2007. It allows developers and DevOps specialists to easily deploy, run, manage, and scale applications written in Ruby, Node.js, Java, Python, Clojure, Scala, Go, and PHP.

 

To learn more about Heroku, head to the Heroku Architecture documentation.

 

Intro to Heroku Logs

Logging in Heroku is modular, similar to gathering system performance metrics. Logs are time-stamped events that can come from any of the processes running in all application containers (Dynos), system components, or backing services. Log streams are aggregated and fed into the Logplex—a high-performance, real-time system for log delivery into a single channel.

 

Run-time activity, as well as dyno restarts and relocations, can be seen in the application logs. This will include logs generated from within application code deployed on Heroku, services like the web server or the database, and the app’s libraries. Scaling, load, and memory usage metrics, among other structural events, can be monitored with system logs. Syslogs collect messages about actions taken by the Heroku platform infrastructure on behalf of your app. These are two of the most recurrent types of logs available on Heroku.

 

To fetch logs from the command line, we can use the heroku logs command. More details on this command, such as output format, filtering, or ordering logs, can be found in the Logging article of Heroku Devcenter.

$ heroku logs 2019-09-16T15:13:46.677020+00:00 app[web.1]: Processing PostController#list (for 208.39.138.12 at 2010-09-16 15:13:46) [GET] 2018-09-16T15:13:46.677902+00:00 app[web.1]: Rendering post/list 2018-09-16T15:13:46.698234+00:00 app[web.1]: Completed in 74ms (View: 31, DB: 40) | 200 OK [http://myapp.heroku.com/] 2018-09-16T15:13:46.723498+00:00 heroku[router]: at=info method=GET path='/posts' host=myapp.herokuapp.com' fwd='204.204.204.204' dyno=web.1 connect=1ms service=18ms status=200 bytes=975   # © 2018 Salesforce.com. All rights reserved.

Heroku Router Logs

Router logs are a special case of logs that exist somewhere between the app logs and the system logs—and are not fully documented on the Heroku website at the time of writing. They carry information about HTTP routing within Heroku Common Runtime, which manages dynos isolated in a single multi-tenant network. Dynos in this network can only receive connections from the routing layer. These routes are the entry and exit points of all web apps or services running on Heroku dynos.

 

Tail router only logs with the heroku logs -tp router CLI command.

$ heroku logs -tp router 2018-08-09T06:24:04.621068+00:00 heroku[router]: at=info method=GET path='/db' host=quiet-caverns-75347.herokuapp.com request_id=661528e0-621c-4b3e-8eef-74ca7b6c1713 fwd='104.163.156.140' dyno=web.1 connect=0ms service=17ms status=301 bytes=462 protocol=https 2018-08-09T06:24:04.902528+00:00 heroku[router]: at=info method=GET path='/db/' host=quiet-caverns-75347.herokuapp.com request_id=298914ca-d274-499b-98ed-e5db229899a8 fwd='104.163.156.140' dyno=web.1 connect=1ms service=211ms status=200 bytes=3196 protocol=https 2018-08-09T06:24:05.002308+00:00 heroku[router]: at=info method=GET path='/stylesheets/main.css' host=quiet-caverns-75347.herokuapp.com request_id=43fac3bb-12ea-4dee-b0b0-2344b58f00cf fwd='104.163.156.140' dyno=web.1 connect=0ms service=3ms status=304 bytes=128 protocol=https 2018-08-09T08:37:32.444929+00:00 heroku[router]: at=info method=GET path='/' host=quiet-caverns-75347.herokuapp.com request_id=2bd88856-8448-46eb-a5a8-cb42d73f53e4 fwd='104.163.156.140' dyno=web.1 connect=0ms service=127ms status=200 bytes=7010 protocol=https   # Fig 1. Heroku router logs in the terminal

Heroku routing logs always start with a timestamp and the “heroku[router]” source/component string, and then a specially formatted message. This message begins with either “at=info”, “at=warning”, or “at=error” (log levels), and can contain up to 14 other detailed fields such as:

  • Heroku error “code” (Optional) – For all errors and warning, and some info messages; Heroku-specific error codes that complement the HTTP status codes.
  • Error “desc” (Optional) – Description of the error, paired to the codes above
  • HTTP request “method” e.g. GET or POST – May be related to some issues
  • HTTP request “path” – URL location for the request; useful for knowing where to check on the application code
  • HTTP request “host” – Host header value
  • The Heroku HTTP Request ID – Can be used to correlate router logs to application logs;
  • HTTP request “fwd” – X-Forwarded-For header value;
  • Which “dyno” serviced the request – Useful for troubleshooting specific containers
  • “Connect” time (ms) spent establishing a connection to the web server(s)
  • “Service” time (ms) spent proxying data between the client and the web server(s)
  • HTTP response code or “status” – Quite informative in case of issues;
  • Number of “bytes” transferred in total for this web request;

 

Common Problems Observed with Router Logs

Examples are manually color-coded in this article. Typical ways to address the issues shown above are also provided for context.

 

Common HTTP Status Codes

404 Not Found Error

Problem: Error accessing nonexistent paths (regardless of HTTP method):

2018-07-30T17:10:18.998146+00:00 heroku[router]: at=info method=POST path='/saycow' host=heroku-app-log.herokuapp.com request_id=e5634f81-ec54-4a30-9767-bc22365a2610 fwd='187.220.208.152' dyno=web.1 connect=0ms service=15ms status=404 bytes=32757 protocol=https 2018-07-27T22:09:14.229118+00:00 heroku[router]: at=info method=GET path='/irobots.txt' host=heroku-app-log.herokuapp.com request_id=7a32a28b-a304-4ae3-9b1b-60ff28ac5547 fwd='187.220.208.152' dyno=web.1 connect=0ms service=31ms status=404 bytes=32769 protocol=https

Solution: Implement or change those URL paths in the application or add the missing files.

500 Server Error

Problem: There’s a bug in the application:

2018-07-31T16:56:25.885628+00:00 heroku[router]: at=info method=GET path='/' host=heroku-app-log.herokuapp.com request_id=9fb92021-6c91-4b14-9175-873bead194d9 fwd='187.220.247.218' dyno=web.1 connect=0ms service=3ms status=500 bytes=169 protocol=https

Solution: The application logs have to be examined to determine the cause of the internal error in the application’s code. Note that HTTP Request IDs can be used to correlate router logs against the web dyno logs for that same request.

Common Heroku Error Codes

 

Other problems commonly detected by router logs can be explored in the Heroku Error Codes. Unlike HTTP codes, these error codes are not standard and only exist in the Heroku platform. They give more specific information on what may be producing HTTP errors.

H14 – No web dynos running

Problem: App has no web dynos setup:

2018-07-30T18:34:46.027673+00:00 heroku[router]: at=error code=H14 desc='No web processes running' method=GET path='/' host=heroku-app-log.herokuapp.com request_id=b8aae23b-ff8b-40db-b2be-03464a59cf6a fwd='187.220.208.152' dyno= connect= service= status=503 bytes= protocol=https

Notice that the above case is an actual error message, which includes both Heroku error code H14 and a description. HTTP 503 means “service currently unavailable.”

Note that Heroku router error pages can be customized. These apply only to errors where the app doesn’t respond to a request e.g. 503.

Solution: Use the heroku ps:scale command to start the app’s web server(s).

 

H12 – Request timeout

Problem: There’s a request timeout (app takes more than 30 seconds to respond):

2018-08-18T07:11:15.487676+00:00 heroku[router]: at=error code=H12 desc='Request timeout' method=GET path='/sleep-30' host=quiet-caverns-75347.herokuapp.com request_id=1a301132-a876-42d4-b6c4-a71f4fe02d05 fwd='189.203.188.236' dyno=web.1 connect=1ms service=30001ms status=503 bytes=0 protocol=https

Error code H12 indicates the app took over 30 seconds to respond to the Heroku router.

Solution: Code that requires more than 30 seconds must run asynchronously (e.g., as a background job) in Heroku. For more info read Request Timeout in the Heroku DevCenter.

H18 – Server Request Interrupted

Problem: The Application encountered too many requests (server overload):

2018-07-31T18:52:54.071892+00:00 heroku[router]: sock=backend at=error code=H18 desc='Server Request Interrupted' method=GET path='/' host=heroku-app-log.herokuapp.com request_id=3a38b360-b9e6-4df4-a764-ef7a2ea59420 fwd='187.220.247.218' dyno=web.1 connect=0ms service=3090ms status=503 bytes= protocol=https

Solution: This problem may indicate that the application needs to be scaled up, or the app performance improved.

H80 – Maintenance mode

Problem: Maintenance mode generates an info router log with error code H18:

2018-07-30T19:07:09.539996+00:00 heroku[router]: at=info code=H80 desc='Maintenance mode' method=GET path='/' host=heroku-app-log.herokuapp.com request_id=1b126dca-1192-4e98-a70f-78317f0d6ad0 fwd='187.220.208.152' dyno= connect= service= status=503 bytes= protocol=https

Solution: Disable maintenance mode with heroku maintenance:off

 

Papertrail

Papertrail™ is a cloud log management service designed to aggregate Heroku app logs, text log files, and syslogs, among many others, in one place. It helps you to monitor, tail, and search logs via a web browser, command-line, or an API. The Papertrail software analyzes log messages to detect trends, and allows you to react instantly with automated alerts.

 

The Event Viewer is a live aggregated log tail with auto-scroll, pause, search, and other unique features. Everything in log messages is searchable, and new logs still stream in real time in the event viewer when searched (or otherwise filtered). Note that Papertrail reformats the timestamp and source in its Event Viewer to make it easier to read.

Viewer Live Pause
Fig 2. The Papertrail Event Viewer. © 2018 Solarwinds. All rights reserved.

Provisioning Papertrail on your Heroku apps is extremely easy: heroku addons:create papertrail from terminal. (See the Papertrail article in Heroku’s DevCenter for more info.) Once setup, the add-on can be open from the Heroku app’s dashboard (Resources section) or with heroku addons:open papertrail in terminal.

 

Troubleshooting Routing Problems Using Papertrail

A great way to examine Heroku router logs is by using the Papertrail solution. It’s easy to isolate them in order to filter out all the noise from multiple log sources: simply click on the “heroku/router” program name in any log message, which will automatically search for “program:heroku/router” in the Event Viewer:

Heroku router viewer
Fig 3. Tail of Heroku router logs in Papertrail, 500 app error selected. © 2018 SolarWinds. All rights reserved.

 

Monitor HTTP 404s

How do you know that your users are finding your content, and that it’s up to date? 404 Not Found errors are what a client receives when the URL’s path is not found. Examples would be a misspelled file name or a missing app route. We want to make sure these types of errors remain uncommon, because otherwise, users are either walking to dead ends or seeing irrelevant content in the app!

 

With Papertrail, setting up an alert to monitor the amount of 404s returned by your app is easy and convenient. One way to do it is to search for “status=404” in the Event Viewer, and then click on the Save Search button. This will bring up the Save Search popup, along with the Save & Setup Alert option:

Save a search
Fig 4. Save a log search and set up an alert with a single action © 2018 SolarWinds. All rights reserved.

 

The following screen will give us the alert delivery options, such as email, Slack message, push notifications, or even publish all matching events as a custom metric for application performance management tools such as AppOptics™.

Troubleshoot 500 errors quickly

500 error on Heroku
Fig 5. HTTP 500 Internal Server Error from herokuapp.com. © 2018 Google LLC. All rights reserved.

 

Let’s say an HTTP 500 error is happening on your app after it’s deployed. A great feature of Papertrail is to make the request_id in log messages clickable. Simply click on it or copy it and search it in the Event Viewer to find all the app logs that are causing the internal problem, along with the detailed error message from your application’s code.

 

Conclusion

Heroku router logs are the glue between web traffic and (sometimes intangible) errors in your application code. It makes sense to give them special focus when monitoring a wide range of issues because they often indicate customer-facing problems that we want to avoid or address ASAP. Add the Papertrail addon to Heroku to get more powerful ways to monitor router logs.

 

Sign up for a 30-day free trial of Papertrail and start aggregating logs from all your Heroku apps and other sources. You may learn more about the Papertrail advanced features in its Heroku Dev Center article.

DevOps engineers wishing to troubleshoot Kubernetes applications can turn to log messages to pinpoint the cause of errors and their impact on the rest of the cluster. When troubleshooting a running application, engineers need real-time access to logs generated across multiple components.

 

Collecting live streaming log data lets engineers:

  • Review container and pod activity
  • Monitor the result of actions, such as creating or modifying a deployment
  • Understand the interactions between containers, pods, and Kubernetes
  • Monitor ingress resources and requests
  • Troubleshoot errors and watch for new or recurring problems

 

The challenge that engineers face is accessing comprehensive, live streams of Kubernetes log data. While some solutions exist today, these are limited in their ability to live tail logs or tail multiple logs. In this article, we’ll present an all-in-one solution for live tailing your Kubernetes logs, no matter the size or complexity of your cluster.

 

The Limitations of Current Logging Solutions

When interacting with Kubernetes logs, engineers frequently use two solutions: the Kubernetes command line interface (CLI), or the Elastic Stack.

 

The Kubernetes CLI (kubectl) is an interactive tool for managing Kubernetes clusters. The default logging tool is the command (kubectl logs) for retrieving logs from a specific pod or container. Running this command with the --follow flag streams logs from the specified resource, allowing you to live tail its logs from your terminal.

 

For example, let’s deploy a Nginx pod under the deployment name papertrail-demo. Using kubectl logs --follow [Pod name], we can view logs from the pod in real time:

$ kubectl logs --follow papertrail-demo-76bf4969df-9gs5w 10.1.1.1 - - [04/Jan/2019:22:42:11 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"

The main limitation of kubectl logs is that it only supports individual Pods. If we deployed two Nginx pod replicas instead of one, we would need to tail each pod separately. For large deployments, this could involve dozens or hundreds of separate kubectl logs instances.

 

The Elastic Stack (previously the ELK Stack) is a popular open-source log management solution. Although it can ingest and display log data using a web-based user interface, unfortunately, it doesn’t offer support for live tailing logs.

 

What is Papertrail, and How Does It Help?

SolarWinds® Papertrail is a cloud-hosted log management solution that lets you live tail your logs from a central location. Using Papertrail, you can view real-time log events from your entire Kubernetes cluster in a single browser window.

 

When a log event is sent from Kubernetes to Papertrail, Papertrail records the log’s contents along with its timestamp and origin pod. You can view these logs in a continuous stream in your browser using the Papertrail Event Viewer, as well as the Papertrail CLI client or Papertrail HTTP API. Papertrail shows all logs by default, but you can limit these to a specific pod, node, or deployment using a flexible search syntax.

 

For example, let’s increase the number of replicas in our Nginx deployment to three. If we used kubectl logs -f, we would need to run it three times: one for each pod. With Papertrail, we can open the Papertrail Event Viewer and create a search that filters the stream to logs originating from the papertrail-demo deployment. Not only does this show us output from each pod in the deployment, but also Kubernetes cluster activity related to each pod:


Filtering a live stream of Kubernetes logs using Papertrail.

 

Sending Logs from Kubernetes to Papertrail

The most effective way to send logs from Kubernetes to Papertrail is via a DaemonSet. DaemonSets run a single instance of a pod on each node in the cluster. The pod used in the DaemonSet automatically collects and forwards log events from other pods, Kubernetes, and the node itself to Papertrail.

 

Papertrail provides two DaemonSets:

  • The Fluentd DaemonSet uses Fluentd to collect logs from containers, pods, Kubernetes, and nodes. This is the preferred method for logging a cluster.
  • The Logspout DaemonSet uses logspout to monitor the Docker log stream. This option is limited to log output from containers, not Kubernetes or nodes.

We’ll demonstrate using the Fluentd DaemonSet. From a computer with kubectl installed, download fluentd-daemonset-papertrail.yaml and open it in a text editor. Change the values of FLUENT_PAPERTRAIL_HOST and FLUENT_PAPERTRAIL_PORT to match your Papertrail log destination. Optionally, you can name your instance by changing FLUENT_HOSTNAME. You can also change the Kubernetes namespace that the DaemonSet runs in by changing the namespace parameter. When you are done, deploy the DaemonSet by running:

$ kubectl create -f fluentd-daemonset-papertrail.yaml

In a few moments, logs will start to appear in Papertrail:


Live feed of Kubernetes logs in Papertrail.

 

Best Practices for Live Tailing Kubernetes Logs

To get the most out of your logs, make sure you’re following these best practices.

 

Log All Applications to STDOUT and STDERR

Kubernetes collects logs from Pods by monitoring their STDOUT and STDERR streams. If your application logs to another location, such as a file or remote service, Kubernetes won’t be able to detect it, and neither will your Papertrail DaemonSet. When deploying an application, make sure to route its logs to the standard output stream.

 

Use the Fluentd DaemonSet

The Logspout DaemonSet is limited to logging containers. The Fluentd DaemonSet, however, will log your containers, pods, and nodes. In addition to logging more resources, Fluentd also logs valuable information such as Pod names, Pod controller activity, and Pod scheduling activity.

 

Open Papertrail Next to Your Terminal

When you’re working on Kubernetes apps and want to debug problems with Pods, have a browser window with Papertrail open either beside or behind your terminal window. This way you can see the results of your actions after you execute them. This also saves you from having to tail manually in your terminal.

 

Group Logs to Make Them Easier to Find

Kubernetes pods (and containers in general) are ephemeral and often have randomly generated names. Unless you specify fixed names, it can be hard to keep track of which pods or containers to filter on. A solution is to use log groups, which let you group logs from a specific application or development team together. This helps you find the logs you need and hide everything else.

 

Save Searches in Papertrail

Papertrail lets you save your searches for creating custom Event Viewer sessions and alerts. You can reopen previously created live tail sessions, share your sessions with team members, or receive an instant notification when new log events arrive in the stream.

 

Conclusion

Kubernetes logs help DevOps teams identify deployment problems and improve the reliability of their application . Live tailing enables faster troubleshooting by helping developers collect, view, and analyze these logs in real time. To get started in SolarWinds Papertrail, sign up and start logging your Kubernetes cluster in a matter of minutes.

SolarWinds uses cookies on its websites to make your online experience easier and better. By using our website, you consent to our use of cookies. For more information on cookies, see our cookie policy.