Monitoring Central

4 Posts authored by: ikongf Employee

It’s easy to recognize problems in Ruby on Rails, but finding each problem’s source can be a challenging task. A problem due to an unexpected event could result in hours of searching through log files and attempting to reproduce the issue. Poor logs will leave you searching, while a helpful log can assist you in finding the cause right away.

 

Ruby on Rails applications automatically create and maintain the basic text logs for each environment, such as development, staging, and production. You can easily format and add extra information to the logs using open-source logging libraries, such as Lograge and Fluentd. These libraries effectively manage small applications, but as you scale your application across many servers, developers need to aggregate logs to troubleshoot problems across all of them.

 

In this tutorial, we will show you how Ruby on Rails applications handle logging natively. Then, we’ll show you how to send the logs to SolarWinds® Papertrail™. This log management solution enables you to centralize your logs in the cloud and provides helpful features like fast search, alerts, and more.

 

Ruby on Rails Native Logging

Ruby offers a built-in logging system. To use it, simply include the following code snippet in your environment.rb (development.rb/production.rb). You can find environments under the config directory of the root project.

config.logger = Logger.new(STDOUT)

Or you can include the following in an initializer:

Rails.logger = Logger.new(STDOUT)

By default, each log is created under #{Rails.root}/log/ and the log file is
named after the environment in which the application is running. The default format gives basic information that includes the date/time of log generation and description (message or exception) of the log.

D, [2018-08-31T14:12:44.116332 #28944] DEBUG -- : Debug message I, [2018-08-31T14:12:44.117330 #28944]  INFO -- : Test message F, [2018-08-31T14:12:44.118348 #28944] FATAL -- : Terminating application, raised unrecoverable error!!! F, [2018-08-31T14:12:44.122350 #28944] FATAL -- : Exception (something bad happened!):

Each log line also includes the severity, otherwise known as log level. The log levels enable you to filter the logs when outputting them or when monitoring problems, such as errors or fatal. The available log levels are :debug, :info, :warn, :error, :fatal, and :unknown. These are
converted to uppercase when output in the log file.

 

Formatting Logs Using Lograge

The default logging in Ruby on Rails during development or in production can be noisy, as you can see below. It also records a limited amount of
information for each page view.

I, [2018-08-31T14:37:44.588288 #27948]  INFO -- : method=GET path=/ format=html controller=Rails::WelcomeController action=index status=200 duration=105.06 view=51.52 db=0.00 params={'controller'=>'rails/welcome', 'action'=>'index'} headers=#<ActionDispatch::Http::Headers:0x046ab950> view_runtime=51.52 db_runtime=0

Lograge adds extra detail and uses a format that is less human readable, but more useful for large-scale analysis through its JSON output option. JSON makes it easier to search, filter, and summarize large volumes of logs. The discrete fields facilitate the process of searching through logs and filtering for the information you need.

I, [2018-08-31T14:51:54.603784 #17752]  INFO -- : {'method':'GET','path':'/','format':'html','controller':'Rails::WelcomeController','action':'index','status':200,'duration':104.06,'view':51.99,'db':0.0,'params':{'controller':'rails/welcome','action':'index'},'headers':'#<ActionDispatch::Http::Headers:0x03b75520>','view_runtime':51.98726899106987,'db_runtime':0}

In order to configure Lograge in a Ruby on Rails app, you need to follow some simple steps:

Step 1: Find the Gemfile under the project root directory and add the following gem.

gem 'lograge'

Step 2: Enable Lograge in each relevant environment (development, production, staging) or in the initializer. You can find all those environments under the config directory of your project. To find the initializer, open up the config directory of your project.

# config/initializers/lograge.rb# OR# config/environments/production.rbRails.application.configure do  config.lograge.enabled = trueend

Step 3: If you’re using Rails 5’s API-only mode and inherit from ActionController::API, you must define it as the controller base class that Lograge will patch:

# config/initializers/lograge.rbRails.application.configure do  config.lograge.base_controller_class = 'ActionController::API'end

With Lograge, you can include additional attributes in log messages, like user ID or request ID, host, source IP, etc. You can read the Lograge documentation to get more information.


Here’s a simple example that captures three attributes:

class ApplicationController < ActionController::Base  before_action :append_info_to_payload  def append_info_to_payload(payload)    super    payload[:user_id] = current_user.try(:id)    payload[:host] = request.host    payload[:source_ip] = request.remote_ip  endend

The above three attributes are logged in environment.rb (production.rb/development.rb) with this block.

config.lograge.custom_options = lambda do |event|  event.payloadend

Troubleshoot Problems Faster Using Papertrail

Papertrail is a popular cloud-hosted log management service that integrates with different logging library solutions. It is easier to centralize all your Ruby on Rails log management in the cloud. You can quickly track real-time activity, making it easier to identify and troubleshoot real-time production applications.

 

Papertrail provides numerous features for handling Ruby on Rails log files, including:

 

Instant log visibility: Papertrail provides fast search and team-wide
access. It also provides analytics reporting and webhook monitoring, which
can be set up typically in less than a minute.

 

Aggregate logs: : Papertrail aggregates logs across your entire deployment, making them available from a single location. It provides you with an easy way to access logs, including application logs, database logs, Apache logs, and more.

2018-10-03-viewer

 

Tail and search logs: Papertrail lets you tail logs in real time from
multiple devices. With the help of advanced searching and filtering tools, you can quickly troubleshoot issues in a production environment.

 

Proactive alert notifications: Almost every application has critical events
that require human attention. That’s precisely why alerts exist. Papertrail gives you the ability to receive alerts via email, Slack, Librato®, PagerDuty, or any custom HTTP webhooks of your choice.

2018-10-03-edit-alert

 

Log archives: You can load the Papertrail log archives into third-party utilities, such as Redshift or Hadoop.

 

Logs scalability: With Papertrail, you can scale your log volume and desired searchable duration.

 

Encryption: For your security, Papertrail supports optional TLS encryption
and certificate-based destination host verification.

Configuring Ruby on Rails to Send Logs to Papertrail

It’s an easy task to get started with Papertrail. If you already have log files,
you can send them to Papertrail using Nxlog or remote_syslog2. This utility will monitor the log files and send new logs to Papertrail. Next, we’ll show you how to send events asynchronously from Ruby on Rails using the remote_syslog_logger.

Add the remote_syslog_logger to your Gemfile. If you are not using a Gemfile, run the following script:

$ gem install remote_syslog_logger

Change the environment configuration file to log via remote_syslog_logger. This is almost always in config/environment.rb (to affect all environments) or config/environments/<environment name>.rb, such as config/environments/production.rb (to affect only a specific environment). Update the host and port to the ones given to you in your Papertrail log destination settings.

config.logger = RemoteSyslogLogger.new('logsN.papertrailapp.com', XXXXX)

It’s that simple! Your logs should now be sent to Papertrail.

 

Papertrail is designed to help you troubleshoot customer problems, resolve error messages, improve slow database queries, and more. It gives you analytical tools to help identify and resolve system anomalies and potential security issues. Learn more about how Papertrail can give you frustration-free log management in the cloud, and sign up for a trial or the free plan to get started.

ikongf

Papertrail for Python Logs

Posted by ikongf Employee Apr 27, 2019

When you’re troubleshooting a problem or tracking down a bug in Python, the first place to look for clues related to server issues is in the application log files.

 

Python includes a robust logging module in the standard library, which provides a flexible framework for emitting log messages. This module is widely used by various Python libraries and is an important reference point for most programmers when it comes to logging.

 

The Python logging module provides a way for applications to configure different log handlers and provides a standard way to route log messages to these handlers. As the Python.org documentation notes, there are four basic classes defined by the Python logging module: Loggers, Handlers, Filters, and Formatters. We’ll provide more details on these below.

 

Getting Started with Python Logs

There are a number of important steps to take when setting up your logs. First, you need to ensure logging is enabled in the applications you use. You also need to categorize your logs by name so they are easy to maintain and search. Naming the logs makes it easier to search through large log files, and to use filters to find the information you need.

 

To send log messages in Python, request a logger object. It should have a unique name to help filter and prioritize how your Python application handles various messages. We are also adding a StreamHandler to print the log on our console output. Here’s a simple example:

import logginglogging.basicConfig(handlers=[logging.StreamHandler()])log = logging.getLogger('test')log.error('Hello, world')

This outputs:

ERROR:test:Hello, world

This message consists of three fields. The first, ERROR, is the log level. The second, test, is the logger name. The third field, “Hello, world”, is the free-form log message.

 

Most problems in production are caused by unexpected or unhandled issues. In Python, such problems generate tracebacks where the interpreter tries to include all important information it could gather. This can sometimes make the traceback a bit hard to read, though. Let’s look at an example traceback. We’ll call a function that isn’t defined and examine the error message.

def test():    nofunction()test()

Which outputs:

Traceback (most recent call last):   File '<stdin>', line 1, in <module>   File '<stdin>', line 2, in test NameError: global name 'nofunction' is not defined

This shows the common parts of a Python traceback. The error message is usually at the end of the traceback. It says “nofunction is not defined,” which is what we expected. The traceback also includes the lines of all stack frames that were touched when this error occurred. Here we can see that it occurred in the test function on line two. Stdin means standard input and refers to the console where we typed this function. If we were using a Python source file, we’d see the file name here instead.

 

Configuring Logging

You should configure the logging module to direct messages to go where you want them. For most applications, you will want to add a Formatter and a Handler to the root logger. Formatters let you specify the fields and timestamps in your logs. Handlers let you define where they are sent. To set these up, Python provides a nifty factory function called basicConfig.

import logginglogging.basicConfig(format='%(asctime)s %(message)s',                  handlers=[logging.StreamHandler()])logging.debug('Hello World!')

By default, Python will output uncaught exceptions to your system’s standard error stream. Alternatively, you could add a handler to the excepthook to send any exceptions through the logging module. This gives you the flexibility to provide custom formatters and handlers. For example, here we log our exceptions to a log file using the FileHandler:

import loggingimport syslogger = logging.getLogger('test')fileHandler = logging.FileHandler('errors.log')logger.addHandler(fileHandler)def my_handler(type, value, tb):  logger.exception('Uncaught exception: {0}'.format(str(value)))# Install exception handlersys.excepthook = my_handler# Throw an errornofunction()

Which results in the following log output:

$ cat errors.log Uncaught exception: name 'nofunction' is not defined None

In addition, you can filter logs by configuring the log level. One way to set the log level is through an environment variable, which gives you the ability to customize the log level in the development or production environment. Here’s how you can use the LOGLEVEL environment variable:

$ export LOGLEVEL='ERROR' $ python >>> import logging >>> logging.basicConfig(handlers=[logging.StreamHandler()]) >>> logging.debug('Hello World!') #prints nothing >>> logging.error('Hello World!') ERROR:root:Hello World!

Logging from Modules

Modules intended for use by other programs should only emit log messages. These modules should not configure how log messages are handled. A standard logging best practice is to let the Python application importing and using the modules handle the configuration.

Another standard best practice to follow is that each module should use a logger named like the module itself. This naming convention makes it easy for the application to distinctly route various modules and helps keep the log code in the module simple.

You need just two lines of code to set up logging using the named logger. Once you do this in Python, the “ name” contains the full name of the current module, and will work in any module. Here’s an example:

import logginglog = logging.getLogger(__name__)def do_something():    log.debug('Doing something!')

Analyzing Your Logs with Papertrail

Python applications on a production server contain millions of lines of log entries. Command line tools like tail and grep are often useful during the development process. However, they may not scale well when analyzing millions of log events spread across multiple servers.

 

Centralized logging can make it easier and faster for developers to manage a large volume of logs. By consolidating log files onto one integrated platform, you can eliminate the need to search for related data that is split across multiple apps, directories, and servers. Also, a log management tool can alert you to critical issues, helping you more quickly identify the root cause of unexpected errors, as well as bugs that may have been missed earlier in the development cycle.

 

For production-scale logging, a log management tool such as Solarwinds® Papertrail™ can help you better manage your data. Papertrail is a cloud-based platform designed to handle logs from any Python application, including Django servers.

 

The Papertrail solution provides a central repository for event logs. It helps you consolidate all of your Python logs using syslog, along with other application and database logs, giving you easy access all in one location. It offers a convenient search interface to find relevant logs. It can also stream logs to your browser in real time, offering a “live tail” experience. Check out the tour of the Papertrail solution’s features.

2018-09-07-viewer

Papertrail is designed to help minimize downtime. You can receive alerts via email, or send them to Slack, Librato, PagerDuty, or any custom HTTP webhooks of your choice. Alerts are also accessible from a web page that enables customized filtering. For example, you can filter by name or tag.

2018-09-07-edit-alert

 

Configuring Papertrail in Your Application

There are many ways to send logs to Papertrail depending on the needs of your application. You can send logs through journald, log files, Django, Heroku, and more. We will review the syslog handler below.

 

Python can send log messages directly to Papertrail with the Python SysLogHandler. Just set the endpoint to the log destination shown in your Papertrail settings. You can optionally format the timestamp or set the log level as shown below.

import loggingimport socketfrom logging.handlers import SysLogHandlersyslog = SysLogHandler(address=('logsN.papertrailapp.com', XXXXX))format = '%(asctime)s YOUR_APP: %(message)s'formatter = logging.Formatter(format, datefmt='%b %d %H:%M:%S')syslog.setFormatter(formatter)logger = logging.getLogger()logger.addHandler(syslog)logger.setLevel(logging.INFO)def my_handler(type, value, tb):  logger.exception('Uncaught exception: {0}'.format(str(value)))# Install exception handlersys.excepthook = my_handlerlogger.info('This is a message')nofunction() #log an uncaught exception

Conclusion

Python offers a well-thought-out framework for logging that makes it simple to enable and manage your log files. Getting started is easy, and a number of tools baked into Python automate the logging process and help ensure ease of use.

Papertrail adds even more functionality and tools for diagnostics and analysis, enabling you to manage your Python logs on a centralized cloud server. Quick to setup and easy to use, Papertrail consolidates your logs on a safe and accessible platform. It simplifies your ability to search log files, analyze them, and then act on them in real time—so that you can focus on debugging and optimizing your applications.

Learn more about how Papertrail can help optimize your development ecosystem.

What are some common problems that can be detected with the handy router logs on Heroku? We’ll explore them and show you how to address them easily and quickly with monitoring of Heroku from SolarWinds Papertrail.

 

One of the first cloud platforms, Heroku is a popular platform as a service (PaaS) that has been in development since June 2007. It allows developers and DevOps specialists to easily deploy, run, manage, and scale applications written in Ruby, Node.js, Java, Python, Clojure, Scala, Go, and PHP.

 

To learn more about Heroku, head to the Heroku Architecture documentation.

 

Intro to Heroku Logs

Logging in Heroku is modular, similar to gathering system performance metrics. Logs are time-stamped events that can come from any of the processes running in all application containers (Dynos), system components, or backing services. Log streams are aggregated and fed into the Logplex—a high-performance, real-time system for log delivery into a single channel.

 

Run-time activity, as well as dyno restarts and relocations, can be seen in the application logs. This will include logs generated from within application code deployed on Heroku, services like the web server or the database, and the app’s libraries. Scaling, load, and memory usage metrics, among other structural events, can be monitored with system logs. Syslogs collect messages about actions taken by the Heroku platform infrastructure on behalf of your app. These are two of the most recurrent types of logs available on Heroku.

 

To fetch logs from the command line, we can use the heroku logs command. More details on this command, such as output format, filtering, or ordering logs, can be found in the Logging article of Heroku Devcenter.

$ heroku logs 2019-09-16T15:13:46.677020+00:00 app[web.1]: Processing PostController#list (for 208.39.138.12 at 2010-09-16 15:13:46) [GET] 2018-09-16T15:13:46.677902+00:00 app[web.1]: Rendering post/list 2018-09-16T15:13:46.698234+00:00 app[web.1]: Completed in 74ms (View: 31, DB: 40) | 200 OK [http://myapp.heroku.com/] 2018-09-16T15:13:46.723498+00:00 heroku[router]: at=info method=GET path='/posts' host=myapp.herokuapp.com' fwd='204.204.204.204' dyno=web.1 connect=1ms service=18ms status=200 bytes=975   # © 2018 Salesforce.com. All rights reserved.

Heroku Router Logs

Router logs are a special case of logs that exist somewhere between the app logs and the system logs—and are not fully documented on the Heroku website at the time of writing. They carry information about HTTP routing within Heroku Common Runtime, which manages dynos isolated in a single multi-tenant network. Dynos in this network can only receive connections from the routing layer. These routes are the entry and exit points of all web apps or services running on Heroku dynos.

 

Tail router only logs with the heroku logs -tp router CLI command.

$ heroku logs -tp router 2018-08-09T06:24:04.621068+00:00 heroku[router]: at=info method=GET path='/db' host=quiet-caverns-75347.herokuapp.com request_id=661528e0-621c-4b3e-8eef-74ca7b6c1713 fwd='104.163.156.140' dyno=web.1 connect=0ms service=17ms status=301 bytes=462 protocol=https 2018-08-09T06:24:04.902528+00:00 heroku[router]: at=info method=GET path='/db/' host=quiet-caverns-75347.herokuapp.com request_id=298914ca-d274-499b-98ed-e5db229899a8 fwd='104.163.156.140' dyno=web.1 connect=1ms service=211ms status=200 bytes=3196 protocol=https 2018-08-09T06:24:05.002308+00:00 heroku[router]: at=info method=GET path='/stylesheets/main.css' host=quiet-caverns-75347.herokuapp.com request_id=43fac3bb-12ea-4dee-b0b0-2344b58f00cf fwd='104.163.156.140' dyno=web.1 connect=0ms service=3ms status=304 bytes=128 protocol=https 2018-08-09T08:37:32.444929+00:00 heroku[router]: at=info method=GET path='/' host=quiet-caverns-75347.herokuapp.com request_id=2bd88856-8448-46eb-a5a8-cb42d73f53e4 fwd='104.163.156.140' dyno=web.1 connect=0ms service=127ms status=200 bytes=7010 protocol=https   # Fig 1. Heroku router logs in the terminal

Heroku routing logs always start with a timestamp and the “heroku[router]” source/component string, and then a specially formatted message. This message begins with either “at=info”, “at=warning”, or “at=error” (log levels), and can contain up to 14 other detailed fields such as:

  • Heroku error “code” (Optional) – For all errors and warning, and some info messages; Heroku-specific error codes that complement the HTTP status codes.
  • Error “desc” (Optional) – Description of the error, paired to the codes above
  • HTTP request “method” e.g. GET or POST – May be related to some issues
  • HTTP request “path” – URL location for the request; useful for knowing where to check on the application code
  • HTTP request “host” – Host header value
  • The Heroku HTTP Request ID – Can be used to correlate router logs to application logs;
  • HTTP request “fwd” – X-Forwarded-For header value;
  • Which “dyno” serviced the request – Useful for troubleshooting specific containers
  • “Connect” time (ms) spent establishing a connection to the web server(s)
  • “Service” time (ms) spent proxying data between the client and the web server(s)
  • HTTP response code or “status” – Quite informative in case of issues;
  • Number of “bytes” transferred in total for this web request;

 

Common Problems Observed with Router Logs

Examples are manually color-coded in this article. Typical ways to address the issues shown above are also provided for context.

 

Common HTTP Status Codes

404 Not Found Error

Problem: Error accessing nonexistent paths (regardless of HTTP method):

2018-07-30T17:10:18.998146+00:00 heroku[router]: at=info method=POST path='/saycow' host=heroku-app-log.herokuapp.com request_id=e5634f81-ec54-4a30-9767-bc22365a2610 fwd='187.220.208.152' dyno=web.1 connect=0ms service=15ms status=404 bytes=32757 protocol=https 2018-07-27T22:09:14.229118+00:00 heroku[router]: at=info method=GET path='/irobots.txt' host=heroku-app-log.herokuapp.com request_id=7a32a28b-a304-4ae3-9b1b-60ff28ac5547 fwd='187.220.208.152' dyno=web.1 connect=0ms service=31ms status=404 bytes=32769 protocol=https

Solution: Implement or change those URL paths in the application or add the missing files.

500 Server Error

Problem: There’s a bug in the application:

2018-07-31T16:56:25.885628+00:00 heroku[router]: at=info method=GET path='/' host=heroku-app-log.herokuapp.com request_id=9fb92021-6c91-4b14-9175-873bead194d9 fwd='187.220.247.218' dyno=web.1 connect=0ms service=3ms status=500 bytes=169 protocol=https

Solution: The application logs have to be examined to determine the cause of the internal error in the application’s code. Note that HTTP Request IDs can be used to correlate router logs against the web dyno logs for that same request.

Common Heroku Error Codes

 

Other problems commonly detected by router logs can be explored in the Heroku Error Codes. Unlike HTTP codes, these error codes are not standard and only exist in the Heroku platform. They give more specific information on what may be producing HTTP errors.

H14 – No web dynos running

Problem: App has no web dynos setup:

2018-07-30T18:34:46.027673+00:00 heroku[router]: at=error code=H14 desc='No web processes running' method=GET path='/' host=heroku-app-log.herokuapp.com request_id=b8aae23b-ff8b-40db-b2be-03464a59cf6a fwd='187.220.208.152' dyno= connect= service= status=503 bytes= protocol=https

Notice that the above case is an actual error message, which includes both Heroku error code H14 and a description. HTTP 503 means “service currently unavailable.”

Note that Heroku router error pages can be customized. These apply only to errors where the app doesn’t respond to a request e.g. 503.

Solution: Use the heroku ps:scale command to start the app’s web server(s).

 

H12 – Request timeout

Problem: There’s a request timeout (app takes more than 30 seconds to respond):

2018-08-18T07:11:15.487676+00:00 heroku[router]: at=error code=H12 desc='Request timeout' method=GET path='/sleep-30' host=quiet-caverns-75347.herokuapp.com request_id=1a301132-a876-42d4-b6c4-a71f4fe02d05 fwd='189.203.188.236' dyno=web.1 connect=1ms service=30001ms status=503 bytes=0 protocol=https

Error code H12 indicates the app took over 30 seconds to respond to the Heroku router.

Solution: Code that requires more than 30 seconds must run asynchronously (e.g., as a background job) in Heroku. For more info read Request Timeout in the Heroku DevCenter.

H18 – Server Request Interrupted

Problem: The Application encountered too many requests (server overload):

2018-07-31T18:52:54.071892+00:00 heroku[router]: sock=backend at=error code=H18 desc='Server Request Interrupted' method=GET path='/' host=heroku-app-log.herokuapp.com request_id=3a38b360-b9e6-4df4-a764-ef7a2ea59420 fwd='187.220.247.218' dyno=web.1 connect=0ms service=3090ms status=503 bytes= protocol=https

Solution: This problem may indicate that the application needs to be scaled up, or the app performance improved.

H80 – Maintenance mode

Problem: Maintenance mode generates an info router log with error code H18:

2018-07-30T19:07:09.539996+00:00 heroku[router]: at=info code=H80 desc='Maintenance mode' method=GET path='/' host=heroku-app-log.herokuapp.com request_id=1b126dca-1192-4e98-a70f-78317f0d6ad0 fwd='187.220.208.152' dyno= connect= service= status=503 bytes= protocol=https

Solution: Disable maintenance mode with heroku maintenance:off

 

Papertrail

Papertrail™ is a cloud log management service designed to aggregate Heroku app logs, text log files, and syslogs, among many others, in one place. It helps you to monitor, tail, and search logs via a web browser, command-line, or an API. The Papertrail software analyzes log messages to detect trends, and allows you to react instantly with automated alerts.

 

The Event Viewer is a live aggregated log tail with auto-scroll, pause, search, and other unique features. Everything in log messages is searchable, and new logs still stream in real time in the event viewer when searched (or otherwise filtered). Note that Papertrail reformats the timestamp and source in its Event Viewer to make it easier to read.

Viewer Live Pause
Fig 2. The Papertrail Event Viewer.

Provisioning Papertrail on your Heroku apps is extremely easy: heroku addons:create papertrail from terminal. (See the Papertrail article in Heroku’s DevCenter for more info.) Once setup, the add-on can be open from the Heroku app’s dashboard (Resources section) or with heroku addons:open papertrail in terminal.

 

Troubleshooting Routing Problems Using Papertrail

A great way to examine Heroku router logs is by using the Papertrail solution. It’s easy to isolate them in order to filter out all the noise from multiple log sources: simply click on the “heroku/router” program name in any log message, which will automatically search for “program:heroku/router” in the Event Viewer:

Heroku router viewer
Fig 3. Tail of Heroku router logs in Papertrail, 500 app error selected. © 2018 SolarWinds. All rights reserved.

 

Monitor HTTP 404s

How do you know that your users are finding your content, and that it’s up to date? 404 Not Found errors are what a client receives when the URL’s path is not found. Examples would be a misspelled file name or a missing app route. We want to make sure these types of errors remain uncommon, because otherwise, users are either walking to dead ends or seeing irrelevant content in the app!

 

With Papertrail, setting up an alert to monitor the amount of 404s returned by your app is easy and convenient. One way to do it is to search for “status=404” in the Event Viewer, and then click on the Save Search button. This will bring up the Save Search popup, along with the Save & Setup Alert option:

Save a search
Fig 4. Save a log search and set up an alert with a single action © 2018 SolarWinds. All rights reserved.

 

The following screen will give us the alert delivery options, such as email, Slack message, push notifications, or even publish all matching events as a custom metric for application performance management tools such as AppOptics™.

Troubleshoot 500 errors quickly

500 error on Heroku
Fig 5. HTTP 500 Internal Server Error from herokuapp.com. © 2018 Google LLC. All rights reserved.

 

Let’s say an HTTP 500 error is happening on your app after it’s deployed. A great feature of Papertrail is to make the request_id in log messages clickable. Simply click on it or copy it and search it in the Event Viewer to find all the app logs that are causing the internal problem, along with the detailed error message from your application’s code.

 

Conclusion

Heroku router logs are the glue between web traffic and (sometimes intangible) errors in your application code. It makes sense to give them special focus when monitoring a wide range of issues because they often indicate customer-facing problems that we want to avoid or address ASAP. Add the Papertrail addon to Heroku to get more powerful ways to monitor router logs.

 

Sign up for a 30-day free trial of Papertrail and start aggregating logs from all your Heroku apps and other sources. You may learn more about the Papertrail advanced features in its Heroku Dev Center article.

DevOps engineers wishing to troubleshoot Kubernetes applications can turn to log messages to pinpoint the cause of errors and their impact on the rest of the cluster. When troubleshooting a running application, engineers need real-time access to logs generated across multiple components.

 

Collecting live streaming log data lets engineers:

  • Review container and pod activity
  • Monitor the result of actions, such as creating or modifying a deployment
  • Understand the interactions between containers, pods, and Kubernetes
  • Monitor ingress resources and requests
  • Troubleshoot errors and watch for new or recurring problems

 

The challenge that engineers face is accessing comprehensive, live streams of Kubernetes log data. While some solutions exist today, these are limited in their ability to live tail logs or tail multiple logs. In this article, we’ll present an all-in-one solution for live tailing your Kubernetes logs, no matter the size or complexity of your cluster.

 

The Limitations of Current Logging Solutions

When interacting with Kubernetes logs, engineers frequently use two solutions: the Kubernetes command line interface (CLI), or the Elastic Stack.

 

The Kubernetes CLI (kubectl) is an interactive tool for managing Kubernetes clusters. The default logging tool is the command (kubectl logs) for retrieving logs from a specific pod or container. Running this command with the --follow flag streams logs from the specified resource, allowing you to live tail its logs from your terminal.

 

For example, let’s deploy a Nginx pod under the deployment name papertrail-demo. Using kubectl logs --follow [Pod name], we can view logs from the pod in real time:

$ kubectl logs --follow papertrail-demo-76bf4969df-9gs5w 10.1.1.1 - - [04/Jan/2019:22:42:11 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"

The main limitation of kubectl logs is that it only supports individual Pods. If we deployed two Nginx pod replicas instead of one, we would need to tail each pod separately. For large deployments, this could involve dozens or hundreds of separate kubectl logs instances.

 

The Elastic Stack (previously the ELK Stack) is a popular open-source log management solution. Although it can ingest and display log data using a web-based user interface, unfortunately, it doesn’t offer support for live tailing logs.

 

What is Papertrail, and How Does It Help?

SolarWinds® Papertrail is a cloud-hosted log management solution that lets you live tail your logs from a central location. Using Papertrail, you can view real-time log events from your entire Kubernetes cluster in a single browser window.

 

When a log event is sent from Kubernetes to Papertrail, Papertrail records the log’s contents along with its timestamp and origin pod. You can view these logs in a continuous stream in your browser using the Papertrail Event Viewer, as well as the Papertrail CLI client or Papertrail HTTP API. Papertrail shows all logs by default, but you can limit these to a specific pod, node, or deployment using a flexible search syntax.

 

For example, let’s increase the number of replicas in our Nginx deployment to three. If we used kubectl logs -f, we would need to run it three times: one for each pod. With Papertrail, we can open the Papertrail Event Viewer and create a search that filters the stream to logs originating from the papertrail-demo deployment. Not only does this show us output from each pod in the deployment, but also Kubernetes cluster activity related to each pod:


Filtering a live stream of Kubernetes logs using Papertrail.

 

Sending Logs from Kubernetes to Papertrail

The most effective way to send logs from Kubernetes to Papertrail is via a DaemonSet. DaemonSets run a single instance of a pod on each node in the cluster. The pod used in the DaemonSet automatically collects and forwards log events from other pods, Kubernetes, and the node itself to Papertrail.

 

Papertrail provides two DaemonSets:

  • The Fluentd DaemonSet uses Fluentd to collect logs from containers, pods, Kubernetes, and nodes. This is the preferred method for logging a cluster.
  • The Logspout DaemonSet uses logspout to monitor the Docker log stream. This option is limited to log output from containers, not Kubernetes or nodes.

We’ll demonstrate using the Fluentd DaemonSet. From a computer with kubectl installed, download fluentd-daemonset-papertrail.yaml and open it in a text editor. Change the values of FLUENT_PAPERTRAIL_HOST and FLUENT_PAPERTRAIL_PORT to match your Papertrail log destination. Optionally, you can name your instance by changing FLUENT_HOSTNAME. You can also change the Kubernetes namespace that the DaemonSet runs in by changing the namespace parameter. When you are done, deploy the DaemonSet by running:

$ kubectl create -f fluentd-daemonset-papertrail.yaml

In a few moments, logs will start to appear in Papertrail:


Live feed of Kubernetes logs in Papertrail.

 

Best Practices for Live Tailing Kubernetes Logs

To get the most out of your logs, make sure you’re following these best practices.

 

Log All Applications to STDOUT and STDERR

Kubernetes collects logs from Pods by monitoring their STDOUT and STDERR streams. If your application logs to another location, such as a file or remote service, Kubernetes won’t be able to detect it, and neither will your Papertrail DaemonSet. When deploying an application, make sure to route its logs to the standard output stream.

 

Use the Fluentd DaemonSet

The Logspout DaemonSet is limited to logging containers. The Fluentd DaemonSet, however, will log your containers, pods, and nodes. In addition to logging more resources, Fluentd also logs valuable information such as Pod names, Pod controller activity, and Pod scheduling activity.

 

Open Papertrail Next to Your Terminal

When you’re working on Kubernetes apps and want to debug problems with Pods, have a browser window with Papertrail open either beside or behind your terminal window. This way you can see the results of your actions after you execute them. This also saves you from having to tail manually in your terminal.

 

Group Logs to Make Them Easier to Find

Kubernetes pods (and containers in general) are ephemeral and often have randomly generated names. Unless you specify fixed names, it can be hard to keep track of which pods or containers to filter on. A solution is to use log groups, which let you group logs from a specific application or development team together. This helps you find the logs you need and hide everything else.

 

Save Searches in Papertrail

Papertrail lets you save your searches for creating custom Event Viewer sessions and alerts. You can reopen previously created live tail sessions, share your sessions with team members, or receive an instant notification when new log events arrive in the stream.

 

Conclusion

Kubernetes logs help DevOps teams identify deployment problems and improve the reliability of their application . Live tailing enables faster troubleshooting by helping developers collect, view, and analyze these logs in real time. To get started in SolarWinds Papertrail, sign up and start logging your Kubernetes cluster in a matter of minutes.

SolarWinds uses cookies on its websites to make your online experience easier and better. By using our website, you consent to our use of cookies. For more information on cookies, see our cookie policy.