How to Improve Your Experience With Log Analyzer and Orion Log Viewer

In this blog post we’ll talk about tips to improve your experience with SolarWinds^® Log Analyzer and its lightweight version, Orion^® Log Viewer.

Log Analyzer is intended for smaller volumes of event ingestion, ranging at around 1,000 events per second (EPS), with short bursts of up to 3,000 EPS. This is regardless of the number of monitored devices and polling engines.

Update to Latest Version

Make sure to update to the latest version—2020.2.6 at the time of writing—as we’re continuously working on improvements. Starting with version 2020.2.5, we’ve significantly modified our event processing pipeline and addressed multiple memory utilization-related issues. The new processing pipeline should be more stable and better equipped to handle short bursts of events.

Database Server Configuration

A consistent load of 1,000 EPS means 86.4 million records per day. For the purposes of scaling, the database storage is on average about 120GB of data for every day of events being stored. Over seven days, which is the default retention period, this means around 1TB of storage. As you can imagine, the higher the amount of data, the more strain it’ll put on the SQL Server instance to query it. Based on our internal performance testing, for this load, this is the minimum recommended hardware configuration for SQL Server hosting only a Log Analyzer database:

• Eight-core processor
• At least 32GB of memory allocated to SQL Server
• Three dedicated SSD drives (or equivalent storage with high IOPS and low latency)
  • SQL data
  • SQL log
  • SQL temp
• Installed on physical hardware

Comparing this to the minimum requirement of quad-core with 16GB ram allocated for SQL Server, we’ve seen about a 50 – 80% decrease in response time in Log Viewer.

Of course, other best practices applicable to the SQL Server instance used by the Orion Platform are applicable here as well. This includes properly configuring max degree of parallelism, data file auto-growth, and more. You can read more about these here.

Dropping Surplus Events

Many devices are sending events with low information value that don’t need to be persisted (DEBUG-level syslog messages, for example). Such events can be discarded simply by configuring processing rules. Dropping these surplus events should allow for higher ingestion rates as long as the number of stored events in the database will still be at the 1,000 EPS level. For example, if there are 10 devices each sending 200 EPS and we drop 50% of the incoming messages without storing them, then we’re still within the recommended limits. This is true even though there are overall 2,000 EPS sent to Log Analyzer. This is because the processing pipelines can process up to 3,000 EPS, with short bursts of up to 5,000 EPS.

To create such a rule, first we configure the condition specifying which events should be discarded.

The next step is to configure two log entry actions:

The first action will cause the message to be discarded instead of stored. The second will ensure we stop processing any other rules for this specific message—this will cause the message to be discarded immediately.

Disable Filter Count

In a situation where there’s already many events stored in the database and the responsiveness of Log Viewer is starting to degrade, it can help to disable filter counts. This will eliminate some of the database-heavy queries and can lead to better performance.

The setting, which is disabled by default, is available in the advanced configuration under the key LogViewerShowAllFilters. After enabling the setting, the filter counts will no longer be provided, and all the filters will be available by default.

Windows Events

In a situation where the Windows event collection and processing is getting backed up on the agent side, it’s usually because the Windows Event Log API we’re using is slow at formatting messages. To retain the order of events, we perform this formatting in only one thread. If the event order isn’t required, it’s possible to increase the amount of processing threads. This can allow for faster processing. Here’s how you can do so:

1. Log in to the agent server
2. Edit C:\Program Files (x86)\SolarWinds\Agent\Plugins\LogManager\SolarWinds.Orion.LogMgmt.Agent.Plugin.exe.config and under the appSettings section, add a new item with key MaxFormattingThreads and set the value to the number of threads that should be used. The recommended number is up to the number of logical processors on the machine. The following is an example of how the section might look with the change applied:
   

   <appSettings>
        <add key="ClientSettingsProvider.ServiceUri" value=""/>
        <add key="MaxFormattingThreads" value="4"/> 
   </appSettings>

3. Save the file and restart the SolarWinds agent service

If you’re looking for a tool capable of handling larger volumes of event ingestion, we recommend looking at our software as a service (SaaS) offerings (SolarWinds Papertrail or SolarWinds Loggly^®) or SolarWinds Security Event Manager for an on-premises solution.