This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.

You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Syslog stops logging with no notification

ttl over 8 years ago

I discovered this morning (only because I didn't receive the nightly report) that two of our Syslog servers stopped logging yesterday afternoon. The nightly archiving and cleanup jobs did not run. The service did not crash. The drive has 63 GB of free space. There are no entries under the Application or System logs in Windows. Under the Errorlog I see this for all of the reporting nodes ("ip.address.#" is placeholder for the actual values in the logs):

2015-05-28 15:38:59 Log to file action - Error: Win32File Object [45600] Unknown error.

2015-05-28 15:38:59 Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: E:\Syslogs\Firewalls\ip.address1.txt

2015-05-28 15:39:00 Log to file action - Error: Win32File Object [45600] Unknown error.

2015-05-28 15:39:00 Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: E:\Syslogs\Firewalls\ip.address.1..txt

2015-05-28 15:39:02 Log to file action - Error: Win32File Object [45600] Unknown error.

2015-05-28 15:39:02 Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: E:\Syslogs\Firewalls\ip.address.2.txt

2015-05-28 15:39:03 Log to file action - Error: Win32File Object [45600] Unknown error.

2015-05-28 15:39:03 Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: E:\Syslogs\ESX\ip.address.3.txt

2015-05-28 15:39:03 Log to file action - Error: Win32File Object [45600] Unknown error.

2015-05-28 15:39:03 Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: E:\Syslogs\Firewalls\ip.address.1.txt

2015-05-28 15:39:06 Log to file action - Error: Win32File Object [45600] Unknown error.

2015-05-28 15:39:06 Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: E:\Syslogs\Firewalls\ip.address.1.txt

2015-05-28 15:39:07 Log to file action - Error: Win32File Object [45600] Unknown error.

2015-05-28 15:39:07 Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: E:\Syslogs\ESX\ip.address.4.txt

2015-05-28 15:39:08 Log to file action - Error: Win32File Object [45600] Unknown error.

2015-05-28 15:39:08 Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: E:\Syslogs\Firewalls\ip.address.1.txt

2015-05-28 15:39:11 Log to file action - Error: Win32File Object [45600] Unknown error.

2015-05-28 15:39:11 Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: E:\Syslogs\Firewalls\ip.address.1.txt

2015-05-28 15:39:16 Log to file action - Error: Win32File Object [45600] Unknown error.

2015-05-28 15:39:16 Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: E:\Syslogs\Firewalls\ip.address.1.txt

2015-05-28 15:39:16 Log to file action - Error: Win32File Object [45600] Unknown error.

2015-05-28 15:39:16 Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: E:\Syslogs\ESX\ip.address.5.txt

The log stops there. When I restart the service I see these additional entries in the Error log:

2015-05-29 07:17:16 Unable to open InterApp listening socket on TCP port 3300

2015-05-29 07:17:16 Unable to open UDP socket on port 514

2015-05-29 07:19:08 Service running, but Service/Manager comm link is not connecting.

2015-05-29 07:19:28 Unable to connect to Service socket on TCP port 3300

2015-05-29 07:19:38 Service running, but Service/Manager comm link is not connecting.

Any ideas?

0 bshopp over 8 years ago

Hard to really tell, best bet to resolution is going to be to open a support ticket on this so we can investigate deeper.
Cancel
Vote Up 0 Vote Down

Cancel
0 kstone over 8 years ago

I see those occasionally also. It is usually a prelude to an out of memory error and service crash.
We use the Keep-Alive feature to send messages from each syslog server to another. Enable Keep-alive in the Inputs section at the bottom of the menu tree.
On the sending server create a rule with an input source of 'Keepalive' and an action of 'Forward to another host'.
On the receiving server setup up a Timeout rule with the desired threshold and actions.
Cancel
Vote Up 0 Vote Down

Cancel
0 bkyle over 8 years ago

Building on what kstone said, how many messages are you receiving per hour?
Cancel
Vote Up 0 Vote Down

Cancel
0 ttl over 8 years ago in reply to bkyle

We're running at about 1.6 - 1.8 million per hour. The messages displaying in the console are bheind by about 11.5 minutes at the moment, I think it's trying to catch up.
Cancel
Vote Up 0 Vote Down

Cancel
0 kstone over 8 years ago in reply to ttl

You're pushing the limit on the engine... How many rules, logs written etc?
Cancel
Vote Up 0 Vote Down

Cancel
0 bkyle over 8 years ago in reply to ttl

You can determine what severity of the messages by going to Manage\Debug Options\Get diagnostic information. You will want to look in the Breakdown of Syslog messages by severity section. Usually the majority of traffic will be informational. If you do not need to log these messages, you can stop sending them from your devices, and this should resolve your issue.
Cancel
Vote Up 0 Vote Down

Cancel
0 hemanathan over 4 years ago in reply to bkyle

30000 messages per hour
Cancel
Vote Up 0 Vote Down

Cancel