This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.

You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Redundancy and High Availability for Kiwi Syslog Server

sgonz over 1 year ago

Hello all,

For the project i'm involved, we need to have a high-availability service to prevent data and log loss.

We have designed an architecture in which data will be replicated across two physically separated nodes, updated synchronously (SQL Always On). Additionally, these data will be asynchronously replicated to a third cloud-based storage. Syslog Server (Kiwi) data will be directly kept in the cloud storage. We will be logging core business activities, executions of various automated tasks, and the operation of external drivers interacting with our system.

Please note that this is not load balancing between the two nodes. All traffic will pass through both physical nodes, each equipped with its own Syslog Server. If load balancing is necessary due to volume, the number of executions, or the risk of message loss due to buffer overflow, additional Kiwi Syslog Servers will be implemented at both locations, functioning as a cluster and sending information to cloud storage. The goal is for one node to take over the syslog if the other fails.

Have any of you faced a similar situation before? (sure you have). Can you please recommend a solution that allows for a failover in case a Kiwi agent fails to send data to cloud storage. In this scenario, a second Kiwi would activate and start transferring data to the same storage avoiding message duplication.

We're reaching out because we couldn't find any topics related to this redundancy and/or high availability in your success center.

Thank you all.

Top Replies

Parents

0 cnorborg over 1 year ago

Hmm.. Just my own $.02, which is worth even less these days!

First thing to remember is that syslog is lossy data, even if you change from UDP syslog messages to TCP, you should never count on getting every single one of them. Will you get the vast, vast majority? Yes.

I don't think Kiwi was ever meant to be an HA solution, there are other logging solutions which profess to be much better. Is it very good? Once again, yes.

The biggest flaw I see is the strategy to log to the cloud. I would think it would be much more reliable to log locally and ship logfiles to the cloud somehow. If using a flatfile structure, just copy them once their full and rotated. If a database, ship logfiles?

The only way I can think of implementing the failover you mention is to use a load-balancer, like an F5, to determine the health of the kiwi server somehow and send syslog messages to healthy nodes?
Cancel
Vote Up +2 Vote Down

Cancel
0 sgonz over 1 year ago in reply to cnorborg

Thank you very much cnorborg and marcrobinson for your help, I have passed it on to the IT-Platform guys for their consideration.

From what they tell me, load balancing may be the ideal solution as you are proposing.

Regarding the cloud stuff, our approach also involves local logging and then sending to the cloud; we'll see what the client says in the end.

Again, thanks for the help!
Cancel
Vote Up 0 Vote Down

Cancel

Reply

0 sgonz over 1 year ago in reply to cnorborg

Thank you very much cnorborg and marcrobinson for your help, I have passed it on to the IT-Platform guys for their consideration.

From what they tell me, load balancing may be the ideal solution as you are proposing.

Regarding the cloud stuff, our approach also involves local logging and then sending to the cloud; we'll see what the client says in the end.

Again, thanks for the help!
Cancel
Vote Up 0 Vote Down

Cancel

Children

No Data