Overlapping realtime change events

Question

We have some Netscreen units (ScreenOS) that only implement stateless configuration via the web interface. There is no configuration/batch mode, changes are applied immediately. Each atomic change then fires off a "System configuration saved by..." syslog event which we use to trigger RTN. There is no event indicating the end of all configuration actions.

We run into issues when multiple changes are being made. With RTN triggering for each change, we end up with multiple overlapping download events. We have "Limit number of simultaneous RTN download operations" enabled, and end up with 4 active downloads and a large number in the download queue.

Concurrent downloads also seem to hit locking issues. 12 events (two devices in a cluster) run the CPU to 100% for about 40 mins, while one event typically complete within 30 seconds. Most of these events timeout the connection before completing.

I would like to see more intelligence in the RTN download system:

1. Only one connection to a single device should be allowed 
2. If another RTN event triggers for an active device, the download should queue until the active connection has been closed
3. If additional RTN events trigger for a device with another download already queued, discard this superfluous event.

Lastly, the download processes should be executed at low/idle priority so they don't impact the rest of the system when they have issues.

ngarratt · Answer

Hi Chris

That workaround would address the performance issue, but will typically miss the rest of the changes until either a later change set or the nightly download (rather defeating the purpose of RTN). I could probably set the event to fire the autodownload event, script up a delayed autodownload event, and suppress for the duration of that delay. This becomes a pretty ugly hack, through.

chris.lapoint · Answer

Thanks for the feedback.   Are there any others experiencing this same issue with other devices?

One workaround would be to setup a Syslog rule specific to each Netscreen devices so that it only triggers RTCD download action for the first event and suppresses further actions for X seconds before triggering again.