nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

NCM RTCD

jonchill

I configured RTCD and after a bit of fiddling got it working.

I came into work this morning and found that the server that has NCM on was running at 100% and that all 4gigs of the memory were in use, when I checked the task manager I found that there were a load of Configmgmtjob.exe and also configautodownload.exe processes running. When I had a look in NMC there were 9 Syslog RTCD jobs trying to run and also our overnight backup job was still running.

In the end I decided to bounce the server and disable RTCD as we couldn't afford to have NPM which runs on the same box running so slowly.

What's causing this hangup and how can I get round it?

ConfigAutoDownload.jpg

Find more posts tagged with

Accepted answers

All comments

chris.lapoint

Had you pushed out any bulk changes to the environment recently? This can cause multiple change detection events to be spawned at once. Also, did you enable logging (File > Settings > Realtime Change Detection > Config Change Settings)? If so, please open a support ticket and attach the RTN.log. You can reference #2344 in dev in your ticket.

jonchill

I pushed out a one line config change to all our switches before I had configured RTCD.

I've logged a call #62768 and sent over the RTN.log.

Thanks

vhcato

We have the same issue, and have since first enabling RTCD with v4.0. I opened a support ticket a while back, worked with Support, and the last correspondence I had with them was pointing to a potential issue with the patch level on my SQL server. To be honest, I haven't looked any further into it, since we plan to upgrade our SQL server soon, along with all of our Orion/Cirrus servers.

AlbanyNYMike

I had the same EXACT problem when I tried to enable RTCD in Cirrus v4.0 I believe this is a database permission issue. The NCM and NPM databases must have full access to eachother (i believe). If NCM DB does not have access to NPM database, your Orion server will jump to 100% CPU, and you will have several jobs running.

I had to disable RTCD in NCM due to this fact. I bet if you log your RTCD jobs, you will see in the log "no IP exists" or something along those lines.

I couldnt get anywhere with SW support on this. Also, it depends on if you are using syslog, or snmp to trigger the RTCD.

pserwe

Denny/all,

RTCD triggered via syslog or trap alert executes the external program ConfigAutoDownload.exe and Configmgmtjob.exe. Unfortunately, those processes have a fairly large memory footprint and 10-20 of them can easily choke a system. Also, they don't seem to pay attention to the concurrency limits configured in NCM under File-->Settings-->Configs-->Config Transfers.

At the bare minimum, a fairly rapidly implemented feature of adding a concurrency awareness and concurrency limit to ConfigAutoDownload and /or ConfigMgmtJob (it seems like ConfigAutoDownload is the parent, and the control should be excercised there) is needed. Whether it be configured via the primary NCM interface, or it has a hard-coded limit of say 5 concurrent sessions as a stop-gap or not depends on how easy it is to implement. Of course, it would be nice to find it under File-->Settings, but it's unusable until it has some sort of concurrency limitation.

Whether by user error (in my case a bad rule) or by bad behavior (in the case where someone rolled out a lot of changes concurrently), it needs this functionality to be usable in a variety of situations.

Peter

jonchill

I have to agree, I had a good long look at the config after our first disastorous go at RTCD and re-enabled it on Monday. But come Wednesday the server had crashed again as it couldn't cope with the number of changes that were being made, were just in the process of re-addressing our network and putting everyone into different VLAN's so every switch in the company is affected. Processes were sitting there not finishing and in the end I had to reboot the box again.

In principle the RTCD is great as you don't need to worry about downloading nightly or weekly it will do it when it sees a change but if the system can't cope and it has to be turned off when multiple changes are being made then its not worth having it on.

Some sort of restrction on the number of consequtive downloads and processes at any one time to stop all the resources being creamed.

Thanks

Jon

chris.lapoint

Thanks everyone for the candid feedback. For our next major release (after 5.1), one of the key drivers is to ensure we can provide more controlled scalability in RTCD. For all those experiencing (or have experienced) issues with RTCD, please post the average and peak load number of unique config changes you expect on a daily basis. This will greatly aid us in our design and planning efforts.

pserwe

Thanks everyone for the candid feedback. For our next major release (after 5.1), one of the key drivers is to ensure we can provide more controlled scalability in RTCD. For all those experiencing (or have experienced) issues with RTCD, please post the average and peak load number of unique config changes you expect on a daily basis. This will greatly aid us in our design and planning efforts.

In theory, I could expect 400 to 1000 configuration changes in a day. Maybe more. This is long-term, down the road, scaling a DIA/VoIP network.

I back up NCM with RANCID. I can make as many changes as I want to any or all of the configurations, and RANCID just chugs along.

Granted, it's polling, and only hourly, it does not attempt to do any sort of RTCD. It runs off perl, expect, and cron. Very simple.

It does however, keep full logs and poll all my routers at the same time, only sequentially. It doesn't halt an overtaxed unix box doing 20 other things.

For real time, there is some expectation of an instantaneous update, that's the theory anyway. With the current memory and cpu footprint on those two applications,
I wouldn't run more than 10 at the same time on ANYBODY's installation. And seriously, I'd want to be able to turn that down to 3-5 on my own. I think there's something
wrong, because what's being done isn't that complicated.

Peter

jonchill

Avg load 1 or 2 changes a day with a peak of 50 - 60.