Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

NCM Crash - Multiple Configuration Downloaders causing CPU to spike 100%

We're running NCM with the Auto Config Downloader using SNMP Traps. We found this afternoon that the Orion box was running at 100% CPU, with about 15 ConfigAutoDownloader.exe processes running at 100% CPU. Also, there was an application crash screen for JobEngineServicev2. Also, the Event Viewer is filled with the following for the TrapService:

An Exception was thrown during while updating the DNS cache. Error Detail-System.Data.SqlClient.SqlException: Transaction (Process ID 107) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction. at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection) at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at SWTrapService.SWTrapService.UpdateCacheEntry(Object ipAddress)

Any idea how to resolve this

Find more posts tagged with

Accepted answers

All comments

rschroeder

It's pretty easy to configure NCM to use fewer simultaneous connections and see if that fixes the problem. I had similar problems until I made a few tweaks to keep things under control.

First, make sure you're not downloading the ARP and Bridge tables for now--they can slow the process a lot. To do this, start up Orion NCM, go to File, Settings, and Device Inventory. Uncheck both the ARP and the Bridge Table options.

Next go to the next area below Device Inventory, called Inventory Engine Settings. Move the slider bar to the left to decrease the number of devices to Inventory Concurrently. I know you're probably not doing a fresh inventory with your auto config downloads, but this is something you can easily change back after the problem's resolved.

Now go to Configs, and highlight Config Transfers. Slide the top bar to the left to reduce the number of simultaneous downloads/uploads. You can always increase it later, but tweak it down to a low number and test to see if that helps resolve your problem.

Then go to Network and make sure your ICMP timeout settings are reasonable. Too short a time will cause the jobs to fail, and too long might make them take forever if one site has slowness issues.

Below Network, check the Telnet/SSH settings. If they timeout too soon or take too long, that will also cause issues in completing jobs in a reasonble window.

In Advanced > Logging, disable Realtime Config Change Detection and Node Monitor, at least until you get this resolved. You can always reenable them later, if you need them.

Lastly, if none of these fix it, and you have proven that you can download the configs manually one at a time, it might be time to open a ticket with Support.

Good luck!

Rick

rschroeder

I found one issue NCM was experiencing, when it wasn't able to complete its jobs, was caused by the size of my Cisco 4510's configuration files versus the method I'd chosen to download them. 384 ports for a fully loaded chassis, 33 lines of configuration code per port, plus port-channels and VLAN's and SVI's and L3 routing . . . the configs were too large to download quickly via the method I was using.

Once I tested via SSH and SFTP and saw the differences, I was able to get the jobs done much more quickly changing to the process that took less time, resulting in NCM's job's being 100% the majority of the time.