NCM INventory Runs very slowly

Is there a reason why an Inventory job runs so slowly in NCM?

We think that we have found the reason by capturing the data on the switch port for the server. It looks like NCM loads the entire job into memory before executing it. Then it does an SNMP GET for a device and waits a timed period (not negotiable) and then does another GET for the same device until it completes each parameter that has been requested via the Inventory set up screen. This happens for each device for each parameter that is specified for each device. The job runs an inordinately long time, up to a day and a half, before we just kill it. We have asked about this in the past, but no reply has come from support.

It is our guess that this was done in testing and the "wait" time was never removed as it went to production. Since we are inventorying some 1300 devices, we would like to see this fixed.

Thanks for the time and effort.

Find more posts tagged with

wait

ncm

slow

inventory

Accepted answers

savell

Well, it certainly resolved my slow inventory issue - the job now completes in 1 hour 21 minutes (the last completed run before the index change took 24 1/2 hours). This is for almost 1700 devices and 20 concurrent inventory processes.

It is even faster than pre version 5.5 - although it's not fair to compare, as I have tweaked (reduced) the snmp timeout and retries which were introduced in 5.5.1.

Dave.

All comments

familyofcrowes

My company is having the same issue. We need to run inventory more often but the time it takes to inventory 1000 devices is too long.

stacy.patten

We are having the same problem.

chris.lapoint

Please try setting the Inventory Engine settings to allow more concurrent processing and let me know if that helps. You can access this setting from File > Settings > Inventory Engine.

mhh351

We have already tried that. While it does help, it is really just a workaround to actual processing problems that the task has.

If we set it to anything greater than 3 or 4 devices at a time, the server goes to 100% CPU (2 dual core processors) and that just keeps everything else from running normally. We have NPM on the same server for some interaction needs so it cannot be held up in its polling cycle.

Can you please look into why the program just plods along in its code?

Thanks!

FormerMember

This started for me in 5.1.1

chris.lapoint

We're going to look into this.

FormerMember

The inventory just hit 3 hours and still running for less than 300 devices. Weird, I took a quick capture and over 17000 packets at the current device. I looked at the trace a bit, restarted it, and it's still blasting the same device. Another 10000 packets. I bet a couple hundred thousand packets have been sent so far to this one device.

chris.lapoint

For those experiencing issues, please open a support ticket and include a reference to this thread. We'll likely need to work with you to troubleshoot the source of your inventory performance issues.

FormerMember

I made some progress on my own. I went back to the device that was getting slammed by 10's of thousands of snmp packets from NCM. I ran inventory for just that device and same thing. Hit 20K packets very quickly. I noticed in the packet trace and the inventory status window that it was trying to go after the IP CIDR Route Table. I shut this off from being collected during inventory, re-tried the same device and made it through as expected.

Going to let a full inventory fly and see what happens. If OK then I will let runs as schedule tonight and see if the server still gets hung up.

FormerMember

Cha-Ching! 286 devices in 67 minutes. No idea why it would hang up on this router or others for the MIB mentioned above. Hopefully I fair well overnight.

mhh351

Thanks for the idea. However, we have had that disabled for about 6 months now.

The last inventory of 1200 devices took slightly over 2 1/2 DAYS! I jsut got the report this morning as the job ended.

We have seen that the job is waiting for a period of time for EACH query to the device.

I think that there is an iddue within the code.

FormerMember

Well, I'm outta the woods. My job ran fine last night and my server did not get hung up. I tracked this down and it started when we upgraded to 5.1.1.

If inventory is taking along time I also noticed it takes the job a ver long time to get through the layer 2 tables. If you're not using NCM to track down IP or MACS on ports, then it may help to also shut that off.

I go after 25 devices simultaneously and you see it just sitting on the layer 2 stuff forever in the status window.

FormerMember

Thanks for the idea. However, we have had that disabled for about 6 months now.
The last inventory of 1200 devices took slightly over 2 1/2 DAYS! I jsut got the report this morning as the job ended.
We have seen that the job is waiting for a period of time for EACH query to the device.
I think that there is an iddue within the code.

Did you turn it off in File Settings or in the actual inventory job? I noticed they are independent of each other.