We have recently heard several partners using the SolarWinds NDS tool, struggling with performance and lack of feedback on the progress that that tool is making during each of the phases of its work.
So I thought that giving the NDS community a bit more information about how it works, would help.
Note that the below relates to a service release - v1.0.4 - recently delivered to all partners who have registered to receive the tool, in the past.
Before we start, a few reminders about NDS:
- This tool leverages the Cisco CDS API and is opened only to Cisco (and SolarWinds) partners that have a CDS API set of credentials. You'll need the Cisco credential when you start NDS.
- This tool requires NCM installed. NDS processes all Cisco nodes that NCM has discovered and inventoried. Nodes discovered by NCM but not Inventoried, won't process successfully in NDS.
- You can get the NDS tool after registering here
Tip: we have detected issues related to the length of the email address used. Using an address that is too long (e.g. about 40 characters) seems to create issues and should be avoided.
Tip: how do I know what version of the NDS tool, i am running? Look at the file properties of the executable file ONCE INSTALLED (not the exe file before installation).
The version v1.0.4 will be depicted as 18.104.22.168:
NDS v2.0, shipping soon, looks like this
The 4 phases of what SolarWinds NDS does, are described in the rest of this blog, and for each, the blog will:
- describe what the NDS screen typically looks like during this phase
- describe briefly what it does and what it needs, to succeed
- How long it typically takes
- How to know about the status of this phase (when possible), and whether resuming it after an interruption is possible, or not.
1- Collecting Cisco device data
During this phase, NDS communicates with NCM and forces it to download fresh configs and execute some CLI scripts. Also NDS performs Inventory tasks of its own directly against the devices (without using NCM).During this phase, NDS requires:
- a connection with the NCM Server
- performing an inventory (directly on the device, i.e. without NCM) of Cisco Devices found in NCM's database. This means that NDS should be able to access these devices and may send SNMP requests
- Internet access
This phase typically lasts between 5 and 10 minutes per node, found in the NCM database.
By default, NDS processes all Cisco devices found in the NCM database.
To reduce this list to a subset of nodes, please read section "Advanced Settings" in NDS 1.0.4 Users Guide.
There is no way to know precisely where NDS is, during this phase.Any interruption during this phase will require re-submitting the request from the start.
2- Submitting Cisco data to Cisco's CDS API
During this phase, the NDS screen looks similar to the previous phase, except that the green progress bar is closer to completion (not shown above).
NDS prepares and submits the request to the the Cisco CDS API.
The request is submitted 20 devices at a time (default setting).
During this phase, NDS requires Internet access (and the Cisco CDS credential, entered when you started NDS).
This phase typically lasts about 30sec per set of 20 devices.
For example, submitting requests for 800 devices should take approximately 800/20*30sec = 1200 sec (20 minutes).There is no way to know precisely where NDS is, during this phase.Any interruption during this phase will require re-submitting the request from the start.
3- Monitoring Cisco's CDS API progress
During this phase, NDS waits for the Cisco CDS API to return the status of the submitted transactions.
Status "Pending" or "Processing Complete" or ‘Partial KTN Pending", means that Cisco still processes the data.
During this phase, NDS requires Internet access.
The only way to get a status of where this phase sits is to connect to this Cisco tool in order to get information about the status of the request, on the Cisco side. This Cisco URL requires the CDS API credential that you have entered in NDS.
The duration of this phase is entirely dependent on Cisco's API activity. It is pretty much impossible to predict how long this phase takes, as the time to respond is based on where your request sits in the queue of requests that the Cisco API has to process (the requests are processed sequentially and not in parallel). It is not rare to see the API taking several days to respond to this phase.
Because of its potentially long duration, you may have to disconnect NDS and resume.
This is supported by NDS.If you close the NDS tool you will be prompted with this window.
You can click Yes, restart NDS later and resume its activity from Phase 3.
4- Generating CDS reports
During this phase, NDS contacts Cisco to download the data required to create the reports.
There is no way to track the progress of this phase, but it is usually not very long (several minutes), and depends essentially on the number of devices that you are processing.
During this phase, NDS requires Internet access.
During this phase, NDS should not be closed, if there is one or more transaction with a status "waiting for report". Closing NDS and restarting NDS during this phase might create duplicate reports. Duplicated reports can be cleaned up manually without consequences.
If NDS was experiencing a problem in this phase:
- Due to a disconnected from the network,
- or due to a problem on a cisco side, such as the one reported in the NDS log file (C:\Program Files (x86)\SolarWinds\NdsAssessmentTool\NDSAssessment.log), looking like this:
“2013-05-09 17:13:55,056  ERROR SolarWinds.CDS.Engine.CDSTransactionFlow - e.Error is not null:System.Exception: inner is null; Type:System.Web.Services.Protocols.SoapException Message: Server error”
you would likely get an "error" status such as this one:
You can usually recover from this state by attempting this:
- fix the connection issue (if any)
- replace the keyword "error" by "pending" in the trans.xml file located under [NDS installation Directory]\CdsRepository
Before: <CDSTransaction Id="SWxxx" Name="TestTrans" Status="Error" Company="…" CreationDate="2013-04-26T09:42:40.9436+02:00" Contact="…" DeviceCount="807" Owner="…" />
After: <CDSTransaction Id="SWxxx" Name="TestTrans" Status="Pending" Company="…" CreationDate="2013-04-26T09:42:40.9436+02:00" Contact="…" DeviceCount="807" Owner="…" />
- Restart NDS
Note: NDS v2.0, available soon to those of you who have registered, does not require the manual operation described above, in order to resume and complete the transactions in error. It is performed automatically as soon as NDS is stopped and restarted.
When NDS completes the reports, the screen looks like this:
Click download and get the reports:
- We have seen this error happening sporadically in the NDS log file (C:\Program Files (x86)\SolarWinds\NdsAssessmentTool\NDSAssessment.log):
Message: ORA-03113: end-of-file on communication channel
We believe this Oracle error is on the Cisco side and should be investigated with Cisco. We recommend re-sumitting the transactions experiencing these issues.
- We have recently seen a few instances of this "Policy Falsified" error. This is being investigated with Cisco, we have no solution / workaround at this time.
We will post here guidance and solution when available.
This "Policy Falsified" issue is now fixed, following a change on the Cisco side. Please let us know if you are still experiencing this issue, by posting on this page.