Job Back Up - High latency

Hi,

I have a massive job with about 5000 nodes to backup configurations. Of this daily job, usually fail about 10-20 nodes due to high latency (+700ms). I have tried to touch the timeout settings of Solarwinds, as well as the number of simultaneous connections in case it is causing the failure, but still the error.
On the other hand, I have seen that if I launch a 1 by 1 backup (also the 10 with the problem), it works.
Is there any way to create a job that runs after the massive one that tries to make backup of these that have failed?
Another way that has occurred to me, would be to create a group with these nodes, but in the dynamic query does not leave the latency field and does not let you do it by SWQL/SQL query.

Any solution?

Find more posts tagged with

config_backups

latency

ncm

backup_configuration

jobs

Accepted answers

All comments

stuartd

My first thoughts are:

why do you need to backup 5k nodes daily? On the presumption you are following some form of Core, Distribution, Access model, then a better approach would be to only backup the Core on a daily basis and the Access weekly or even less frequently. I'll wager the config on a lot of the nodes never changes so why daily?.

My second thought, is why have one single job for all 5k nodes?

From a load perspective you'd be better off distributing them across groups (defined by whatever means you have or by using a custom property) and stagger the backups across the week

Anyway, you've sort of answered your own question. Just create a second job with the nodes that fail and run that at a completely different time to the other job.

adam.beedell

You can alert on the timeout scenario, and trigger a manual backup off that

I've done the former, not the latter yet, trade ya?

Left Join NCM.Nodes ncm on nodes.nodeid = ncm.corenodeID
WHERE ncm.LoginStatus not like 'Login Ok' 
and 
ncm.LoginStatus = 'Unable to login to device : Timeout'

duck1

Yeah, I can create a second job for the ones that fail, but the list changes daily. Is there a way to automatically mark the failed nodes and apply the second job only to them?

On the other hand, could we also increase the timeout and retry settings for this last job so that they are different from the rest of the jobs?

duck1

Thanks for your idea, I will use it to make an alarm that modifies a custom property and then a job is executed according to that custom property, it is the only way to make it automatic.
I would like to comment that in my case, those that fail due to high latency, the login status does not show any timeout, I'm going to filter by LatestTransferJobStatus.ErrorMessage LIKE ‘%connection refused%’,

LEFT JOIN NCM.Nodes AS NCMNodes  
    ON Nodes.NodeID = NCMNodes.CoreNodeID  
LEFT JOIN NCM.LatestTransferJobStatus AS LatestTransferJobStatus  
    ON NCMNodes.NodeID = LatestTransferJobStatus.NodeID  
WHERE  
    LatestTransferJobStatus.TransferState = 3
    AND Nodes.Status = 1  
    AND LatestTransferJobStatus.ErrorMessage LIKE '%connection refused%';

EDIT: the correct SWQL is the following:

LEFT JOIN NCM.Nodes AS NCMNodes  
    ON Nodes.NodeID = NCMNodes.CoreNodeID

LEFT JOIN NCM.ConfigBackupStatus AS NCMConfigBackupStatus  
    ON NCMNodes.NodeID = NCMConfigBackupStatus.NodeID

LEFT JOIN NCM.LatestTransferJobStatus AS LatestTransferJobStatus  
    ON NCMNodes.NodeID = LatestTransferJobStatus.NodeID

WHERE  
    NCMConfigBackupStatus.BackupStatus = 14
    AND LatestTransferJobStatus.TransferState = 3
    AND Nodes.Status = 1  -- Solo nodos activos  
    AND LatestTransferJobStatus.ErrorMessage LIKE '%connection refused%'