I'm attempting to run a discovery profile that imports all "up" volumes and interfaces and then a series of Get-SwisData and Remove-SwisObject to only keep the ones our organization cares about. About half way through the Remove-SwisObject, I end up getting the following error:
Remove-SwisObject : Access to Orion.NPM.Interfaces denied.
Then I attempt to refresh the Orion web console and get the following error:
There was an error retrieving data from SolarWinds Information Service
Error: A query to the SolarWinds Information Service failed.
It's not a ton of objects, since is a smaller network (only 319 volumes and 1341 interfaces to remove). Here's a stripped down version of the PowerShell that's being run:
$swis = Connect-Swis -Hostname localhost -Credential $cred $query = @" SELECT ProfileID, Name FROM Orion.DiscoveryProfiles WHERE Name = 'Discovery Automation' "@ $discoveryProfile = Get-SwisData -SwisConnection $swis -Query $query Invoke-SwisVerb $swis Orion.Discovery StartDiscoveryProfile $discoveryProfile.ProfileID do { Start-Sleep -Seconds 60 $Status = Get-SwisData $swis "SELECT Status FROM Orion.DiscoveryProfiles WHERE ProfileID = @profileId" @{profileId = $($discoveryProfile.ProfileID)} Write-Verbose -Message "TimeStamp: $(Get-Date -Format yyyy-MM-dd_HH:mm:ss) | Discovery not completed yet. Sleeping..." } while ($Status -eq 1)
Once that's finished, this is run:
$queryVolume = @" SELECT CP.Z_AUTOMATION_BYPASS_Volumes, CP.UNS_Responsible_Team, N.Vendor, N.MachineType, N.NodeName, V.Type, V.FullName, V.Uri FROM Orion.Volumes V LEFT JOIN Orion.NodesCustomProperties CP ON CP.NodeID = V.NodeID LEFT JOIN Orion.Nodes N ON N.NodeID = V.NodeID WHERE Z_AUTOMATION_BYPASS_Volumes = 'False' AND Vendor = 'Windows' AND Type != 'Fixed Disk' AND Type != 'NetworkDisk' AND Type != 'Mount Point' AND Type != 'MountPoint' OR (Type = 'Unknown' AND (VolumeDescription LIKE '%Label:Recovery%' OR VolumeDescription = 'Physical Memory' OR VolumeDescription = 'Virtual Memory' OR VolumeDescription = 'Memory buffers' OR VolumeDescription = 'Cached memory' OR VolumeDescription = 'Shared memory')) OR (Type = 'Other' AND (VolumeDescription = 'Physical Memory' OR VolumeDescription = 'Virtual Memory')) "@ $volumes = Get-SwisData -SwisConnection $swis -Query $queryVolume Foreach ($volume in $volumes){ Write-Verbose -Message "*****Removing object: TimeStamp: $(Get-Date -Format yyyy-MM-dd_HH:mm:ss) | $($volume | Out-String)" Remove-SwisObject -SwisConnection $swis -Uri $volume.Uri Start-Sleep -Milliseconds 500 } $queryInterface = @" SELECT CP.Z_AUTOMATION_BYPASS_Interfaces, CP.UNS_Responsible_Team, N.Vendor, N.MachineType, N.NodeName , INT.Alias, INT.MAC, INT.InterfaceTypeName, INT.Uri FROM Orion.NPM.Interfaces INT LEFT JOIN Orion.NodesCustomProperties CP ON CP.NodeID = INT.NodeID LEFT JOIN Orion.Nodes N On N.NodeID = INT.NodeID WHERE Z_AUTOMATION_BYPASS_Interfaces = 'False' AND Vendor = 'Windows' AND Alias NOT LIKE '%[[]MON]' "@ $interfaces = Get-SwisData -SwisConnection $swis -Query $queryInterface Foreach ($interface in $interfaces){ Write-Verbose -Message "*****Removing object: TimeStamp: $(Get-Date -Format yyyy-MM-dd_HH:mm:ss) | $($interface | Out-String)" Remove-SwisObject -SwisConnection $swis -Uri $interface.Uri Start-Sleep -Milliseconds 500 }
About half way through the interfaces it bombs out. Do I need to have a longer sleep? Is there a better way to do this? I like the flexibility of Get-SwisData in using any type of logic we want to select the appropriate volumes and interfaces.
Anyway, I end up rebooting the polling engine and everything is once again magically delicious. In fact, once it comes back up, I'm able to re-run the second portion of the code above, which has fewer interfaces this time and no volumes, and it works. On the second run there are 0 volumes and 377 interfaces (it made it through 964 interfaces on the first run). I thought it might be a message queue issue. We pull in the message queue every 2 minutes and it never even went above 0. CPU and memory never spike. There's no expensive queries or processes with waits on the SQL server.
A possibly notable error:
2020-05-19 18:24:38,727 [95] ERROR SolarWinds.InformationService.Core.InformationService - (null) (null) Exception caught in method SolarWinds.InformationService.Core.InformationService.RunQuery System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Anyway, I appreciate any help!