Download config running- error details connection refused...
Do I need to do this?
Not sure what you're asking, need more details?
If you're getting that error and trying to figure out what is wrong, the best first step is to install an SSH client, such as putty, onto the Solarwinds poller/server and try the SSH connection using it. Be sure to use the same credentials that Solarwinds would be using. If you can't log in via putty for whatever reason (ACL, credentials, etc) then Solarwinds can't either.