Are there any special considerations when uninstalling Patch Manager and removing it from my environment?
We were using it via sccm so I do not have any agents installed.
No, there shouldn't be. The program should uninstall cleanly.
There may be a SolarWinds WMI Provider or SolarWinds Client Components in Programs and Features; if so you should be able to uninstall that from Control Panel.
The only other thing that might typically be left over is that Patch Manager may have created a certificate on the WSUS server, which would be used to sign any published updates. Note, however, that we are not the only product that may have created such a cert, so if you want to delete that cert from the WSUS, Trusted Publishers, and Trusted Root Certification Authorities Certificate stores, just be sure that you don't need it. It's just a cert that is basically only used for 3rd party updates, so many people just choose to leave it.