Maybe when onboarding those external users as guest users in the tenant where SSO is configured?
If that does not work, you need to setup some sort of directory where all users are synced to from the different sources and set up SSO to that directory.
