Groups being at the Organization level seems to be a legacy feature, and may be fine for some groups (while I can't really think of any.) But more often than not a group is going to be specific to a Service Provider. For example, IT doesn't need to have a group called "Payroll", and Finance doesn't need to have a group called "Service Desk" or "Help Desk".
Yet right now, since groups can only be defined at the organization level, these groups are available in all Service Providers and only serve to unnecessarily add to the list of users and groups. This is especially annoying when assigning a ticket and now my list is much longer than it needs to be, because there are a bunch of groups which I will never use (and can't use.)
The only way I can think to avoid this is to change permissions in a Service Provider to not allow users to see those groups. I don't know if that is even possible, but even if it is, I - as an administrator - will still see those groups in my Service Provider which will literally never be used.
At the very least allow me to hide groups in the "Assigned to" dropdown (and now the "Group Assignment" dropdown) which have no agent users in that Service Provider. If all members of that group are at the "Requester" level (or other non-licensed level) then there is no reason for that group to be in my dropdown. (Similarly, if that group is empty there is no reason for it to be in my dropdown.)