When you are in the Setup tab and select Configuration Summary does the appliance performance look ok (as seen in the pic below)? If the performance window shows the appliance as good then check the following items.
1. Confirm there are not large or multiple snapshots on the appliance
2. Confirm the network drops are not occurring during a backup
3. Confirm Live Migrations are not occurring during the drops.
If the following does not yield any results then we may need to get on with support to gather logs from your Virtualization Manager appliance. Let me know what you find so we can get this resolved for you asap.