You can lower the following retention policies in the System Properties section of Advanced Setup:
Days to retain performance data hourly rollups (default: 90)
Days to retain raw performance data (default: 14)
This will not decrease the database size because the database keeps the once occupied space, but it will reduce the space necessary to keep Virtualization Manager running and the database will not grow anymore.
If you want to free the space which the database no longer needs after that, there's an interactive cleanup script located in /var/hyper9/bin/db_cleanup.sh on the virtual appliance. It will ask you about compacting the database and if you answer yes, it will perform VACUUM FULL on it. You should not do this unless necessary, because it usually takes a very, very long time to complete.