So far I run everything of mine (bar the database) on one single server. Most crucial factor for speed is database access. My database lives in a SQL environment.
If your speed is fine currently, IPAM shouldn't really load it up greatly. How big is the address space you're looking at scanning?
as superfly99 has responded, there should not be a problem of running IPAM with existing product from above - we can talk about DB load if you want to scan for more than 1 million IP addresses.