Internet bandwidth upgrade decision should be based on which parameters?

Here are some ideas to consider:

• Average utilization or percentile may not be good measurements for determining when to upgrade ISP bandwidth unless you average across a very tiny time span. 
• Average trending can help you estimate a date by which you might anticipate problems occurring because ISP bandwidth will be 100% utilized.  But it's not the tool you need yet.
• Content filtering and caching may save you a considerable amount of money in Internet bandwidth by preventing users from repeatedly downloading the exact same content, or from accessing resources that may be inappropriate for your business to support.

Imagine an average Internet utilization for 30 days.  Mine (for 40,000 devices and 10,000 Guest Internet clients) shows average Transmit and Receive Bits Per Second over the last 30 days stayed about 30

Mb/s. 

That's VERY deceiving because it's an average.  Peak spikes on that graph show between 125 Mb/s and 160 Mb's .  The average is thrown off because my business uses only 15% of the daytime Internet bandwidth.  Weekends throws the average off even more.

No, you're better off measuring the spikes and putting yourself in the position of your clients when you've theoretically run out of bandwidth.  If you bought 50 Mb/s based on the above 30-day average that includes nighttime and weekends, your users would run out of bandwidth quickly in the morning, and would complain and be unable to perform business transactions all day long.

Suppose you look at the spikes in that graph and say to yourself "I'm OK with buying 160 Mb/s, because that was the highest average I saw in the last 30 days." Well, you'd be out of bandwidth again, because that average doesn't show you the true spikes.  During that same time, I observed spikes between 350 Mb/s and 900 Mb/s on a 1 Gb/s service.  I could never see that information using any Averaging or trending tool.

A better tool for this is the Engineer's Tool Set Bandwidth Gauges, set to show Historical Statistics in a graph. 

I configure it to poll my Edge Internet switches' Internet-facing ports ever ten seconds, and I see real-time utilization and trends over 24 hours in a much more accurate and useful view than NPM's 30-day or 7-day or 24-hour averaging views.  It shows what the customers are really doing.

And that's what you want, isn't it--to know the actual demand during the day.  And you'll see that trend spiking for extended periods if you allow employees or Guests to use your Internet pipe with their cell phones.  Every time Apple provides a new iOS upgrade, cell phones eat up an additional 200 Mb/s of Internet bandwidth for three or four days in my environment.