I was apparently more tired than I realized last night.
In order to enlighten anyone else that might have the same question:
I was seeing this issue because I built the View from a NetFlow template. I was thinking I needed that since I’m showing NetFlow data.
However when using custom objects it doesn’t matter what the template actually is. So the fix here is to build the template from a Summery page and it eliminates the items I mentioned above. If you have directlink enabled or use an account that has no menu items set it makes a rather nice management report.