When I've had issues with page limitations and resource limitations producing an error when combined, I've rewritten my resource as a SWQL query. I never ultimately determined the cause of the issue, but proceeded with SWQL workaround. (DIdn't invest time in identifying due to dozens of other higher priorities and the constant battle between HIPS/ group policies/ and my application's functionality.) Perhaps showing alerts via a report would also be a possible workaround. I admit it was not satisfying when root causes weren't identified, but there just weren't enough hours in the day.
If you come up with a fix for the condition you described, I'd be very curious what you find.
I have the same issue on 11.5.3
its a bug and I open a ticket for that problem.
open a ticket and refere it to 991728