You're right on track in regards to the file size. By default I believe the limit is 384MB. We have an unpublished tool that will allow you to increase this limit.
The EXE has built in help, just run the EXE to get the help. The EXE needs to be run on a machine with the WSUS console installed – preferably the WSUS server or the Patch Manager server.
The syntax to be used is:
/action setmaxcabsize /targetwsusname WSUSServerName /targetwsusport portnumber /maxcabfilesize sizeInMB
There is also a /targetwsususessl parameter, which should default to NO. If the WSUS server is using SSL, or if the command doesn’t like the missing parameter, then also add this parameter with the correct value YES | NO.
That did the trick. I was forced to use /targetwsususessl NO. Thanks David! -mark