At the end of last year we had a site with a 100 meg line. We suggested WAN acceleration, but the customer decided that increasing the pipe size was what they wanted. We put in a Gig connection to them (30 miles away) and the speed appeared the same, lots of delays and complaints about performance. Working with our firewall vendor and the ISP everything seemed to function as expected. Even though latency appeared to be pretty good the ISP finally agreed to "optimize" the connection. i.e. instead of just tossing the traffic to the "cloud" they set up a hard coded pathing between the two sites. Once that was done everything performed much better.
Bottom line it turned out to be latency. We tried tinkering with the windows size but since the traffic is also in a VPN tunnel, that didn't ultimately have much effect.
30 miles shouldn't have produced appreciable latency, IMHO. I'd expect <5 ms delay for that distance, based on my sites that are that far away. It makes me wonder what the latency restrictions for the vendor's application or hardware might be, and what latency you were experiencing across that distance. But I'm using direct fiber--your distance could have been over wireless for all I know.
Optimizing a connection by selecting more efficient / shorter paths certainly is a factor. Years ago I opened a new hospital WAN connect with some bundled T1's. The distance as the crow flies was 75 miles, but latency was in the 250 ms range. A traceroute revealed my packets to that site went from Duluth, Minnesota to St. Louis, Missouri, then to Nashville, then to Chicago, then to Minneapolis, Minnesota, and finally back up to the new hospital at Hinckley, Minnesota. That 75-mile path was actually traveling over 2,600 miles to get to the new site, via this "un-optimized" path. That was with ATT/Qwest. Eventually we got that resolved and the latency dropped to 7 ms.
Certainly a VPN tunnel adds overhead & doesn't help latency at all.
The physical distance is about 30 miles, but the real distance could vary greatly depending upon the route that the traffic was taking. Keeping in mind in an area that has far less density a left turn instead of a right turn could actually add hundreds of miles to the actual route. Kind of like if you are in a major city and make a wrong turn at an intersection you might add a mile or so to the overall distance, but when you are out here if you turn on the wrong country road it could be 20 miles before you get the chance to turn to get back in the right direction and another 20 to the next street and another 30 to get back on the right destination road. It's the same way with telecom - sort of.
You're right on, regarding paths and distance.
One of my regional hubs is 125 miles from my main site, and the two resilient paths are very different geographically. One runs about that same 125 miles, the other is closer to 300 miles. Sadly, the WAN provider insists the longer path be the primary path. The users love maintenance windows, when the primary path fails over to the shorter path. They can tell the improvement in latency by noticeably better application performance. And they know right away when the longer path becomes primary again.
Over the years I have come across many applications that are not "WAN friendly". If an application can't handle running at 52Mbps then it probably isn't even LAN friendly.
Another thing is verify where your traffic actually goes. I have seen a vendor sell a "virtual Ethernet" circuit, with both ends in one small city. The only switch they had in the state (they were running on another vendor's fiber) was a couple of hundred miles away. It was still pretty quick (~15 ms latency) but if I we had bought the fiber providers circuit, latency would have been <4 ms.
- Does apply to your environment? This applies to every environment; Hold the application vendors to account.
- Have you run into the issue and improved throughput by adjusting TCP window size? No, I have never adjusted the window size.
- Are you using WAN Accelerators to address the problem? I have, in my previous position, used WAN Accelerators.
- If you're using WAN accelerators, what brands & models have you tested, what did you like, what was the cost per end site and per core site? I don't know the cost, but Cisco WAAS is very solid.
- What problems did you discover on the way? Were there things that simply couldn't be addressed & corrected? Some things just can't be accelerated.
- How do you adjust TCP windows size on a PC and on a server? Not sure....
- Does the TCP window size need to be adjusted on both PC and Server, or only on one side or the other? Honestly, most modern operating systems allow for Selective ACK and Window Scaling, so I can't imagine that this would be necessary.
Thanks for your input!
1 of 1 people found this helpful
1. We totally run into this issue; in our case our WAN links are 100Gbps(*) and stretch from Tokyo (through Seattle) to Chicago and the bandwidth-latency product is an issue,
2. For large datasets we use bbcp
$ bbcp -N io -v -s 64 -W 16M -P 10 'tar -c .' username@host:'tar -x -C /gscratch/groupdir’
if you have smaller data sets and non-bulk transfers then the RTT will affect performance since you will have 2*RTT for the acknowledgement (before the application sending data builds up a buffer)
if your data chunks are very small then then this will kill performance as you never get to take advantage of the TCP window.
4. do not bother -- for bulk transfer we use bbcp
5. no. it's not the 1990's -- but Firewalls and IPS devices can totally destroy the ability to use larger windows, or packet sizes.
For our truly large bulk transfers we have special systems that sit outside our security perimeter specifically designed for bulk data transfer
if you can't send unfragmented 9000 byte packets to the destination then that will also affect performance.
(* I hate it when vendors talk about WANs when they mean T1 circuits)