[net-perf] bad performance w/o jumbo frames over large BDPs

[Michael Sinatra]

Michael Van Norman wrote:
> I'm getting about a 6x increase in sending by turning off tcp 
> segmentation offload.  This was theoretically fixed in the 2.6.12 
> kernel, but I'm running  2.6.17.6-web100, so maybe not.  I am still not 
> getting the performance levels I should, but it is much better with tso 
> off.

I have managed to get a dramatic improvement in performance in Linux by 
using the advanced congestion control features in the newer kernels (I 
think it's present in 2.6.17, but I am using 2.6.18-web100 right now). 
You need to enable it at compile time and then use sysctl -w 
net.ipv4.tcp_congestion_control = <value>, where <value> is one of the 
of those listed in Tierney's TCP Tuning Guide for Linux.  (see 
<http://dsd.lbl.gov/TCP-tuning/linux.html>)

Now I am seeing 700-750 mb/s sending (although bwctl wants to set the 
window size at something outrageously high--60-70MB, or else performance 
degrades), and 800-850 mb/s receiving, with more normal window sizes. 
Remember that this is all with *non*-jumbo frames.

I'll take a look at the segmentation offload as well.

Jeff, can you do a 'uname -a' on the same box where you sent us the 
helpful sysctl info?  I'd like to see the exact kernel version.

michael