[net-perf] bad performance w/o jumbo frames over large BDPs

[Michael Sinatra]

Mark Boolootian wrote:
>> In other words, if we're just interested in making sure there's no 
>> packet loss on the abilene backbone, then it doesn't matter what the 
>> nodes use, as long as they do RFC-compliant TCP, so that they're 
>> approximating the type of production traffic that we would be sending. 
> 
> I don't think this is true.  I think Linux has a reputation for pushing
> out new TCP features and tweaking the network stack in a far more ad-hoc
> way than BSD.

I agree.  I think my statement is true only if we're NOT interested in 
seeing how the network performs with certain OSes, only how well it can 
possibly perform.  It doesn't sound like anyone here is just interested 
in that.

> I believe Internet2 should run a few *BSD boxes in addition
> to the Linux fleet.  While web100 for BSD may not (yet) be available,
> the stability and use of the BSD network stack as a reference makes it
> important to have available for testing.  It can potentially help in
> debugging the kinds of problems the Michaels are seeing (I'm not actually
> clear on whether the problem is also being experienced with FreeBSD or
> not).

The problem is being experienced in FreeBSD 6.x and 7.x with Intel gig 
interfaces.  It is also experienced on Linux 2.6.1[78].x with Intel gig 
and Neterion 10GE interfaces.  Only by altering the congestion control 
algorithm am I able to get good performance in a non-jumbo configuration.

All of the hardware is amd64.  Most is SMP, but I also see the problem 
on UP machines.  I am in the process of testing a 2.9Ghz P4 running a 
recent Fedorka distro (i386 with Broadcom tigon3 interfaces).

I thought about suggesting *BSD as well.  Of course there are a lot of 
differences within BSD, most notably, the process of fine-grain locking 
of the network stack in FreeBSD 5.x, and moreso in 6.x.  I plan on 
testing 4.11 when I can get another interface from my border box hooked 
up.  And then there's NetBSD and OpenBSD, and IIRC, NetBSD has been 
benchmarking quite well on the network side.

Setting aside the within-BSD differences, there's the question of how 
many OSes need to be represented in any measurement infrastructure. 
Putting FreeBSD in there leaves a good chance of catching issues that 
might also affect MacOS.  But then there's the big fat windows elephant 
in the room.  Should there also be Windows hosts in the measurement 
infrastructure?

By the way, here's the reason we're trying to make sure we have a clean 
path between UCB and UIUC:

http://www.citris-uc.org/Dec8-2006-Dance

michael