[net-perf] Re: [HPR-TAC] Progress on Pipes?

Fri Apr 22 13:41:01 PDT 2005

> the requirements for participation 
> (http://e2epi.internet2.edu/network-perf-wk/requirements.html) 

The above link for requirements doesn't provide much useful info on 
what the hardware should look like.  Below are a couple of emails on 
what Abilene and ESnet have been doing for hardware for BWCTL boxes 
for both GigE and 10 GigE (the 2nd message is focused on 10 GigE).

The I2 piPEs performance measurement point info web page lists
the hardware used for the existing set of BWCTL/OWAMP/NDT servers
in use in Abilene:

  http://e2epi.internet2.edu/pipes/ami/pmp-info.html

mb

----

Date: Tue, 19 Apr 2005 21:55:41 -0400
>From: Matthew J Zekauskas <matt at internet2.edu>
To: booloo at ucsc.edu
Cc: "Jeff W. Boote" <boote at internet2.edu>,
        Matthew J Zekauskas <matt at internet2.edu>

Hi Mark,

What you use for bwctl (or any throughtput measurements) will
depend on what you want your target rate to be.  We have found
that the chipset and memory subsystem is critical...  in particular
you definitely want to be able to use multiple banks for memory.

Background:   our original attempts to build a good gigE server
are here:
http://people.internet2.edu/~shalunov/gigatcp/

We bought the computers used in Abilene from ioncomputer.com.
They've been OK... we had some issues with the wrong serial
connector being delivered, and Jeff (and others) have had
compatability problems with our database systems.  The big
plus is 48vdc supplies.  The abilene specs are interspersed
in here:
<http://e2epi.internet2.edu/pipes/ami/pmp-info.html>

We are just installing some dell 1750's into Abilene as
additional gigE bwctl machines (they had some that could
be ordered with 48vdc supplies; apparently no longer).

Here's some additional Dell and 10GE information that's about
6 months old:

> The copper GigE's built into the 2650's work great (BCM5703X
> now using the tg3 driver).  If you need fiber (SX), the ONLY
> NICs I tested in there that ran NetPipe w/o failure were the
> Intel's.  Syskonnect and Netgear GA620's (my old favorites)
> both had problems in the 2650's.
>
> If you want 10GigE, I have only tried the 1st generation
> Intel's so far.  S2io couldn't deliver yet and the 2nd gen
> Intel's aren't out in LR yet (and none of our equipment is
> SR yet).  In my 2.4 GHz Xeon, 400 MHz FSB 2650's, I max out
> around 3.8 Gbps TCP.  Good enough for OC48 but clearly not
> even full PCI-X speed.  The newer ones with a 533 MHz FSB
> do a bit better.  2nd generation NICs would probably do
> better still.
>
> It is possible btw to get 2650's with DC power supplies and/or
> 2-post rack mounts, but you can't configure them that way via
> the web site!  We had to work through our sales guy for those.
>
> I'm quite happy with the 2650's - I've got ~30 in the field -
> and they are fine up to OC48.  In the long run (end of this
> year?) I am hoping to find something like the 2650 with PCI
> Express 10GigE support that can do wire speed 10GigE.
>

Here's what Sylvain Ravot says about 10GE (christian todorov of
I2 says that Sun actually has some boxes like this that are
reasonably priced):

>
> The best hardware for 10 GE interfaces is based on 64 bit CPU
> architecture.
>
> The best 10 GE card in term of throughput is the s2io adapter.
> http://www.s2io.com/products/xframe.html. From time to time the server
> hangs because of the card. We have tested the LR (long reach) adapter
> but there is a new SR (short reach adapter) that seems to be very
> stable.
>
> SR adapters are cheaper and more robust. We have broken several LR
> adapters.
>
> The max. throughput measured with Intel cards was 5,8 Gbps and the max.
> throughput measured with S2io was 6,5 Gbps.
>
> We have tested Itanium2 (Intel) and Opteron (AMD) systems. The Opteron
> system is slightly more performant than the Itanium2.
>
> Opteron config: Tyan Thunder K8S Pro (S2882), 2x Opteron 2.2 GHz , 2 GB
> memory
>
> Itanium config: HP rx4640, 4x 1.5GHz Itanium-2, zx1 chipset, 8GB memory
>

Hope this helps,

--Matt

-------------------------------------------------------------------------------

Date: Wednesday, April 20, 2005 9:37 AM -0500
>From: Joe Metzger <metzger at es.net>
To: Matthew J Zekauskas <matt at internet2.edu>
Subject: Re: specs for PC for 10GE testers?

Matt,
We bought these from a local integrator Finetec, who builds systems on request,
has a history of very prompt service to the lab. In fact, he drives over
and delivers them as soon as they are complete!

SYS-OP-252D-2GB-4U	$3,435.00ea.
**Includes**
Tyan S2895A2NRF IDE Version
http://www.tyan.com/products/html/thunderk8we.html
Dual AMD 252 Opteron CPUs with Fans
2GB DDR400 ECC/Reg. Memory
Floppy Drive and CDROM
120GB 7200RPM 8MB Buffer SATA Drive
AIC 4U Rackmount Case with 460W PS and Rail
**End**

Then we put in 2 Neterion X-Frame PCI-X NIC's in 2 of the systems and 1 NIC in the other two
systems.

After a bit of effort, our system administrator was able to load
Fedora Core 3, and apply appropriate patches to get it to support
the on-board Serial ATA controler and load the latest Neterion
Driver version 1.7.7.

Then we disabled ipchains, bumped up the MTU to 9K added the sysctl
settings recommended by Neterion.

After that, simple iperf runs with no tuning arguments achive 7.5 Gbps
on the 133 Mhz PCI-X bus and 5.5 Gbps on the 100 Mhz PCI-X bus.

I have not completed my other capabilities testing of the box due
to them being deployed to start circuit acceptance testing. However,
I hope to get them back into the lab and run some tests and maybe
put together a short 15-20 minute talk for the Joint Techs or ESCC
meeting this summer.

BTW, here are the Neterion recommended sysctl settings. This file is
included in their driver packages.

# some of the defaults may be different for your kernel
# call this file with sysctl -p <this file>
# these are just suggested values that worked well to increase throughput in
# several network benchmark tests, your mileage may vary

### IPV4 specific settings
# turns TCP timestamp support off, default 1, reduces CPU use
net.ipv4.tcp_timestamps = 0
# turn SACK support off, default on
net.ipv4.tcp_sack = 0
# on systems with a VERY fast bus -> memory interface this is the big gainer
# sets min/default/max TCP read buffer, default 4096 87380 174760
net.ipv4.tcp_rmem = 10000000 10000000 10000000
# sets min/pressure/max TCP write buffer, default 4096 16384 131072
net.ipv4.tcp_wmem = 10000000 10000000 10000000
# sets min/pressure/max TCP buffer space, default 31744 32256 32768
net.ipv4.tcp_mem = 10000000 10000000 10000000

### CORE settings (mostly for socket and UDP effect)
# maximum receive socket buffer size, default 131071
net.core.rmem_max = 524287
# maximum send socket buffer size, default 131071
net.core.wmem_max = 524287
# default receive socket buffer size, default 65535
net.core.rmem_default = 524287
# default send socket buffer size, default 65535
net.core.wmem_default = 524287
# maximum amount of option memory buffers, default 10240
net.core.optmem_max = 524287
# number of unprocessed input packets before kernel starts dropping them, default 300
net.core.netdev_max_backlog = 300000

Matthew J Zekauskas wrote:
> I'm curious as to the specs for the PC you are using for
> the 10GE testers that you are building.  Mostly to just
> be able to compare notes later :).  If you tell me when
> you get a chance?  Also the supplier would be of interest.
>
> --Matt

-- 
Joe Metzger
metzger at es.net
Network Engineering,  ESnet
Lawrence Berkeley National Laboratory