How to use IPerf properly – additions to the TCP throughput post

IPerf is a tool to measure network throughput.



It is very efficient but if one use it without proper knowledge the
results can lead you to trouble.

It can measure with both TCP and UDP.

You should know how TCP and UDP works. In TCP we have
acknowledges periodically by the nature of the operation. In UDP
there are no acknowledges by the protocol, the application should
handle the packetloss (if any..) and the control.

Let’s see two examples.

FTP/SCP copy. This connection is using TCP. Before we can send
the next packets we should wait for the acknowledge of the sent
packets. Send-send-wait4ack-send-send-wait4ack-send-send-wait..
If some packets lost TCP will resend it.

VoIP telephony. This connection is using UDP. The phones sending
packets periodically at a predefined rate with predefined size
of packets therefore at a predefined bandwidth. No matter what
happens (drops can occure) IP phone will send-send-send-send…

You can see that the TCP flow is “under control”, but UDP can go
“out of control”. This we can use for measuring the throughput.


The IPerf tool is a client-server application. Basicaly the client
sends the data to the server. After they joined the client starts
to send data to the server based on the parameters on both sides.
We can set to make a single direction, bidirectional one-after-one
and bidirectional parallel measurement. We can also set to do the
test with one or several other parallel sessions and we can set
the size of the packets also. This means that with the proper
settings you can simulate file transfers (FTP, SCP, SMB..) or for
example VoIP calls (if you know the codec sample rate/sample size
you can calculate the proper packet size and bandwidth parameters).

Because when you measure you send packets from client to server
you should consider this behavior when you set up the application
instances. The direction is the key.

If you want to test the FTP download speed from A to B you should
start the server at site B and the client on site A but the FTP
server is at site A and the client is at site B. To test the upload
from A to B you should swap the roles: A should be the client and
B should be the server. This is important because you should set
the receive window size to the value you want to measure at. Here
are some examples: standard is 64KB, Windows XP/Vista/Win7 17.5KB,
FTP 8KB, SCP 64KB, SMB 4K blokk size.. Or you should check your
application what you want to test (SAP for example). This should be
set at least the server side, but you should set it on client side also if
you want to measure both directions parallel.

Testing/simulating VoIP is also not so complicated, you only need
the codec/sample rate/sample size/sample interval and you can
calculate the packet payload size and the bandwidth parameters.
You can use the already attached picture above (assembled by me).

Let’s start the command line part.

Usage: iperf [-s|-c host] [-p port] [-t secs] [-w bytes] …
       iperf [-h|–help] [-v|–version]

Client/Server:
  -f, –format    [kmKM]   format to report: Kbits, Mbits, KBytes, MBytes
  -i, –interval  #        seconds between periodic bandwidth reports
  -l, –len       #[KM]    length of buffer to read or write (default 8 KB)
  -m, –print_mss          print TCP maximum segment size (MTU – TCP/IP header)
  -p, –port      #        server port to listen on/connect to
  -u, –udp                use UDP rather than TCP
  -w, –window    #[KM]    TCP window size (socket buffer size)
  -B, –bind         bind to , an interface or multicast address
  -M, –mss       #        set TCP maximum segment size (MTU – 40 bytes)
  -N, –nodelay            set TCP no delay, disabling Nagle’s Algorithm
  -V, –IPv6Version        Set the domain to IPv6

Server specific:
  -s, –server             run in server mode
  -D, –daemon    run the server as a daemon

Client specific:
  -a, –tcp_bandwidth \    for TCP, bandwidth to send at in bits/sec
  #[KM]    (default no bandwidth limit used)
  -b, –bandwidth #[KM]    for UDP, bandwidth to send at in bits/sec
                           (default 1 Mbit/sec, implies -u)
  -c, –client       run in client mode, connecting to 
  -n, –num       #[KM]    number of bytes to transmit (instead of -t)
  -t, –time      #        time in seconds to transmit for (default 10 secs)
  -F, –fileinput    input the data to be transmitted from a file
  -I, –stdin              input the data to be transmitted from stdin
  -P, –parallel  #        number of parallel client threads to run
  -S, –tos       #        set type-of-service for outgoing packets
  -T, –ttl       #        time-to-live, for multicast (default 1)
  -W, –windowSizeSuggest  Run the client so as to suggest a suitable window size (default off)

Miscellaneous:
  -h, –help               print this message and quit
  -v, –version            print version information and quit

[KM] Indicates options that support a K or M suffix for kilo- or mega-

The TCP window size option can be set by the environment variable
TCP_WINDOW_SIZE. Most other options can be set by an environment variable
IPERF_, such as IPERF_BANDWIDTH.


To measure the throughput for FTP connection from A to B you should issue
Site B: iperf -s -w 8K
Site A: iperf -c site.B.IP.address -fk -i 10 -m -t 60
This will start a simple FTP-like connection and do it for 60 seconds,
make bandwidth report in every 10s in kbps (printing MSS).


To measure the throughput for SCP connection from A to B issue this:
Site B: iperf -s -w 64K
Site A: iperf -c site.B.IP.address -fk -i 10 -m -t 60
This will start a simple SCP-like connection and do it for 60 seconds,
make bandwidth report in every 10s in kbps (printing MSS).


Please check the TCP throughput calculations article to evaluate the results.


To generate VoIP/G711-like, bidirectional traffic for 4 “calls” issue this:
Site A: iperf -c site.B.IP.address -u -mss 160 -b 65000 -S 184 -P 4 -fk -i 10 -t 300
Site B: iperf -s -u -mss 160 -b 65000 -S 184 -P 4 -fk -i 10
This will start 4 simple bidirectional VoIP/G711-like flows for 300 seconds
marked with DSCP EF (TOS 184), make bandwidth report in every 10s in kbps.


To check a link to see if you can use it at the full capacity you
can saturate the link with these commands (the -b value must be
high enough, equal or more than the link capacity):
Site A: iperf -c site.B.IP.address -u -mss 160 -b 10M -fk -i 10 -t 300
Site B: iperf -s -u -b 10M -fk -i 10
You can do 2 separate or one simple bidirectional measurement.


The JPerf is a GUI for IPerf, you can set the parameters in a Java GUI.


A functionality decreased IPerf is implemented in the Cisco IOSes.
It’s name is TTCP, it is unsupported and you should know that in
this case you utilise the router CPU and this can limit/tamper the
results, use it on your own risk and and not trust it.


Please always check both sides when you evaluate the results.
You should check what is sent and what is received and see the
correct conclusion. Hope this helps.

Maximum TCP throughput – the user experiences “low bandwidth” when up/downloading

Maximum TCP throughput calculations:
There are several important things what you should know before reading this.
Please check TCP/IP fundamentals at  http://www.tcpipguide.com/free/index.htm
Common window sizes:
Standard 64KB
Windows XP 17.5KB
FTP 8KB
SCP 64KB
SMB 16K/4K

TCP throughput:

http://wand.net.nz/~perry/max_download.php << this is a calculator, it can show you the throughut weakness
http://www.babinszki.com/Networking/Max-Ethernet-and-TCP-Throughput.html

http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/ 

Data throughput in TCP based on TCP window size and latency:
FTP TCP window 8KB -> 65356bit
RTT=60ms=0.06s
Maximum TCP transfer = 65356bit/0.06s=1.038Mbps

 

General TCP window 64KB -> 524288bit
RTT=60ms=0.06s
Maximum TCP transfer = 524288bit/0.06s=8738133,33bps = 8.533.33Mbps

 

DataMover TCP window 256KB -> 2097152bit
RTT=60ms=0.06s
Maximum TCP transfer = 2097152bit/0.06s=33.33Mbps

 

Throughput can never exceed window size divided by round-trip time.
window = 17520byte = 140160
rtt 60ms = 0.06s
Max link throughput (based on window and rtt) = 140160bit/0.06s=2336000 = 2281,25kbps

 

The capacity of a pipe is its bandwidth multiplied by round-trip time.
BW = 2Mbps = 2097152bps
rtt = 60ms = 0.06s
Max link capacity (based on BW and rtt) = 125829,12bit = 15728,64byte

 

Formula to calculate the optimal TCP window size:
Bandwidth-in-bits-per-second * Round-trip-latency-in-seconds = TCP window size in bits / 8 = TCP window size in bytes
BW = 2Mbps = 2097152bps
rtt = 60ms = 0.06s
2097152bps * 0.06s = 125829,12bit = 15728,64byte — ideal window size

 

Formula to calculate Maximum Latency for a desired throughput
TCP-window-size-bits / Desired-throughput-in-bits-per-second = Maximum RTT Latency
TCP window size = 17520byte = 140160bit
desired BW = 2Mbps = 2097152bps
140160bit / 2097152bps = ~0.067ms is the max RTT for the desired 2M throughput

 

 

Please consider that these values are IP packets, not frames (or Layer2 bandwidths) or the packets’ data transers.
Layer2 overheads will “decrease” the link capacity:  http://sd.wareonearth.com/~phil/net/overhead/

2 hosts on different LANs cannot communicate to each other..

[HOST1]=[ETH SWITCH]=[ROUTER1 ** CLOUD ** ROUTER2]=[ETHSWITCH]=[HOST2]

HOST1: 10.0.0.100/24 – def.gw: 10.0.0.1
HOST2: 10.0.1.100/24 – def.gw: 10.0.1.1

You have access only to the routers.

HOST1 is pingable from ROUTER1 but not pingable from HOST2/ROUTER2.
Traceroute from HOST2 to HOST1 stops at the ROUTER1.
Traceroute from HOST1 to HOST2 has no hops.

Possible problems:

@0: Please check the router interface configurations: IPs and masks are correct?

@1: Please never trust the host settings:
If HOST1 has a bad default gateway then it can routes the ping/traceroute into a black hole.

@2: Please never trust the host settings:
If HOST1 has a bad subnet mask (255.255.254.0 instead of 255.255.255.0)
then HOST1 thinks that HOST2 is in the same LAN segment of her!
10.0.0.0/23 -> hosts: 10.0.0.1 .. 10.0.1.254 -> HOST2 is falling into this range.

You can see (on a Cisco router) if the problem is one of these by:

Please issue term mon and then deb arp and watch the incoming
ARP requests and look for the unreachable HOST1’s entries.
Ask HOST1’s operator to clear the ARP cache, HOST1 should resolve
the MAC address of the default gateway and you should see this
request on the router (ARP is L2 broadcast). You also should see
if HOST1 tries to resolve the MAC of HOST2. (If you have Junipers
you can user the monitor interface traffic commands to achieve the
same ARP wathing function).

Then ask the operator to check the host IP configuration (static/DHCP IPs?).

If this not lead into success please ask to check the HOST1 firewall rules
(may be HOST2 also) and the ACLs applied on the interfaces in the path.

Another problem:
HOST1 is conf’d to use DHCP to obtain IP address.
If the DHCP server not send an offer (not reachable for HOST1 or there
is a restriction for HOST1’s MAC address.. etc..) then a Windows host
will give an address to the interface of the link-local range 169.254.0.0/16
(zero config networking). If this happens the trace from HOST2 also stops
at ROUTER1. You can check this if you configure the interface IP of ROUTER1
(facing to HOST1) to 169.254.0.1/16 and ping the address 169.254.255.255
and HOST1 will reply, you will see the ARP entry in ARP cache and you can
say the MAC address of HOST1 to the operator to check if it is really HOST1.

 

How to simulate WAN delay/latency in GNS3

Download or create a QEMU linux where tc (traffic control) is working.
I chosed Debian Lenny standard from http://people.debian.org/~aurel32/qemu/armel/
o Run it one time outside GNS3, we need bridging utilities to be installed:
 qemu -name QEMU1 -m 256 -serial telnet:127.0.0.1:3000,server,nowait -no-acpi debian_lenny_i386_standard.img
o telnet to the host
 telnet 127.0.0.1 3000 or use putty
o Install bridge-utils
 aptitude install bridge-utils
o After you installed bridge-utils add the QEMU host to GNS3 (like in Linux microcore article)
o Add the host to GNS3 working area, connect the links to the routers/hosts and start the network:
o Set the delay and jitter and drop values for this connection:
 (I usually add 6 eth connections, in the mentioned QEMU image the first eth link is not eth0 but eth1)

 # Bring up the interfaces without IP addresses
 ifconfig eth1 0.0.0.0 up
 ifconfig eth2 0.0.0.0 up
 ifconfig eth3 0.0.0.0 up
 ifconfig eth4 0.0.0.0 up
 ifconfig eth5 0.0.0.0 up
 ifconfig eth6 0.0.0.0 up
 # Create a bridge group
 brctl addbr br0
 # Add the interfaces to the bridge group
 brctl addif br0 eth1
 brctl addif br0 eth2
 brctl addif br0 eth3
 brctl addif br0 eth4
 brctl addif br0 eth5
 brctl addif br0 eth6
 # Bring the bridge group up
 ifconfig br0 up
 # Check the interfaces – all up, no problems?
 ifconfig
 # Remove the root traffic control from the interfaces
 tc qdisc del dev eth1 root
 tc qdisc del dev eth2 root
 tc qdisc del dev eth3 root
 tc qdisc del dev eth4 root
 tc qdisc del dev eth5 root
 tc qdisc del dev eth6 root
 tc qdisc add dev eth1 root handle 1:0 netem delay 120msec
 tc qdisc add dev eth2 root handle 1:0 netem delay 120msec
 tc qdisc add dev eth3 root handle 1:0 netem delay 120msec
 tc qdisc add dev eth4 root handle 1:0 netem delay 120msec
 tc qdisc add dev eth5 root handle 1:0 netem delay 120msec
 tc qdisc add dev eth6 root handle 1:0 netem delay 120msec

 o Real WAN shows variable delay sometime, this adds 100ms ± 10ms with a correlation value (the next random element depending 25% on the last one).
 tc qdisc change dev eth0 root netem delay 100ms 10ms 25%
 o Delay distribution
 tc qdisc change dev eth0 root netem delay 100ms 20ms distribution normal
 o Random packet loss
 tc qdisc change dev eth0 root netem loss 0.1%
 o Packet duplication
 tc qdisc change dev eth0 root netem duplicate 1%
 o Packet corruption
 tc qdisc change dev eth0 root netem corrupt 0.1% 
o Packet re-ordering, 2 method, please do not mix them (the 2nd is more realistic)
– 1st way: gap uses a fixed sequence and reorders every Nth packet
 tc qdisc change dev eth0 root netem gap 5 delay 10ms
– 2nd way: a certain percentage of the packets to get mis-ordered:
 tc qdisc change dev eth0 root netem delay 10ms reorder 25% 50%

Add Microcore linux host to GNS3

o Download the image linux-microcore-4.0.2-clean.img from http://sourceforge.net/projects/gns-3/files/Qemu%20Appliances/
o Start the image, we have to install some applications:
 qemu -name QEMU -m 256 -serial telnet:127.0.0.1:3000,server,nowait -no-acpi linux-microcore-4.0.2-clean.img
o Telnet to the host’s console – username root / password root – sudo su -> get root
 telnet 127.0.0.1 3000 or use putty
o Install some applications to the host (iPerf, freeradius, lighttpd, proftpd, ntpd… not as root, not in “sudo su”):
 tc@box:~$ tce-load -iw bwmon cifs-utils d-itg dhcpcd e2fsprogs elinks freeradius freeradius-dev freeradius-doc htop iperf iptraf jed less lighttpd net-snmp net-tools netperf nmap ntop ntpclient proftpd rsync screen sed sftp-server tcpdump tmux vim wget bridge-utils openssh
o Add the host to GNS3 – Edit -> Preferences -> QEMU -> QEMU host -> Name the host, browse the image, RAM 128M, NICs 2 (or whatever you choose), NIC model e1000, Qemu options: -no-acpi -> Save

o Add the host to GNS3 working area, connect the links to the routers/hosts and start the network:
 Configure hostname, eth interfaces and add default route like this:
 sudo su
 hostname Qemu1
 ifconfig eth0 10.10.10.2 netmask 255.255.255.0 up
 route add default gw 10.10.10.1

Now you have a fully functional linux host in GNS3 with iPerf, SSH etc..

Switch port port-security maximum MAC address when using IP phone

Switch port port-security maximum MAC address when using IP phone
on the switch link with microswitch for PC connectivity on the phone:

Problem: [switch]–[IP Phone]–[PC]
If the user’s switchport config is like this:

errdisable recovery cause psecure-violation
errdisable recovery interval 600
!
interface x/y/z
switchport port-security aging time 5
switchport port-security aging type inactivity
switchport port-security violation shutdown
switchport port-security maximum 2
switchport port-security maximum 1 vlan access
switchport port-security maximum 1 vlan voice

Ideally this will work. But it will cause problems.

When the user turn on the PC he/she will see a system notification that
the link went down (and the phone switched of of PoE powered or loss
the connectivity to the switch). This is because of the port-security config
and a strange bahavior of the IP Phone.

At start the phone is using it’s MAC for CDP in both VLANs, in the native
and the voice also because at startup the the phone not know which is
the voice VLAN -> 2 MAC. When the PC come up and sends a single
packet (DHCP, ARP..) then we have a 3rd MAC. Until the MAC of the phone
in the data (native) VLAN not aged out the port will be secured. If you are
lucky then the phone remains up and got power and not forget the voice
VLAN. After the aging time expires and the auto recovery recovered the
port the PC can communicate because the MAC aging already deleted the
MAC of the phone from the data VLAN. And another problem: some phones
have a weird behavior. The phones have a feature of CDP: the host presence
type length value (TLV) to notify the switch of changes in the attached device’s
port link state. The phones generates this CDP frame in the voice VLAN – this
is a control message to the switch to remove the PC’s MAC from the CAM table.
But some phone plattforms (in my experience) generates this message in the
data (native) VLAN so the switch sees the phone’s MAC in both VLANs. This
leads to the problem that when the PC booted and send the first packet the
switch will secure the port because of the MAC violation. After the MAC aging
time and the automatic error recovery of the port the PC will be able to send
and receive packets on the port.

Solution:
interface x/y/z
switchport port-security maximum 3
switchport port-security maximum 2 vlan access
switchport port-security maximum 1 vlan voice

linkek — Telecommunication lines