How to use IPerf properly – additions to the TCP throughput post

IPerf is a tool to measure network throughput.



It is very efficient but if one use it without proper knowledge the
results can lead you to trouble.

It can measure with both TCP and UDP.

You should know how TCP and UDP works. In TCP we have
acknowledges periodically by the nature of the operation. In UDP
there are no acknowledges by the protocol, the application should
handle the packetloss (if any..) and the control.

Let’s see two examples.

FTP/SCP copy. This connection is using TCP. Before we can send
the next packets we should wait for the acknowledge of the sent
packets. Send-send-wait4ack-send-send-wait4ack-send-send-wait..
If some packets lost TCP will resend it.

VoIP telephony. This connection is using UDP. The phones sending
packets periodically at a predefined rate with predefined size
of packets therefore at a predefined bandwidth. No matter what
happens (drops can occure) IP phone will send-send-send-send…

You can see that the TCP flow is “under control”, but UDP can go
“out of control”. This we can use for measuring the throughput.


The IPerf tool is a client-server application. Basicaly the client
sends the data to the server. After they joined the client starts
to send data to the server based on the parameters on both sides.
We can set to make a single direction, bidirectional one-after-one
and bidirectional parallel measurement. We can also set to do the
test with one or several other parallel sessions and we can set
the size of the packets also. This means that with the proper
settings you can simulate file transfers (FTP, SCP, SMB..) or for
example VoIP calls (if you know the codec sample rate/sample size
you can calculate the proper packet size and bandwidth parameters).

Because when you measure you send packets from client to server
you should consider this behavior when you set up the application
instances. The direction is the key.

If you want to test the FTP download speed from A to B you should
start the server at site B and the client on site A but the FTP
server is at site A and the client is at site B. To test the upload
from A to B you should swap the roles: A should be the client and
B should be the server. This is important because you should set
the receive window size to the value you want to measure at. Here
are some examples: standard is 64KB, Windows XP/Vista/Win7 17.5KB,
FTP 8KB, SCP 64KB, SMB 4K blokk size.. Or you should check your
application what you want to test (SAP for example). This should be
set at least the server side, but you should set it on client side also if
you want to measure both directions parallel.

Testing/simulating VoIP is also not so complicated, you only need
the codec/sample rate/sample size/sample interval and you can
calculate the packet payload size and the bandwidth parameters.
You can use the already attached picture above (assembled by me).

Let’s start the command line part.

Usage: iperf [-s|-c host] [-p port] [-t secs] [-w bytes] …
       iperf [-h|–help] [-v|–version]

Client/Server:
  -f, –format    [kmKM]   format to report: Kbits, Mbits, KBytes, MBytes
  -i, –interval  #        seconds between periodic bandwidth reports
  -l, –len       #[KM]    length of buffer to read or write (default 8 KB)
  -m, –print_mss          print TCP maximum segment size (MTU – TCP/IP header)
  -p, –port      #        server port to listen on/connect to
  -u, –udp                use UDP rather than TCP
  -w, –window    #[KM]    TCP window size (socket buffer size)
  -B, –bind         bind to , an interface or multicast address
  -M, –mss       #        set TCP maximum segment size (MTU – 40 bytes)
  -N, –nodelay            set TCP no delay, disabling Nagle’s Algorithm
  -V, –IPv6Version        Set the domain to IPv6

Server specific:
  -s, –server             run in server mode
  -D, –daemon    run the server as a daemon

Client specific:
  -a, –tcp_bandwidth \    for TCP, bandwidth to send at in bits/sec
  #[KM]    (default no bandwidth limit used)
  -b, –bandwidth #[KM]    for UDP, bandwidth to send at in bits/sec
                           (default 1 Mbit/sec, implies -u)
  -c, –client       run in client mode, connecting to 
  -n, –num       #[KM]    number of bytes to transmit (instead of -t)
  -t, –time      #        time in seconds to transmit for (default 10 secs)
  -F, –fileinput    input the data to be transmitted from a file
  -I, –stdin              input the data to be transmitted from stdin
  -P, –parallel  #        number of parallel client threads to run
  -S, –tos       #        set type-of-service for outgoing packets
  -T, –ttl       #        time-to-live, for multicast (default 1)
  -W, –windowSizeSuggest  Run the client so as to suggest a suitable window size (default off)

Miscellaneous:
  -h, –help               print this message and quit
  -v, –version            print version information and quit

[KM] Indicates options that support a K or M suffix for kilo- or mega-

The TCP window size option can be set by the environment variable
TCP_WINDOW_SIZE. Most other options can be set by an environment variable
IPERF_, such as IPERF_BANDWIDTH.


To measure the throughput for FTP connection from A to B you should issue
Site B: iperf -s -w 8K
Site A: iperf -c site.B.IP.address -fk -i 10 -m -t 60
This will start a simple FTP-like connection and do it for 60 seconds,
make bandwidth report in every 10s in kbps (printing MSS).


To measure the throughput for SCP connection from A to B issue this:
Site B: iperf -s -w 64K
Site A: iperf -c site.B.IP.address -fk -i 10 -m -t 60
This will start a simple SCP-like connection and do it for 60 seconds,
make bandwidth report in every 10s in kbps (printing MSS).


Please check the TCP throughput calculations article to evaluate the results.


To generate VoIP/G711-like, bidirectional traffic for 4 “calls” issue this:
Site A: iperf -c site.B.IP.address -u -mss 160 -b 65000 -S 184 -P 4 -fk -i 10 -t 300
Site B: iperf -s -u -mss 160 -b 65000 -S 184 -P 4 -fk -i 10
This will start 4 simple bidirectional VoIP/G711-like flows for 300 seconds
marked with DSCP EF (TOS 184), make bandwidth report in every 10s in kbps.


To check a link to see if you can use it at the full capacity you
can saturate the link with these commands (the -b value must be
high enough, equal or more than the link capacity):
Site A: iperf -c site.B.IP.address -u -mss 160 -b 10M -fk -i 10 -t 300
Site B: iperf -s -u -b 10M -fk -i 10
You can do 2 separate or one simple bidirectional measurement.


The JPerf is a GUI for IPerf, you can set the parameters in a Java GUI.


A functionality decreased IPerf is implemented in the Cisco IOSes.
It’s name is TTCP, it is unsupported and you should know that in
this case you utilise the router CPU and this can limit/tamper the
results, use it on your own risk and and not trust it.


Please always check both sides when you evaluate the results.
You should check what is sent and what is received and see the
correct conclusion. Hope this helps.

Maximum TCP throughput – the user experiences “low bandwidth” when up/downloading

Maximum TCP throughput calculations:
There are several important things what you should know before reading this.
Please check TCP/IP fundamentals at  http://www.tcpipguide.com/free/index.htm
Common window sizes:
Standard 64KB
Windows XP 17.5KB
FTP 8KB
SCP 64KB
SMB 16K/4K

TCP throughput:

http://wand.net.nz/~perry/max_download.php << this is a calculator, it can show you the throughut weakness
http://www.babinszki.com/Networking/Max-Ethernet-and-TCP-Throughput.html

http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/ 

Data throughput in TCP based on TCP window size and latency:
FTP TCP window 8KB -> 65356bit
RTT=60ms=0.06s
Maximum TCP transfer = 65356bit/0.06s=1.038Mbps

 

General TCP window 64KB -> 524288bit
RTT=60ms=0.06s
Maximum TCP transfer = 524288bit/0.06s=8738133,33bps = 8.533.33Mbps

 

DataMover TCP window 256KB -> 2097152bit
RTT=60ms=0.06s
Maximum TCP transfer = 2097152bit/0.06s=33.33Mbps

 

Throughput can never exceed window size divided by round-trip time.
window = 17520byte = 140160
rtt 60ms = 0.06s
Max link throughput (based on window and rtt) = 140160bit/0.06s=2336000 = 2281,25kbps

 

The capacity of a pipe is its bandwidth multiplied by round-trip time.
BW = 2Mbps = 2097152bps
rtt = 60ms = 0.06s
Max link capacity (based on BW and rtt) = 125829,12bit = 15728,64byte

 

Formula to calculate the optimal TCP window size:
Bandwidth-in-bits-per-second * Round-trip-latency-in-seconds = TCP window size in bits / 8 = TCP window size in bytes
BW = 2Mbps = 2097152bps
rtt = 60ms = 0.06s
2097152bps * 0.06s = 125829,12bit = 15728,64byte — ideal window size

 

Formula to calculate Maximum Latency for a desired throughput
TCP-window-size-bits / Desired-throughput-in-bits-per-second = Maximum RTT Latency
TCP window size = 17520byte = 140160bit
desired BW = 2Mbps = 2097152bps
140160bit / 2097152bps = ~0.067ms is the max RTT for the desired 2M throughput

 

 

Please consider that these values are IP packets, not frames (or Layer2 bandwidths) or the packets’ data transers.
Layer2 overheads will “decrease” the link capacity:  http://sd.wareonearth.com/~phil/net/overhead/

2 hosts on different LANs cannot communicate to each other..

[HOST1]=[ETH SWITCH]=[ROUTER1 ** CLOUD ** ROUTER2]=[ETHSWITCH]=[HOST2]

HOST1: 10.0.0.100/24 – def.gw: 10.0.0.1
HOST2: 10.0.1.100/24 – def.gw: 10.0.1.1

You have access only to the routers.

HOST1 is pingable from ROUTER1 but not pingable from HOST2/ROUTER2.
Traceroute from HOST2 to HOST1 stops at the ROUTER1.
Traceroute from HOST1 to HOST2 has no hops.

Possible problems:

@0: Please check the router interface configurations: IPs and masks are correct?

@1: Please never trust the host settings:
If HOST1 has a bad default gateway then it can routes the ping/traceroute into a black hole.

@2: Please never trust the host settings:
If HOST1 has a bad subnet mask (255.255.254.0 instead of 255.255.255.0)
then HOST1 thinks that HOST2 is in the same LAN segment of her!
10.0.0.0/23 -> hosts: 10.0.0.1 .. 10.0.1.254 -> HOST2 is falling into this range.

You can see (on a Cisco router) if the problem is one of these by:

Please issue term mon and then deb arp and watch the incoming
ARP requests and look for the unreachable HOST1’s entries.
Ask HOST1’s operator to clear the ARP cache, HOST1 should resolve
the MAC address of the default gateway and you should see this
request on the router (ARP is L2 broadcast). You also should see
if HOST1 tries to resolve the MAC of HOST2. (If you have Junipers
you can user the monitor interface traffic commands to achieve the
same ARP wathing function).

Then ask the operator to check the host IP configuration (static/DHCP IPs?).

If this not lead into success please ask to check the HOST1 firewall rules
(may be HOST2 also) and the ACLs applied on the interfaces in the path.

Another problem:
HOST1 is conf’d to use DHCP to obtain IP address.
If the DHCP server not send an offer (not reachable for HOST1 or there
is a restriction for HOST1’s MAC address.. etc..) then a Windows host
will give an address to the interface of the link-local range 169.254.0.0/16
(zero config networking). If this happens the trace from HOST2 also stops
at ROUTER1. You can check this if you configure the interface IP of ROUTER1
(facing to HOST1) to 169.254.0.1/16 and ping the address 169.254.255.255
and HOST1 will reply, you will see the ARP entry in ARP cache and you can
say the MAC address of HOST1 to the operator to check if it is really HOST1.

 

How to simulate WAN delay/latency in GNS3

Download or create a QEMU linux where tc (traffic control) is working.
I chosed Debian Lenny standard from http://people.debian.org/~aurel32/qemu/armel/
o Run it one time outside GNS3, we need bridging utilities to be installed:
 qemu -name QEMU1 -m 256 -serial telnet:127.0.0.1:3000,server,nowait -no-acpi debian_lenny_i386_standard.img
o telnet to the host
 telnet 127.0.0.1 3000 or use putty
o Install bridge-utils
 aptitude install bridge-utils
o After you installed bridge-utils add the QEMU host to GNS3 (like in Linux microcore article)
o Add the host to GNS3 working area, connect the links to the routers/hosts and start the network:
o Set the delay and jitter and drop values for this connection:
 (I usually add 6 eth connections, in the mentioned QEMU image the first eth link is not eth0 but eth1)

 # Bring up the interfaces without IP addresses
 ifconfig eth1 0.0.0.0 up
 ifconfig eth2 0.0.0.0 up
 ifconfig eth3 0.0.0.0 up
 ifconfig eth4 0.0.0.0 up
 ifconfig eth5 0.0.0.0 up
 ifconfig eth6 0.0.0.0 up
 # Create a bridge group
 brctl addbr br0
 # Add the interfaces to the bridge group
 brctl addif br0 eth1
 brctl addif br0 eth2
 brctl addif br0 eth3
 brctl addif br0 eth4
 brctl addif br0 eth5
 brctl addif br0 eth6
 # Bring the bridge group up
 ifconfig br0 up
 # Check the interfaces – all up, no problems?
 ifconfig
 # Remove the root traffic control from the interfaces
 tc qdisc del dev eth1 root
 tc qdisc del dev eth2 root
 tc qdisc del dev eth3 root
 tc qdisc del dev eth4 root
 tc qdisc del dev eth5 root
 tc qdisc del dev eth6 root
 tc qdisc add dev eth1 root handle 1:0 netem delay 120msec
 tc qdisc add dev eth2 root handle 1:0 netem delay 120msec
 tc qdisc add dev eth3 root handle 1:0 netem delay 120msec
 tc qdisc add dev eth4 root handle 1:0 netem delay 120msec
 tc qdisc add dev eth5 root handle 1:0 netem delay 120msec
 tc qdisc add dev eth6 root handle 1:0 netem delay 120msec

 o Real WAN shows variable delay sometime, this adds 100ms ± 10ms with a correlation value (the next random element depending 25% on the last one).
 tc qdisc change dev eth0 root netem delay 100ms 10ms 25%
 o Delay distribution
 tc qdisc change dev eth0 root netem delay 100ms 20ms distribution normal
 o Random packet loss
 tc qdisc change dev eth0 root netem loss 0.1%
 o Packet duplication
 tc qdisc change dev eth0 root netem duplicate 1%
 o Packet corruption
 tc qdisc change dev eth0 root netem corrupt 0.1% 
o Packet re-ordering, 2 method, please do not mix them (the 2nd is more realistic)
– 1st way: gap uses a fixed sequence and reorders every Nth packet
 tc qdisc change dev eth0 root netem gap 5 delay 10ms
– 2nd way: a certain percentage of the packets to get mis-ordered:
 tc qdisc change dev eth0 root netem delay 10ms reorder 25% 50%

Add Microcore linux host to GNS3

o Download the image linux-microcore-4.0.2-clean.img from http://sourceforge.net/projects/gns-3/files/Qemu%20Appliances/
o Start the image, we have to install some applications:
 qemu -name QEMU -m 256 -serial telnet:127.0.0.1:3000,server,nowait -no-acpi linux-microcore-4.0.2-clean.img
o Telnet to the host’s console – username root / password root – sudo su -> get root
 telnet 127.0.0.1 3000 or use putty
o Install some applications to the host (iPerf, freeradius, lighttpd, proftpd, ntpd… not as root, not in “sudo su”):
 tc@box:~$ tce-load -iw bwmon cifs-utils d-itg dhcpcd e2fsprogs elinks freeradius freeradius-dev freeradius-doc htop iperf iptraf jed less lighttpd net-snmp net-tools netperf nmap ntop ntpclient proftpd rsync screen sed sftp-server tcpdump tmux vim wget bridge-utils openssh
o Add the host to GNS3 – Edit -> Preferences -> QEMU -> QEMU host -> Name the host, browse the image, RAM 128M, NICs 2 (or whatever you choose), NIC model e1000, Qemu options: -no-acpi -> Save

o Add the host to GNS3 working area, connect the links to the routers/hosts and start the network:
 Configure hostname, eth interfaces and add default route like this:
 sudo su
 hostname Qemu1
 ifconfig eth0 10.10.10.2 netmask 255.255.255.0 up
 route add default gw 10.10.10.1

Now you have a fully functional linux host in GNS3 with iPerf, SSH etc..

Switch port port-security maximum MAC address when using IP phone

Switch port port-security maximum MAC address when using IP phone
on the switch link with microswitch for PC connectivity on the phone:

Problem: [switch]–[IP Phone]–[PC]
If the user’s switchport config is like this:

errdisable recovery cause psecure-violation
errdisable recovery interval 600
!
interface x/y/z
switchport port-security aging time 5
switchport port-security aging type inactivity
switchport port-security violation shutdown
switchport port-security maximum 2
switchport port-security maximum 1 vlan access
switchport port-security maximum 1 vlan voice

Ideally this will work. But it will cause problems.

When the user turn on the PC he/she will see a system notification that
the link went down (and the phone switched of of PoE powered or loss
the connectivity to the switch). This is because of the port-security config
and a strange bahavior of the IP Phone.

At start the phone is using it’s MAC for CDP in both VLANs, in the native
and the voice also because at startup the the phone not know which is
the voice VLAN -> 2 MAC. When the PC come up and sends a single
packet (DHCP, ARP..) then we have a 3rd MAC. Until the MAC of the phone
in the data (native) VLAN not aged out the port will be secured. If you are
lucky then the phone remains up and got power and not forget the voice
VLAN. After the aging time expires and the auto recovery recovered the
port the PC can communicate because the MAC aging already deleted the
MAC of the phone from the data VLAN. And another problem: some phones
have a weird behavior. The phones have a feature of CDP: the host presence
type length value (TLV) to notify the switch of changes in the attached device’s
port link state. The phones generates this CDP frame in the voice VLAN – this
is a control message to the switch to remove the PC’s MAC from the CAM table.
But some phone plattforms (in my experience) generates this message in the
data (native) VLAN so the switch sees the phone’s MAC in both VLANs. This
leads to the problem that when the PC booted and send the first packet the
switch will secure the port because of the MAC violation. After the MAC aging
time and the automatic error recovery of the port the PC will be able to send
and receive packets on the port.

Solution:
interface x/y/z
switchport port-security maximum 3
switchport port-security maximum 2 vlan access
switchport port-security maximum 1 vlan voice

How to check packet marking and traffic instantly

#How to check packet marking and traffic instantly

##################################################
#Config part
interface ...
ip flow ingress
ip flow egress
# To see the result:
sh ip cache flow  --  complete list of flows
sh ip cache verbose flow  --  complete list of flows, each flow will be verbosely showed (example type of service field)
sh ip cache ip.network.to.check network.mask.for.that verbose flow  --  flow will be verbosely showed for a specifig range of IP(s) (example type of service field ; netmask can be 255.255.255.255 to see only one host)
# If you have MLS:
# Config part
mls ip multicast flow-stat-timer 9
mls aging fast time 16
mls aging long 64
mls aging normal 64
mls flow ip interface-full
no mls flow ipv6
mls nde sender
mls qos
mls cef error action reset
# To check MLS QoS netflow
sh mls netflow ip <source|destination> <IP.you.want.to.see> qos nowrap | i <whatever..>

Sample outputs:
#sh ip cac fl
IP packet size distribution (20259790 total packets):
1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
.000 .364 .343 .094 .114 .001 .001 .000 .000 .000 .050 .000 .000 .000 .000
packet size distribution  —  64 byte and .364 means 36.4% of the traffic
is between packet size 32byte and 64byte and so forth

512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
.000 .019 .000 .006 .001 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes
13 active, 4083 inactive, 11467910 added
277423652 ager polls, 0 flow alloc failures
Active flows timeout in 5 minutes
Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 34056 bytes
13 active, 1011 inactive, 11467886 added, 11467886 added to flow
0 alloc failures, 0 force free
1 chunk, 1 chunk added
last clearing of statistics never
Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
——–         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-Telnet         568      0.0        36    41      0.0       3.3      12.3
TCP-WWW            184      0.0         3   328      0.0       0.6       7.3
TCP-BGP        1423795      0.3         5    56      1.9      35.1      14.5
TCP-other         7708      0.0        27   341      0.0       8.5       7.2
UDP-DNS           3539      0.0         1    74      0.0       0.3      15.5
UDP-NTP         150741      0.0         1    76      0.0       0.0      15.3
UDP-Frag             3      0.0         2   373      0.0       0.0      15.5
UDP-other      8590603      2.0         1   152      2.2       0.1      15.4
ICMP           1156084      0.2         1    96      0.4       4.4      15.4
Total:        11333225      2.6         1   108      4.7       5.0      15.3

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts
Gi0/1.100     10.207.3.17     Local         10.207.3.18     06 00B3 2B19     2

Protocol 06 (hex value) = TCP
Protocol 11 (hex value) = UDP
Protocol 01 (hex value) = ICMP

#sh ip cac ve fl
IP packet size distribution (20259839 total packets):
1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
.000 .364 .343 .094 .114 .001 .001 .000 .000 .000 .050 .000 .000 .000 .000

512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
.000 .019 .000 .006 .001 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes
9 active, 4087 inactive, 11467921 added
277423910 ager polls, 0 flow alloc failures
Active flows timeout in 5 minutes
Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 34056 bytes
9 active, 1015 inactive, 11467897 added, 11467897 added to flow
0 alloc failures, 0 force free
1 chunk, 1 chunk added
last clearing of statistics never
Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
——–         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-Telnet         569      0.0        36    41      0.0       3.3      12.3
TCP-WWW            184      0.0         3   328      0.0       0.6       7.3
TCP-BGP        1423797      0.3         5    56      1.9      35.1      14.5
TCP-other         7710      0.0        27   341      0.0       8.5       7.2
UDP-DNS           3539      0.0         1    74      0.0       0.3      15.5
UDP-NTP         150741      0.0         1    76      0.0       0.0      15.3
UDP-Frag             3      0.0         2   373      0.0       0.0      15.5
UDP-other      8590613      2.0         1   152      2.2       0.1      15.4
ICMP           1156084      0.2         1    96      0.4       4.4      15.4
Total:        11333240      2.6         1   108      4.7       5.0      15.3

SrcIf          SrcIPaddress    DstIf          DstIPaddress    Pr TOS Flgs  Pkts
Port Msk AS                    Port Msk AS    NextHop              B/Pk  Active
Gi0/1.100——xx.xxx.xxx.xxx—Local———–10.212.200.132–11-00—10——–1
incoming and outgoing interface//source and destination IP addresses (next hops may be..)
007B /26 0                     007B /32 0     0.0.0.0                76     0.0

Gi0/0          10.207.3.26     Local          10.207.3.25     06 C0–18——-3
ToS value is the 8 byte of the IP field : C0 -> 11000000 bin – DSCP CS6 – IP Precedence 6
423B /30 0                     00B3 /0  0     0.0.0.0                52    13.3

Gi0/1.100      xx.xxx.xxx.xx   Local          10.212.200.132  06 00  1A       3—-  packet means packets per active seconds
0031 /26 0                     D08D /32 0     0.0.0.0                47     0.1—-  active seconds

Gi0/1.100      xx.xxx.xxx.xx   Local          10.212.200.132  11 00  10       2
9BE7 /26 0                     00A1 /32 0     0.0.0.0                76—–0.0—-  bytes per packets

ToS <> DSCP

ToS    ToS     ToS         ToS     DSCP    DSCP       DSCP    DSCP
dec    hex     bin         prec    bin     hex        dec     name
bin

0      0×00   00000000     000    000000   0×00       0       none/default
32     0×20   00100000     001    001000   0×08       8       cs1
40     0×28   00101000     001    001010   0x0A       10      af11
48     0×30   00110000     001    001100   0x0C       12      af12
56     0×38   00111000     001    001110   0x0E       14      af13
64     0×40   01000000     010    010000   0×10       16      cs2
72     0×48   01001000     010    010010   0×12       18      af21
80     0×50   01010000     010    010100   0×14       20      af22
88     0×58   01011000     010    010110   0×16       22      af23
96     0×60   01100000     011    011000   0×18       24      cs3
104    0×68   01101000     011    011010   0x1A       26      af31
112    0×70   01110000     011    011100   0x1C       28      af32
120    0×78   01111000     011    011110   0x1E       30      af33
128    0×80   10000000     100    100000   0×20       32      cs4
136    0×88   10001000     100    100010   0×22       34      af41
144    0×90   10010000     100    100100   0×24       36      af42
152    0×98   10011000     100    100110   0×26       38      af43
160    0xA0   10100000     101    101000   0×28       40      cs5
184    0xB8   10111000     101    101110   0x2E       46      ef
192    0xC0   11000000     110    110000   0×30       48      cs6
224    0xE0   11100000     111    111000   0×38       56      cs7

Cisco — How to change telnet port for a Cisco router — not the rotary group solution!

R2#tel 172.16.0.100 23
Trying 172.16.0.100 …
% Connection timed out; remote host not responding

R2#tel 172.16.0.100 9999
Trying 172.16.0.100, 9999 … Open

User Access Verification

Password:
R3>

<..omitted..>
Config on the R3 router:
___________________
ip nat inside source static tcp <desired IP address, ex. the loopback..> 23 <same IP as the previous..> <desired port>
interface <interface for the desired IP>
ip nat outside