How to capture TCP retransmissions on Linux
You are probably experiencing perfomance issues during file transfers on a Linux box and you would like to find out if there are some network issues recognized. First you may want to find out if there are a lot of re-transmissions during transfers.
Produce Data
Write data to the destination you experience performance issues with, create a small script for that. You may want to use java jcifs to transfer files. See below an example of jcifs_copy.sh
#!/bin/bash
COUNTERSTART=1
COUNTEREND=50
TARGET=//storso11/nas1001
USERNAME=user
PASSWORD=pass
FILE=//opt/sybase/12.5.4/ASE-12_5/bin/dataserver
LOGFILE=/logs/$(basename $0)date > ${LOGFILE}
while [ ${COUNTERSTART} -le ${COUNTEREND} ]; do
echo Run: ${COUNTERSTART}
java -Djcifs.smb.client.useExtendedSecurity=0 -Djcifs.smb.lmCompatibility=0 -cp /local/flamingo/lib/ext/jcifs.jar NasWriteTest smb:${TARGET} ${USERNAME} ${PASSWORD} ${FILE} >> ${LOGFILE}
let COUNTERSTART=${COUNTERSTART}+1
sleep 5
done
In case you have mounted a network share, you may want to use the dd command (press CTRL-C to abort and to see the results) to generate data.
# dd if=/dev/urandom of=testfileR bs=8k count=10000; sync;
# dd if=/dev/urandom of=testfileR bs=8k count=10000; sync;
Capture ReTransmissions
By using tshark
Hint: Make sure you have tshark installed, if you haven’t, install it by executing following command
# yum install wireshark
Loaded plugins: fastestmirror, langpacksLoading mirror speeds from cached hostfile * base: mirror.switch.ch * epel: mirror.23media.de * extras: mirror.switch.ch * updates: mirror.switch.chResolving Dependencies--> Running transaction check---> Package wireshark.x86_64 0:1.10.14-10.el7 will be installed--> Processing Dependency: libsmi.so.2()(64bit) for package: wireshark-1.10.14-10.el7.x86_64--> Processing Dependency: libcares.so.2()(64bit) for package: wireshark-1.10.14-10.el7.x86_64--> Running transaction check---> Package c-ares.x86_64 0:1.10.0-3.el7 will be installed---> Package libsmi.x86_64 0:0.4.8-13.el7 will be installed--> Finished Dependency ResolutionDependencies Resolved================================================================================================= Package Arch Version Repository Size=================================================================================================Installing: wireshark x86_64 1.10.14-10.el7 base 13 MInstalling for dependencies: c-ares x86_64 1.10.0-3.el7 base 78 k libsmi x86_64 0.4.8-13.el7 base 2.3 MTransaction Summary=================================================================================================Install 1 Package (+2 Dependent packages)Total download size: 15 MInstalled size: 83 MIs this ok [y/d/N]: yDownloading packages:(1/3): c-ares-1.10.0-3.el7.x86_64.rpm | 78 kB 00:00:00(2/3): libsmi-0.4.8-13.el7.x86_64.rpm | 2.3 MB 00:00:01(3/3): wireshark-1.10.14-10.el7.x86_64.rpm | 13 MB 00:00:02--------------------------------------------------------------------------------------------------------------------------------------------Total 5.6 MB/s | 15 MB 00:00:02Running transaction checkRunning transaction testTransaction test succeededRunning transaction Installing : c-ares-1.10.0-3.el7.x86_64 1/3 Installing : libsmi-0.4.8-13.el7.x86_64 2/3 Installing : wireshark-1.10.14-10.el7.x86_64 3/3 Verifying : libsmi-0.4.8-13.el7.x86_64 1/3 Verifying : c-ares-1.10.0-3.el7.x86_64 2/3 Verifying : wireshark-1.10.14-10.el7.x86_64 3/3Installed: wireshark.x86_64 0:1.10.14-10.el7Dependency Installed: c-ares.x86_64 0:1.10.0-3.el7 libsmi.x86_64 0:0.4.8-13.el7Complete!
Collect data and show statistics at the end:
# tshark -q -z io,stat,30,"COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission"
Realtime output of source and destination
# tshark -R tcp.analysis.retransmission
Record all data into a binary file:
# tshark -i bond0 -w /temp_data/test.pcap -F libpcap
View detailed output:
# tshark -r /temp_data/test.pcap -q -z io,stat,1,\
> "COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission",\
> "COUNT(tcp.analysis.duplicate_ack)tcp.analysis.duplicate_ack",\
> "COUNT(tcp.analysis.lost_segment) tcp.analysis.lost_segment",\
> "COUNT(tcp.analysis.fast_retransmission) tcp.analysis.fast_retransmission"
Sample output:
===================================================================
IO Statistics
Interval: 1.000 secs
Column #0: COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission
Column #1: COUNT(tcp.analysis.duplicate_ack)tcp.analysis.duplicate_ack
Column #2: COUNT(tcp.analysis.lost_segment) tcp.analysis.lost_segment
Column #3: COUNT(tcp.analysis.fast_retransmission) tcp.analysis.fast_retransmission
| Column #0 | Column #1 | Column #2 | Column #3
Time | COUNT | COUNT | COUNT | COUNT
000.000-001.000 0 0 2 0
001.000-002.000 0 0 0 0
002.000-003.000 0 0 0 0
003.000-004.000 0 0 0 0
004.000-005.000 0 0 0 0
005.000-006.000 0 0 0 0
006.000-007.000 0 0 0 0
007.000-008.000 0 0 0 0
008.000-009.000 0 0 0 0
009.000-010.000 0 0 0 0
010.000-011.000 0 0 0 0
011.000-012.000 0 0 0 0
012.000-013.000 0 0 0 0
013.000-014.000 0 0 0 0
014.000-015.000 5 56 6 5
015.000-016.000 7 46 4 4
016.000-017.000 9 30 5 3
017.000-018.000 6 51 6 6
018.000-019.000 4 54 4 4
019.000-020.000 1 24 0 1
020.000-021.000 0 0 6 0
021.000-022.000 0 0 16 0
022.000-023.000 4 21 8 4
023.000-024.000 14 0 1 0
024.000-025.000 25 17 3 1
025.000-026.000 0 0 4 0
026.000-027.000 21 19 11 2
027.000-028.000 34 27 5 3
028.000-029.000 3 34 3 3
029.000-030.000 3 36 9 3
030.000-031.000 0 0 7 0
031.000-032.000 23 5 2 1
032.000-033.000 0 0 0 0
033.000-034.000 3 30 3 3
034.000-035.000 1 3 2 1
035.000-036.000 14 47 4 4
036.000-037.000 2 29 4 2
037.000-038.000 1 20 2 1
038.000-039.000 0 0 0 0
039.000-040.000 0 0 0 0
===================================================================
By using netstat
Get statistics about retransmissions
# netstat -s | grep segments
Get statistics about packet loss
# netstat -s | egrep -i 'loss|retran'
Press CTRL-C to abort and to see the results
Results
By using tshark
By using netstat
