Introduction

When latency, jitter, or unexplained throughput drops affect your applications, the first step to resolution is often packet-level visibility. For administrators, developers, and site operators, tcpdump remains a lightweight, powerful tool to capture and analyze live traffic and reveal the root causes of network performance bottlenecks. This article provides a practical, detailed guide to using tcpdump to identify common issues — from retransmissions and windowing problems to MTU/fragmentation and asymmetric routing — with concrete commands, interpretation tips, and best practices.

Why tcpdump?

tcpdump is widely available on Unix-like systems, consumes minimal resources when used correctly, and outputs precise packet details. It can capture packets for real-time troubleshooting or generate pcap files for offline analysis with tools like Wireshark or tshark. Unlike higher-level logs, tcpdump shows the exact sequence of packets, flags, timestamps, sizes, and IP/TCP headers, which is crucial to diagnosing subtle performance problems.

Preparing for a Capture

Before you start capturing, consider the environment and impact. Full packet captures can be large and CPU-intensive on busy hosts. Follow these preparatory steps:

  • Identify the correct interface (use ip link or ifconfig).
  • Prefer capturing on a mirrored port or a dedicated monitoring interface where possible, to avoid interfering with production interfaces.
  • Decide between live inspection and writing to a file for offline analysis.
  • Ensure sufficient disk space and use ring buffers for long captures.
  • Capture with maximum snapshot length (-s 0) to avoid truncating relevant packet payload or headers.

Basic Command Options and Practical Examples

Here are several essential tcpdump options you will use frequently, with explanations and example usage.

-i (interface)

Specifies the network interface to capture from. Example: tcpdump -i eth0.

-s (snapshot length)

Controls how many bytes of each packet are captured. Use -s 0 to capture entire packets — important for TCP diagnostics where payload and options matter.

-w and -r (write/read pcap)

Write to a file: tcpdump -i eth0 -s 0 -w capture.pcap. Read later: tcpdump -r capture.pcap or open with Wireshark.

-n, -nn (disable name resolution)

Disable DNS and port name lookups to reduce capture overhead and speed analysis: -n (no DNS), -nn (no DNS and no port to service name).

-tttt and -tt

Timestamps: -tttt prints human-readable timestamps with date, which is useful when correlating events across systems.

-B (buffer size)

Increase the capture buffer with -B (kilobytes). On high-throughput systems, a larger buffer reduces packet loss for tcpdump itself. Example: -B 4096.

-C and -W (ring buffers)

Rotate capture files to avoid giant pcaps. Example: -C 100 -W 10 -w capture%%.pcap creates files of 100MB and cycles through 10 files.

-S, -v, -vv, -e

-S prints absolute TCP sequence numbers (useful for calculating bytes in flight). -v/-vv increases verbosity for header details. -e displays link-layer headers (MAC addresses), essential when diagnosing switch-level issues or VLAN tags.

Filters: Capture Only What Matters

Use BPF filters to reduce noise. Examples:

  • Single host: host 203.0.113.5
  • TCP on port 443: tcp port 443
  • Between two hosts: host 203.0.113.5 and host 198.51.100.2
  • Exclude management traffic: not port 22 and tcp

Combining filters reduces file size and improves focus when looking for specific patterns like retransmissions or handshake failures.

Identifying Common Bottlenecks

This section shows what to look for in tcpdump output and how to interpret the traces to pinpoint problems.

1. Packet Loss and Retransmissions

Retransmissions are a primary sign of packet loss. In tcpdump output, look for repeated segments labeled as retransmissions or duplicate ACKs. Use -S -vv and inspect TCP flags and sequence numbers. A pattern of repeated ACKs from the receiver (three DUP ACKs) followed by retransmissions from the sender indicates network loss on the path.

Key indicators:

  • Sender retransmitting the same sequence number soon after the initial transmission.
  • Repeated ACKs with identical acknowledgment numbers (duplicate ACKs).
  • High frequency of retransmits relative to total packets suggests an overloaded link or faulty hardware.

2. Round-Trip Time (RTT) and Latency

Timestamps allow you to compute RTT by correlating a packet and its corresponding ACK. Capture with -tttt and note the timestamps of the SYN, SYN/ACK, and subsequent ACK. For established connections, calculate RTT as the difference between the data packet timestamp and the ACK that acknowledges that data (matching sequence numbers).

Large or variable RTTs may indicate queuing delays, overloaded links, or poor routing. If RTT increases during high throughput, investigate bufferbloat on the sender or intermediate routers.

3. TCP Window and Flow Control

Check the advertised receive window in the TCP header. If the receiver’s window is consistently small, throughput will be limited regardless of link capacity. Use -vv to view the window size and TCP options like window scaling. Watch for zero-window conditions (receiver advertises zero window) which cause sender to stall.

4. SYN/Handshake Issues

Failed or delayed connection establishments can be identified by inspecting SYN/SYN/ACK/ACK sequence. Repeated SYNs without SYN/ACK responses suggest reachability or firewall issues, while SYNs followed by ICMP unreachable messages indicate routing or ACL errors.

5. IP Fragmentation and MTU Problems

Fragmentation or ICMP “Fragmentation Needed” messages point to MTU mismatches. Look for IP fragments (identified by fragment offset and MF flag) and corresponding ICMP messages. Large numbers of fragments reduce performance; fix by adjusting MTU, enabling Path MTU Discovery, or tuning TCP MSS (Maximum Segment Size) via iptables or server configuration.

6. Asymmetric Routing and Missing Packets

Asymmetric paths can make a capture on a single host misleading—packets may not be seen in both directions. If ACKs seem missing or conversations look incomplete, capture on both endpoints or on a mirrored switch port. Use MAC addresses (-e) to verify if packets traversed expected network segments.

Advanced Techniques and Examples

Here are practical patterns to capture targeted symptoms:

Capture retransmits and duplicate ACKs

Run a filter that focuses on TCP retransmissions and duplicate ACKs. While tcpdump itself does not categorize retransmissions explicitly, you can capture TCP segments and post-process with tshark or Wireshark to detect retransmits. Alternatively, capture full TCP conversations with -s 0 -w and analyze sequences in Wireshark using built-in “TCP Analysis” flags.

Correlate application timestamps with packet traces

Add precise timestamps to packet captures and synchronize with application logs. Capture with -tttt and convert application logs to the same timezone/format. Correlating the two shows whether slow responses are due to network delays or server processing.

Use buffer tuning to avoid dropped packets by tcpdump

In high-throughput scenarios, tcpdump may drop packets if the kernel capture buffer is insufficient. Increase it with -B, and consider running the capture in a ring buffer to avoid losing historical context: tcpdump -i eth0 -s 0 -B 4096 -C 200 -W 5 -w /var/log/net/capture%%.pcap.

Post-Capture Analysis

Often the most effective analysis happens offline. Use Wireshark or tshark to:

  • Apply TCP conversation filters and follow TCP streams.
  • Enable expert info to surface retransmissions, zero windows, and spurious retransmits.
  • Plot TCP throughput over time and examine sequence number graphs to assess bytes-in-flight and congestion window behavior.

tshark can automate detection of problems: run it to extract statistics or to flag anomalies in large captures.

Practical Troubleshooting Workflow

When a performance issue is reported, follow a systematic approach:

  • Reproduce the issue while capturing relevant interfaces with targeted filters.
  • Capture complete packets (-s 0) and use ring buffers for long sessions.
  • Record application logs and system metrics (CPU, interrupts, NIC errors) concurrently.
  • Look for packet loss (retransmits), delayed ACKs (large RTT), small receiver windows (flow control), MTU/fragmentation events, and ICMP errors.
  • Validate that network devices (switches, routers, NICs) are not dropping or offloading packets unexpectedly (check NIC offload settings and driver behavior).

Best Practices and Caveats

Keep these practical tips in mind:

  • Prefer mirrored ports for non-intrusive captures: capturing on production interfaces can affect performance.
  • Disable offloads when needed: because features like TCP segmentation offload (TSO) and large receive offload (LRO) can change what you see in captures; disable them for accurate packet-level inspection when necessary.
  • Time synchronization: ensure hosts and capture devices use NTP for accurate timestamp correlation.
  • Legal/privacy considerations: captures may contain sensitive data — handle pcaps securely and redact or filter payloads when sharing.

Conclusion

tcpdump is an indispensable tool for uncovering network performance bottlenecks. With careful capture planning, targeted filters, and detailed analysis — focusing on retransmissions, RTT, TCP window behavior, MTU issues, and asymmetric routing — you can diagnose and resolve many classes of performance problems. Combine tcpdump captures with system metrics and application logs for a full picture of the stack.

For more guides and resources on secure connectivity and troubleshooting best practices, visit Dedicated-IP-VPN at https://dedicated-ip-vpn.com/.