When deploying or maintaining IKEv2-based VPNs, fragmented UDP packets are a common and often confusing source of connectivity and performance problems. This article walks through the technical causes of fragmentation in IKEv2, practical diagnostics you can run, and targeted fixes that work for server and client environments. The goal is to equip sysadmins, developers, and network engineers with reproducible steps and configuration guidance so they can quickly resolve real-world issues.

Why UDP fragmentation happens with IKEv2

IKEv2 negotiates IPsec SAs using UDP, typically over port 500 for IKE messages and 4500 when NAT Traversal (NAT-T) is used. Fragmentation occurs when the size of an IP packet exceeds the path MTU (PMTU) between sender and receiver. Multiple factors can increase IKEv2 packet size:

  • Large certificates or long certificate chains (X.509).
  • Multiple or large vendor ID / notification payloads.
  • Complex proposals or large configuration payloads.
  • UDP encapsulation (NAT-T) adds 20–28 bytes overhead, pushing some packets over PMTU.

At the IP layer fragmentation can be handled in two ways:

  • IP-level fragmentation: The original IP packet is split into fragments and reassembled at the receiver.
  • IKEv2 fragmentation (RFC 7383): The IKEv2 layer splits large IKE messages into IKE fragments that fit within a single IP/UDP datagram.

IP fragmentation is fragile because many middleboxes (NATs, firewalls) mishandle fragments: they may drop them, rewrite headers incorrectly, or fail to buffer fragments for reassembly. This results in failed IKE negotiations, stalled tunnels, or one-way traffic.

Common symptoms of fragmented UDP problems

Look for the following when diagnosing IKEv2 fragmentation issues:

  • IKE_SA_INIT or IKE_AUTH messages fail to complete, often hanging after the first packet.
  • Partial connectivity: rekey works but initial connection fails.
  • Packet captures show IP fragments (IP frag offset > 0) or UDP packets with “fragmented” flags.
  • VPN clients behind certain ISPs or specific NAT devices can’t connect while others can.
  • Logs from IPsec software (strongSwan, libreswan, Windows) mentioning “fragmentation” or “message too long”.

Practical diagnostic steps

1) Reproduce and capture traffic

First, reproduce the problem while capturing packets on both ends of the connection. Use tcpdump or tshark.

Example capture command:

tcpdump -n -s 0 -w ike_frag.pcap udp port 500 or udp port 4500

Open the capture in Wireshark and filter on the flow (src/dst). Inspect for:

  • IP fragments (frame shows “Fragmented IP protocol (proto UDP)” or ip.frag_offset > 0).
  • Presence (or absence) of the IKEv2 Fragmentation Payload (type 46 per RFC 7383).
  • UDP length fields that exceed PMTU minus headers.

2) Verify PMTU and MTU endpoints

Check the MTU on interfaces and whether a PPPoE or VPN encapsulation reduces effective MTU:

ip link show dev eth0

Use path MTU utilities:

  • Linux: tracepath or tracepath6 to determine PMTU.
  • ICMP-based checks: ping -M do -s <size> to test whether large packets pass without fragmentation (adjust for IPv4/IPv6 differences).

Remember that PMTU is affected by all overheads: IP (20 bytes), UDP (8), ESP (if not UDP encapsulated), UDP encapsulation (additional 4 bytes for NAT-T), and any extra tunneling (e.g., GRE, VLAN tags).

3) Inspect IPsec/IKE logs

Enable verbose logging on your IKE implementation:

  • strongSwan: set charon logging to charon { filelog { /var/log/charon.log; filelog format = json; } } and increase log level.
  • libreswan/Openswan: increase pluto log verbosity.
  • Windows: use Event Viewer and enable IKE logging.
  • VPN client logs often show “message too large” or “received fragmented packet” warnings.

Look for specific messages about fragmentation or payload size, such as “Message too long” or “No response to fragment”.

4) Check firewall/NAT behavior

Many middleboxes drop non-initial fragments because they cannot associate them with a connection yet. Use iptables to inspect or allow fragments:

iptables -t raw -L -v

On Linux, consider checking nf_conntrack and fragment handling. Firewalls should accept fragments for UDP 4500/500 and for ESP if encapsulated. If a hardware firewall is in-line, consult its documentation regarding fragment reassembly and UDP/ESP handling.

Targeted fixes and configuration changes

1) Enable IKEv2 fragmentation (RFC 7383)

This is the most robust fix because fragmentation is handled at the IKE layer, avoiding IP fragmentation issues across middleboxes. Many modern implementations support IKEv2 fragmentation:

  • strongSwan: fragmentation option (v5.x) is enabled by default in many builds; check configs:
    charon.plugins.ikev2.enable_fragments = yes
  • libreswan: check for ikev2fragmentation options and upgrade if necessary.
  • Windows: newer Windows versions (10/Server 2016+) support IKEv2 fragmentation, but behavior may depend on client version.

When both peers support IKEv2 fragmentation, large IKE messages are split into multiple IKEv2 fragment messages and reassembled by the peer—no IP fragmentation required.

2) Reduce MTU/MSS or enable MSS clamping

If upgrading or enabling IKE fragmentation is not possible, clamp TCP MSS on the firewall or configure smaller MTU on VPN interfaces so that packets never exceed PMTU:

  • iptables on Linux (example for PPPoE path):
    iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
  • Set tunnel interface MTU: ip link set dev ipsec0 mtu 1400 (adjust as required)
  • Clients can also be configured to use smaller MTU values.

Clamping MSS prevents TCP flows from sending large segments that force fragmentation later in the path. This primarily helps post-establishment traffic but will not fix IKE negotiation packets that are inherently UDP-based and possibly too large.

3) Adjust IPsec/UDP encapsulation behavior

If NAT-T is used, UDP encapsulation on port 4500 will be present. Some implementations allow forcing or disabling NAT-T:

  • Force UDP encapsulation even when no NAT detected to keep behavior consistent: some endpoints prefer consistent 4500 traffic.
  • Alternatively, disable UDP encapsulation if both peers are public addresses and PMTU allows (rarely required).

Keep in mind that NAT-T adds a few bytes of overhead which can trigger fragmentation on marginal MTUs. You may need to adjust MTU accordingly.

4) Allow IP fragments through firewalls and NATs

If you must rely on IP fragmentation (not recommended), ensure intermediate firewalls allow fragments and that connection tracking is configured appropriately:

  • On Linux, iptables’ raw table and connection tracking can be tuned to permit fragments.
    Example to accept fragments: iptables -A INPUT -f -j ACCEPT (but scope this carefully)
  • Enterprise firewalls often have a setting to reassemble fragments before inspection—enable this if safe and supported.

Note: Accepting fragments can expose you to certain fragment-based fingerprinting or evasion attacks; apply strict source/destination scoping when opening fragment rules.

5) Reduce IKE message size: use shorter certificates or ECDSA

A low-level mitigation is to reduce the size of IKE payloads that cause fragmentation:

  • Use ECDSA certificates instead of RSA to shrink certificate size.
  • Remove unused certificate chain elements, or switch to OCSP stapling to reduce the need to send the full chain (where supported).
  • Limit vendor ID or large notification payloads where configurable.

Smaller IKE messages are less likely to exceed PMTU and trigger fragmentation.

Vendor-specific notes and gotchas

strongSwan

strongSwan supports IKEv2 fragmentation (RFC 7383). Ensure your version includes the fragmentation plugin and that it’s enabled. Also check

/etc/strongswan.conf and /etc/ipsec.conf for fragmentation-related options. Per-connection options like leftmtu and rightmtu can also help.

Windows

Windows IKEv2 clients usually handle fragmentation well in modern builds, but older clients or certain enterprise versions may fail. Ensure clients are up to date and test with Windows event logs for IKE errors.

Hardware firewalls and middleboxes

Many hardware devices have limited fragment reassembly buffers or strict security policies that drop fragments. Where possible, enable stateful fragment reassembly or bypass deep packet inspection for IPsec traffic.

Example troubleshooting workflow

  • Capture traffic on client and server while reproducing the failure.
  • Look for IP fragments or absence of IKE fragmentation payloads in the capture.
  • Run tracepath or ping tests to determine PMTU.
  • Enable IKEv2 fragmentation on both peers (strongSwan/libreswan/Windows) and retry.
  • If fragmentation support is unavailable, clamp MSS or reduce MTU on the path.
  • If necessary, allow fragments in firewall rules scoped to specific hosts and ports as a last resort.

Security considerations

Fragment handling affects security. Disabling inspection for fragmented packets or relaxing firewall rules may be necessary to restore functionality, but this increases exposure to fragment-based evasion techniques. Always:

  • Limit fragment acceptance to known endpoints and ports.
  • Monitor and log fragment-related traffic.
  • Prefer application-layer fixes (IKE fragmentation, smaller certs, MTU adjustments) over loosening firewall policies.

Summary

Fragmented UDP packets with IKEv2 are usually caused by oversized IKE messages hitting a path MTU smaller than the IP/UDP payload. The best long-term fix is to enable IKEv2 fragmentation (RFC 7383) on both peers so that IKE messages are split at the IKE layer rather than relying on IP fragmentation. When that is not possible, mitigating strategies include reducing MTU/MSS, shortening certificates and payloads, and carefully permitting fragments through security devices. Detailed packet captures and PMTU tests are the essential first steps to diagnose and determine the appropriate remediation.

For implementation examples and managed VPN setups that minimize MTU-related issues, see Dedicated-IP-VPN at https://dedicated-ip-vpn.com/.