Understanding how IKEv2 establishes secure VPN connections is essential for system administrators, network engineers, and developers building or managing secure remote-access or site-to-site solutions. This article dives into the technical details of the Internet Key Exchange Version 2 (IKEv2) protocol: how it negotiates security associations, performs key exchange, derives encryption and authentication keys, and how those keys are applied to IPsec. The goal is to demystify the cryptographic flow and operational components so you can make informed design and troubleshooting decisions.

IKEv2 at a glance: roles and core concepts

IKEv2 is the control-plane protocol used to create and manage IPsec Security Associations (SAs). It runs over UDP/500 (and UDP/4500 for NAT traversal), and its primary tasks are:

  • Negotiate an IKE Security Association (IKE_SA) to protect further control messages.
  • Authenticate peers (certificates, pre-shared keys, or EAP mechanisms).
  • Perform Diffie-Hellman key exchange and derive cryptographic material.
  • Create CHILD_SAs which carry the actual IPsec Encapsulating Security Payload (ESP) or AH traffic.

IKEv2 uses two main message exchanges to establish an IKE_SA and one or more CHILD_SAs. Compared to IKEv1, IKEv2 is leaner, more modular, and designed to be resilient to packet loss and NAT devices.

Message flow and payloads: IKE_SA_INIT and IKE_AUTH

IKEv2 setup typically proceeds in two primary round trips:

1) IKE_SA_INIT

This is the unauthenticated Diffie-Hellman (DH) exchange that negotiates cryptographic algorithms and provides shared secret material. The responder and initiator exchange the following payloads:

  • SA — lists proposals (encryption algorithm, PRF, integrity algorithm, DH group, and lifetime)
  • KE — public Diffie-Hellman values (based on a chosen group such as MODP/2048, 3072, or Curve25519)
  • Ni/Nr — nonces from Initiator/Responder
  • NAT detection payloads — optional, for NAT traversal checks

From the DH exchange the parties compute the shared secret g^ir (or its elliptic counterpart) and use the nonces plus the agreed PRF to compute an initial seed value, typically called SKEYSEED.

2) IKE_AUTH

IKE_AUTH carries identity and authentication payloads and establishes the first CHILD_SA (unless a later CHILD_SA is negotiated). Key payloads include:

  • IDi/IDr — identity of the initiator and responder (IP address, FQDN, user@realm, etc.)
  • AUTH — authentication data computed using keys derived from SKEYSEED (could be signature, MAC, or EAP)
  • SA (CHILD) — proposals for the CHILD_SA (ESP transforms)
  • TS — traffic selectors indicating which subnets/addresses are protected

Once IKE_AUTH succeeds, the peers hold an authenticated IKE_SA and at least one CHILD_SA for IPsec traffic.

Key derivation: how IKEv2 generates the keys

Key derivation in IKEv2 follows a structured use of PRFs and hashing over exchanged material. The primary steps are:

  • SKEYSEED = prf(Ni | Nr, g^ir) — the initial seed generated from DH shared secret and nonces.
  • From SKEYSEED, you derive a set of keys used for IKE SA protection (SK_d, SK_ai, SK_ar, SK_ei, SK_er, SK_pi, SK_pr).

Typical key items and their purposes:

  • SK_d — used by IKE to derive keys for CHILD_SAs (a seed for further KDF)
  • SK_ei / SK_er — encryption keys for IKE message encryption, initiator and responder
  • SK_ai / SK_ar — integrity (MAC) keys for IKE messages
  • SK_pi / SK_pr — used for peer authentication data (e.g., for signing or MAC)

Child SAs use a separate derivation step: CHILD_SA keys are derived from SK_d (and optionally a new Diffie-Hellman exchange if Perfect Forward Secrecy is requested). The resulting keys include:

  • SK_d’ (new) — future derivation seed for rekeying
  • SK_e — encryption keys for ESP
  • SK_a — integrity keys for ESP (if not using an AEAD cipher)
  • SK_p — keys for IPcomp or other payloads when used

Detailed KDF description

IKEv2 uses a keyed PRF+ to expand secrets. The general pattern for deriving CHILD_SA keys is:

SK_d | SK_ai | SK_ar | SK_ei | SK_er | SK_pi | SK_pr = prf+(SKEYSEED, Ni | Nr | SPIi | SPIr)

Where prf+ is a repeated PRF application to generate sufficient key material. For CHILD SA keys, the prf+ uses SK_d and concatenates values such as nonces and SPIs to produce SK_e, SK_a, and a new SK_d.

Cryptographic choices: ciphers, PRFs, and integrity

Algorithm selection during the SA negotiation is crucial. Common choices include:

  • DH Groups: 2048/3072/4096 MODP, ECP groups like P-256/P-384, and Curve25519/Curve448 for faster ECDH.
  • Encryption: AES-GCM (AEAD) is preferred because it provides confidentiality and integrity in one primitive. AES-CBC combined with HMAC-SHA(2) is supported but more complex and less recommended.
  • Integrity: HMAC-SHA1 was common historically; modern deployments should use HMAC-SHA256/384/512 or AEAD ciphers which avoid separate integrity keys.
  • PRF: Usually HMAC-SHA256 or SHA-based PRFs aligned with the chosen integrity algorithm.

Best practice: prefer AEAD modes (AES-GCM or ChaCha20-Poly1305) and ECDH groups like Curve25519 for performance and security.

Modes, lifetimes, and rekeying

CHILD_SAs operate in either tunnel or transport mode. Tunnel mode encapsulates an entire IP packet inside an encrypted packet (used for gateway-to-gateway or client-to-gateway VPNs), while transport mode encrypts only the payload (used often with host-to-host scenarios).

Lifetimes are negotiated per SA and typically expressed in seconds. When a CHILD_SA is close to expiring, rekeying occurs:

  • Rekeying the CHILD_SA can reuse the IKE_SA and SK_d material; optionally a new DH exchange yields fresh PFS.
  • Ikev2 uses Rekey messages which replace old SPIs with new ones; this avoids tearing down the tunnel during key rollover.
  • After a rekey, old traffic continues to be accepted for a short grace period to avoid packet loss in flight.

Authentication and identity

IKEv2 supports multiple authentication mechanisms:

  • Certificates: X.509 certs with RSA or ECDSA signatures are common in enterprises. Certificates are validated against trust anchors and CRLs/OCSP optionally.
  • Pre-Shared Keys (PSK): Simple to configure but less scalable and prone to credential sharing risks.
  • EAP methods: EAP-MSCHAPv2, EAP-TLS, EAP-TTLS, etc., allow integration with RADIUS and multi-factor authentication systems for user-level auth.

Identity payloads (IDi/IDr) can carry different identity types. Mapping IDs to certificates or policies is essential for correct tunnel authorization.

NAT traversal, MOBIKE, and resilience

IKEv2 includes mechanisms to cope with NATs and mobility:

  • NAT-T (UDP encapsulation): When either peer detects NAT, IPsec ESP packets are encapsulated within UDP/4500 so NAT devices can translate ports.
  • MOBIKE: a protocol extension that allows an IKE_SA to survive changes of IP addresses (useful for mobile clients switching networks).
  • Dead Peer Detection (DPD): keeps state synchronized and allows fast recovery when the peer becomes unreachable.

Common operational considerations and pitfalls

Real-world deployments require attention to interoperability and robustness:

  • Algorithm mismatches: Ensure both sides advertise compatible SA proposals and preferred algorithms in the right preference order.
  • MTU and fragmentation: Encapsulation increases packet size; path MTU discovery and appropriate MTU settings (and TCP MSS clamping) prevent fragmentation issues.
  • Certificates and PKI: Certificate chains and trusted CA configuration often cause failures; ensure the full chain is presented if necessary and CRLs/OCSP accessibility is considered.
  • NAT hairpinning and client networks: Some client-side NATs or carrier-grade NATs require additional tuning for NAT-T and keepalive timers.
  • Logging and debugging: Use debug-level logs for IKE (such as strongSwan or libreswan logs) and packet captures to inspect IKE payloads, SPIs, and proposal rejections.

Putting it together: example handshake (conceptual)

Example minimal flow for an authenticated IKEv2 tunnel using ECDH and certificates:

  • Initiator -> Responder: IKE_SA_INIT (SA proposal with AES-GCM+PRF, KEi (X25519), Ni)
  • Responder -> Initiator: IKE_SA_INIT response (SA selection, KEr, Nr)
  • Both compute SKEYSEED and preliminary SK_* values
  • Initiator -> Responder: IKE_AUTH (IDi, CERTREQ/Cert, AUTHi, SA (CHILD proposals), TSi)
  • Responder -> Initiator: IKE_AUTH response (IDr, Cert, AUTHr, selected CHILD SA, TSr)
  • CHILD_SA keys derived and ESP traffic begins using negotiated AEAD cipher in tunnel mode

Conclusion and recommendations

IKEv2 combines a modular key-exchange architecture with robust operational features for modern VPNs. For secure, high-performance deployments:

  • Prefer AEAD ciphers (AES-GCM, ChaCha20-Poly1305) and strong ECDH groups (Curve25519 or P-256+ where compliant).
  • Use certificates or EAP with centralized authentication for scalability and auditability; avoid global PSKs where possible.
  • Enable NAT-T and MOBIKE on mobile environments, and configure sensible keepalives and DPD timers for resilience.
  • Monitor lifetimes and rekey operations to ensure seamless key rollover and PFS where required.

With a clear understanding of the IKEv2 message flow, key derivation, and cryptographic choices, you can design VPN solutions that deliver both security and operational reliability. For additional resources and service options, visit Dedicated-IP-VPN at https://dedicated-ip-vpn.com/.