Understanding L2TP authentication end-to-end requires looking beneath the abstract flow diagrams and inspecting the actual packets on the wire. In real deployments, L2TP is commonly used together with IPsec to provide confidentiality and integrity, while PPP handles user authentication and IP configuration. This article walks through the complete authentication flow—from IPsec negotiation through L2TP control setup, PPP authentication exchanges (PAP/CHAP/MS-CHAPv2), to IPCP addressing—using packet-capture artifacts and practical tips for analysis with tools such as Wireshark.

High-level flow and where to capture

Before we dive into specific packets, it helps to know the logical stages you will see in a capture:

  • IPsec (IKE) Phase 1 — establish a secure channel for negotiating SAs (UDP/500 or UDP/4500 for NAT-T).
  • IPsec (IKE) Phase 2 — Quick Mode to negotiate ESP SAs (could see ESP or UDP/4500-encapsulated ESP packets).
  • L2TP control channel — establishment of tunnel and session (UDP/1701 control messages).
  • PPP LCP negotiation inside L2TP — link options and MRU, authentication negotiation.
  • PPP authentication — PAP, CHAP or MS-CHAPv2 exchanges inside L2TP data packets.
  • IPCP negotiation — IP address assignment and routing information.
  • Encrypted user traffic — packets carried inside L2TP data or ESP once session is up.

Recommended capture points: the client’s network interface and the server’s interface (if possible) to see both sides. If the VPN is using IPsec with ESP, remember ESP contents are encrypted; you’ll need key material (or the plain-text before IPsec) to view PPP/L2TP payloads in Wireshark.

Useful Wireshark filters

  • IKE (ISAKMP): udp.port == 500
  • NAT-T (ESP-in-UDP): udp.port == 4500
  • ESP: esp
  • L2TP: udp.port == 1701 or l2tp
  • PPP and PPP-related protocols: ppp, chap, pap, mschap

IPsec/IKE details (Phase 1 & 2)

The authentication flow for an L2TP/IPsec VPN almost always starts with IKE. In a typical capture you will see:

  • IKE Main Mode (or Aggressive Mode) exchanges — proposals, key exchange (DH), nonces, and peer authentication (PSK or certificates). These packets use UDP port 500.
  • After Phase 1 completes, Quick Mode packets negotiate ESP SAs (including SPI values and encryption/integrity algorithms). If NAT-T is present you will see UDP/4500 used, and ESP payloads encapsulated in UDP.

How to correlate IKE to ESP: note the SPI values (Security Parameter Index) in Quick Mode responses; the SPI appears in subsequent ESP packets. In Wireshark, click a Quick Mode packet and inspect the ESP SA parameters—then find matching ESP frames with the same SPI to find encrypted payloads belonging to that SA.

Decrypting ESP: ESP is encrypted by design. To see L2TP/PPP inside ESP, you must provide decryption keys to Wireshark or capture traffic before encryption. For PSK-based setups, Wireshark cannot automatically decrypt ESP unless you supply IPsec SA keys (the platform or VPN endpoint may log SAs or you may configure the VPN to export keys). For certificate-based authentication, RSA private keys (if available) can sometimes be used to decrypt IKE, but not ESP unless you have derived SAs.

L2TP control plane — establishing tunnel and session

Once the IPsec layer is in place (or in non-IPsec L2TP deployments where L2TP runs in the clear), the L2TP control channel begins on UDP/1701. Key control messages appear in a clear sequence:

  • Start-Control-Connection-Request (SCCRQ)
  • Start-Control-Connection-Reply (SCCRP)
  • Start-Control-Connection-Connected (SCCCN)
  • Outgoing and incoming Echo-Request/Reply, StopCCN, and ZLB (zero-length body) messages for ACK semantics

L2TP control headers include flags that are crucial when analyzing captures: the Control (C) bit, Length (L) bit, Sequence (S) bit, and Offset (O) bit. A common filter shows these fields in Wireshark; you can expand the L2TP header to see the assigned Tunnel ID and Session ID. Tunnel ID identifies the control connection, while Session ID is assigned during call/session establishment and will be used for the PPP traffic that follows.

Common L2TP control attributes to inspect

  • Assigned Tunnel ID and Peer Tunnel ID
  • Assigned Session ID
  • Framing capabilities and vendor-specific AVPs
  • Call serial number and bearer type (for PPP sessions)

Tracking the SCCRQ → SCCRP → SCCCN three-way handshake confirms that both endpoints agree on session parameters. If the capture shows a repeated SCCRQ with no SCCRP, the control packets may be blocked by a firewall or the IPsec policy is mismatched.

PPP LCP negotiation and authentication options

After the L2TP control phase, PPP LCP packets traverse inside L2TP Data messages. LCP (protocol 0xC021) negotiates link features such as MRU, authentication protocol preference, and whether compression will be used. Look for LCP Configure-Request and Configure-Ack packets.

One important LCP option is the Authentication Protocol. The candidate options you’ll encounter include:

  • PAP (Password Authentication Protocol, PPP protocol 0xC023) — clear-text password exchange with a Request/ACK model
  • CHAP (Challenge-Handshake Authentication Protocol, PPP protocol 0xC223) — challenge/response using MD5 (more secure than PAP)
  • MS-CHAPv2 — Microsoft’s variant used in many Windows VPN setups; provides mutual authentication and uses NT-style password hashing and the NT-Response

Wireshark decodes these and shows the PPP protocol field; proper decoding requires having access to the decrypted L2TP payload (if protected by IPsec).

Deep dive: CHAP and MS-CHAPv2 packet flow

CHAP sequence (CHAP v1/v2 common structure):

  • CHAP Challenge (Server → Client): contains an identifier (id) and a random challenge value.
  • CHAP Response (Client → Server): id, and a response = MD5(id | password | challenge).
  • Authentication Success/Failure (Server → Client).

In a packet capture you can inspect the CHAP Challenge and Response fields. The response is an MD5 digest; knowing the challenge, the id, and the password would allow recomputing the digest and validating credentials (this is why CHAP still relies on a secret not transmitted on the wire).

MS-CHAPv2 is more complex. The exchange includes:

  • Server challenge (16 bytes) and name
  • Client response containing Peer Challenge, NT-Response, and Username
  • Server sends Authenticate-Response and Final Success/Failure

The NT-Response is derived from the server and peer challenges plus the user password transformed into the NT hash. Tools such as chapcrack or specialized forensic scripts can attempt to brute-force NT-Response when a weak password is used. In captures you will not see the raw password, but the challenge/response artifacts are visible inside the PPP frames.

PAP: what a capture looks like

PAP is the simplest PPP authentication: the client sends a Authenticate-Request packet containing the user name and password in clear text, and the server replies with Authenticate-Ack or Authenticate-Nak. If your capture shows PAP, the credentials are visible unless they were protected under IPsec. PAP is easy to debug but should be avoided in unencrypted environments.

IPCP and IP address assignment

After successful authentication, PPP transitions to IPCP (Internet Protocol Control Protocol — protocol 0x8021) negotiation. IPCP Configure-Request/Configure-Ack frames negotiate IP addressing, DNS, and other options. In captures you’ll see the client and server exchange IPCP packets until both sides agree on an IP address—this is the moment the VPN data path becomes usable for IP traffic.

Putting it all together: sample packet sequence

  • UDP 500: IKE_SA_INIT (DH, nonces)
  • UDP 500: IKE_AUTH (identity, authentication — PSK or certificate)
  • UDP 500: Quick Mode negotiations (IPsec SA proposals)
  • UDP 4500 or ESP: resulting ESP packets (encrypted payloads)
  • UDP 1701 over the encrypted channel: SCCRQ → SCCRP → SCCCN
  • L2TP Data: PPP LCP Configure-Request/ACK
  • PPP (CHAP or PAP): Challenge → Response → Success
  • PPP IPCP: Configure-Request/ACK for IP address
  • UDP/ESP: encrypted user traffic flows

Practical analysis tips

  • Correlate timestamps: offset jitter in captures can make it hard to see exact ordering—use timestamps to follow the flow between IKE and L2TP events.
  • Track Tunnel/Session IDs: L2TP Tunnel ID and Session ID distinguish concurrent sessions from the same peers.
  • Filter by PPP protocol codes: Use ppp.protocols to locate LCP (0xC021), PAP (0xC023), CHAP (0xC223), IPCP (0x8021).
  • ESP correlation: match SPI values from Quick Mode with subsequent ESP packets to isolate encrypted sessions.
  • Decrypt when possible: capture VPN server logs that export SAs or configure debug output to capture keys; importing these into Wireshark is often the only way to see PPP inside ESP.
  • Watch for NAT-T: presence of UDP/4500 indicates NAT in the path—this alters packet headers and can affect firewall traversal.

Security and operational takeaways

L2TP by itself does not provide encryption — PPP authentication and IP addressing happen over the L2TP tunnel. In production, always pair L2TP with IPsec. CHAP and MS-CHAPv2 are better than PAP but each has known weaknesses (especially MS-CHAPv2 against offline cracking). Design choices to reduce risk:

  • Prefer certificate-based IKE authentication or strong PSKs.
  • Avoid PAP unless used over an already encrypted channel.
  • Harden endpoints and use strong passwords to prevent CHAP/ MS-CHAPv2 offline attacks.
  • Monitor L2TP control messages for repeated failures — they can indicate brute-force attempts or misconfiguration.

Analyzing packet captures across the whole stack—from IKE negotiation to IPCP acknowledgment—gives a full picture of where authentication succeeds or fails. Capturing both sides (client and server) and correlating SPIs, Tunnel/Session IDs, and timestamps makes it straightforward to identify the root cause of authentication or connectivity problems.

For more technical guides and practical troubleshooting advice on VPN deployments, visit Dedicated-IP-VPN at https://dedicated-ip-vpn.com/.