Overview: Why understanding the PPTP control and encryption handshake matters

Point-to-Point Tunneling Protocol (PPTP) remains in use in many legacy environments and some network products due to its simplicity and broad client support. However, PPTP is often criticized for weak security. For system administrators, developers and enterprise architects, it is important to understand the exact sequence and cryptographic elements involved in a PPTP VPN connection so they can assess risk, troubleshoot interoperability, and plan migration paths.

Architecture basics: PPTP, GRE and PPP

PPTP is not a single monolithic protocol; it is a packaging around two distinct channels:

  • Control channel: TCP port 1723 is used to carry PPTP control messages for session setup and management.
  • Tunnel data channel: Generic Routing Encapsulation (GRE) carries the encapsulated PPP frames (the actual user data and PPP control packets).

Inside the GRE tunnel sits the full PPP stack. That means PPP configuration and authentication protocols (LCP, NCP, CHAP, PAP, EAP, and Microsoft extensions such as MS-CHAPv1/v2 and MPPE) determine both authentication and encryption parameters for the tunneled session. The control plane and data plane are separate, which is why the negotiation is conceptually split between establishing the tunnel and then negotiating PPP options inside it.

High-level handshake flow

At a high level, establishing a PPTP connection proceeds in these phases:

  • PPTP control channel setup (TCP 1723) and PPTP Call-Request / Call-Connected exchange.
  • Exchange of GRE keys/identifiers and creation of the GRE data path.
  • PPP Link Control Protocol (LCP) negotiation inside the GRE tunnel to agree on PPP options.
  • Authentication phase (commonly MS-CHAP v2, or optionally PAP/CHAP/EAP), which also triggers encryption key derivation if MPPE is used.
  • Network Control Protocols (NCPs) like IPCP for IP address/ DNS assignment and optional compression/encryption parameters.

1. PPTP control channel and GRE establishment

The initiating client opens a TCP connection to the server on port 1723 and sends a PPTP Start-Control-Connection-Request. The server responds with Start-Control-Connection-Reply, and the two sides exchange control messages to negotiate versioning and establish a virtual call. The server will then issue a Call-Request and the client will reply with Call-Connected. These messages contain call IDs used to correlate GRE streams.

Once the Call-Connected step completes, the GRE encapsulation is active and PPP frames can flow inside GRE packets. Note that the control channel itself is not encrypted — only the GRE-encapsulated PPP payload can be protected by MPPE.

2. PPP LCP negotiation

PPP Link Control Protocol (LCP) runs inside the GRE tunnel first. LCP options negotiated here influence subsequent behavior, such as:

  • MRU (Maximum Receive Unit)
  • Authentication protocol preference (CHAP, PAP, EAP, MS-CHAP-v2)
  • Whether the peer should be allowed to authenticate itself

If LCP completes successfully, PPP will then proceed to the authentication phase.

3. Authentication: MS-CHAP v2 in practice

In practice, the most common authentication mechanism used with PPTP is MS-CHAP v2. The MS-CHAP v2 exchange contains several steps that both authenticate the peer and produce secret material used for encryption keys:

  • The server sends an 8-byte challenge (Server Challenge).
  • The client generates a 16-byte NT Response using the NT hash of the user’s password and the server challenge, and also returns a 24-byte LM/NT response in older variants.
  • The client also sends a 16-byte Peer Challenge in v2 to help prevent precomputation attacks.
  • The server verifies responses and then sends an authenticator that proves it knows the session key.

The critical point is that MS-CHAP v2 derives the Master Session Key (MSK) or an equivalent secret that MPPE uses to generate session keys for encrypting the PPP payload.

4. MPPE key derivation

When MPPE (Microsoft Point-to-Point Encryption) is negotiated (through PPP Authentication and the Compression/Encryption Control Protocol, or CCP), session keys are derived from the MS-CHAP v2 result. The common flow:

  • From the NT password hash and the MS-CHAPv2 exchange, derive the Master Key (MK) / Master Session Key.
  • From MK, derive the Session Key(s) that MPPE will use for data encryption in both directions.
  • MPPE supports 40-bit, 56-bit and 128-bit modes, but on many implementations the effective entropy is limited by the NT hash and the weak derivation steps.

Under MPPE, the symmetric cipher historically used is RC4. The session key is extended/padded to the required length and applied to the RC4 stream. MPPE uses rekeying and salt sequence numbers to mitigate some stream-cipher weaknesses, but the underlying choice of RC4 and the weak derivation from MS-CHAPv2 limit security.

Detailed step-by-step handshake (message-level)

Below is a more granular view of the messages and operations that occur once the TCP control channel and GRE are established:

  • LCP Configure-Request / Configure-Ack: negotiate PPP options.
  • LCP Echo-Request/Echo-Reply: link keepalive and status checks.
  • Authentication: MS-CHAP v2 challenge/response messages inside PPP Authentication Protocol (PAP/CHAP/CHAPv2 wrapper). The client sends Username and the computed NT-Response, and the server returns a Success/Failure message.
  • Upon success, CCP (Compression Control Protocol) may negotiate MPPE via CCP Configure-Request containing MPPE options. The peers agree on encryption bit-length (40/56/128) and header formats.
  • MPPE Session Key derivation: uses the MS-CHAP-derived keys and optional session salts to create per-direction encryption keys.
  • IPCP (IP Control Protocol): negotiate IP addresses, DNS, and NetBIOS name servers.
  • Once NCPs are configured, the PPP session moves to the Network phase and user IP traffic is encapsulated in GRE and encrypted by MPPE (if enabled).

Security assessment: where the weaknesses are

Understanding the handshake clarifies the root causes of PPTP’s poor security posture:

  • MS-CHAP v2 vulnerabilities: MS-CHAP v2’s challenge/response can be reduced to an NT hash recovery problem. In practice, tools and cloud cracking services can recover passwords from a single captured handshake.
  • RC4 and MPPE limitations: RC4 has known biases and weaknesses. MPPE attempts rekeying, but the effective key material is limited by the NT hash and the design of the key derivation.
  • No forward secrecy: PPTP + MS-CHAPv2 provides no forward secrecy—compromise of the password or long-term secret allows retroactive decryption of captured sessions.
  • Control channel exposure: The PPTP control channel is unencrypted; attackers can observe and manipulate PPTP control messages in certain network scenarios (e.g., middleboxes) unless protected by lower-layer security.

Interoperability and troubleshooting tips

For administrators and developers integrating or maintaining PPTP endpoints, practical points to watch for:

  • Ensure LCP timeouts and retries are consistent on client and server; mismatched timeouts often cause failed negotiations.
  • When authentication fails, capture the PPP packets and inspect MS-CHAPv2 fields (Peer/Server Challenges and NT-Responses) to verify that the client is computing responses correctly and that usernames match expected NT-hashes on the server.
  • MPPE negotiation failures typically appear as PPPs that authenticate successfully but then drop when IPCP is attempted; check CCP message exchanges to ensure MPPE bit-length and state are agreed.
  • RADIUS integration: many enterprise setups use RADIUS for authentication. Ensure RADIUS and NAS devices agree on return attributes that enable MPPE (e.g., MS-MPPE-Keys attributes) and that shared secrets are synchronized.

Mitigations and recommended alternatives

Given the documented weaknesses, the consensus best practice is to migrate away from PPTP for sensitive traffic. Recommended alternatives:

  • IPsec (IKEv2): Provides strong authentication (certificates or EAP methods) and modern cipher suites with support for Perfect Forward Secrecy (PFS).
  • L2TP over IPsec: Commonly used where client support for L2TP exists; encryption handled by IPsec.
  • OpenVPN: TLS-based VPN that supports modern ciphers, strong authentication, and flexible configuration.
  • WireGuard: Lightweight modern VPN protocol with a simple handshake and contemporary cryptography that is easier to audit and typically faster.

Where legacy PPTP must be supported for compatibility reasons, the following mitigations reduce exposure:

  • Limit PPTP access to trusted networks and hosts.
  • Require strong, high-entropy passwords and rotate credentials frequently to reduce brute-force risk.
  • Combine PPTP with an outer transport that provides confidentiality (e.g., SSH tunnels) to protect control messages, understanding this adds complexity.

Conclusion

Examining the PPTP handshake makes clear that the protocol is really a wrapper around PPP and MS-CHAPv2/MPPE — and that most of the security outcome hinges on the authentication and key derivation performed by MS-CHAPv2. While PPTP can be simple and convenient, its reliance on flawed primitives like MS-CHAPv2 and RC4 means it no longer meets modern security expectations.

For site operators and developers responsible for VPN deployments, the best course is to evaluate current usage patterns, plan migration to stronger protocols, and where PPTP remains unavoidable, lock it down with strict access controls and monitoring.

Dedicated-IP-VPN — https://dedicated-ip-vpn.com/