Moving away from pre-shared keys (PSKs) for L2TP/IPsec VPNs is a necessary step for organizations that require stronger authentication, better key management, and improved auditability. PSKs are convenient but scale poorly and present a single point of compromise: if one client leaks the PSK, every connection is potentially compromised. Replacing PSKs with certificate-based authentication (public key infrastructure, PKI) yields stronger security, non-repudiation, and simpler rotation. This article provides a practical, technical guide to implementing L2TP/IPsec with certificate-based authentication, covering PKI basics, certificate generation, server and client configuration (Linux strongSwan + xl2tpd example and notes for Windows/macOS), networking considerations, and operational best practices.

Why use certificates instead of PSKs?

Before diving into configuration, it is useful to enumerate the concrete security advantages that certificates bring:

  • Per-entity credentials — each server and client has its own keypair and certificate, so a leaked client credential affects only that client.
  • Stronger cryptography — certificates allow use of modern key types (RSA-2048/3072/4096, ECDSA P-256/P-384) and algorithms supported by IKE implementations.
  • Expiration and revocation — certificates can expire and be revoked with CRLs or OCSP, enabling controlled credential lifecycle management.
  • Non-repudiation and auditability — certificate metadata and signing provide evidence about who or what was authorized.

Overview of the architecture

L2TP/IPsec combines two protocols:

  • IPsec (IKE) — provides authentication, key exchange and encryption for the link (ESP). We will use IKEv1 or IKEv2 depending on the stack; certificate authentication replaces PSK here.
  • L2TP over PPP — carries PPP frames which can include user-level authentication (PAP/CHAP or MS-CHAPv2). When using certificate-based IKE, PPP can still use username/password if desired, but for the strongest setup use client certificates for IKE and enforce PPP noauth or restrict accounts to avoid redundant weak auth.

Generate a simple PKI with OpenSSL

The following steps create a simple CA, a server certificate, and a client certificate using OpenSSL. In production, use an established CA, hardware security modules (HSM), or an enterprise PKI.

Commands (execute on a secure build host):

  • Create a CA private key: openssl genpkey -algorithm RSA -out ca.key -pkeyopt rsa_keygen_bits:4096
  • Create a CA self-signed certificate: openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -subj "/CN=My VPN CA" -out ca.crt
  • Create server key and CSR: openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:4096 and openssl req -new -key server.key -subj "/CN=vpn.example.com" -out server.csr
  • Sign the server certificate: openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 1825 -sha256
  • Create client key, CSR and sign similarly: openssl genpkey -algorithm RSA -out client.key -pkeyopt rsa_keygen_bits:2048, openssl req -new -key client.key -subj "/CN=client1@example.com" -out client.csr, openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 825 -sha256
  • Optionally create a PKCS#12 bundle for Windows/macOS: openssl pkcs12 -export -inkey client.key -in client.crt -certfile ca.crt -name "VPN Client 1" -out client.p12

Linux server: strongSwan + xl2tpd configuration

On most Linux distributions, strongSwan is a robust IPsec/IKE implementation. For L2TP payload handling, pair it with xl2tpd and PPP. The high-level steps are:

  • Install packages: apt install strongswan xl2tpd ppp (Debian/Ubuntu)
  • Enable IP forwarding: sysctl -w net.ipv4.ip_forward=1 and persist in /etc/sysctl.conf
  • Open firewall ports: UDP 500 and UDP 4500 for IKE and NAT-T, and allow ESP protocol (IP protocol 50) if not using NAT-T.

strongSwan config snippets

Example /etc/ipsec.conf (IKEv2-like style with certificate auth):

config setup
charondebug="ike 2, knl 2, cfg 2"

conn l2tp-vpn
left=%any
leftid=@vpn.example.com
leftcert=server.crt
leftsendcert=always
leftsubnet=0.0.0.0/0
leftfirewall=yes
right=%any
rightauth=pubkey
rightsourceip=10.10.10.0/24
keyexchange=ikev2
ike=aes256-sha2_256-modp2048!
esp=aes256-sha2_256!
rekey=no
auto=add

Notes:

  • leftcert refers to the server certificate file placed under /etc/ipsec.d/certs. The private key goes into /etc/ipsec.d/private.
  • rightauth=pubkey enforces certificate-based client auth. For client certificates, strongSwan will verify signing against the CA in /etc/ipsec.d/cacerts.

/etc/ipsec.secrets

Put the RSA private key entry here:

: RSA "server.key"

Do not place PSKs. The file must be readable only by root (chmod 600).

xl2tpd and PPP

xl2tpd provides L2TP layer. Configure /etc/xl2tpd/xl2tpd.conf with a connection section and point ppp options to a file (e.g., /etc/ppp/options.xl2tpd). For PPP, you can avoid username/password authentication by using noauth (if you trust the IKE client certs) or continue to use MS-CHAPv2. A minimal /etc/ppp/options.xl2tpd might include:

  • require-mschap-v2 (if you use username/password)
  • ms-dns <DNS_IP>
  • ipcp-accept-local and ipcp-accept-remote
  • mtu 1400 and mru 1400 (adjust for fragmentation/NAT)

To minimize weak auth vectors, use PPP noauth only if you are comfortable that IKE client certificate authentication is enforced and that authorization mapping exists (e.g., one certificate per user).

Client configuration (Windows, macOS, Linux)

Most OS VPN clients support certificate-based IPsec. On Windows, use the built-in “Windows VPN (L2TP/IPsec)” client but configure the connection to use a certificate for machine or user authentication by importing the .p12 into the Personal certificate store and selecting certificate authentication in the VPN properties. On macOS, import the client .p12 into Keychain and configure L2TP with “Machine” or “User” certificate depending on the profile.

On strongSwan-based Linux clients, place client.crt in /etc/ipsec.d/certs, client.key in /etc/ipsec.d/private, and CA in /etc/ipsec.d/cacerts. Example client /etc/ipsec.conf:

conn myvpn
left=%defaultroute
leftcert=client.crt
leftauth=pubkey
right=vpn.example.com
rightid=@vpn.example.com
rightsubnet=0.0.0.0/0
auto=add

Firewall, NAT, and routing caveats

  • Enable NAT-T (UDP 4500) so clients behind NAT can connect; strongSwan and modern stacks enable this automatically but ensure firewall allows UDP 500 and 4500.
  • If NAT is in front of the server, ensure proper ALG rules or that ESP passthrough is allowed. Prefer NAT-T to avoid ESP issues.
  • IP forwarding is required on the server; set up NAT masquerading if clients should access the internet via the VPN (iptables or nftables rules).
  • Set appropriate MTU/MRU values in PPP options to avoid fragmentation over IPsec tunneling (common values: 1400 or 1300).

Operational and security best practices

  • Use strong keys and algorithms: RSA-2048 or ECDSA P-256 minimum; prefer AES-256-GCM or AES-256-SHA2 for ESP and SHA-256+ for IKE.
  • Protect private keys: server.key and client private keys must be root-only and, where feasible, protected by hardware (HSM/TPM).
  • Enable certificate revocation: publish CRLs on a web server and configure strongSwan with cacert and CRL checking, or use OCSP stapling if supported.
  • Rotate certificates: issue short validity or a policy-driven rotation cadence to limit exposure on compromise.
  • Audit and monitor: enable logging (charondebug), watch for failed authentication and unusual IP pool usage, and integrate with SIEM solutions.
  • Test failures and recovery: simulate certificate revocation, expired certs, and key compromise to validate operational playbooks.

Troubleshooting checklist

  • Verify strongSwan status: ipsec statusall and check journalctl -u strongswan for certificate verification errors.
  • Confirm certificates are installed in the correct IPsec directories and that filenames in config match actual files.
  • Check firewall rules and that UDP 500/4500 are reachable from client networks.
  • Use packet capture (tcpdump) to inspect IKE exchanges; look for AUTH failures or mismatched IDs.
  • For Windows clients, inspect Event Viewer for IKE/RA/PHASE1 errors and ensure the client certificate is present in Personal store and trusted by the CA certificate under Trusted Root Certification Authorities.

Certificate-based authentication for L2TP/IPsec adds management overhead upfront but delivers significant security and operational benefits: per-client credentials, revocation capability, stronger cryptography, and improved audit trails. Implementing this with strongSwan and xl2tpd on Linux is a common approach that supports modern cryptographic suites and scalable credential management. Remember to integrate your certificate lifecycle into your operational processes and to protect private keys with stringent access controls.

For deployment-ready guidance, appliance recommendations, or managed dedicated-IP VPN offerings, see Dedicated-IP-VPN at https://dedicated-ip-vpn.com/.