Deploying L2TP over IPsec with strong authentication requires a robust certificate infrastructure. While many deployments rely on pre-shared keys (PSKs), using X.509 server certificates provides stronger security, easier client management, and better scalability for enterprise and developer environments. This article walks through how to generate and manage server certificates using OpenSSL, and how to integrate them into an L2TP/IPsec stack (libreswan/strongSwan + xl2tpd or similar) with practical configuration and hardening advice.

Why use server certificates for L2TP/IPsec?

Certificates provide several advantages over PSKs:

  • Stronger authentication: Private keys protected on servers and client certificates reduce the risk of shared-key disclosure.
  • Scalability: You can revoke individual client certificates without rotating PSKs for all users.
  • Compatibility: Certificate-based authentication is standard across IKEv2 and many IKEv1 implementations and integrates with enterprise PKI.
  • Auditing and policy: Certificates can encode metadata (Subject, SANs) and allow finer-grained access control.

Design considerations before you build a CA

Decide a few architecture points:

  • Will you run an internal CA or integrate with an enterprise CA? For most VPN providers and dev environments, a private CA is sufficient.
  • Do you need client certificates as well (mutual TLS / certificate authentication)? If so, build client templates and revocation workflows.
  • Key algorithms: RSA 3072 or 4096 vs ECDSA (prime256v1 or secp384r1). ECDSA offers smaller keys and faster operations but ensure clients support it.
  • Certificate lifetime: shorter lifetimes (1–3 years for server certs) are safer; clients may prefer auto-rotation mechanisms.

OpenSSL CA: Minimal but secure setup

Below are the essential steps to create a small private CA, produce a server certificate with SANs, and export it for use in IPsec. Commands assume a Linux shell and OpenSSL installed. Use a dedicated build directory (e.g., /root/vpn-ca).

1) Create a CA key and self-signed certificate

Prefer encrypted CA private keys and store them offline if possible.

Commands:

mkdir -p /root/vpn-ca && cd /root/vpn-ca

openssl genpkey -algorithm RSA -out ca.key -pkeyopt rsa_keygen_bits:4096 (for an encrypted key, add -aes256)

openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.crt -subj "/C=US/ST=State/L=City/O=Org/OU=VPN/CN=VPN-CA"

2) Generate server key and CSR (using SANs)

IPsec VPN servers are typically reached by hostname or IP. Modern TLS/SSL checks use subjectAltName (SAN) rather than CN, so include SANs.

Create an OpenSSL config snippet (server_openssl.cnf):

[ req ]
default_bits = 3072
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no
[ req_distinguished_name ]
C = US
ST = State
L = City
O = ExampleCorp
OU = VPN
CN = vpn.example.com
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = vpn.example.com
IP.1 = 203.0.113.10

Then run:

openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:3072

openssl req -new -key server.key -out server.csr -config server_openssl.cnf

3) Sign the CSR with the CA

Create a minimal extensions file for signing (server_ext.cnf):

basicConstraints = CA:FALSE
nsCertType = server
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = vpn.example.com
IP.1 = 203.0.113.10

Sign:

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 825 -sha256 -extfile server_ext.cnf

The 825-day limit is a guideline; choose 1–3 years per your policy.

4) Export to PKCS#12 for software that expects a bundle

Some IPsec implementations (or management GUIs) accept PKCS#12 bundles that contain the key and cert chain. Protect the exported file with a strong password.

openssl pkcs12 -export -out server.p12 -inkey server.key -in server.crt -certfile ca.crt -name "vpn.example.com"

Integrating the certificate into an IPsec/L2TP stack

Common Linux stacks are strongSwan or Libreswan for IPsec and xl2tpd for L2TP. Below are key points for each.

strongSwan (IKEv2 or IKEv1)

Place files in /etc/ipsec.d/:

  • /etc/ipsec.d/cacerts/ca.crt
  • /etc/ipsec.d/certs/server.crt
  • /etc/ipsec.d/private/server.key (chmod 600)

Example strongSwan connection snippet (/etc/ipsec.conf or /etc/ipsec.d/conn.conf):

conn L2TP-PSK
left=%any
leftcert=vpn.example.com
leftsendcert=always
leftid=@vpn.example.com
right=%any
authby=rsasig
auto=add

Note: IKEv2 prefers certificate-based auth; if you must support legacy clients using L2TP/IPsec with PSK + certificate, test client compatibility carefully. Put keys under root-only access and consider using a hardware security module (HSM) or PKCS#11 provider for production.

Libreswan

Libreswan uses /etc/ipsec.d/ similarly. Example in /etc/ipsec.conf:

conn L2TP
left=%defaultroute
leftcert=vpn.example.com
leftsendcert=always
leftid=@vpn.example.com
right=%any
auth=rsasig

xl2tpd and PPP

L2TP (PPP sessions) remains separate. Typically PPP authentication is still handled by PAP/CHAP or radius. Using certificate-based IPsec secures the tunnel and then PPP authenticates user sessions. Ensure pppd options (e.g., /etc/ppp/options.xl2tpd) are hardened and that no PAP is accepted unless necessary.

Important certificate and OpenSSL configuration details

Pay attention to the following technical specifics:

  • subjectAltName (SAN): Always include DNS and IP SANs used by clients. Many clients ignore CN if SAN exists.
  • Key usage and EKU: For server certificates, include keyEncipherment and serverAuth. If using client certs, include clientAuth.
  • CRL and OCSP: Publish a CRL or OCSP responder URL in the certificate if you plan to revoke certs. Example extension: authorityInfoAccess and crlDistributionPoints.
  • DH parameters: For IPsec, configure strongSwan/libreswan with modern DH groups (e.g., 24/25/26 or ECP groups). Avoid group 1/2/5. If using OpenSSL for other services, use a 2048+ DH or ECDH.
  • Permissions: server.key should be readable only by root (chmod 600). CA private key ideally kept offline.
  • Certificate chain: Present the full chain (server cert + intermediate(s) + root if required) so clients can validate without fetching intermediates.

Revocation and lifecycle

Establish processes for certificate revocation and renewal:

  • Generate a CRL with openssl ca -gencrl or openssl ca -revoke flows if you run a full CA directory structure.
  • Host CRL and OCSP endpoints over HTTPS and reference them in cert extensions so clients and servers can check status.
  • Automate renewal before expiry. Use monitoring to alert when certs approach expiration.
  • For client certificates, implement a secure enrollment and revocation workflow (e.g., SCEP, ACME-like automation for internal CA, or a management portal).

Hardening and best practices

Follow these operational security tips:

  • Use strong algorithms: RSA >=3072 or ECDSA prime256v1/secp384r1. Avoid deprecated algorithms and SHA-1.
  • Minimize certificate validity periods and rotate keys on compromise.
  • Keep the CA offline if possible; use an intermediate CA for daily signing to reduce exposure of the root key.
  • Monitor logs for failed certificate authentication attempts. Ensure ipsec logs are captured (strongSwan logs, syslog).
  • Use kernel and firewall hardening: ensure IP forwarding, NAT, and firewall rules allow only required traffic for IPsec and L2TP, and block management ports from open Internet when unnecessary.
  • Consider Perfect Forward Secrecy (PFS) by including appropriate DH/ECDH groups in IPsec proposals.

Troubleshooting tips

Common issues and checks:

  • Certificate not accepted — verify SANs and EKU fields: openssl x509 -in server.crt -text -noout.
  • Wrong chain — ensure clients receive the intermediate(s). Use strongSwan’s charon-cmd or check logs for verification failures.
  • Permissions errors — ensure server.key readable by the IPsec daemon user (typically root) and file SELinux contexts are correct if enabled.
  • CRL/OCSP failures — ensure CRL is reachable and timestamps are valid. Some clients cache CRLs; force a refresh if testing revocation.

Automation and scaling

For larger deployments, manual OpenSSL workflows become cumbersome. Consider:

  • Using a private ACME server (e.g., Boulder or smallstep) to automate issuance and rotation.
  • Integrating with an enterprise PKI (AD CS) and automating enrollment via SCEP/EST.
  • Maintaining configuration-as-code for IPsec and cert deployments (Ansible, Terraform, or custom scripts) and securely distributing keys (Vault, HSM).

Implementing certificate-based authentication for L2TP/IPsec significantly raises the security posture of your VPN infrastructure. With careful OpenSSL configuration (SANs, EKU, strong keys), disciplined CA practices (offline roots, CRLs), and proper IPsec integration (strongSwan/libreswan + xl2tpd), you can run a scalable, auditable VPN service suitable for enterprise and developer environments.

For implementation details, presets, or a checklist tailored to your environment, visit Dedicated-IP-VPN at https://dedicated-ip-vpn.com/