Why strong server certificates matter for SSTP

SSTP (Secure Socket Tunneling Protocol) rides on TLS, using TCP port 443 to encapsulate PPP frames. Because SSTP hands off security to the TLS layer, the server certificate is the single most critical element in establishing trust and protecting user credentials and traffic. A misconfigured certificate can cause client failures, downgrade attacks, or allow man-in-the-middle interception. This guide shows how to create and manage proper server certificates with OpenSSL so SSTP deployments are secure, interoperable, and practical for production.

Design considerations before generating certificates

Before you run commands, decide on these items:

  • Certificate authority (CA) model: Use either an internal CA for internal VPNs, or a public CA (recommended for wide client support) if you need trust by default without manual client configuration.
  • Host identity: The certificate must match how clients connect — use a DNS name rather than a raw IP when possible. If you must use IP, ensure the IP is in the certificate’s subjectAltName (SAN).
  • Key strength and algorithms: Use at least RSA 2048 or preferably 4096 or ECC keys (e.g., prime256v1) depending on your environment and client compatibility.
  • Extended Key Usage: Server certs should include serverAuth. Key usage should allow digitalSignature and keyEncipherment for TLS.
  • Certificate lifetime: Shorter lifetimes reduce risk; 1–3 years is common for server certs in controlled deployments. Rotate keys periodically.

Practical OpenSSL workflow overview

The step-by-step flow used in this guide:

  • Create a CA private key and self-signed CA certificate (if you use an internal CA).
  • Create a server private key and a Certificate Signing Request (CSR) with SAN entries.

    • <li.Sign the CSR with the CA to produce a server certificate that includes appropriate extensions.

  • Optionally create a PKCS#12 (.pfx/.p12) bundle for Windows SSTP (RRAS) or devices that expect a single file containing the private key and certificate chain.
  • Deploy certificate, configure SSTP server (e.g., RRAS on Windows), and verify with clients.

1) Set up a CA (internal)

On a secure machine (offline is ideal for root CA), generate a CA key and certificate. Example uses RSA 4096 and SHA-256:

openssl genpkey -algorithm RSA -out ca.key -pkeyopt rsa_keygen_bits:4096

openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -subj “/C=US/ST=State/L=City/O=Org/OU=VPN/CN=Example-Internal-CA” -out ca.crt

Notes: Keep ca.key highly secured. If you are using a public CA, skip these CA steps and request a certificate from the CA following their CSR process.

2) Create the server key

Generate a secure server private key. You can choose RSA or ECC; here is RSA 4096:

openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:4096

Or ECC (prime256v1):

openssl ecparam -name prime256v1 -genkey -noout -out server.key

3) Create a CSR with SAN (subjectAltName)

TLS clients require SAN entries. Modern clients ignore the CN and rely solely on SAN. Create an OpenSSL config file fragment to include SAN and proper extendedKeyUsage. Example minimal file (server-openssl.cnf):

[ req ]distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[ req_distinguished_name ]C = US
ST = State
L = City
O = ExampleOrg
OU = VPN
CN = vpn.example.com

[ v3_req ]keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[ alt_names ]DNS.1 = vpn.example.com
IP.1 = 203.0.113.10

Then generate the CSR using this config:

openssl req -new -key server.key -out server.csr -config server-openssl.cnf

Tip: Replace DNS and IP with your actual hostnames and IPs. If clients use a load balancer or dedicated IP, ensure that name/IP is included.

4) Sign the CSR with your CA (create server certificate)

You must ensure the certificate includes serverAuth and SAN. Use a signing config or -extfile to add extensions.

Create a file server-ext.cnf with v3 extensions:

[ v3_ca ]subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[ alt_names ]DNS.1 = vpn.example.com
IP.1 = 203.0.113.10

Sign the CSR for one year with CA key/cert:

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256 -extfile server-ext.cnf -extensions v3_ca

You can adjust -days and algorithm as needed. The -CAcreateserial creates ca.srl; track it for CA management.

5) Verify the certificate

Check that SAN and EKU are present:

openssl x509 -in server.crt -noout -text

Look for:

  • Subject Alternative Name: your DNS and IP entries
  • X509v3 Extended Key Usage: TLS Web Server Authentication
  • Key Usage includes digitalSignature, keyEncipherment

6) Create a PKCS#12 bundle for Windows SSTP

Windows SSTP (RRAS) commonly expects a certificate installed in the machine store. If you manage servers manually or supply to admins, package the private key and cert chain into a PFX:

openssl pkcs12 -export -out server.pfx -inkey server.key -in server.crt -certfile ca.crt

You will be prompted for an export passphrase. Use a strong passphrase and transfer the .pfx securely. On Windows import into the Local ComputerPersonal store and then bind the certificate to RRAS SSTP interface.

Operational guidance and hardening

Beyond generation, consider these operational points:

  • Private key protection: Use filesystem permissions, HSMs, or TPM-backed keys for better protection. Never store unencrypted keys on publicly accessible machines.
  • Certificate chain: Ensure clients have the CA certificate in their trust store if using an internal CA. For Windows clients, you can distribute the CA cert via Group Policy.
  • CRL and OCSP: Publish a CRL or run an OCSP responder for revocation. Configure the CA to include CRL/OCSP AIA URLs in issued certificates so clients can check revocation.
  • Cipher suites and TLS versions: Configure the SSTP/TLS endpoint to prefer TLS 1.2/1.3 and disable old ciphers (e.g., RC4, 3DES). SSTP support of TLS 1.3 depends on platform; Windows Server 2019+ supports modern TLS stacks.
  • Monitoring: Monitor certificate expirations (automate alerts) and keep the certificate lifecycle mapped in your change management system.

Common gotchas and troubleshooting

Here are frequent issues and how to quickly diagnose them:

  • Clients reject the certificate: Confirm SAN contains the name clients use. Verify the CA is trusted by the client.
  • SSTP connection stalls during TLS handshake: Check server certificates, private key match, and that the server isn’t missing intermediate certs in the chain.
  • Wrong EKU or Key Usage: Some clients will reject certs that lack serverAuth; reissue ensuring extendedKeyUsage is set.
  • Certificate expired: Set up monitoring for certificate expiration and renew well before expiry.
  • Mixed IPv4/IPv6: If clients connect over IPv6, include IPv6 addresses in SAN as IP.2, etc.

Example minimal command sequence (quick reference)

For an internal CA you can run these commands (adjust names and SAN content):

  • openssl genpkey -algorithm RSA -out ca.key -pkeyopt rsa_keygen_bits:4096
  • openssl req -x509 -new -key ca.key -sha256 -days 3650 -subj “/CN=My-Internal-CA” -out ca.crt
  • openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:4096
  • Create CSR with config containing SAN (see earlier example), then: openssl req -new -key server.key -out server.csr -config server-openssl.cnf
  • Sign: openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256 -extfile server-ext.cnf -extensions v3_ca
  • Package for Windows: openssl pkcs12 -export -out server.pfx -inkey server.key -in server.crt -certfile ca.crt

Final checklist before going live

  • Certificate contains SAN entries matching client endpoints.
  • Certificate EKU includes serverAuth.
  • Private key is protected and accessible only by the service account running SSTP/TLS.
  • CA chain is correctly installed on the server and distributed to clients if internal CA is used.
  • Revocation infrastructure (CRL/OCSP) is available and referenced in certificates if you need quick revocation.
  • TLS protocol and cipher configuration follows best practices for the OS and platform.

Implementing SSTP with carefully built OpenSSL certificates gives you full control and transparency over trust, validity, and lifecycle. This approach is suitable for small-to-medium deployments and for testing before transitioning to certificates from public CAs.

For deployment templates, configuration snippets, and managed service recommendations tailored to enterprise environments, visit Dedicated-IP-VPN at https://dedicated-ip-vpn.com/.