Protecting SSL Private Keys in NGINX with HashiCorp Vault

Original: https://www.nginx.com/blog/protecting-ssl-private-keys-nginx-hashicorp-vault/

In the first post in this series, we describe several approaches to improving the security of your SSL private keys. The post finished with a demonstration of a remote password distribution point (PDP) used to securely share encryption passwords with NGINX instances.

Secrets management systems like HashiCorp Vault operate in a similar fashion to that sample PDP:

In this post, we show how to set up HashiCorp Vault to distribute SSL passwords. For even more security, you can set up an external hardware security module (HSM).

This post applies to both NGINX Open Source and NGINX Plus. For ease of reading, we’ll refer to NGINX throughout.

Using HashiCorp Vault to Protect SSL Private Keys

The instructions in this section set up a central PDP server using Vault to distribute SSL passwords. They are based on DigitalOcean’s instructions; modify them as necessary to comply with your own Vault policies.

In our example, each remote web server has a unique authentication token. These tokens can be used to access secrets in Vault’s secret/webservers/ path, and we store the SSL passwords in secret/webservers/ssl_passwords.

We will see how to secure the tokens, and how to revoke individual authentication tokens when necessary.

Download, Install, and Initialize HashiCorp Vault on the PDP Server

  1. Follow the DigitalOcean instructions to download and extract Vault on your PDP server. We’re using the following sample /etc/vault.hcl file to make Vault remotely accessible, and to disable TLS (for ease of use when testing):

    backend "file" {
        path = "/var/lib/vault"
    }
    
    listener "tcp" {
        address = "0.0.0.0:8200"
        tls_disable = 1
    }
  2. Start Vault using the startup scripts (if you created them), or manually:

    [email protected]:~$ sudo /usr/local/bin/vault server -config=/etc/vault.hcl
  3. Initialize Vault and obtain the Initial Root Token:

    [email protected]:~$ export VAULT_ADDR=http://localhost:8200
    [email protected]:~$ vault init -key-shares=3 -key-threshold=2
    [email protected]:~$ vault operator unseal
    Initial Root Token: 86c5c2a4-8ab2-24dd-1816-48449c83114e
  4. We need to provide the Initial Root Token in many of the following commands. For convenience, we assign it to the root_token shell variable:

    [email protected]:~$ root_token=86c5c2a4-8ab2-24dd-1816-48449c83114e

Store the Secrets

  1. Still working on the PDP server, create a temporary file called /tmp/ssl_passwords.txt with the passwords in it.
  2. Store this file as a secret in Vault, and verify that you can retrieve it:

    [email protected]:~$ VAULT_TOKEN=$root_token vault kv put secret/webservers/ssl_passwords [email protected]/tmp/ssl_passwords.txt
    
    [email protected]:~$ VAULT_TOKEN=$root_token vault kv get -field=value secret/webservers/ssl_passwords
    password1
    password2
    ...
  3. For security, delete /tmp/ssl_passwords.txt.
  4. Create a policy specification in a file called web.hcl with the following contents:

    path "secret/webservers/*" {
        capabilities = ["read"]
    }
  5. Load the policy into Vault, naming it web:

    [email protected]:~$ VAULT_TOKEN=$root_token vault policy write web web.hcl
  6. Create a new authentication token, associate it with the web policy, and optionally include the display-name parameter to give it a user‑friendly name. Make a note of the token and token_accessor values; you’ll use them in subsequent commands:

    [email protected]:~$ VAULT_TOKEN=$root_token vault token create -policy=web -display-name=webserver1
    Key                  Value
    ---                  -----
    token                dcf75ffd-a245-860f-6960-dc9e834d3385
    token_accessor       0c1d6181-7adf-7b42-27be-b70cfa264048

    The NGINX web server uses this token to retrieve the SSL passwords. The web policy prevents the web server from retrieving secrets outside the secret/webservers/* path.

Verify the Web Server Can Retrieve the Token

  1. Working on the NGINX web server, install the Vault binary.
  2. Declare the location of the remote Vault server (here, http://pdp:8200), and then verify that the web server machine can retrieve the SSL passwords using the token:

    [email protected]:~$ export VAULT_ADDR=http://pdp:8200
    [email protected]:~$ VAULT_TOKEN=dcf75ffd-a245-860f-6960-dc9e834d3385 vault kv get -field=value secret/webservers/ssl_passwords
    password1
    password2
    ...

Configure the NGINX Vault Connector on the Web Server

  1. As part of setting up the sample PDP in the first post we created a shell script called connector.sh on the NGINX host (web server machine). Here we modify it to use Vault:

    #!/bin/sh
    # Usage: connector_v.sh  
    
    CONNECTOR=$1
    CREDS=$2
    
    [ -e $CONNECTOR ] && /bin/rm -f $CONNECTOR
    
    mkfifo $CONNECTOR; chmod 600 $CONNECTOR
    
    export VAULT_ADDR=http://pdp:8200
    export VAULT_TOKEN=$CREDS
    
    while true; do
        vault kv get -field=value secret/webservers/ssl_passwords > $CONNECTOR
        sleep 0.1 # race condition, ensures EOF
    done
  2. Run the script as a background process, invoked as follows:

    [email protected]:~# ./connector_v.sh /var/run/nginx/ssl_passwords \
    dcf75ffd-a245-860f-6960-dc9e834d3385 &
  3. Test the connector by reading from the connector path:

    [email protected]:~$ cat /var/run/nginx/ssl_passwords
    password1
    password2
    ...
  4. Configure NGINX to read the ssl_passwords file on startup, and to use the contents as passwords to decrypt encrypted private keys. You can include the ssl_password_file directive either in a server block (like the one created for the standard configuration in the first post) or in the http context to apply it to multiple virtual servers:

    ssl_password_file /var/run/nginx/ssl_passwords;
  5. Verify that NGINX can read the password and decrypt the SSL keys:

    [email protected]:~# nginx -t
    nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
    nginx: configuration file /etc/nginx/nginx.conf test is successful

Revoking a Web Server’s Authentication Token

You can easily revoke access if the web server is compromised or when it is decommissioned. To do so, you can directly revoke the authentication token used by the web server:

[email protected]:~$ VAULT_TOKEN=$root_token vault token revoke dcf75ffd-a245-860f-6960-dc9e834d3385

Vault tokens must be kept secret, so it’s not a best practice to type their actual identifiers on the command line. It’s more common to identify a token using its accessor, which gives limited rights over the token and cannot be used to retrieve the token value.

If you need to determine the accessor for a web server’s authentication token, run the vault list command to retrieve the list of accessors, and the vault token lookup command on each accessor to find the one with the relevant display name and policy:

[email protected]:~$ VAULT_TOKEN=$root_token vault list /auth/token/accessors
Keys
----
83be5a73-9025-1221-cb70-4b0e8a3ba8df
0c1d6181-7adf-7b42-27be-b70cfa264048
f043b145-7a63-01db-ea85-9f22f413c55e
[email protected]:~$ VAULT_TOKEN=$root_token vault token lookup -accessor 0c1d6181-7adf-7b42-27be-b70cfa264048
Key                 Value
---                 -----
...
display-name        webserver1       
...
policy              web
...

You can then revoke the token using its accessor:

[email protected]:~$ VAULT_TOKEN=$root_token vault token revoke -accessor 0c1d6181-7adf-7b42-27be-b70cfa264048

Security Implications

Using Vault has a similar security profile to the sample PDP described in the first post. SSL private keys can only be obtained if the corresponding password is obtained, and for this, an attacker needs to know the value of a current authentication token.

The primary benefit of using Vault is to automate and scale the secret store.

Using an External HSM to Manage Private Keys

None of the solutions we’ve covered so far in the series protect the private key when an attacker gains root access to the NGINX server. Alternatively, if an attacker can access NGINX’s runtime memory or generate a core dump, there are well‑known techniques to scan the process’s memory and locate private key data.

External hardware security modules (HSMs) address this by storing the SSL private keys in external, tamper‑proof hardware. They offer decryption as a service, and NGINX accesses that service whenever it needs to perform an SSL operation that requires the kay.

The NGINX server never sees the SSL private key data. An attacker who gains root access on the server cannot obtain the SSL private key, but can decrypt data on demand by accessing the HSM decryption service using the NGINX credentials.

Configuring NGINX to Access an HSM

NGINX delegates all SSL private key operations to a crypto library called OpenSSL. Third‑party HSM devices can be made available to NGINX by using the HSM vendor’s OpenSSL engine.

The NGINX configuration is specific to each vendor HSM, but generally follows a straightforward path:

For an example of HSM setup, refer to Amazon’s CloudHSM documentation.

Security Implications

External HSMs are a highly secure method of storing SSL private keys. An attacker with root access to the NGINX server is able to leverage the NGINX credentials to decrypt arbitrary data using the HSM but is not able to obtain the unencrypted private key. An HSM makes it significantly harder for an attacker to impersonate a website or to decrypt arbitrary data.

Conclusion

It’s essential to ensure that secret data such as the SSL private key is fully protected because the consequences of disclosure are very serious.

For many organizations with appropriate security processes in place, it’s sufficient to store the private key on the frontend load balancers and then limit and audit all access to those servers (the standard configuration described on the first post).

For organizations that need to deploy NGINX configuration frequently, the measures in this post and the first one can be used to limit the users or entities who can see the private key data.

In the final post in this series, we’ll explain how to automate the provisioning of keys and certificates from Vault to NGINX Plus’s key‑value store, using the NGINX Plus API.

Note again that none of these methods reduces the need to fully secure the running NGINX instances from remote access or configuration manipulation.

Retrieved by Nick Shadrin from nginx.com website.