Manual Install

The following steps provide a step-by-step guide to installing Keysafe 5 and its dependencies into an existing Kubernetes cluster.

An alternative to this guide is the Keysafe 5 Quick Start Guide which provides a scripted means of installing Keysafe 5.

These steps install Keysafe 5 and its dependencies. They should be followed to set up a demo environment for evaluation purposes and should not be used for production environments.

Please see Hardening The Deployment for steps to harden the deployment. Entrust recommends these steps as a minimum and that additional hardening may be required dependent on your own requirements.

A production deployment will have as a minimum the following:

Maintained and patched versions of all the dependencies.
A secure CA with TLS v1.3 support for certificates. The deploy script can provide a local insecure CA.
A secure Kubernetes installation. The deploy script can install K3s locally.
A secure MongoDB database. The deploy script can provide a replicated MongoDB with X.509 authentication running in Kubernetes.
A secure RabbitMQ message broker. The deploy script can provide a RabbitMQ with X.509 authentication running in Kubernetes.
HTTPS secured by a trusted certificate for the Keysafe 5 endpoints. The deploy script will enable HTTPS connections with a self-signed insecure certificate.
Require authentication to access Keysafe 5. OIDC & OAUTH2 are currently supported in Keysafe 5. The deploy script will not set up authenticated access.

Unpack the release

$ mkdir keysafe5-install
$ tar -xf nshield-keysafe5-1.1.1.tar.gz -C keysafe5-install
$ cd keysafe5-install

Docker images

Push the Keysafe 5 images to your private docker registry:

# Load the Docker images to your local Docker
$ docker load < docker-images/hsm-mgmt.tar
$ docker load < docker-images/sw-mgmt.tar
$ docker load < docker-images/ui.tar

# We need to use a private registry in many places
$ export DOCKER_REGISTRY=private.registry.local

# Tag the Docker images for a private registry
$ docker tag hsm-mgmt:1.1.1 $DOCKER_REGISTRY/keysafe5/hsm-mgmt:1.1.1
$ docker tag sw-mgmt:1.1.1 $DOCKER_REGISTRY/keysafe5/sw-mgmt:1.1.1
$ docker tag mgmt-ui:1.1.1 $DOCKER_REGISTRY/keysafe5/mgmt-ui:1.1.1

# Log in to ensure pushes succeed
$ docker login $DOCKER_REGISTRY

# And push
$ docker push $DOCKER_REGISTRY/keysafe5/hsm-mgmt:1.1.1
$ docker push $DOCKER_REGISTRY/keysafe5/sw-mgmt:1.1.1
$ docker push $DOCKER_REGISTRY/keysafe5/mgmt-ui:1.1.1

Set up a CA

You should use your existing CA for a production system. This is simply used as an example for the purposes of having a working demo system.

The CA we will create is based on OpenSSL 1.1.1 - and is run inside a directory of your choosing, but in the examples here we will use /home/user/keysafe5-install/demoCA. In that directory, create the file demoCA.conf with the contents:

[ ca ]
default_ca      = CA_default                          # The default ca section

[ CA_default ]

dir             = /home/user/keysafe5-install/demoCA  # The directory of the CA
database        = $dir/index.txt                      # index file.
new_certs_dir   = $dir/newcerts                       # new certs dir

certificate     = $dir/cacert.pem                     # The CA cert
serial          = $dir/serial                         # serial no file
#rand_serial    = yes                                 # for random serial#'s
private_key     = $dir/private/cakey.pem              # CA private key
RANDFILE        = $dir/private/.rand                  # random number file

default_days    = 15                                  # how long to certify for
default_crl_days= 5                                   # how long before next CRL
default_md      = sha256                              # Message Digest
policy          = test_root_ca_policy
x509_extensions = certificate_extensions
unique_subject  = no
# This copy_extensions setting should not be used in a production system.
# It is simply used to simplify the demo system.
copy_extensions = copy

[ test_root_ca_policy ]
commonName = supplied
stateOrProvinceName = optional
countryName = optional
emailAddress = optional
organizationName = optional
organizationalUnitName = optional
domainComponent = optional

[ certificate_extensions ]
basicConstraints = CA:false

[ req ]
default_bits       = 4096
default_md         = sha256
prompt             = yes
distinguished_name = root_ca_distinguished_name
x509_extensions    = root_ca_extensions

[ root_ca_distinguished_name ]
commonName = hostname

[ root_ca_extensions ]
basicConstraints       = CA:true
keyUsage               = keyCertSign, cRLSign
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints       = critical,CA:true

Remember to update the dir value to the directory in which the demoCA.conf and the other CA files will be stored.

To generate the long-term CA key and random number source, we first make a directory called private:

$ mkdir ~/keysafe5-install/demoCA/private

Then run:

$ openssl genrsa -out ~/keysafe5-install/demoCA/private/cakey.pem 4096
$ openssl rand -out ~/keysafe5-install/demoCA/private/.rand 1024

The CA needs a self-signed certificate; as this is a short-term demo it will be valid for 90 days:

$ openssl req -x509 -new -nodes \
  -key demoCA/private/cakey.pem \
  -subj "/CN=demoCA" -days 90 \
  -out demoCA/cacert.pem \
  -config demoCA/demoCA.conf
$ cp demoCA/cacert.pem ca.crt

And finally, to finish off the configuration:

$ mkdir demoCA/newcerts
$ echo 01 > demoCA/serial
$ touch demoCA/index.txt

Install and set up the supporting software

Kubernetes namespace

Create a namespace in Kubernetes for Keysafe 5 installation.

$ kubectl create namespace nshieldkeysafe5

Istio

The version of Istio installed will align with the software version of istioctl.

$ istioctl install -y

RabbitMQ

Entrust recommends that you use your standard secure RabbitMQ installation, along with your policies for authentication and virtual hosts on your production system; this is only a demo system.

First, we need to generate the TLS keys, and guest password. We need to add the network addresses through which RabbitMQ will be accessed to the certificate, and are very dependent on the configuration of the Kubernetes cluster.

$ openssl genrsa -out ~/keysafe5-install/rabbit.key 4096
$ export DNS1="*.rabbit-chart-rabbitmq-headless.rabbitns.svc.cluster.local"
$ export DNS2=rabbit-chart-rabbitmq.rabbitns.svc
$ export DNS3=rabbitmq.rabbitns.svc.cluster.local
$ export DNS4=host.docker.internal
$ export LOCALIP=127.0.0.1
$ export HOSTIP=$(hostname -I | cut -f1 -d" ")
$ openssl req -new -key ~/keysafe5-install/rabbit.key \
  -out ~/keysafe5-install/rabbitmq.csr -subj \
  "/CN=rabbitmq/C=GB/L=Cambridge" \
  -addext "keyUsage=digitalSignature" \
  -addext "extendedKeyUsage=serverAuth" \
  -addext "subjectAltName=DNS:rabbitmq,DNS:${DNS1},DNS:${DNS2},DNS:${DNS3},DNS:${DNS4},DNS:${HOSTNAME},IP:${LOCALIP},IP:${HOSTIP}"
$ openssl ca -config ~/keysafe5-install/demoCA/demoCA.conf \
  -out rabbit.crt \
  -in rabbitmq.csr -batch
$ rm rabbitmq.csr

Now we create the kubernetes secrets for the RabbitMQ service:

$ kubectl create namespace rabbitns
$ kubectl create secret generic rabbitmq-certificates \
  --namespace=rabbitns \
  --from-file=ca.crt \
  --from-file=tls.crt=rabbit.crt \
  --from-file=tls.key=rabbit.key
$ kubectl -n rabbitns create secret generic rabbitmq-pw \
  --from-literal=rabbitmq-password=guest

Then install RabbitMQ.

$ helm repo add bitnami https://charts.bitnami.com/bitnami && helm repo update
$ helm install rabbit-chart \
  --namespace=rabbitns
  --set image.tag=3.11.3-debian-11-r2 \
  --set auth.username=guest \
  --set auth.existingPasswordSecret=rabbitmq-pw \
  --set auth.tls.enabled=true \
  --set auth.tls.existingSecret=rabbitmq-certificates \
  --set replicaCount=2 \
  --set service.type=LoadBalancer \
  --set extraConfiguration='
    listeners.ssl.default = 5671
    ssl_options.versions.1 = tlsv1.3
    ssl_options.depth=0
    ssl_options.verify = verify_peer
    ssl_options.fail_if_no_peer_cert = true
    auth_mechanisms.1 = EXTERNAL
    ssl_cert_login_from = subject_alternative_name
    ssl_cert_login_san_type = dns
    ssl_cert_login_san_index = 0' \
  --set plugins="" \
  --set extraPlugins="rabbitmq_auth_mechanism_ssl" \
  --wait --timeout 10m  bitnami/rabbitmq --version 11.1.2

As we have the service.type set to LoadBalancer we use the IP address of this machine for AMQP connections. We have also added ${HOSTNAME} to the certificate’s subjectAltName as a DNS entry so it may also be used instead of the IP address.

$ export RABBIT_URL=${HOSTIP}:5671

Add the virtual host that will be used for Keysafe 5 communication.

$ export RUN_RABBIT="kubectl -n rabbitns exec rabbit-chart-rabbitmq-0 -c rabbitmq -- "
$ export RABBIT_VHOST=nshieldvhost
$ ${RUN_RABBIT} rabbitmqctl add_vhost ${RABBIT_VHOST}

Then add the X.509 users for Keysafe 5, enable X.509 authentication, then make RabbitMQ accessible from outside the cluster. It’s also a good idea to set up the X.509 keys and certificates for Keysafe 5 and its agents.

$ export AGENT_USER=keysafe5-agent
$ export KS5_USER=ks5
$ for x509user in $AGENT_USER $KS5_USER
  do
    ${RUN_RABBIT} rabbitmqctl add_user $x509user "ephemeralpw"
    ${RUN_RABBIT} rabbitmqctl set_permissions -p $RABBIT_VHOST $x509user ".*" ".*" ".*"
    ${RUN_RABBIT} rabbitmqctl clear_password $x509user
    openssl genrsa -out $x509user.key 4096
    openssl req -new -key $x509user.key -out $x509user.csr \
      -subj "/CN=${x509user}/C=GB/L=Cambridge" \
      -addext "keyUsage=digitalSignature" \
      -addext "extendedKeyUsage=clientAuth" \
      -addext "subjectAltName=DNS:${x509user}"
    openssl ca -config ~/keysafe5-install/demoCA/demoCA.conf \
      -out ${x509user}.crt -in ${x509user}.csr -batch
    rm ${x509user}.csr
  done
$ kubectl create secret generic ks5-amqptls \
  --namespace nshieldkeysafe5 \
  --from-file=ca.crt \
  --from-file=tls.crt=ks5.crt \
  --from-file=tls.key=ks5.key

We can now remove access for the default guest user.

$ ${RUN_RABBIT} rabbitmqctl delete_user guest

Note that we need to keep a set of AMQP TLS credentials for the Keysafe 5 agent.

$ tar -zcf ~/keysafe5-install/agent-amqptls.tar.gz \
  keysafe5-agent.key keysafe5-agent.crt ca.crt

MongoDB

Entrust recommends that you use your standard secure MongoDB Replica Set installation. This is just an example, and not production-ready.

$ kubectl create namespace mongons
$ helm install mongo-chart \
  --set image.tag=5.0.13-debian-11-r16 \
  --set architecture=replicaset \
  --set auth.enabled=true \
  --set auth.username=dummyuser \
  --set auth.password=dummypassword \
  --set auth.database=authdb \
  --set tls.enabled=true \
  --namespace=mongons \
  bitnami/mongodb --version 12.1.31

There will be a message listing the MongoDB server addresses. We save them to a variable for use later.

$ export MONGO1=mongo-chart-mongodb-0.mongo-chart-mongodb-headless.mongons.svc.cluster.local:27017
$ export MONGO2=mongo-chart-mongodb-1.mongo-chart-mongodb-headless.mongons.svc.cluster.local:27017
$ export MONGODB=${MONGO1},${MONGO2}

We then pick up the secrets in the MongoDB configuration.

$ kubectl get secret --namespace mongons mongo-chart-mongodb-ca \
  -o jsonpath="{.data.client-pem}" | base64 --decode | \
  openssl pkey -out mongo-client-key.pem
$ kubectl get secret --namespace mongons mongo-chart-mongodb-ca \
  -o jsonpath="{.data.client-pem}" | base64 --decode | \
  openssl x509 -out mongo-client-cert.pem
$ kubectl get secret --namespace mongons mongo-chart-mongodb-ca \
  -o jsonpath="{.data.mongodb-ca-cert}" | base64 --decode > mongo-ca-cert.pem

We add those secrets in a format that Keysafe 5 can accept.

$ kubectl create secret generic mongodb-client-tls \
  --namespace=nshieldkeysafe5 \
  --from-file=ca.crt=mongo-ca-cert.pem \
  --from-file=tls.crt=mongo-client-cert.pem \
  --from-file=tls.key=mongo-client-key.pem
$ rm mongo-ca-cert.pem mongo-client-key.pem

Access the MongoDB shell to create a user with read/write permissions on the hsm-mgmt-db and sw-mgmt-db collections. Note that the username needs to match the subject of the client certificate, as found by the following command.

$ openssl x509 -in mongo-client-cert.pem -subject | head -n 1

In this example we will use mongo-chart-mongodb.mongons.svc.cluster.local

We then run the mongo client container.

$ export MONGO_RUN="kubectl -n mongons exec mongo-chart-mongodb-0 0 -- "
$ export TLS_PRIVKEY="$(${MONGO_RUN} bash -c 'cat /certs/mongodb.pem')"
$ export TLS_CERT="$(${MONGO_RUN} bash -c 'cat /certs/mongodb-ca-cert')"
$ export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace mongons \
  mongo-chart-mongodb -o jsonpath="{.data.mongodb-root-password}" \
  | base64 --decode)
$ kubectl run --namespace mongons mongo-chart-mongodb-client \
  --rm --tty -i --restart='Never' --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" \
  --env="TLS_PRIVKEY=$TLS_PRIVKEY" --env="TLS_CERT=$TLS_CERT" --env="MONGODB=$MONGODB" \
  --image bitnami/mongodb:5.0.13-debian-11-r1 --command -- bash

Once inside the mongo client container, we need to set up a connection to the server before we can start mongo admin and create the user. After we finish creating the user, we need to exit mongo admin, and then the mongo-client container.

$ echo "$TLS_CERT" > /tmp/tls.crt
$ echo "$TLS_PRIVKEY" > /tmp/tls.key
$ mongo admin --tls --tlsCAFile /tmp/tls.crt --tlsCertificateKeyFile /tmp/tls.key \
  --host $MONGODB --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD

> use $external
> x509_user = {
   "user" : "CN=mongo-chart-mongodb.mongons.svc.cluster.local",
   "roles" : [
     {"role": "readWrite", "db": "hsm-mgmt-db" },
     {"role": "readWrite", "db": "sw-mgmt-db" },
   ]
 }
> db.createUser(x509_user)
> exit
$ exit

Install Keysafe 5

# Get Ingress IP address
$ export INGRESS_IP=$(kubectl --namespace istio-system get svc -l app=istio-ingressgateway -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

# Install the KeySafe 5 backend services
$ helm install keysafe5-backend \
  --namespace=nshieldkeysafe5 \
  --set hsm_mgmt.image=$DOCKER_REGISTRY/keysafe5/hsm-mgmt:1.0.0 \
  --set sw_mgmt.image=$DOCKER_REGISTRY/keysafe5/sw-mgmt:1.0.0 \
  --set database.type=mongo \
  --set database.mongo.hosts="$MONGO1\,$MONGO2" \
  --set database.mongo.replicaSet=rs0 \
  --set database.mongo.auth.type=tls \
  --set database.mongo.auth.authDatabase=authdb \
  --set database.mongo.tls.enabled=true \
  --set database.mongo.tls.existingSecret=mongodb-client-tls \
  --set amqp.URL=${RABBIT_URL}/${RABBIT_VHOST} \
  --set amqp.auth.type=tls \
  --set amqp.tls.enabled=true \
  --set amqp.tls.existingSecret=ks5-amqptls \
  --wait --timeout 10m \
 helm-charts/nshield-keysafe5-backend-1.1.1.tgz

# Install the KeySafe 5 UI
$ helm install keysafe5-ui \
  --namespace=nshieldkeysafe5 \
  --set ui.image=$DOCKER_REGISTRY/keysafe5/mgmt-ui:1.0.0 \
  --set svcEndpoint="https://${HOSTNAME}" \
  --set authMethod=none \
  --wait --timeout 10m \
helm-charts/nshield-keysafe5-ui-1.1.1.tgz

# Create the TLS secret for the Istio Ingress Gateway
$ openssl genrsa -out istio.key 4096
$ openssl req -new -key istio.key -out istio.csr \
  -subj "/CN=${HOSTNAME}" \
  -addext "keyUsage=digitalSignature" \
  -addext "extendedKeyUsage=serverAuth" \
  -addext "subjectAltName=DNS:${HOSTNAME},IP:${INGRESS_IP}"
$ openssl ca -config ~/keysafe5-install/demoCA/demoCA.conf \
  -out istio.crt -in istio.csr -batch
$ kubectl -n istio-system create secret tls \
  keysafe5-server-credential --cert=istio.crt --key=istio.key

# Configure Istio Ingress Gateway for KeySafe 5
$ helm install keysafe5-istio \
  --namespace=nshieldkeysafe5 \
  --set tls.existingSecret=keysafe5-server-credential \
  --set requireAuthn=false \
  --wait --timeout 1m \
helm-charts/nshield-keysafe5-istio-1.1.1.tgz

Access Keysafe 5

You can now access Keysafe 5 on either https://$INGRESS_IP or https://$HOSTNAME.

For example, you could send curl requests as demonstrated below.

$ curl -X GET --cacert demoCA/cacert.pem https://${HOSTNAME}/mgmt/v1/hsms | jq
$ curl -X GET --cacert demoCA/cacert.pem https://${HOSTNAME}/mgmt/v1/hosts | jq
$ curl -X GET --cacert demoCA/cacert.pem https://${HOSTNAME}/mgmt/v1/pools | jq
$ curl -X GET --cacert demoCA/cacert.pem https://${HOSTNAME}/mgmt/v1/feature-certificates | jq
$ curl -X GET --cacert demoCA/cacert.pem https://${HOSTNAME}/mgmt/v1/worlds | jq

You can access the Management UI in a web browser at https://$HOSTNAME.

Configure nShield client machines

To configure a host machine to be managed and monitored by this deployment, run the Keysafe 5 agent binary on the nShield client machine containing the relevant Security World or HSMs.

Configure this Keysafe 5 agent to communicate with the same AMQP service as the previous deployment.

You will need to copy the agent-amqptls.tar.gz created above.

Ensure no firewall rules are blocking the AMQP port communication between the machine exposing the AMQP port from Kubernetes and the machine running the agent.

$ sudo tar -xf keysafe5-install/keysafe5-agent/keysafe5-1.1.1-Linux-keysafe5-agent.tar.gz -C /
$ export KS5CONF=/opt/nfast/keysafe5/conf
$ cat << EOF | sudo tee $KS5CONF/config.yaml
  logging:
    level: info
    format: json
    file:
      enabled: true
      path: /opt/nfast/log/keysafe5-agent.log

  amqp:
    url: "${INGRESS_IP}:5671/${RABBIT_VHOST}"
    auth_type: "tls"
  update_interval: 15s
  health_interval: 15s
EOF

$ sudo mkdir -p $KS5CONF/amqp/tls
$ sudo tar xf agent-amqptls.tar.gz -C $KS5CONF/amqp/tls
$ sudo mv $KS5CONF/amqp/tls/keysafe5-agent.key $KS5CONF/amqp/tls/tls.key
$ sudo mv $KS5CONF/amqp/tls/keysafe5-agent.crt $KS5CONF/amqp/tls/tls.crt

If the hardserver is already running, use the Keysafe 5 install script to not restart it when installing the Keysafe 5 agent.

$ sudo /opt/nfast/keysafe5/sbin/install

Otherwise, use the nShield install script which will start both the nShield Security World software and the Keysafe 5 agent.

$ sudo /opt/nfast/sbin/install

Uninstall

helm --namespace nshieldkeysafe5 uninstall keysafe5-istio
helm --namespace nshieldkeysafe5 uninstall keysafe5-backend
helm --namespace nshieldkeysafe5 uninstall keysafe5-ui
helm --namespace rabbitns uninstall rabbit-chart
helm --namespace mongons uninstall mongo-chart

To uninstall the Keysafe 5 agent, run the Keysafe 5 uninstaller.

$ sudo /opt/nfast/keysafe5/sbin/install -u