Manual Install

The following steps provide a step-by-step guide to installing KeySafe 5 and its dependencies into an existing Kubernetes cluster.

An alternative to this guide is the KeySafe 5 Quick Start Guide which provides a scripted means of installing KeySafe 5.

These steps install KeySafe 5 and its dependencies. They should be followed to set up a demo environment for evaluation purposes and should not be used for production environments.

Please see Hardening The Deployment for steps to harden the deployment. Entrust recommends these steps as a minimum and that additional hardening may be required dependent on your own requirements.

A production deployment will have as a minimum the following:

Maintained and patched versions of all the dependencies.
A secure CA with TLS v1.3 support for certificates. The deploy script can provide a local insecure CA.
A secure Kubernetes installation. The deploy script can install K3s locally.
A secure MongoDB database. The deploy script can provide a replicated MongoDB with X.509 authentication running in Kubernetes.
A secure RabbitMQ message broker. The deploy script can provide a RabbitMQ with X.509 authentication running in Kubernetes.
A secure means of large object storage. The deploy script can provide local object storage within the Kubernetes cluster or be configured for using an NFS for object storage.
HTTPS secured by a trusted certificate for the KeySafe 5 endpoints. The deploy script will enable HTTPS connections with a self-signed insecure certificate.
Require authentication to access KeySafe 5. OIDC & OAUTH2 are currently supported in KeySafe 5. The deploy script will not set up authenticated access.

Unpack the release

$ mkdir ~/keysafe5-install
$ tar -xf nshield-keysafe5-1.2.0.tar.gz -C ~/keysafe5-install
$ cd ~/keysafe5-install

Docker images

The Docker images need to be loaded in a Docker registry that each node in your Kubernetes cluster can pull the images from.

# Load the Docker images to your local Docker
$ docker load < docker-images/codesafe-mgmt.tar
$ docker load < docker-images/hsm-mgmt.tar
$ docker load < docker-images/sw-mgmt.tar
$ docker load < docker-images/ui.tar

To run a local private Docker registry, see Deploy a registry server.

Push the KeySafe 5 images to your private docker registry:

$ export DOCKER_REGISTRY=private.registry.local

# Tag the Docker images for a private registry
$ docker tag codesafe-mgmt:1.2.0 $DOCKER_REGISTRY/keysafe5/codesafe-mgmt:1.2.0
$ docker tag hsm-mgmt:1.2.0 $DOCKER_REGISTRY/keysafe5/hsm-mgmt:1.2.0
$ docker tag sw-mgmt:1.2.0 $DOCKER_REGISTRY/keysafe5/sw-mgmt:1.2.0
$ docker tag mgmt-ui:1.2.0 $DOCKER_REGISTRY/keysafe5/mgmt-ui:1.2.0

# Log in to ensure pushes succeed
$ docker login $DOCKER_REGISTRY

# And push
$ docker push $DOCKER_REGISTRY/keysafe5/codesafe-mgmt:1.2.0
$ docker push $DOCKER_REGISTRY/keysafe5/hsm-mgmt:1.2.0
$ docker push $DOCKER_REGISTRY/keysafe5/sw-mgmt:1.2.0
$ docker push $DOCKER_REGISTRY/keysafe5/mgmt-ui:1.2.0

Set up a Certificate Authority

You should use your existing CA for a production system. This is simply used as an example for the purposes of having a working demo system.

Either OpenSSL 3.0 or OpenSSL 1.1.1 may be used to create the CA, and the CA may be created in a directory of your choosing. In these examples, /home/user/keysafe5-install/internalCA is the example directory used. In that directory, create the file internalCA.conf with the contents:

[ ca ]
default_ca      = CA_default                          # The default ca section

[ CA_default ]

dir             = /home/user/keysafe5-install/internalCA  # The directory of the CA
database        = $dir/index.txt                      # index file.
new_certs_dir   = $dir/newcerts                       # new certs dir

certificate     = $dir/cacert.pem                     # The CA cert
serial          = $dir/serial                         # serial no file
#rand_serial    = yes                                 # for random serial#'s
private_key     = $dir/private/cakey.pem              # CA private key
RANDFILE        = $dir/private/.rand                  # random number file

default_days    = 15                                  # how long to certify for
default_crl_days= 5                                   # how long before next CRL
default_md      = sha256                              # Message Digest
policy          = test_root_ca_policy
x509_extensions = certificate_extensions
unique_subject  = no
# This copy_extensions setting should not be used in a production system.
# It is simply used to simplify the demo system.
copy_extensions = copy

[ test_root_ca_policy ]
commonName = supplied
stateOrProvinceName = optional
countryName = optional
emailAddress = optional
organizationName = optional
organizationalUnitName = optional
domainComponent = optional

[ certificate_extensions ]
basicConstraints = CA:false

[ req ]
default_bits       = 4096
default_md         = sha256
prompt             = yes
distinguished_name = root_ca_distinguished_name
x509_extensions    = root_ca_extensions

[ root_ca_distinguished_name ]
commonName = hostname

[ root_ca_extensions ]
basicConstraints       = CA:true
keyUsage               = keyCertSign, cRLSign
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints       = critical,CA:true

Remember to update the dir value to the directory in which the internalCA.conf and the other CA files will be stored.

To generate the long-term CA key and random number source, create a directory called private:

$ mkdir ~/keysafe5-install/internalCA/private

Then run:

$ openssl genrsa -out ~/keysafe5-install/internalCA/private/cakey.pem 4096
$ openssl rand -out ~/keysafe5-install/internalCA/private/.rand 1024

The CA needs a self-signed certificate; as this is a short-term demo it will be valid for 90 days:

$ openssl req -x509 -new -nodes \
  -key internalCA/private/cakey.pem \
  -subj "/CN=internalCA" -days 90 \
  -out internalCA/cacert.pem \
  -config internalCA/internalCA.conf
$ cp internalCA/cacert.pem ca.crt

And finally, to finish off the configuration:

$ mkdir internalCA/newcerts
$ echo 01 > internalCA/serial
$ touch internalCA/index.txt

Install and set up the supporting software

Kubernetes namespace

Create a namespace in Kubernetes for KeySafe 5 installation.

$ kubectl create namespace nshieldkeysafe5

Istio

The version of Istio installed will align with the software version of istioctl.

$ istioctl install -y

RabbitMQ

Entrust recommends that you use your standard secure RabbitMQ installation, along with your policies for authentication and virtual hosts on your production system; this is only a demo system.

First, you must generate the TLS keys and guest password. You must add the network addresses through which RabbitMQ will be accessed to the certificate, and are very dependent on the configuration of the Kubernetes cluster.

$ openssl genrsa -out ~/keysafe5-install/rabbit.key 4096
$ export DNS1="*.rabbit-chart-rabbitmq-headless.rabbitns.svc.cluster.local"
$ export DNS2=rabbit-chart-rabbitmq.rabbitns.svc
$ export DNS3=rabbitmq.rabbitns.svc.cluster.local
$ export DNS4=host.docker.internal
$ export LOCALIP=127.0.0.1
$ export HOSTIP=$(hostname -I | cut -f1 -d" ")
$ openssl req -new -key ~/keysafe5-install/rabbit.key \
  -out ~/keysafe5-install/rabbitmq.csr -subj \
  "/CN=rabbitmq/C=GB/L=Cambridge" \
  -addext "keyUsage=digitalSignature" \
  -addext "extendedKeyUsage=serverAuth" \
  -addext "subjectAltName=DNS:rabbitmq,DNS:${DNS1},DNS:${DNS2},DNS:${DNS3},DNS:${DNS4},DNS:${HOSTNAME},IP:${LOCALIP},IP:${HOSTIP}"
$ openssl ca -config ~/keysafe5-install/internalCA/internalCA.conf \
  -out rabbit.crt \
  -in rabbitmq.csr -batch
$ rm rabbitmq.csr

Now create the kubernetes secrets for the RabbitMQ service:

$ kubectl create namespace rabbitns
$ kubectl create secret generic rabbitmq-certificates \
  --namespace=rabbitns \
  --from-file=ca.crt \
  --from-file=tls.crt=rabbit.crt \
  --from-file=tls.key=rabbit.key
$ kubectl -n rabbitns create secret generic rabbitmq-pw \
  --from-literal=rabbitmq-password=guest

Then install RabbitMQ.

$ helm repo add bitnami https://charts.bitnami.com/bitnami && helm repo update
$ helm install rabbit-chart \
  --namespace=rabbitns \
  --set image.tag=3.11.19-debian-11-r18 \
  --set auth.username=guest \
  --set auth.existingPasswordSecret=rabbitmq-pw \
  --set auth.tls.enabled=true \
  --set auth.tls.existingSecret=rabbitmq-certificates \
  --set replicaCount=2 \
  --set service.type=LoadBalancer \
  --set extraConfiguration='
    listeners.ssl.default = 5671
    ssl_options.versions.1 = tlsv1.3
    ssl_options.depth=0
    ssl_options.verify = verify_peer
    ssl_options.fail_if_no_peer_cert = true
    auth_mechanisms.1 = EXTERNAL
    ssl_cert_login_from = subject_alternative_name
    ssl_cert_login_san_type = dns
    ssl_cert_login_san_index = 0' \
  --set plugins="" \
  --set extraPlugins="rabbitmq_auth_mechanism_ssl" \
  --wait --timeout 10m  bitnami/rabbitmq --version 11.16.2

As we have the service.type set to LoadBalancer we use the IP address of this machine for AMQP connections. We have also added ${HOSTNAME} to the certificate’s subjectAltName as a DNS entry so it may also be used instead of the IP address.

$ export RABBIT_URL=${HOSTIP}:5671

Add the virtual host that will be used for KeySafe 5 communication.

$ export RUN_RABBIT="kubectl -n rabbitns exec rabbit-chart-rabbitmq-0 -c rabbitmq -- "
$ export RABBIT_VHOST=nshieldvhost
$ ${RUN_RABBIT} rabbitmqctl add_vhost ${RABBIT_VHOST}

Then add and configure the X.509 user for the KeySafe 5 application to communicate with RabbitMQ.

$ export KS5_USER=ks5
$ ${RUN_RABBIT} rabbitmqctl add_user $KS5_USER "ephemeralpw"
$ ${RUN_RABBIT} rabbitmqctl set_permissions -p $RABBIT_VHOST $KS5_USER ".*" ".*" ".*"
$ ${RUN_RABBIT} rabbitmqctl clear_password $KS5_USER

You should then create the X.509 key and certificate for this user.

openssl genrsa -out $KS5_USER.key 4096
openssl req -new -key $KS5_USER.key -out $KS5_USER.csr \
  -subj "/CN=${KS5_USER}/C=GB/L=Cambridge" \
  -addext "keyUsage=digitalSignature" \
  -addext "extendedKeyUsage=clientAuth" \
  -addext "subjectAltName=DNS:${KS5_USER}"
openssl ca -config ~/keysafe5-install/internalCA/internalCA.conf \
  -out ${KS5_USER}.crt -in ${KS5_USER}.csr -batch
rm ${KS5_USER}.csr
$ kubectl create secret generic ks5-amqptls \
  --namespace nshieldkeysafe5 \
  --from-file=ca.crt \
  --from-file=tls.crt=ks5.crt \
  --from-file=tls.key=ks5.key

Now remove access for the default guest user.

$ ${RUN_RABBIT} rabbitmqctl delete_user guest

MongoDB

Entrust recommends that you use your standard secure MongoDB Replica Set installation. This is just an example, and not production-ready.

$ kubectl create namespace mongons
$ helm install mongo-chart \
  --set image.tag=5.0.19-debian-11-r3 \
  --set architecture=replicaset \
  --set auth.enabled=true \
  --set auth.username=dummyuser \
  --set auth.password=dummypassword \
  --set auth.database=authdb \
  --set tls.enabled=true \
  --namespace=mongons \
  --wait --timeout 10m  bitnami/mongodb --version 12.1.31

There will be a message listing the MongoDB server addresses. Save the addresses to environment variables for use later.

$ export MONGO1=mongo-chart-mongodb-0.mongo-chart-mongodb-headless.mongons.svc.cluster.local:27017
$ export MONGO2=mongo-chart-mongodb-1.mongo-chart-mongodb-headless.mongons.svc.cluster.local:27017
$ export MONGODB=${MONGO1},${MONGO2}

We then pick up the secrets in the MongoDB configuration.

$ kubectl get secret --namespace mongons mongo-chart-mongodb-ca \
  -o jsonpath="{.data.client-pem}" | base64 --decode | \
  openssl pkey -out mongo-client-key.pem
$ kubectl get secret --namespace mongons mongo-chart-mongodb-ca \
  -o jsonpath="{.data.client-pem}" | base64 --decode | \
  openssl x509 -out mongo-client-cert.pem
$ kubectl get secret --namespace mongons mongo-chart-mongodb-ca \
  -o jsonpath="{.data.mongodb-ca-cert}" | base64 --decode > mongo-ca-cert.pem

We add those secrets in a format that KeySafe 5 can accept.

$ kubectl create secret generic mongodb-client-tls \
  --namespace=nshieldkeysafe5 \
  --from-file=ca.crt=mongo-ca-cert.pem \
  --from-file=tls.crt=mongo-client-cert.pem \
  --from-file=tls.key=mongo-client-key.pem
$ rm mongo-ca-cert.pem mongo-client-key.pem

Access the MongoDB shell to create roles with minimal required permissions on the codesafe-mgmt-db, hsm-mgmt-db and sw-mgmt-db collections, followed by creating a user with these roles. Note that the username needs to match the subject of the client certificate, as found by the following command.

$ openssl x509 -in mongo-client-cert.pem -subject | head -n 1

In this example, mongo-chart-mongodb.mongons.svc.cluster.local is used.

Run the Mongo client container.

$ export MONGO_RUN="kubectl -n mongons exec mongo-chart-mongodb-0 -c mongodb -- "
$ export TLS_PRIVKEY="$(${MONGO_RUN} bash -c 'cat /certs/mongodb.pem')"
$ export TLS_CERT="$(${MONGO_RUN} bash -c 'cat /certs/mongodb-ca-cert')"
$ export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace mongons \
  mongo-chart-mongodb -o jsonpath="{.data.mongodb-root-password}" \
  | base64 --decode)
$ kubectl run --namespace mongons mongo-chart-mongodb-client \
  --rm --tty -i --restart='Never' --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" \
  --env="TLS_PRIVKEY=$TLS_PRIVKEY" --env="TLS_CERT=$TLS_CERT" --env="MONGODB=$MONGODB" \
  --image bitnami/mongodb:5.0.19-debian-11-r3 --command -- bash

Once inside the Mongo client container, you must set up a connection to the server before starting mongo admin and creating the roles and user. After the user is created, exit Mongo admin and the Mongo client container.

$ echo "$TLS_CERT" > /tmp/tls.crt
$ echo "$TLS_PRIVKEY" > /tmp/tls.key
$ mongo admin --tls --tlsCAFile /tmp/tls.crt --tlsCertificateKeyFile /tmp/tls.key \
  --host $MONGODB --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD

> use admin
> db.createRole(
  {
    role: "hsm-mgmt-db-user",
    privileges: [
        {
          "resource": {"db": "hsm-mgmt-db", "collection": ""},
          "actions": ["createIndex", "find", "insert", "remove", "update"]
        },
      ],
    roles: []
  }
)
> db.createRole(
  {
    role: "sw-mgmt-db-user",
    privileges: [
        {
          "resource": {"db": "sw-mgmt-db", "collection": ""},
          "actions": ["createIndex", "dropCollection", "find", "insert", "remove", "update"]
        },
      ],
    roles: []
  }
)
> db.createRole(
  {
    role: "codesafe-mgmt-db-user",
    privileges: [
        {
          "resource": {"db": "codesafe-mgmt-db", "collection": ""},
          "actions": ["createIndex", "find", "insert", "remove", "update"]
        },
      ],
    roles: []
  }
)
> use $external
> x509_user = {
   "user" : "CN=mongo-chart-mongodb.mongons.svc.cluster.local",
   "roles" : [
     {"role": "codesafe-mgmt-db-user", "db": "admin" },
     {"role": "hsm-mgmt-db-user", "db": "admin" },
     {"role": "sw-mgmt-db-user", "db": "admin" },
   ]
 }
> db.createUser(x509_user)
> exit
$ exit

Object Storage

For large object storage, create a Persistent Volume Claim, in the nshieldkeysafe5 Kubernetes namespace (the same namespace that we will deploy the application to).

Cluster-local Object Storage

If your Kubernetes cluster only has 1 worker node, you can choose to use local storage.

$ cat << EOF | kubectl -n nshieldkeysafe5 apply -f -
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: data-nshield-keysafe5
  spec:
    accessModes:
      - ReadWriteOnce
    storageClassName: local-path
    resources:
      requests:
        storage: 2Gi
EOF

NFS Object Storage

If your Kubernetes cluster has more than 1 worker node, you must use a type of storage that supports distributed access, such as NFS. For details on creating a PVC for NFS object storage, please see NFS Object Storage Configuration.

Install KeySafe 5

# Get Ingress IP address
$ export INGRESS_IP=$(kubectl --namespace istio-system get svc -l app=istio-ingressgateway -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

# Install the KeySafe 5 backend services
$ helm install keysafe5-backend \
  --namespace=nshieldkeysafe5 \
  --set codesafe_mgmt.image=$DOCKER_REGISTRY/keysafe5/codesafe-mgmt:1.2.0 \
  --set hsm_mgmt.image=$DOCKER_REGISTRY/keysafe5/hsm-mgmt:1.2.0 \
  --set sw_mgmt.image=$DOCKER_REGISTRY/keysafe5/sw-mgmt:1.2.0 \
  --set database.type=mongo \
  --set database.mongo.hosts="$MONGO1\,$MONGO2" \
  --set database.mongo.replicaSet=rs0 \
  --set database.mongo.auth.type=tls \
  --set database.mongo.auth.authDatabase=authdb \
  --set database.mongo.tls.enabled=true \
  --set database.mongo.tls.existingSecret=mongodb-client-tls \
  --set amqp.URL=${RABBIT_URL}/${RABBIT_VHOST} \
  --set amqp.auth.type=tls \
  --set amqp.tls.enabled=true \
  --set amqp.tls.existingSecret=ks5-amqptls \
  --set objectStore.pvc=data-nshield-keysafe5 \
  --wait --timeout 10m \
 helm-charts/nshield-keysafe5-backend-1.2.0.tgz

# Install the KeySafe 5 UI
$ helm install keysafe5-ui \
  --namespace=nshieldkeysafe5 \
  --set ui.image=$DOCKER_REGISTRY/keysafe5/mgmt-ui:1.2.0 \
  --set svcEndpoint="https://${INGRESS_IP}" \
  --set authMethod=none \
  --wait --timeout 10m \
helm-charts/nshield-keysafe5-ui-1.2.0.tgz

# Create the TLS secret for the Istio Ingress Gateway
$ openssl genrsa -out istio.key 4096
$ openssl req -new -key istio.key -out istio.csr \
  -subj "/CN=${HOSTNAME}" \
  -addext "keyUsage=digitalSignature" \
  -addext "extendedKeyUsage=serverAuth" \
  -addext "subjectAltName=DNS:${HOSTNAME},IP:${INGRESS_IP}"
$ openssl ca -config ~/keysafe5-install/internalCA/internalCA.conf \
  -out istio.crt -in istio.csr -batch
$ kubectl -n istio-system create secret tls \
  keysafe5-server-credential --cert=istio.crt --key=istio.key

# Configure Istio Ingress Gateway for KeySafe 5
$ helm install keysafe5-istio \
  --namespace=nshieldkeysafe5 \
  --set tls.existingSecret=keysafe5-server-credential \
  --set requireAuthn=false \
  --wait --timeout 1m \
helm-charts/nshield-keysafe5-istio-1.2.0.tgz

Access KeySafe 5

You can now access KeySafe 5 at https://$INGRESS_IP.

For example, you could send curl requests as demonstrated below.

$ curl -X GET --cacert ca.crt https://${INGRESS_IP}/mgmt/v1/hsms | jq
$ curl -X GET --cacert ca.crt https://${INGRESS_IP}/mgmt/v1/hosts | jq
$ curl -X GET --cacert ca.crt https://${INGRESS_IP}/mgmt/v1/pools | jq
$ curl -X GET --cacert ca.crt https://${INGRESS_IP}/mgmt/v1/feature-certificates | jq
$ curl -X GET --cacert ca.crt https://${INGRESS_IP}/mgmt/v1/worlds | jq
$ curl -X GET --cacert ca.crt https://${INGRESS_IP}/codesafe/v1/images | jq
$ curl -X GET --cacert ca.crt https://${INGRESS_IP}/codesafe/v1/certificates | jq

You can access the Management UI in a web browser at https://$INGRESS_IP.

Configure nShield client machines

To configure a host machine to be managed and monitored by this deployment, run the KeySafe 5 agent binary on the nShield client machine containing the relevant Security World or HSMs.

Configure this KeySafe 5 agent to communicate with the RabbitMQ server installed previously.

Ensure no firewall rules are blocking the AMQP port communication between the machine exposing the AMQP port from Kubernetes and the machine running the agent.

$ sudo tar -xf ~/keysafe5-install/keysafe5-agent/keysafe5-1.2.0-Linux-keysafe5-agent.tar.gz -C /
$ export KS5CONF=/opt/nfast/keysafe5/conf
$ sudo cp $KS5CONF/config.yaml.example $KS5CONF/config.yaml
$ sudo sed -i "s|^  url: 127.0.0.1:5671|  url: ${INGRESS_IP}:5671/${RABBIT_VHOST}|g" /opt/nfast/keysafe5/conf/config.yaml

Create the amqp/tls configuration directory and copy the ca.crt file created in the keysafe5-install directory on the deployment machine.

$ sudo mkdir -p $KS5CONF/amqp/tls
$ sudo cp ca.crt $KS5CONF/amqp/tls/ca.crt

Create the private key and a certificate signing request (CSR) for this specific KeySafe 5 agent.

$ sudo /opt/nfast/keysafe5/bin/amqptls -keypath=/opt/nfast/keysafe5/conf/amqp/tls/tls.key -keygen
$ sudo /opt/nfast/keysafe5/bin/amqptls -keypath=/opt/nfast/keysafe5/conf/amqp/tls/tls.key -csrgen

The CSR should be provided to a KeySafe 5 administrator who, in a secure location/environment, creates a RabbitMQ service client TLS certificate using the CA trusted by the RabbitMQ server.

On the machine that you created the Demo Certificate Authority on, use the agentcert.sh script to create a client TLS certificate for this KeySafe 5 agent using the CSR and the demo CA.

$ sudo chmod +r ks5agent_demohost.csr
$ ./agentcert.sh ks5agent_demohost.csr 365

Using the username printed in the output of the previous command, configure the RabbitMQ server to allow access for this X.509 user in the appropriate virtual host.

$ export x509user=ks5agent_demohost
$ export RUN_RABBIT="kubectl -n rabbitns exec rabbit-chart-rabbitmq-0 -c rabbitmq -- "
$ ${RUN_RABBIT} rabbitmqctl add_user $x509user "ephemeralpw"
$ ${RUN_RABBIT} rabbitmqctl set_permissions -p $RABBIT_VHOST $x509user ".*" ".*" ".*"
$ ${RUN_RABBIT} rabbitmqctl clear_password $x509user

Transfer the resulting certificate ks5agent_demohost.crt to the nShield client machine at /opt/nfast/keysafe5/conf/amqp/tls/tls.crt.

On the nShield client machine, if the hardserver is already running, use the KeySafe 5 install script to not restart it when installing the KeySafe 5 agent.

$ sudo /opt/nfast/keysafe5/sbin/install

Otherwise, use the nShield install script which will start both the nShield Security World software and the KeySafe 5 agent.

$ sudo /opt/nfast/sbin/install

Uninstall

helm --namespace nshieldkeysafe5 uninstall keysafe5-istio
helm --namespace nshieldkeysafe5 uninstall keysafe5-backend
helm --namespace nshieldkeysafe5 uninstall keysafe5-ui
helm --namespace rabbitns uninstall rabbit-chart
helm --namespace mongons uninstall mongo-chart

To uninstall the KeySafe 5 agent, run the KeySafe 5 uninstaller.

$ sudo /opt/nfast/keysafe5/sbin/install -u