Upgrade

This chapter details how to update an existing KeySafe 5 install to the latest version.

When upgrading KeySafe 5 it is recommended to first update the Helm charts installed in the central platform and then update all KeySafe 5 agent installs on host machines being managed by KeySafe 5.

Ensure all pods are healthy prior to performing an upgrade, unhealthy pods can prevent helm from fully completing an upgrade.

Helm Chart Upgrade

To upgrade the release of a Helm Chart we do a helm upgrade command, see Helm Upgrade

List all installed releases using helm list -A.

$ helm list -A
NAME            	NAMESPACE	    REVISION	UPDATED                                	STATUS  	CHART                         	APP VERSION
keysafe5-backend	nshieldkeysafe5	1       	2023-07-03 12:23:50.419332325 +0100 BST	deployed	nshield-keysafe5-backend-1.1.0	1.1.0
keysafe5-istio  	nshieldkeysafe5	1       	2023-07-03 12:24:29.852443337 +0100 BST	deployed	nshield-keysafe5-istio-1.1.0  	1.1.0
mongo-chart     	mongons        	1       	2023-07-03 12:21:42.118470777 +0100 BST	deployed	mongodb-12.1.31               	5.0.19
rabbit-chart    	rabbitns       	1       	2023-07-03 12:21:44.58324842 +0100 BST 	deployed	rabbitmq-11.1.2               	3.11.19

To upgrade or modify the existing my-release deployment:

  1. Obtain the current configuration of the currently installed helm chart release.

  2. Run a helm upgrade command, overriding any existing configuration values using the --set directive.

The example shown below is for the helm-keysafe5-backend Chart but the same process applies also to helm-keysafe5-ui and helm-keysafe5-istio.

Documentation for each configurable value in the KeySafe 5 Helm charts can be found by untarring the chart.tgz and viewing the contents of either README.md or the default values.yaml file.

 
$ helm -n nshieldkeysafe5 get values --all --output yaml keysafe5-backend > keysafe5-backend-values.yaml

$ helm upgrade --install keysafe5-backend \
  --namespace=nshieldkeysafe5 \
  --values keysafe5-backend-values.yaml \
  --set example.ConfigKey=updatedValue \
  helm-keysafe5-backend/

List all installed releases using helm list to see the upgraded Helm chart.

$ helm list -A
NAME            	NAMESPACE	    REVISION	UPDATED                                	STATUS  	CHART                         	APP VERSION
keysafe5-backend	nshieldkeysafe5	2       	2023-07-03 14:17:49.456099223 +0100 BST	deployed	nshield-keysafe5-backend-1.2.0	1.2.0
keysafe5-istio  	nshieldkeysafe5	1       	2023-07-03 12:24:29.852443337 +0100 BST	deployed	nshield-keysafe5-istio-1.1.0  	1.1.0
mongo-chart     	mongons        	1       	2023-07-03 12:21:42.118470777 +0100 BST	deployed	mongodb-12.1.31               	5.0.19
rabbit-chart    	rabbitns       	1       	2023-07-03 12:21:44.58324842 +0100 BST 	deployed	rabbitmq-11.1.2               	3.11.19

To view the release history for a Helm Chart using the helm history command.

$ helm history -n nshieldkeysafe5 keysafe5-backend
REVISION	UPDATED                 	STATUS    	CHART                         	APP VERSION	DESCRIPTION
1       	Mon Jul  3 12:23:50 2023	superseded	nshield-keysafe5-backend-1.1.0	1.1.0      	Install complete
2       	Mon Jul  3 14:17:49 2023	deployed  	nshield-keysafe5-backend-1.2.0	1.2.0      	Upgrade complete

To revert to a previous installed release, use the helm rollback command.

$ helm -n nshieldkeysafe5 rollback keysafe5-backend 1

$ helm history -n nshieldkeysafe5 keysafe5-backend
REVISION	UPDATED                 	STATUS    	CHART                         	APP VERSION	DESCRIPTION
1       	Mon Jul  3 12:23:50 2023	superseded	nshield-keysafe5-backend-1.1.0	1.1.0      	Install complete
2       	Mon Jul  3 14:17:49 2023	superseded	nshield-keysafe5-backend-1.2.0	1.2.0      	Upgrade complete
3       	Mon Jul  3 14:22:03 2023	deployed  	nshield-keysafe5-backend-1.1.0	1.1.0      	Rollback to 1

Upgrade from KeySafe 5 1.1

To upgrade from KeySafe 5 1.1, see the following example commands.

KeySafe 5 1.2 requires the addition of a new database and additional database user configuration. It also adds a requirement for a persistent object storage. For more details, see MongoDB User Configuration and Object Storage.
The paths to Docker images should be updated to point to where you have stored the shipped Docker images. See Docker Images. The Helm chart directories pointed to when upgrading should be the directory for the Helm chart version that you are upgrading to. 
helm -n nshieldkeysafe5 get values --all --output yaml keysafe5-backend > keysafe5-backend-values.yaml
helm upgrade --install keysafe5-backend \
  --namespace=nshieldkeysafe5 \
  --values keysafe5-backend-values.yaml \
  --set hsm_mgmt.image=private.registry.local/keysafe5/hsm-mgmt:1.2.0 \
  --set hsm_mgmt.pullPolicy=Always \
  --set sw_mgmt.image=private.registry.local/keysafe5/sw-mgmt:1.2.0 \
  --set sw_mgmt.pullPolicy=Always \
  --set codesafe_mgmt.image=private.registry.local/keysafe5/codesafe-mgmt:1.2.0 \
  --set codesafe_mgmt.pullPolicy=Always \
  --set objectStore.pvc=data-nshield-keysafe5 \
  helm-keysafe5-backend/

helm -n nshieldkeysafe5 get values --all --output yaml keysafe5-ui > keysafe5-ui-values.yaml
helm upgrade --install keysafe5-ui \
  --namespace=nshieldkeysafe5 \
  --values keysafe5-ui-values.yaml \
  --set ui.image=private.registry.local/keysafe5/mgmt-ui:1.2.0 \
  --set ui.pullPolicy=Always \
  helm-keysafe5-ui/

helm -n nshieldkeysafe5 get values --all --output yaml keysafe5-istio > keysafe5-istio-values.yaml
helm upgrade --install keysafe5-istio \
  --namespace=nshieldkeysafe5 \
  --values keysafe5-istio-values.yaml \
  helm-keysafe5-istio/

Agent Upgrade

To update the KeySafe 5 agent installed on a machine:

  • Take a backup of the agent config directory located at %NFAST_DATA_HOME%/keysafe5/conf.

  • Uninstall the existing KeySafe 5 agent as detailed in the KeySafe 5 Installation Guide for the currently installed version of the product.

  • Install the new KeySafe 5 agent as detailed in chapter KeySafe 5 Agent Installation.

  • Restore existing agent configuration and restart the agent.

Upgrading supporting software

Upgrade from KeySafe 5 1.1 recommended versions

KeySafe 5 1.1 recommended RabbitMQ 3.11.3 and MongoDB 5.0.10. This section details how to upgrade the software from these versions to the latest recommended compatible versions.

MongoDB 5.0.10 to 5.0.19

To update a non-Kubernetes existing Mongo install to a Mongo 5.0.19 install, see the official documentation at Upgrade to the Latest Revision of MongoDB.

To update a Mongo 5.0.10 install deployed via Bitnami Helm Charts:

# Obtain details of currently deployed helm charts
# Substitute chart and namespace values in the commands below as required
helm list -A

# Fetch current MongoDB helm chart deployed values
helm -n mongons get values --all --output yaml mongo-chart > mongo-chart-values.yaml

# Obtain the names of the existing mongo secrets
# Substitute secret names in the commands below as required
kubectl get secrets -n mongons

# Make copies of the existing secrets, this is required as the existing ones will be removed during the upgrade process
kubectl get secret mongodb-server-certificates -n=mongons -o yaml \
  | sed 's/mongodb-server-certificates/mongodb-server-certificates-upgrade/' \
  | kubectl apply -f -
kubectl get secret mongo-chart-mongodb -n=mongons -o yaml \
  | sed 's/mongo-chart-mongodb/mongo-chart-mongodb-upgrade/' \
  | kubectl apply -f -

# Upgrade helm chart based on existing deployed values
helm upgrade --install mongo-chart \
    --namespace=mongons \
    --values mongo-chart-values.yaml \
    --set image.tag=5.0.19-debian-11-r3 \
    --set auth.existingSecret=mongo-chart-mongodb-upgrade \
    --set tls.autoGenerated=false \
    --set tls.existingSecret=mongodb-server-certificates-upgrade\
    --wait --timeout 5m \
    bitnami/mongodb --version 12.1.31

# Obtain details of newly deployed helm charts
helm list -A

The helm listing for mongo-chart may fail to update app version from 5.0.10 to 5.0.19. This bug is only visual and the image will have updated successfully. This can be confirmed by using kubectl.

MongoDB User Configuration

To configure the MongoDB database correctly, access the MongoDB shell to create a user with the minimal permissions required for using the codesafe-mgmt-db, hsm-mgmt-db and sw-mgmt-db databases. See Database: User Roles.

Note that the username needs to match the subject of the client certificate, as found by the following command.

$ kubectl get secret --namespace nshieldkeysafe5 mongodb-client-certificates \
  -o jsonpath="{.data.tls\.crt}" | base64 --decode | \
  openssl x509 -out mongo-client-cert.pem
$ openssl x509 -in mongo-client-cert.pem -subject | head -n 1

In this example we will use ks5-mongo-user

We then run the mongo client container.

$ export MONGO1=mongo-chart-mongodb-0.mongo-chart-mongodb-headless.mongons.svc.cluster.local:27017
$ export MONGO2=mongo-chart-mongodb-1.mongo-chart-mongodb-headless.mongons.svc.cluster.local:27017
$ export MONGODB=${MONGO1},${MONGO2}
$ export MONGO_RUN="kubectl -n mongons exec mongo-chart-mongodb-0 0 -- "
$ export TLS_PRIVKEY="$(${MONGO_RUN} bash -c 'cat /certs/mongodb.pem')"
$ export TLS_CERT="$(${MONGO_RUN} bash -c 'cat /certs/mongodb-ca-cert')"
$ export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace mongons \
  mongo-chart-mongodb-upgrade -o jsonpath="{.data.mongodb-root-password}" \
  | base64 --decode)
$ kubectl run --namespace mongons mongo-chart-mongodb-client \
  --rm --tty -i --restart='Never' --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" \
  --env="TLS_PRIVKEY=$TLS_PRIVKEY" --env="TLS_CERT=$TLS_CERT" --env="MONGODB=$MONGODB" \
  --image bitnami/mongodb:5.0.19-debian-11-r3 --command -- bash

Once inside the mongo client container, we need to set up a connection to the server before we can start mongo admin and create the user. After we finish creating the user, we need to exit mongo admin, and then the mongo-client container.

$ echo "$TLS_CERT" > /tmp/tls.crt
$ echo "$TLS_PRIVKEY" > /tmp/tls.key
$ mongo admin --tls --tlsCAFile /tmp/tls.crt --tlsCertificateKeyFile /tmp/tls.key \
  --host $MONGODB --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD

> use admin
> db.createRole(
  {
    role: "hsm-mgmt-db-user",
    privileges: [
        {
          "resource": {"db": "hsm-mgmt-db", "collection": ""},
          "actions": ["createIndex", "find", "insert", "remove", "update"]
        },
      ],
    roles: []
  }
)
> db.createRole(
  {
    role: "sw-mgmt-db-user",
    privileges: [
        {
          "resource": {"db": "sw-mgmt-db", "collection": ""},
          "actions": ["createIndex", "dropCollection", "find", "insert", "remove", "update"]
        },
      ],
    roles: []
  }
)
> db.createRole(
  {
    role: "codesafe-mgmt-db-user",
    privileges: [
        {
          "resource": {"db": "codesafe-mgmt-db", "collection": ""},
          "actions": ["createIndex", "find", "insert", "remove", "update"]
        },
      ],
    roles: []
  }
)
> use $external
> x509_user = {
   "roles" : [
     {"role": "codesafe-mgmt-db-user", "db": "admin" },
     {"role": "hsm-mgmt-db-user", "db": "admin" },
     {"role": "sw-mgmt-db-user", "db": "admin" },
   ]
 }
> db.updateUser("CN=ks5-mongo-user", x509_user)
> exit
$ exit

RabbitMQ 3.11.3 to 3.11.19

To update a non-Kubernetes existing RabbitMQ install, see the official documentation at Upgrading RabbitMQ.

To update a RabbitMQ 3.11.3 install deployed via Bitnami Helm Charts:

# Obtain details of currently deployed helm charts
# Substitute chart and namespace values in the commands below as required
helm list -A

# Fetch current rabbitmq helm chart deployed values
helm -n rabbitns get values --all --output yaml rabbit-chart > rabbit-chart-values.yaml

# Upgrade helm chart based on existing deployed values
helm upgrade --install rabbit-chart \
  --namespace=rabbitns \
  --values rabbit-chart-values.yaml \
  --set image.tag=3.11.19-debian-11-r18 \
  --set service.type=LoadBalancer \
  --set extraPlugins="rabbitmq_auth_mechanism_ssl" \
  --wait --timeout 10m \
  bitnami/rabbitmq --version 11.16.2

# Obtain details of newly deployed helm charts
helm list -A

Istio

The version of Istio installed aligns with the software version of istioctl.

For details of how to upgrade Istio, see Upgrade Istio.

Object Storage

Upgrading to KeySafe 5 1.2 requires a Persistent Volume Claim in the nshieldkeysafe5 Kubernetes namespace.

For details on how to create a PVC, see Object Storage.