Upgrade

This chapter details how to update an existing KeySafe 5 install to the latest version.

If the environment that is being upgraded contains a WSOP3.x instance, it must be upgraded to at least WSOP v3.3.0 prior to upgrading KeySafe 5.

When upgrading KeySafe 5 it is recommended to first update the Helm charts installed in the central platform and then update all KeySafe 5 Agent installs on host machines being managed by KeySafe 5.

Upgrading from KeySafe 5 1.2

To upgrade the release of a Helm Chart we do a helm upgrade command, see Helm Upgrade

List all installed releases using helm list -A.

$ helm list -A
NAME            	NAMESPACE      	REVISION	UPDATED                                	STATUS  	CHART                         	APP VERSION
keysafe5-backend	nshieldkeysafe5	1       	2024-04-25 15:59:40.995994525 +0100 BST	deployed	nshield-keysafe5-backend-1.2.0	1.2.0
keysafe5-istio  	nshieldkeysafe5	1       	2024-04-25 15:58:09.344300669 +0100 BST	deployed	nshield-keysafe5-istio-1.2.0  	1.2.0
keysafe5-ui     	nshieldkeysafe5	1       	2024-04-25 15:57:42.260802671 +0100 BST	deployed	nshield-keysafe5-ui-1.2.0     	1.2.0
mongo-chart     	mongons        	1       	2024-04-25 15:55:05.825098514 +0100 BST	deployed	mongodb-12.1.31               	5.0.10
rabbit-chart    	rabbitns       	1       	2024-04-25 15:58:19.881365343 +0100 BST	deployed	rabbitmq-11.16.2              	3.11.18

Ensure all pods are healthy prior to performing an upgrade, unhealthy pods can prevent helm from fully completing an upgrade.

The process involves upgrading the charts in the following order:

keysafe5-backend
keysafe5-ui
keysafe5-istio
mongo-chart
rabbit-chart

Unpack the source

mkdir ~/keysafe5-upgrade
tar -C ~/keysafe5-upgrade -xf nshield-keysafe5-1.3.0.tar.gz
cd ~/keysafe5-upgrade

Docker Images

The Docker images need to be loaded onto a Docker registry that each node in your Kubernetes cluster can pull the images from. See Docker Images for more details.

Upgrading the KeySafe 5 backend

The parameters used for the running Helm chart need to be retrieved into a file called keysafe5-backed-values.yaml.

helm -n nshieldkeysafe5 get values --all --output yaml keysafe5-backend > keysafe5-backend-values.yaml

This file will need updating with the new KeySafe 5 1.3 settings. To rename the amqp block to messageBus you may use sed:

sed -i -es'/^amqp/messageBus/' keysafe5-backend-values.yaml

If there are other changes you wish to make, you may do so.

helm upgrade --install keysafe5-backend \
  --namespace=nshieldkeysafe5 \
  --wait --timeout 3m \
  --values keysafe5-backend-values.yaml \
  --set hsm_mgmt.image=$DOCKER_REGISTRY/keysafe5/hsm-mgmt:1.3.0 \
  --set sw_mgmt.image=$DOCKER_REGISTRY/keysafe5/sw-mgmt:1.3.0 \
  --set codesafe_mgmt.image=$DOCKER_REGISTRY/keysafe5/codesafe-mgmt:1.3.0 \
  helm-charts/nshield-keysafe5-backend-1.3.0.tgz

Upgrading the KeySafe 5 UI, and Istio

The same process as for the backend is also used for the UI, and Istio:

helm -n nshieldkeysafe5 get values --all --output yaml keysafe5-ui > keysafe5-ui-values.yaml
helm -n nshieldkeysafe5 get values --all --output yaml keysafe5-istio > keysafe5-istio-values.yaml

You may make changes to the yaml files before upgrading though this is not required.

helm upgrade --install keysafe5-ui \
  --namespace=nshieldkeysafe5 \
  --wait --timeout 3m \
  --values keysafe5-ui-values.yaml \
  --set ui.image=$DOCKER_REGISTRY/keysafe5/mgmt-ui:1.3.0 \
  --set ui.pullPolicy=Always \
  helm-charts/nshield-keysafe5-ui-1.3.0.tgz

helm upgrade --install keysafe5-istio \
  --namespace=nshieldkeysafe5 \
  --wait --timeout 3m \
  --values keysafe5-istio-values.yaml \
  helm-charts/nshield-keysafe5-istio-1.3.0.tgz

Agent Upgrade

To update the KeySafe 5 Agent installed on a machine:

Take a backup of the Agent config directory located at %NFAST_DATA_HOME%/keysafe5/conf.
Uninstall the existing KeySafe 5 Agent as detailed in the KeySafe 5 Installation Guide for the currently installed version of the product.
Install the new KeySafe 5 Agent as detailed in chapter KeySafe 5 Agent Installation.

The configuration between the KeySafe 5 Agent v1.2 and v1.3 has changed. You may either configure the Agent afresh, or use the new example configuration file to create a new configuration file, with values from the old configuration. The existing keys may be reused, but note the new directory structure.

Upgrading supporting software

KeySafe 5 1.2 recommended RabbitMQ 3.11.19 and MongoDB 5.0.19. This section details how to upgrade the software from these versions to the latest recommended compatible versions.

MongoDB 5.0.19 to 7.0.7

Upgrading from MongoDB 5.0 to MongoDB 7.0 requires an interim upgrade to version 6.

To update a non-Kubernetes existing MongoDB install to a MongoDB 7.0 install, see the official documentation at Upgrade to the Latest Revision of MongoDB.

To update a MongoDB 5.0.19 install deployed via Bitnami Helm Charts:

First ensure that MongoDB is running.

helm list -A

We need to fetch the existing helm chart’s settings

helm -n mongons get values --output yaml mongo-chart > mongo-chart-values.yaml

We also need to fetch the root password, and the replicaset key. They need to be passed to the new MongoDB instance, and to run commands on the MongoDB server.

MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace mongons mongo-chart-mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 -d)
MONGODB_REPLICA_SET_KEY=$(kubectl get secret --namespace mongons mongo-chart-mongodb -o jsonpath="{.data.mongodb-replica-set-key}" | base64 -d)

Now we will upgrade the chart, also picking up the existing chart’s settings.

helm upgrade mongo-chart bitnami/mongodb \
  --namespace mongons \
  --set architecture=replicaset \
  --set auth.replicaSetKey="$MONGODB_REPLICA_SET_KEY" \
  --set auth.rootPassword="$MONGODB_ROOT_PASSWORD" \
  --set image.tag=6.0.10-debian-11-r8 \
  --wait --timeout 3m \
  --values mongo-chart-values.yaml \
  --version 13.18.5

This will take about a minute, but once it succeeds the KeySafe 5 backend services will fail, then restart as its database broke communications.

Once kubectl -n nshieldkeysafe5 get pods shows that the KeySafe 5 backend services are running again, it is safe to continue.

We will now set the FeatureCompatibilityVersion to 6.0 to ensure that we can upgrade to MongoDB 7.0.

We create a simple script to allow us to run commands directly on the database. These are different to the ones shown when upgrading MongoDB, but that is because we use TLS.

RUN_MONGO="kubectl -n mongons exec --stdin=true mongo-chart-mongodb-0 0 --"
RUN_MONGO_ADMIN="$RUN_MONGO mongosh admin --tls --tlsCAFile /certs/mongodb-ca-cert --tlsCertificateKeyFile /certs/mongodb.pem --host 127.0.0.1 --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD"

Using this we can now update the FeatureCompatibilityVersion.

$RUN_MONGO_ADMIN --eval 'db.adminCommand({setFeatureCompatibilityVersion: "6.0"})'

Now we may upgrade MongoDB to version 7.0.

Again, fetch the existing helm chart’s settings

helm -n mongons get values --output yaml mongo-chart > mongo-chart-values.yaml

The password and replicaset key have not changed, so we can upgrade.

helm upgrade mongo-chart bitnami/mongodb \
  --namespace mongons \
  --wait --timeout 3m \
  --set image.tag=7.0.7-debian-12-r0 \
  --values mongo-chart-values.yaml \
  --version 15.0.2

After this has finished, wait until the KeySafe 5 backend services are running again.

Obtain details of newly deployed helm charts

helm list -A

And finally for MongoDB, update the FeatureCompatibilityVersion.

$RUN_MONGO_ADMIN --eval 'db.adminCommand({setFeatureCompatibilityVersion: "7.0", confirm: true})'

RabbitMQ 3.11.19 to 3.12.13

To update a non-Kubernetes existing RabbitMQ install, see the official documentation at Upgrading RabbitMQ.

To update a RabbitMQ 3.11.3 install deployed via Bitnami Helm Charts:

# Obtain details of currently deployed helm charts
# Substitute chart and namespace values in the commands below as required
helm list -A

# Fetch current rabbitmq helm chart deployed values
helm -n rabbitns get values --output yaml rabbit-chart > rabbit-chart-values.yaml

# Upgrade helm chart based on existing deployed values
helm upgrade --install rabbit-chart \
  --namespace=rabbitns \
  --values rabbit-chart-values.yaml \
  --set image.tag=3.12.13-debian-12-r1 \
  --set service.type=LoadBalancer \
  --set extraPlugins="rabbitmq_auth_mechanism_ssl" \
  --wait --timeout 10m \
  bitnami/rabbitmq --version 12.13.1

# Obtain details of newly deployed helm charts
helm list -A

Istio

The version of Istio installed aligns with the software version of istioctl.

For details of how to upgrade Istio, see Upgrade Istio.