Procedures

RedHat Linux Server

On a RedHat Linux server do the following:

Install the OpenShift Client tools.
Install docker:
```
% sudo dnf install docker
```

Deploy a KeyControl Vault cluster

For this integration, KeyControl Vault was deployed as a two-node cluster.

Follow the installation and setup instructions in KeyControl Vault Installation and Upgrade Guide.

Create a Secrets Vault to be used with Openshift

Sign in to the KeyControl Vault Manager.
In the home page, select Create Vault.

The Create Vault dialog appears.
In the Type drop-down box, select Secrets. Enter the required information.
Select Create Vault.

For example:
When you receive an email with a URL and sign-in credentials to the KeyControl vault, bookmark the URL and save the credentials.

You can also copy the sign-in credentials when the vault details gets displayed.
Sign in to the URL provided.
Change the initial password when prompted.

Create a secret in the Secrets Vault.

After you sign in to the secrets vault, create a box that will contain the secret.

Select Manage in the Secrets Vault Home tab, then select Manage Boxes.
In the Manage Boxes Tab, select Add a Box Now.
In the Create a Box Window, enter the Name and Description.

In this integration guide and the configuration file examples it contains, the box will be named box1.
Select Continue.
In Checkout Details, select Continue.
In Rotation Details, select Create. The box gets created.
Select the new box.
In the Secrets Pane, select Add a Secret Now.
From the Choose a type of secret to create list, select Text.
In the Create Secret: Text window, enter the following:
1. Name: Enter the name of the secret. This will be used later in configuration files in this integration.
2. Description: Enter a brief description.
3. Secret Data: Enter the value of the secret. This is the value the Openshift containers will try to retrieve during the integration.
Select Create.

Setup and Configuration

Configure the environment for the integration. From this point on, all setup and configuration will be done from the linux server that was set up earlier.

Setup KUBECONFIG

This should give you access to the Openshift cluster.

% export KUBECONFIG=<path-to-your-kubeconfig-file>

Check that you can see the Openshift cluster nodes:

% oc get nodes

NAME                           STATUS   ROLES                  AGE    VERSION
ocp4147-r758r-master-0         Ready    control-plane,master   3d1h   v1.27.8+4fab27b
ocp4147-r758r-master-1         Ready    control-plane,master   3d1h   v1.27.8+4fab27b
ocp4147-r758r-master-2         Ready    control-plane,master   3d1h   v1.27.8+4fab27b
ocp4147-r758r-worker-0-ctl28   Ready    worker                 3d     v1.27.8+4fab27b
ocp4147-r758r-worker-0-t7lfl   Ready    worker                 3d     v1.27.8+4fab27b
ocp4147-r758r-worker-0-vbwrb   Ready    worker                 3d     v1.27.8+4fab27b

Set up namespaces

The integration will use two namespaces in Openshift.

Set up the mutating webhook namespace.
1. Create a yaml file containing the following code:
  apiVersion: v1 kind: Namespace metadata: name: mutatingwebhook
2. Name the file mutatingwebhooknamespace.yaml
3. Create the namespace:
  % oc create -f mutatingwebhooknamespace.yaml
Set up the test namespace.
1. Create a yaml file containing the following code:
  apiVersion: v1 kind: Namespace metadata: name: testnamespace
2. Name the file testnamespace.yaml.
3. Create the namespace:
  % oc create -f testnamespace.yaml

Docker registry

Set up DOCKER_CONFIG

Export DOCKER_CONFIG to the directory where your docker configuration will be stored:
```
% export DOCKER_CONFIG=~/.docker
```

% docker login -u YOURUSERID <registry-url>

Deploy the init container image.

The init container image needs to be deployed to the docker registry.

Download the init container image:

% wget https://github.com/EntrustCorporation/PASM-Vault-Kubernetes-Agent/releases/download/v1.0/init-container.tar

Load the provided image init-container.tar into the docker registry:
```
% docker load --input init-container.tar
```

Check the image:

% docker images

REPOSITORY                            TAG         IMAGE ID      CREATED        SIZE
localhost/init-container              latest      1c476f57b72c  10 months ago  186 MB

Tag the Image:

% docker tag localhost/init-container:latest <registry-url>/init-container

Push the image into the docker registry that is used within Openshift:
```
% docker push <registry-url>/init-container:latest
```

Deploy the mutating webhook image.

The webhook code is in the form of image and needs to be deployed as container. It needs to be deployed to the docker registry.

Download the webhook container image:

% wget https://github.com/EntrustCorporation/PASM-Vault-Kubernetes-Agent/releases/download/v1.0/mutating-webhook.tar

Load the provided image mutating-webhook.tar into the docker registry:
```
% docker load --input mutating-webhook.tar
```

Check the image:

% docker images

REPOSITORY                  TAG         IMAGE ID      CREATED        SIZE
localhost/mutating-webhook  latest      805efa734095  10 months ago  222 MB

Tag the Image:

% docker tag localhost/mutating-webhook:latest <registry-url>/mutating-webhook

Push the image into the docker registry that is used within Openshift:
```
% docker push <registry-url>/mutating-webhook:latest
```

Get the Secrets Vault Authentication token for the vault Admin User.

Log in to the Secrets vault as admin using the KeyControl Vault Rest API and get the vault authentication token.
https://<VAULT-IP>/vault/1.0/Login/<VAULT-UUID>
This URL is visible on vault management UI of the KeyControl. Also note the VAULT-UUID from the URL. This will be required in further steps.

Here is an example:
https://xx.xxx.xxx.xxx/vault/1.0/Login/7e6e8646-20f0-41a5-8bb6-a17dff64aff2
The request body should be in the form:
{ "username":"admin", "password":"password" }
The username should be one of the administrators of the vault. The examples uses admin, but this should be the actual admin user name.

Get the token using the following curl command:

% curl -k -X POST https://xx.xxx.xxx.xxx/vault/1.0/Login/7e6e8646-20f0-41a5-8bb6-a17dff64aff2/ -d '{ "username": "admin", "password":"xxxxxxx" }'

The response body for the command above should be similar to this:

{
  "is_user": false,
  "box_admin": false,
  "appliance_id": "bc1a085c-70c9-476c-b4f6-abff0c73cce4",
  "is_secondary_approver": false,
  "access_token": "N2U2ZTg2NDYtMjBmMC00MWE1LThiYjYtYTE3ZGZmNjRhZmYy.eyJkYXRhIjoiU0ZSWFVBRUFldWJCSDNXamtrazRFWkI4ZFA4dzlRMFNtWnpuaEVFNGtnaFRLU1FOVWRMRWlxbVlVUy9aWHR0WUFYTUFwWU9SejZsS1Vhb3NnYitkcGh1N3lRQUFBQ1FBQUFCbWFycFdJbmpGU1NlQk1ZTjRVZTZHT2llcWVFVVNaRjhhai95bGQ5ZFlxY21xN2FmQ1JSRGgxYTlCSVNBYlREcTJ1dFdUWGI3QnZNODRvakYzZFdPZG54Ym1pUGpQRUJxblB5cTFuL2ZpYnpUdmx5SHVWb05aSnh6WXM0cjJXQTNvU1AzeDgwK0ZmMHR5cm4xZHBJSFRUNG4zWHhrL1JRSEdFZ3pSR2pkOUI0WVN0ZGRwV3hMV1gvSGIyMzBIUUhPYVNvWURzdnJQU3BSb1lyQVZWK2JkMDVkYXV0RzN4OERYRTlJS2VFd0VBbjZwcG9xN3hJSHZTdTJrdDdLaWpRb1p6Y3pORmNLTzJlSTFaVGszTVRaall5MHpaRGxpTFRRME9UQXRPV1ppWkMxa01UVXpOamd6WkdGaVltTT0iLCJzcGVjIjoxLCJpZCI6IjVlOTcxNmNjLTNkOWItNDQ5MC05ZmJkLWQxNTM2ODNkYWJiYyJ9",
  "expires_at": "2024-08-27T13:22:02.216046Z",
  "admin": true,
  "user": "admin"
}

Copy the token received in the response body in the field access_token.

This is required when executing APIs described in further steps and is referred to as Vault Authentication Token. The access_token field will be used in the curl headers from now on for the rest of the configuration calls. For example:

curl -H "X-Vault-Auth: N2U2ZTg2NDYtMjBmMC00MWE1LThiYjYtYTE3ZGZmNjRhZmYy.eyJkYXRhIjoiU0ZSWFVBRUFldWJCSDNXamtrazRFWkI4ZFA4dzlRMFNtWnpuaEVFNGtnaFRLU1FOVWRMRWlxbVlVUy9aWHR0WUFYTUFwWU9SejZsS1Vhb3NnYitkcGh1N3lRQUFBQ1FBQUFCbWFycFdJbmpGU1NlQk1ZTjRVZTZHT2llcWVFVVNaRjhhai95bGQ5ZFlxY21xN2FmQ1JSRGgxYTlCSVNBYlREcTJ1dFdUWGI3QnZNODRvakYzZFdPZG54Ym1pUGpQRUJxblB5cTFuL2ZpYnpUdmx5SHVWb05aSnh6WXM0cjJXQTNvU1AzeDgwK0ZmMHR5cm4xZHBJSFRUNG4zWHhrL1JRSEdFZ3pSR2pkOUI0WVN0ZGRwV3hMV1gvSGIyMzBIUUhPYVNvWURzdnJQU3BSb1lyQVZWK2JkMDVkYXV0RzN4OERYRTlJS2VFd0VBbjZwcG9xN3hJSHZTdTJrdDdLaWpRb1p6Y3pORmNLTzJlSTFaVGszTVRaall5MHpaRGxpTFRRME9UQXRPV1ppWkMxa01UVXpOamd6WkdGaVltTT0iLCJzcGVjIjoxLCJpZCI6IjVlOTcxNmNjLTNkOWItNDQ5MC05ZmJkLWQxNTM2ODNkYWJiYyJ9" ...

Configure Openshift cluster with KeyControl Secrets vault.

To configure an Openshift cluster in the Keycontrol vault, the following API needs to be executed.

https://<VAULT-IP>/vault/1.0/SetK8sConfiguration

The headers must have the vault authentication token, X-Vault-Auth*: <VAULT-AUTHENTICATION-TOKEN>.

The request body should be in the form:

{
 // URL of Openshift API server (or Loadbalancer if one is setup in front of
 // control plane) along with port. This URL must be accessible by PASM vault.
 // If Cluster is behind a service mesh, a proper ingress should be configured
 // so that API server is accessible.
 "k8s_url": "https://<KC-ACCESSIBLE-IP>/<FQDN of K8s API server>:6443",

 // Base64 encoded string of the certificate presented by API server during SSL handshake
 "k8s_ca_string": "<BASE64-ENCODED-CA-STRING>",

 // Indicates whether this configuration should be enabled or not. Valid values
 // are 'enabled' or 'disabled'. If the value is 'disabled', then "k8s_url" and
 // "k8s_ca_string" won't be validated and won't be saved in the configuration.
 "k8s_status": "enabled",
}

The certificate string can be obtained from the kubeconfig file or by using following command:

% oc config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}'

If Openshift is configured with multiple clusters, replace the number 0 by the appropriate cluster number. The certificate string retrieved is already base64 encoded. You do not need to encode it again.

The final request body:

{
 "k8s_url": "https://api.ocp4147.interop.local:6443",
 "k8s_ca_string": "<certificate-string>",
 "k8s_status": "enabled"
}

Save the request body in a file named SetK8sConfiguration.body and execute the following command:

% curl -H "X-Vault-Auth: N2U2ZTg2NDYtMjBmMC00MWE1LThiYjYtYTE3ZGZmNjRhZmYy.eyJkYXRhIjoiU0ZSWFVBRUFldWJCSDNXamtrazRFWkI4ZFA4dzlRMFNtWnpuaEVFNGtnaFRLU1FOVWRMRWlxbVlVUy9aWHR0WUFYTUFwWU9SejZsS1Vhb3NnYitkcGh1N3lRQUFBQ1FBQUFCbWFycFdJbmpGU1NlQk1ZTjRVZTZHT2llcWVFVVNaRjhhai95bGQ5ZFlxY21xN2FmQ1JSRGgxYTlCSVNBYlREcTJ1dFdUWGI3QnZNODRvakYzZFdPZG54Ym1pUGpQRUJxblB5cTFuL2ZpYnpUdmx5SHVWb05aSnh6WXM0cjJXQTNvU1AzeDgwK0ZmMHR5cm4xZHBJSFRUNG4zWHhrL1JRSEdFZ3pSR2pkOUI0WVN0ZGRwV3hMV1gvSGIyMzBIUUhPYVNvWURzdnJQU3BSb1lyQVZWK2JkMDVkYXV0RzN4OERYRTlJS2VFd0VBbjZwcG9xN3hJSHZTdTJrdDdLaWpRb1p6Y3pORmNLTzJlSTFaVGszTVRaall5MHpaRGxpTFRRME9UQXRPV1ppWkMxa01UVXpOamd6WkdGaVltTT0iLCJzcGVjIjoxLCJpZCI6IjVlOTcxNmNjLTNkOWItNDQ5MC05ZmJkLWQxNTM2ODNkYWJiYyJ9" -k -X POST https://xx.xxx.xxx.xxx/vault/1.0/SetK8sConfiguration/ --data @./SetK8sConfiguration.body

The response should be:

{
  "Status": "Success",
  "Message": "Kubernetes configuration updated successfully"
}

Set the namespace.

Before proceeding with the rest of the configuration, set the namespace in openshift to the mutatingwebhook namespace created earlier.

% oc config set-context --current --namespace=mutatingwebhook

Create the registry secrets inside the namespace

The credentials for the external docker registry access need to be created so they can be mentioned in the pod deployment specification.

Create the secret in the namespace:

% oc create secret generic regcred --from-file=.dockerconfigjson=$HOME/.docker/config.json --type=kubernetes.io/dockerconfigjson

Confirm that the secret was created:
```
% oc get secret regcred
```

Deploy the webhook container.

Use the following yaml specification to deploy the webhook container in Openshift:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mutatingwebhook
  namespace: mutatingwebhook
  labels:
    app: mutatingwebhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mutatingwebhook
  template:
    metadata:
      labels:
        app: mutatingwebhook
    spec:
      containers:
      - name: mutatingwebhook
        image: 1.2.3.4:5000/mutating-webhook:latest
        ports:
        - containerPort: 5000
        env:
        # Specify these variables either as a mutatingwebhook container environment variables or
        # add appropriate annotations in the pod specification. Refer documentation further to get
        # the details about annotations. The annotations in pod specifications, if specified, take
        # precedance over these environment variables.
        - name: ENTRUST_VAULT_IPS
          value: <comma-separated-list-of-vault-ips>
        - name: ENTRUST_VAULT_UUID
          value: <uuid-of-vault-from-which-secrets-to-be-fetched>
        - name: ENTRUST_VAULT_INIT_CONTAINER_URL
          value: <complete-url-of-init-container-image-from-image-registry>
        securityContext:				#Required, only if deploying on OpenShift
          allowPrivilegeEscalation: false 		#Required, only if deploying on OpenShift
          capabilities: 	    			#Required, only if deploying on OpenShift
            drop: 					#Required, only if deploying on OpenShift
              - ALL 					#Required, only if deploying on OpenShift
    securityContext: 					#Required, only if deploying on OpenShift
      runAsNonRoot: true 				#Required, only if deploying on OpenShift
      runAsUser: 1000700000 				#Required, only if deploying on OpenShift
      seccompProfile: 					#Required, only if deploying on OpenShift
        type: RuntimeDefault 				#Required, only if deploying on OpenShift

If image registry requires authentication, then make sure to add the credentials as secret in the yaml specification by adding the imagePullSecrets tag in the containers section.

Save the yaml in a file called mutatingwebhook.yaml.

Example mutatingwebhook file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mutatingwebhook
  namespace: mutatingwebhook
  labels:
    app: mutatingwebhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mutatingwebhook
  template:
    metadata:
      labels:
        app: mutatingwebhook
    spec:
      imagePullSecrets:
        - name: regcred                         # external registry secret created earlier
      containers:
      - name: mutatingwebhook
        image: >-
          <registry-url>/mutating-webhook
        ports:
        - containerPort: 5000
        env:
        # Specify these variables either as a mutatingwebhook container environment variables or
        # add appropriate annotations in the pod specification. Refer documentation further to get
        # the details about annotations. The annotations in pod specifications, if specified, take
        # precedance over these environment variables.
        - name: ENTRUST_VAULT_IPS
          value: 1x.19x.14x.20x,1x.19x.14x.21x
        - name: ENTRUST_VAULT_UUID
          value: 7e6e8646-20f0-41a5-8bb6-a17dff64aff2
        - name: ENTRUST_VAULT_INIT_CONTAINER_URL
          value: <registry-url>/init-container
        securityContext:				#Required, only if deploying on OpenShift
          allowPrivilegeEscalation: false 		#Required, only if deploying on OpenShift
          capabilities: 	    			#Required, only if deploying on OpenShift
            drop: 					#Required, only if deploying on OpenShift
              - ALL 					#Required, only if deploying on OpenShift
    securityContext: 					#Required, only if deploying on OpenShift
      runAsNonRoot: true 				#Required, only if deploying on OpenShift
      runAsUser: 1000700000 				#Required, only if deploying on OpenShift
      seccompProfile: 					#Required, only if deploying on OpenShift
        type: RuntimeDefault 				#Required, only if deploying on OpenShift

Pay attention to these sections and adjust according to your environment.

      imagePullSecrets:
        - name: regcred                         # external registry secret created earlier

        image: >-
          <registry-url>/mutating-webhook

        - name: ENTRUST_VAULT_IPS
          value: 1x.19x.14x.20x,1x.19x.14x.21x
        - name: ENTRUST_VAULT_UUID
          value: 7e6e8646-20f0-41a5-8bb6-a17dff64aff2
        - name: ENTRUST_VAULT_INIT_CONTAINER_URL
          value: <registry-url>/init-container

Deploy the file:

% oc create -f mutatingwebhook.yaml

Check if the multating webhook deployment is running:

$ oc get pods
NAME                              READY   STATUS    RESTARTS   AGE
mutatingwebhook-c7958b7f9-dwd9x   1/1     Running   0          5s

$ oc describe pod mutatingwebhook-c7958b7f9-dwd9x
Name:             mutatingwebhook-c7958b7f9-dwd9x
Namespace:        mutatingwebhook
Priority:         0
Service Account:  default
Node:             ocp4147-r758r-worker-0-t7lfl/1x.19x.14x.2xx
Start Time:       Tue, 27 Aug 2024 13:20:12 -0400
Labels:           app=mutatingwebhook
                  pod-template-hash=c7958b7f9
Annotations:      k8s.ovn.org/pod-networks:
                    {"default":{"ip_addresses":["10.129.2.136/23"],"mac_address":"0a:58:0a:81:02:88","gateway_ips":["10.129.2.1"],"routes":[{"dest":"10.128.0....
                  k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "ovn-kubernetes",
                        "interface": "eth0",
                        "ips": [
                            "10.129.2.136"
                        ],
                        "mac": "0a:58:0a:81:02:88",
                        "default": true,
                        "dns": {}
                    }]
                  openshift.io/scc: restricted-v2
                  seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:           Running
SeccompProfile:   RuntimeDefault
IP:               10.129.2.136
IPs:
  IP:           10.129.2.136
Controlled By:  ReplicaSet/mutatingwebhook-c7958b7f9
Containers:
  mutatingwebhook:
    Container ID:   cri-o://a05bed83d3c0d5ac39a1d421e9dc992cdbf6cb3550826520dd978eb3da927a02
    Image:          <registry-url>/mutating-webhook
    Image ID:       <registry-url>/mutating-webhook@sha256:d5379f9b116f725c1e34cda7f10cba2ef7ed369681a768fb985d098cd24f1b1d
    Port:           5000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 27 Aug 2024 13:20:14 -0400
    Ready:          True
    Restart Count:  0
    Environment:
      ENTRUST_VAULT_IPS:                 1x.19x.14x.20x,1x.19x.14x.21x
      ENTRUST_VAULT_UUID:                7e6e8646-20f0-41a5-8bb6-a17dff64aff2
      ENTRUST_VAULT_INIT_CONTAINER_URL:  <registry-url>/init-container
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qpv28 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-qpv28:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       17s   default-scheduler  Successfully assigned mutatingwebhook/mutatingwebhook-c7958b7f9-dwd9x to ocp4147-r758r-worker-0-t7lfl
  Normal  AddedInterface  17s   multus             Add eth0 [10.129.2.136/23] from ovn-kubernetes
  Normal  Pulling         17s   kubelet            Pulling image "registry.eselab.net/mutating-webhook"
  Normal  Pulled          16s   kubelet            Successfully pulled image "<registry-url>/mutating-webhook" in 1.14339284s (1.143433867s including waiting)
  Normal  Created         16s   kubelet            Created container mutatingwebhook
  Normal  Started         16s   kubelet            Started container mutatingwebhook

Deploy the entrust-pasm service.

Create a yaml file named entrustpasmservice.yaml that contains the following code:

apiVersion: v1
kind: Service
metadata:
  name: entrust-pasm		# DO NOT CHANGE THIS
  namespace: mutatingwebhook 	# DO NOT CHANGE THIS
spec:
  selector:
    app: mutatingwebhook
  ports:
    - protocol: TCP
      port: 5000
      targetPort: 5000

Run the following command to create the service:
```
% oc create -f entustpasmservice.yaml
```

Check the service:

% oc get service
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
entrust-pasm   ClusterIP   172.30.233.96   <none>        5000/TCP   8s

Obtain the webhook server certificate.

After the container is deployed, obtain the webhook server certificate. This certificate will be needed to configure the webhook in openshift. Use the following command:

% oc exec -it -n mutatingwebhook $(kubectl get pods --no-headers -o custom-columns=":metadata.name" -n mutatingwebhook) -- wget -q -O- localhost:8080/ca.pem

The output should be something like this:

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZTekNDQXpPZ0F3SUJBZ0lVRUFiWFlYazZHVUVJM2lrUGZQbnp4MVRMRjdVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1FERXhNQzhHQTFVRUF3d29TSGxVY25WemRDQkxaWGxEYjI1MGNtOXNJRU5sY25ScFptbGpZWFJsSUVGMQpkR2h2Y21sMGVURUxNQWtHQTFVRUJoTUNWVk13SGhjTk1qUXdPREkzTVRjeU1ERTNXaGNOTXpRd09ESTFNVGN5Ck1ERTNXakJBTVRFd0x3WURWUVFERENoSWVWUnlkWE4wSUV0bGVVTnZiblJ5YjJ3Z1EyVnlkR2xtYVdOaGRHVWcKUVhWMGFHOXlhWFI1TVFzd0NRWURWUVFHRXdKVlV6Q0NBaUl3RFFZSktvWklodmNOQVFFQkJRQURnZ0lQQURDQwpBZ29DZ2dJQkFMTlVMQk5KVVVzYU1nSHlrR3hHOWcvdFBtZXdFTi9DODVlSFJjMmREOHZyZndiNkZ5dFg3UG1BCnRhSVliMmJ4OUdTaldXVFlNNXlPL0g5WkY5L3BjUmJuYkdaNFBMTzBGdG1vM2FicCszRVJjTENWREZCamszQlcKTW9hOWQ1VVBkU2kzYnFBdGFrT2xFZHdOQ2ZvSWhlMDZNOGF2TDcrVWtZV1FQWGppMzBDODZCUTRRWE01dkFKQwpwaFg1NmhCY3N1RjJDc0w2cGZYUjlJbFVjVTgwZFBKN1BiT3ZubWdNbGpYakpYUnpHREJOUG9VS25HNmJ5N0tpCjV2TkNBaXYxRmJPQUxXUlZCZWdRUHZEdmpJRmVoMzBZUDRkamlydk5Yc1pNZnRiamFzZFVPWGhIMGFYK1BYN3cKYTNFRUpZL290TjR4VVlLaTFGRlhteDRRWTZaSTVheWdnTzNHSTZEWDJUcENQUzFqcnpXL2xETzhtd1QzNEhFSApKSno1T05abkRGME40YXJJWnFKa2dUVTd2Q1RaL3ltaWFsc2lDNkJQM1lUTkl5NmlKY1loczFUYWRQUVpmaElnCmxqeFhPQ2dDTXdLV1poNjdPdEtWTkNmeC83ZzBsbHhyeW1YY0xoY25aRy81azRDbmlmZHVTQWl4a1hRRVpXaTcKenRJL0ZIZjJMVGNrVkNnSms5NDlJSlNpSTdVREQwQkFHTkVnTE5hVnFBQ0pRbTNlQmVTS24xZ0c0ODNHMTRQawp5MkYyMEpEelBreG5ZVlBoVkVpK29scXZ3ZFVPVlFNWFV5UTUyQTZ5RTQwZFZsSjBIZGJoTjdzOUJZQWJSUjB4Ck9xalcvTW0yL2trUE1rZ3IyRlNsK2JhcVlzS1hKUktsNFVIemZlQUxNRXNIeCs3L2lMK0hBZ01CQUFHalBUQTcKTUF3R0ExVWRFd0VCL3dRQ01BQXdLd1lEVlIwUkJDUXdJb0lnWlc1MGNuVnpkQzF3WVhOdExtMTFkR0YwYVc1bgpkMlZpYUc5dmF5NXpkbU13RFFZSktvWklodmNOQVFFTEJRQURnZ0lCQU1aVFpPNmhSS29HVEs0eHB3b1hqY3p1CnloSzZCbURsejg0akd2ek85Q083dkZJblFRSkVlK1Mrc3lBb1l6N3kwNlpnUDZJYi9lZEx5by9aTFB0UDAralAKSnloSUNoN3lXVldhNkJyNEE3VExBT1BpSWEvQmtaZHBaR3NMdENrbE5uckF1MFBBbXY0QWpUZUZzZWF2aXREeApBTWlYSUlTYk1TZG55bEhFVTUrMWREdzRMRTBtN01UR2JlU2E3TnZEelVZdWhCNkxDVlhxTWlUQTQ3MCtpdXF4Cm10dkZ3WVU5RVgvV3ZLdlNibWFZZjRKR24rL3pBMlNpU0d5V0R1NW03RCtjazZoTXBKenRXWW91YlNROWxQQTUKWmZxcDhBYkdnc1J3a1IyTjF1VzZFK1BJNWEzb1llU2NKMUJJeG5GTTBxNjJuV3ppdHB3SlRQN2VsNTlkSWFEQwpYaEc3UWVxbDFCSUpWUHR6M2Nra1dCTzhuNlJhOHNtVWxTMVNSNkdUWDZ5RGIrV1YzVzVranhmZ0lrNXJCRnlyClRXeEVPSHlJQkdQc1d2UnAvNFlyTHRzOW10ekpPUVJqeVpLUVZlaUhmaGI1T1JTSE9MT1Y3MERESHd5K2hIc0wKemtKRkc3Mm9ocDJYeDkweHdoeUVFdlhWYU5mRFpTekdKVXpWSFhTSDRWdG1WaXFBbkFYQkhLYXdHVlA1OXBRVwphOEtuZWNVZkk3VExlSk9ya09HelZLeU05YUFHaS83MW92WFUwZnZOWXZscTlENU9FajVKME5mQ1hzWXdlcDE5CnVyYlFpRUJJRFNMbElpekIxK1lUYUsvOTJHQ1IwaU1JeG5ON2NWU3l1UEJ1emRZSkt4QjdNZEhhZXFsM1FERzQKcUpjSm5BNmRhUFZ1dzFqMi9sZmgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

This is a base64 encoded certificate. With this value, configure webhook in openshift.

Configure the webhook in Openshift.

In sample YAML spec replace the caBundle value with the value obtained in the previous section and configure the webhook with the oc apply command.

Create a webhook.yaml with the following code:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: "entrust-pasm.mutatingwebhook.com"
webhooks:
  - name: "entrust-pasm.mutatingwebhook.com"
    objectSelector:
      matchLabels:
        entrust.vault.inject.secret: enabled
    rules:
    - apiGroups: ["*"]
      apiVersions: ["*"]
      operations: ["CREATE"]
      resources: ["deployments", "jobs", "pods", "statefulsets"]
    clientConfig:
      service:
        namespace: "mutatingwebhook"
        name: "entrust-pasm"
        path: "/mutate"
        port: 5000
        # Replace value below with the value obtained from command
      caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZTekNDQXpPZ0F3SUJBZ0lVRUFiWFlYazZHVUVJM2lrUGZQbnp4MVRMRjdVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1FERXhNQzhHQTFVRUF3d29TSGxVY25WemRDQkxaWGxEYjI1MGNtOXNJRU5sY25ScFptbGpZWFJsSUVGMQpkR2h2Y21sMGVURUxNQWtHQTFVRUJoTUNWVk13SGhjTk1qUXdPREkzTVRjeU1ERTNXaGNOTXpRd09ESTFNVGN5Ck1ERTNXakJBTVRFd0x3WURWUVFERENoSWVWUnlkWE4wSUV0bGVVTnZiblJ5YjJ3Z1EyVnlkR2xtYVdOaGRHVWcKUVhWMGFHOXlhWFI1TVFzd0NRWURWUVFHRXdKVlV6Q0NBaUl3RFFZSktvWklodmNOQVFFQkJRQURnZ0lQQURDQwpBZ29DZ2dJQkFMTlVMQk5KVVVzYU1nSHlrR3hHOWcvdFBtZXdFTi9DODVlSFJjMmREOHZyZndiNkZ5dFg3UG1BCnRhSVliMmJ4OUdTaldXVFlNNXlPL0g5WkY5L3BjUmJuYkdaNFBMTzBGdG1vM2FicCszRVJjTENWREZCamszQlcKTW9hOWQ1VVBkU2kzYnFBdGFrT2xFZHdOQ2ZvSWhlMDZNOGF2TDcrVWtZV1FQWGppMzBDODZCUTRRWE01dkFKQwpwaFg1NmhCY3N1RjJDc0w2cGZYUjlJbFVjVTgwZFBKN1BiT3ZubWdNbGpYakpYUnpHREJOUG9VS25HNmJ5N0tpCjV2TkNBaXYxRmJPQUxXUlZCZWdRUHZEdmpJRmVoMzBZUDRkamlydk5Yc1pNZnRiamFzZFVPWGhIMGFYK1BYN3cKYTNFRUpZL290TjR4VVlLaTFGRlhteDRRWTZaSTVheWdnTzNHSTZEWDJUcENQUzFqcnpXL2xETzhtd1QzNEhFSApKSno1T05abkRGME40YXJJWnFKa2dUVTd2Q1RaL3ltaWFsc2lDNkJQM1lUTkl5NmlKY1loczFUYWRQUVpmaElnCmxqeFhPQ2dDTXdLV1poNjdPdEtWTkNmeC83ZzBsbHhyeW1YY0xoY25aRy81azRDbmlmZHVTQWl4a1hRRVpXaTcKenRJL0ZIZjJMVGNrVkNnSms5NDlJSlNpSTdVREQwQkFHTkVnTE5hVnFBQ0pRbTNlQmVTS24xZ0c0ODNHMTRQawp5MkYyMEpEelBreG5ZVlBoVkVpK29scXZ3ZFVPVlFNWFV5UTUyQTZ5RTQwZFZsSjBIZGJoTjdzOUJZQWJSUjB4Ck9xalcvTW0yL2trUE1rZ3IyRlNsK2JhcVlzS1hKUktsNFVIemZlQUxNRXNIeCs3L2lMK0hBZ01CQUFHalBUQTcKTUF3R0ExVWRFd0VCL3dRQ01BQXdLd1lEVlIwUkJDUXdJb0lnWlc1MGNuVnpkQzF3WVhOdExtMTFkR0YwYVc1bgpkMlZpYUc5dmF5NXpkbU13RFFZSktvWklodmNOQVFFTEJRQURnZ0lCQU1aVFpPNmhSS29HVEs0eHB3b1hqY3p1CnloSzZCbURsejg0akd2ek85Q083dkZJblFRSkVlK1Mrc3lBb1l6N3kwNlpnUDZJYi9lZEx5by9aTFB0UDAralAKSnloSUNoN3lXVldhNkJyNEE3VExBT1BpSWEvQmtaZHBaR3NMdENrbE5uckF1MFBBbXY0QWpUZUZzZWF2aXREeApBTWlYSUlTYk1TZG55bEhFVTUrMWREdzRMRTBtN01UR2JlU2E3TnZEelVZdWhCNkxDVlhxTWlUQTQ3MCtpdXF4Cm10dkZ3WVU5RVgvV3ZLdlNibWFZZjRKR24rL3pBMlNpU0d5V0R1NW03RCtjazZoTXBKenRXWW91YlNROWxQQTUKWmZxcDhBYkdnc1J3a1IyTjF1VzZFK1BJNWEzb1llU2NKMUJJeG5GTTBxNjJuV3ppdHB3SlRQN2VsNTlkSWFEQwpYaEc3UWVxbDFCSUpWUHR6M2Nra1dCTzhuNlJhOHNtVWxTMVNSNkdUWDZ5RGIrV1YzVzVranhmZ0lrNXJCRnlyClRXeEVPSHlJQkdQc1d2UnAvNFlyTHRzOW10ekpPUVJqeVpLUVZlaUhmaGI1T1JTSE9MT1Y3MERESHd5K2hIc0wKemtKRkc3Mm9ocDJYeDkweHdoeUVFdlhWYU5mRFpTekdKVXpWSFhTSDRWdG1WaXFBbkFYQkhLYXdHVlA1OXBRVwphOEtuZWNVZkk3VExlSk9ya09HelZLeU05YUFHaS83MW92WFUwZnZOWXZscTlENU9FajVKME5mQ1hzWXdlcDE5CnVyYlFpRUJJRFNMbElpekIxK1lUYUsvOTJHQ1IwaU1JeG5ON2NWU3l1UEJ1emRZSkt4QjdNZEhhZXFsM1FERzQKcUpjSm5BNmRhUFZ1dzFqMi9sZmgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    admissionReviewVersions: ["v1", "v1beta1"]
    sideEffects: None
    timeoutSeconds: 5

Run the following command to apply the changes:

% oc apply -f webhook.yaml

mutatingwebhookconfiguration.admissionregistration.k8s.io/entrust-pasm.mutatingwebhook.com created

Testing

Now that the openshift cluster and Keycontrol Vault are properly configured and set up let’s deploy a couple of pods that will attempt to use the ocsecret created earlier inside the pod containers.

Important points:

The deployment of pod (in which the secrets need to be fetched) must happen by the service accounts mentioned in the policy created above.
The service accounts added in policy must have system:auth-delegator ClusterRole at the openshift side.

Set the namespace.

We created two namespaces earlier. Currently the integration should be set to the mutatingwebhook namespace. Change the namespace so it points to the test namespace.

> oc config set-context --current --namespace=testnamespace

Create the registry secrets inside the namespace

The credentials for the external docker registry access need to be created so they can be mentioned in the pod deployment specification.

Create the secret in the namespace:

% oc create secret generic regcred --from-file=.dockerconfigjson=$HOME/.docker/config.json --type=kubernetes.io/dockerconfigjson

Confirm that the secret was created:
```
% oc get secret regcred
```

Create a Policy in Secrets Vault for openshift service accounts

Create a policy in the Secrets Vault that will enable Openshift service accounts to read secrets from the vault. To create such policy, the following API needs to be executed.

URL: https://<VAULT-IP>/vault/1.0/CreatePolicy

The headers must have the vault authentication token.

X-Vault-Auth: <VAULT-AUTHENTICATION-TOKEN>

The request body should be in the form:

{
  "name": "vault_user_policy",		   // Name of the policy to be created
  "role": "Vault User Role",          	   // Role of user. DO NOT CHANGE THIS
  "principals": [
     {
       "k8s_user": {
         "k8s_namespace": "default",       // Namespace in which the service account resides
         "k8s_service_account": "default"  // Name of the service account
       }
     }
  ],
  "resources": [
     {
       "box_id": "box1", // Name of the box in which the secret(s) that need access reside. Can be '*' to indicate all boxes
       "secret_id": [ 	 // List of secrets to which access needs to be granted. Can be '*' to indicate all secrets.
         "secret1",
         "secret2
       ]
     }
  ]
}

Match the namespace k8s_namespace to the same namespace the service account resides in and where you will be deploying the pods.

Save the request body for the environment to a file named createpolicy.body:

{
  "name": "openshift_vault_user_policy",
  "role": "Vault User Role",
  "principals": [
     {
       "k8s_user": {
         "k8s_namespace": "testnamespace",
         "k8s_service_account": "default"
       }
     }
  ],
  "resources": [
     {
       "box_id": "box1",
       "secret_id": [
         "*"
       ]
     }
  ]
}

In this example, the policy will allow the default user in the testnamespace to have access to any secret in box1 in the Keycontrol Secrets vault.

Check and adjust the following sections according to your environment:

       "k8s_user": {
         "k8s_namespace": "testnamespace",
         "k8s_service_account": "default"

       "box_id": "box1",
       "secret_id": [
         "*"
       ]

Create the policy:

curl -H "X-Vault-Auth: <VAULT-IDENTIFICATION-TOKEN>" -k -X POST https://10.194.148.206/vault/1.0/CreatePolicy/ --data @./createpolicy.body

{"policy_id": "openshift_vault_user_policy-82756a"}

Create the clusterrolebinding.

Create the clusterrolebinding to allow the default user access to the secrets:

% oc create clusterrolebinding authdelegator --clusterrole=system:auth-delegator --serviceaccount=testnamespace:default

Test the access by running the following command:

% oc auth can-i create tokenreviews --as=system:serviceaccount:testnamespace:default

The output should be yes if set up correctly.

Deploying Pod with Secrets

Secrets can be added to the pod either as volume mounts or as environment variables.

Adding secrets as volume mounts to the pod

The sample pod specification along with labels and annotations required for successfully pulling secrets as volumes mounts:

apiVersion: v1
kind: Pod
metadata:
  name: pod1
  namespace: testnamespace
  labels:
    app: test
    entrust.vault.inject.secret: enabled		# Must have this label
  annotations:
    entrust.vault.ips: <comma-separated-list-of-vault-ips>
    entrust.vault.uuid: <uuid-of-vault-from-which-secrets-to-be-fetched>
    entrust.vault.init.container.url: <complete-url-of-init-container-image-from-image-registry>
    entrust.vault.secret.file.k8s-box.ca-cert: output/cert.pem	# box and secret name from vault and value denoting location of secret on the container. The location
spec:
  serviceAccountName: k8suser	       # Service account name configured in PASM Vault Policy
  containers:
  - name: ubuntu
    image: 10.254.154.247:5000/ubuntu:latest
    command: ['cat','/output/ans.txt']
    imagePullPolicy: IfNotPresent
    securityContext:		       #Required, only if deploying on Openshift
      allowPrivilegeEscalation: false  #Required, only if deploying on Openshift
      capabilities: 		       #Required, only if deploying on OpenShift
        drop: 	    	       	       #Required, only if deploying on OpenShift
          - ALL 	      	       #Required, only if deploying on OpenShift
  securityContext: 	   	       #Required, only if deploying on OpenShift
    runAsNonRoot: true 	      	       #Required, only if deploying on OpenShift
    runAsUser: 1000700000 	       #Required, only if deploying on OpenShift
    seccompProfile: 	  	       #Required, only if deploying on OpenShift
      type: RuntimeDefault     	       #Required, only if deploying on OpenShift

The pod specification must have the following annotations:

entrust.vault.inject.secret: enabled: This label indicates that it needs secret.
entrust.vault.ips: This value is a comma-separated list of IPs of vaults which are in the cluster.

This annotation, if specified, will override the ENTRUST_VAULT_IPS environment variable from the mutating webhook configuration.
entrust.vault.uuid: This value denotes the UUID of the vault from which we need to fetch the secrets.

This annotation, if specified, will override the ENTRUST_VAULT_UUID environment variable from the mutating webhook configuration.
entrust.vault.init.container.url: This value denotes the complete URL of the init container image pushed into image registry.

This annotation, if specified, will override the ENTRUST_VAULT_INIT_CONTAINER_URL environment variable from the mutating webhook configuration.
Annotations in the form of entrust.vault.secret.file.{box-name}.{secret-name}.

The box-name and secret-name placeholders within the annotation should denote the name of the box and the secret name within that box that needs to be pulled.

For example, if the secret was named db-secret and the box was k8s-box, then the name of the annotation would be entrust.vault.secret.file.k8s-box.db-secret. The value of the annotation should be the path to the file where the secret needs to be stored within the container. The path should be relative to the / directory, meaning that if the secret needs to be present in /output/cert.pem, then the value of the annotation should be output/cert.pem
In the spec section, the serviceAccountName value should be the name of the service account that was added in the policy in the Secrets vault.

To add the secrets as volume mounts to the pod:

Create a file named pod1.yaml with the yaml for the environment. We are using the box and secret we created earlier in the Secrets vault. (box1 and ocsecret). We also had to provide the credentials for the docker registry.

apiVersion: v1
kind: Pod
metadata:
  name: pod1
  namespace: testnamespace
  labels:
    app: test
    entrust.vault.inject.secret: enabled                # Must have this label
  annotations:
    entrust.vault.ips: 1x.19x.14x.20x,1x.19x.14x.21x
    entrust.vault.uuid: 7e6e8646-20f0-41a5-8bb6-a17dff64aff2
    entrust.vault.init.container.url: <registry-url>/init-container
    entrust.vault.secret.file.box1.ocsecret: output/ocsecret.txt
spec:
  imagePullSecrets:
    - name: regcred                    # external registry secret created earlier
  serviceAccountName: default          # Service account name configured in PASM Vault Policy
  containers:
  - name: ubuntu
    image: ubuntu
    command: ["sh", "-c"]
    args:
        - echo "Getting secret from KeyControl Secrets Vault";
          cat /output/ocsecret.txt;
          echo;
          echo "DONE" && sleep 3600
    imagePullPolicy: IfNotPresent
    securityContext:                   #Required, only if deploying on OpenShift
      allowPrivilegeEscalation: false  #Required, only if deploying on OpenShift
      capabilities:                    #Required, only if deploying on OpenShift
        drop:                          #Required, only if deploying on OpenShift
          - ALL                        #Required, only if deploying on OpenShift
  securityContext:                     #Required, only if deploying on OpenShift
    runAsNonRoot: true                 #Required, only if deploying on OpenShift
    runAsUser: 1000700000              #Required, only if deploying on OpenShift
    seccompProfile:                    #Required, only if deploying on OpenShift
      type: RuntimeDefault             #Required, only if deploying on OpenShift

Check and adjust the following sections according to your environment:

  annotations:
    entrust.vault.ips: 1x.19x.14x.20x,1x.19x.14x.21x
    entrust.vault.uuid: 7e6e8646-20f0-41a5-8bb6-a17dff64aff2
    entrust.vault.init.container.url: <registry-url>/init-container
    entrust.vault.secret.file.box1.ocsecret: output/ocsecret.txt

  imagePullSecrets:
    - name: regcred                    # external registry secret created earlier

    command: ["sh", "-c"]
    args:
        - echo "Getting secret from KeyControl Secrets Vault";
          cat /output/ocsecret.txt;
          echo;
          echo "DONE" && sleep 3600

Deploy the pod:
```
% oc create -f pod1.yaml
```

Check the pod output to verify that it was capable of pulling the secret from the KeyControl Secrets vault:

% oc logs pod/pod1

Defaulted container "ubuntu" out of: ubuntu, secret-v (init)
Getting secret from KeyControl Secrets Vault
This is the secret coming from KCV to Openshift.
DONE

Pulling secret as environment variable within Pod

Openshift supports initiating environment variables for a container either directly or from Openshift secrets. For security purposes, the secrets vault takes the later approach. Also, if secrets are injected using environment variables, a sidecar container will be added which will delete the Openshift secrets after the successful injection of KeyControl Secret Vault secrets as environment variables in the main application container. Since we need to create and delete Openshift secrets for this, the service account must also have the required permissions to create and delete the Openshift secrets.

Grant the service account with required permissions: (replace the namespace and serviceaccount values appropriately)

% oc create rolebinding secretrole --namespace testnamespace --clusterrole=edit --serviceaccount=testnamespace:default

To check if proper permissions are set up for the service account, use the following commands:

% oc auth can-i create secrets -n testnamespace --as=system:serviceaccount:testnamespace:default
% oc auth can-i delete secrets -n testnamespace --as=system:serviceaccount:testnamespace:default

The output of both the above commands should be yes.

Save the following sample pod specification yaml in a file called pod2.yaml. It shows how to deploy a pod with secrets as environment variables.

apiVersion: v1
kind: Pod
metadata:
  name: pod2
  namespace: testnamespace
  labels:
    app: test2
    entrust.vault.inject.secret: enabled		# Must have this label
  annotations:
    entrust.vault.ips: 10.19x.14x.20x.,10.19x.14x.21x
    entrust.vault.uuid: 7e6e8646-20f0-41a5-8bb6-a17dff64aff2
    entrust.vault.init.container.url: <registry-url>/init-container
    entrust.vault.secret.env.box1.ocsecret: OCSECRET
spec:
  imagePullSecrets:
    - name: regcred                    # external registry secret created earlier
  serviceAccountName: default	       # Service account name configured in PASM Vault Policy
  containers:
  - name: ubuntu
    image: ubuntu
    command: ["sh", "-c"]
    args:
        - echo "Getting secret from KeyControl Secrets Vault";
          printenv OCSECRET;
          echo "DONE" && sleep 3600
    imagePullPolicy: IfNotPresent
    securityContext:		       #Required, only if deploying on OpenShift
      allowPrivilegeEscalation: false  #Required, only if deploying on OpenShift
      capabilities: 		       #Required, only if deploying on OpenShift
        drop: 	    	       	       #Required, only if deploying on OpenShift
          - ALL 	      	       #Required, only if deploying on OpenShift
  securityContext: 	   	       #Required, only if deploying on OpenShift
    runAsNonRoot: true 	      	       #Required, only if deploying on OpenShift
    runAsUser: 1000700000 	       #Required, only if deploying on OpenShift
    seccompProfile: 	  	       #Required, only if deploying on OpenShift
      type: RuntimeDefault     	       #Required, only if deploying on OpenShift

To pull the secret as an environment variable, add an annotation of the form entrust.vault.secret.env.{box-name}.{secret-name}. The value of the annotation should indicate the name of the environment variable in which the secret data is expected to be present.

Check and adjust the following sections according to your environment. These annotations are as documented in the previous section.

  annotations:
    entrust.vault.ips: 10.19x.14x.20x.,10.19x.14x.21x
    entrust.vault.uuid: 7e6e8646-20f0-41a5-8bb6-a17dff64aff2
    entrust.vault.init.container.url: <registry-url>/init-container
    entrust.vault.secret.env.box1.ocsecret: OCSECRET

  imagePullSecrets:
    - name: regcred                    # external registry secret created earlier

    command: ["sh", "-c"]
    args:
        - echo "Getting secret from KeyControl Secrets Vault";
          printenv OCSECRET;
          echo "DONE" && sleep 3600

Create and test the pod:
```
% oc create -f pod2.yaml
```

Check the pod output to verify that it was capable of pulling the secret from the KeyControl Secrets vault:

% oc logs pod/pod2

Defaulted container "ubuntu" out of: ubuntu, pasm-sidecar, secret-v (init)
Getting secret from KeyControl Secrets Vault
This is the secret coming from KCV to Openshift.
DONE