Apache nShield on Kubernetes

This section describes how to build, deploy and confirm an instance of the Apache WebServer running with PKCS #11 with a nShield HSM on Kubernetes. This guide uses Red Hat OpenShift for the Kubernetes environment of choice. This section of the guide has only been tested with the Connect XC.

Prerequisites

  1. This integration procedure uses two servers:

    • Server #1 will be used to build the Apache Docker image.

      This server will be referred to as "the build machine". It should be the same server on which you did the Apache PKCS #11 integration.

    • Server #2 is the server from which you have access to your Kubernetes environment.

      This guide has been tested using a server that has Red Hat CodeReady OpenShift installed and also with a OpenShift Cluster deployed in a VMware vSphere environment. This will be used to deploy the image. This server will be referred to as the "the Kubernetes server".

      It is possible to perform this procedure using a single server, but Entrust strongly suggests the use of two servers.

  2. Security World software is already installed on the build machine. This was performed during the initial integration, refer to Install the Security World software and create the Security World.

  3. OpenSSL configured on the build machine.

  4. nCOP v1.1.1 installed on the build machine with Docker.

  5. Access to a Kubernetes cluster in the Kubernetes server.

Entrust strongly suggests you familiarise with the procedures in the Red Hat OpenShift nShield HSM Integration Guide. This chapter includes many of the same procedures from that guide, the key difference being the creation of the Apache Container.

Generate the key and certificate and save Security World files

This procedure is performed on the build machine.

  1. Edit /opt/nfast/cknfastrc and make sure it contains the following:

    CKNFAST_FAKE_ACCELERATOR_LOGIN=1
    Make sure CKNFAST_LOADSHARING is commented out.

  2. Create a working directory:

    % mkdir ~/working; cd ~/working

  3. Configure your OpenSSL configuration to point to the OpenSSL configuration file used in this integration, see Set up the PKCS11 engine.

    % export OPENSSL_CONF=/etc/pki/tls/openssl.pkcs11.cnf

  4. Check the engine:

    % openssl engine -tt -c -v

    The following should be included in the output:

    (pkcs11) pkcs11 engine
[RSA, rsaEncryption, id-ecPublicKey]
[ available ]
SO_PATH, MODULE_PATH, PIN, VERBOSE, QUIET, INIT_ARGS, FORCE_LOGIN

  5. Generate the key, specifying module protection:

    % /opt/nfast/bin/generatekey pkcs11 type=RSA size=2048 pubexp= plainname=apachecontainerkey nvram=no protect=module

key generation parameters:
 operation    Operation to perform               generate
 application  Application                        pkcs11
 protect      Protected by                       module
 verify       Verify security of key             yes
 type         Key type                           RSA
 size         Key size                           2048
 pubexp       Public exponent for RSA key (hex)
 plainname    Key name                           apachecontainerkey
 nvram        Blob in NVRAM (needs ACS)          no
Key successfully generated.
Path to key: /opt/nfast/kmdata/local/key_pkcs11_uaa522cc0b4c329ab1138a1ff90b49a408e46f630d

  6. Copy the key file from /opt/nfast/kmdata/local into the kmdl directory in your working directory:

    % mkdir -p ~/working/kmdl; cd ~/work/kmdl
% cp /opt/nfast/kmdata/local/key_pkcs11_uaa522cc0b4c329ab1138a1ff90b49a408e46f630d ~/working/kmdl/.

  7. Generate the certificate:

    % /usr/bin/openssl req -engine pkcs11 -x509 -out certificate.pem -days 365 -key "pkcs11:token=accelerator;object=apachecontainerkey;" -keyform ENGINE -subj "/CN=apachecontainerkey"

  8. Copy the world and module files from /opt/nfast/kmdata/local into the working/kmdl directory:

    % cp /opt/nfast/kmdata/local/world ~/working/kmdl/.
% cp /opt/nfast/kmdata/local/module_XXXXXX ~/working/kmdl/.

    The working/kmdl directory should contain the world file, the module file, the key and the certificate.pem file. For example:

    % ls -al1

-rw-rw-r--.  1 cviana cviana  1017 Dec 20 11:40 certificate.pem
-rw-rw-r--.  1 cviana cviana 10052 Dec 20 11:39 key_pkcs11_uaa522cc0b4c329ab1138a1ff90b49a408e46f630d
-rwxrwxr-x.  1 cviana cviana  5000 Dec 20 11:41 module_201E-03E0-D947
-rwxrwxr-x.  1 cviana cviana 39968 Dec 20 11:40 world

  9. Copy certificate.pem to /etc/pki/tls/certs/:

    % cp certificate.pem /etc/pki/tls/certs/.

  10. Edit /etc/httpd/conf.d/ssl.conf and set the following lines:

    SSLCertificateFile /etc/pki/tls/certs/certificate.pem
SSLCertificateKeyFile "pkcs11:object=apachecontainerkey;token=accelerator"
SSLCryptoDevice pkcs11

Install nCOP 1.1.1

The installation process involves extracting the nCOP tarball into /opt/ncop.

  1. Make the installation directory:

    % sudo mkdir -p /opt/ncop

  2. Extract the tarball:

    % sudo tar -xvf NCOPTARFILE -C /opt/ncop

Build the nCOP containers

This process will build nCOP containers for the hardserver and application Note that:

  • This guide uses the Red Hat flavor of the container.

  • Docker needs to be installed for this process to be successful.

  • You will also need the Security World ISO file to be able to build nCOP.

  • To configure the containers, you will need the HSM IP address, world and module files.

  • In this example, version 12.80.4 of the Security World client is in use.

  • The steps below are performed on the build machine.

To build the nCOP containers:

  1. Before building the images, stop the local hardserver:

    % sudo /opt/nfast/sbin/init.d-ncipher stop

  2. Mount the Security World Software ISO file:

    % sudo mount -t iso9660 -o loop ISOFILE.iso /mnt/iso

  3. Build the nShield container for the hardserver and application (Red Hat):

    % cd /opt/ncop
% sudo ./make-nshield-hwsp --from registry.access.redhat.com/ubi8/ubi --tag nshield-hwsp-pkcs11-redhat /mnt/iso
% sudo ./make-nshield-application --from registry.access.redhat.com/ubi8/ubi --tag nshield-app-pkcs11-redhat /mnt/iso

  4. Unmount the Security World file:

    % sudo umount /mnt/iso

  5. Validate that the images have been built:

    % sudo docker images

REPOSITORY                            TAG       IMAGE ID       CREATED         SIZE
nshield-app-pkcs11-redhat             latest    27cc224fe5f7   2 minutes ago   950MB
nshield-hwsp-pkcs11-redhat            latest    bfe9eafa3197   2 minutes ago   486MB
registry.access.redhat.com/ubi8/ubi   latest    cc0656847854   6 weeks ago     216MB

Build the Apache container image

The steps below are performed on the build machine.

  1. Copy the required files to your working directory:

    % cd ~/working
% cp /etc/pki/tls/openssl.pkcs11.cnf openssl.cnf
% cp /etc/httpd/conf.d/ssl.conf .
% cp /etc/httpd/conf/httpd.conf httpd.conf

    The required files are:

    • openssl.conf is already set, based on the Apache PKCS #11 integration.

    • ssl.cnf needs to be edited with the keys and certificates generated previously. Also change the SSL port to 9443.

    • httpd.conf.

  2. Create a file in your working directory to build your Apache container:

    % vi Dockerfile

    The file should contain the following:

    FROM nshield-app-pkcs11-redhat
RUN yum -y install httpd mod_ssl openssl-pkcs11
COPY openssl.cnf /etc/pki/tls/openssl.cnf
COPY ssl.conf /etc/httpd/conf.d/ssl.conf
COPY httpd.conf /etc/httpd/conf/httpd.conf
EXPOSE 9443/tcp
COPY kmdl /opt/nfast/kmdata/local
COPY kmdl/certificate.pem /opt/nfast/kmdata/local/
COPY kmdl/certificate.pem /etc/pki/tls/certs/.
#(for openshift)
RUN chmod 777 /etc/httpd/run
ENTRYPOINT ["/usr/sbin/httpd", "-e", "info", "-D", "FOREGROUND"]

  3. Build the Apache container image:

    % sudo docker build . -t nshield-apache-pkcs11-redhat

  4. Once the build is complete, the new image will be included in the list of Docker images:

    % sudo docker images

REPOSITORY                            TAG       IMAGE ID       CREATED          SIZE
nshield-apache-pkcs11-redhat          latest    80cd2a476deb   18 seconds ago   982MB
nshield-app-pkcs11-redhat             latest    27cc224fe5f7   19 minutes ago   950MB
nshield-hwsp-pkcs11-redhat            latest    bfe9eafa3197   20 minutes ago   486MB
registry.access.redhat.com/ubi8/ubi   latest    cc0656847854   6 weeks ago      216MB

Build the test image container

The steps below are performed on the build machine. The test image will be the base image when testing the container integration inside the Kubernetes cluster.

  1. In the working directory, create a testimage directory and copy the needed files to build the test image:

    % cd ~/working
% mkdir testimage
% cd testimage
% cp ../kmdl/certificate.pem .

  2. In the testimage directory create the abi.sh script. This will be used to test the performance of the Apache web server when integrated.

    vi abi.sh

    The content of the files should be the following:

    ab -c50 -n10000 https://localhost:9443/icons/apache_pb2.gif

  3. In the testimage, directory create the osc.sh script. This will be used to connect to the Apache webserver when integrated, to test if the server is using the key and certificates generated previously.

    vi osc.sh

    The content of the files should be the following:

    openssl s_client -crlf -connect localhost:9443 -CAfile certificate.pem

  4. Change the permissions of the scripts:

    % chmod 777 abi.sh
% chmod 777 osc.sh

  5. Create a Docker file to build the test image:

    % vi Dockerfile

    The contents of the file should be the following:

    FROM registry.access.redhat.com/ubi8/ubi
RUN yum -y install net-tools httpd-tools openssl nano
COPY certificate.pem /opt/test/
COPY abi.sh /opt/test/
COPY osc.sh /opt/test/
RUN chmod 777 /opt/test/*.sh

  6. Build the test image:

    % sudo docker build . -t test-nshield-apache-pkcs11-redhat

  7. List the Docker images:

    % sudo docker images

REPOSITORY                            TAG       IMAGE ID       CREATED             SIZE
test-nshield-apache-pkcs11-redhat     latest    7dbedf1c97d9   About an hour ago   242MB
nshield-apache-pkcs11-redhat          latest    80cd2a476deb   2 hours ago         982MB
nshield-app-pkcs11-redhat             latest    27cc224fe5f7   2 hours ago         950MB
nshield-hwsp-pkcs11-redhat            latest    bfe9eafa3197   2 hours ago         486MB
registry.access.redhat.com/ubi8/ubi   latest    cc0656847854   6 weeks ago         216MB

Test the configuration locally

The steps below are performed on the build machine.

  1. Edit the /opt/ncop/config1/config file. Change HSM settings according to the HSM that is being used and loaded with the Security World that matches your files in /opt/nfast/kmdata/local.

    % sudo vi /opt/ncop/config1/config

    The contents of the file should be the following:

    syntax-version=1

[nethsm_imports]
local_module=1
remote_esn=201E-03E0-D947
remote_ip=10.194.148.33
remote_port=9004
keyhash=84800d1bfff6515ed5806fe443bbaca812d73733
privileged=1

  2. Create a Docker socket:

    % sudo docker volume create socket1

  3. Start testing the implementation:

    % sudo docker run --rm -it -v socket1:/opt/nfast/sockets -v /opt/ncop/config1/config:/opt/nfast/kmdata/config/config nshield-hwsp-pkcs11-redhat
% sudo docker run --rm -it -v socket1:/opt/nfast/sockets -p9443:9443 nshield-apache-pkcs11-redhat

  4. Run the osc.sh script:

    % osc.sh

    This script test connects using the apachecontainerkey key from the HSM created earlier.

    CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = apachecontainerkey
verify return:1
---
Certificate chain
 0 s:CN = apachecontainerkey
   i:CN = apachecontainerkey
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICwTCCAakCFAml778s/6aHXzKK84DsM6cvO2AhMA0GCSqGSIb3DQEBCwUAMB0x
GzAZBgNVBAMMEmFwYWNoZWNvbnRhaW5lcmtleTAeFw0yMTEyMjAxNjQwMjFaFw0y
MjEyMjAxNjQwMjFaMB0xGzAZBgNVBAMMEmFwYWNoZWNvbnRhaW5lcmtleTCCASIw
DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ48W0IR6OD5wcMuQv/aYCWnYa8u
htz+Pq2UiXcmwhd9fSSkeVGHVEOP5+dtdJKjI6DTMwWeFg+u515HLM+rteeQCicD
noXEm/WV8HRffNknznHtKMiOW75UNcsOxuavc0hKCdn/q/Yn5AsBQTbjmUSFQhDQ
Z4saNuN6PSgNJh/grjZilEhuRyh4neAEyiNMBIULXecbnBpoMnMtQLm6PT1IRZQC
QQ3L8yONTPXqiap/hcSdUxOZjaTwL7oDsnTaK42eXOx2MytfSMULw+KTwnCrKYNc
+DLiRCMJsVbdHwFahjMsidnLgheBnc7BN5LAx5rH0a2J1G9lDQv/5UFa+4sCAwEA
ATANBgkqhkiG9w0BAQsFAAOCAQEAaTAWb/jaEicXCvV/80IyDnSyBHSyDT3yKMrW
dEn4O6lUgEwo1tU1MgNHPr7ripKl2UQ4GS7vrntWU7d9k7Rzq2QQB5NkhvC1xzsB
RV/miBKTd4cNGMxfB+zmEWpv82nMKsWjfQmbwPVXoIHcprpCNrJKMBOIempbOsGo
i9Y2xBxJsT8HGZ027j1rB3yMmZHSBFMRHuDRGlPMlFbDmnzsC/6VMvB+iNMA1fgs
az3bKQMYJ7Q3BzwKWV730tAyOcVVk79aRRQBLpohjCBkJqsepuRMP2EP/aPkNJ9S
sgeFjwyrsyZE4SImJQORAThXwDbYbC5cbTknOVLCLkidg+Gfuw==
-----END CERTIFICATE-----
subject=CN = apachecontainerkey

issuer=CN = apachecontainerkey

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1265 bytes and written 369 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 9EE98D136ABC1E77DC175D590226602E009BFFD76DDD3D2E67E610F10C1504B7
    Session-ID-ctx:
    Resumption PSK: 32F4D1F4A4BD0CE70A037EB5F773AFF3B19BD88F8127E03E4DB1ED3079917099004F1583C80D8C6F15CA6E0573058AD5
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - ef e2 e0 5f 17 7c 72 3b-01 b7 9b 53 f9 73 d7 c9   ..._.|r;...S.s..
    0010 - d5 fe b3 cc 03 b8 98 82-b8 3b df 92 fc 00 a1 f6   .........;......
    0020 - c3 3e 63 1b f3 27 84 fd-cb 3d 8c a5 89 61 18 1f   .>c..'...=...a..
    0030 - dd 71 98 9e 19 dd 50 e3-e5 46 81 8b df a9 75 6b   .q....P..F....uk
    0040 - 0b 18 cb 51 85 56 d1 1f-68 86 bb 41 c6 97 97 e7   ...Q.V..h..A....
    0050 - 86 ea b1 66 14 c2 20 40-41 e4 e1 f5 72 b6 0a 00   ...f.. @A...r...
    0060 - e0 6e 25 39 10 e8 c3 51-2b 41 47 6a 9e 7b fa f3   .n%9...Q+AGj.{..
    0070 - 2d 51 8e b8 81 2d 86 51-74 89 8c ea 90 26 15 90   -Q...-.Qt....&..
    0080 - 50 9e e8 23 54 0f 5f 5f-89 bf ca 92 cf 72 fc fc   P..#T.__.....r..
    0090 - 69 c0 61 e8 be b4 05 c9-da 87 c0 63 ad 8c e2 13   i.a........c....
    00a0 - 3d 88 84 4e 15 c2 0b 43-8e 6b 3a 15 f6 aa 86 8b   =..N...C.k:.....
    00b0 - fa 57 3e 25 f7 63 11 19-36 e4 c1 c3 48 46 05 25   .W>%.c..6...HF.%
    00c0 - ba 46 64 ee 86 2d 19 b9-0f 02 47 86 05 be c5 7c   .Fd..-....G....|
    00d0 - f2 51 ba 7a 14 5b f5 1b-57 5c ad 8e 3e dc 29 64   .Q.z.[..W\..>.)d

    Start Time: 1640029791
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: F3BE76408FB218FEABAB99F2172166FB0C97A0068BDCE2ECEE37D4F842D68DBD
    Session-ID-ctx:
    Resumption PSK: 36B592796087A64B4B99FB18C83D8CD824C08F5DF51A0F5BCADEAF286D24895B42ACF0698E79BC2459124F6B918D3392
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - ef e2 e0 5f 17 7c 72 3b-01 b7 9b 53 f9 73 d7 c9   ..._.|r;...S.s..
    0010 - f1 db 38 c3 49 50 9e e4-39 50 1a 81 ab 69 f8 85   ..8.IP..9P...i..
    0020 - ea a7 27 f3 4e 7d 73 ee-2e 8b eb 2c 13 38 dc aa   ..'.N}s....,.8..
    0030 - 19 ff e7 c6 e2 b6 41 2d-a6 94 f4 cb 2b 66 20 09   ......A-....+f .
    0040 - fd 01 4f 0a dc 6e 2e 3c-b1 e5 fe 96 61 3f e4 47   ..O..n.<....a?.G
    0050 - 5d ec e3 6b 03 50 94 6a-c9 62 22 72 65 4a 86 69   ]..k.P.j.b"reJ.i
    0060 - a2 7f ab 88 3c 77 03 c5-d9 64 5c 84 7b d4 41 5f   ....<w...d\.{.A_
    0070 - cb 44 2b 64 9e 3a e2 bb-56 5b 4d 2f c6 43 a9 8a   .D+d.:..V[M/.C..
    0080 - 40 49 9b 39 07 ad 75 33-90 52 d1 e8 d5 d1 c7 50   @I.9..u3.R.....P
    0090 - 50 42 e9 72 24 54 60 86-63 32 bb 25 1b 9e 83 99   PB.r$T`.c2.%....
    00a0 - d3 4f c3 5a bb a8 d1 65-d9 f4 6c 48 d6 09 bd 1c   .O.Z...e..lH....
    00b0 - 83 d9 1b d6 28 b9 25 32-86 4a 2b c9 8c 5e 68 bc   ....(.%2.J+..^h.
    00c0 - c7 f7 8c f7 c3 5e fd e6-02 33 65 fa 51 e8 bb 19   .....^...3e.Q...
    00d0 - e2 3b 54 56 e6 ee e9 2a-5c 46 d6 6e b6 76 61 91   .;TV...*\F.n.va.

    Start Time: 1640029791
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

  5. Run the abi.sh script.

    % abi.sh

    This script tests performance against the server running in the container.

    This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software:        Apache/2.4.37
Server Hostname:        localhost
Server Port:            9443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        localhost

Document Path:          /icons/apache_pb2.gif
Document Length:        4234 bytes

Concurrency Level:      50
Time taken for tests:   94.090 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      45170000 bytes
HTML transferred:       42340000 bytes
Requests per second:    106.28 [#/sec] (mean)
Time per request:       470.449 [ms] (mean)
Time per request:       9.409 [ms] (mean, across all concurrent requests)
Transfer rate:          468.82 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       49  465 457.3    333    5856
Processing:     0    1   1.1      1      44
Waiting:        0    1   1.0      1      41
Total:         49  466 457.4    334    5857

Percentage of the requests served within a certain time (ms)
  50%    334
  66%    480
  75%    603
  80%    660
  90%   1022
  95%   1347
  98%   1824
  99%   2236
 100%   5857 (longest request)

Register the Apache and nCOP containers to an external registry

In this guide, the external registry is indicated as <external-docker-registry-IP-address>. Register the Docker container images to a Docker registry, so they can be used when you deploy Kubernetes pods into the cluster.

The distribution of the nShield container image is not permitted, because the software components are under strict export controls.

  1. Log in to the Docker registry using your site credentials:

    % sudo docker login -u YOURUSERID https://<external-docker-registry-IP-address>

  2. Tag images:

    % sudo docker tag test-nshield-apache-pkcs11-redhat <external-docker-registry-IP-address>/cv-test-nshield-apache-pkcs11-redhat
% sudo docker tag nshield-apache-pkcs11-redhat <external-docker-registry-IP-address>/cv-nshield-apache-pkcs11-redhat
% sudo docker tag nshield-app-pkcs11-redhat <external-docker-registry-IP-address>/cv-nshield-app-pkcs11-redhat
% sudo docker tag nshield-hwsp-pkcs11-redhat <external-docker-registry-IP-address>/cv-nshield-hwsp-pkcs11-redhat

  3. Push images:

    % sudo docker push <external-docker-registry-IP-address>/cv-test-nshield-apache-pkcs11-redhat
% sudo docker push <external-docker-registry-IP-address>/cv-nshield-apache-pkcs11-redhat
% sudo docker push <external-docker-registry-IP-address>/cv-nshield-app-pkcs11-redhat
% sudo docker push <external-docker-registry-IP-address>/cv-nshield-hwsp-pkcs11-redhat

  4. Remove local images:

    % sudo docker rmi <external-docker-registry-IP-address>/cv-test-nshield-apache-pkcs11-redhat
% sudo docker rmi <external-docker-registry-IP-address>/cv-nshield-apache-pkcs11-redhat
% sudo docker rmi <external-docker-registry-IP-address>/cv-nshield-app-pkcs11-redhat
% sudo docker rmi <external-docker-registry-IP-address>/cv-nshield-hwsp-pkcs11-redhat

  5. Show images:

    % sudo docker images

  6. Pull images from the registry:

    % sudo docker pull <external-docker-registry-IP-address>/cv-test-nshield-apache-pkcs11-redhat
% sudo docker pull <external-docker-registry-IP-address>/cv-nshield-apache-pkcs11-redhat
% sudo docker pull <external-docker-registry-IP-address>/cv-nshield-app-pkcs11-redhat
% sudo docker pull <external-docker-registry-IP-address>/cv-nshield-hwsp-pkcs11-redhat

  7. Show images:

    % sudo docker images

Deploy on Kubernetes

For this integration, the Kubernetes environment is Red Hat OpenShift. This can be applied to any other Kubernetes environment as required. The steps below are performed on the Kubernetes server. This procedure requires that the Kubernetes environment is already created.

  1. Log in to the Kubernetes server.

    This server will be used to deploy the Kubernetes pod that will run the nShield Apache application.

  2. Install the Security World software.

    Install the latest version of the Security World Software as described in the Installation Guide for the HSM.

    Entrust recommends that you uninstall any existing nShield software before installing the new nShield software.

  3. Add the HSM used during the Apache container creation as a client to the OpenShift server.

    Install the HSM by following the instructions in the Installation Guide for the HSM. Once the Kubernetes server has been added as a client in the HSM, enroll the HSM in the Kubernetes server:

    % sudo /opt/nfast/bin/nethsmenroll --privileged 10.194.148.33

    When using an OpenShift cluster it is important to add all master and worker nodes of the cluster as clients in the HSM.

  4. Create YAML directory and files needed to deploy the application:

    1. Create a directory named yaml. Use this directory when creating all required files.

    2. Create the project.yaml project YAML file needed to deploy the application. For example:

      apiVersion: project.openshift.io/v1
kind: Project
metadata:
  annotations:
    openshift.io/description: ""
    openshift.io/display-name: apache
    openshift.io/requester: kube:admin
  name: apache
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

    3. Create the cm.yaml config map YAML file needed by the application. For example:

      apiVersion: v1
kind: ConfigMap
metadata:
  name: ncop-config-apache
  namespace: apache
data:
  config: |
    syntax-version=1

    [nethsm_imports]
    local_module=1
    remote_esn=201E-03E0-D947
    remote_ip=10.194.148.33
    remote_port=9004
    keyhash=84800d1bfff6515ed5806fe443bbaca812d73733
    privileged=1

    4. Create the secret.yaml Docker secret YAML file. This contains the required credentials to pull the container images from the Docker registry being used. For example:

      apiVersion: v1
data:
  .dockerconfigjson: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
kind: Secret
metadata:
  creationTimestamp: "2021-09-20T18:40:23Z"
  name: regcred
  namespace: apache
  resourceVersion: "41868365"
  selfLink: /api/v1/namespaces/default/secrets/regcred
  uid: 96b9d6a4-de30-48cc-9c9a-bbf60730cf92
type: kubernetes.io/dockerconfigjson

    5. Create the pod_apache.yaml pod YAML file that will be used to deploy the application. For example:

      kind: Pod
apiVersion: v1
metadata:
  generateName: nshield-apache-example-
  namespace: apache
  labels:
    app: nshield-apache
spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: test-nshield-apache-pkcs11
      command:
        - sh
        - '-c'
        - sleep 7200
      image: >-
        registry.eselab.net/cv-test-nshield-apache-pkcs11-redhat
      imagePullPolicy: Always
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File

    - name: nshield-hwsp-pkcs11
      image: >-
        registry.eselab.net/cv-nshield-hwsp-pkcs11-redhat
      imagePullPolicy: Always
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      ports:
        - containerPort: 8080
          protocol: TCP
      resources: {}
      volumeMounts:
        - name: ncop-config-apache
          mountPath: /opt/nfast/kmdata/config
        - name: ncop-hardserver
          mountPath: /opt/nfast/kmdata/hardserver.d
        - name: ncop-sockets
          mountPath: /opt/nfast/sockets
        - name: ncop-sockets-priv
          mountPath: /opt/nfast/sockets-priv
    - name: nshield-apache-pkcs11
      image: >-
        registry.eselab.net/cv-nshield-apache-pkcs11-redhat
      imagePullPolicy: Always
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      ports:
        - containerPort: 9443
          protocol: TCP
      resources: {}
      volumeMounts:
        - name: ncop-sockets
          mountPath: /opt/nfast/sockets
  securityContext: {}
  volumes:
    - configMap:
        name: ncop-config-apache
        defaultMode: 420
      name: ncop-config-apache
    - name: ncop-hardserver
      emptyDir: {}
    - name: ncop-sockets
      emptyDir: {}
    - name: ncop-sockets-priv
      emptyDir: {}

Deploy the application

The steps below are performed in the Kubernetes server. This procedure requires that the Kubernetes environment is already created.

  1. Set the environment.

    1. CodeReady OpenShift Environment

      This step assumes a CodeReady OpenShift environment. Perform what is required from your Kubernetes environment. For example:

      % eval $(crc oc-env)

      Log into OpenShift:

      % oc login -u kubeadmin https://api.crc.testing:6443

    2. Openshift Cluster

      % export KUBECONFIG=/path/to/the/cluster/kubeconfig/file

  2. Change directory to where the YAML files are located:

    % cd yaml

  3. Create the project:

    % oc create -f project.yaml

project.project.openshift.io/apache created

  4. Change from the current project to the newly created project:

    % oc project apache

Now using project "apache" on server "https://api.crc.testing:6443".

  5. Create the namespace (CodeReady Only)

    % oc create namespace apache

namespace/apache created

  6. Create the registry secret:

    % oc create -f secret.yaml

secret/regcred created

  7. Create the config map:

    % oc create -f cm.yaml

configmap/ncop-config-apache created

  8. Deploy the Apache application containers:

    % oc create -f pod_apache.yaml

pod/nshield-apache-example-9nl74 created

    With the latest security policy changes on Openshift 4.11.20, you may see the following warning which does not affect the pod deployment:

    Warning: would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "test-nshield-apache-pkcs11" must set securityContext.allowPri\
vilegeEscalation=false), unrestricted capabilities (container "test-nshield-apache-pkcs11" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != tr\
ue (pod or containers "test-nshield-apache-pkcs11", "nshield-hwsp-pkcs11", "nshield-apache-pkcs11" must set securityContext.runAsNonRoot=true), seccompProfile (po\
d or containers "test-nshield-apache-pkcs11", "nshield-hwsp-pkcs11", "nshield-apache-pkcs11" must set securityContext.seccompProfile.type to "RuntimeDefault" or "\
Localhost")

  9. List the containers in the pod:

    % oc get pods nshield-apache-example-9nl74 -n apache -o jsonpath="{.spec.containers[*].image}"

registry.eselab.net/cv-test-nshield-apache-pkcs11-redhat registry.eselab.net/cv-nshield-hwsp-pkcs11-redhat registry.eselab.net/cv-nshield-apache-pkcs11-redhat

  10. Verify Apache is running:

    % oc get pods --all-namespaces | grep apache

apache      nshield-apache-example-9nl74     3/3     Running     0             5m

  11. Open a debug session on the pod so you can check the Apache application:

    % oc debug pod/nshield-apache-example-9nl74

Defaulting container name to test-nshield-apache-pkcs11.
Use 'oc describe pod/nshield-apache-example-9nl74-debug -n apache' to see all of the containers in this pod.

Starting pod/nshield-apache-example-9nl74-debug, command was: sh -c sleep 7200
Pod IP: 10.217.0.118
If you don't see a command prompt, try pressing enter.
sh-4.4#

  12. In the debug session, run the osc.sh script to check the connection with the Apache server that is running in the container.

    You should see that it is using the key created when you created the Docker container. In this case, this is the apachecontainerkey key.

    sh-4.4# cd /opt/test/osc.sh

CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = apachecontainerkey
verify return:1
---
Certificate chain
 0 s:CN = apachecontainerkey
   i:CN = apachecontainerkey
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICwTCCAakCFAml778s/6aHXzKK84DsM6cvO2AhMA0GCSqGSIb3DQEBCwUAMB0x
GzAZBgNVBAMMEmFwYWNoZWNvbnRhaW5lcmtleTAeFw0yMTEyMjAxNjQwMjFaFw0y
MjEyMjAxNjQwMjFaMB0xGzAZBgNVBAMMEmFwYWNoZWNvbnRhaW5lcmtleTCCASIw
DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ48W0IR6OD5wcMuQv/aYCWnYa8u
htz+Pq2UiXcmwhd9fSSkeVGHVEOP5+dtdJKjI6DTMwWeFg+u515HLM+rteeQCicD
noXEm/WV8HRffNknznHtKMiOW75UNcsOxuavc0hKCdn/q/Yn5AsBQTbjmUSFQhDQ
Z4saNuN6PSgNJh/grjZilEhuRyh4neAEyiNMBIULXecbnBpoMnMtQLm6PT1IRZQC
QQ3L8yONTPXqiap/hcSdUxOZjaTwL7oDsnTaK42eXOx2MytfSMULw+KTwnCrKYNc
+DLiRCMJsVbdHwFahjMsidnLgheBnc7BN5LAx5rH0a2J1G9lDQv/5UFa+4sCAwEA
ATANBgkqhkiG9w0BAQsFAAOCAQEAaTAWb/jaEicXCvV/80IyDnSyBHSyDT3yKMrW
dEn4O6lUgEwo1tU1MgNHPr7ripKl2UQ4GS7vrntWU7d9k7Rzq2QQB5NkhvC1xzsB
RV/miBKTd4cNGMxfB+zmEWpv82nMKsWjfQmbwPVXoIHcprpCNrJKMBOIempbOsGo
i9Y2xBxJsT8HGZ027j1rB3yMmZHSBFMRHuDRGlPMlFbDmnzsC/6VMvB+iNMA1fgs
az3bKQMYJ7Q3BzwKWV730tAyOcVVk79aRRQBLpohjCBkJqsepuRMP2EP/aPkNJ9S
sgeFjwyrsyZE4SImJQORAThXwDbYbC5cbTknOVLCLkidg+Gfuw==
-----END CERTIFICATE-----
subject=CN = apachecontainerkey

issuer=CN = apachecontainerkey

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1265 bytes and written 369 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 3FB4E1ACACEB5FC796DBB2EAC0BC92EBFC795AF7199BDAE680CAEBB606DE5D4F
    Session-ID-ctx:
    Resumption PSK: A41DA3DEBDE08228F09051E6671BCC1D3C0F715AF538D5476342067CDC3BFA717F88A1D3F9AE3DAA8F4AE3B954ADBAD3
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 29 42 88 f9 42 e6 a2 6c-5e c8 82 21 d1 4d 9d 8e   )B..B..l^..!.M..
    0010 - 58 0f 9d 98 7d 1d 9f cd-b7 1f 44 9f 69 7c b5 c3   X...}.....D.i|..
    0020 - 0c 47 e4 18 db a5 7e d4-93 81 d2 46 a1 51 b8 97   .G....~....F.Q..
    0030 - 50 cb d7 af ee 1a 4f 07-4c 20 5e 15 ca 7b ce e0   P.....O.L ^..{..
    0040 - a8 9f fe 9e 6e 4c 21 21-43 e1 ca ce 0a 79 e0 f2   ....nL!!C....y..
    0050 - 76 1c a6 e5 b9 ac eb 77-05 52 7a b0 1a 78 e8 4c   v......w.Rz..x.L
    0060 - 28 67 6c ef 01 5a 29 04-c8 b6 eb 17 b7 2c a5 8b   (gl..Z)......,..
    0070 - a7 5b da 33 eb dd 95 3d-58 05 03 b8 43 72 c8 70   .[.3...=X...Cr.p
    0080 - cd b1 3c e4 fa 95 92 d4-6f a8 49 4f b3 03 02 53   ..<.....o.IO...S
    0090 - 42 6c 7c d5 62 82 ee 01-80 09 cc 44 09 53 e0 25   Bl|.b......D.S.%
    00a0 - 98 7a cf 55 9f 2a fb 14-c1 9f ea 20 a3 cc 31 d2   .z.U.*..... ..1.
    00b0 - 8c 8d bd 09 4c 30 c2 07-83 b5 c2 5a 34 18 ce 74   ....L0.....Z4..t
    00c0 - b8 66 35 1f 6d 41 a3 8a-aa d4 cb e8 97 4e b4 09   .f5.mA.......N..
    00d0 - 76 8e 38 5b 9b a4 76 fa-0c 6d 01 a3 dd f6 7a 83   v.8[..v..m....z.

    Start Time: 1640110099
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 65EEA38183B8D613B0E866114C0CF84CE8A8EFE0879D137083607FD28251DFC6
    Session-ID-ctx:
    Resumption PSK: E5E878DCC69F627043889B7A2CCFDEEE753F16B4F2BDA9A28670E01D9459BBBEE10380F5304784D7D38981CF40F86353
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 29 42 88 f9 42 e6 a2 6c-5e c8 82 21 d1 4d 9d 8e   )B..B..l^..!.M..
    0010 - 99 eb 3f 7b 17 11 3b e6-3c 7c 83 44 80 dd 4d 1b   ..?{..;.<|.D..M.
    0020 - 75 fe bf 5a c0 34 5b ca-07 1a fa 54 1e 41 be 8c   u..Z.4[....T.A..
    0030 - 85 ed 87 a4 c1 21 5b 88-fa f0 71 b3 ab d3 2d d1   .....![...q...-.
    0040 - ec 69 99 6c 77 94 5e e9-11 8d 9c c6 9d 2f d7 c2   .i.lw.^....../..
    0050 - a8 a1 5c 70 75 18 80 36-bd 08 27 79 98 57 94 91   ..\pu..6..'y.W..
    0060 - 6e 4c f5 6c d5 aa aa 8e-57 a2 dd e6 ca 5e 43 76   nL.l....W....^Cv
    0070 - f5 30 48 95 30 8b 1b 93-af 71 15 5f c2 7e ee 4e   .0H.0....q._.~.N
    0080 - 10 83 f6 4a 1c b1 88 54-f8 d0 73 63 57 ed 63 37   ...J...T..scW.c7
    0090 - 26 ed 48 4b b4 c9 cf ff-fc d6 2f de df 40 ee e6   &.HK....../..@..
    00a0 - c7 41 1d 92 9f de fd a9-c0 a8 40 d0 69 22 cb 74   .A........@.i".t
    00b0 - 08 0b b2 60 70 d2 02 3c-8a a0 4c 49 48 bb c9 0a   ...`p..<..LIH...
    00c0 - 52 cd 97 ea 69 3b 1d f8-65 56 f1 32 f0 5d df a1   R...i;..eV.2.]..
    00d0 - 44 d7 f5 73 a5 a4 fa 9f-62 09 8b c5 c3 4a 26 cf   D..s....b....J&.

    Start Time: 1640110099
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

  13. In the debug session, run the abi.sh script to check the performance of the Apache server running in the container:

    sh-4.4# /opt/test/abi.sh

This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        Apache/2.4.37
Server Hostname:        localhost
Server Port:            9443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        localhost

Document Path:          /icons/apache_pb2.gif
Document Length:        4234 bytes

Concurrency Level:      50
Time taken for tests:   87.555 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      45170000 bytes
HTML transferred:       42340000 bytes
Requests per second:    114.21 [#/sec] (mean)
Time per request:       437.775 [ms] (mean)
Time per request:       8.755 [ms] (mean, across all concurrent requests)
Transfer rate:          503.81 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       49  434 476.9    262    5149
Processing:     0    1   4.9      1     242
Waiting:        0    1   4.8      0     241
Total:         49  435 477.0    264    5149

Percentage of the requests served within a certain time (ms)
  50%    264
  66%    434
  75%    569
  80%    658
  90%   1036
  95%   1389
  98%   1895
  99%   2255
 100%   5149 (longest request)

The integration of Apache using PKC #11 running inside a Kubernetes container is now complete.