Test Integration

This procedure requires test scripts obtained from NetApp. Send the output files generated by the test scripts to NetApp for verification.

Load the test scripts into NetApp ONTAP

  1. Open a command window and sign in to the NetApp ONTAP Cluster Management.

  2. Unlock and set the password for the diag user:

    mycluster::> security login unlock -username diag
mycluster::> security login password -username diag

  3. Set diagnostics:

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

  4. Enter system shell.

    Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

  5. Copy the test script files from a server of your choice into the Systemshell of the NetApp ONTAP node.

    Provide the password when prompted.

    mycluster-01% scp root@xx.xxx.xxx.xxx:/root/Downloads/kmip_before_reboot_test.sh .
kmip_before_reboot_test.sh                                                            100% 7346   731.0KB/s   00:00
SSH terminating : scp.c : main : 690,errs = 0.

mycluster-01% scp root@xx.xxx.xxx.xxx:/root/Downloads/kmip_post_reboot_test.sh .
kmip_post_reboot_test.sh                                                              100% 6047     3.6MB/s   00:00
SSH terminating : scp.c : main : 690,errs = 0.
    The test scripts were provided by NetApp.

  6. Verify the test script files are in the current directory:

    mycluster-01% ls
kmip_before_reboot_test.sh      kmip_post_reboot_test.sh

Execute the kmip_before_reboot_test.sh test script

  1. Open a command window and sign in to the NetApp ONTAP Cluster Management.

  2. Set diagnostics:

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

  3. Enter Systemshell:

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

  4. Execute the kmip_before_reboot_test.sh test script and redirect the output to kmip_before_reboot_test.txt.

    Cryptographic Security Platform Key Management Vault presents itself as a single entity even though it may be composed of multiple nodes (two in this test case). If the Please enter whether this is a clustered key-server config prompt appears, enter no.

    mycluster-01% bash kmip_before_reboot_test.sh | tee kmip_before_reboot_test.txt

Please enter key server name: CSP Vault
Please enter key server version: 10.5.3
Please enter whether this is a clustered key-server config (yes or no): no
    Example output 
    Executing script kmip_before_reboot_test - version 2.3
Testing DOT: NetApp Release 9.17.1P9: Wed Jun 24 13:09:36 UTC 2026 <1O>
 with Key Manager: CSP Vault 10.5.3
Tesing with clustered key servers: no
Step 1 - Get local node name
Local node name is mycluster-01
Step 2 - Get admin vserver name where EKM is configured
Admin vserver name is mycluster
Step 3 - Check if key-servers are registered
Key server is configured and status is available

                Node: mycluster-01
             Vserver: mycluster
     Key Server Port: 5696
 KMIP is operational: true

Key Server           Role           Server Status     Reason
-------------------  ------------   ---------------   ------
xx.xxx.xxx.xxx       primary        available         -

Clustered key servers are not configured as expected
Step 4 - Turn on logging for key management

343 entries were modified.

Step 5 - Enable KMIP logging for key management

1 entry was modified.

Step 6 - Create data storage aggregate - test_aggr
[Job 32] Job succeeded: DONE

Sleeping for 10 seconds before checking if aggregate was created...
Step 7 - Verify aggregate exists
Aggregate was created successfully.
Step 8 - Create data vserver - test_vserver
[Job 33] Sleeping for 10 seconds before checking if vserver was created...
[Job 33] Job succeeded:
Vserver creation completed.

Step 9 - Verify vserver exists
Vserver was created successfully.
Step 10 - Create 2 encrypted volumes
[Job 34] Job succeeded: Successful

[Job 35] Job succeeded: Successful

Step 11 - Verify encrypted volumes are online
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
test_vserver test_vol_1 test_aggr   online     RW         20MB    18.75MB    1%
test_vserver test_vol_2 test_aggr   online     RW         20MB    18.78MB    1%
2 entries were displayed.

Volume test_vol_1 was created successfully.
Volume test_vol_2 was created successfully.
Step 12 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xxx:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
6101d2fb-8443-11f1-b8da-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050038c0edc571a3036eced32d03f3588a490000000000000000
5f12713a-8443-11f1-b8da-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500b66f9fb3479471d9b7ba982e3da265b70000000000000000
2 entries were displayed.

Step 13 - Create NSE key
NSE key id is  00000000000000000200000000000100fe73b73aee19dec2141a0d72066bca0f0000000000000000
Step 14 - Get the NSE key
NSE key id is 00000000000000000200000000000100fe73b73aee19dec2141a0d72066bca0f0000000000000000
Step 15 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xxx:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 00000000000000000200000000000100fe73b73aee19dec2141a0d72066bca0f0000000000000000
6101d2fb-8443-11f1-b8da-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050038c0edc571a3036eced32d03f3588a490000000000000000
5f12713a-8443-11f1-b8da-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500b66f9fb3479471d9b7ba982e3da265b70000000000000000
3 entries were displayed.

Step 16 - Run debug smdb table cryptomodKeyTable show
cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                                                                                              key-type    key-digest
------------ --------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------- ----------- ----------------------------------------------------------------
mycluster-01 0         00000000000000000200000000000500b66f9fb3479471d9b7ba982e3da265b70000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 b50d922347f7a4c01b801bc5c501178636b9d465e8d609caa77d98fc30c823c1
mycluster-01 1         0000000000000000020000000000050038c0edc571a3036eced32d03f3588a490000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 8c2d9619a2d2c1e39a0c4d1dc6302f576c562d177284b1b3c912a9c02193cf3b
mycluster-01 2         00000000000000000200000000000100fe73b73aee19dec2141a0d72066bca0f0000000000000000 0000000000000000000000000000000000000000000000000000000000000000                                                                 NSE-AK      4f69121ddb4418e13eb64b4374f1c3a2747d144e59751645aefee01f4b605fa0
3 entries were displayed.

Step 17 - Check if key-servers are registered
Key server is configured and status is available
Step 18 - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
xx.xxx.xxx.xxx:5696.host=xx.xxx.xxx.xxx
xx.xxx.xxx.xxx:5696.port=5696
xx.xxx.xxx.xxx:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xxx:5696.protocol=KMIP1_4
xx.xxx.xxx.xxx:5696.timeout=25
xx.xxx.xxx.xxx:5696.nbio=1
xx.xxx.xxx.xxx:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xxx:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xxx:5696.ciphers="TLSv1.2:kRSA:!CAMELLIA:!IDEA:!RC2:!RC4:!SEED:!eNULL:!aNULL"
xx.xxx.xxx.xxx:5696.verify=true
xx.xxx.xxx.xxx:5696.netapp_keystore_uuid=c8cad565-8442-11f1-b8da-005056b1584e

Step 19 - Get output of /cfcard/kmip/kmipcmd.log file
KmipDiscoverVersions succeeded
Step 20 - Turn on AUTOBOOT

  (system node systemshell)

Node: mycluster-01
AUTOBOOT="true"
1 entry was acted on.

Manually reboot the local node and wait 10 minutes before logging back and in running kmip_post_reboot_test.sh

  5. Exit Systemshell:

    mycluster-01% exit

  6. Reboot the node.

    Wait 10 minutes before signing in to the cluster again.

    mycluster::*> reboot -node mycluster-01

Execute the kmip_post_reboot_test.sh test script

  1. Open a command window and sign in to the NetApp ONTAP Cluster Management.

  2. Set diagnostics:

    mycluster::> set diag

  3. Enter Systemshell.

    Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01

  4. Execute the kmip_post_reboot_test.sh test script and redirect the output to kmip_post_reboot_test.txt:

    mycluster-01% bash kmip_post_reboot_test.sh | tee kmip_post_reboot_test.txt

Please enter key server name: CSP Vault
Please enter key server version: 10.5.3
Please enter whether this is a clustered key-server config (yes or no): no
    Example output 
    Executing script kmip_post_reboot_test - version 2.3
Testing DOT: NetApp Release 9.17.1P9: Wed Jun 24 13:09:36 UTC 2026 <1O>
 with Key Manager: CSP Vault 10.5.3
Tesing with clustered key servers: no
Step 1 - Get local node name
Local node name is mycluster-01
Step 2 - Get admin vserver name where EKM is configured
Admin vserver name is mycluster
Step 3 - Check if key-servers are registered
Key server is configured and status is available

                Node: mycluster-01
             Vserver: mycluster
     Key Server Port: 5696
 KMIP is operational: true

Key Server           Role           Server Status     Reason
-------------------  ------------   ---------------   ------
xx.xxx.xxx.xxx       primary        available         -

Clustered key servers are not configured as expected
Step 4 - Post Reboot - Verify encrypted volumes are online
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
test_vserver test_vol_1 test_aggr   online     RW         20MB    18.74MB    1%
test_vserver test_vol_2 test_aggr   online     RW         20MB    18.75MB    1%
2 entries were displayed.

Volume test_vol_1 is online as expected.
Volume test_vol_2 is online as expected.
Step 5 - Post Reboot - Get the NSE key
NSE key id is 00000000000000000200000000000100fe73b73aee19dec2141a0d72066bca0f0000000000000000
Step 6 - Post Reboot - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xxx:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 00000000000000000200000000000100fe73b73aee19dec2141a0d72066bca0f0000000000000000
6101d2fb-8443-11f1-b8da-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050038c0edc571a3036eced32d03f3588a490000000000000000
5f12713a-8443-11f1-b8da-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500b66f9fb3479471d9b7ba982e3da265b70000000000000000
3 entries were displayed.

Step 7 - Post Reboot - Run debug smdb table cryptomodKeyTable show
cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                              key-type key-digest
------------ --------- -------------------------------------------------------------------------------- ---------------------------------------------------------------- -------- ----------------------------------------------------------------
mycluster-01 0         00000000000000000200000000000100fe73b73aee19dec2141a0d72066bca0f0000000000000000 0000000000000000000000000000000000000000000000000000000000000000 NSE-AK   4f69121ddb4418e13eb64b4374f1c3a2747d144e59751645aefee01f4b605fa0
mycluster-01 1         00000000000000000200000000000500b66f9fb3479471d9b7ba982e3da265b70000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 b50d922347f7a4c01b801bc5c501178636b9d465e8d609caa77d98fc30c823c1
mycluster-01 2         0000000000000000020000000000050038c0edc571a3036eced32d03f3588a490000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 8c2d9619a2d2c1e39a0c4d1dc6302f576c562d177284b1b3c912a9c02193cf3b
3 entries were displayed.

Step 8 - Post Reboot - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
xx.xxx.xxx.xxx:5696.host=xx.xxx.xxx.xxx
xx.xxx.xxx.xxx:5696.port=5696
xx.xxx.xxx.xxx:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xxx:5696.protocol=KMIP1_4
xx.xxx.xxx.xxx:5696.timeout=25
xx.xxx.xxx.xxx:5696.nbio=1
xx.xxx.xxx.xxx:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xxx:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xxx:5696.ciphers="TLSv1.2:kRSA:!CAMELLIA:!IDEA:!RC2:!RC4:!SEED:!eNULL:!aNULL"
xx.xxx.xxx.xxx:5696.verify=true
xx.xxx.xxx.xxx:5696.netapp_keystore_uuid=c8cad565-8442-11f1-b8da-005056b1584e

Step 9 - Post Reboot - Compare /cfcard/kmip/servers.cfg files
The /cfcard/kmip/servers.cfg output before reboot is the same after rebooting
Step 10 - Post Reboot - Delete the NSE key


Step 11 - Post Reboot - Delete the encrypted volumes

[Job 38] Job succeeded: Successful
[Job 39] Job succeeded: Successful
2 entries were acted on.

Step 12 - Post Reboot - Delete the data vserver - test_vserver
[Job 40]
Step 13 - Post Reboot - Delete the data aggregate - test_aggr
[Job 42] Job succeeded: DONE

Step 14 - Turn off logging for key management

343 entries were modified.

Step 15 - Enable KMIP logging for key management

1 entry was modified.

Step 16 - Post Reboot - Verify no keys are observed in key query
No keys are on the cluster as expected.

  5. Exit Systemshell:

    mycluster-01% exit

Enable FIPS mode

  1. Open a command window and sign in to the NetApp ONTAP Cluster Management.

  2. Set diagnostics:

    mycluster::> set diag

  3. Enable FIPS mode:

    mycluster::*> security config modify -interface SSL -is-fips-enabled true

Warning: This command will enable FIPS compliance and can potentially cause some non-compliant components to fail.
         MetroCluster and Vserver DR require FIPS to be enabled on both sites in order to be compatible. An SNMP users
         or SNMP traphosts that are non-compliant to FIPS will be deleted automatically. An SNMPv1 user, SNMPv2c user
         or SNMPv3 user (with none or MD5 as authentication protocol or none or DES as encryption protocol or both) is
         non-compliant to FIPS. An SNMPv1 traphost or SNMPv3 traphost (configured with an SNMPv3 user non-compliant to
         FIPS) is non-compliant to FIPS.
Do you want to continue? {y|n}: y

  4. Reboot all nodes in the cluster.

    Wait 10 minutes before signing in to the cluster again.

    mycluster::*> reboot -node *
  (system node reboot)

Warning: Are you sure you want to reboot node "mycluster-01"? {y|n}: Y
1 entry was acted on.

Connection to xx.xxx.xxx.xxx closed.

  5. Sign in to the NetApp ONTAP Cluster Management again.

  6. Set diagnostics:

    mycluster::> set diag

  7. Verify that FIPS mode is enabled:

    mycluster::*> security config show
Cluster    Supported
FIPS Mode  Protocols Supported Cipher Suites
---------- --------- ----------------------------------------------------------
true       TLSv1.3,  TLS_RSA_WITH_AES_128_CCM, TLS_RSA_WITH_AES_128_CCM_8,
           TLSv1.2   TLS_RSA_WITH_AES_128_GCM_SHA256,
                     TLS_RSA_WITH_AES_128_CBC_SHA,
                     TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CCM,

 ...

                     TLS_SRP_SHA_DSS_WITH_AES_256_CBC_SHA,
                     TLS_SRP_SHA_RSA_WITH_AES_128_CBC_SHA,
                     TLS_SRP_SHA_RSA_WITH_AES_256_CBC_SHA,
                     TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384

Execute the before and post test scripts a second time

  1. Open a command window and sign in to the NetApp ONTAP Cluster Management.

  2. Set diagnostics:

    mycluster::> set diag

  3. Enter Systemshell.

    Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01

  4. Execute the kmip_before_reboot_test.sh test script and redirect the output to kmip_before_reboot_test_fips.txt.

    mycluster-01% bash kmip_before_reboot_test.sh | tee kmip_before_reboot_test_fips.txt

Please enter key server name: CSP Vault
Please enter key server version: 10.5.3
Please enter whether this is a clustered key-server config (yes or no): no
    Example output 
    Executing script kmip_before_reboot_test - version 2.3
Testing DOT: NetApp Release 9.17.1P9: Wed Jun 24 13:09:36 UTC 2026 <1O>
 with Key Manager: CSP Vault 10.5.3
Tesing with clustered key servers: no
Step 1 - Get local node name
Local node name is mycluster-01
Step 2 - Get admin vserver name where EKM is configured
Admin vserver name is mycluster
Step 3 - Check if key-servers are registered
Key server is configured and status is available

                Node: mycluster-01
             Vserver: mycluster
     Key Server Port: 5696
 KMIP is operational: true

Key Server           Role           Server Status     Reason
-------------------  ------------   ---------------   ------
xx.xxx.xxx.xxx       primary        available         -

Clustered key servers are not configured as expected
Step 4 - Turn on logging for key management

343 entries were modified.

Step 5 - Enable KMIP logging for key management

1 entry was modified.

Step 6 - Create data storage aggregate - test_aggr
[Job 45] Job succeeded: DONE

Sleeping for 10 seconds before checking if aggregate was created...
Step 7 - Verify aggregate exists
Aggregate was created successfully.
Step 8 - Create data vserver - test_vserver
[Job 46]
Vserver creation completed.Sleeping for 10 seconds before checking if vserver was created...
[Job 46] Job succeeded:
Vserver creation completed.

Step 9 - Verify vserver exists
Vserver was created successfully.
Step 10 - Create 2 encrypted volumes
[Job 47] Job succeeded: Successful

[Job 48] Job succeeded: Successful

Step 11 - Verify encrypted volumes are online
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
test_vserver test_vol_1 test_aggr   online     RW         20MB    18.76MB    1%
test_vserver test_vol_2 test_aggr   online     RW         20MB    18.78MB    1%
2 entries were displayed.

Volume test_vol_1 was created successfully.
Volume test_vol_2 was created successfully.
Step 12 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xxx:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
f3eed91d-8445-11f1-8792-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050071e8a5cdfad40ebd604f6995b14381350000000000000000
f2084ec4-8445-11f1-8792-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500ad6132942b75b7b577a3064c840507540000000000000000
2 entries were displayed.

Step 13 - Create NSE key
NSE key id is  00000000000000000200000000000100c73d13795c2040a1abbfc5d48444bfe40000000000000000
Step 14 - Get the NSE key
NSE key id is 00000000000000000200000000000100c73d13795c2040a1abbfc5d48444bfe40000000000000000
Step 15 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xxx:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 00000000000000000200000000000100c73d13795c2040a1abbfc5d48444bfe40000000000000000
f3eed91d-8445-11f1-8792-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050071e8a5cdfad40ebd604f6995b14381350000000000000000
f2084ec4-8445-11f1-8792-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500ad6132942b75b7b577a3064c840507540000000000000000
3 entries were displayed.

Step 16 - Run debug smdb table cryptomodKeyTable show
cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                                                                                              key-type    key-digest
------------ --------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------- ----------- ----------------------------------------------------------------
mycluster-01 0         00000000000000000200000000000500ad6132942b75b7b577a3064c840507540000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 844c23e85be1f4dadf882bd917a41a40238eb04550ad79706971fc545fa65378
mycluster-01 1         0000000000000000020000000000050071e8a5cdfad40ebd604f6995b14381350000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 cbc0de60041c27c9b22e2722ec9b3e554587af62bb8a1f0ab606d4fb5462f761
mycluster-01 2         00000000000000000200000000000100c73d13795c2040a1abbfc5d48444bfe40000000000000000 0000000000000000000000000000000000000000000000000000000000000000                                                                 NSE-AK      a2b1e0c59acbaff10a4db7a70985b6b5f67441359bc4eabc24b740879ed5861c
3 entries were displayed.

Step 17 - Check if key-servers are registered
Key server is configured and status is available
Step 18 - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
xx.xxx.xxx.xxx:5696.host=xx.xxx.xxx.xxx
xx.xxx.xxx.xxx:5696.port=5696
xx.xxx.xxx.xxx:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xxx:5696.protocol=KMIP1_4
xx.xxx.xxx.xxx:5696.timeout=25
xx.xxx.xxx.xxx:5696.nbio=1
xx.xxx.xxx.xxx:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xxx:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xxx:5696.ciphers="TLSv1.2+FIPS:!eNULL:!aNULL"
xx.xxx.xxx.xxx:5696.verify=true
xx.xxx.xxx.xxx:5696.netapp_keystore_uuid=c8cad565-8442-11f1-b8da-005056b1584e

Step 19 - Get output of /cfcard/kmip/kmipcmd.log file
KmipDiscoverVersions succeeded
Step 20 - Turn on AUTOBOOT

  (system node systemshell)

Node: mycluster-01
AUTOBOOT="true"
1 entry was acted on.

Manually reboot the local node and wait 10 minutes before logging back and in running kmip_post_reboot_test.sh

  5. Exit Systemshell:

    mycluster-01% exit

  6. Reboot the node.

    Wait 10 minutes before signing in to the cluster again.

    mycluster::*> reboot -node mycluster-01

  7. Sign in to the NetApp ONTAP Cluster Management again.

  8. Set diagnostics:

    mycluster::> set diag

  9. Enter Systemshell:

    mycluster::*> systemshell -node mycluster-01

  10. Execute the kmip_post_reboot_test.sh test script and redirect the output to kmip_post_reboot_test_fips.txt:

    mycluster-01% bash kmip_post_reboot_test.sh | tee kmip_post_reboot_test_fips.txt
    Example output 
    Volume test_vol_1 is online as expected.
Volume test_vol_2 is online as expected.
Step 5 - Post Reboot - Get the NSE key
NSE key id is 00000000000000000200000000000100c73d13795c2040a1abbfc5d48444bfe40000000000000000
Step 6 - Post Reboot - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xxx:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 00000000000000000200000000000100c73d13795c2040a1abbfc5d48444bfe40000000000000000
f3eed91d-8445-11f1-8792-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050071e8a5cdfad40ebd604f6995b14381350000000000000000
f2084ec4-8445-11f1-8792-005056b1584e  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500ad6132942b75b7b577a3064c840507540000000000000000
3 entries were displayed.

Step 7 - Post Reboot - Run debug smdb table cryptomodKeyTable show
cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                              key-type key-digest
------------ --------- -------------------------------------------------------------------------------- ---------------------------------------------------------------- -------- ----------------------------------------------------------------
mycluster-01 0         00000000000000000200000000000100c73d13795c2040a1abbfc5d48444bfe40000000000000000 0000000000000000000000000000000000000000000000000000000000000000 NSE-AK   a2b1e0c59acbaff10a4db7a70985b6b5f67441359bc4eabc24b740879ed5861c
mycluster-01 1         0000000000000000020000000000050071e8a5cdfad40ebd604f6995b14381350000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 cbc0de60041c27c9b22e2722ec9b3e554587af62bb8a1f0ab606d4fb5462f761
mycluster-01 2         00000000000000000200000000000500ad6132942b75b7b577a3064c840507540000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 844c23e85be1f4dadf882bd917a41a40238eb04550ad79706971fc545fa65378
3 entries were displayed.

Step 8 - Post Reboot - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
xx.xxx.xxx.xxx:5696.host=xx.xxx.xxx.xxx
xx.xxx.xxx.xxx:5696.port=5696
xx.xxx.xxx.xxx:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xxx:5696.protocol=KMIP1_4
xx.xxx.xxx.xxx:5696.timeout=25
xx.xxx.xxx.xxx:5696.nbio=1
xx.xxx.xxx.xxx:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xxx:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xxx:5696.ciphers="TLSv1.2+FIPS:!eNULL:!aNULL"
xx.xxx.xxx.xxx:5696.verify=true
xx.xxx.xxx.xxx:5696.netapp_keystore_uuid=c8cad565-8442-11f1-b8da-005056b1584e

Step 9 - Post Reboot - Compare /cfcard/kmip/servers.cfg files
The /cfcard/kmip/servers.cfg output before reboot is the same after rebooting
Step 10 - Post Reboot - Delete the NSE key


Step 11 - Post Reboot - Delete the encrypted volumes

[Job 51] Job succeeded: Successful
[Job 52] Job succeeded: Successful
2 entries were acted on.

Step 12 - Post Reboot - Delete the data vserver - test_vserver
[Job 53]
Step 13 - Post Reboot - Delete the data aggregate - test_aggr
[Job 55] Job succeeded: DONE

Step 14 - Turn off logging for key management

343 entries were modified.

Step 15 - Enable KMIP logging for key management

1 entry was modified.

Step 16 - Post Reboot - Verify no keys are observed in key query
No keys are on the cluster as expected.

  11. Copy the test script output files to a server of your choice:

    mycluster-01% scp *.txt root@xxx.xxx.xxx.xxx:/root/Downloads/.

kmip_before_reboot_test.txt                                                                                                                                                   100%   16KB   4.9MB/s   00:00
kmip_before_reboot_test_fips.txt                                                                                                                                              100%   14KB   7.3MB/s   00:00
kmip_post_reboot_test.txt                                                                                                                                                     100%   14KB   9.5MB/s   00:00
kmip_post_reboot_test_fips.txt                                                                                                                                                100%   14KB  15.0MB/s   00:00
SSH terminating : scp.c : main : 690,errs = 0.

  12. Send these output files to NetApp for verification.

Verify FIPS mode is unchanged after reboot

  1. Exit Systemshell:

    mycluster-01% exit

  2. Disable FIPS mode:

    mycluster::*> security config modify -interface SSL -is-fips-enabled false

  3. Reboot all nodes in the cluster:

    mycluster::*> reboot -node *

  4. Sign in to the NetApp ONTAP Cluster Management again.

  5. Set diagnostics:

    mycluster::> set diag

  6. Verify that FIPS mode is disabled on the cluster:

    mycluster::*> security config show
Cluster    Supported
FIPS Mode  Protocols Supported Cipher Suites
---------- --------- ----------------------------------------------------------
false      TLSv1.3,  TLS_RSA_WITH_AES_128_CCM, TLS_RSA_WITH_AES_128_CCM_8,
           TLSv1.2   TLS_RSA_WITH_AES_128_GCM_SHA256,
                     TLS_RSA_WITH_AES_128_CBC_SHA,
                     TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CCM,

 ...

                     TLS_SRP_SHA_RSA_WITH_AES_128_CBC_SHA,
                     TLS_SRP_SHA_RSA_WITH_AES_256_CBC_SHA,
                     TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384,
                     TLS_CHACHA20_POLY1305_SHA256