Test integration

This test procedure requires test scripts available from NetApp. The output files resulting from executing the test scripts need to be sent back to NetApp for verification.

Load the test scripts into NetApp ONTAP

  1. Open a command window and remote login into the NetApp ONTAP Cluster Management.

  2. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  3. Enter system shell.

    Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

mycluster-01%

  4. Copy the test script files from a server of your choice into the Systemshell of the NetApp ONTAP node.

    Provide the password when prompted.

    mycluster-01% scp root@xx.xxx.xxx.xxx:/root/Downloads/kmip_before_reboot_test.sh .
kmip_before_reboot_test.sh                                                            100% 7346   731.0KB/s   00:00
SSH terminating : scp.c : main : 690,errs = 0.

mycluster-01% scp root@xx.xxx.xxx.xxx:/root/Downloads/kmip_post_reboot_test.sh .
kmip_post_reboot_test.sh                                                              100% 6047     3.6MB/s   00:00
SSH terminating : scp.c : main : 690,errs = 0.
    The test scripts were provided by NetApp.

  5. Verify the test scripts files are in the current directory.

    mycluster-01% ls
kmip_before_reboot_test.sh      kmip_post_reboot_test.sh

Execute the kmip_before_reboot_test.sh test script

  1. Open a command window and remote login into the NetApp ONTAP Cluster Management.

  2. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  3. Enter Systemshell.

    Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

mycluster-01%

  4. Execute the kmip_before_reboot_test.sh test script and redirect the output to file kmip_before_reboot_test.txt.

    KeyControl presents itself as a single entity even though it may be composed of multiple nodes (two in this test case). Therefore, select no if the Please enter whether this is a clustered key-server config (yes or no): question is shown.

    mycluster-01% bash kmip_before_reboot_test.sh | tee kmip_before_reboot_test.txt

Please enter key server name: KeyControl
Please enter key server version: 10.4.3
Please enter whether this is a clustered key-server config (yes or no): no
Executing script kmip_before_reboot_test - version 2.3
Testing DOT: NetApp Release 9.16.1P3: Thu Apr 24 02:50:10 UTC 2025 <1O>
 with Key Manager: KeyControl 10.4.3
Tesing with clustered key servers: no
Step 1 - Get local node name
Local node name is mycluster-01
Step 2 - Get admin vserver name where EKM is configured
Admin vserver name is mycluster
Step 3 - Check if key-servers are registered
Key server is configured and status is available

                Node: mycluster-01
             Vserver: mycluster
     Key Server Port: 5696
 KMIP is operational: true

Key Server           Role           Server Status     Reason
-------------------  ------------   ---------------   ------
XX.XXX.XXX.XX6       primary        available         -

Clustered key servers are not configured as expected
                                                    Step 4 - Turn on logging for key management

284 entries were modified.

Step 5 - Enable KMIP logging for key management

1 entry was modified.

Step 6 - Create data storage aggregate - test_aggr
[Job 161] Job succeeded: DONE

Sleeping for 10 seconds before checking if aggregate was created...
Step 7 - Verify aggregate exists
Aggregate was created successfully.
Step 8 - Create data vserver - test_vserver
[Job 162] Job succeeded:
Vserver creation completed.

Sleeping for 10 seconds before checking if vserver was created...
Step 9 - Verify vserver exists
Vserver was created successfully.
Step 10 - Create 2 encrypted volumes
[Job 163] Job succeeded: Successful

[Job 164] Job succeeded: Successful

Step 11 - Verify encrypted volumes are online
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
test_vserver test_vol_1 test_aggr   online     RW         20MB    18.76MB    1%
test_vserver test_vol_2 test_aggr   online     RW         20MB    18.76MB    1%
2 entries were displayed.

Volume test_vol_1 was created successfully.
Volume test_vol_2 was created successfully.
Step 12 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: XX.XXX.XXX.XX6:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
693e2a8f-506b-11f0-be40-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 000000000000000002000000000005004102ebb412bcc8fdc78e34151553a2f50000000000000000
67c26dfa-506b-11f0-be40-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500bb9c5cba1c36533832e0521d2c2b04c90000000000000000
2 entries were displayed.

Step 13 - Create NSE key
                        NSE key id is  0000000000000000020000000000010044a2413d1cbeddbe4ec7f520a20b2cf10000000000000000
                                                                                                                       Step 14 - Get the NSE key
                                                                                                                                                NSE key id is 0000000000000000020000000000010044a2413d1cbeddbe4ec7f520a20b2cf10000000000000000
Step 15 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: XX.XXX.XXX.XX6:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 0000000000000000020000000000010044a2413d1cbeddbe4ec7f520a20b2cf10000000000000000
693e2a8f-506b-11f0-be40-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 000000000000000002000000000005004102ebb412bcc8fdc78e34151553a2f50000000000000000
67c26dfa-506b-11f0-be40-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500bb9c5cba1c36533832e0521d2c2b04c90000000000000000
3 entries were displayed.

Step 16 - Run debug smdb table cryptomodKeyTable show
                                                     cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                                                                                              key-type    key-digest
------------ --------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------- ----------- ----------------------------------------------------------------
mycluster-01 0         00000000000000000200000000000500bb9c5cba1c36533832e0521d2c2b04c90000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 9dbb47b40182c845ac7d1b3929a69c4f153a093c0b0dc3d39d7c25c3f1738bea
mycluster-01 1         000000000000000002000000000005004102ebb412bcc8fdc78e34151553a2f50000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 2a9b3355b4a88cca2a7f1d6219de2674c92baeda0ecc78aeea5b8e14931188eb
mycluster-01 2         0000000000000000020000000000010044a2413d1cbeddbe4ec7f520a20b2cf10000000000000000 0000000000000000000000000000000000000000000000000000000000000000                                                                 NSE-AK      a571b55cb95a398dd89ea9f10788fb26d72366c66d7ce9d7eb4a69aefed67890
3 entries were displayed.

Step 17 - Check if key-servers are registered
                                             Key server is configured and status is available
Step 18 - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
XX.XXX.XXX.XX6:5696.host=XX.XXX.XXX.XX6
XX.XXX.XXX.XX6:5696.port=5696
XX.XXX.XXX.XX6:5696.trusted_file=/cfcard/kmip/certs/CA.pem
XX.XXX.XXX.XX6:5696.protocol=KMIP1_4
XX.XXX.XXX.XX6:5696.timeout=25
XX.XXX.XXX.XX6:5696.nbio=1
XX.XXX.XXX.XX6:5696.cert_file=/cfcard/kmip/certs/client.crt
XX.XXX.XXX.XX6:5696.key_file=/cfcard/kmip/certs/client.key
XX.XXX.XXX.XX6:5696.ciphers="TLSv1.2:kRSA:!CAMELLIA:!IDEA:!RC2:!RC4:!SEED:!eNULL:!aNULL"
XX.XXX.XXX.XX6:5696.verify=true
XX.XXX.XXX.XX6:5696.netapp_keystore_uuid=559433ba-42e4-11f0-9158-0050568b2de8

Step 19 - Get output of /cfcard/kmip/kmipcmd.log file
                                                     KmipDiscoverVersions succeeded
Step 20 - Turn on AUTOBOOT

  (system node systemshell)

Node: mycluster-01
AUTOBOOT="true"
1 entry was acted on.

Manually reboot the local node and wait 10 minutes before logging back and in running kmip_post_reboot_test.sh

  5. Exit Systemshell.

    mycluster-01% exit

  6. Reboot the node.

    Wait 10 minutes before logging back into the cluster.

    mycluster::*> reboot -node mycluster-01
  (system node reboot)

Warning: Are you sure you want to reboot node "mycluster-01"? {y|n}: y


Connection to xxx.xxx.xxx.xxx closed.

Execute the kmip_post_reboot_test.sh test script

  1. Open a command window and remote login into the NetApp ONTAP Cluster Management.

  2. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  3. Enter Systemshell.

    Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

mycluster-01%

  4. Execute the kmip_post_reboot_test.sh test script and redirect the output to file kmip_post_reboot_test.txt.

    mycluster-01% bash kmip_post_reboot_test.sh | tee kmip_post_reboot_test.txt

Please enter key server name: KeyControl
Please enter key server version: 10.4.3
Please enter whether this is a clustered key-server config (yes or no): no
Executing script kmip_post_reboot_test - version 2.3
Testing DOT: NetApp Release 9.16.1P3: Thu Apr 24 02:50:10 UTC 2025 <1O>
 with Key Manager: KeyControl 10.4.3
Tesing with clustered key servers: no
Step 1 - Get local node name
Local node name is mycluster-01
Step 2 - Get admin vserver name where EKM is configured
Admin vserver name is mycluster
Step 3 - Check if key-servers are registered
Key server is configured and status is available

                Node: mycluster-01
             Vserver: mycluster
     Key Server Port: 5696
 KMIP is operational: true

Key Server           Role           Server Status     Reason
-------------------  ------------   ---------------   ------
XX.XXX.XXX.XX6       primary        available         -

Clustered key servers are not configured as expected
Step 4 - Post Reboot - Verify encrypted volumes are online
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
test_vserver test_vol_1 test_aggr   online     RW         20MB    18.75MB    1%
test_vserver test_vol_2 test_aggr   online     RW         20MB    18.75MB    1%
2 entries were displayed.

Volume test_vol_1 is online as expected.
Volume test_vol_2 is online as expected.
Step 5 - Post Reboot - Get the NSE key
NSE key id is 0000000000000000020000000000010044a2413d1cbeddbe4ec7f520a20b2cf10000000000000000
Step 6 - Post Reboot - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: XX.XXX.XXX.XX6:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 0000000000000000020000000000010044a2413d1cbeddbe4ec7f520a20b2cf10000000000000000
693e2a8f-506b-11f0-be40-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 000000000000000002000000000005004102ebb412bcc8fdc78e34151553a2f50000000000000000
67c26dfa-506b-11f0-be40-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500bb9c5cba1c36533832e0521d2c2b04c90000000000000000
3 entries were displayed.

Step 7 - Post Reboot - Run debug smdb table cryptomodKeyTable show
cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                              key-type key-digest
------------ --------- -------------------------------------------------------------------------------- ---------------------------------------------------------------- -------- ----------------------------------------------------------------
mycluster-01 0         0000000000000000020000000000010044a2413d1cbeddbe4ec7f520a20b2cf10000000000000000 0000000000000000000000000000000000000000000000000000000000000000 NSE-AK   a571b55cb95a398dd89ea9f10788fb26d72366c66d7ce9d7eb4a69aefed67890
mycluster-01 1         000000000000000002000000000005004102ebb412bcc8fdc78e34151553a2f50000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 2a9b3355b4a88cca2a7f1d6219de2674c92baeda0ecc78aeea5b8e14931188eb
mycluster-01 2         00000000000000000200000000000500bb9c5cba1c36533832e0521d2c2b04c90000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 9dbb47b40182c845ac7d1b3929a69c4f153a093c0b0dc3d39d7c25c3f1738bea
3 entries were displayed.

Step 8 - Post Reboot - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
XX.XXX.XXX.XX6:5696.host=XX.XXX.XXX.XX6
XX.XXX.XXX.XX6:5696.port=5696
XX.XXX.XXX.XX6:5696.trusted_file=/cfcard/kmip/certs/CA.pem
XX.XXX.XXX.XX6:5696.protocol=KMIP1_4
XX.XXX.XXX.XX6:5696.timeout=25
XX.XXX.XXX.XX6:5696.nbio=1
XX.XXX.XXX.XX6:5696.cert_file=/cfcard/kmip/certs/client.crt
XX.XXX.XXX.XX6:5696.key_file=/cfcard/kmip/certs/client.key
XX.XXX.XXX.XX6:5696.ciphers="TLSv1.2:kRSA:!CAMELLIA:!IDEA:!RC2:!RC4:!SEED:!eNULL:!aNULL"
XX.XXX.XXX.XX6:5696.verify=true
XX.XXX.XXX.XX6:5696.netapp_keystore_uuid=559433ba-42e4-11f0-9158-0050568b2de8

Step 9 - Post Reboot - Compare /cfcard/kmip/servers.cfg files
The /cfcard/kmip/servers.cfg output before reboot is the same after rebooting
Step 10 - Post Reboot - Delete the NSE key


Step 11 - Post Reboot - Delete the encrypted volumes

[Job 167] Job succeeded: Successful
[Job 168] Job succeeded: Successful
2 entries were acted on.

Step 12 - Post Reboot - Delete the data vserver - test_vserver
[Job 169]
Step 13 - Post Reboot - Delete the data aggregate - test_aggr
[Job 171] Job succeeded: DONE

Step 14 - Turn off logging for key management

284 entries were modified.

Step 15 - Enable KMIP logging for key management

1 entry was modified.

Step 16 - Post Reboot - Verify no keys are observed in key query
No keys are on the cluster as expected.

  5. Exit Systemshell.

    mycluster-01% exit

Enable FIPS mode

  1. Open a command window and remote login into the NetApp ONTAP Cluster Management.

  2. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  3. Enable FIPS mode.

    mycluster::*> security config modify -interface SSL -is-fips-enabled true

Warning: This command will enable FIPS compliance and can potentially cause some non-compliant components to fail.
         MetroCluster and Vserver DR require FIPS to be enabled on both sites in order to be compatible. An SNMP users
         or SNMP traphosts that are non-compliant to FIPS will be deleted automatically. An SNMPv1 user, SNMPv2c user
         or SNMPv3 user (with none or MD5 as authentication protocol or none or DES as encryption protocol or both) is
         non-compliant to FIPS. An SNMPv1 traphost or SNMPv3 traphost (configured with an SNMPv3 user non-compliant to
         FIPS) is non-compliant to FIPS.
Do you want to continue? {y|n}: y

  4. Reboot all nodes in the cluster.

    Wait 10 minutes before logging back into the cluster.

    mycluster::*> reboot -node *
  (system node reboot)

Warning: Are you sure you want to reboot node "mycluster-01"? {y|n}: Y
1 entry was acted on.


Connection to xx.xxx.xxx.xxx closed.

  5. Log back into the NetApp ONTAP Cluster Management.

  6. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  7. Verify FIPS mode is enabled.

    mycluster::*> security config show
Cluster    Supported
FIPS Mode  Protocols Supported Cipher Suites
---------- --------- ----------------------------------------------------------
true       TLSv1.3,  TLS_RSA_WITH_AES_128_CCM, TLS_RSA_WITH_AES_128_CCM_8,
           TLSv1.2   TLS_RSA_WITH_AES_128_GCM_SHA256,
                     TLS_RSA_WITH_AES_128_CBC_SHA,
                     TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CCM,

 ...

                     TLS_SRP_SHA_DSS_WITH_AES_256_CBC_SHA,
                     TLS_SRP_SHA_RSA_WITH_AES_128_CBC_SHA,
                     TLS_SRP_SHA_RSA_WITH_AES_256_CBC_SHA,
                     TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384

Execute the before and post test scripts a second time

  1. Open a command window and remote login into the NetApp ONTAP Cluster Management.

  2. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  3. Enter Systemshell.

    Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

mycluster-01%

  4. Execute the kmip_before_reboot_test.sh test script and redirect the output to file kmip_before_reboot_test_fips.txt.

    mycluster-01% bash kmip_before_reboot_test.sh | tee kmip_before_reboot_test_fips.txt

Please enter key server name: KeyControl
Please enter key server version: 10.4.3
Please enter whether this is a clustered key-server config (yes or no): no
Executing script kmip_before_reboot_test - version 2.3
Testing DOT: NetApp Release 9.16.1P3: Thu Apr 24 02:50:10 UTC 2025 <1O>
 with Key Manager: KeyControl 10.4.3
Tesing with clustered key servers: no
Step 1 - Get local node name
Local node name is mycluster-01
Step 2 - Get admin vserver name where EKM is configured
Admin vserver name is mycluster
Step 3 - Check if key-servers are registered
Key server is configured and status is available

                Node: mycluster-01
             Vserver: mycluster
     Key Server Port: 5696
 KMIP is operational: true

Key Server           Role           Server Status     Reason
-------------------  ------------   ---------------   ------
XX.XXX.XXX.XX6       primary        available         -

Clustered key servers are not configured as expected
Step 4 - Turn on logging for key management

284 entries were modified.

Step 5 - Enable KMIP logging for key management

1 entry was modified.

Step 6 - Create data storage aggregate - test_aggr
[Job 177] Job succeeded: DONE

Sleeping for 10 seconds before checking if aggregate was created...
Step 7 - Verify aggregate exists
Aggregate was created successfully.
Step 8 - Create data vserver - test_vserver
[Job 178] Sleeping for 10 seconds before checking if vserver was created...
[Job 178] Job succeeded:
Vserver creation completed.

Step 9 - Verify vserver exists
Vserver was created successfully.
Step 10 - Create 2 encrypted volumes
[Job 179] Job succeeded: Successful

[Job 180] Job succeeded: Successful

Step 11 - Verify encrypted volumes are online
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
test_vserver test_vol_1 test_aggr   online     RW         20MB    18.75MB    1%
test_vserver test_vol_2 test_aggr   online     RW         20MB    18.76MB    1%
2 entries were displayed.

Volume test_vol_1 was created successfully.
Volume test_vol_2 was created successfully.
Step 12 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: XX.XXX.XXX.XX6:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
40f653bd-5103-11f0-9478-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500bc9dbeec5db9a3106c920c4c65af30860000000000000000
3ee2ce15-5103-11f0-9478-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500cdbdc1f1aad97bdeff91cecf93c4a7910000000000000000
2 entries were displayed.

Step 13 - Create NSE key
NSE key id is  000000000000000002000000000001007e8c53f2b60ce82be0cea3e55085fa140000000000000000
Step 14 - Get the NSE key
                         NSE key id is 000000000000000002000000000001007e8c53f2b60ce82be0cea3e55085fa140000000000000000
Step 15 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: XX.XXX.XXX.XX6:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 000000000000000002000000000001007e8c53f2b60ce82be0cea3e55085fa140000000000000000
40f653bd-5103-11f0-9478-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500bc9dbeec5db9a3106c920c4c65af30860000000000000000
3ee2ce15-5103-11f0-9478-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500cdbdc1f1aad97bdeff91cecf93c4a7910000000000000000
3 entries were displayed.

Step 16 - Run debug smdb table cryptomodKeyTable show
                                                     cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                                                                                              key-type    key-digest
------------ --------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------- ----------- ----------------------------------------------------------------
mycluster-01 0         00000000000000000200000000000500cdbdc1f1aad97bdeff91cecf93c4a7910000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 e6015f0ec69b1caa10b7c1e68a2f04dfc144b2884819ecb2fd64fd5b765c0198
mycluster-01 1         00000000000000000200000000000500bc9dbeec5db9a3106c920c4c65af30860000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 4a0c5ab1545b9bba62791cb3e7ba42bc7e295d002cb8a91072c57e54a3632a56
mycluster-01 2         000000000000000002000000000001007e8c53f2b60ce82be0cea3e55085fa140000000000000000 0000000000000000000000000000000000000000000000000000000000000000                                                                 NSE-AK      ccdb85becaaa34f3301748938ce9d6ea63dfe809f99fe70cd7867dc272654a87
3 entries were displayed.

Step 17 - Check if key-servers are registered
                                             Key server is configured and status is available
                                                                                             Step 18 - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
XX.XXX.XXX.XX6:5696.host=XX.XXX.XXX.XX6
XX.XXX.XXX.XX6:5696.port=5696
XX.XXX.XXX.XX6:5696.trusted_file=/cfcard/kmip/certs/CA.pem
XX.XXX.XXX.XX6:5696.protocol=KMIP1_4
XX.XXX.XXX.XX6:5696.timeout=25
XX.XXX.XXX.XX6:5696.nbio=1
XX.XXX.XXX.XX6:5696.cert_file=/cfcard/kmip/certs/client.crt
XX.XXX.XXX.XX6:5696.key_file=/cfcard/kmip/certs/client.key
XX.XXX.XXX.XX6:5696.ciphers="TLSv1.2+FIPS:!eNULL:!aNULL"
XX.XXX.XXX.XX6:5696.verify=true
XX.XXX.XXX.XX6:5696.netapp_keystore_uuid=559433ba-42e4-11f0-9158-0050568b2de8

Step 19 - Get output of /cfcard/kmip/kmipcmd.log file
                                                     KmipDiscoverVersions succeeded
Step 20 - Turn on AUTOBOOT

  (system node systemshell)

Node: mycluster-01
AUTOBOOT="true"
1 entry was acted on.

Manually reboot the local node and wait 10 minutes before logging back and in running kmip_post_reboot_test.sh

  5. Exit Systemshell.

    mycluster-01% exit

  6. Reboot the node.

    Wait 10 minutes before logging back into the cluster.

    mycluster::*> reboot -node mycluster-01
  (system node reboot)

Warning: Are you sure you want to reboot node "mycluster-01"? {y|n}: y


Connection to xxx.xxx.xxx.xxx closed.

  7. Log back into the NetApp ONTAP Cluster Management.

  8. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  9. Enter Systemshell. Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

mycluster-01%

  10. Execute the kmip_post_reboot_test.sh test script and redirect the output to file kmip_post_reboot_test_fips.txt.

    mycluster-01% bash kmip_post_reboot_test.sh | tee kmip_post_reboot_test_fips.txt

Please enter key server name: KeyControl
Please enter key server version: 10.4.3
Please enter whether this is a clustered key-server config (yes or no): no
Executing script kmip_post_reboot_test - version 2.3
Testing DOT: NetApp Release 9.16.1P3: Thu Apr 24 02:50:10 UTC 2025 <1O>
 with Key Manager: KeyControl 10.4.3
Tesing with clustered key servers: no
Step 1 - Get local node name
Local node name is mycluster-01
Step 2 - Get admin vserver name where EKM is configured
Admin vserver name is mycluster
Step 3 - Check if key-servers are registered
Key server is configured and status is available

                Node: mycluster-01
             Vserver: mycluster
     Key Server Port: 5696
 KMIP is operational: true

Key Server           Role           Server Status     Reason
-------------------  ------------   ---------------   ------
XX.XXX.XXX.XX6       primary        available         -

Clustered key servers are not configured as expected
Step 4 - Post Reboot - Verify encrypted volumes are online
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
test_vserver test_vol_1 test_aggr   online     RW         20MB    18.75MB    1%
test_vserver test_vol_2 test_aggr   online     RW         20MB    18.75MB    1%
2 entries were displayed.

Volume test_vol_1 is online as expected.
Volume test_vol_2 is online as expected.
Step 5 - Post Reboot - Get the NSE key
NSE key id is 000000000000000002000000000001007e8c53f2b60ce82be0cea3e55085fa140000000000000000
                                                                                              Step 6 - Post Reboot - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: XX.XXX.XXX.XX6:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 000000000000000002000000000001007e8c53f2b60ce82be0cea3e55085fa140000000000000000
40f653bd-5103-11f0-9478-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500bc9dbeec5db9a3106c920c4c65af30860000000000000000
3ee2ce15-5103-11f0-9478-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500cdbdc1f1aad97bdeff91cecf93c4a7910000000000000000
3 entries were displayed.

Step 7 - Post Reboot - Run debug smdb table cryptomodKeyTable show
cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                              key-type key-digest
------------ --------- -------------------------------------------------------------------------------- ---------------------------------------------------------------- -------- ----------------------------------------------------------------
mycluster-01 0         000000000000000002000000000001007e8c53f2b60ce82be0cea3e55085fa140000000000000000 0000000000000000000000000000000000000000000000000000000000000000 NSE-AK   ccdb85becaaa34f3301748938ce9d6ea63dfe809f99fe70cd7867dc272654a87
mycluster-01 1         00000000000000000200000000000500cdbdc1f1aad97bdeff91cecf93c4a7910000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 e6015f0ec69b1caa10b7c1e68a2f04dfc144b2884819ecb2fd64fd5b765c0198
mycluster-01 2         00000000000000000200000000000500bc9dbeec5db9a3106c920c4c65af30860000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 4a0c5ab1545b9bba62791cb3e7ba42bc7e295d002cb8a91072c57e54a3632a56
3 entries were displayed.

Step 8 - Post Reboot - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
XX.XXX.XXX.XX6:5696.host=XX.XXX.XXX.XX6
XX.XXX.XXX.XX6:5696.port=5696
XX.XXX.XXX.XX6:5696.trusted_file=/cfcard/kmip/certs/CA.pem
XX.XXX.XXX.XX6:5696.protocol=KMIP1_4
XX.XXX.XXX.XX6:5696.timeout=25
XX.XXX.XXX.XX6:5696.nbio=1
XX.XXX.XXX.XX6:5696.cert_file=/cfcard/kmip/certs/client.crt
XX.XXX.XXX.XX6:5696.key_file=/cfcard/kmip/certs/client.key
XX.XXX.XXX.XX6:5696.ciphers="TLSv1.2+FIPS:!eNULL:!aNULL"
XX.XXX.XXX.XX6:5696.verify=true
XX.XXX.XXX.XX6:5696.netapp_keystore_uuid=559433ba-42e4-11f0-9158-0050568b2de8

Step 9 - Post Reboot - Compare /cfcard/kmip/servers.cfg files
The /cfcard/kmip/servers.cfg output before reboot is the same after rebooting
Step 10 - Post Reboot - Delete the NSE key


Step 11 - Post Reboot - Delete the encrypted volumes

[Job 184] Job succeeded: Successful
[Job 185] Job succeeded: Successful
2 entries were acted on.

Step 12 - Post Reboot - Delete the data vserver - test_vserver
[Job 186]
Step 13 - Post Reboot - Delete the data aggregate - test_aggr
[Job 188] Job succeeded: DONE

Step 14 - Turn off logging for key management

284 entries were modified.

Step 15 - Enable KMIP logging for key management

1 entry was modified.

Step 16 - Post Reboot - Verify no keys are observed in key query
No keys are on the cluster as expected.

  11. Copy the test script output files to a server of your choice.

    Provide the password when prompted.

    mycluster-01% scp *.txt root@xxx.xxx.xxx.xxx:/root/Downloads/.

kmip_before_reboot_test.txt                                                                                                                                                   100%   16KB   4.9MB/s   00:00
kmip_before_reboot_test_fips.txt                                                                                                                                              100%   14KB   7.3MB/s   00:00
kmip_post_reboot_test.txt                                                                                                                                                     100%   14KB   9.5MB/s   00:00
kmip_post_reboot_test_fips.txt                                                                                                                                                100%   14KB  15.0MB/s   00:00
SSH terminating : scp.c : main : 690,errs = 0.

  12. Send these output files to NetApp for verification.

Verify FIPS mode is unchanged after reboot

  1. Exit Systemshell.

    mycluster-01% exit

  2. Disable FIPS mode.

    mycluster::*> security config modify -interface SSL -is-fips-enabled false

  3. Reboot all nodes in the cluster.

    mycluster::*> reboot -node *
  (system node reboot)

Warning: Are you sure you want to reboot node "mycluster-01"? {y|n}: Y
1 entry was acted on.


Connection to xx.xxx.xxx.xxx closed.

  4. Log back into the NetApp ONTAP Cluster Management.

  5. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  6. Verify FIPS mode is disabled on the cluster.

    mycluster::*> security config show
Cluster    Supported
FIPS Mode  Protocols Supported Cipher Suites
---------- --------- ----------------------------------------------------------
false      TLSv1.3,  TLS_RSA_WITH_AES_128_CCM, TLS_RSA_WITH_AES_128_CCM_8,
           TLSv1.2   TLS_RSA_WITH_AES_128_GCM_SHA256,
                     TLS_RSA_WITH_AES_128_CBC_SHA,
                     TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CCM,

 ...

                     TLS_SRP_SHA_RSA_WITH_AES_128_CBC_SHA,
                     TLS_SRP_SHA_RSA_WITH_AES_256_CBC_SHA,
                     TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384,
                     TLS_CHACHA20_POLY1305_SHA256