Test integration

This test procedure requires test scripts available from NetApp. The output files resulting from executing the test scripts need to be sent back to NetApp for verification.

Load the test scripts into NetApp ONTAP

  1. Open a command window and remote login into the NetApp ONTAP Cluster Management.

  2. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  3. Enter system shell.

    Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

mycluster-01%

  4. Copy the test script files from a server of your choice into the Systemshell of the NetApp ONTAP node.

    Provide the password when prompted.

    mycluster-01% scp root@xx.xxx.xxx.xxx:/root/Downloads/kmip_before_reboot_test.sh .
kmip_before_reboot_test.sh                                                            100% 7346   731.0KB/s   00:00
SSH terminating : scp.c : main : 690,errs = 0.

mycluster-01% scp root@xx.xxx.xxx.xxx:/root/Downloads/kmip_post_reboot_test.sh .
kmip_post_reboot_test.sh                                                              100% 6047     3.6MB/s   00:00
SSH terminating : scp.c : main : 690,errs = 0.
    The test scripts were provided by NetApp.

  5. Verify the test scripts files are in the current directory.

    mycluster-01% ls
kmip_before_reboot_test.sh      kmip_post_reboot_test.sh

Execute the kmip_before_reboot_test.sh test script

  1. Open a command window and remote login into the NetApp ONTAP Cluster Management.

  2. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  3. Enter Systemshell.

    Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

mycluster-01%

  4. Execute the kmip_before_reboot_test.sh test script and redirect the output to file kmip_before_reboot_test.txt.

    KeyControl presents itself as a single entity even though it may be composed of multiple nodes (two in this test case). Therefore, select no if the Please enter whether this is a clustered key-server config (yes or no): question is shown.

    mycluster-01% bash kmip_before_reboot_test.sh | tee kmip_before_reboot_test.txt

Please enter key server name: KeyControl
Please enter key server version: 10.4.1
Executing script kmip_before_reboot_test - version 2.0
Testing DOT: NetApp Release 9.14.1P10: Thu Nov 28 12:32:16 UTC 2024 <1O>
 with Key Manager: KeyControl 10.4.1
Step 1 - Get local node name
Local node name is mycluster-01
Step 2 - Check if key-servers are registered
Key server is configured and status is available
Step 3 - Turn on logging for key management

216 entries were modified.

Step 4 - Create a KMIP log file


Step 5 - Create data storage aggregate - test_aggr
[Job 32] Job succeeded: DONE

Sleeping for 10 seconds before checking if aggregate was created...
Step 6 - Verify aggregate exists
Aggregate was created successfully.
Step 7 - Create data vserver - test_vserver
[Job 33] Sleeping for 10 seconds before checking if vserver was created...
[Job 33] Job succeeded:
Vserver creation completed.

Step 8 - Verify vserver exists
Vserver was created successfully.
Step 9 - Create 2 encrypted volumes
[Job 34] Job succeeded: Successful

[Job 35] Job succeeded: Successful

Step 10 - Verify encrypted volumes are online
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
test_vserver test_vol_1 test_aggr   online     RW         20MB    18.77MB    1%
test_vserver test_vol_2 test_aggr   online     RW         20MB    18.79MB    1%
2 entries were displayed.

Volume test_vol_1 was created successfully.
Volume test_vol_2 was created successfully.
Step 11 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xxx:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
09f0e909-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500903f4e84f2b556f26f515687f506a7b30000000000000000
06ac08eb-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500d84075559fbc352b558db71f7a73f4da0000000000000000

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xxx:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
09f0e909-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500903f4e84f2b556f26f515687f506a7b30000000000000000
06ac08eb-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500d84075559fbc352b558db71f7a73f4da0000000000000000
4 entries were displayed.

Step 12 - Create NSE key
NSE key id is  000000000000000002000000000001008e2e389af67414b030ecc5315f6580840000000000000000
Step 13 - Get the NSE key
NSE key id is displayed.
Step 14 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xxx:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 000000000000000002000000000001008e2e389af67414b030ecc5315f6580840000000000000000
09f0e909-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500903f4e84f2b556f26f515687f506a7b30000000000000000
06ac08eb-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500d84075559fbc352b558db71f7a73f4da0000000000000000

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xxx:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 000000000000000002000000000001008e2e389af67414b030ecc5315f6580840000000000000000
09f0e909-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500903f4e84f2b556f26f515687f506a7b30000000000000000
06ac08eb-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500d84075559fbc352b558db71f7a73f4da0000000000000000
6 entries were displayed.

Step 15 - Run debug smdb table cryptomodKeyTable show
                                                     cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                                                                                              key-type    key-digest
------------ --------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------- ----------- ----------------------------------------------------------------
mycluster-01 0         00000000000000000200000000000500d84075559fbc352b558db71f7a73f4da0000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 85b99396413ad70c512eb2b242d8387f030f37985e3961573b4f80744a1ea437
mycluster-01 1         00000000000000000200000000000500903f4e84f2b556f26f515687f506a7b30000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 460dcd3713837df4b766a73da1c97511a6ce5912b560e418978a1f19960d8fef
mycluster-01 2         000000000000000002000000000001008e2e389af67414b030ecc5315f6580840000000000000000 0000000000000000000000000000000000000000000000000000000000000000                                                                 NSE-AK      60db26cb257a0089a48e98743e21e216d3266e15da0742ec243eb2287148412b
3 entries were displayed.

Step 16 - Check if key-servers are registered
                                             Key server is configured and status is available
Step 17 - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
xx.xxx.xxx.xxx:5696.host=xx.xxx.xxx.xxx
xx.xxx.xxx.xxx:5696.port=5696
xx.xxx.xxx.xxx:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xxx:5696.protocol=KMIP1_4
xx.xxx.xxx.xxx:5696.timeout=25
xx.xxx.xxx.xxx:5696.nbio=1
xx.xxx.xxx.xxx:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xxx:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xxx:5696.ciphers="TLSv1.2:kRSA:!CAMELLIA:!IDEA:!RC2:!RC4:!SEED:!eNULL:!aNULL"
xx.xxx.xxx.xxx:5696.verify=true

xx.xxx.xxx.xxx:5696.host=xx.xxx.xxx.xxx
xx.xxx.xxx.xxx:5696.port=5696
xx.xxx.xxx.xxx:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xxx:5696.protocol=KMIP1_4
xx.xxx.xxx.xxx:5696.timeout=25
xx.xxx.xxx.xxx:5696.nbio=1
xx.xxx.xxx.xxx:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xxx:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xxx:5696.ciphers="TLSv1.2:kRSA:!CAMELLIA:!IDEA:!RC2:!RC4:!SEED:!eNULL:!aNULL"
xx.xxx.xxx.xxx:5696.verify=true

Step 18 - Get output of /cfcard/kmip/kmipcmd.log file
KmipDiscoverVersions succeeded
Step 19 - Turn on AUTOBOOT

  (system node systemshell)

Node: mycluster-01
AUTOBOOT="true"
1 entry was acted on.

Manually reboot the local node and wait 10 minutes before logging back and in running kmip_post_reboot_test.sh

  5. Exit Systemshell.

    mycluster-01% exit

  6. Reboot the node.

    Wait 10 minutes before logging back into the cluster.

    mycluster::*> reboot -node mycluster-01
  (system node reboot)

Warning: Are you sure you want to reboot node "mycluster-01"? {y|n}: y


Connection to xxx.xxx.xxx.xxx closed.

Execute the kmip_post_reboot_test.sh test script

  1. Open a command window and remote login into the NetApp ONTAP Cluster Management.

  2. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  3. Enter Systemshell.

    Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

mycluster-01%

  4. Execute the kmip_post_reboot_test.sh test script and redirect the output to file kmip_post_reboot_test.txt.

    mycluster-01% bash kmip_post_reboot_test.sh | tee kmip_post_reboot_test.txt

Please enter key server name: KeyControl
Please enter key server version: 10.4.1
Executing script kmip_post_reboot_test - version 2.0
Testing DOT: NetApp Release 9.14.1P10: Thu Nov 28 12:32:16 UTC 2024 <1O>
 with Key Manager: KeyControl 10.4.1
Step 1 - Get local node name
Local node name is mycluster-01
Step 2 - Check if key-servers are registered
Key server is configured and status is available
Step 3 - Post Reboot - Verify encrypted volumes are online
                                                          Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
test_vserver test_vol_1 test_aggr   online     RW         20MB    18.76MB    1%
test_vserver test_vol_2 test_aggr   online     RW         20MB    18.76MB    1%
2 entries were displayed.

Volume test_vol_1 is online as expected.
                                        Volume test_vol_2 is online as expected.
Step 4 - Post Reboot - Get the NSE key
NSE key id is 000000000000000002000000000001008e2e389af67414b030ecc5315f6580840000000000000000
Step 5 - Post Reboot - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xx6:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 000000000000000002000000000001008e2e389af67414b030ecc5315f6580840000000000000000
09f0e909-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500903f4e84f2b556f26f515687f506a7b30000000000000000
06ac08eb-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500d84075559fbc352b558db71f7a73f4da0000000000000000

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xx7:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 000000000000000002000000000001008e2e389af67414b030ecc5315f6580840000000000000000
09f0e909-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500903f4e84f2b556f26f515687f506a7b30000000000000000
06ac08eb-dce0-11ef-8bd5-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500d84075559fbc352b558db71f7a73f4da0000000000000000
6 entries were displayed.

Step 6 - Post Reboot - Run debug smdb table cryptomodKeyTable show
cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                              key-type key-digest
------------ --------- -------------------------------------------------------------------------------- ---------------------------------------------------------------- -------- ----------------------------------------------------------------
mycluster-01 0         000000000000000002000000000001008e2e389af67414b030ecc5315f6580840000000000000000 0000000000000000000000000000000000000000000000000000000000000000 NSE-AK   60db26cb257a0089a48e98743e21e216d3266e15da0742ec243eb2287148412b
mycluster-01 1         00000000000000000200000000000500903f4e84f2b556f26f515687f506a7b30000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 460dcd3713837df4b766a73da1c97511a6ce5912b560e418978a1f19960d8fef
mycluster-01 2         00000000000000000200000000000500d84075559fbc352b558db71f7a73f4da0000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 85b99396413ad70c512eb2b242d8387f030f37985e3961573b4f80744a1ea437
3 entries were displayed.

Step 7 - Post Reboot - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
xx.xxx.xxx.xx6:5696.host=xx.xxx.xxx.xx6
xx.xxx.xxx.xx6:5696.port=5696
xx.xxx.xxx.xx6:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xx6:5696.protocol=KMIP1_4
xx.xxx.xxx.xx6:5696.timeout=25
xx.xxx.xxx.xx6:5696.nbio=1
xx.xxx.xxx.xx6:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xx6:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xx6:5696.ciphers="TLSv1.2:kRSA:!CAMELLIA:!IDEA:!RC2:!RC4:!SEED:!eNULL:!aNULL"
xx.xxx.xxx.xx6:5696.verify=true

xx.xxx.xxx.xx7:5696.host=xx.xxx.xxx.xx7
xx.xxx.xxx.xx7:5696.port=5696
xx.xxx.xxx.xx7:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xx7:5696.protocol=KMIP1_4
xx.xxx.xxx.xx7:5696.timeout=25
xx.xxx.xxx.xx7:5696.nbio=1
xx.xxx.xxx.xx7:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xx7:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xx7:5696.ciphers="TLSv1.2:kRSA:!CAMELLIA:!IDEA:!RC2:!RC4:!SEED:!eNULL:!aNULL"
xx.xxx.xxx.xx7:5696.verify=true

Step 8 - Post Reboot - Compare /cfcard/kmip/servers.cfg files
The /cfcard/kmip/servers.cfg output before reboot is the same after rebooting
Step 9 - Post Reboot - Delete the NSE key


Step 10 - Post Reboot - Delete the encrypted volumes

[Job 38] Job succeeded: Successful
[Job 39] Job succeeded: Successful
2 entries were acted on.

Step 11 - Post Reboot - Delete the data vserver - test_vserver
[Job 40]
Step 12 - Post Reboot - Delete the data aggregate - test_aggr
[Job 42] Job succeeded: DONE

Step 13 - Turn off logging for key management

216 entries were modified.

Step 14 - Delete a KMIP log file


Step 15 - Post Reboot - Verify no keys are observed in key query
No keys are on the cluster as expected.

  5. Exit Systemshell.

    mycluster-01% exit

Enable FIPS mode

  1. Open a command window and remote login into the NetApp ONTAP Cluster Management.

  2. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  3. Enable FIPS mode.

    mycluster::*> security config modify -interface SSL -is-fips-enabled true

Warning: This command will enable FIPS compliance and can potentially cause some non-compliant components to fail.
         MetroCluster and Vserver DR require FIPS to be enabled on both sites in order to be compatible. An SNMP users
         or SNMP traphosts that are non-compliant to FIPS will be deleted automatically. An SNMPv1 user, SNMPv2c user
         or SNMPv3 user (with none or MD5 as authentication protocol or none or DES as encryption protocol or both) is
         non-compliant to FIPS. An SNMPv1 traphost or SNMPv3 traphost (configured with an SNMPv3 user non-compliant to
         FIPS) is non-compliant to FIPS.
Do you want to continue? {y|n}: y

  4. Reboot all nodes in the cluster.

    Wait 10 minutes before logging back into the cluster.

    mycluster::*> reboot -node *
  (system node reboot)

Warning: Are you sure you want to reboot node "mycluster-01"? {y|n}: Y
1 entry was acted on.


Connection to xx.xxx.xxx.xxx closed.

  5. Log back into the NetApp ONTAP Cluster Management.

  6. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  7. Verify FIPS mode is enabled.

    mycluster::*> security config show
Cluster    Supported
FIPS Mode  Protocols Supported Cipher Suites
---------- --------- ----------------------------------------------------------
true       TLSv1.3,  TLS_RSA_WITH_AES_128_CCM, TLS_RSA_WITH_AES_128_CCM_8,
           TLSv1.2   TLS_RSA_WITH_AES_128_GCM_SHA256,
                     TLS_RSA_WITH_AES_128_CBC_SHA,
                     TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CCM,

 ...

                     TLS_SRP_SHA_DSS_WITH_AES_256_CBC_SHA,
                     TLS_SRP_SHA_RSA_WITH_AES_128_CBC_SHA,
                     TLS_SRP_SHA_RSA_WITH_AES_256_CBC_SHA,
                     TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384

Execute the before and post test scripts a second time

  1. Open a command window and remote login into the NetApp ONTAP Cluster Management.

  2. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  3. Enter Systemshell.

    Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

mycluster-01%

  4. Execute the kmip_before_reboot_test.sh test script and redirect the output to file kmip_before_reboot_test_fips.txt.

    mycluster-01% bash kmip_before_reboot_test.sh | tee kmip_before_reboot_test_fips.txt

Please enter key server name: KeyControl
Please enter key server version: 10.4.1
Executing script kmip_before_reboot_test - version 2.0
Testing DOT: NetApp Release 9.14.1P10: Thu Nov 28 12:32:16 UTC 2024 <1O>
 with Key Manager: KeyControl 10.4.1
Step 1 - Get local node name
Local node name is mycluster-01
Step 2 - Check if key-servers are registered
Key server is configured and status is available
Step 3 - Turn on logging for key management

216 entries were modified.

Step 4 - Create a KMIP log file


Step 5 - Create data storage aggregate - test_aggr
[Job 45] Job succeeded: DONE

Sleeping for 10 seconds before checking if aggregate was created...
Step 6 - Verify aggregate exists
Aggregate was created successfully.
Step 7 - Create data vserver - test_vserver
[Job 46] Sleeping for 10 seconds before checking if vserver was created...
[Job 46] Job succeeded:
Vserver creation completed.

Step 8 - Verify vserver exists
Vserver was created successfully.
Step 9 - Create 2 encrypted volumes
[Job 47] Job succeeded: Successful

[Job 48] Job succeeded: Successful

Step 10 - Verify encrypted volumes are online
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
test_vserver test_vol_1 test_aggr   online     RW         20MB    18.77MB    1%
test_vserver test_vol_2 test_aggr   online     RW         20MB    18.79MB    1%
2 entries were displayed.

Volume test_vol_1 was created successfully.
Volume test_vol_2 was created successfully.
Step 11 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xx6:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
ddb9ecd0-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050041f10f2d23caf84391b6579a45ee8a5f0000000000000000
dab1e555-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500ef91891b7c136f55c266c1740cc959f90000000000000000

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xx7:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
ddb9ecd0-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050041f10f2d23caf84391b6579a45ee8a5f0000000000000000
dab1e555-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500ef91891b7c136f55c266c1740cc959f90000000000000000
4 entries were displayed.

Step 12 - Create NSE key
NSE key id is  000000000000000002000000000001008a457ba6bf6e5b7a30ee1280dc56a6050000000000000000
Step 13 - Get the NSE key
NSE key id is displayed.
Step 14 - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xx6:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 000000000000000002000000000001008a457ba6bf6e5b7a30ee1280dc56a6050000000000000000
ddb9ecd0-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050041f10f2d23caf84391b6579a45ee8a5f0000000000000000
dab1e555-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500ef91891b7c136f55c266c1740cc959f90000000000000000

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xx7:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 000000000000000002000000000001008a457ba6bf6e5b7a30ee1280dc56a6050000000000000000
ddb9ecd0-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050041f10f2d23caf84391b6579a45ee8a5f0000000000000000
dab1e555-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500ef91891b7c136f55c266c1740cc959f90000000000000000
6 entries were displayed.

Step 15 - Run debug smdb table cryptomodKeyTable show
cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                                              key-type    key-digest
------------ --------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- ----------- ----------------------------------------------------------------
mycluster-01 0         00000000000000000200000000000500ef91891b7c136f55c266c1740cc959f90000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 0572e25a6da0af547827a838db9cd19a1bc292e31665e2d0d93d15866a8819f3
mycluster-01 1         0000000000000000020000000000050041f10f2d23caf84391b6579a45ee8a5f0000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 XTS-AES-256 cf9d08c00dc49ee483e59185cccb2d0cb428c2b9b0b5ec916adb9d803a2668a6
mycluster-01 2         000000000000000002000000000001008a457ba6bf6e5b7a30ee1280dc56a6050000000000000000 000000000000000000000000000000000000000000000000000000                           NSE-AK      79d2a7fbbe927b0f76291b760e919c90efd183cc5f3f312c568133f58172e30a
3 entries were displayed.

Step 16 - Check if key-servers are registered
                                             Key server is configured and status is available
Step 17 - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
xx.xxx.xxx.xx6:5696.host=xx.xxx.xxx.xx6
xx.xxx.xxx.xx6:5696.port=5696
xx.xxx.xxx.xx6:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xx6:5696.protocol=KMIP1_4
xx.xxx.xxx.xx6:5696.timeout=25
xx.xxx.xxx.xx6:5696.nbio=1
xx.xxx.xxx.xx6:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xx6:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xx6:5696.ciphers="TLSv1.2+FIPS:!eNULL:!aNULL"
xx.xxx.xxx.xx6:5696.verify=true

xx.xxx.xxx.xx7:5696.host=xx.xxx.xxx.xx7
xx.xxx.xxx.xx7:5696.port=5696
xx.xxx.xxx.xx7:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xx7:5696.protocol=KMIP1_4
xx.xxx.xxx.xx7:5696.timeout=25
xx.xxx.xxx.xx7:5696.nbio=1
xx.xxx.xxx.xx7:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xx7:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xx7:5696.ciphers="TLSv1.2+FIPS:!eNULL:!aNULL"
xx.xxx.xxx.xx7:5696.verify=true

Step 18 - Get output of /cfcard/kmip/kmipcmd.log file
KmipDiscoverVersions succeeded
Step 19 - Turn on AUTOBOOT

  (system node systemshell)

Node: mycluster-01
AUTOBOOT="true"
1 entry was acted on.

Manually reboot the local node and wait 10 minutes before logging back and in running kmip_post_reboot_test.sh

  5. Exit Systemshell.

    mycluster-01% exit

  6. Reboot the node.

    Wait 10 minutes before logging back into the cluster.

    mycluster::*> reboot -node mycluster-01
  (system node reboot)

Warning: Are you sure you want to reboot node "mycluster-01"? {y|n}: y


Connection to xxx.xxx.xxx.xxx closed.

  7. Log back into the NetApp ONTAP Cluster Management.

  8. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  9. Enter Systemshell. Provide the password when prompted.

    mycluster::*> systemshell -node mycluster-01
  (system node systemshell)
diag@127.0.0.1's password:

Warning:  The system shell provides access to low-level
diagnostic tools that can cause irreparable damage to
the system if not used properly.  Use this environment
only when directed to do so by support personnel.

mycluster-01%

  10. Execute the kmip_post_reboot_test.sh test script and redirect the output to file kmip_post_reboot_test_fips.txt.

    mycluster-01% bash kmip_post_reboot_test.sh | tee kmip_post_reboot_test_fips.txt

Please enter key server name: KeyControl
Please enter key server version: 10.4.1
Executing script kmip_post_reboot_test - version 2.0
Testing DOT: NetApp Release 9.14.1P10: Thu Nov 28 12:32:16 UTC 2024 <1O>
 with Key Manager: KeyControl 10.4.1
Step 1 - Get local node name
Local node name is mycluster-01
Step 2 - Check if key-servers are registered
Key server is configured and status is available
Step 3 - Post Reboot - Verify encrypted volumes are online
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
test_vserver test_vol_1 test_aggr   online     RW         20MB    18.46MB    2%
test_vserver test_vol_2 test_aggr   online     RW         20MB    18.47MB    2%
2 entries were displayed.

Volume test_vol_1 is online as expected.
Volume test_vol_2 is online as expected.
Step 4 - Post Reboot - Get the NSE key
NSE key id is 000000000000000002000000000001008a457ba6bf6e5b7a30ee1280dc56a6050000000000000000
Step 5 - Post Reboot - Run key-manager key query

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xx6:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 000000000000000002000000000001008a457ba6bf6e5b7a30ee1280dc56a6050000000000000000
ddb9ecd0-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050041f10f2d23caf84391b6579a45ee8a5f0000000000000000
dab1e555-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500ef91891b7c136f55c266c1740cc959f90000000000000000

               Node: mycluster-01
            Vserver: mycluster
        Key Manager: xx.xxx.xxx.xx7:5696
   Key Manager Type: KMIP
 Key Manager Policy: -

Key Tag                               Key Type Encryption   Restored
------------------------------------  -------- ------------ --------
test                                  NSE-AK   AES-256      true
    Key ID: 000000000000000002000000000001008a457ba6bf6e5b7a30ee1280dc56a6050000000000000000
ddb9ecd0-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 0000000000000000020000000000050041f10f2d23caf84391b6579a45ee8a5f0000000000000000
dab1e555-dce2-11ef-a576-0050568b2de8  VEK      XTS-AES-256  true
    Key ID: 00000000000000000200000000000500ef91891b7c136f55c266c1740cc959f90000000000000000
6 entries were displayed.

Step 6 - Post Reboot - Run debug smdb table cryptomodKeyTable show
cryptomodKeyTable show output is
node         key-index key-id                                                                           key                                                              key-type key-digest
------------ --------- -------------------------------------------------------------------------------- ---------------------------------------------------------------- -------- ----------------------------  ------------------------------------
mycluster-01 0         000000000000000002000000000001008a457ba6bf6e5b7a30ee1280dc56a6050000000000000000 0000000000000000000000000000000000000000000000000000000000000000 NSE-AK   79d2a7fbbe927b0f76291b760e91  9c90efd183cc5f3f312c568133f58172e30a
mycluster-01 1         0000000000000000020000000000050041f10f2d23caf84391b6579a45ee8a5f0000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000  00000000000000000000000000 XTS-AES-256 cf9d08c00dc49ee483e59185cccb2d0cb428c2b9b0b5ec916adb9d803a2668a6
mycluster-01 2         00000000000000000200000000000500ef91891b7c136f55c266c1740cc959f90000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000  00000000000000000000000000 XTS-AES-256 0572e25a6da0af547827a838db9cd19a1bc292e31665e2d0d93d15866a8819f3
3 entries were displayed.

Step 7 - Post Reboot - Get output of /cfcard/kmip/servers.cfg file

  (system node systemshell)
xx.xxx.xxx.xx6:5696.host=xx.xxx.xxx.xx6
xx.xxx.xxx.xx6:5696.port=5696
xx.xxx.xxx.xx6:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xx6:5696.protocol=KMIP1_4
xx.xxx.xxx.xx6:5696.timeout=25
xx.xxx.xxx.xx6:5696.nbio=1
xx.xxx.xxx.xx6:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xx6:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xx6:5696.ciphers="TLSv1.2+FIPS:!eNULL:!aNULL"
xx.xxx.xxx.xx6:5696.verify=true

xx.xxx.xxx.xx7:5696.host=xx.xxx.xxx.xx7
xx.xxx.xxx.xx7:5696.port=5696
xx.xxx.xxx.xx7:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xx.xxx.xxx.xx7:5696.protocol=KMIP1_4
xx.xxx.xxx.xx7:5696.timeout=25
xx.xxx.xxx.xx7:5696.nbio=1
xx.xxx.xxx.xx7:5696.cert_file=/cfcard/kmip/certs/client.crt
xx.xxx.xxx.xx7:5696.key_file=/cfcard/kmip/certs/client.key
xx.xxx.xxx.xx7:5696.ciphers="TLSv1.2+FIPS:!eNULL:!aNULL"
xx.xxx.xxx.xx7:5696.verify=true

Step 8 - Post Reboot - Compare /cfcard/kmip/servers.cfg files
The /cfcard/kmip/servers.cfg output before reboot is the same after rebooting
Step 9 - Post Reboot - Delete the NSE key


Step 10 - Post Reboot - Delete the encrypted volumes

[Job 55] Job succeeded: Successful
[Job 56] Job succeeded: Successful
2 entries were acted on.

Step 11 - Post Reboot - Delete the data vserver - test_vserver
[Job 57]
Step 12 - Post Reboot - Delete the data aggregate - test_aggr
[Job 59] Job succeeded: DONE

Step 13 - Turn off logging for key management

216 entries were modified.

Step 14 - Delete a KMIP log file


Step 15 - Post Reboot - Verify no keys are observed in key query
No keys are on the cluster as expected.

  11. Copy the test script output files to a server of your choice.

    Provide the password when prompted.

    mycluster-01% scp *.txt root@xxx.xxx.xxx.xxx:/root/Downloads/.

kmip_before_reboot_test.txt                                                                                                                                                   100%   16KB   4.9MB/s   00:00
kmip_before_reboot_test_fips.txt                                                                                                                                              100%   14KB   7.3MB/s   00:00
kmip_post_reboot_test.txt                                                                                                                                                     100%   14KB   9.5MB/s   00:00
kmip_post_reboot_test_fips.txt                                                                                                                                                100%   14KB  15.0MB/s   00:00
SSH terminating : scp.c : main : 690,errs = 0.

  12. Send these output files to NetApp for verification.

Verify FIPS mode is unchanged after reboot

  1. Exit Systemshell.

    mycluster-01% exit

  2. Disable FIPS mode.

    mycluster::*> security config modify -interface SSL -is-fips-enabled false

  3. Reboot all nodes in the cluster.

    mycluster::*> reboot -node *
  (system node reboot)

Warning: Are you sure you want to reboot node "mycluster-01"? {y|n}: Y
1 entry was acted on.


Connection to xx.xxx.xxx.xxx closed.

  4. Log back into the NetApp ONTAP Cluster Management.

  5. Set diagnostics.

    mycluster::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

mycluster::*>

  6. Verify FIPS mode is disabled on the cluster.

    mycluster::*> security config show
Cluster    Supported
FIPS Mode  Protocols Supported Cipher Suites
---------- --------- ----------------------------------------------------------
false      TLSv1.3,  TLS_RSA_WITH_AES_128_CCM, TLS_RSA_WITH_AES_128_CCM_8,
           TLSv1.2   TLS_RSA_WITH_AES_128_GCM_SHA256,
                     TLS_RSA_WITH_AES_128_CBC_SHA,
                     TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CCM,

 ...

                     TLS_SRP_SHA_RSA_WITH_AES_128_CBC_SHA,
                     TLS_SRP_SHA_RSA_WITH_AES_256_CBC_SHA,
                     TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384,
                     TLS_CHACHA20_POLY1305_SHA256