System logging (nShield 5 HSMs)

This section describes how you can access the logging information generated by the platform. This information is separate from that produced by ncoreapi. See Platform services and (nShield 5 HSMs) for more information about platform services and ncoreapi.

For information about the logging and debugging information generated by ncoreapi, see Logging, debugging, and diagnostics and Audit Logging.

Two system logs are automatically created by the system. These are the init log and the system log.

The init log records information created by the system when it boots.

The system log continuously records system level information whenever the system is running.

Both logs record information automatically and there is no user configuration required. The information recorded is determined by the system and there is no user configuration of the level of information recorded.

The commands used to retrieve and clear logs are described in hsmadmin logs.

Maximum log size

The init and system log share a common non-volatile storage area within the HSM. This storage area has the capacity to store the logs for a long period of normal usage but, if the logs are not periodically retrieved and cleared, it could eventually become full.

The system log is normally much larger than the init log but Entrust recommend that you clear both logs at the same time. This is because there are no specific warnings produced by actions that write to the init log. From firmware versions 13.5 onwards the init log sized is fixed and it is not necessary to clear the init log.

For HSMs running firmware version 13.5 or later, the system will detect when the logs have exceeded a safe working capacity and will prohibit the following actions:

These actions will continue to result in errors until the log volume is reduced back to a safe working level.

If the logs are not cleared promptly when the safe working capacity is exceeded the log volume may reach a critical capacity and at this point the system will enter an error state and display an error code on the LED. If this happens all actions are prevented other than retrieving and clearing the logs and it will be necessary to reboot the HSM to clear the error state.
It is not normally possible to clear the system log from HSMs running v13.5 or later firmware without first exporting the entries. However it is possible to do this in recovery mode. See Recovery Mode.

Interaction with the system clock

It is important that the timestamps in the system logs are accurate so that events can be correlated across the whole network in which the HSM is operating.

For HSMs running firmware version 13.5 or later, if the system clock is lost, for instance due to the HSM running on battery power for an extended period of time, a number of administrator actions will be prohibited until the system clock is restored, including:

See System interaction with the system clock for more information.

Init log

The init log records information that is produced each time the system boots.

The amount of information recorded depends on the firmware version loaded on the HSM.

The information is intended for use by Entrust Support to diagnose any issues that cause an HSM to fail to boot and thus the log is normally retrieved from recovery mode, see Recovery Mode.

Retrieving the init log

The init log can be retrieved with the following command:

hsmadmin logs get --esn <ESN> --log init [--json | --out <OUTFILE>]

Entrust recommend that you always direct the output to a file using the --out parameter and save the file for forwarding to Entrust Support.

Clearing the init log

The init log can be cleared with the following command:

hsmadmin logs clear [--esn <ESN>] --log init

Before running this command, place the unit in maintenance mode using nopclearfail -M -m <MODULEID> -w.

From firmware versions 13.5 onwards it is not necessary to clear the init log because it is cleared automatically each time the system successfully boots.

System log

The system log records system level events and warnings.

For HSMs running firmware version 13.5 or later these logs are produced in a signed format. HSMs running firmware earlier than 13.5 produce logs in an unsigned format.

The procedures for managing logs differ depending on whether the logs are signed or unsigned.

Signed system logs

Signed system logs are produced by firmware version 13.5 or later and have a number of benefits.

Signed logs can be verified to ensure that they have not been tampered with. See Verifying Signed Logs.

Signed logs contain a sequence number that prevents unintentional deletion of logs and can be used to help order stored logs and identify any gaps. See Log Sequence Number.

Log sequence number

Signed system logs use a sequence number to prevent the unintentional loss of logs and to aid in sorting logs.

The HSM holds an internal sequence number for the system log currently being written. This sequence number is persistent over both reboots and factory state operations.

When the system log is exported see Retrieving Signed Logs the sequence number is incremented and a new log is started with the new sequence number. The previous log is not deleted by the export command but remains stored internally within the HSM.

To prevent unintentional duplication of logs, signed logs can normally only be exported once. If it is required to export the same log for a second time the --saved option must be used with the export command.

Once a log has been successfully exported it can be cleared from the HSM as described in Expiring Signed Logs. The use of the sequence number in this command ensures that no logs are deleted that have not been exported.

It is possible to export and expire logs in a single command but Entrust do not recommend doing this because if the command fails for any reason there is a risk that logs may be lost. The recommended procedure is to export the logs first and then expire the logs as a separate procedure only once the export has completed successfully.

Each exported log begins with one entry showing its own sequence number and another entry showing the sequence number of the preceeding log. This allows logs to be chained together to identify any missing gaps.

Retrieving the system log

Retrieving unsigned logs

The system log can be retrieved with the following command:

hsmadmin logs get --esn <ESN> --log system [--json | --out <OUTFILE>]

This command will retrieve the logs in unsigned format and will work for HSMs running any firmware.

Retrieving signed logs

For HSMs running firmware version 13.5 or later Entrust recommend that system logs are automatically retrieved by configuring the nShield Audit Log service as described in nShield Audit Log Service. If you wish to retrieve the logs manually you can do so with the following command:

hsmadmin logs export --esn <ESN> [--json | --outdir <OUTDIR>] [--saved] [--expire]

This command automatically verifies the logs as part of the export process

Logs should normally be exported only once. If an attempt is made to export a log that has already been exported the command will fail with a warning. If you wish to export a log that has previously been exported you should use the --saved option with the above command.
When upgrading to Firmware version 13.5 or later from a firmware version lower than 13.5, there may initially be a saved log in the system created by the previous firmware. You should export this log using the --saved option and then expire it.
It is possible to automatically expire logs whilst exporting them by use of the --expire option but this is not recommended as it may result in loss of logs should the command fail for any reason.

Clearing the system log

The procedures for clearing the system log differ depending on whether the logs are produced in signed or unsigned format. If your HSM is running firmware version 13.5 or later your logs are produced in signed format. If your HSM is running a firmware version earlier than 13.5 your logs are produced in unsigned format.

The process of clearing a signed log is referred to as expiring.

Clearing unsigned logs

For HSMs running firmware versions earlier than 13.5 the log can be cleared with the following command:

hsmadmin logs clear [--esn <ESN>] --log system
For HSMs running firmware versions later than 13.5 this command will fail. You must follow the procedures described in Expiring Signed Logs.
It is possible to use this command to clear signed logs if the HSM is in recovery mode. See Recovery Mode. Entrust do not recommend this procedure unless instructed to do do by Entrust Support.

Expiring signed logs

If your HSM is running a firmware version of 13.5 or later the logs will be in signed format. To prevent unintentional loss of logs, signed logs must be exported before they can be cleared. You can export signed logs by following the procedures at Retrieving Signed Logs.

Logs that have previously been exported can be expired with the following command:

hsmadmin logs expire --esn <ESN> --seq SEQ_NO

This will expire the log with the sequence number SEQ_NO. The sequence number is included as the first line of any exported log.

It is possible to automatically expire logs whilst exporting them by use of the --expire option on the export command but this is not recommended as it may result in loss of logs should the command fail for any reason.

Verifying signed logs

Signed logs are automatically verified as part of the export process.

It is also possible to verify exported logs at any time in the future should you wish to do so, provided that you have access to the verification key. This verification can be conducted without access to the HSM.

The verification key is persistent over reboots but will be changed by a factory state operation. Therefore it is recommended that you record the verification key as soon as possible after any factory state operation. See Return to Factory State for information about factory state.

The system log will contain an entry for the factory state event. This log entry will contain the value of the verification key so it will always be possible to obtain the verification key if you forget to record it.

The current log verification key can be obtained with the following command:

hsmadmin logs getkey --esn <ESN> [--json | --out <OUTFILE>]

This command automatically validates the certificate protecting the verification key and produces a warning if it fails to find a certificate for the key. This warning is expected if the HSM:

  • is in recovery mode

  • has been upgraded to a firmware version of 13.5 or later but has not performed a factory state operation since the upgrade.

If you receive this warning in any other circumstance, contact https://nshieldsupport.entrust.com.

If you have upgraded to firmware version 13.5 or later, but have not performed a factory state operation since the upgrade, perform a factory state as soon as possible. This generates an internal certificate for the log signing key, which allows validation of the key all the way back to the root of trust.