Transaction IDs

Introduction

Transaction IDs, also known as correlation IDs, are identifiers that allow a request to be traced through different layers, components, and log files of a system.

nCore supports Transaction IDs in conjunction with the Audit Logging scheme, provided in firmware versions v13.5 and later, for nShield Solo XC, Connect XC, 5s, and 5c HSMs.

Transaction IDs are represented in nCore as UTF-8 strings that are permitted to be up to 88 bytes in length, not including the terminator. This is sufficient to represent the base64 of a SHA512 or SHA3-512 hash.

By using human-readable strings, you can choose how the IDs appear when printed in nCore client logs and audit logs. You can also use this string to encode data such as sequence numbers, UUIDs, cryptographic hashes, or hex or base64 encodings of user-supplied data.

Limitations

Some limitations are present in this release.

Sending commands with Transaction IDs set to HSMs running firmware versions older than v13.5 is not supported.
- If the attempt is made, commands will fail with error Status_UnknownFlag.
Transaction IDs are preserved where the command is directly forwarded from the client to the HSM. Where commands are implemented internally by the Security World software, the linkage might not be preserved.
- In particular, this can apply to Cmd_Destroy when sent from client-side applications, as objects are reference counted in the hardserver, so the actual HSM Cmd_Destroy command is issued by the hardserver when the reference count reaches 0. This limitation does not apply to CodeSafe, which does not use hardserver.
- Client-side code sending large commands, such as Cmd_Decrypt, with a symmetric key on a large ciphertext might be automatically split by library code into multiple commands of smaller buffers. Linkage is not preserved when the Transaction ID is set directly on the M_Command object, but if the Transaction ID is set on the connection, as a thread-local, or on any of the other supported interfaces besides setting on command directly, then it will be.

Unicode Notes

Because the Transaction ID strings can be UTF-8, the following filters and features are present to mitigate potential issues when displaying them in JSON or text output in the nshieldaudit tool. Where filtering occurs, one or more consecutive filtered characters are replaced with an underscore _ character as the replacement character.

Control characters other than new-line, carriage-return, and tab are filtered.
Whitespace, including non-ASCII Unicode whitespace, is filtered when outputting in the single-line text representation, but is not filtered in the full JSON output.
Bidirectional text formatting characters are filtered.
The option to restrict to only ASCII characters can be enabled by setting the environment variable NSHIELDAUDIT_ASCII_ONLY=1 in the environment of the nshieldaudit tool when performing the export.

Additional options may be provided for controlling permitted characters in a future release.

Setting Transaction IDs

Transaction IDs are associated with an nCore command submitted by client code.

nShield client-side software and libraries support use of the NFAST_TRANSACTION_ID environment variable as a way to inject a Transaction ID into any operation.

It can also be set programmatically in nCore client libraries, as described in the following sections. In each case, library code will automatically copy and truncate the string to the 88 byte limit.

nCore C (Generic Stub)

When a command is submitted, for example by using NFastApp_Submit or NFastApp_Transact, library code searches for a Transaction ID that has been set by the following functions, in the following order of precedence:

Set directly on an M_Command object using the helper functions NFastApp_SetTransactionID for heap-allocation, where command will be freed, or NFastApp_SetTransactionID_NoFree where all memory is on stack.
A user-supplied callback function, "upcall", for retrieving the Transaction ID from user call context or transaction context that has been provided to the library when NFastApp_InitEx was called.
Thread-local value set via NFastApp_SetThreadTransactionID.
Set for a given NFastApp_Connection using NFastApp_SetConnTransactionID.
Set for a given NFast_AppHandle using NFastApp_SetAppTransactionID.
NFAST_TRANSACTION_ID environment variable.

Corresponding "Get" functions also exist for each of these "Set" functions.

The caller provides Transaction ID strings as const char * objects that can contain UTF-8 characters.

SEElib (CodeSafe CSEE)

Helper functions SEElib_SetTransactionID and SEElib_SetTransactionID_NoFree provide the corresponding functionality to their NFastApp_* equivalents for setting a UTF-8 Transaction ID string on an M_Command directly.

Other mechanisms of setting the Transaction ID are not provided in this case. This includes the NFAST_TRANSACTION_ID environment variable.

nCore Python (nfpython)

When a command is submitted, for example by using the submit() or transact() methods of an nfpython.connection object, library code searches for a Transaction ID that has been set by the following functions, in the following order of precedence:

Set directly on an nfpython.Command object using the nfpython.set_transaction_id() function.
Thread-local value set using the nfpython.set_thread_transaction_id() function.
Set for a given nfpython.connection object using the set_transaction_id() method of the object.
NFAST_TRANSACTION_ID environment variable.

Corresponding "Get" functions also exist for each of these "Set" functions.

nCore Java (nfjava)

When a command is submitted, for example by using the NFConnection.submit() or NFConnection.transact() methods, library code searches for a Transaction ID that has been set by the following functions. It observes the following order of precedence:

Set directly on an M_Command object using the static helper function NFUtils.setTransactionID().
Thread-local value set via static helper function NFConnection.setThreadTransactionID.
Set for a given NFConnection instance using NFConnection.setTransactionID method of the connection object.
NFAST_TRANSACTION_ID Java system property.
NFAST_TRANSACTION_ID environment variable.

Corresponding "Get" functions also exist for each of these "Set" functions.

Higher-level APIs

The thread-local Transaction ID setting function NFastApp_SetThreadTransactionID, and its Python and Java equivalents, does not take any library handles as arguments. This means that, in many cases, it can be called by higher-level code to provide the Transaction ID without having to directly modify or pass through to lower layers.

Because the storage for this function is thread-local, it can safely be used in a multi-threaded application. NFastApp_SetThreadTransactionID can also be be called with a NULL pointer to unset the thread-local, preventing subsequent code from unintentionally using the Transaction ID outside of the intended scope.

Setting the NFAST_TRANSACTION_ID environment variable also provides an implementation-agnostic and language-agnostic way to provide a Transaction ID. This can be done for a whole program, for example, where a whole request is self-contained, which might be the case with something like signtool. It can also be used dynamically in-process, because library code re-checks the environment variable each time an nCore command is submitted. Updating dynamically is only suitable when a single thread of the application is submitting nCore commands, because environment variables are process-wide.

Supporting additional setters for particular higher-level APIs like PKCS #11 or CNG, for example supporting setting the Transaction ID via an attribute or property on a relevant object or handle in the higher-level API, may be considered in future if there is enough demand and if existing interfaces are insufficient for user needs.

Transaction ID logging

Client debug dogs

Transaction IDs appear in client-side debug logs as part of the traced nCore commands, as demonstrated in the following example produced from the nCore C Generic Stub library using the NFLOG_SEVERITY=DEBUG1 environment variable.

The Transaction ID string in this example is ExampleTransactionID1234.

Generic Stub nCore Log

00:40:56 DEBUG1: NFastApp_Submit tag=17cf08be; conn=0x55851a740800; reply=0x55851a771e30; time=1732495256
  command.tag= 0x00000000 0
         .cmd= Sign
         .status zero
         .flags= sessioninfo_present 0x00000200
         .state absent
         .args.sign.flags= none 0x00000000
                   .key= 0x9e4fa111 2656018705 2656018705
                   .mech= RSApPKCS1
                   .plain.type= Bignum
                         .data.bignum.m= 128 bytes
                   .given_iv absent
         .certs absent
         .extractstate absent
         .sessionid absent
         .sessioninfo.v= 0x00000000 0
                     .flags= transaction_present 0x00000004
                     .transaction= "ExampleTransactionID1234"

nCore audit logs

Transaction IDs also appear in nCore audit logs emitted by firmware version v13.5 or later.

Text audit logs

nCore audit logs can be exported from the audit database, using the nshieldaudit command, in a condensed human-readable text format summarizing the most important information in one audit entry per line.

The Transaction ID is referenced with the trID= field in these logs, for example:

nCore text audit log

2024-11-25 00:40:56.254 idx=1940686 src=Host trID=ExampleTransactionID1234 cmd=Sign rc=OK obj=982

nCore JSON audit logs

nCore audit logs can be exported from the audit database in JSON format showing the full detail of the underlying nCore audit data. This is most useful if more context is needed or where a machine-readable format is needed for additional processing.

The following example displays the portion of a JSON audit segment that describes an audit event for a Sign command, similar to the one shown in the Client debug log:

nCore JSON audit segment

{
    "v": 0,
    "timestamp": 1732495256254,
    "source": "Host",
    "infos": [
    {
        "type": "Command",
        "body": {
        "flags": [
            "sessioninfo_present"
        ],
        "cmd": "Sign",
        "info": {},
        "sessioninfo": {
            "v": 0,
            "flags": [
            "transaction_present"
            ],
            "transaction": "ExampleTransactionID1234"
        },
        "status": "OK"
        }
    },
    {
        "type": "ObjectUse",
        "body": {
        "v": 0,
        "objid": 982,
        "action": {
            "type": "OpPermissions",
            "details": {
            "perms": [
                "Sign"
            ]
            }
        }
        }
    }
    ]
}