Datetime (YYYY-MM-DD HH:MM:SS)

The Datetime token type was introduced in response to requirements to allow specific date parts to remain in the clear and for date tokens to be distinguishable from real dates. The Datetime token type allows time to be tokenized (HH:MM:SS) in fractions of a second, including milliseconds (MMM), microseconds (mmmmmm), and nanoseconds (nnnnnnnnn).

Extended DateTime Tokenization with Timezone Offsets

The ISO 8601 DateTime format with timezone offsets are supported only in Application Protector .NET 10.0.1. Other protectors do not support ISO 8601 formatted DateTime inputs.

The extended DateTime tokenization now supports ISO 8601 formatted dates including timezone offsets. For example, +05:30. The tokenizer applies protection only to the date and time element up to seconds. The fractional seconds and additional identifiers remains unchanged. The Delimiters are preserved.

Supported Format:

• Examples of valid input:
  • YYYY-MM-DD HH:MM:SS+hh:mm
  • YYYY.MM.DD HH:MM:SS+hh:mm
• Maximum length: 37 bytes
  For example, the longest supported format is -

  2019-11-07 13:37:00:000000000+05:30
  

Table: Datetime Tokenization Type properties

Note:
^*1 - Limitation. For more information, refer to Extended DateTime Tokenization with Timezone Offsets.

The Tokenize Time property defines whether the time part (HH:MM:SS) will be tokenized. If Tokenize Time is set to “No”, the time part will be treated as a delimiter. It will be added to the date after tokenization.

The Distinguishable Date property defines whether the tokenized values will be outside of the normal date range.

If the Distinguishable Date option is enabled, then all tokenized dates will be in the range from year 5596-09-06 to 8334-08-03. The tokenized value will become recognizable. As an example, tokenizing “2012-04-25” can result in “6457-07-12”, which is distinguishable.

If the Distinguishable Date option is disabled, then the tokenized dates will be in the range from year 0600-01-01 to 3337-11-27. As an example, tokenizing “2012-04-25” will result in “1856-12-03”, which is non-distinguishable.

The Date in Clear property defines whether Month or Year will be left in the clear in the tokenized value.

Note: You cannot use enabled Distinguishable Date and select month or year to be left in the clear at the same time.

The following points are applicable when you tokenize the Dates with Year as 3337 by setting the Year part to be in clear:

• The tokenized Date value can be outside of the accepted Date range.
• The tokenized Date value can be de-tokenized to obtain the original Date value.

For example, if the Date 3337-11-27 is tokenized by setting the Year part 3337 in clear, then the resultant tokenized value 3337-12-15 is outside of the accepted Date range. The detokenization of this tokenized value returns the original Date 3337-11-27.

The following table shows examples of the way in which a value will be tokenized with the Datetime token.

Table: Examples of Tokenization for DateTime Values

Note:
^*1 - Limitation. For more information, refer to Extended DateTime Tokenization with Timezone Offsets.

Datetime Tokenization for Cutover Dates of the Proleptic Gregorian Calendar
The data systems, such as, Oracle or Java-based systems, do not accept the cutover dates of the Proleptic Gregorian Calendar. The cutover dates of the Proleptic Gregorian Calendar fall in the interval 1582-10-05 to 1582-10-14. These dates are converted to 1582-10-15. When using Oracle, conversion occurs by adding ten days to the source date. Due to this conversion, data loss occurs as the system is not capable to return the actual date value after the de-tokenization.

Note: The tokenization of the Date values in the cutover Date range of the Proleptic Gregorian Calendar results in an “Invalid Input” error.

The following points are applicable when the Distinguishable Date option is disabled:

• If the Distinguishable Date option is disabled, then the tokenized dates are in the range 0600-01-01 to 3337-11-27, which also includes the cutover date range. During tokenization, an internal validation is performed to check whether the value is tokenized to the cutover date. If it is a cutover date, then the Year part (1582) of the tokenized value is converted to 3338 and then returned.
• During de-tokenization, an internal check is performed to validate whether the Year is 3338. If the Year is 3338, then it is internally converted to 1582.

The following points are applicable when you tokenize the dates from the Year 1582 by setting the Year part to be in clear:

• The tokenized value can result in the cutover Date range. In such a scenario, the Year part of the tokenized Date value is converted to 3338.
• During de-tokenization, the Year part of the Date value is converted to 1582 to obtain the original date value.

For example, if the date 1582.04.30 12:12:12 is tokenized by setting the Year part in clear and the resultant tokenized value falls in the cutover Date range, then the Year part is converted to 3338 resulting in a tokenized value as 3338.10.10 12:12:12. The de-tokenization of this tokenized value returns the original Date 1582.04.30 12:12:12.

Note:
The tokenization accepts the date range 0600-01-01 to 3337-11-27 excluding the cutover date range.
The de-tokenization accepts the date range 0600-01-01 to 3337-11-27 and date values from the Year 3338. The year 3338 is accepted due to our support for tokenized value from the cutover date range.

Consider a scenario where you are migrating the protected data from Protector 1 to Protector 2. The Protector 1 includes the Datetime tokenizer update to process the cutover dates of the Proleptic Gregorian Calendar as input. The Protector 2 does not include this update. In such a scenario, an “Invalid Date Format” error occurs in Protector 2, when you try to unprotect the protected data as it fails to accept the input year 3338. The following steps must be performed to mitigate this issue:

1. Unprotect the protected data from Protector 1.
2. Migrate the unprotected data to Protector 2.
3. Protect the data from Protector 2.

Time zone Normalization for Datetime Tokens
The Datetime tokenizer does not normalize the timestamp with respect to the timezone before protecting the data.

In a few Protectors, the timezone normalization is done by the APIs that are used by the Protectors to retrieve the timestamp. However, this behavior can also be configured.

There are differences in handling timestamps. Therefore, you cannot rely on Datetime tokens for migration or transfer to different systems or timezones.

So, before migrating the Datetime tokens, ensure that the timestamps are normalized for timezones so that unprotecting the token value returns the original expected value.

Datetime Tokenization Properties for different protectors

Application Protector

The following table shows supported input data types for Application protectors with the Datetime token.

Table: Supported input data types for Application protectors with Datetime token

^*1 - The API accepts and returns data in BYTE[] format. The customer application needs to convert the input into byte arrays before calling the API, and similarly, convert the output from byte arrays after receiving the response from the API.

^*2 - The Protegrity Application Protectors only support bytes converted from the string data type. If int, short, or long format data is directly converted to bytes and passed as input to the Application Protector APIs that support byte as input and provide byte as output, then data corruption might occur.

For more information about Application protectors, refer to Application Protector.