Delimiters
A delimiter refers to a group of one or more characters which are used in data, such as mathematical expressions or plain text to separate data.
Protegrity tokenization can generate the same token regardless of how the data is formatted. Any character in the input that does not comply with the token types in the Tokenization Types is generally treated as a delimiter and remains unchanged during tokenization.
The following table shows how the Protegrity Token types handles delimiters and spaces as compared to plain numerical data.
Table: Tokenization with Delimiters
Note: Some tokenizers can tokenize delimiters. Unicode Gen2, lower ASCII, printable, and binary are examples of tokenizers that can tokenize delimiters.
| Input | Value returned by Protegrity Tokenization |
|---|---|
| 5332711989955364 | 8344588301109112 |
| 5332-7119-8995-5364 | 8344-5883-0110-9112 |
| 5332 7119 8995 5364 | 8344 5883 0110 9112 |
Feedback
Was this page helpful?