Static Lookup Table (SLT) Tokenizers
A static lookup table (SLT) contains a pre-generated list of all possible values from a given set of characters. An alphabetic lookup table for instance might contain all values from “Aa” to “Zz”. All entries are then shuffled so that they are in random order.
SLT tokenizer uses multiple SLTs to generate tokens. This is done by first dividing the input value into smaller pieces, called token blocks, which correspond to entries in the lookup tables. The token blocks are then substituted with values from the SLTs and chained together to form the final token value. This means that the token is a result of multiple lookups in multiple SLTs.
Another benefit of SLT tokenizers is that tokenization can be done locally on the protector. With this solution, tokenization is performed locally within the protector environment.
For more information, refer to Working with Data Elements.
There are several types of SLT tokenizers from which you can choose. They are distinguished by their block size and the number of lookup tables.
Table: SLT Tokenizer with block size and lookup tables
| Tokenizer | Allow Short Tokens | No. of lookup tables | Block size |
|---|---|---|---|
| SLT_1_3 | Yes | 1 | 1 |
| 1 | 2 | ||
| 1 | 3 | ||
No, return input as it is No, generate error | 1 | 3 | |
| SLT_2_3 | Yes | 2 | 1 |
| 2 | 2 | ||
| 2 | 3 | ||
No, return input as it is No, generate error | 2 | 3 | |
| SLT_1_6 | Yes | 1 | 1 |
| 1 | 2 | ||
| 1 | 3 | ||
| 1 | 6 | ||
No, return input as it is No, generate error | 1 | 6 | |
SLT_2_6 | Yes | 2 | 1 |
| 2 | 2 | ||
| 2 | 3 | ||
| 2 | 6 | ||
No, return input as it is No, generate error | 2 | 6 | |
| SLT_6_DECIMAL | NA | Multiple lookup tables: One for each input length in the range 1 to 5 One for input lengths >= 6 | |
| SLT_DATETIME | NA | Multiple lookup tables | |
| SLT_X_1 | Yes | 5-98*1 | 1 |
No, return input as it is No, generate error | 3-96*1 | 1 | |
*1 - For the SLT_X_1 tokenizer, the number of lookup tables used for the security operations is determined during the creation of the data elements.
The following table describes the types of SLT tokenizers and compares their characteristics.
Table: SLT Tokenizer Memory Footprint for Token Types
| Token Type | Tokenizer | Allow Short Tokens | Size of Token Tables (number of entries) | Size of Token Tables (kB) | Amount of Memory used in the Protector (kB) | Comments |
|---|---|---|---|---|---|---|
| Numeric | SLT_1_3 SLT_2_3 SLT_1_6 SLT_2_6 | No, generate error No, return input as it is | 1,000 2,000 1,000,000 2,000,000 | 4 8 3,906 7,812 | 8 16 7,812 15,624 | |
| Yes | 1,110 2,220 1,001,110 2,002,220 | 4.33 8.66 3,910.58 7,821.17 | 8.66 17.32 7,821.17 15,642.34 | |||
| Integer | SLT_1_3 | NA | 4096 | 16 | 32 | |
| Credit Card | SLT 1_3 SLT 2_3 SLT 1_6 SLT 2_6 | NA | 1,000 2,000 1,000,000 2,000,000 | 4 8 3,906 7,812 | 8 16 7,812 15,624 | |
| Alpha | SLT 1_3 SLT 2_3 | No, generate error No, return input as it is | 140,608 281,216 | 549 1,098 | 1,098 2,196 | |
| Yes | 143,364 286,728 | 560.01 1,120.02 | 1,120.02 2,240.04 | |||
| Upper-case Alpha | SLT 1_3 SLT 2_3 | No, generate error No, return input as it is | 17,576 35,152 | 69 138 | 138 276 | |
| Yes | 18,278 36,556 | 71.39 142.79 | 142.79 285.59 | |||
| Alpha-Numeric | SLT 1_3 SLT 2_3 | No, generate error No, return input as it is | 238,328 476,656 | 931 1,862 | 1,862 3,724 | |
| Yes | 242,234 484,468 | 946.22 1,892.45 | 1,892.45 3,784.90 | |||
| Upper-Case Alpha-Numeric | SLT 1_3 SLT 2_3 | No, generate error No, return input as it is | 46,656 93,312 | 182 364 | 364 728 | |
| Yes | 47,988 95,976 | 187.45 374.90 | 374.90 749.81 | |||
| Lower ASCII | SLT 1_3 | No, generate error No, return input as it is | 830,584 | 3,244 | 6,488 | |
| Yes | 839,514 | 3,279.35 | 6,558.70 | |||
| Datetime | SLT_DATETIME | NA | 1,086,400 | 4,244 | 8,488 | Maximum memory is used when both date part and time part will be tokenized |
| Decimal | SLT_6_DECIMAL | NA | 597,870 | 2,335 | 4,670 | |
| Unicode Gen2 | SLT_1_3 SLT_X_1 | No, generate error No, generate error No, return input as it is | 4,096,000 359,994 | 16,384 1,440 | 32,768 2,880 | |
SLT_1_3 SLT_X_1 | Yes Yes | 4,121,760 500,000 | 16,488 2,000 | 32,975 4,000 | ||
| Binary | SLT_1_3 SLT_2_3 | NA | 238,328 476,656 | 931 1,862 | 1,862 3,724 | Same tokenizers and other values as for Alpha-Numeric token element |
SLT_1_3 SLT_2_3 | No, generate error No, return input as it is | 238,328 476,656 | 931 1,862 | 1,862 3,724 | Same tokenizers and other values as for Alpha-Numeric token element | |
| Yes | 242,234 484,468 | 946.22 1,892.45 | 1,892.45 3,784.90 |
Note: The amount of memory used in the protector is twice the size of the token tables (kB) because an inverted SLT is stored in the memory, in addition to the original SLT.
Table: SLT Tokenizer Characteristics for Deprecated Token Types
| Token Type | Tokenizer | Allow Short Tokens | Size of Token Tables (number of entries) | Size of Token Tables (kB) | Amount of Memory used in the Protector (kB) | Comments |
|---|---|---|---|---|---|---|
| Printable | SLT 1_3 | No, generate error No, return input as it is | 6,967,871 | 27,218 | 54,436 | |
| Yes | 7,004,543 | 27,361.49 | 54,722.99 | |||
| Date YYYY-MM-DD | SLT_1_3 SLT_2_3 SLT_1_6 SLT_2_6 | NA | 1,000 2,000 1,000,000 2,000,000 | 4 8 3,906 7,812 | 8 16 7,812 15,624 | |
| Date DD/MM/YYYY | SLT_1_3 SLT_2_3 SLT_1_6 SLT_2_6 | NA | 1,000 2,000 1,000,000 2,000,000 | 4 8 3,906 7,812 | 8 16 7,812 15,624 | |
| Date MM.DD.YYYY | SLT_1_3 SLT_2_3 SLT_1_6 SLT_2_6 | NA | 1,000 2,000 1,000,000 2,000,000 | 4 8 3,906 7,812 | 8 16 7,812 15,624 | |
| Unicode | SLT_1_3 SLT_2_3 | No, generate error No, return input as it is | 238,328 476,656 | 931 1,862 | 1,862 3,724 | Same tokenizers and other values as for Alpha-Numeric token element |
| Yes | ||||||
| Unicode Base64 | SLT_1_3 SLT_2_3 | No, generate error No, return input as it is | 274,625 549,250 | 1,073 2,146 | 2,146 4,292 | Same tokenizers and other values as for Alpha-Numeric token elements. It also includes +, /, and =. |
| Yes |
Feedback
Was this page helpful?