Data Type and Alphabet
The data type specifies the data that should be tokenized, for instance with the characters to expect as input and the output to generate.
An alphabet contains all characters considered for tokenization, it is derived from the tokenization type. Characters outside the alphabet are considered delimiters.
Note: This is applicable only for Unicode Gen2 token.
Refer to Tokenization Types for the full list of supported token types.
Feedback
Was this page helpful?