Extract

• 1: Adobe Action Message Format
• 2: Amazon S3 Object
• 3: Binary Payload
• 4: CSV Payload
• 5: Common Event Format (CEF)
• 6: XML Payload
• 7: Date Time Format
• 8: XML with Tree-of-Trees (ToT)
• 9: Fixed Width
• 10: HTML Form Media Payload
• 11: HTTP Message Payload
• 12: Enhanced Adobe PDF Codec
• 13: JSON Payload
• 14: JSON with Tree-of-Trees (ToT)
• 15: Microsoft Office Documents
• 16: Multipart Mime Payload
• 17: PDF Payload
• 18: Protocol Buffer Payload
• 19: Secure File Transfer Payload
• 20: Shared File
• 21: SMTP Message Payload
• 22: Text Payload
• 23: URL Payload
• 24: User Defined Extraction Payload
• 25: ZIP Compressed File Payload

1 - Adobe Action Message Format

This payload extracts AMF format from the request and lets you define regex to control precise extraction.

The fields for Adobe Action Message Format (AMF) payload are as seen in the following figure.

The properties for the AMF payload are explained in the following table.

* The following options are available for the Method field.

• Serialize: Configure the AMF payload only to be exposed in learn mode. This will be useful for debugging while creating rules for learn mode.
• Serialized String Value: Configure the AMF payload as string and extract the matched Pattern.
• String Value: Configure the data using the matched Pattern. The data is not serialized ahead of the pattern matching.
• String Value by Key Name: The data is expected to come in key-value pairs. The parameters are matched using the Pattern. The value for the matched parameter is extracted.

2 - Amazon S3 Object

This payload extracts Amazon S3 object from the request and lets you define regex to control precise extraction. It is generally used with the Amazon S3 service.

The following figure illustrates the Amazon S3 Object payload fields.

The properties for the Amazon S3 Object payload are explained in the following table.

3 - Binary Payload

This payload extracts binary data from the request and lets you define regex to control precise extraction.

The fields for Binary payload are as seen in the following figure.

The properties for the Binary payload are explained in the following table.

The following table describes the fields for Encoding.

The following options are available for the Encoding field:

• No Encoding
• Standard
• External
• Proprietary

No Encoding

If the No Encoding option is selected, then no encoding is applied.

Standard

The Standard Encoding consists of built-in codecs of standard character encodings or mapping tables, including UTF-8, UTF-16, ASCII and more.

For more information about the complete list of encoding methods. refer to the section Standard Encoding Method List.

External

When external encoding is applied, you must select a codec.

The following table describes the codecs for the External encoding.

Proprietary

When proprietary encoding is selected, codecs linked are displayed.

The following table describes the codecs for the Proprietary encoding.

The following encryption methods are not supported for the High ASCII codec and the Base128 Unicode CJK codec:

• AES-128
• AES-256
• 3DES
• CUSP AES-128
• CUSP AES-256
• CUSP 3DES
• FPE NIST 800-38G Unicode (Basic Latin and Latin-1 Supplement Alpha)
• FPE NIST 800-38G Unicode (Basic Latin and Latin-1 Supplement Alpha-Numeric)

The following tokenization data types are not supported for the High ASCII codec and the Base128 Unicode CJK codec:

• Binary
• Printable

The input data for the Base128 Unicode CJK and High ASCII codecs must contain only ASCII characters. For example, if input data consisting of non-english characters is tokenized using the Alpha tokenization, then the Alpha tokenization treats the non-english characters as a delimiter and the tokenized output will include the non-english characters. As a result, the protection or unprotection operation will fail.

4 - CSV Payload

This payload extracts CSV format from the request and lets you define regex to control precise extraction.

With the Row and Column Index, you can now define how the column positions can be calculated. For example, consider a CSV input as provided in the following snippet, where the first column begins at the 0th field in the row, that are padded with commas until the next column, ex5 begins. This applies when the indexing is 0-based. If you choose to use 1-based indexing, the first column begins at 1 and subsequent fields are 2, 3, and so on. Based on these definitions, you can define the rule and its properties.

first, ex5, last, pick, ex6, city, ex1, ex2
John, wwww, Smith, mister, wwww, stamford, 333, 444
Adam, wwww, Martin, mister, wwww, fairfield, 333, 444

It is recommended to use the External Base64 encoding method in the Transform action type for the CSV codec. If the Standard Base64 method is used, then additional newline feeds are generated in the output.

The CSV implementation in the DSG does not support the following:

• The fields contain line breaks.
• If double quotes are used to enclose fields, then the double quotes appearing inside a field are escaped by preceding them with another double quote.

The fields for the CSV payload are as seen in the following figure.

If the CSV input includes NON-ASCII or Unicode data, then the Binary extract rule must be used before using the CSV extract rule.

If the CSV input file includes non-printable special characters, then to transform the data successfully, the user must add the csv-bytes-parsing parameter in the features.json file.

To add the parameter in the features.json file, perform the following steps.

1. Login to the ESA Web UI.
2. Navigate to Settings > System > Files.
3. Open the features.json file for editing.
4. Add the csv-bytes-parsing parameter in the features.json file. The csv-bytes-parsing parameter must be added in the following format:
   { "features": [ "csv-bytes-parsing" ] }

The properties for the CSV payload are explained in the following table.

5 - Common Event Format (CEF)

If you want to protect fields that are part of a CEF log file, you can use the CEF payload to extract the required fields.

The properties for the Common Event Format (CEF) payload are explained in the following table.

6 - XML Payload

This payload extracts the XML format content from the request and lets you extract the exact XML element value with it.

The fields for the XML payload are as seen in the following figure.

The properties for the XML payload are explained in the following table.

* The Advance XML Parser options field provides the following parsing parameters that can be configured.

Consider the following example to understand the Advance XML Parser options available in the XML codec. In this example, a request is sent from a client to remove the whitespaces between the XML tags from a sample XML payload in the message body of the HTTP/REST request. The following Ruleset is created for this example.

1. Create an extract rule for the HTTP message payload using the default RuleSet template defined under the REST API service.

   Consider the following sample XML payload in the HTTP message body.

   <?xml version = "1.0" encoding = "ASCII" ?>
   <class_list>
       <!--Students grades are uploaded by months-->
      <student>
            <name>John Doe</name>
         <grade>A</grade>
      </student>
   </class_list>
   

   In the example, a lot of white spaces are used for indentation. The payload contain spaces, carriage returns, and line feeds between the <class_list>, <student>, and <name> XML tags.

2. The extract rule for extracting the HTTP message body is as seen in the following figure.

   

3. Under the Extract rule, create another child rule to extract the XML payload from the HTTP Message.

   In this child rule, provide /class_list/student/name to parse the XML payload in the XPath List field and set the remove_blank_text parameter to true in the Advance XML Parser options field in the JSON format.

   

4. Under this extract rule, create another child rule to extract the sensitive data between the <name> and <name> tags. The fields for this child extract rule are as seen in the following figure.

   

5. Under the extract rule, create a transform rule to protect the sensitive data between the <name> and the <name> tags using Regex Replace with a pattern xxxxx. The fields for the transform rule are as seen in the following figure.

   

6. Click Deploy or Deploy to Node Groups to apply the configuration changes.

When a request is sent to the configured URI, the DSG processes the request and the following response appears with the whitespaces removed from the XML payload. In addition, the sensitive data between the <name> and the </name> tags is protected.

<?xml version='1.0' encoding='ASCII'?>
<class_list><!--Students grades are uploaded by months--><student><name>xxxxx xxxxx</name><grade>A</grade></student></class_list>

7 - Date Time Format

The Datetime format payload is used to convert custom datetime formats, which are not supported by tokenization datetime or date data element, to a supported format that can be processed by DSG.

Consider an example, where you provide a time format, such as DD/MM/YYYY HH:MM:SS as an input to an Extract rule with the Datetime payload. The given format is not supported by the datetime or date tokenization data element. The Extract rule converts the format to an acceptable format, a transform rule protects the datetime. The Datetime payload converts the protected value to the input format and returns this value to the user.

When you request DSG to unprotect the protected datetime value, an extract rule identifies the protected datetime value, a subsequent transform rule unprotects the value and returns the original datetime format, which is DD/MM/YYYY HH:MM:SS.

Ensure that the input sent to the extract rule for Date Time extraction is exactly in the same input format as configured in the rule. If you are unsure of the input that might be sent to the extract rule, then ensure that before you rollout for production, Ruleset configuration is thoroughly checked.

The following figure illustrates the Date Time format payload fields.

Before you begin:

Ensure that the following pre-requisites are completed:

• The datetime data element defined in the policy on ESA is used to perform protect or unprotect operation.

The following table describes the fields for Datetime codec.

*These fields appear only when Unprotect is selected as Mode of Operation.

8 - XML with Tree-of-Trees (ToT)

The XML with Tree-of-Trees (ToT) codec extracts the XML element defined in the XPath field. The XML with ToT codec allows you to process the multiple XML elements in an extract rule.

The fields for the XML with ToT payload is as seen in the following figure.

To understand the XML with ToT payload, consider the following example where the student details, such as, name, age, subject, and gender can be sent as a part of the request. In this example, the XML with ToT rule extracts and protects the name and the age element.

<?xml version='1.0' encoding='UTF-8'?>
<students>
 <student>
   <name>Rick Grimes</name>
   <age>35</age>
   <subject>Maths</subject>
   <gender>Male</gender>
 </student>
 <student>
   <name>Daryl Dixon </name>
   <age>33</age>
   <subject>Science</subject>
   <gender>Male</gender>
 </student>
 <student>
   <name>Maggie</name>
   <age>36</age>
   <subject>Arts</subject>
   <gender>Female</gender>
 </student>
</students>

The following figure illustrates one extraction rule for multiple XML elements.

In the Figure 14-25, the XML ToT Extract rule extracts the two different XML elements, name and age. The /students/student/name path extracts the name element and protect it with the transform rule. Similarly, the /students/student/age path extracts the age element and protect it with the transform rule. Same data elements are used to protect both the XML elements. You can use different data elements to transform XML elements as per your requirements. It is recommended to use the Profile reference, from the drop-down that appears on the Profile Name field. This helps to process the extraction and transformation in one rule. In addition, it reduces the transform overhead of defining one element at a time for the same XML file. If the Profile Name field is left empty then the extracted value is passed to the child rule for transformation.

For more information about profile referencing, refer to the section Profile Reference.

The properties for the XML with ToT payload are explained in the following table.

9 - Fixed Width

In scenarios where the input data is sent to DSG in a fixed width format, the Fixed Width codec is used. In a fixed width input, the data columns are specified in terms of exact column start character offset and fixed column width in terms of number of characters that define column width.

For example, consider a fixed width input as provided in the following snippet. The Name column begins at the 0th character in a row, has a fixed width of 20 characters and is padded with spaces until the next column Number begins. The Number column begins at the 20th character in a row and has a fixed width of 12 characters.

With the Row and Column Index, you can now define how the column positions can be calculated. If you choose to use 1-based indexing, the Name column begins at 1 and for fixed width of 20 characters, subsequent column will begin at the 21st character. While, if you use 0-based indexing, the Name column begins at 0 and for fixed width of 20 characters, subsequent column will begin at the 20th character. Based on these definitions, you can define the rule and its properties.

Name                         Number
John Smith                   418-Y11-4111
Mary Hartford               319-Z19-4341
Evan Nolan                   465-R45-4567

The fields for the Fixed Width payload are as seen in the following figure.

Note: If the input file includes non-printable special characters, then to transform the data successfully, the user must add the fw-bytes-parsing parameter in the features.json file.
To add the parameter in the features.json file, perform the following steps.

1. Login to the ESA Web UI.
2. Navigate to Settings > System > Files.
3. Open the features.json file for editing.
4. Add the fw-bytes-parsing parameter in the features.json file. The fw-bytes-parsing parameter must be added in the following format:
   { "features": [ "fw-bytes-parsing" ] }

The properties for the Fixed Width payload are explained in the following table.

10 - HTML Form Media Payload

This payload extracts HTML form media format from the request and lets you define regex to control precise extraction.

The fields for the HTML Form Media Payload (X-WWW-FORM-URLENCODED) payload are as seen in the following figure.

The properties for the X-WWW-FORM-URLENCODED payload are explained in the following table.

11 - HTTP Message Payload

This payload extracts HTTP message format from the request and lets you define regex to control precise extraction.

The following figure illustrates the HTTP Message payload fields.

The properties for the HTTP Message payload are explained in the following table.

12 - Enhanced Adobe PDF Codec

The Enhanced Adobe PDF codec extracts the PDF payload from the request and lets you define Regex to control precise extraction. This payload is available when the Action type is selected as Extract.

As part of the ruleset construction for this codec, it is mandatory to include a child Text extract rule under the Enhanced Adobe PDF codec extract rule. You must not use any other rule apart from the child Text extract rule under the Enhanced Adobe PDF codec extract rule.

In the DSG, some font files are already added to the /opt/protegrity/alliance/config/pdf_fonts directory. By default, the following font file is set in the gateway.json file.

"pdf_codec_default_font":{
"name": "OpenSans-Regular.ttf"
 }

Note: The Advanced Settings can be used to configure the default font file for a specific rule.

If you want to process a PDF file that contains custom fonts, then upload it to the /opt/protegrity/alliance/config/pdf_fonts directory. If the custom fonts are not uploaded to the mentioned directory, then the OpenSans-Regular.ttf font file will be used to process the PDF file.

For more information about how-to examples to detokenize a PDF, refer to the section Using Amazon S3 to Detokenize a PDF and Using HTTP Tunnel to Detokenize a PDF in the Protegrity Data Security Gateway How-to Guide.

The following figure displays the Enhanced Adobe PDF payload fields.

The properties for the Enhanced Adobe PDF payload are explained in the following table.

Note: The configurations in the Advanced Settings are only applicable for that specific rule.

Known Limitations

The following list describes the known limitations for this release.

1. The Enhanced Adobe PDF codec does not support detokenization for sensitive data that splits into multiple lines. It is expected that the prefix, data to be detokenized, and the suffix are in a single line and do not break into multiple lines.
2. The embedded fonts are not supported. Ensure that when you are uploading the fonts, the entire character set for that font family is uploaded to the DSG.
3. The prefix and suffix used to identify the data to be detokenized must be unique and not a part of the data.
4. The PDFs created with the rotate operator are not supported for detokenization.
5. The Enhanced Adobe PDF codec does not process password protected PDFs.
6. The detokenized data appears spaced out with extra white spaces.

13 - JSON Payload

This codec extracts the JSON element from the JSON request as per the JSONPATH defined.

Consider the following sample input that will be processed using the JSON codec to extract a unique JSON element:

{  
    "entities":[  
        {  
            "entity_type":"CostCenter",
            "properties":{  
                "Id":"10097",
                "LastUpdateTime":1455383881190,
                "Currency":"USD",
                "ApproveThresholdAmount":100000,
                "AccountingCode":"5555",
                "CostCenterAttachments":"{\"complexTypeProperties\":[]}"
            }
        }
    ],
    "operation":"UPDATE"
}

In the Extract rule, assuming that the AccountingCode needs to be protected, the JSONPath that will be set is entities[*].properties.AccountingCode. Based on the input JSON structure, the JSONPath value differs.

The following figure illustrates the JSON payload fields.

The properties for the JSON payload are explained in the following tab

The following table describes the additional configuration option for the Recurse mode.

JSONPath Examples

This section provides guidance on the type of JSONPath expressions that DSG understands. This guidance must be considered before you define the acceptable JSONPath to be extracted when using the JSON codec.

The DSG supports the following operators.

CAUTION: The $ operator is not supported.

To understand the JSONPath, consider the following example JSON. The subsequent table provides JSONPath examples that can be used with the example JSON.

{
  "store": {
    "book": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      {
        "category": "fiction",
        "author": "J. R. R. Rowling",
        "title": "Harry Potter and Chamber of Secrets",
        "isbn": "0-395-12345-8",
        "price": 29.99
      },
      {
        "category": "fiction",
        "author": "J. R. R. Tolkien ",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99
      },
      {
        "category": "fiction",
        "author": "Arthur Conan Doyle ",
        "title": "Sherlock Homes",
        "isbn": "0-795-19395-8",
        "price": 9
      }
    ]
  }
}

The following table provides the JSONPath examples based on the JSON example.

14 - JSON with Tree-of-Trees (ToT)

This section provides an overview of the JSON with Tree-of-Trees (ToT) payload. The JSON ToT payload allows you to use the advantages offered by Tree of Trees to extract the JSON payload from the request and provide protection according to the data element defined. Profile Reference can also be used to process different elements of the JSON.

The following figure illustrates the JSON ToT fields.

The properties for the JSON ToT payload are explained in the following table:

The following table describes the additional configuration option for the Recurse mode.

15 - Microsoft Office Documents

This payload extracts Microsoft Office documents from the request and lets you define regex to control precise extraction.

The following figure illustrates the MS Office payload fields.

The properties for the Microsoft Office documents payload are explained in the following table.

16 - Multipart Mime Payload

This payload extracts mime payload from the request and lets you define regex to control precise extraction.

The following figure illustrates the Multipart Mime payload.

The properties for the Multipart Mime payload are explained in the following table.

17 - PDF Payload

This payload extracts PDF payload from the request and lets you define regex to control precise extraction.

The following figure illustrates the PDF payload fields.

The properties for the PDF payload are explained in the following table.

Note: The DSG PDF codec supports only text formats in PDFs.
For any assistance in supporting additional text formats, contact Protegrity Professional Services.

18 - Protocol Buffer Payload

The PBpath defines a way to address fields in binary encoded protocolbuf messages. It uses field ids to construct an address messages or fields in a nested message hierarchy.

An example for the PBpath field is shown as follows:

 1.101.2.201.301.2.401.701.2.802

In DSG, protocol buffer version 2 is used.

The following figure illustrates the Protocol Buffer (protobuf) payload fields.

The properties for the Protocol Buffer payload are explained in the following table.

19 - Secure File Transfer Payload

This payload extracts SFTP message from the request and lets you further processing to be done on the files.

The following figure illustrates the Secure File Transfer payload fields.

The properties for the Secured File Transfer payload are explained in the following table.

20 - Shared File

This payload extracts file from the request and lets you define regex to control precise extraction. It is generally used with Mounted services, namely NFS and CIFS.

The following figure illustrates the NFS/CIFS share-related Shared File payload fields.

The properties for the Shared File payload are explained in the following table.

21 - SMTP Message Payload

This payload extracts SMTP payload from the request and lets you define regex to control precise extraction.

The following figure illustrates the SMTP message payload fields.

The properties for the SMTP payload are explained in the following table.

22 - Text Payload

This payload extracts text payload from the request and lets you define regex to control precise extraction.

The following figure illustrates the Text payload fields.

The properties for the Text payload are explained in the following table.

23 - URL Payload

This payload extracts URL payload from the request and extract precise object based on selection.

The following figure illustrates the URL payload fields.

The properties for the URL payload are explained in the following table.

24 - User Defined Extraction Payload

This codec lets you define custom extraction logic and pass arguments to the next rule. The language that is currently supported for extraction is Python.

From DSG 3.0.0.0, the Python version is upgraded to python 3. The UDFs written in Python v2.7 will not be compatible with Python v3.10. To migrate the UDFs from python 2 to python 3, refer to the section Migrating the UDFs to Python 3.

The following figure illustrates the User Defined Extraction payload fields.

The properties for the User Defined Extraction payload are explained in the following table.

Note: The DSG supports the usage of the PyJwt python library in custom UDF creations. PyJWT is a python library that is used to implement Open Authentication (OAuth) using JSON Web Tokens (JWT). JSON Web Tokens (JWT) is an open standard that defines how to transmit information between a sender and a receiver as a JSON object. To authenticate JWT for OAuth, you must write a custom UDF. The PyJwt library version supported by the DSG is 1.7.1.
For more information about writing a custom UDF on the DSG, refer to the section User Defined Functions (UDF).

Note: The DSG supports the usage of the Kafka python library in custom UDF creations. Kafka is a python library that is used for storing, processing, and forwarding for applications in a distributed environment. For example, the DSG uses the Kafka library to forward Transaction Metrics logs to external applications. The Kafka library version supported by the DSG is 2.0.2.
For more information about writing a custom UDF on the DSG, refer to the section User Defined Functions (UDF).

Note: The DSG supports the usage of the Openpyxl Python library in custom UDF creations. Openpyxl is a Python library that is used to parse Excel xlsx, xlsm, xltx, xltm files. This library enables column-based transformation for Microsoft Office Excel. The Openpyxl library version supported by the DSG is 2.6.4.

Note: The DSG uses the in-built tarfile python module for custom UDF creation. This module is used in the DSG to parse .tar and .tgz packages. Using the tarfile module, you can extract and decompress .tar and .tgz packages.

25 - ZIP Compressed File Payload

This payload extracts ZIP file from the request and lets you extract file name or file content.

The following figure illustrates the ZIP Compressed File payload fields.

The properties for the ZIP payload are explained in the following table.