Actions

• 1: Error
• 2: Exit
• 3: Extract
• 3.1: Adobe Action Message Format
• 3.2: Amazon S3 Object
• 3.3: Binary Payload
• 3.4: CSV Payload
• 3.5: Common Event Format (CEF)
• 3.6: XML Payload
• 3.7: Date Time Format
• 3.8: XML with Tree-of-Trees (ToT)
• 3.9: Fixed Width
• 3.10: HTML Form Media Payload
• 3.11: HTTP Message Payload
• 3.12: Enhanced Adobe PDF Codec
• 3.13: JSON Payload
• 3.14: JSON with Tree-of-Trees (ToT)
• 3.15: Microsoft Office Documents
• 3.16: Multipart Mime Payload
• 3.17: PDF Payload
• 3.18: Protocol Buffer Payload
• 3.19: Secure File Transfer Payload
• 3.20: Shared File
• 3.21: SMTP Message Payload
• 3.22: Text Payload
• 3.23: URL Payload
• 3.24: User Defined Extraction Payload
• 3.25: ZIP Compressed File Payload
• 4: Log
• 5: Profile Reference
• 6: Set User Identity
• 7: Set Context Variable
• 8: Transform
• 8.1: GNU Privacy Guard (GPG)
• 8.2: Protegrity Data Protection method
• 8.3: Regular expression replace
• 8.4: Security Assertion Markup Language (SAML) codec
• 8.5: User defined transformation
• 9: Dynamic Injection

1 - Error

In HTTP protocol, you can send custom response messages for requests with invalid content, while for other protocols, such as SFTP or SMTP the connection is terminated.

The fields for the Error action are as seen in the following figure.

The following table describes the fields applicable to the Error action.

The Error action type must always be created as a leaf node - a rule without any child nodes.

2 - Exit

The Exit option acts as a terminating action and the rules are not processed further. The exit option must always be created as a leaf node, a rule without a child node.

The fields for the Exit action are as seen in the following figure.

The following table describes the fields for the Exit Action.

* The following are the available options for the Scope field.

• Branch: Stop processing the child rules.
• Profile: Stop processing the rules under the same profile.
• RuleSet: Stop processing all the rules.

3 - Extract

The Extract action defines the payloads supported by the DSG.

The following payloads are supported in DSG.

• Adobe Action Message Format (AMF)

• Binary

• Character Separated Values (CSV)

• Common Event Format (CEF)

• eXtensible Markup Language (XML)

• Extensible Markup Language (XML) with Tree-of-Tress (ToT)

• Fixed Width

• HTML Form Media Type (X-WWW-FORM-URLENCODED)

• HTTP Message

• JavaScript Object Notation (JSON)

• JavaScript Object Notation (JSON) with Tree-of-Tress (ToT)

• Multipart Mime

• Microsoft Office 2007 Excel Document

• Microsoft Office 2013 Document

• Adobe Portable Document Format (PDF)

• Enhanced Adobe Portable Document Format (PDF)

• Protocol Buffer (protobuf)

• Secured File Transfer

• Amazon S3 Object

• SMTP Message

• Text

• Uniform Resource Locator

• User Defined Extraction

  Note: JWT token encoded in base64 and without padding characters can only be extracted using the UDF extraction rule.

• ZIP Compressed File

3.1 - Adobe Action Message Format

This payload extracts AMF format from the request and lets you define regex to control precise extraction.

The fields for Adobe Action Message Format (AMF) payload are as seen in the following figure.

The properties for the AMF payload are explained in the following table.

* The following options are available for the Method field.

• Serialize: Configure the AMF payload only to be exposed in learn mode. This will be useful for debugging while creating rules for learn mode.
• Serialized String Value: Configure the AMF payload as string and extract the matched Pattern.
• String Value: Configure the data using the matched Pattern. The data is not serialized ahead of the pattern matching.
• String Value by Key Name: The data is expected to come in key-value pairs. The parameters are matched using the Pattern. The value for the matched parameter is extracted.

3.2 - Amazon S3 Object

This payload extracts Amazon S3 object from the request and lets you define regex to control precise extraction. It is generally used with the Amazon S3 service.

The following figure illustrates the Amazon S3 Object payload fields.

The properties for the Amazon S3 Object payload are explained in the following table.

3.3 - Binary Payload

This payload extracts binary data from the request and lets you define regex to control precise extraction.

The fields for Binary payload are as seen in the following figure.

The properties for the Binary payload are explained in the following table.

The following table describes the fields for Encoding.

The following options are available for the Encoding field:

• No Encoding
• Standard
• External
• Proprietary

No Encoding

If the No Encoding option is selected, then no encoding is applied.

Standard

The Standard Encoding consists of built-in codecs of standard character encodings or mapping tables, including UTF-8, UTF-16, ASCII and more.

For more information about the complete list of encoding methods. refer to the section Standard Encoding Method List.

External

When external encoding is applied, you must select a codec.

The following table describes the codecs for the External encoding.

Proprietary

When proprietary encoding is selected, codecs linked are displayed.

The following table describes the codecs for the Proprietary encoding.

The following encryption methods are not supported for the High ASCII codec and the Base128 Unicode CJK codec:

• AES-128
• AES-256
• 3DES
• CUSP AES-128
• CUSP AES-256
• CUSP 3DES
• FPE NIST 800-38G Unicode (Basic Latin and Latin-1 Supplement Alpha)
• FPE NIST 800-38G Unicode (Basic Latin and Latin-1 Supplement Alpha-Numeric)

The following tokenization data types are not supported for the High ASCII codec and the Base128 Unicode CJK codec:

• Binary
• Printable

The input data for the Base128 Unicode CJK and High ASCII codecs must contain only ASCII characters. For example, if input data consisting of non-english characters is tokenized using the Alpha tokenization, then the Alpha tokenization treats the non-english characters as a delimiter and the tokenized output will include the non-english characters. As a result, the protection or unprotection operation will fail.

3.4 - CSV Payload

This payload extracts CSV format from the request and lets you define regex to control precise extraction.

With the Row and Column Index, you can now define how the column positions can be calculated. For example, consider a CSV input as provided in the following snippet, where the first column begins at the 0th field in the row, that are padded with commas until the next column, ex5 begins. This applies when the indexing is 0-based. If you choose to use 1-based indexing, the first column begins at 1 and subsequent fields are 2, 3, and so on. Based on these definitions, you can define the rule and its properties.

first, ex5, last, pick, ex6, city, ex1, ex2
John, wwww, Smith, mister, wwww, stamford, 333, 444
Adam, wwww, Martin, mister, wwww, fairfield, 333, 444

It is recommended to use the External Base64 encoding method in the Transform action type for the CSV codec. If the Standard Base64 method is used, then additional newline feeds are generated in the output.

The CSV implementation in the DSG does not support the following:

• The fields contain line breaks.
• If double quotes are used to enclose fields, then the double quotes appearing inside a field are escaped by preceding them with another double quote.

The fields for the CSV payload are as seen in the following figure.

If the CSV input includes NON-ASCII or Unicode data, then the Binary extract rule must be used before using the CSV extract rule.

If the CSV input file includes non-printable special characters, then to transform the data successfully, the user must add the csv-bytes-parsing parameter in the features.json file.

To add the parameter in the features.json file, perform the following steps.

1. Login to the ESA Web UI.
2. Navigate to Settings > System > Files.
3. Open the features.json file for editing.
4. Add the csv-bytes-parsing parameter in the features.json file. The csv-bytes-parsing parameter must be added in the following format:
   { "features": [ "csv-bytes-parsing" ] }

The properties for the CSV payload are explained in the following table.

3.5 - Common Event Format (CEF)

If you want to protect fields that are part of a CEF log file, you can use the CEF payload to extract the required fields.

The properties for the Common Event Format (CEF) payload are explained in the following table.

3.6 - XML Payload

This payload extracts the XML format content from the request and lets you extract the exact XML element value with it.

The fields for the XML payload are as seen in the following figure.

The properties for the XML payload are explained in the following table.

* The Advance XML Parser options field provides the following parsing parameters that can be configured.

Consider the following example to understand the Advance XML Parser options available in the XML codec. In this example, a request is sent from a client to remove the whitespaces between the XML tags from a sample XML payload in the message body of the HTTP/REST request. The following Ruleset is created for this example.

1. Create an extract rule for the HTTP message payload using the default RuleSet template defined under the REST API service.

   Consider the following sample XML payload in the HTTP message body.

   <?xml version = "1.0" encoding = "ASCII" ?>
   <class_list>
       <!--Students grades are uploaded by months-->
      <student>
            <name>John Doe</name>
         <grade>A</grade>
      </student>
   </class_list>
   

   In the example, a lot of white spaces are used for indentation. The payload contain spaces, carriage returns, and line feeds between the <class_list>, <student>, and <name> XML tags.

2. The extract rule for extracting the HTTP message body is as seen in the following figure.

   

3. Under the Extract rule, create another child rule to extract the XML payload from the HTTP Message.

   In this child rule, provide /class_list/student/name to parse the XML payload in the XPath List field and set the remove_blank_text parameter to true in the Advance XML Parser options field in the JSON format.

   

4. Under this extract rule, create another child rule to extract the sensitive data between the <name> and <name> tags. The fields for this child extract rule are as seen in the following figure.

   

5. Under the extract rule, create a transform rule to protect the sensitive data between the <name> and the <name> tags using Regex Replace with a pattern xxxxx. The fields for the transform rule are as seen in the following figure.

   

6. Click Deploy or Deploy to Node Groups to apply the configuration changes.

When a request is sent to the configured URI, the DSG processes the request and the following response appears with the whitespaces removed from the XML payload. In addition, the sensitive data between the <name> and the </name> tags is protected.

<?xml version='1.0' encoding='ASCII'?>
<class_list><!--Students grades are uploaded by months--><student><name>xxxxx xxxxx</name><grade>A</grade></student></class_list>

3.7 - Date Time Format

The Datetime format payload is used to convert custom datetime formats, which are not supported by tokenization datetime or date data element, to a supported format that can be processed by DSG.

Consider an example, where you provide a time format, such as DD/MM/YYYY HH:MM:SS as an input to an Extract rule with the Datetime payload. The given format is not supported by the datetime or date tokenization data element. The Extract rule converts the format to an acceptable format, a transform rule protects the datetime. The Datetime payload converts the protected value to the input format and returns this value to the user.

When you request DSG to unprotect the protected datetime value, an extract rule identifies the protected datetime value, a subsequent transform rule unprotects the value and returns the original datetime format, which is DD/MM/YYYY HH:MM:SS.

Ensure that the input sent to the extract rule for Date Time extraction is exactly in the same input format as configured in the rule. If you are unsure of the input that might be sent to the extract rule, then ensure that before you rollout for production, Ruleset configuration is thoroughly checked.

The following figure illustrates the Date Time format payload fields.

Before you begin:

Ensure that the following pre-requisites are completed:

• The datetime data element defined in the policy on ESA is used to perform protect or unprotect operation.

The following table describes the fields for Datetime codec.

*These fields appear only when Unprotect is selected as Mode of Operation.

3.8 - XML with Tree-of-Trees (ToT)

The XML with Tree-of-Trees (ToT) codec extracts the XML element defined in the XPath field. The XML with ToT codec allows you to process the multiple XML elements in an extract rule.

The fields for the XML with ToT payload is as seen in the following figure.

To understand the XML with ToT payload, consider the following example where the student details, such as, name, age, subject, and gender can be sent as a part of the request. In this example, the XML with ToT rule extracts and protects the name and the age element.

<?xml version='1.0' encoding='UTF-8'?>
<students>
 <student>
   <name>Rick Grimes</name>
   <age>35</age>
   <subject>Maths</subject>
   <gender>Male</gender>
 </student>
 <student>
   <name>Daryl Dixon </name>
   <age>33</age>
   <subject>Science</subject>
   <gender>Male</gender>
 </student>
 <student>
   <name>Maggie</name>
   <age>36</age>
   <subject>Arts</subject>
   <gender>Female</gender>
 </student>
</students>

The following figure illustrates one extraction rule for multiple XML elements.

In the Figure 14-25, the XML ToT Extract rule extracts the two different XML elements, name and age. The /students/student/name path extracts the name element and protect it with the transform rule. Similarly, the /students/student/age path extracts the age element and protect it with the transform rule. Same data elements are used to protect both the XML elements. You can use different data elements to transform XML elements as per your requirements. It is recommended to use the Profile reference, from the drop-down that appears on the Profile Name field. This helps to process the extraction and transformation in one rule. In addition, it reduces the transform overhead of defining one element at a time for the same XML file. If the Profile Name field is left empty then the extracted value is passed to the child rule for transformation.

For more information about profile referencing, refer to the section Profile Reference.

The properties for the XML with ToT payload are explained in the following table.

3.9 - Fixed Width

In scenarios where the input data is sent to DSG in a fixed width format, the Fixed Width codec is used. In a fixed width input, the data columns are specified in terms of exact column start character offset and fixed column width in terms of number of characters that define column width.

For example, consider a fixed width input as provided in the following snippet. The Name column begins at the 0th character in a row, has a fixed width of 20 characters and is padded with spaces until the next column Number begins. The Number column begins at the 20th character in a row and has a fixed width of 12 characters.

With the Row and Column Index, you can now define how the column positions can be calculated. If you choose to use 1-based indexing, the Name column begins at 1 and for fixed width of 20 characters, subsequent column will begin at the 21st character. While, if you use 0-based indexing, the Name column begins at 0 and for fixed width of 20 characters, subsequent column will begin at the 20th character. Based on these definitions, you can define the rule and its properties.

Name                         Number
John Smith                   418-Y11-4111
Mary Hartford               319-Z19-4341
Evan Nolan                   465-R45-4567

The fields for the Fixed Width payload are as seen in the following figure.

Note: If the input file includes non-printable special characters, then to transform the data successfully, the user must add the fw-bytes-parsing parameter in the features.json file.
To add the parameter in the features.json file, perform the following steps.

1. Login to the ESA Web UI.
2. Navigate to Settings > System > Files.
3. Open the features.json file for editing.
4. Add the fw-bytes-parsing parameter in the features.json file. The fw-bytes-parsing parameter must be added in the following format:
   { "features": [ "fw-bytes-parsing" ] }

The properties for the Fixed Width payload are explained in the following table.

3.10 - HTML Form Media Payload

This payload extracts HTML form media format from the request and lets you define regex to control precise extraction.

The fields for the HTML Form Media Payload (X-WWW-FORM-URLENCODED) payload are as seen in the following figure.

The properties for the X-WWW-FORM-URLENCODED payload are explained in the following table.

3.11 - HTTP Message Payload

This payload extracts HTTP message format from the request and lets you define regex to control precise extraction.

The following figure illustrates the HTTP Message payload fields.

The properties for the HTTP Message payload are explained in the following table.

3.12 - Enhanced Adobe PDF Codec

The Enhanced Adobe PDF codec extracts the PDF payload from the request and lets you define Regex to control precise extraction. This payload is available when the Action type is selected as Extract.

As part of the ruleset construction for this codec, it is mandatory to include a child Text extract rule under the Enhanced Adobe PDF codec extract rule. You must not use any other rule apart from the child Text extract rule under the Enhanced Adobe PDF codec extract rule.

In the DSG, some font files are already added to the /opt/protegrity/alliance/config/pdf_fonts directory. By default, the following font file is set in the gateway.json file.

"pdf_codec_default_font":{
"name": "OpenSans-Regular.ttf"
 }

Note: The Advanced Settings can be used to configure the default font file for a specific rule.

If you want to process a PDF file that contains custom fonts, then upload it to the /opt/protegrity/alliance/config/pdf_fonts directory. If the custom fonts are not uploaded to the mentioned directory, then the OpenSans-Regular.ttf font file will be used to process the PDF file.

For more information about how-to examples to detokenize a PDF, refer to the section Using Amazon S3 to Detokenize a PDF and Using HTTP Tunnel to Detokenize a PDF in the Protegrity Data Security Gateway How-to Guide.

The following figure displays the Enhanced Adobe PDF payload fields.

The properties for the Enhanced Adobe PDF payload are explained in the following table.

Note: The configurations in the Advanced Settings are only applicable for that specific rule.

Known Limitations

The following list describes the known limitations for this release.

1. The Enhanced Adobe PDF codec does not support detokenization for sensitive data that splits into multiple lines. It is expected that the prefix, data to be detokenized, and the suffix are in a single line and do not break into multiple lines.
2. The embedded fonts are not supported. Ensure that when you are uploading the fonts, the entire character set for that font family is uploaded to the DSG.
3. The prefix and suffix used to identify the data to be detokenized must be unique and not a part of the data.
4. The PDFs created with the rotate operator are not supported for detokenization.
5. The Enhanced Adobe PDF codec does not process password protected PDFs.
6. The detokenized data appears spaced out with extra white spaces.

3.13 - JSON Payload

This codec extracts the JSON element from the JSON request as per the JSONPATH defined.

Consider the following sample input that will be processed using the JSON codec to extract a unique JSON element:

{  
    "entities":[  
        {  
            "entity_type":"CostCenter",
            "properties":{  
                "Id":"10097",
                "LastUpdateTime":1455383881190,
                "Currency":"USD",
                "ApproveThresholdAmount":100000,
                "AccountingCode":"5555",
                "CostCenterAttachments":"{\"complexTypeProperties\":[]}"
            }
        }
    ],
    "operation":"UPDATE"
}

In the Extract rule, assuming that the AccountingCode needs to be protected, the JSONPath that will be set is entities[*].properties.AccountingCode. Based on the input JSON structure, the JSONPath value differs.

The following figure illustrates the JSON payload fields.

The properties for the JSON payload are explained in the following tab

The following table describes the additional configuration option for the Recurse mode.

JSONPath Examples

This section provides guidance on the type of JSONPath expressions that DSG understands. This guidance must be considered before you define the acceptable JSONPath to be extracted when using the JSON codec.

The DSG supports the following operators.

CAUTION: The $ operator is not supported.

To understand the JSONPath, consider the following example JSON. The subsequent table provides JSONPath examples that can be used with the example JSON.

{
  "store": {
    "book": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      {
        "category": "fiction",
        "author": "J. R. R. Rowling",
        "title": "Harry Potter and Chamber of Secrets",
        "isbn": "0-395-12345-8",
        "price": 29.99
      },
      {
        "category": "fiction",
        "author": "J. R. R. Tolkien ",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99
      },
      {
        "category": "fiction",
        "author": "Arthur Conan Doyle ",
        "title": "Sherlock Homes",
        "isbn": "0-795-19395-8",
        "price": 9
      }
    ]
  }
}

The following table provides the JSONPath examples based on the JSON example.

3.14 - JSON with Tree-of-Trees (ToT)

This section provides an overview of the JSON with Tree-of-Trees (ToT) payload. The JSON ToT payload allows you to use the advantages offered by Tree of Trees to extract the JSON payload from the request and provide protection according to the data element defined. Profile Reference can also be used to process different elements of the JSON.

The following figure illustrates the JSON ToT fields.

The properties for the JSON ToT payload are explained in the following table:

The following table describes the additional configuration option for the Recurse mode.

3.15 - Microsoft Office Documents

This payload extracts Microsoft Office documents from the request and lets you define regex to control precise extraction.

The following figure illustrates the MS Office payload fields.

The properties for the Microsoft Office documents payload are explained in the following table.

3.16 - Multipart Mime Payload

This payload extracts mime payload from the request and lets you define regex to control precise extraction.

The following figure illustrates the Multipart Mime payload.

The properties for the Multipart Mime payload are explained in the following table.

3.17 - PDF Payload

This payload extracts PDF payload from the request and lets you define regex to control precise extraction.

The following figure illustrates the PDF payload fields.

The properties for the PDF payload are explained in the following table.

Note: The DSG PDF codec supports only text formats in PDFs.
For any assistance in supporting additional text formats, contact Protegrity Professional Services.

3.18 - Protocol Buffer Payload

The PBpath defines a way to address fields in binary encoded protocolbuf messages. It uses field ids to construct an address messages or fields in a nested message hierarchy.

An example for the PBpath field is shown as follows:

 1.101.2.201.301.2.401.701.2.802

In DSG, protocol buffer version 2 is used.

The following figure illustrates the Protocol Buffer (protobuf) payload fields.

The properties for the Protocol Buffer payload are explained in the following table.

3.19 - Secure File Transfer Payload

This payload extracts SFTP message from the request and lets you further processing to be done on the files.

The following figure illustrates the Secure File Transfer payload fields.

The properties for the Secured File Transfer payload are explained in the following table.

3.20 - Shared File

This payload extracts file from the request and lets you define regex to control precise extraction. It is generally used with Mounted services, namely NFS and CIFS.

The following figure illustrates the NFS/CIFS share-related Shared File payload fields.

The properties for the Shared File payload are explained in the following table.

3.21 - SMTP Message Payload

This payload extracts SMTP payload from the request and lets you define regex to control precise extraction.

The following figure illustrates the SMTP message payload fields.

The properties for the SMTP payload are explained in the following table.

3.22 - Text Payload

This payload extracts text payload from the request and lets you define regex to control precise extraction.

The following figure illustrates the Text payload fields.

The properties for the Text payload are explained in the following table.

3.23 - URL Payload

This payload extracts URL payload from the request and extract precise object based on selection.

The following figure illustrates the URL payload fields.

The properties for the URL payload are explained in the following table.

3.24 - User Defined Extraction Payload

This codec lets you define custom extraction logic and pass arguments to the next rule. The language that is currently supported for extraction is Python.

From DSG 3.0.0.0, the Python version is upgraded to python 3. The UDFs written in Python v2.7 will not be compatible with Python v3.10. To migrate the UDFs from python 2 to python 3, refer to the section Migrating the UDFs to Python 3.

The following figure illustrates the User Defined Extraction payload fields.

The properties for the User Defined Extraction payload are explained in the following table.

Note: The DSG supports the usage of the PyJwt python library in custom UDF creations. PyJWT is a python library that is used to implement Open Authentication (OAuth) using JSON Web Tokens (JWT). JSON Web Tokens (JWT) is an open standard that defines how to transmit information between a sender and a receiver as a JSON object. To authenticate JWT for OAuth, you must write a custom UDF. The PyJwt library version supported by the DSG is 1.7.1.
For more information about writing a custom UDF on the DSG, refer to the section User Defined Functions (UDF).

Note: The DSG supports the usage of the Kafka python library in custom UDF creations. Kafka is a python library that is used for storing, processing, and forwarding for applications in a distributed environment. For example, the DSG uses the Kafka library to forward Transaction Metrics logs to external applications. The Kafka library version supported by the DSG is 2.0.2.
For more information about writing a custom UDF on the DSG, refer to the section User Defined Functions (UDF).

Note: The DSG supports the usage of the Openpyxl Python library in custom UDF creations. Openpyxl is a Python library that is used to parse Excel xlsx, xlsm, xltx, xltm files. This library enables column-based transformation for Microsoft Office Excel. The Openpyxl library version supported by the DSG is 2.6.4.

Note: The DSG uses the in-built tarfile python module for custom UDF creation. This module is used in the DSG to parse .tar and .tgz packages. Using the tarfile module, you can extract and decompress .tar and .tgz packages.

3.25 - ZIP Compressed File Payload

This payload extracts ZIP file from the request and lets you extract file name or file content.

The following figure illustrates the ZIP Compressed File payload fields.

The properties for the ZIP payload are explained in the following table.

4 - Log

Create a rule in the child node of the extract rule defined for the payload. The payload can be defined for services, such as, HTTP, SFTP, REST, and so on. When the extract rule is invoked for the payload, the log rule will display the contents of the payload in the gateway log.

The fields for the Logs action type are as seen in the following figure.

The properties for the Log action are explained in the following table.

* -In the Log action, depending on the severity of the message, you can define different types of log levels. The following table shows the DSG log level setting and the corresponding log entry that is seen in Forensics.

** The following options are available for the Destination field:

• Forensics: Log is sent to the Forensics and the DSG internal log.
• Internal: Log is sent only to the DSG internal log.

Consider the following example to understand the functionality of the Log action type. In this example, the Log action type functionality is used to find the transactionID from a sample text message in the HTTP payload. The following RuleSet structure is created to find the transactionID:

1. Create an extract rule for the HTTP payload using the default RuleSet template defined under the REST API service. The HTTP payload consists of the following sample text message:

   {"source":{"merchantId":"LCPLC-lphf21","storeId":"LPHF2100303086",
   "terminalId":"0","operatorId":"10","terminalType":"GASPRO",
   "offline":false},
   "message":{"transactionType":"WALLET","messageType":"VERIFYPARTNER",
   "transactionTime":"2018-05-09T11:48:47+0000",
   "transactionId":"20180509114847LPHF2100301237000090"},
   "identity":{"key":"4169671111","entryType":"KEYEDOPERATOR"}}
   

2. The fields for the extract rule for the HTTP message payload are as seen in the following figure:

   

3. Create a child extract rule to extract the text message in the payload.

   1. Create a RegEx expression, “transactionID”:(.*)", to match the pattern for the transactionID. The fields for the child extract rule are as seen in the following figure:

      

4. Create another child rule for the Log action type. The fields for the Log action type are as seen in the following figure:

   

5. Add the following values in the Log action type.

   1. Action: Log
   2. Level: Warning
   3. Destination: Internal
   4. Title Format: %(name)s
   5. Message Format: %(value)s

6. After the log rule is processed, the following message will be displayed in the DSG internal log:

   

5 - Profile Reference

While creating Rulesets for Dynamic CoP, use the Profile Reference rule is used for data transformation instead of the Transform rule. The security benefits of using Profile Reference rule are higher than the Transform rule as the requests are triggered on the fly.

The fields for the Profile Reference action are as seen in the following figure.

The following table describes the fields for the Profile Reference action.

Note: The Profile Reference action type must always be created as a leaf node (a rule without any child nodes).

6 - Set User Identity

The fields for the Set User Identity action are as seen in the following figure.

Note: If the Set User Identity rule is followed by a Transform rule, then any data processing logs generated in Forensics are logged with the user set using the Set User Identity rule.

In addition, the user set using the Set User Identity rule overrides the user set in the Transform rules for auditing any data processing logs in Forensics.

Note: If the “Set User Identity” rule is configured along with Basic Authentication, then the “user_name” field in the transaction metrics will be set with the username configured in the “Set User Identity” rule.

If Basic Authentication is configured without the “Set User Identity” rule, then the “user_name” field in the transaction metrics displays the username configured in the Basic Authentication.

7 - Set Context Variable

You can The value set due to this rule will be maintained throughout the rule lifecycle.

The following table describes the Variable Name type supported by the Set Context Variable option.

The Set Context Variable action type must always be created as a leaf node - a rule without any child nodes.

User IP address

Record the IP of the client that sents a request to a DSG node in the audit log. When a client request is sent to the load balancer that distributes incoming requests to the cluster of DSG nodes, the load balancer appends a header to the request. This header captures the client IP address.

The types of headers can be X-Forwarded-For, which is most commonly used, or X-Client-IP, User-Agent, and so on.

Before a Set Context Variable with the User IP Addr Variable Name type rule is created, an extract rule that extracts the Header with the given header name, such as X-Forwarded-For, from a request would be created.

If a request header sends an IP address 192.168.0.0 as the X-Forwarded-For value, the following image shows the client IP in the Forensics log displaying this IP address value.

The fields for the Variable Name type are as seen in the following figure.

The following table describes the fields.

Value External IV protect

You can send an external IV value that will used along with the protect or unprotect algorithm in the request header to create more secure encrypted data. External IV values add additional layer of randomness and help in creating secure tokens.

Note: This value overrides the value set in the Default External IV field in the Transform rule.

The fields for the Variable Name type are as seen in the following figure.

The following table describes the fields.

Note: If an External IV value is sent in the header to protect or unprotect sensitive data, with the case-preserving and position-preserving property enabled in the Alpha-Numeric (0-9, a-z, A-Z) token type, then the External IV property is not supported.

Value-External IV Reprotect

You can send an external IV value that will used along with the reprotect algorithm in the request header to create more secure encrypted data. External IV values add additional layer of randomness and help in creating secure tokens.

Note: This value overrides the value set in the Default External IV field in the Transform rule.

The fields for the Variable Name type are as seen in the following figure.

The following table describes the fields.

Note: If an External IV value is sent in the header to protect or unprotect sensitive data, with the case-preserving and position-preserving property enabled in the Alpha-Numeric (0-9, a-z, A-Z) token type, then the External IV property is not supported.

Dynamic Rule

The Dynamic Rule provides a hook in form of the URI in the preceding rule and a logical endpoint for the Dynamic CoP header request to join the rule tree.

After you define the Dynamic Rule variable name type, you can proceed with creating the Dynamic Injection action type.

The fields for the Variable Name type are as seen in the following figure.

The following table describes the fields.

Client Correlation Handle

The client correlation handle captures the Linux epoch time when the protect or unprotect operation is successful.

When you define rulesets, the rules are structured such that the extract rule identifies the protect successful event from the input message. This rule is followed by the extraction of the timestamp using a UDF rule.

The set context variable rule follows next to set the variable to the extracted timestamp. You can further create a rule that converts this timestamp to a hex value followed by a Log rule to display the exact time of protect and unprotect operation in the ESA Forensics or DSG logs.

The fields for the Variable Name type are as seen in the following figure.

The following table describes the fields.

"Value configured by user is ignored as it exceeds 
20 Characters (Maximum Limit)"

If this parameter is not configured, the context variable value is truncated to 20 characters by default.

If this check box is not selected and the context variable value passed to the Set Context Variable rule exceeds the maximumInputLength parameter value set in the Rule Advanced Settings, then the transaction fails with an error.| | |Rule Advanced Settings|Set the parameter maximumInputLength such that data beyond this length is not set as the context variable. The datatype for this option is number of characters.|20|

User Defined Headers

User Defined Headers are meant to provide additional information about an HTTP Response Header that can be helpful for troubleshooting purposes. The User Defined Headers can include information such as custom cookies, state information, and provide information to the load balancer, for example, CPU utilization of a particular node behind the load balancer.

The fields for the Variable Name type are as seen in the following figure.

The following table describes the fields.

8 - Transform

For any Transform action, if you click the ( ) icon to add a new rule, a message TRANSFORM rule cannot have a child rule under it appears.

The fields for the Transform action are as seen in the following figure.

The following table describes the methods applicable to the Transform action.

The Transform action type must always be created as a leaf node (a rule without any child nodes).

If an External IV value is configured in the Transform rule, with the case-preserving and position-preserving property enabled in Alpha-Numeric (0-9, a-z, A-Z) token type, then the External IV property cannot be used to transform sensitive data.

8.1 - GNU Privacy Guard (GPG)

The GPG encrypted data is first optionally compressed, encrypted with a one-time generated session key, and this session key is then encrypted with the public key. The extracted data from execution of RuleSet can be transformed using the GPG method in the Transform action.

From the DSG Web UI, in the Operation field, you can either select Encrypt or Decrypt operation. The options for each operation vary based on the selection. The DSG appliance is compatible with GPG v2.2. Refer to the GPG documentation at https://www.gnupg.org/faq/gnupg-faq.html

Importing keys

Run the following steps to import public and private keys generated outside DSG.

To import keys:

1. Upload the public key from the ESA Web UI. Navigate to Cloud Gateway > 3.3.0.0 {build number} > Transport > Certificate/Key Material.

   The Certificate/Key Material screen appears.

   1. On the Certificate/Key Material screen, click Upload.

   2. Click Choose File and select the public key to be uploaded.

2. Upload the private key to ESA using an FTP tool.

3. On the DSG CLI Manager, navigate to the /opt/protegrity/alliance/3.3.0.0.<build number>-1/config/resources/ directory. Verify that the private key and public key are available in this directory.

4. Run the following command. docker ps A list of all the available docker images is displayed. For example

   CONTAINER ID   IMAGE   COMMAND    CREATED        STATUS          PORTS     NAMES
   28791aa86a02   gpg-agent:3.3.0.0.51    "gpg-agent --server …"   15 hours ago   Up 25 minutes             gpg-agent-3.3.0.0.51-1
   

5. Under the NAMES column, note the name of the image corresponding to the gpg-agent:3.3.0.0<build number>.

6. Run the following command to import the public key.

   docker exec -it <Name of the GPG container> gpg --homedir /opt/protegrity/alliance/config/resources --import /opt/protegrity/alliance/config/resources/<public_key_file_name>
   

   For example,

   docker exec -it gpg-agent-3.3.0.0.51-1 gpg --homedir /opt/protegrity/alliance/config/resources --import /opt/protegrity/alliance/config/resources/test.gpg
   

7. Import the private key by running the following command:

   docker exec -it <Name of the GPG container> gpg --homedir /opt/protegrity/alliance/config/resources --allow-secret-key-import --pinentry-mode loopback --import /opt/protegrity/alliance/config/resources/<private_key_file_name>
   

   For example,

   docker exec -it gpg-agent-3.3.0.0.51-1 gpg --homedir /opt/protegrity/alliance/config/resources --allow-secret-key-import --pinentry-mode loopback --import /opt/protegrity/alliance/config/resources/secret.gpg
   

8. Trust the imported keys as ultimate keys by running the following command:

   docker exec -it <Name of the GPG container> gpg --homedir /opt/protegrity/alliance/config/resources --edit-key <Name>
   

For example,

```
docker exec -it gpg-agent-3.3.0.0.51-1 gpg --homedir /opt/protegrity/alliance/config/resources --edit-key test.user@sample.com
gpg> trust
#enter 5<RETURN>
#enter y<RETURN>
gpg> quit
```

Generating GPG keys

Steps to generate the GPG keys on ESA.

1. Login to ESA CLI Manager.

2. Run the following command to generate the key.

   docker exec -it <Name of GPG container> --homedir /opt/protegrity/alliance/config/resources/ --pinentry-mode=loopback --full-generate-key
   

   For example,

   docker exec -it gpg-agent-3.3.0.0.51-1 gpg --homedir /opt/protegrity/alliance/config/resources/ --pinentry-mode=loopback --full-generate-key
   

3. Select the type of key that you want to generate from the following options.

   • (1) RSA and RSA (default)
   • (2) DSA and Elgamal
   • (3) DSA (sign only)
   • (4) RSA (sign only)

4. Enter the keysize for the key. The keysize can range between 1024 to 4096.

5. Select the validity of the key from the following options.

   • 0 = key does not expire
   • <n> = key expires in n days
   • <n>w = key expires in n weeks
   • <n>m = key expires in n months
   • <n>y = key expires in n years

6. Enter the real name that identifies the key.

7. Enter the email address for the key.

8. Enter a comment for the key. The public key in GPG includes a key and user ID information that identifies the key with the user ID.

9. Select (O) to confirm the user ID details.

10. Press Enter or provide a passphrase. The passphrase is used during decryption.

11. Run the following command to verify the key is generated.

    docker exec -it <Name of the container> gpg --homedir /opt/protegrity/alliance/config/resources/ --list-keys
    

    For example,

    docker exec -it gpg-agent-3.3.0.0.51-1 gpg --homedir /opt/protegrity/alliance/config/resources/ --list-keys
    

The gpg directory must include the following files after you generate a GPG key successfully:

• pubring.gpg
• secring.gpg
• trustdb.gpg
• random_seed
• s.gpg-agent
• s.gpg-agent.ssh
• s.gpg-agent.extra
• s.gpg-agent.browser
• private-keys-v1.d
• openpgp-revocs.d

Encrypt operation

The encrypt operation transform rule related options for GPG rule implementation are listed in this section.

The following table describes the fields for Encrypt operation in the GNU Privacy Guard method.

Decrypt operation

The decrypt operation transform rule-related options for the GNU Privacy Guard (GPG) rule implementation are listed in this section.

The following table describes the fields for the Decrypt operation in the GPG method.

8.2 - Protegrity Data Protection method

The following table describes the fields for Protegrity Data Protection method.

Field	Sub-Field	Description	Notes
Protection Method		Specify the action performed (protection, unprotection, or re-protection).
Data Element Name		Specify the Data element used (for protection, unprotection, or re-protection).	The Data Element Name drop down list populates data elements from the deployed policy.
Encryption Data Element		Select to process the encryption data element
Default External IV		Default value to be used as an external initialization vector.
Reprotect New Data Element Name		New data element name that will be used to reprotect data.	The Data Element Name drop down list populates data elements from the deployed policy.
Reprotect New Default External IV		New default value to be used as an external initialization vector.
Default Username		Policy Username used for the user.	Ensure that you do not use the OS user alliance as a policy user. If the user is not specified, by default the data security operation will be performed by the alliance user. Ensure that the username does not exceed 256 characters.
Encoding		Encoding method to be used.
Codec		Based on the encoding selected, select the codec to be used.For more information about codec types, refer to the section Codecs.
Prefix		Prefix text to be padded before the protected value. This helps in identifying protected text from clear text.
Suffix		Suffix text to be padded after the protected value. This helps in identifying protected text from clear text.
Padding Character		Characters to be added to raise the number of characters to the minimum required size by the Protection method.
Minimum Input length		Number of characters that define if input is too short for the Protection method to be padded with the Padding Character.	If a data element is created with the Length Preservation property and it allows short data, then the input data will be tokenized if the input data is short. If a data element is created with the Length Preservation property and it does not allow short data, then the input data will not be tokenized and an error message will be generated.
Advanced Settings
	Permissive Error Handling	Click to expand.The underlying Application Protector API used in the DSG might encounter input that is not ideal in nature, such as, input too short. In such cases, you can use this option to decide how such inputs should be handled gracefully, as opposed to failing the entire data body processing.
	Enabled	Select to enable permissive handling of error generated due to distorted input.
	Error strings	Regex pattern to identify the errors that need to be handled permissively. You can also provide the exact error message.For example, if the error message on the Log viewer screen is “The input is too short”, then you can enter the exact message “The input is too short” in this field. Other error message examples are “The input is too long”, “License is invalid”, “Permission denied”, “Policy not available”, and so on. Based on the error message that you encounter and want to handle differently, the value in this field should be adjusted accordingly. For example, a pattern, such as, too short, too long, Permission denied can be used to gracefully handle the respective three errors.
	Output Data	Regex substitution pattern that dictates how output values for erroneous input values are substituted. For example, if this value is set to “????”, then the distorted input will be replaced with this value, thus allowing the rule to process instead of failing due to distorted input. Users may choose such fixed substitution strings to spot individual erroneous input data values post processing of data. You can also add prefix and suffix to the input. The regex must follow the “<prefix>\g<0><suffix>” REGEX substitution pattern. For example, if you want the input to be identified with the “#*_” as the input prefix and “_#*” as the input suffix, the regex pattern with be “#*_\g(0)_#* “.

8.3 - Regular expression replace

The following table describes the fields for Regular Expression Replace method.

8.4 - Security Assertion Markup Language (SAML) codec

Any SAML implementation involves the following two entities, namely, the Single Sign-On (SSO) application or the Service Provider that the user is trying to access and the Identity Provider (IDP) responsible for authenticating the user.

A typical SAML implementation is illustrated in the following diagram.

Consider that the user wants to access the Service Provider (SP). The SP redirects the user’s authentication request to the Identity Provider (IDP) through the DSG.

When a user tries to access a Service Provider (SP), an authentication request is generated. The DSG receives and forwards the request to the IDP. The IDP authenticates the user and reverts with a SAML assertion response. The DSG updates the domain names in the SAML assertion response, verifies the inbound SAML signature, and re-signs the modified SAML response to ensure that the response is secure and not tampered. The response is then forwarded to the SP. The SP verifies the response and authenticates the user.

After this authentication is complete, when the same user tries to access any other SP on the same network, authentication is no longer required. The IDP identifies the user from the previous active session and reverts with a SAML response that user is already authenticated.

Note: SAML codec is tested with the following the SalesForce SP and the Microsoft Azure AD IDP.

Before you begin:

Ensure that the following prerequisites are met:

• The IDP’s public certificate, which was used to sign the message response, is uploaded to the ESA using the Certificate screen.
• The certificate and private key that the customer wants to use to re-sign the SAML response using the DSG must be uploaded to the ESA.
• The certificate that is used to re-sign the message must be uploaded to the SP for validating the response.
• As the user requests to access the SP are redirected to the IDP through the DSG, ensure that in the SP configurations, the IDP redirect URLs are configured.

For example, if the IDP is https:\\login.abc.com, then in the SP configurations ensure that the redirect URLs are set to https:\\secure.login.abc.com.

• As the SAML response from the IDP, that is used to authenticate the user, is redirected through the DSG to the SP, ensure that the IDP configurations are set as required.

  For example, if the SP is https:\\biloxi.com, then in the IDP configurations, ensure that the redirect URLs are set to https:\\secure.biloxi.com.

Note: After you upload the certificates to the ESA, ensure that you deploy configurations from the ESA such that the certificates and keys are pushed to all the DSG nodes in the cluster.

The SAML codec fields can be seen in the following image.

The following table describes the fields for the SAML codec.

8.5 - User defined transformation

The following figure illustrates the User Defined Transformation payload fields.

The following table describes the fields for User Defined Transformation method.

The DSG supports the usage of the PyJwt python library in custom UDF creations. PyJWT is a python library that is used to implement Open Authentication (OAuth) using JSON Web Tokens (JWT). JSON Web Tokens (JWT) is an open standard that defines how to transmit information between a sender and a receiver as a JSON object. To authenticate JWT for OAuth, you must write a custom UDF. The PyJwt library version supported by the DSG is 1.7.1.

The DSG supports the usage of the Kafka python library in custom UDF creations. Kafka is a python library that is used for storing, processing, and forwarding for applications in a distributed environment. For example, the DSG uses the Kafka library to forward Transaction Metrics logs to external applications. The Kafka library version supported by the DSG is 2.0.2.

The DSG supports the usage of the Openpyxl Python library in custom UDF creations. Openpyxl is a Python library that is used to parse Excel xlsx, xlsm, xltx, xltm files. This library enables column-based transformation for Microsoft Office Excel. The Openpyxl library version supported by the DSG is 2.6.4.

9 - Dynamic Injection

After defining a Dynamic Rule variable name type to store the request header that sends the Dynamic CoP structure, configure the Dynamic Injection action type to process Dynamic rules and protect data on-the-fly.

While creating Rulesets for Dynamic CoP, use the Profile Reference rule is for data transformation instead of the Transform rule. The security benefits of using Profile Reference rule are higher than the Transform rule as the requests can be triggered out of the secure network perimeter of an organization.

The fields for the Dynamic Injection action are as seen in the following figure.

The following table describes the Authorized Rule Types applicable to the Dynamic Injection action.

Note: The Dynamic Injection action type must always be created as a leaf node (a rule without any child nodes).

Creating a Dynamic Injection rule

The dynamic injection rule must be created to send dynamic Ruleset requests.

To create a Dynamic Injection Rule:

1. Create an extract rule under an existing profile or a new profile to extract the request header with the Dynamic Rule.

   

2. Under this extract rule, create a Set Context Variable rule to extract the Dynamic rule and set as a variable.

   

3. Under the same profile, create an extract rule to extract the request message.

   

4. Under the extract rule created in step 3, create the Dynamic Injection rule. The Dynamic rule received in the request header in step 2 will be hooked to this rule for further processing.

   

5. Click Deploy or Deploy to Node Groups to apply configuration changes.

   When a request is received at the configured URI, the request header and request body are processed, and then the response is sent.