DSG REST API

In addition to providing a conceptual overview, the discussion steps through a use case where DSG is configured with a RuleSet object behind a REST API URL end-point.

The RuleSet has been designed to provide fine-grained protection of an XML document where the sensitive data is identified through an XPath. The sections also contain code samples in Java, Python and Scala that show example invocation of the API from client applications written in these languages.

Overview

In addition to offering In-Band mechanism for data security, DSG offers a RESTful Web Service API. This mechanism of utilizing DSG’s capabilities is referred to as On-Demand data security.

Client applications invoke DSG’s REST API by sending it HTTP requests pointed at pre-configured URLs in the DSG. These URLs are internally backed by request processing behaviors which are specified through RuleSets configuration objects. In general, RuleSets allow a user to define hierarchical, step-by-step processing of data conveyed in any Layer-7 protocol messages. Example Rule nodes of a RuleSet tree might include extraction of HTTP POST request body, parsing of the body content according to a certain format (typically specified in HTTP Content-Type header), extraction of sensitive data within the message body and protection of extracted data.

While invoking a REST API call on DSG, a client conveys a document of certain format embedded in an HTTP request method (e.g. HTTP POST) to a pre-configured DSG REST URL. As an outcome of processing the request, DSG sends a 200 response to the client. The response message carries a modified version of the original document wherein select sensitive data pieces in plaintext are substituted with their cryptographic equivalent (either cipher text or tokens).

The following figure shows an example usage of DSG’s REST API. It illustrates a fine-grained data security scenario where DSG accepts an incoming request from a client where the request carries a document with certain sensitive data elements embedded in it. DSG parses the input document, extracts sensitive data pieces in it and protects those extracted data fragments as per preconfigured data security policies.

The protected data fragments are substituted in-place at the same location of their plaintext equivalents in the original decoded document. The decoded document is then encoded back to its original format and delivered back to the client as part of the API response.

While the illustration above shows data protection, one can certainly conclude that the same mechanism can be used for any other form of data transformation natively available in Protegrity core data security platform including data un-protection, re-protection, masking and hashing.

REST API Authentication

DSG supports basic authentication and mutual authentication.

Basic Authentication

DSG supports user authentication using the HTTP basic authentication mechanism.

The user credentials (username and password) are validated against ESA LDAP, which may be connected to any external directory source. Successful user authentication results are cached for a configurable period thus saving the authentication round trips for performance efficiency reasons. The identity of the authenticated users is automatically used as their ‘policy user’ identity for performing data security operations.

In this authentication type, the username and password are included in the authorization header.

For permissions and access control, there are three types of roles in DSG:

• Cloud Gateway Admin
• Cloud Gateway Viewer
• Cloud Gateway Auth

The following table describes each of these roles.

In DSG, you can allow users linked to the Cloud Gateway Authentication role to perform REST API authentication. For a REST API call, the client sends the username and password in the authorization header to the server. After the authentication is successful, all the rules that are part of the REST API call are processed. The authentication is performed at the parent level of every rule and the authentication is cached for every child rule.

The following steps explain the process for the REST API authentication:

1. The user makes a REST API call to the gateway.
2. The authorization parameters, username and password, are verified against LDAP.
3. On successful authentication, the username-password combination is cached for the transaction.
4. The gateway then sends a response to the REST API call sent by the user.
5. If the authentication fails, the server sends a 401 error response stating that the authorization parameters are invalid. The REST API call then terminates.

To enable REST API authentication from the ESA Web UI, ensure that the user is linked to the Cloud Gateway Auth role as shown in the following figure.

Enabling Rest API Authentication

You can enable REST authentication using the HTTP message payload.

To enable REST API Authentication:

1. In the ESA Web UI, navigate to Cloud Gateway > 3.3.0.0 {build number} > RuleSet.

2. Select the required rule.

3. In the Authentication drop down list, select Basic.

   

4. Save the changes.

   You can also request client certificates as a part of authorization to make a REST API call. After the certificate is verified successfully, all the rules that are a part of the REST API call are executed. If the certificate is invalid, a 401 error response is sent to the user.

Enabling a Client Certificate for a Rule

Apart from specifying client certificate at tunnel level, you can also specify client certificate at service level.

To enable Client Certificate for a Rule

1. In the ESA Web UI, navigate to **Cloud Gateway > 3.3.0.0 {build number}**Cloud Gateway > 3.3.0.0 {build number}> RuleSet

2. Select the required rule.

3. Check the Require Client Certificate checkbox.

   

4. Save the changes.

TLS Mutual Authentication

DSG can be configured with trusted root CAs and/or the individual client machine certificates for the machines that will be allowed to connect to DSG’s REST API TLS tunnel.

Client machines that fail to offer a valid client certificate will not be able to connect to DSG’s REST API TLS ports.

When communicating with DSG, you can add an additional layer of security by requesting the HTTP client to present a client certificate. You can do so by enabling mutual authentication in DSG. The certificate presented by the client must be trusted by DSG in order for the TLS connection to be successfully established.

Before rolling out to production, in the testing environment, you can use DSG to generate the client key and certificate. The client key and certificate are self-signed by DSG using a self-generated CA.

Enabling Mutual Authentication

DSG supports TLS mutual authentication and you can use steps in this section to enable it.

Ensure that the following prerequisites are completed in the given order.

1. Ensure that you generate custom CA certificates and keys, for example, a certificate file ca_test.crt, a key file ca_test.key, a pem file ca_test.pem and upload it from the Certificate/Key Material screen on the DSG.
2. Ensure that you generate custom server certificates and keys, for example, a certificate file server_test.crt, a key file server_test.key, a pem file server_test.pem, and sign it with the CA certificates generated in step 1. After creating the certificates and keys, upload it from the Certificate/Key Material screen on the DSG.
3. Ensure that you generate custom client certificates and keys, for example, a certificate file client_test.crt, a key file client_test.key, a pem file client_test.pem, and sign it with the CA certificates generated in step 1. You must use the client certificate while sending a request to the DSG.

To enable Mutual Authentication:

1. Configuring Tunnel to enable Mutual Authentication.

   1. On the ESA Web UI, navigate to the Tunnels screen and click the HTTP tunnel that you want to edit.

   2. In TLS Mutual Authentication, set the value as CERT_OPTIONAL.

   3. In CA Certificates, provide the absolute path of the ca_test.pem certificate.

2. Configuring rule to enable Mutual Authentication.

   1. In the extract rule, ensure that you select the Require Client Certificate check box.

Protecting an XML Document through DSG REST API

Let’s review a use case where a client requests protection of certain sensitive information within an XML document through DSG’s REST API.

While this use case describes fine-grained protection of an XML document, one can easily translate this into other structured data formats such as CSV, JSON etc. or even unstructured data formats.

As a precursor to understanding DSG RuleSet configuration, let’s first review the REST API input and the expected output. The following snippet shows sample REST API request and response messages. The client sends an HTTP POST request message that carries an XML document in it. The expectation is that the content of <Person><Name>…</Name></Person> XML hierarchy be protected at word boundaries. DSG responds back with an updated XML document wherein the specified sensitive data been substituted with tokens.

Request:

POST /protect/xml/xpath HTTP/1.1
Host: restapi
Content-Length: 85
Content-Type: application/xml
<Person>
  <Title>Mr.</Title>
  <Name>Joe Smith</Name>
  <Gender>Male</Gender>
</Person>

Response:

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Server: Protegrity Cloud Gateway 1.0.0.170

<Person>
  <Title>Mr.</Title>
  <Name>nM9M 4NFuRl9</Name>
  <Gender>Male</Gender>

To produce the API output shown above, DSG is configured with a RuleSet object ahead of time. The RuleSet object is tied to a REST API URL (/protect/xml/xpath in this example). As mentioned earlier, a RuleSet object is a collection of all the rules responsible for handing requests arriving on a specific URL. The handling of this request is decomposed in four cascading steps as shown in the following figure.

Step 1: Extract body of the HTTP request message. The extracted body content will be the entire XML document. The extracted output of this Rule will be fed to all its children sequentially. In this example, there is only one child of this Rule node.

Step 2: Parse the input as an XML document such that an XPath can be evaluated on it to find position offsets of the sensitive data content.

Note: One may choose to treat the XML document as a flat text file and run a REGEX on it instead and get the same results. The choice depends on studying the XML and where all sensitive data lies and which mechanism will yield more accurate, maintainable and high performant results.

Step 3: Split the extracted data from the previous rule into words. This will be done by running a simple REGEX on the input. Each word will be fed into children rule nodes of this rule one by one. In this use case there is only one child to this Rule.

Step 4: Protect input content. Since this Rule is a leaf node (a node without any children), return resulting ciphertext or token to the parent.

At the end of Step 4, the RuleSet traversal stack will unwind and each branch Rule node will reverse its previous action such that the overall data can be brought back to its original format. Going back in the reverse direction, Step 4 will return tokens to Step 3 which will concatenate them together into a string. Step 2 will substitute the string yielded back from Step 3 into the original XML document in place of the original plaintext string pointed at by the configured XPath. Step 1 that was originally responsible for extracting body of the HTTP request will now replace what has been extracted with the modified XML document. A layer of platform logic outside of RuleSet tree execution will create an HTTP response message which will convey the modified XML document to the client.

Let us translate the request handling design described above into real DSG configuration. The figures from Step 2 Rule Node Configuration to Postman Client Example depict Rule nodes creation for Step 1 through Step 4 in the ESA Web UI under the Cloud Gateway section.

Java Client using Native java.net.HttpURLConnection

This section describes REST API example with JAVA client.

import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

/*
 * DSG REST API Client Example:
 * 
 * This example simulates a DSG REST API client. The example uses Java native 
 * HttpURLConnection utility for sending an HTTP POST request to a configured
 * REST API end-point URL in DSG ("http://restapi/protect/xml/regex" or 
 * "http://restapi/protect/xml/xpath").
 * 
 * The example sends an XML document with sensitive content in plain-text and
 * expects to receive an HTTP response (200) where such content has been 
 * substituted by cipher-text or tokens. In order to support this example, the 
 * DSG has been configured with a corresponding REST API RuleSet that protects 
 * information contained in <Name>...</Name> XML tag.
 * 
 * The DSG REST API consumes the entire XML document, finds the sensitive 
 * information in it either based on a configured REGEX or XPath, and responds 
 * back with a modified XML document where the sensitive information has been
 * protected. No other part of the input document is modified. 
 *   
 * While this example demonstrates XML as the subject content, DSG supports a
 * variety of content types including all textual formats (XML, JSON, HTML, CSV,
 * JS, HTML etc.) as well as complex formats such as XLS(X), PPT(X), DOC(X),
 * TSV, MHT, ZIP and PDF. 
 */

public class RestApiSampleHttpURLConnection {
  public static void main(String[] args) {

    // DSG's REST API end point URL - tied to REGEX or XPath based RuleSet
String dsgRestApiUrl = "http://restapi/protect/xml/xpath";
    // String dsgRestApiUrl = "http://restapi/protect/xml/regex";

    HttpURLConnection conn = null;  
    try {
      // Create connection
      URL url = new URL(dsgRestApiUrl);
      conn = (HttpURLConnection)url.openConnection();

      // Set headers and body
      conn.setRequestMethod("POST");
      conn.addRequestProperty("Content-Type", "application/xml");
      conn.setDoOutput(true);

      // Send request
      DataOutputStream wr = new DataOutputStream (conn.getOutputStream());
      String inXml = "<Person>"
                  +     "<Title>Mr.</Title>"
                  +     "<Name>Joe Smith</Name>"
                  +     "<Gender>Male</Gender>"
                  +  "</Person>";
      wr.writeBytes(inXml);
      wr.close();
      System.out.println("REQUEST: \n" + "POST " + dsgRestApiUrl + "\n" + inXml);	

      // Receive response line by line  	
      BufferedReader in = new BufferedReader(
        new InputStreamReader(conn.getInputStream()));
      String inputLine;
      StringBuffer response = new StringBuffer();
      while ((inputLine = in.readLine()) != null) {
        response.append(inputLine);
      }
      in.close();
      System.out.println("RESPONSE: \n" + response.toString());	
    } catch (Exception e) {
      e.printStackTrace();
    } 
  }
}

Python Client

This section showcases REST API example with Python client.

#!/usr/bin/python
import http.client
reqBody =  "<Person>               \
            <Title>Mr.</Title>     \
            <Name>Joe Smith</Name> \
            <Gender>Male</Gender>  \
            </Person>"
headers = {'Content-type': 'application/xml'}
conn = http.client.HTTPConnection('restapi')
conn.request('POST', '/protect/xml/xpath', reqBody, headers)
print(conn.getresponse().read())

Scala Client using Apache HTTP Client

This section showcases REST API example with SCALA client.

import org.apache.http.client.methods.HttpPost
import org.apache.http.impl.client.DefaultHttpClient
import org.apache.http.entity.StringEntity
import org.apache.http.util.EntityUtils;

object RestApiClient {
  def main(args: Array[String]): Unit = {
    val post = new HttpPost("http://restapi/protect/xml/xpath")
    post.addHeader("Content-Type","application/xml")
    val inXml = "<Person>"                 +
                    "<Title>Mr.</Title>"     +
                    "<Name>Joe Smith</Name>" +
                    "<Gender>Male</Gender>"  +
                "</Person>";
    val client = new DefaultHttpClient
    val params = client.getParams    
    post.setEntity(new StringEntity(inXml))
    val response = client.execute(post)
    val rspBodyString = EntityUtils.toString(response.getEntity)
    println("RESPONSE: \n" + rspBodyString)
  }
}

Postman (Chrome Plugin) Client

This section describes a REST API example with the Postman client.