Configuring the Protector

The process of configuring the Big Data Protector involves the process of editing the cluster configuration.

After executing the configurator script and creating the UDFs, update the cluster to include the following configurations:

1. Inclusion of the environment variables.
2. Attach the BigDataProtector-Init-Script_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.sh script to the Databricks compute.

Ensure that the ESA is started and in a running state before restarting the Databricks cluster after updating the configurations.

Note: The instructions mentioned in the section apply only to the Application Protector REST approach.

Before you begin

1. Log in to PPC node through SSH.
2. To get the CA certificate, run the following command:

   kubectl -n api-gateway get secret ingress-certificate-secret -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/ProtegrityCA.pem
   

3. Optional: To verify the certificate, run the following command:

   openssl x509 -in /tmp/ProtegrityCA.pem -noout -subject -issuer
   

4. To encode the certificate as a single-line base64 value, run the following command:

   base64 -w0 /tmp/ProtegrityCA.pem
   

5. Store the output of step 4 in the PTY_ESA_CA_CERTIFICATE variable.

   Note: For security reasons, Protegrity recommends storing the base64-encoded value from step 4 in a Databricks Secret and referencing it as {{secrets/<scope_name>/<key_name>}}. For more information about using Databricks Secrets, refer to Secret management.

Editing the Cluster Configuration

1. Log in to the Databricks portal.

2. Edit the required cluster.

3. Expand the Advanced section.

4. Click the Spark tab.

5. Under Environment variables, add the variables, with their values, listed in the table:

   Variable	Value
   PTY_ESA_IP	Enter ESA or PPC IP address.
   PTY_ESA_PORT	Enter the port number to connect to ESA or PPC.
   PTY_ESA_TOKEN	Enter the JWT token to connect to ESA or PPC.
   PTY_ESA_ADMINISTRATOR_USERNAME	Enter the username to connect to ESA or PPC. This is required only if a token is not used.
   PTY_ESA_ADMINISTRATOR_PASSWORD	{{secrets/<scope_name>/<key_name>}}This is required only if a token is not used.
   PTY_AUDIT_STORE_IP_PORT	Enter the IP address to connect to the Audit Store. The value is a comma-separated string of <audit_store_ip>:<audit_store_port>. For example, 11.22.33.44:9200, 55.66.77.88:9200
   PTY_PROTECTOR_CONFIGURATION	Enter the protector configuration values. The values will be a single string of comma-separated configurations. For example: [Policy management] policyrefreshinterval = 60 [Policy management] emptystring = null
   PTY_ESA_CA_CERTIFICATE	{{secrets/<scope_name>/<key_name>}} Set this configuration only to interact with PPC. Enter the base64-encoded PEM of the ESA signing CA. Required when the ESA presents a CA-signed certificate without its CA chain.
   PTY_ESA_FQDN	Enter the Fully Qualified Domain Name of ESA or PPC. This is an optional parameter.

   Note: To store the ESA or PPC password and PPC CA certificate, it is recommended to use Databricks Secrets. For more information about using Databricks Secrets, refer to Secret management.

6. Click the Init scripts tab.

7. From the Source list, select Volumes.

8. In the File path box, enter the location of the initialization script.

9. To save the changes and restart the cluster, click Confirm and restart.

   Note: If the initialization script fails with a non-zero exit code, enable cluster logging to view the error log files for troubleshooting purposes.

   When the cluster is restarted, the initialization script starts the Application Protector REST service on every node in the cluster. After the Application Protector REST service is started, use the Unity Catalog Batch Python UDFs to protect and unprotect data.

   Note: The process to execute the initialization script will take some time before the cluster is ready to use for performing protect and unprotect operations. For more information on using the UDFs for protect and unprotect operations, refer to Unity Catalog Batch Python UDFs.