Configuring the Protector

The process of configuring the Big Data Protector involves the process of editing the cluster configuration.

After executing the configurator script and creating the UDFs, update the cluster to include the following configurations:

1. Inclusion of the environment variables.
2. Attach the BigDataProtector-Init-Script_Linux-ALL-64_<ARCH>_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh script to the Databricks compute.

Ensure that the ESA is started and in a running state before restarting the Databricks cluster after updating the configurations.

Note: The instructions mentioned in the section apply only to the Application Protector REST approach.

To edit the cluster:

1. Log in to the Databricks portal.

2. Edit the required cluster.

3. Expand the Advanced section.

4. Click the Spark tab.

5. Under Environment variables, add the variables, with their values, listed in the table:

   • x86-64
   • ARM64

   Variable	Value
   PTY_ESA_IP	Enter the ESA IP address.
   PTY_ESA_PORT	Enter the port number to connect to ESA.
   PTY_ESA_TOKEN	Enter the JWT token to connect to ESA.
   PTY_ESA_ADMINISTRATOR_USERNAME	Enter the username to connect to ESA. This is required only if a token is not used.
   PTY_ESA_ADMINISTRATOR_PASSWORD	{{secrets/<scope_name>/<key_name>}} This is required only if a token is not used.
   PTY_AUDIT_STORE_IP_PORT	Enter the IP address to connect to the Audit Store. The value is a comma-separated string of <audit_store_ip>:<audit_store_port>. For example, 11.22.33.44:9200, 55.66.77.88:9200
   PTY_PROTECTOR_CONFIGURATION	Specify the values as [core]emptystring=empty,[sync]interval=10

   Note: To store the ESA password, it is recommended to use Databricks Secrets. For more information about using Databricks Secrets, refer to Secret management.

   Variable	Value
   PTY_ESA_IP	Enter the ESA IP address.
   PTY_ESA_PORT	Enter the port number to connect to ESA.
   PTY_ESA_TOKEN	Enter the JWT token to connect to ESA.
   PTY_ESA_ADMINISTRATOR_USERNAME	Enter the username to connect to ESA. This is required only if a token is not used.
   PTY_ESA_ADMINISTRATOR_PASSWORD	{{secrets/<scope_name>/<key_name>}} This is required only if a token is not used.
   PTY_AUDIT_STORE_IP_PORT	Enter the IP address to connect to the Audit Store. The value is a comma-separated string of <audit_store_ip>:<audit_store_port>. For example, 11.22.33.44:9200, 55.66.77.88:9200
   PTY_PROTECTOR_CONFIGURATION	Specify the values as [core]emptystring=empty,[sync]interval=10
   PTY_ESA_CA_CERTIFICATE	{{secrets/<scope_name>/<key_name>}} Set this configuration only to interact with PPC. Enter the base64-encoded PEM of the ESA signing CA. Required when the ESA presents a CA-signed certificate without its CA chain.
   PTY_ESA_FQDN	Enter the Fully Qualified Domain Name of ESA or PPC. This is an optional parameter.

   Note: To store the ESA password, it is recommended to use Databricks Secrets. For more information about using Databricks Secrets, refer to Secret management.

6. Click the Init scripts tab.

7. From the Source list, select Volumes.

8. In the File path box, enter the location of the initialization script.

9. To save the changes and restart the cluster, click Confirm and restart.

   Note: If the initialization script fails with a non-zero exit code, enable cluster logging to view the error log files for troubleshooting purposes. When the cluster is restarted, the initialization script starts the Application Protector REST service on every node in the cluster. After the Application Protector REST service is started, use the Unity Catalog Batch Python UDFs to protect and unprotect data.

   Note: The process to execute the initialization script will take some time before the cluster is ready to use for performing protect and unprotect operations. For more information on using the UDFs for protect and unprotect operations, refer to Unity Catalog Batch Python UDFs.