This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

AWS Databricks

Big Data Protector on AWS Databricks

1: Understanding the architecture

1.1: For the Application Protector REST Approach
1.2: For the Cloud Protector Approach

2: System Requirements

2.1: For the Application Protector REST Approach
2.2: For the Cloud Protector Approach

3: Preparing the Environment

3.1: Extracting the Installation Package
3.2: Working with the Configurator Script
3.3: Retrieving the IP Address
3.4: Uploading the Secrets

4: Installing the Protector

5: Configuring the Protector

6: Uninstalling the Protector

The Protegrity Big Data Protector for AWS Databricks delivers end‑to‑end data protection. Organizations deploying the Big Data Protector rely on modern, supported storage options such as Workspace storage, Unity Catalog Volumes, and cloud object storage like Amazon S3.

Designed to secure sensitive data across analytics pipelines, the Big Data Protector applies advanced tokenization and encryption during Spark execution and enforces centralized, policy‑driven controls. Whether installed via Workspace-backed paths or deployed using S3 buckets for configuration and script delivery, the Protector ensures resilient execution across AWS Databricks clusters.

By embracing cloud‑native storage paths, this approach ensures long‑term compatibility with Databricks platform changes while maintaining Protegrity’s standard of seamless and transparent protection. Organizations can continue to process high‑value datasets on AWS Databricks with confidence knowing that sensitive information is secured across its lifecycle, even as the underlying platform evolves.

The Protegrity Big Data Protector for AWS Databricks empowers organizations to secure sensitive data across their analytics pipelines by combining high‑performance protection mechanisms with flexible deployment models tailored for modern cloud architectures. Central to this capability are two approaches:

Each approach is designed to address different customer requirements around scalability, infrastructure usage, and cost optimization. Together, these two approaches provide enterprises the flexibility to choose a data protection strategy aligned with their architectural, cost, and compliance requirements. For example, a fully cluster-local data protection strategy can be leveraged using AP REST. On the other hand, a data protection strategy can also be centrally managed via Cloud Protector, or in hybrid deployments. This dual-path model ensures that AWS Databricks customers can achieve seamless, transparent, policy-based data protection while continuing to extract high-value insights from their data securely and efficiently.

Note: The documentation covers instructions to install the Big Data Protector on AWS Databricks using the x86-64 and ARM64 architectures. For ease of access, the commands and or terminal logs will appear in a tabbed format for each of the architectures.

Application Protector REST Approach

The AP REST model enables data protection directly within the Databricks cluster itself, eliminating the need for a separate Cloud API infrastructure. This approach is particularly suitable for customers who want to avoid maintaining additional cloud-native services for protection operations.

With AP REST, protection workflows are executed through REST endpoints running on the cluster, allowing seamless scaling along with Databricks’ auto-scaling compute. This ensures that sensitive data remains protected throughout processing while also adapting automatically to dynamically assigned IPs in auto-scaling environments. This results in an operationally efficient fit for Spark-driven workloads on AWS.

For the Application Protector REST Approach, the following cluster types are supported:

Databricks Dedicated Compute
Databricks Standard Compute

For the Application Protector REST approach, the following sections are applicable:

Cloud Protector Approach

The Cloud Protector approach extends protection capabilities by offering centralized, cloud-hosted security services for environments that require externally managed protection layers. It enables highly scalable, policy-driven tokenization and encryption without requiring protection logic to reside inside the Databricks compute itself.

In contexts where Cloud Protector is integrated with the Big Data Protector, organizations benefit from lifecycle-wide protection that spans storage, compute, and inter-system data transfers. Cloud Protector provides the foundation for UDF-driven protections, including Spark and Unity Catalog–level enforcement, ensuring centralized governance across distributed analytics ecosystems.

For the Cloud Protector approach, the following cluster types are supported:

Databricks Dedicated Compute
Databricks Standard Compute
Databricks SQL Warehouse

For the Cloud Protector approach, the following sections are applicable:

1 - Understanding the architecture

Architecture for the Big Data Protector on AWS Databricks

1.1 - For the Application Protector REST Approach

The architecture for installing the AWS Databricks protector using the Application Protector REST approach is depicted in the image below.

An outline of the steps in the workflow is explained below.

Download the AWS Databricks build from the customer portal and extract the configurator script.
Execute the configurator script to retrieve the IP address of the Application Protector REST server.
Use the IP address to generate the CA, client, and server certificates.
Store the content of the CA and the client certificates as Secrets in the Secret Manager.
Create a Databricks Unity Catalog Service Credential to access the Secrets from the Secret Manager .
Execute the configurator script to create the Unity Catalog Batch Python UDFs.
Edit the cluster configuration to include the environment variables and attach the initialization script.

1.2 - For the Cloud Protector Approach

The architecture for installing the AWS Databricks protector using the Cloud Protector approach is depicted in the image below.

An outline of the steps in the workflow is explained below.

Install and configure the Cloud Protector.
Create an AWS Databricks Unity Catalog Service Credential and connect it with AWS IAM roles.
Create a Databricks Compute.
On a Linux staging machine, download and extract the installation package for AWS Databricks from the customer portal, for the Databricks Compute.
Execute the configurator script to create the Batch Python UDFs at the Unity Catalog level.
Attach an AWS Databricks Notebook to the Databricks Compute.
Execute the Unity Catalog Batch Python UDFs to protect and unprotect data.

2 - System Requirements

Understand the system requirements to install the Big Data Protector

2.1 - For the Application Protector REST Approach

Understand the system requirements to install the Big Data Protector using the Application Protector REST Approach

Ensure that the following prerequisites are available before installing the Big Data Protector:

Python3 along with the requests module is installed on the machine to execute the configurator script.
A compatible version of the ESA is installed, configured, and running.
Access to the Databricks workspace is available.
A Databricks cluster, of any one of the following types, is created and is in the running state:
- Dedicated Compute
- Standard Compute
Create the Databricks Service Principal.
The Databricks Service Principal must have the Can attach to permission on the cluster.
Create the following certificates for mutual TLS authorization:
- CA Certificate
- Server Certificate
- Non-encrypted Server Key
- Client Certificate
- Non-encrypted Client Key
Note: These certificates must be generated ONLY after retrieving the IP address of the Application Protector REST server.
Permission to create a Secrets Manager and store secrets is available.
Create an AWS Databricks Unity Catalog Service Credential.
Note: For more information about creating the credential, refer to Create service credentials.
The Databricks Service Principal must have the access permissions on the Databricks Unity Catalog Service Credential.
A Databricks Unity Catalog Volume is available with a Catalog and a Schema and the following permissions:
- The Databricks Service Principal must have the Read volume and Write volume permission on the Databricks Unity Catalog Volume.
- The Databricks Service Principal must have the Use catalog permission at the Catalog level.
- The Databricks Service Principal must have the Use schema permission at the Schema level.
- The Databricks Service Principal must have the Create function permission at the Schema level.
- The Databricks Service Principal must have the manage permission at the Schema level.

2.2 - For the Cloud Protector Approach

Understand the system requirements to install the Big Data Protector using the Cloud Protector Approach

The prerequisites required to install and run the Big Data Protector on a Databricks Compute are listed below.

Python3 along with the requests module is installed on the machine to execute the configurator script.
A compatible version of the ESA is installed, configured, and running.
Access to the Databricks workspace is available.
A Databricks cluster, of any one of the following types, is created and is in the running state:
- Dedicated Compute
- Standard Compute
- SQL Warehouse
Create the Databricks Service Principal.
The Databricks Service Principal must have the Can attach to permission on the cluster.
Install and configure the Cloud API on AWS.
Note: For more information about installing and configuring the Cloud API on AWS, refer to Cloud API.
To modify the core parameters for RPSync, refer to Policy Agent Lambda Configuration.
Install and configure a compatible version of the ESA.
Note: For more information about compatible ESA versions, refer to Cloud API.
Create an AWS Databricks Unity Catalog Service Credential.
Note: For more information about creating the credential, refer to Create service credentials.
Assigned the ACCESS privilege to the principals that will be using the AWS Databricks Unity Catalog Service Credential.
Create a service principal and OAuth secret to deploy the UDFs.
Note: For more information, refer to Authorize service principal access to Databricks with OAuth.
(Optional) Configure private connectivity to the Protegrity Cloud API.
Note: For more information, refer to Configure private connectivity to resources in your VPC.
A Databricks Unity Catalog Volume is available with a Catalog and a Schema and the following permissions:
- The Databricks Service Principal must have the ATTACH or MANAGE permission on the compute.
- The Databricks Service Principal must have the Read volume and Write volume permission on the Databricks Unity Catalog Volume.
- The Databricks Service Principal must have the Use catalog permission at the Catalog level.
- The Databricks Service Principal must have the Use schema permission at the Schema level.
- The Databricks Service Principal must have the Create function permission at the Schema level.
- The Databricks Service Principal must have the manage permission at the Schema level.
To use a SQL Warehouse with the Cloud Protector approach, create a SQL Warehouse. For more information, refer to Create a SQL warehouse.

3 - Preparing the Environment

Prepare the environment to install the Big Data Protector

3.1 - Extracting the Installation Package

Extract the files from the installation package

Extract the contents of the installation package to access the configurator script. This script generates the required files to install the Big Data Protector.

To extract the files from the installation package:

Download the Big Data Protector package to any local directory.

BigDataProtector_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz

BigDataProtector_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz

To extract the files from the installation package, run the following command:

tar -xvf BigDataProtector_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz

tar -xvf BigDataProtector_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz

Press ENTER.
The command extracts the installation package and the GPG signature files.

BigDataProtector_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz
signatures/
signatures/BigDataProtector_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz_10.0.sig

BigDataProtector_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz
signatures/
signatures/BigDataProtector_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz_10.0.sig

Note: Verify the authenticity of the build using the signatures folder. For more information, refer to Verification of Signed Protector Build.

To extract the configurator script, run the following command:

tar -xvf BigDataProtector_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz

tar -xvf BigDataProtector_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz

Press ENTER.
The command extracts the configurator script.

BigDataProtector-Configurator_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

BigDataProtector-Configurator_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

3.2 - Working with the Configurator Script

Understand help options available in the Configurator script

The configurator script performs the following tasks:

Generate the IP address for the Application Protector REST server.
Create the UDFs.
Delete the UDFs.

The configurator script provides the --help option to understand the options and the arguments to be provided.

To understand the options and the arguments for the configurator script:

To view the options and the arguments, run the following command:

./BigDataProtector-Configurator_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh --help

./BigDataProtector-Configurator_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh --help

Press ENTER. The command displays all the options and the arguments required to execute the configurator script.

This script needs the following inputs as a string:
 1. The ID of the operation.
    ----------------------------------------------------------
    | ID | Operation                                         |
    ----------------------------------------------------------
    |  1 | Get Application Protector REST's Server IP        |
    |  2 | Create Databricks Unity Catalog Batch Python UDFs |
    |  3 | Delete Databricks Unity Catalog Batch Python UDFs |
    ----------------------------------------------------------
 2. The URL of the Databricks Workspace.
 3. The Application ID of the Databricks Service Principal
 4. The OAuth Secret of the Databricks Service Principal
 5. The ID of the Databricks Compute.

If the ID of the operation is specified as "2" or "3", then the script will require the following additional inputs as a string:
 6. The name of the Databricks Unity Catalog Catalog-Schema.
 7. The ID of the approach.
    -----------------------------------
    | ID | Approach                   |
    -----------------------------------
    |  1 | Application Protector REST |
    |  2 | Cloud Protector            |
    -----------------------------------

If the ID of the operation is specified as "2" and the ID of the approach is specified as "1", then the script will require the following additional inputs as a string:
 8. The path of the CA Certificate.
 9. The path of the Server Certificate.
10. The path of the Server Key.
11. The name of the AWS Secret.
12. The name of the AWS Secret's AWS Region.
13. The name of the Databricks Unity Catalog Service Credential.
14. The path of the Databricks Unity Catalog Volume.

If the ID of the operation is specified as "2" and the ID of the approach is specified as "2", then the script will require the following additional inputs as a string:
 8. The name of the AWS Lambda Function.
 9. The name of the AWS Lambda Function's AWS Region.
10. The name of the Databricks Unity Catalog Service Credential.

If the ID of the operation is specified as "3" and the ID of the approach is specified as "1", then the script will require the following additional input as a string:
 8. The path of the Databricks Unity Catalog Volume.

This script accepts the above-mentioned inputs in any one of the following ways:
 1. Using .cfg file (pass the path of the .cfg file to this script as a command-line argument).
 2. Using command-line arguments.
 3. Using interactive prompts.

Structure of the .cfg file:
operation_id = "operation_id"
databricks_workspace_url = "databricks_workspace_url"
databricks_service_principal_application_id = "databricks_service_principal_application_id"
databricks_service_principal_oauth_secret = "databricks_service_principal_oauth_secret"
databricks_compute_id = "databricks_compute_id"
databricks_unity_catalog_catalog_schema_name = "databricks_unity_catalog_catalog_schema_name"
approach_id = "approach_id"
ca_certificate_path = "ca_certificate_path"
server_certificate_path = "server_certificate_path"
server_key_path = "server_key_path"
aws_secret_name = "aws_secret_name"
aws_secret_aws_region_name = "aws_secret_aws_region_name"
databricks_unity_catalog_service_credential_name = "databricks_unity_catalog_service_credential_name"
databricks_unity_catalog_volume_path = "databricks_unity_catalog_volume_path"
aws_lambda_function_name = "aws_lambda_function_name"
aws_lambda_function_aws_region_name = "aws_lambda_function_aws_region_name"

Syntax of the command-line arguments:
--operation_id "operation_id"
--databricks_workspace_url "databricks_workspace_url"
--databricks_service_principal_application_id "databricks_service_principal_application_id"
--databricks_service_principal_oauth_secret "databricks_service_principal_oauth_secret"
--databricks_compute_id "databricks_compute_id"
--databricks_unity_catalog_catalog_schema_name "databricks_unity_catalog_catalog_schema_name"
--approach_id "approach_id"
--ca_certificate_path "ca_certificate_path"
--server_certificate_path "server_certificate_path"
--server_key_path "server_key_path"
--aws_secret_name "aws_secret_name"
--aws_secret_aws_region_name "aws_secret_aws_region_name"
--databricks_unity_catalog_service_credential_name "databricks_unity_catalog_service_credential_name"
--databricks_unity_catalog_volume_path "databricks_unity_catalog_volume_path"
--aws_lambda_function_name "aws_lambda_function_name"
--aws_lambda_function_aws_region_name "aws_lambda_function_aws_region_name"

This script utilizes one of the following approaches:
-----------------------------------
| ID | Approach                   |
-----------------------------------
|  1 | Application Protector REST |
|  2 | Cloud Protector            |
-----------------------------------

###########################################################################################

If you want to use Application Protector REST approach, following operations are supported:
----------------------------------------------------------
| ID | Operation                                         |
----------------------------------------------------------
|  1 | Get Application Protector REST's Server IP        |
|  2 | Create Databricks Unity Catalog Batch Python UDFs |
|  3 | Delete Databricks Unity Catalog Batch Python UDFs |
----------------------------------------------------------

To Get Application Protector REST's Server IP, following inputs are required:
Using interactive prompts:
1. The ID of the operation as "1"
2. The URL of the Databricks Workspace
3. The Application ID of the Databricks Service Principal
4. The OAuth Secret of the Databricks Service Principal
5. The ID of the Databricks Compute

Using command-line arguments:
--operation_id "1"
--databricks_workspace_url "databricks_workspace_url"
--databricks_service_principal_application_id "databricks_service_principal_application_id"
--databricks_service_principal_oauth_secret "databricks_service_principal_oauth_secret"
--databricks_compute_id "databricks_compute_id"

Using .cfg file (pass the path of the .cfg file to this script as a command-line argument):
operation_id = "1"
databricks_workspace_url = "databricks_workspace_url"
databricks_service_principal_application_id = "databricks_service_principal_application_id"
databricks_service_principal_oauth_secret = "databricks_service_principal_oauth_secret"
databricks_compute_id = "databricks_compute_id"

To Create Databricks Unity Catalog Batch Python UDFs, following inputs are required:
Using interactive prompts:
1. The ID of the operation as "2"
2. The URL of the Databricks Workspace
3. The Application ID of the Databricks Service Principal
4. The OAuth Secret of the Databricks Service Principal
5. The ID of the Databricks Compute
6. The name of the Databricks Unity Catalog Catalog-Schema
7. The ID of the approach as "1"
8. The path of the CA Certificate
9. The path of the Server Certificate
10. The path of the Server Key
11. The name of the AWS Secret
12. The name of the AWS Secret's AWS Region
13. The name of the Databricks Unity Catalog Service Credential
14. The path of the Databricks Unity Catalog Volume

Using command-line arguments:
--operation_id "2"
--databricks_workspace_url "databricks_workspace_url"
--databricks_service_principal_application_id "databricks_service_principal_application_id"
--databricks_service_principal_oauth_secret "databricks_service_principal_oauth_secret"
--databricks_compute_id "databricks_compute_id"
--databricks_unity_catalog_catalog_schema_name "databricks_unity_catalog_catalog_schema_name"
--approach_id "1"
--ca_certificate_path "ca_certificate_path"
--server_certificate_path "server_certificate_path"
--server_key_path "server_key_path"
--aws_secret_name "aws_secret_name"
--aws_secret_aws_region_name "aws_secret_aws_region_name"
--databricks_unity_catalog_service_credential_name "databricks_unity_catalog_service_credential_name"
--databricks_unity_catalog_volume_path "databricks_unity_catalog_volume_path"

Using .cfg file (pass the path of the .cfg file to this script as a command-line argument):
operation_id = "2"
databricks_workspace_url = "databricks_workspace_url"
databricks_service_principal_application_id = "databricks_service_principal_application_id"
databricks_service_principal_oauth_secret = "databricks_service_principal_oauth_secret"
databricks_compute_id = "databricks_compute_id"
databricks_unity_catalog_catalog_schema_name = "databricks_unity_catalog_catalog_schema_name"
approach_id = "1"
ca_certificate_path = "ca_certificate_path"
server_certificate_path = "server_certificate_path"
server_key_path = "server_key_path"
aws_secret_name = "aws_secret_name"
aws_secret_aws_region_name = "aws_secret_aws_region_name"
databricks_unity_catalog_service_credential_name = "databricks_unity_catalog_service_credential_name"
databricks_unity_catalog_volume_path = "databricks_unity_catalog_volume_path"

To Delete Databricks Unity Catalog Batch Python UDFs, following inputs are required:
Using interactive prompts:
1. The ID of the operation as "3"
2. The URL of the Databricks Workspace
3. The Application ID of the Databricks Service Principal
4. The OAuth Secret of the Databricks Service Principal
5. The ID of the Databricks Compute
6. The name of the Databricks Unity Catalog Catalog-Schema
7. The ID of the approach as "1"
8. The path of the Databricks Unity Catalog Volume

Using command-line arguments:
--operation_id "3"
--databricks_workspace_url "databricks_workspace_url"
--databricks_service_principal_application_id "databricks_service_principal_application_id"
--databricks_service_principal_oauth_secret "databricks_service_principal_oauth_secret"
--databricks_compute_id "databricks_compute_id"
--databricks_unity_catalog_catalog_schema_name "databricks_unity_catalog_catalog_schema_name"
--approach_id "1"
--databricks_unity_catalog_volume_path "databricks_unity_catalog_volume_path"

Using .cfg file (pass the path of the .cfg file to this script as a command-line argument):
operation_id = "3"
databricks_workspace_url = "databricks_workspace_url"
databricks_service_principal_application_id = "databricks_service_principal_application_id"
databricks_service_principal_oauth_secret = "databricks_service_principal_oauth_secret"
databricks_compute_id = "databricks_compute_id"
databricks_unity_catalog_catalog_schema_name = "databricks_unity_catalog_catalog_schema_name"
approach_id = "1"
databricks_unity_catalog_volume_path = "databricks_unity_catalog_volume_path"

################################################################################

If you want to use Cloud Protector approach, following operations are supported:
----------------------------------------------------------
| ID | Operation                                         |
----------------------------------------------------------
|  1 | Create Databricks Unity Catalog Batch Python UDFs |
|  2 | Delete Databricks Unity Catalog Batch Python UDFs |
----------------------------------------------------------

To Create Databricks Unity Catalog Batch Python UDFs, following inputs are required:
Using interactive prompts:
1. The ID of the operation as "2"
2. The URL of the Databricks Workspace
3. The Application ID of the Databricks Service Principal
4. The OAuth Secret of the Databricks Service Principal
5. The ID of the Databricks Compute
6. The name of the Databricks Unity Catalog Catalog-Schema
7. The ID of the approach as "2"
8. The name of the AWS Lambda Function
9. The name of the AWS Lambda Function's AWS Region
10. The name of the Databricks Unity Catalog Service Credential

Using command-line arguments:
--operation_id "2"
--databricks_workspace_url "databricks_workspace_url"
--databricks_service_principal_application_id "databricks_service_principal_application_id"
--databricks_service_principal_oauth_secret "databricks_service_principal_oauth_secret"
--databricks_compute_id "databricks_compute_id"
--databricks_unity_catalog_catalog_schema_name "databricks_unity_catalog_catalog_schema_name"
--approach_id "2"
--aws_lambda_function_name "aws_lambda_function_name"
--aws_lambda_function_aws_region_name "aws_lambda_function_aws_region_name"
--databricks_unity_catalog_service_credential_name "databricks_unity_catalog_service_credential_name"

Using .cfg file (pass the path of the .cfg file to this script as a command-line argument):
operation_id = "2"
databricks_workspace_url = "databricks_workspace_url"
databricks_service_principal_application_id = "databricks_service_principal_application_id"
databricks_service_principal_oauth_secret = "databricks_service_principal_oauth_secret"
databricks_compute_id = "databricks_compute_id"
databricks_unity_catalog_catalog_schema_name = "databricks_unity_catalog_catalog_schema_name"
approach_id = "2"
aws_lambda_function_name = "aws_lambda_function_name"
aws_lambda_function_aws_region_name = "aws_lambda_function_aws_region_name"
databricks_unity_catalog_service_credential_name = "databricks_unity_catalog_service_credential_name"

To Delete Databricks Unity Catalog Batch Python UDFs, following inputs are required:
Using interactive prompts:
1. The ID of the operation as "3"
2. The URL of the Databricks Workspace
3. The Application ID of the Databricks Service Principal
4. The OAuth Secret of the Databricks Service Principal
5. The ID of the Databricks Compute
6. The name of the Databricks Unity Catalog Catalog-Schema
7. The ID of the approach as "2"

Using command-line arguments:
--operation_id "3"
--databricks_workspace_url "databricks_workspace_url"
--databricks_service_principal_application_id "databricks_service_principal_application_id"
--databricks_service_principal_oauth_secret "databricks_service_principal_oauth_secret"
--databricks_compute_id "databricks_compute_id"
--databricks_unity_catalog_catalog_schema_name "databricks_unity_catalog_catalog_schema_name"
--approach_id "2"

Using .cfg file (pass the path of the .cfg file to this script as a command-line argument):
operation_id = "3"
databricks_workspace_url = "databricks_workspace_url"
databricks_service_principal_application_id = "databricks_service_principal_application_id"
databricks_service_principal_oauth_secret = "databricks_service_principal_oauth_secret"
databricks_compute_id = "databricks_compute_id"
databricks_unity_catalog_catalog_schema_name = "databricks_unity_catalog_catalog_schema_name"
approach_id = "2"

3.3 - Retrieving the IP Address

Retrieve the IP Address for the Application Protector REST server

The IP address for the Application Protector REST approach is required to generate the certificates. The certificates must be created using the retrieved IP address. These certificates will be used to establish a mutual trust between the Unity Catalog Batch Python UDFs and the Application Protector REST Server.

Note: The instructions mentioned in the section apply only to the Application Protector REST approach.

To retrieve the IP address:

To execute the configurator script, run the following command:

./BigDataProtector-Configurator_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

./BigDataProtector-Configurator_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

Press ENTER.
The prompt to enter the operation ID appears.

Creating installation files...
Created installation files.

Enter the ID of the operation:

Creating installation files...
Created installation files.

The ID of the operation needs to be one of the following:
----------------------------------------------------------
| ID | Operation                                         |
----------------------------------------------------------
|  1 | Get Application Protector REST's Server IP        |
|  2 | Create Databricks Unity Catalog Batch Python UDFs |
|  3 | Delete Databricks Unity Catalog Batch Python UDFs |
----------------------------------------------------------
Enter the ID of the operation:

To retrieve the IP address of the Application Protector REST server, type 1.
Press ENTER.
The prompt to enter the Databricks Workspace URL appears.
```
Enter the URL of the Databricks Workspace:
```
Enter the Databricks Workspace URL.
Press ENTER.
The prompt to enter the application ID of the Databricks Service Principal appears.
```
Enter the Application ID of the Databricks Service Principal:
```
Enter the Application ID of the Databricks Service Principal.
Press ENTER.
The prompt to enter the OAuth secret for the Service Principal appears.
```
Enter the OAuth Secret of the Databricks Service Principal:
```
Enter the OAuth secret.
Press ENTER.
The prompt to enter the cluster ID appears.
```
Enter the ID of the Databricks Compute:
```
Enter the Cluster ID.

Press ENTER.
The script retrieves the IP address of the Application Protector REST server.

Executing specified operation...

APREST Protector's Server IP: x.x.x.x

Executed specified operation.

3.4 - Uploading the Secrets

Upload the secrets to the Secret Manager for the Application Protector REST approach

The CA and the Client certificates are important entities in the mutual trust process. These certificates determine the authentication and authorization for the Application Protector REST server. As a result, it is critical to store these certificates in a secured location. Therefore, the certificates must be uploaded to the Secrets Manager in AWS where they will be stored as secrets.

Note: The instructions mentioned in the section apply only to the Application Protector REST approach.

To upload the secrets:

Create a Secrets Manager in AWS to upload the secrets.

Assign the required access permissions to the Secrets Manager. For example:

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "VisualEditor0",
        "Effect": "Allow",
        "Action": [
            "secretsmanager:*"
        ],
        "Resource": [
            "arn:aws:secretsmanager:<aws_region_name>:<aws_account>:secret:*"
        ]
    },
    {
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::<aws_account>:role/<aws_iam_role>",
        "Effect": "Allow"
    }
]
}

Log in to the machine where the certificates are created.
Launch the python console.
To view the contents of the CA.pem file and store it as PTY-APPLICATION-PROTECTOR-REST-CA-CERTIFICATE, run the following command:
```
with open("ca/CA.pem") as file:
    file.read()
Store CA cert as PTY-APPLICATION-PROTECTOR-REST-CA-CERTIFICATE
```
Press ENTER.
The command displays the contents of the CA.pem file.

To view the contents of the client.pem file and store it as PTY-APPLICATION-PROTECTOR-REST-CLIENT-CERTIFICATE, run the following command:

with open("client/client.pem") as file:
    file.read()
Store client cert as PTY-APPLICATION-PROTECTOR-REST-CLIENT-CERTIFICATE

Press ENTER.
The command displays the contents of the client.pem file.
To view the contents of the client.key file and store it as PTY-APPLICATION-PROTECTOR-REST-CLIENT-KEY, run the following command:
```
with open("client/client.key") as file:
    file.read()
Store client key as PTY-APPLICATION-PROTECTOR-REST-CLIENT-KEY
```
Press ENTER.
The command displays the contents of the client.key file.
Log in to the AWS portal.
Navigate to the required Secrets Manager.
Click Store a new secret.
The Choose secret type page appears.
From the Secret type section, select Other type of secret.

Enter the details, as listed in the table, in a new row.

Key	Value
PTY-APPLICATION-PROTECTOR-REST-CA-CERTIFICATE	1. In the Key box, enter PTY-APPLICATION-PROTECTOR-REST-CA-CERTIFICATE. 2. In the Value box, enter the contents of the CA.pem file.
PTY-APPLICATION-PROTECTOR-REST-CLIENT-CERTIFICATE	1. In the Key box, enter PTY-APPLICATION-PROTECTOR-REST-CLIENT-CERTIFICATE. 2. In the Value box, enter the contents of the client.pem file.
PTY-APPLICATION-PROTECTOR-REST-CLIENT-KEY	1. In the Key box, enter PTY-APPLICATION-PROTECTOR-REST-CLIENT-KEY. 2. In the Value box, enter the contents of the client.key file.

Click Next.
The Configure secret page appears.
In the Secret name box, enter a name to identify the secret.
Click Next.
The Configure rotation page appears.
Click Next.
The Review page appears.
Verify the details.
Click Store.
The secrets are stored as per the specified details.

4 - Installing the Protector

Install the Big Data Protector on AWS Databricks

The process of installing the Big Data Protector involves the process of creating the user-defined functions.

The following combinations will work for a successful execution of the configurator script:

Databricks Dedicated Compute + Application Protector REST approach
Databricks Dedicated Compute + Cloud Protector approach
Databricks Standard Compute + Application Protector REST approach
Databricks Standard Compute + Cloud Protector approach
Databricks SQL Warehouse + Cloud Protector approach

The Databricks SQL Warehouse + Application Protector REST approach combination will not work. This is because Protegrity executes a few Python commands on the Databricks Compute to retrieve a listening IP for the Application Protector REST’s Server. When the Databricks Compute is a SQL Warehouse, the Python commands fail to execute. This occurs because the SQL Warehouse supports only SQL commands.

Creating the UDFs For the Application Protector REST Approach

The configurator script is used to create the UDFs. These Unity Catalog Batch Python UDFs are designed to perform data protection and unprotection operations. Select the required approach and the operation ID to create the UDFs using the Application Protector REST server. This section explains the process to create the UDFs using the interactive method of installation.

To create the UDFs:

Log in to the staging machine.
Navigate to the directory where the installation files are extracted.

To execute the configurator script, run the following command:

./BigDataProtector-Configurator_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

./BigDataProtector-Configurator_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

Press ENTER.
The prompt to enter the operation ID appears.

Creating installation files...
Created installation files.
 
Enter the ID of the operation:

Creating installation files...
Created installation files.

The ID of the operation needs to be one of the following:
----------------------------------------------------------
| ID | Operation                                         |
----------------------------------------------------------
|  1 | Get Application Protector REST's Server IP        |
|  2 | Create Databricks Unity Catalog Batch Python UDFs |
|  3 | Delete Databricks Unity Catalog Batch Python UDFs |
----------------------------------------------------------
Enter the ID of the operation:

To create the UDFs, type 2.
Press ENTER.
The prompt to enter the Databricks Workspace URL appears.
```
Enter the URL of the Databricks Workspace:
```
Enter the Databricks Workspace URL.
Press ENTER.
The prompt to enter the application ID of the Databricks Service Principal appears.
```
Enter the Application ID of the Databricks Service Principal:
```
Enter the Application ID of the Databricks Service Principal.
Press ENTER.
The prompt to enter the OAuth secret for the Service Principal appears.
```
Enter the OAuth Secret of the Databricks Service Principal:
```
Enter the OAuth secret.
Press ENTER.
The prompt to enter the cluster ID appears.
```
Enter the ID of the Databricks Compute:
```
Note: The Cluster ID can be either for Standard Compute or Dedicated Compute. For more information about identifying the Cluster ID, refer to Get identifiers for workspace objects.
Enter the Cluster ID.

Press ENTER.
The prompt to enter the name of the schema appears.

Enter the name of the Databricks Unity Catalog Catalog-Schema:

Enter the name of the catalog and the schema in the <catalog_name.schema_name> format.

Press ENTER.
The prompt to select the approach appears.

Enter the ID of the approach:

The ID of the approach needs to be one of the following:
-----------------------------------
| ID | Approach                   |
-----------------------------------
|  1 | Application Protector REST |
|  2 | Cloud Protector            |
-----------------------------------
Enter the ID of the approach:

To create the UDFs using the Application Protector REST approach, type 1.
Press ENTER.
The prompt to enter the path of the CA Certificate appears.
```
Enter the path of the CA Certificate:
```
Enter the path of the CA Certificate.
Press ENTER.
The prompt to enter the path of the Server Certificate appears.
```
Enter the path of the Server Certificate:
```
Enter the path of the Server Certificate.
Press ENTER.
The prompt to enter the path of the Server key appears.
```
Enter the path of the Server Key:
```
Enter the path of the Server Key.
Press ENTER.
The prompt to enter the name of the AWS Secret appears.
```
Enter the name of the AWS Secret:
```
Enter the name of the AWS Secret.
Press ENTER.
The prompt to enter the region of the Secret appears.
```
Enter the name of the AWS Secret's AWS Region:
```
Enter the region where the Secret is created.
Press ENTER.
The prompt to enter the name of the Service Credential appears.
```
Enter the name of the Databricks Unity Catalog Service Credential:
```
Enter the name of the Databricks Unity Catalog Service Credential.
Press ENTER.
The prompt to enter the path of the Unity Catalog Volume appears.
```
Enter the path of the Databricks Unity Catalog Volume:
```
Enter the path of the Databricks Unity Catalog Volume.

Press ENTER.
The script creates the UDFs at the specified location.

Executing specified operation...

1. Create the following environment variables in the Spark section of the Advanced properties of the Databricks Compute:
PTY_ESA_IP=PTY_ESA_IP
PTY_ESA_PORT=PTY_ESA_PORT
Either PTY_ESA_TOKEN=PTY_ESA_TOKEN or PTY_ESA_ADMINISTRATOR_USERNAME=PTY_ESA_ADMINISTRATOR_USERNAME and PTY_ESA_ADMINISTRATOR_PASSWORD=PTY_ESA_ADMINISTRATOR_PASSWORD
PTY_AUDIT_STORE_IP_PORT=PTY_AUDIT_STORE_IP_PORT
PTY_PROTECTOR_CONFIGURATION=PTY_PROTECTOR_CONFIGURATION
2. Attach "DATABRICKS_UNITY_CATALOG_VOLUME_PATH/DATABRICKS_INIT_SCRIPT_NAME" as an Init Script to the Databricks Compute.
3. Restart the Databricks Compute.

Executed specified operation.

Executing specified operation...
Creating <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_string_fpe Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_string_fpe Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_string_fpe Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_string_fpe Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF.
Creating /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz file...
Created /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz file.
Creating /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector-Init-Script_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh file...
Created /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector-Init-Script_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh file.

1. Create the following environment variables in the Spark section of the Advanced properties of the Databricks Compute:
PTY_ESA_IP=PTY_ESA_IP
PTY_ESA_PORT=PTY_ESA_PORT
Either PTY_ESA_TOKEN=PTY_ESA_TOKEN or PTY_ESA_ADMINISTRATOR_USERNAME=PTY_ESA_ADMINISTRATOR_USERNAME and PTY_ESA_ADMINISTRATOR_PASSWORD=PTY_ESA_ADMINISTRATOR_PASSWORD
PTY_AUDIT_STORE_IP_PORT=PTY_AUDIT_STORE_IP_PORT
PTY_PROTECTOR_CONFIGURATION=PTY_PROTECTOR_CONFIGURATION
2. Attach "/Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector-Init-Script_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh" as an Init Script to the Databricks Compute.
3. Restart the Databricks Compute.

Executed specified operation.

Creating the UDFs For the Cloud Protector Approach

The configurator script is used to create the UDFs. These Unity Catalog Batch Python UDFs are used to perform data protection and unprotection operations. Select the required approach and the operation ID to create the UDFs using the Cloud Protector. This section explains the process to create the UDFs using the interactive method of installation.

To create the UDFs:

Log in to the staging machine.
Navigate to the directory where the installation files are extracted.

To execute the configurator script, run the following command:

./BigDataProtector-Configurator_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

./BigDataProtector-Configurator_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

Press ENTER.
The prompt to enter the operation ID appears.

Creating installation files...
Created installation files.

Enter the ID of the operation:

Creating installation files...
Created installation files.

The ID of the operation needs to be one of the following:
----------------------------------------------------------
| ID | Operation                                         |
----------------------------------------------------------
|  1 | Get Application Protector REST's Server IP        |
|  2 | Create Databricks Unity Catalog Batch Python UDFs |
|  3 | Delete Databricks Unity Catalog Batch Python UDFs |
----------------------------------------------------------
Enter the ID of the operation:

To create the UDFs, type 2.
Press ENTER.
The prompt to enter the Databricks Workspace URL appears.
```
Enter the URL of the Databricks Workspace:
```
Enter the Databricks Workspace URL.
Press ENTER.
The prompt to enter the application ID of the Databricks Service Principal appears.
```
Enter the Application ID of the Databricks Service Principal:
```
Enter the Application ID of the Databricks Service Principal.
Press ENTER.
The prompt to enter the OAuth secret for the Service Principal appears.
```
Enter the OAuth Secret of the Databricks Service Principal:
```
Enter the OAuth secret.
Press ENTER.
The prompt to enter the cluster ID appears.
```
Enter the ID of the Databricks Compute:
```
Note: The Cluster ID can be either for Standard Compute or Dedicated Compute. For more information about identifying the Cluster ID, refer to Get identifiers for workspace objects.
Enter the Cluster ID.

Press ENTER.
The prompt to enter the name of the schema appears.

Enter the name of the Databricks Unity Catalog Catalog-Schema:

Enter the name of the catalog and the schema in the <catalog_name.schema_name> format.

Press ENTER.
The prompt to select the approach appears.

Enter the ID of the approach:

The ID of the approach needs to be one of the following:
-----------------------------------
| ID | Approach                   |
-----------------------------------
|  1 | Application Protector REST |
|  2 | Cloud Protector            |
-----------------------------------
Enter the ID of the approach:

To create the UDFs using the Cloud Protector approach, type 2.
Press ENTER.
The prompt to enter the name of the AWS Lambda Function appears.
```
Enter the name of the AWS Lambda Function:
```
Enter the name of the AWS Lambda Function.
Press ENTER.
The prompt to enter the region of the AWS Lambda function appears.
```
Enter the name of the AWS Lambda Function's AWS Region:
```
Enter the region name.
Press ENTER.
The prompt to enter the name of the Service Credential appears.
```
Enter the name of the Databricks Unity Catalog Service Credential:
```
Enter the name of the Databricks Unity Catalog Service Credential.

Press ENTER.
The script creates the UDFs at the specified location.

Executing specified operation...
Executed specified operation.

Executing specified operation...
Creating <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_protect_string_fpe Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_protect_string_fpe Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_unprotect_string_fpe Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_unprotect_string_fpe Databricks Unity Catalog Batch Python UDF.
Creating <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF...
Created <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF.
Executed specified operation.

5 - Configuring the Protector

Modify the Cluster Configuration for the Application Protector REST approach

The process of configuring the Big Data Protector involves the process of editing the cluster configuration.

After executing the configurator script and creating the UDFs, update the cluster to include the following configurations:

Inclusion of the environment variables.
Attach the BigDataProtector-Init-Script_Linux-ALL-64_<ARCH>_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh script to the Databricks compute.

Ensure that the ESA is started and in a running state before restarting the Databricks cluster after updating the configurations.

Note: The instructions mentioned in the section apply only to the Application Protector REST approach.

To edit the cluster:

Log in to the Databricks portal.
Edit the required cluster.
Expand the Advanced section.
Click the Spark tab.

Under Environment variables, add the variables, with their values, listed in the table:

Variable	Value
`PTY_ESA_IP`	Enter the ESA IP address.
`PTY_ESA_PORT`	Enter the port number to connect to ESA.
`PTY_ESA_TOKEN`	Enter the JWT token to connect to ESA.
`PTY_ESA_ADMINISTRATOR_USERNAME`	Enter the username to connect to ESA. This is required only if a token is not used.
`PTY_ESA_ADMINISTRATOR_PASSWORD`	`{{secrets/<scope_name>/<key_name>}}` This is required only if a token is not used.
`PTY_AUDIT_STORE_IP_PORT`	Enter the IP address to connect to the Audit Store. The value is a comma-separated string of `<audit_store_ip>:<audit_store_port>`. For example, 11.22.33.44:9200, 55.66.77.88:9200
`PTY_PROTECTOR_CONFIGURATION`	Specify the values as [core]emptystring=empty,[sync]interval=10

Note: To store the ESA password, it is recommended to use Databricks Secrets. For more information about using Databricks Secrets, refer to Secret management.

Variable	Value
`PTY_ESA_IP`	Enter the ESA IP address.
`PTY_ESA_PORT`	Enter the port number to connect to ESA.
`PTY_ESA_TOKEN`	Enter the JWT token to connect to ESA.
`PTY_ESA_ADMINISTRATOR_USERNAME`	Enter the username to connect to ESA. This is required only if a token is not used.
`PTY_ESA_ADMINISTRATOR_PASSWORD`	`{{secrets/<scope_name>/<key_name>}}` This is required only if a token is not used.
`PTY_AUDIT_STORE_IP_PORT`	Enter the IP address to connect to the Audit Store. The value is a comma-separated string of `<audit_store_ip>:<audit_store_port>`. For example, 11.22.33.44:9200, 55.66.77.88:9200
`PTY_PROTECTOR_CONFIGURATION`	Specify the values as [core]emptystring=empty,[sync]interval=10
`PTY_ESA_CA_CERTIFICATE`	`{{secrets/<scope_name>/<key_name>}}` Set this configuration only to interact with PPC. Enter the base64-encoded PEM of the ESA signing CA. Required when the ESA presents a CA-signed certificate without its CA chain.
`PTY_ESA_FQDN`	Enter the Fully Qualified Domain Name of ESA or PPC. This is an optional parameter.

Note: To store the ESA password, it is recommended to use Databricks Secrets. For more information about using Databricks Secrets, refer to Secret management.

Click the Init scripts tab.
From the Source list, select Volumes.
In the File path box, enter the location of the initialization script.
To save the changes and restart the cluster, click Confirm and restart.
Note: If the initialization script fails with a non-zero exit code, enable cluster logging to view the error log files for troubleshooting purposes. When the cluster is restarted, the initialization script starts the Application Protector REST service on every node in the cluster. After the Application Protector REST service is started, use the Unity Catalog Batch Python UDFs to protect and unprotect data.
Note: The process to execute the initialization script will take some time before the cluster is ready to use for performing protect and unprotect operations. For more information on using the UDFs for protect and unprotect operations, refer to Unity Catalog Batch Python UDFs.

6 - Uninstalling the Protector

Uninstall the Big Data Protector

The process of uninstalling the Big Data Protector involves the process of removing the user-defined functions.

Deleting the UDFs For the Application Protector REST Approach

Deleting the UDFs is an optional step. Delete the UDFs only to clean up the Databricks cluster. The configurator script is used to delete the UDFs. You must select the required approach and the operation ID to delete. This section explains the process to delete the UDFs using the interactive method of installation.

To delete the UDFs:

Log in to the staging machine.
Navigate to the directory where the installation files are extracted.

To execute the configurator script, run the following command:

./BigDataProtector-Configurator_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

./BigDataProtector-Configurator_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

Press ENTER.
The prompt to enter the operation ID appears.

Creating installation files...
Created installation files.
 
Enter the ID of the operation:

Creating installation files...
Created installation files.

The ID of the operation needs to be one of the following:
----------------------------------------------------------
| ID | Operation                                         |
----------------------------------------------------------
|  1 | Get Application Protector REST's Server IP        |
|  2 | Create Databricks Unity Catalog Batch Python UDFs |
|  3 | Delete Databricks Unity Catalog Batch Python UDFs |
----------------------------------------------------------
Enter the ID of the operation:

To delete the UDFs, type 3.
Press ENTER.
The prompt to enter the Databricks Workspace URL appears.
```
Enter the URL of the Databricks Workspace:
```
Enter the Databricks Workspace URL.
Press ENTER.
The prompt to enter the application ID of the Databricks Service Principal appears.
```
Enter the Application ID of the Databricks Service Principal:
```
Enter the Application ID of the Databricks Service Principal.
Press ENTER.
The prompt to enter the OAuth secret for the Service Principal appears.
```
Enter the OAuth Secret of the Databricks Service Principal:
```
Enter the OAuth secret.
Press ENTER.
The prompt to enter the cluster ID appears.
```
Enter the ID of the Databricks Compute:
```
Enter the Cluster ID.
Note: The Cluster ID can be either for a Standard Compute or Dedicated Compute. For more information about identifying the Cluster ID, refer to Get identifiers for workspace objects.

Press ENTER.
The prompt to enter the name of the schema appears.

Enter the name of the Databricks Unity Catalog Catalog-Schema:

Enter the name of the catalog and the schema in the <catalog_name.schema_name> format.

Press ENTER. The prompt to select the approach appears.

Enter the ID of the approach:

The ID of the approach needs to be one of the following:
-----------------------------------
| ID | Approach                   |
-----------------------------------
|  1 | Application Protector REST |
|  2 | Cloud Protector            |
-----------------------------------
Enter the ID of the approach:

To delete the UDFs using the Application Protector REST approach, type 1.
Press ENTER.
The prompt to specify the Unity Catalog Volume path appears.
```
Enter the path of the Databricks Unity Catalog Volume:
```
Enter the path of the Databricks Unity Catalog Volume.

Press ENTER.
The script deletes the UDFs from the specified location.

Executing specified operation...

Executed specified operation.

Executing specified operation...
Deleting <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_string_fpe Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_string_fpe Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_string_fpe Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_string_fpe Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF.
Deleting /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz file...
Deleted /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.tgz file.
Deleting /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector-Init-Script_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh file...
Deleted /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector-Init-Script_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh file.
Executed specified operation.

Deleting the UDFs For the Cloud Protector Approach

Deleting the UDFs is an optional step. Delete the UDFs only to clean up the Databricks cluster. The configurator script is used to delete the UDFs. Select the required approach and the operation ID to delete the UDFs. This section explains the process to delete the UDFs using the interactive method of installation.

To delete the UDFs:

Log in to the staging machine.
Navigate to the directory where the installation files are extracted.

To execute the configurator script, run the following command:

./BigDataProtector-Configurator_Linux-ALL-64_x86-64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

./BigDataProtector-Configurator_Linux-ALL-64_ARM64_AWS.Databricks-<DBR_version>-64_<BDP_version>.sh

Press ENTER.
The prompt to enter the operation ID appears.

Creating installation files...
Created installation files.
 
Enter the ID of the operation:

Creating installation files...
Created installation files.

The ID of the operation needs to be one of the following:
----------------------------------------------------------
| ID | Operation                                         |
----------------------------------------------------------
|  1 | Get Application Protector REST's Server IP        |
|  2 | Create Databricks Unity Catalog Batch Python UDFs |
|  3 | Delete Databricks Unity Catalog Batch Python UDFs |
----------------------------------------------------------
Enter the ID of the operation:

To delete the UDFs, type 3.
Press ENTER.
The prompt to enter the Databricks Workspace URL appears.
```
Enter the URL of the Databricks Workspace:
```
Enter the Databricks Workspace URL.
Press ENTER.
The prompt to enter the application ID of the Databricks Service Principal appears.
```
Enter the Application ID of the Databricks Service Principal:
```
Enter the Application ID of the Databricks Service Principal.
Press ENTER.
The prompt to enter the OAuth secret for the Service Principal appears.
```
Enter the OAuth Secret of the Databricks Service Principal:
```
Enter the OAuth secret.
Press ENTER.
The prompt to enter the cluster ID appears.
```
Enter the ID of the Databricks Compute:
```
Enter the Cluster ID.
Note: The Cluster ID can be either for a Standard Compute or Dedicated Compute. For more information about identifying the Cluster ID, refer to Get identifiers for workspace objects.

Press ENTER.
The prompt to enter the name of the schema appears.

Enter the name of the Databricks Unity Catalog Catalog-Schema:

Enter the name of the catalog and the schema in the <catalog_name.schema_name> format.

Press ENTER. The prompt to select the approach appears.

Enter the ID of the approach:

The ID of the approach needs to be one of the following:
-----------------------------------
| ID | Approach                   |
-----------------------------------
|  1 | Application Protector REST |
|  2 | Cloud Protector            |
-----------------------------------
Enter the ID of the approach:

To delete the UDFs using the Cloud Protector approach, type 2.

Press ENTER.
The script deletes the UDFs from the specified location.

Executing specified operation...

Executed specified operation.

Executing specified operation...
Deleting <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_protect_string_fpe Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_protect_string_fpe Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_unprotect_string_fpe Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_unprotect_string_fpe Databricks Unity Catalog Batch Python UDF.
Deleting <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF...
Deleted <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF.
Executed specified operation.