Installing the Protector
The process of installing the Big Data Protector involves the process of creating the user-defined functions.
The following combinations will work for a successful execution of the configurator script:
- Databricks Dedicated Compute + Application Protector REST approach
- Databricks Dedicated Compute + Cloud Protector approach
- Databricks Standard Compute + Application Protector REST approach
- Databricks Standard Compute + Cloud Protector approach
- Databricks SQL Warehouse + Cloud Protector approach
The Databricks SQL Warehouse + Application Protector REST approach combination will not work. This is because Protegrity executes a few Python commands on the Databricks Compute to retrieve a listening IP for the Application Protector REST’s Server. When the Databricks Compute is a SQL Warehouse, the Python commands fail to execute. This occurs because the SQL Warehouse supports only SQL commands.
Creating the UDFs for the Application Protector REST Approach
The configurator script creates the UDFs. These Unity Catalog Batch Python UDFs are designed to perform data protection and unprotection operations. Select the required approach and the operation ID to create the UDFs using the Application Protector REST server. This section explains the process to create the UDFs using the interactive method of installation.
To create the UDFs:
Log in to the node having connectivity to ESA/PPC.
Navigate to the directory where the installation files are extracted.
To execute the configurator script, run the following command:
./BigDataProtector-Configurator_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.shPress ENTER.
The prompt to enter the operation ID appears.Creating installation files... Created installation files. The ID of the operation needs to be one of the following: ---------------------------------------------------------- | ID | Operation | ---------------------------------------------------------- | 1 | Get Application Protector REST's Server IP | | 2 | Create Databricks Unity Catalog Batch Python UDFs | | 3 | Delete Databricks Unity Catalog Batch Python UDFs | ---------------------------------------------------------- Enter the ID of the operation:To create the UDFs, type
2.Press ENTER.
The prompt to enter the Databricks Workspace URL appears.Enter the URL of the Databricks Workspace:Enter the Databricks Workspace URL.
Press ENTER.
The prompt to enter the application ID of the Databricks Service Principal appears.Enter the Application ID of the Databricks Service Principal:Enter the Application ID of the Databricks Service Principal.
Press ENTER.
The prompt to enter the OAuth secret for the Service Principal appears.Enter the OAuth Secret of the Databricks Service Principal:Enter the OAuth secret.
Press ENTER.
The prompt to enter the cluster ID appears.Enter the ID of the Databricks Compute:Note: The Cluster ID can be either for Standard Compute or Dedicated Compute. For more information about identifying the Cluster ID, refer to Get identifiers for workspace objects.
Enter the Cluster ID.
Press ENTER.
The prompt to enter the name of the schema appears.Enter the name of the Databricks Unity Catalog Catalog-Schema:Enter the name of the catalog and the schema in the
<catalog_name.schema_name>format.Press ENTER.
The prompt to select the approach appears.The ID of the approach needs to be one of the following: ----------------------------------- | ID | Approach | ----------------------------------- | 1 | Application Protector REST | | 2 | Cloud Protector | ----------------------------------- Enter the ID of the approach:To create the UDFs using the Application Protector REST approach, type
1.Press ENTER.
The prompt to enter the path of the CA Certificate appears.Enter the path of the CA Certificate:Enter the path of the CA Certificate.
Press ENTER.
The prompt to enter the path of the Server Certificate appears.Enter the path of the Server Certificate:Enter the path of the Server Certificate.
Press ENTER.
The prompt to enter the path of the Server key appears.Enter the path of the Server Key:Enter the path of the Server Key.
Press ENTER.
The prompt to enter the name of the key vault appears.Enter the name of the Azure Key Vault:Enter the name of the Azure Key Vault.
Press ENTER.
The prompt to enter the name of the Service Credential appears.Enter the name of the Databricks Unity Catalog Service Credential:Enter the name of the Databricks Unity Catalog Service Credential.
Press ENTER.
The prompt to enter the path of the Unity Catalog Volume appears.Enter the path of the Databricks Unity Catalog Volume:Enter the path of the Databricks Unity Catalog Volume.
Press ENTER.
The script creates the UDFs at the specified location.Executing specified operation... Creating /Volumes/<catalog_name>/<schema_name>/<volume_name>/azure_identity-1.25.3-py3-none-any.whl file... Created /Volumes/<catalog_name>/<schema_name>/<volume_name>/azure_identity-1.25.3-py3-none-any.whl file. Creating /Volumes/<catalog_name>/<schema_name>/<volume_name>/azure_keyvault_secrets-4.11.0-py3-none-any.whl file... Created /Volumes/<catalog_name>/<schema_name>/<volume_name>/azure_keyvault_secrets-4.11.0-py3-none-any.whl file. Creating <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_protect_string_fpe Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_protect_string_fpe Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_unprotect_string_fpe Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_unprotect_string_fpe Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF. Creating /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.tgz file... Created /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.tgz file. Creating /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector-Init-Script_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.sh file... Created /Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector-Init-Script_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.sh file. 1. Create the following environment variables in the Spark section of the Advanced properties of the Databricks Compute: PTY_ESA_IP=PTY_ESA_IP PTY_ESA_PORT=PTY_ESA_PORT Either PTY_ESA_TOKEN=PTY_ESA_TOKEN or PTY_ESA_ADMINISTRATOR_USERNAME=PTY_ESA_ADMINISTRATOR_USERNAME and PTY_ESA_ADMINISTRATOR_PASSWORD=PTY_ESA_ADMINISTRATOR_PASSWORD PTY_AUDIT_STORE_IP_PORT=PTY_AUDIT_STORE_IP_PORT PTY_PROTECTOR_CONFIGURATION=PTY_PROTECTOR_CONFIGURATION PTY_ESA_CA_CERTIFICATE=PTY_ESA_CA_CERTIFICATE PTY_ESA_FQDN=PTY_ESA_FQDN 2. Attach "/Volumes/<catalog_name>/<schema_name>/<volume_name>/BigDataProtector-Init-Script_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.sh" as an Init Script to the Databricks Compute. 3. Restart the Databricks Compute. Executed specified operation.
Creating the UDFs for the Cloud Protector Approach
The configurator script creates the UDFs. These Unity Catalog Batch Python UDFs can perform data protection and unprotection operations. Select the required approach and the operation ID to create the UDFs using the Cloud Protector. This section explains the process to create the UDFs using the interactive method of installation.
To create the UDFs:
Log in to the node having connectivity to ESA/PPC.
Navigate to the directory where the installation files are extracted.
To execute the configurator script, run the following command:
./BigDataProtector-Configurator_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.shPress ENTER.
The prompt to enter the operation ID appears.Creating installation files... Created installation files. The ID of the operation needs to be one of the following: ---------------------------------------------------------- | ID | Operation | ---------------------------------------------------------- | 1 | Get Application Protector REST's Server IP | | 2 | Create Databricks Unity Catalog Batch Python UDFs | | 3 | Delete Databricks Unity Catalog Batch Python UDFs | ---------------------------------------------------------- Enter the ID of the operation:To create the UDFs, type
2.Press ENTER.
The prompt to enter the Databricks Workspace URL appears.Enter the URL of the Databricks Workspace:Enter the Databricks Workspace URL.
Press ENTER.
The prompt to enter the application ID of the Databricks Service Principal appears.Enter the Application ID of the Databricks Service Principal:Enter the Application ID of the Databricks Service Principal.
Press ENTER.
The prompt to enter the OAuth secret for the Service Principal appears.Enter the OAuth Secret of the Databricks Service Principal:Enter the OAuth secret.
Press ENTER.
The prompt to enter the cluster ID appears.Enter the ID of the Databricks Compute:Note: The Cluster ID can be either for SQL Warehouse, Standard Compute or Dedicated Compute. For more information about identifying the Cluster ID, refer to Get identifiers for workspace objects.
Enter the Cluster ID.
Press ENTER.
The prompt to enter the name of the schema appears.Enter the name of the Databricks Unity Catalog Catalog-Schema:Enter the name of the catalog and the schema in the
<catalog_name.schema_name>format.Press ENTER.
The prompt to select the approach appears.The ID of the approach needs to be one of the following: ----------------------------------- | ID | Approach | ----------------------------------- | 1 | Application Protector REST | | 2 | Cloud Protector | ----------------------------------- Enter the ID of the approach:To create the UDFs using the Cloud Protector approach, type
2.Press ENTER.
The prompt to enter the protection endpoint appears.Enter the URL of the Azure Function App's Protect Function:Enter the protection endpoint.
Press ENTER.
The prompt to enter the unprotection endpoint appears.Enter the URL of the Azure Function App's Unprotect Function:Enter the unprotection endpoint.
Press ENTER.
The prompt to enter the name of the key vault appears.Enter the name of the Azure Key Vault:Enter the name of the Azure Key Vault.
Press ENTER.
The prompt to enter the name of the Service Credential appears.Enter the name of the Databricks Unity Catalog Service Credential:Enter the name of the Databricks Unity Catalog Service Credential.
Press ENTER.
The prompt to enter the path of the Unity Catalog volume appears.Enter the path of the Databricks Unity Catalog Volume:Enter the location of the Unity Catalog volume.
Press ENTER.
The script creates the UDFs at the specified location.Executing specified operation... Creating /Volumes/<catalog_name>/<schema_name>/<volume_name>/azure_identity-1.25.3-py3-none-any.whl file... Created /Volumes/<catalog_name>/<schema_name>/<volume_name>/azure_identity-1.25.3-py3-none-any.whl file. Creating /Volumes/<catalog_name>/<schema_name>/<volume_name>/azure_keyvault_secrets-4.11.0-py3-none-any.whl file... Created /Volumes/<catalog_name>/<schema_name>/<volume_name>/azure_keyvault_secrets-4.11.0-py3-none-any.whl file. Creating <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_decrypt_string Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_encrypt_string Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_get_version Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_get_version_extended Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_protect_binary Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_protect_date Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_protect_int Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_protect_smallint Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_protect_string Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_unprotect_binary Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_unprotect_date Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_unprotect_int Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_unprotect_smallint Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_unprotect_string Databricks Unity Catalog Batch Python UDF. Creating <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF... Created <catalog_name>.<schema_name>.pty_who_am_i Databricks Unity Catalog Batch Python UDF. Executed specified operation.
Feedback
Was this page helpful?