Preparing the Environment on

Extracting the Installation Package

Mon, 01 Jan 0001 00:00:00 +0000

Extract the contents of the installation package to access the configurator script. This script generates the required files to install the Big Data Protector.

To extract the files from the installation package:

Log in to the Linux machine that has connectivity to ESA/PPC.
Download the Big Data Protector package BigDataProtector_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.tgz to any local directory.

To extract the files from the installation packaage, run the following command:

tar -xvf BigDataProtector_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.tgz

Press ENTER.
The command extracts the installation package and the GPG signature files.

Working with the Configurator Script

Mon, 01 Jan 0001 00:00:00 +0000

Viewing the Help Options

The configurator script performs the following tasks:

Generate the IP address for the Application Protector REST server.
Create the UDFs.
Delete the UDFs.

The configurator script provides the --help option to understand the options and the arguments to be provided.

To understand the options and the arguments for the configurator script:

To view the options and the arguments, run the following command:

./BigDataProtector-Configurator_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.sh --help

Press ENTER.
The command displays all the options and the arguments required to execute the configurator script.

This script utilizes one of the following approaches:
-----------------------------------
| ID | Approach |
-----------------------------------
| 1 | Application Protector REST |
| 2 | Cloud Protector |
-----------------------------------

###########################################################################################

If you want to use Application Protector REST approach, following operations are supported:
----------------------------------------------------------
| ID | Operation |
----------------------------------------------------------
| 1 | Get Application Protector REST's Server IP |
| 2 | Create Databricks Unity Catalog Batch Python UDFs |
| 3 | Delete Databricks Unity Catalog Batch Python UDFs |
----------------------------------------------------------

To Get Application Protector REST's Server IP, following inputs are required:
Using interactive prompts:
1. The ID of the operation as "1"
2. The URL of the Databricks Workspace
3. The Application ID of the Databricks Service Principal
4. The OAuth Secret of the Databricks Service Principal
5. The ID of the Databricks Compute

Using command-line arguments:
--operation_id "1"
--databricks_workspace_url "databricks_workspace_url"
--databricks_service_principal_application_id "databricks_service_principal_application_id"
--databricks_service_principal_oauth_secret "databricks_service_principal_oauth_secret"
--databricks_compute_id "databricks_compute_id"

Using .cfg file (pass the path of the .cfg file to this script as a command-line argument):
operation_id = "1"
databricks_workspace_url = "databricks_workspace_url"
databricks_service_principal_application_id = "databricks_service_principal_application_id"
databricks_service_principal_oauth_secret = "databricks_service_principal_oauth_secret"
databricks_compute_id = "databricks_compute_id"

To Create Databricks Unity Catalog Batch Python UDFs, following inputs are required:
Using interactive prompts:
1. The ID of the operation as "2"
2. The URL of the Databricks Workspace
3. The Application ID of the Databricks Service Principal
4. The OAuth Secret of the Databricks Service Principal
5. The ID of the Databricks Compute
6. The name of the Databricks Unity Catalog Catalog-Schema
7. The ID of the approach as "1"
8. The path of the CA Certificate
9. The path of the Server Certificate
10. The path of the Server Key
11. The name of the Azure Key Vault
12. The name of the Databricks Unity Catalog Service Credential
13. The path of the Databricks Unity Catalog Volume

Using command-line arguments:
--operation_id "2"
--databricks_workspace_url "databricks_workspace_url"
--databricks_service_principal_application_id "databricks_service_principal_application_id"
--databricks_service_principal_oauth_secret "databricks_service_principal_oauth_secret"
--databricks_compute_id "databricks_compute_id"
--databricks_unity_catalog_catalog_schema_name "databricks_unity_catalog_catalog_schema_name"
--approach_id "1"
--ca_certificate_path "ca_certificate_path"
--server_certificate_path "server_certificate_path"
--server_key_path "server_key_path"
--azure_key_vault_name "azure_key_vault_name"
--databricks_unity_catalog_service_credential_name "databricks_unity_catalog_service_credential_name"
--databricks_unity_catalog_volume_path "databricks_unity_catalog_volume_path"

Using .cfg file (pass the path of the .cfg file to this script as a command-line argument):
operation_id = "2"
databricks_workspace_url = "databricks_workspace_url"
databricks_service_principal_application_id = "databricks_service_principal_application_id"
databricks_service_principal_oauth_secret = "databricks_service_principal_oauth_secret"
databricks_compute_id = "databricks_compute_id"
databricks_unity_catalog_catalog_schema_name = "databricks_unity_catalog_catalog_schema_name"
approach_id = "1"
ca_certificate_path = "ca_certificate_path"
server_certificate_path = "server_certificate_path"
server_key_path = "server_key_path"
azure_key_vault_name = "azure_key_vault_name"
databricks_unity_catalog_service_credential_name = "databricks_unity_catalog_service_credential_name"
databricks_unity_catalog_volume_path = "databricks_unity_catalog_volume_path"

To Delete Databricks Unity Catalog Batch Python UDFs, following inputs are required:
Using interactive prompts:
1. The ID of the operation as "3"
2. The URL of the Databricks Workspace
3. The Application ID of the Databricks Service Principal
4. The OAuth Secret of the Databricks Service Principal
5. The ID of the Databricks Compute
6. The name of the Databricks Unity Catalog Catalog-Schema
7. The ID of the approach as "1"
8. The path of the Databricks Unity Catalog Volume

Using command-line arguments:
--operation_id "3"
--databricks_workspace_url "databricks_workspace_url"
--databricks_service_principal_application_id "databricks_service_principal_application_id"
--databricks_service_principal_oauth_secret "databricks_service_principal_oauth_secret"
--databricks_compute_id "databricks_compute_id"
--databricks_unity_catalog_catalog_schema_name "databricks_unity_catalog_catalog_schema_name"
--approach_id "1"
--databricks_unity_catalog_volume_path "databricks_unity_catalog_volume_path"

Using .cfg file (pass the path of the .cfg file to this script as a command-line argument):
operation_id = "3"
databricks_workspace_url = "databricks_workspace_url"
databricks_service_principal_application_id = "databricks_service_principal_application_id"
databricks_service_principal_oauth_secret = "databricks_service_principal_oauth_secret"
databricks_compute_id = "databricks_compute_id"
databricks_unity_catalog_catalog_schema_name = "databricks_unity_catalog_catalog_schema_name"
approach_id = "1"
databricks_unity_catalog_volume_path = "databricks_unity_catalog_volume_path"

################################################################################

If you want to use Cloud Protector approach, following operations are supported:
----------------------------------------------------------
| ID | Operation |
----------------------------------------------------------
| 1 | Create Databricks Unity Catalog Batch Python UDFs |
| 2 | Delete Databricks Unity Catalog Batch Python UDFs |
----------------------------------------------------------

To Create Databricks Unity Catalog Batch Python UDFs, following inputs are required:
Using interactive prompts:
1. The ID of the operation as "2"
2. The URL of the Databricks Workspace
3. The Application ID of the Databricks Service Principal
4. The OAuth Secret of the Databricks Service Principal
5. The ID of the Databricks Compute
6. The name of the Databricks Unity Catalog Catalog-Schema
7. The ID of the approach as "2"
8. The URL of the Azure Function App's Protect Function
9. The URL of the Azure Function App's Unprotect Function
10. The name of the Azure Key Vault
11. The name of the Databricks Unity Catalog Service Credential
12. The path of the Databricks Unity Catalog Volume

Using command-line arguments:
--operation_id "2"
--databricks_workspace_url "databricks_workspace_url"
--databricks_service_principal_application_id "databricks_service_principal_application_id"
--databricks_service_principal_oauth_secret "databricks_service_principal_oauth_secret"
--databricks_compute_id "databricks_compute_id"
--databricks_unity_catalog_catalog_schema_name "databricks_unity_catalog_catalog_schema_name"
--approach_id "2"
--azure_function_app_protect_function_url "azure_function_app_protect_function_url"
--azure_function_app_unprotect_function_url "azure_function_app_unprotect_function_url"
--azure_key_vault_name "azure_key_vault_name"
--databricks_unity_catalog_service_credential_name "databricks_unity_catalog_service_credential_name"
--databricks_unity_catalog_volume_path "databricks_unity_catalog_volume_path"

Using .cfg file (pass the path of the .cfg file to this script as a command-line argument):
operation_id = "2"
databricks_workspace_url = "databricks_workspace_url"
databricks_service_principal_application_id = "databricks_service_principal_application_id"
databricks_service_principal_oauth_secret = "databricks_service_principal_oauth_secret"
databricks_compute_id = "databricks_compute_id"
databricks_unity_catalog_catalog_schema_name = "databricks_unity_catalog_catalog_schema_name"
approach_id = "2"
azure_function_app_protect_function_url = "azure_function_app_protect_function_url"
azure_function_app_unprotect_function_url = "azure_function_app_unprotect_function_url"
azure_key_vault_name = "azure_key_vault_name"
databricks_unity_catalog_service_credential_name = "databricks_unity_catalog_service_credential_name"
databricks_unity_catalog_volume_path = "databricks_unity_catalog_volume_path"

To Delete Databricks Unity Catalog Batch Python UDFs, following inputs are required:
Using interactive prompts:
1. The ID of the operation as "3"
2. The URL of the Databricks Workspace
3. The Application ID of the Databricks Service Principal
4. The OAuth Secret of the Databricks Service Principal
5. The ID of the Databricks Compute
6. The name of the Databricks Unity Catalog Catalog-Schema
7. The ID of the approach as "2"
8. The path of the Databricks Unity Catalog Volume

Using command-line arguments:
--operation_id "3"
--databricks_workspace_url "databricks_workspace_url"
--databricks_service_principal_application_id "databricks_service_principal_application_id"
--databricks_service_principal_oauth_secret "databricks_service_principal_oauth_secret"
--databricks_compute_id "databricks_compute_id"
--databricks_unity_catalog_catalog_schema_name "databricks_unity_catalog_catalog_schema_name"
--approach_id "2"
--databricks_unity_catalog_volume_path "databricks_unity_catalog_volume_path"

Using .cfg file (pass the path of the .cfg file to this script as a command-line argument):
operation_id = "3"
databricks_workspace_url = "databricks_workspace_url"
databricks_service_principal_application_id = "databricks_service_principal_application_id"
databricks_service_principal_oauth_secret = "databricks_service_principal_oauth_secret"
databricks_compute_id = "databricks_compute_id"
databricks_unity_catalog_catalog_schema_name = "databricks_unity_catalog_catalog_schema_name"
approach_id = "2"
databricks_unity_catalog_volume_path = "databricks_unity_catalog_volume_path"

Retrieving the IP Address

Mon, 01 Jan 0001 00:00:00 +0000

The IP address for the Application Protector REST approach is required to generate the certificates. The certificates must be created using the retrieved IP address. These certificates will be used to establish a mutual trust between the Unity Catalog Batch Python UDFs and the Application Protector REST Server.

Note: The instructions mentioned in the section apply only to the Application Protector REST approach.

To execute the configurator script, run the following command:

./BigDataProtector-Configurator_Linux-ALL-64_x86-64_Azure.Databricks-<DBR_version>-64_<BDP_version>.sh

Press ENTER.
The prompt to enter the operation ID appears.

Creating installation files...
Created installation files.

The ID of the operation needs to be one of the following:
----------------------------------------------------------
| ID | Operation |
----------------------------------------------------------
| 1 | Get Application Protector REST's Server IP |
| 2 | Create Databricks Unity Catalog Batch Python UDFs |
| 3 | Delete Databricks Unity Catalog Batch Python UDFs |
----------------------------------------------------------
Enter the ID of the operation:

To retrieve the IP address of the Application Protector REST server, type 1.
Press ENTER.
The prompt to enter the Databricks Workspace URL appears.
```
Enter the URL of the Databricks Workspace:
```
Enter the Databricks Workspace URL.
Press ENTER.
The prompt to enter the application ID of the Databricks Service Principal appears.
```
Enter the Application ID of the Databricks Service Principal:
```
Enter the Application ID of the Databricks Service Principal.
Press ENTER.
The prompt to enter the OAuth secret for the Service Principal appears.
```
Enter the OAuth Secret of the Databricks Service Principal:
```
Enter the OAuth secret.
Press ENTER.
The prompt to enter the cluster ID appears.
```
Enter the ID of the Databricks Compute:
```
Enter the Cluster ID.

Note: The Cluster ID can be either for Standard Compute or Dedicated Compute. For more information about identifying the Cluster ID, refer to Get identifiers for workspace objects.

Uploading the Secrets to the Azure Key Vault

Mon, 01 Jan 0001 00:00:00 +0000

The CA and the Client certificates are important entities in the mutual trust process. These certificates determine the authentication and authorization for the Application Protector REST server. As a result, it is critical to store these certificates in a secured location. Therefore, upload the certificates to the Azure Key Vault in Azure where they will be stored as secrets.

Note: The instructions mentioned in the section apply only to the Application Protector REST approach.