This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

CDP AWS DataHub

Big Data Protector on CDP Public Cloud AWS

1 - Understanding the architecture

The architecture for the CDP-AWS-DataHub distribution of the Big Data Protector is depicted in the image below.

ComponentDescription
RPAgentIs a daemon running on each node that downloads the package from ESA over a TLS channel using the installed Certificates.
Log ForwarderIs a daemon running on each node that routes the audit logs and application logs to ESA/Audit Store.
config.iniIs a file on each node containing the set of configuration parameters to modify the protector behavior.
BDP LayerContains the Big Data Protector UDFs and APIs executing in CDP service processes.
JcoreLiteIs the JNI library that provides a Java API layer to the Core libraries.
CoreIs the set of various libraries that provide the Protegrity Core functionality.

2 - System Requirements

Ensure that the following prerequisites are met, before installing the Big Data Protector from the Cloudera Manager:

  • The Hadoop cluster is installed, configured, and running CDP-AWS-Datahub (Cloudera Runtime 7.3.1).
  • The ESA appliance, version 10.0.x or v10.1.x, is installed, configured, and running.
  • The ports that are configured on ESA and the nodes in the cluster, which will run the Big Data Protector, are listed in the following table:
Destination PortProtocolSourceDestinationDescription
8443TLSRPAgent on the Big Data Protector cluster nodeESAThe RPAgent communicates with ESA through port
8443 to download a policy.
9200TLSLog Forwarder on the Big Data Protector Cluster nodeProtegrity Audit
Store appliance		The Log Forwarder sends all the logs to
the Protegrity Audit Appliance through port 9200.
15780TCPProtector on the Big Data Protector
cluster node		Log Forwarder
on the Big Data
Protector cluster
node		The Big Data Protector writes Audit Logs to
localhost through port 15780. The Application
Logs are also written to localhost through
port 15780. The Log Forwarder reads the logs from that
socket.
  • The user, installing the Big Data Protector, has the requisite permissions to perform the following tasks:
    • Copy the Big Data Protector parcels and CSDs to the Cloudera Manager repository directories
    • Restart the Cloudera SCM Server
  • If you are installing the Big Data Protector on a cluster, then ensure that it is installed on all the nodes in the cluster.
  • The group ptyitusr and the user ptyitusr, responsible to manage the Big Data Protector-related services are managed by Cloudera Manager. The user and group are unavailable on the cluster nodes.

This build supports both Spark 2 and Spark 3 on the cluster using a single pepspark jar.
For more information about installing Spark3 on CDP AWS DataHub cluster, refer https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/cds-3/topics/spark-install-spark-3-parcel.html.

3 - Preparing the Environment

3.1 - Extracting the package

Extract the contents of the installation package to access the configurator script. This script generates the required files to install the Big Data Protector.

To extract the files from the installation package:

  1. Log in to the Linux machine that has connectivity to ESA.

  2. Download the Big Data Protector package BigDataProtector_Linux-ALL-64_x86-64_AWS.Generic.CDP-Datahub-7.3-64_<BDP_version>.tgz to any local directory.

  3. To extract the files from the installation pacakage, run the following command:

    tar -xvf BigDataProtector_Linux-ALL-64_x86-64_AWS.Generic.CDP-Datahub-7.3-64_<BDP_version>.tgz

  4. Press ENTER. The command extracts the installation package and the GPG signature files.

    BigDataProtector_Linux-ALL-64_x86-64_AWS.Generic.CDP-Datahub-7.3-64_<BDP_version>.tgz
signatures/
signatures/BigDataProtector_Linux-ALL-64_x86-64_AWS.Generic.CDP-Datahub-7.3-64_<BDP_version>.tgz_10.0.sig

    Verify the authenticity of the build using the signatures folder. For more information, refer Verification of Signed Protector Build.

  5. To extract the configurator script, run the following command:

    tar -xvf BigDataProtector_Linux-ALL-64_x86-64_AWS.Generic.CDP-Datahub-7.3-64_<BDP_version>.tgz

  6. Press ENTER. The command extracts the configurator script.

    BDPConfigurator_CDP-AWS-DataHub-7.3_<BDP_version>.sh

3.2 - Executing the configurator script

Execute the Big Data Protector configurator script to:

  1. Download the certificates from ESA.
  2. Create the installation files for the Big Data Protector.

To execute the configurator script:

  1. Log in to the staging machine that has connectivity to ESA.

  2. To execute the configurator script, run the following command:

    ./BDPConfigurator_CDP-AWS-DataHub-7.3_<BDP_Version>.sh

  3. Press ENTER.

    The prompt to continue the configuration of Big Data Protector appears.

    *******************************************************************************
         Welcome to the Big Data Protector Configurator Wizard
*******************************************************************************
This will setup the Big Data Protector Installation Files for CDP AWS Data Hub.

Do you want to continue? [yes or no]:

  4. To continue, type yes.

  5. Press ENTER. The prompt to select the type of installation files appears.

    Big Data Protector Configurator started...
Unpacking...
Extracting files...


Select the type of Installation files you want to generate.
[ 1: Create All ]      : Creates entire Big Data Protector CSDs, Parcels, Recipes and other files.
[ 2: Update PTY_CERT ] : Creates new PTY_CERT parcel with an incremented patch version.
                        Use this if you have updated the ESA certificates.
[ 3: Update PTY_LOGFORWARDER_CONF ]
                    : Creates new PTY_LOGFORWARDER_CONF parcel with an incremented patch version.
                        Use this if you want to set Custom LogForwarder configuration files to
                        forward logs to an External Audit Store.

[ 1, 2 or 3 ]:

    Note: From v10.0.0, the PTY_FLUENTBIT_CONF parcel is renamed to PTY_LOGFORWARDER_CONF.

  6. To create the Big Data Protector parcels and CSDs, type 1.

  7. To update the PTY_CERT parcels with an incremented patch version, type 2.

  8. To update the PTY_LOGFORWARDER_CONF parcel with an incremented patch version, type 3.

  9. Press ENTER. The prompt to select the operating system for the Cloudera Manager parcel appears.

    Select the OS version for Cloudera Manager Parcel.
This will be used as the OS Distro suffix in the Parcel name.

[ 1: el7 ]    :  RHEL 7 and clones (CentOS, Scientific Linux, etc)
[ 2: el8 ]    :  RHEL 8 and clones (CentOS, Scientific Linux, etc)
[ 3: el9 ]    :  RHEL 9 and clones (CentOS, Scientific Linux, etc)
[ 4: sles12 ] :  SuSE Linux Enterprise Server 12.x
[ 5: sles15 ] :  SuSE Linux Enterprise Server 15.x

Enter the no.:

  10. Depending on the requirements, type 1, 2, 3, 4 or 5 to select the operating system version for the Big Data Protector parcels.

  11. Press ENTER. The prompt to enter the S3 URI to upload the installation file appears.

    Enter the S3 URI where the BDP Installation files are to be uploaded.

(E.g. s3://examplebucket/folder):

  12. Enter the location to upload the installation files.

  13. Press ENTER. The prompt to select the upload method appears.

    Choose one option among the following for BDP Installation files:
[ 1 ] : Upload files to 's3://<bucket_name>/<directory_name>/' S3 URI.
[ 2 ] : Generate files locally to current working directory. (You would have to manually upload the files to the specified S3 URI)

[ 1 or 2 ]:

  14. To upload the files, type 1.

  15. Press ENTER. The prompt to select the authentication option appears.

    Choose the Type of AWS Access Keys from the following options:
[ 1 ] : IAM User Access Keys (Permanent access key id & secret access key)
[ 2 ] : Temporary Security Credentials (Temporary access key id, secret access key & session token)

[ 1 or 2 ]:

  16. Depending upon the authentication option, the script will prompt for the following inputs:

    OptionDescription
    1Prompts to enter the following permanent IAM user access keys:
    AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY
    2Prompts to enter the following temporary security credentials:
    AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY
    AWS_SESSION_TOKEN

  17. Enter the required credentials.

  18. Press ENTER. The prompt to enter ESA hostname or IP address appears.

    Enter ESA Hostname or IP Address:

  19. Enter ESA IP address.

  20. Press ENTER. The prompt to enter ESA listening port appears.

    Enter ESA host listening port [8443]:

  21. Enter the listening port number.

  22. Press ENTER.

    The prompt to enter ESA JSON Web Token appears.

    If you have an existing ESA JSON Web Token (JWT) with Export Certificates role, enter it otherwise enter 'no':

    Note: The script silently reads the user input. Therefore, the user will be unable to see the entered JWT or no.

  23. Enter the JWT token.

    a. If you do not have an existing ESA JSON Web Token (JWT), type no.

    b. Press ENTER.
    The prompt to enter the user name with Export Certificates permission appears.

    ```
JWT was not provided. Script will now prompt for ESA username and password.
Enter ESA Username with Export Certificates role:
```

    c. Enter the username that has permissions to export the certificates.

    d. Press ENTER.
    The prompt to enter the password appears. Enter the password for username <user_name>: e. Enter the password.

    f. Press ENTER.
    The script retrieves the JWT from ESA, validates it, and the prompt to package custom log forwarder configuration appears.

     Fetching JWT from ESA....

 Fetching Certificates from ESA....

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 100 11264  100 11264    0     0   202k      0 --:--:-- --:--:-- --:--:--  203k

 -------------------------------------------------------------------------------


 Do you want to package any custom LogForwarder configuration files for External Audit Store?
 [ yes ] : Create a PTY_LOGFORWARDER_CONF parcel containing configuration files to be used with External Audit Store.
 [ no ]  : Skip this step.

 [ yes or no ]:

 ```

  24. To package the Log Forwarder configuration file(s) for an external Audit Store, type yes.

  25. Press ENTER.
    The prompt to enter the local directory path containing the Log Forwarder configuration files appears.

    Do you want to package any custom LogForwarder configuration files for External Audit Store?
[ yes ] : Create a PTY_LOGFORWARDER_CONF parcel containing configuration files to be used with External Audit Store.
[ no ]  : Skip this step.

[ yes or no ]: yes

Creation of PTY_LOGFORWARDER_CONF parcel is enabled.

Enter the local directory path on this machine that stores the LogForwarder configuration files for External Audit Store:

    Note: The PTY_LOGFORWARDER_CONF parcel is used to package any custom Log Forwarder configuration files that the user provides and can be distributed across the CDP nodes through the Cloudera Manager. Ensure that you name the custom Log Forwarder configuration files for the external Audit Store with the .conf extension.

  26. Enter the local directory path that contains the Log Forwarder configuration files.

  27. Press ENTER.
    The script generates the installation files and uploads them to the specified S3 URI.

    Generating Installation files...

****************************************************************************************************************************************

Retrieving the S3 bucket's AWS Region via AWS S3 REST API...
Successfully retrieved S3 bucket's AWS region: <region_name>

Started uploading the Installation files to S3 bucket using REST API.

Uploading BDP_PEP-<BDP_version>.jar...
-> File uploaded to s3://<bucket_name>/<directory_name>/CSDandParcels/BDP_PEP-<BDP_version>.jar

Uploading PTY_BDP-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel...
-> File uploaded to s3://<bucket_name>/<directory_name>/CSDandParcels/PTY_BDP-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel

Uploading PTY_BDP-<BDP_Version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel.sha...
-> File uploaded to s3://<bucket_name>/<directory_name>/CSDandParcels/PTY_BDP-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel.sha

Uploading PTY_CERT-<BDP_Version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel...
-> File uploaded to s3://<bucket_name>/<directory_name>/CSDandParcels/PTY_CERT-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel

Uploading PTY_CERT-<BDP_Version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel.sha...
-> File uploaded to s3://<bucket_name>/<directory_name>/CSDandParcels/PTY_CERT-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel.sha

Uploading pepimpala4_0_RHEL.so...
-> File uploaded to s3://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so

Uploading createobjects.sql...
-> File uploaded to s3://<bucket_name>/<directory_name>/pepimpala/sqlscripts/createobjects.sql

Uploading dropobjects.sql...
-> File uploaded to s3://<bucket_name>/<directory_name>/pepimpala/sqlscripts/dropobjects.sql

Uploading BDP_Pre-Service-Deployment_Recipe_<BDP_version>.sh...
-> File uploaded to s3://<bucket_name>/<directory_name>/RecipesAndTemplates/BDP_Pre-Service-Deployment_Recipe_<BDP_version>.sh

Uploading BDP_Post-Cloudera-Manager-Start_Recipe_<BDP_version>.sh...
-> File uploaded to s3://<bucket_name>/<directory_name>/RecipesAndTemplates/BDP_Post-Cloudera-Manager-Start_Recipe_<BDP_version>.sh

Uploading custom_properties_template.json...
-> File uploaded to s3://<bucket_name>/<directory_name>/RecipesAndTemplates/custom_properties_template.json

Uploading guide_to_create_cluster_template_with_bdp.txt...
-> File uploaded to s3://<bucket_name>/<directory_name>/RecipesAndTemplates/guide_to_create_cluster_template_with_bdp.txt

Uploading PTY_LOGFORWARDER_CONF-<BDP_Version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel...
-> File uploaded to s3://<bucket_name>/<directory_name>/CSDandParcels/PTY_LOGFORWARDER_CONF-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel

Uploading PTY_LOGFORWARDER_CONF-<BDP_Version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel.sha...
-> File uploaded to s3://<bucket_name>/<directory_name>/CSDandParcels/PTY_LOGFORWARDER_CONF-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel.sha

Successfully uploaded Installation files under ./Installation_Files to S3 URI: s3://<bucket_name>/directory_name

****************************************************************************************************************************************

* The BDP CSD & Parcels (and checksums) are generated locally in ./Installation_Files/CSDandParcels/ directory.

* BDP Recipes, Custom Properties and Custom Cluster Template creation guide are generated locally in ./Installation_Files/RecipesAndTemplates/ directory.
-> Follow the guide to create a custom Cluster Template and use it along with the 2 Recipes and Custom Properties on CDP AWS.

* The pepimpala .so library is generated locally in ./Installation_Files/pepimpala/ directory.

* The pepimpala SQL scripts to create and drop Impala UDFs is generated locally in ./Installation_Files/pepimpala/sqlscripts/ directory.
-> Use these scripts as reference to register Protegrity Impala UDFs if you plan to use the Impala Service.
Note: The location clause in the Create Function query points to the S3 URI of the pepimpala*.so

****************************************************************************************************************************************

Successfully configured the Big Data Protector Installaton files for CDP AWS DataHub.

    If you select the option to generate the installation files locally, the configurator script creates the files in a local directory.

    Generating Installation files...

****************************************************************************************************************************************

* The BDP CSD & Parcels (and checksums) are generated locally in ./Installation_Files/CSDandParcels/ directory.
-> Manually upload them to 's3://<bucket_name>/<directory_name>/CSDandParcels/' [This step is Required]

* BDP Recipes, Custom Properties and Custom Cluster Template creation guide are generated locally in ./Installation_Files/RecipesAndTemplates/ directory.
-> Follow the guide to create a custom Cluster Template and use it along with the 2 Recipes and Custom Properties on CDP AWS.
-> Manually upload them to 's3://<bucket_name>/<directory_name>/RecipesAndTemplates/' [This step is Optional]

* You can use the ./Installation_Files/set_unset_bdp_config.sh helper script for setting/unsetting BDP configs in Cloudera Manager.

* The pepimpala .so library is generated locally in ./Installation_Files/pepimpala/ directory.
-> Manually upload the library to 's3://<bucket_name>/<directory_name>/pepimpala/' [This step is Required]

* The pepimpala SQL scripts to create and drop Impala UDFs is generated locally in ./Installation_Files/pepimpala/sqlscripts/ directory.
-> Use these scripts as reference to register Protegrity Impala UDFs if you plan to use the Impala Service.
Note: The location clause in the Create Function query points to the S3 URI of the pepimpala*.so
-> Manually upload them to 's3://<bucket_name>/<directory_name>/pepimpala/sqlscripts/' [This step is Optional]

****************************************************************************************************************************************

Successfully configured the Big Data Protector Installaton files for CDP AWS DataHub.

3.3 - Registering the Recipe Scripts

The Big Data Protector provides the following recipe scripts:

  • BDP_Pre-Service-Deployment_Recipe_<BDP_version>.sh - downloads the Big Data Protector CSD and parcels from the S3 bucket to the Cloudera Manager local CSD and Parcel repository before the Cloudera Manager server starts.
  • BDP_Post-Cloudera-Manager-Start_Recipe_<BDP_version>.sh - runs after the Cloudera Manager Server starts. It creates and executes the secondary scripts as background processes for each available Protegrity Parcel. The background processes will check when the Cloudera Manager Server API endpoint would be open and then sends the requests to distribute and activate the PTY_BDP, PTY_CERT, and PTY_LOGFORWARDER_CONF (if present) parcels.

By default, the execution logs for the Recipe scripts can be found in the /var/log/recipes/ directory. The execution logs of the secondary scripts (executing in the background) can be found in the /tmp/protegrity/ directory.

To register each recipe script:

  1. Log in to the Cloudera Management Console.
  2. Navigate to Shared Resources > Recipes. The Recipes page appears.
  3. Click Register Recipe. The Register Recipe page appears.
  4. In the Name box, enter a name to identify the recipe script.
  5. From the Type list, select the required recipe script type. Note: You must register both the pre-service-deployment-recipe and post-cloudera-manager-start recipe scripts.
  6. Enter the optional recipe description.
  7. Select the File option to upload the file, to the Cloudera manager UI, that contains the recipe script.
    Note: To upload the pre-service-deployment-recipe script, select the BDP_Pre-Service-Deployment_Recipe_<BDP_version>.sh script. To upload the post-cloudera-manager-start recipe script, select the BDP_Post-Cloudera-Manager-Start_Recipe_<BDP_version>.sh script.
  8. Click Register.

3.4 - Creating and Registering the Custom Cluster Template

Create and register the custom cluster template to add BDP_PEP service and the required service configurations to the Data Hub cluster.

To Create the Custom Cluster Template with the BDP PEP service:

  1. Log in to the Cloudera Management Console.
  2. Navigate to Shared Resources > Cluster Templates. The Cluster Templates page appears.
  3. Click Create Cluster Template. The Register Cluster Template page appears.
  4. In the Name box, enter a name for the template.
  5. In the Description box, enter an explanation for the template.
  6. Open any text editor.
  7. Copy the JSON content of the required cluster template that you want to use. For example, Data Engineering or Data Mart.
  8. On the cluster template JSON, search for the services key, whose value is a JSON array of JSON objects.
  9. Add the following JSON objects to that array of services:
    {
  "refName": "bdp_pep",
  "serviceType": "BDP_PEP",
  "displayName": "BDP PEP",
  "roleConfigGroups": [
    {
      "refName": "bdp_pep-PTY_RPAGENT-BASE",
      "roleType": "PTY_RPAGENT",
      "base": true,
      "configs": [
        {
          "name": "rpa_sync_host",
          "value": "{{{rpa_sync_host}}}"
        }
      ]
    },
    {
      "refName": "bdp_pep-PTY_LOGFORWARDER-BASE",
      "roleType": "PTY_LOGFORWARDER",
      "base": true,
      "configs": [
        {
          "name": "auditstore_ip_port_list",
          "value": "{{{auditstore_ip_port_list}}}"
        },
        {
          "name": "auditstore_type",
          "value": "{{{auditstore_type}}}"
        },
        {
          "name": "enable_applog_file",
          "value": true
        }
      ]
    }
  ]
}

    The service object is position-independent within the array and can be placed at the beginning or end of the array.
    Adding the BDP_PEP service to the array of services will ensure that the service is added to the Data Hub cluster during the cluster creation, when the Cloudera Manager imports the cluster template.
    Ensure that the values, such as the {{{esa_address}}} should be written as it is (called Mustache template “{{{…}}}”). The actual value is set by adding the custom properties during the creation of the CDP data hub cluster. For more information about the format of the custom properties, check the custom_properties_template.json file in the S3 bucket containing installation files of Big Data Protector.

  10. After adding the bdp_pep service object, search for the hostTemplates key in the cluster template, whose value is an array of hostTemplate objects for master, worker, and compute nodes etc.
  11. For each hostTemplate in that array, search for the key roleConfigGroupsRefNames, which has a value of array of strings.
  12. Add the bdp_pep-PTY_RPAGENT-BASE and the bdp_pep-PTY_LOGFORWARDER-BASE strings in the roleConfigGroupsRefNames array.
        {
    ...
    "hostTemplates": [
        {
        "refName": "...",
        "cardinality" : ...,
        "roleConfigGroupsRefNames": [
        ...,
        "bdp_pep-PTY_RPAGENT-BASE",
        "bdp_pep-PTY_LOGFORWARDER-BASE"
        ],
        ...
        },
        {
        ...
        },
        ...
        ]
        ...
        }
  13. Click Register.

4 - Installing the Big Data Protector

4.1 - Creating the DataHub Cluster

Create a new Data Hub cluster with Big Data Protector, use the registered cluster template with the two Recipes generated and custom properties.

To create a Data Hub Cluster:

  1. Log in to the Cloudera Management Console.

  2. Click Data Hub Clusters. The Data Hubs page appears.

  3. Click Create Data Hub. The Provision Data Hub page appears.

  4. From the environment list, select the CDP AWS environment.

  5. Select the Custom tab.

  6. From the Cluster Template list, select the previously created customized cluster template with the Big Data Protector.

  7. In the Cluster Name box, enter a name to identify the cluster.

  8. Click Advanced Options.

  9. Click the Image Catalog tab.

  10. Select the required image for the operating system.

  11. Click the Hardware and Storage tab.
    Note the host group which will install the Cloudera Manager (CM) Server. Usually, the host is the Master node.

  12. Click the Cluster Extensions tab.

  13. Attach the two previously registered Recipes to the host group that would host the Cloudera Manager Server.

  14. Add the contents of the custom_properties_template.json file to the Custom Properties section. For example:

    {
"rpa_sync_host": "Replace with ESA IP Address or Hostname",
"auditstore_type": "Replace with Audit Store type. Allowed Values - <Protegrity Audit Store|External Audit Store|Protegrity Audit Store + External Audit Store>",
"auditstore_ip_port_list": "Replace with list of hostnames and/or ports of Protegrity Audit Store(s). Allowed Syntax - hostname[:port][,hostname[:port],hostname[:port]...]"
}

    Warning: After the cluster startup, the Cloudera manager username and password are seen in clear in the log files generated in the /var/log/recipes/ directory.

  15. Click Provision Cluster. Cloudera Manager creates the cluster as per the specifics mentioned in the scripts and templates.

5 - Configuring the Big Data Protector

The instructions to configure the parameters for the Big Data Protector are explained in this section. Apart from manually setting the configuration parameters, the installation package also provides the helper script to set and restore the configurations.

5.1 - Setting the Parameters Manually

To manually set the configuration parameters for the Big Data Protector, refer to the following table:

From v10.0.0 onwards, the BDP pep* jar files will be installed under the /opt/cloudera/parcels/PTY_BDP/bdp/lib/ directory. In addition, the BDP version would be added to the .jar file names.

ServiceBDP Configuration
Hive on TezIn the Hive on Tez Service Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh and Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh:
Key: HIVE_CLASSPATH
Value: /opt/cloudera/parcels/PTY_BDP/bdp/lib/jcorelite.jar:/opt/cloudera/parcels/PTY_BDP/bdp/lib/pephive-<hive_version>_v<bdp_version>.jar:${HIVE_CLASSPATH}

For example: /opt/cloudera/parcels/PTY_BDP/bdp/lib/jcorelite.jar:/opt/cloudera/parcels/PTY_BDP/bdp/lib/pephive-3.1.3000_v10.0.0+4.jar:${HIVE_CLASSPATH}

In the Hive on Tez Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml:
Name: hive.exec.pre.hooks<br>Value: com.protegrity.hive.PtyHiveUserPreHook
TezName: tez.cluster.additional.classpath.prefix
Value: /opt/cloudera/parcels/PTY_BDP/bdp/lib/jcorelite.jar:/opt/cloudera/parcels/PTY_BDP/bdp/lib/pephive-<hive_version>_v<bdp_version>.jar
HBaseName: hbase.coprocessor.region.classes
Value: com.protegrity.hbase.PTYRegionObserver
Spark on YarnIn Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh:
SPARK_DIST_CLASSPATH=/opt/cloudera/parcels/PTY_BDP/bdp/lib/jcorelite.jar:/opt/cloudera/parcels/PTY_BDP/bdp/lib/pepspark-<spark_version>_v<bdp_version>.jar:/opt/cloudera/parcels/PTY_BDP/bdp/lib/pephive-<hive_version>_v<bdp_version>.jar:${SPARK_DIST_CLASSPATH}
Spark 3 on YarnIn Spark 3 Service Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-env.sh:
SPARK_DIST_CLASSPATH=/opt/cloudera/parcels/PTY_BDP/bdp/lib/jcorelite.jar:/opt/cloudera/parcels/PTY_BDP/bdp/lib/pepspark-<spark_version>_v<bdp_version>.jar:/opt/cloudera/parcels/PTY_BDP/bdp/lib/pephive-<hive_version>_v<bdp_version>.jar:${SPARK_DIST_CLASSPATH}
ImpalaIn the Impala Daemon Environment Advanced Configuration Snippet (Safety Valve):
Key: PTY_CONFIGPATH
Value: /opt/cloudera/parcels/PTY_BDP/bdp/data/config.ini

Warning: Ensure that you do not override the BDP configurations at the client side. Overriding the configurations can result in the component failure.

After you set BDP configurations, restart the services that are in the Stale configuration state on Cloudera Manager. Ensure to Redeploy the client configuration.

5.2 - Setting the Parameters using the Helper Script

After installing the Big Data Protector, set the configuration parameters. These parameters will vary depending on the services that will be used. Protegrity now provides the set_unset_bdp_config.sh script to set the configuration parameters for the required services.

Important: To uninstall the Big Data Protector, ensure to roll back the configuration parameters, to their previous values, that are set after installing the Big Data Protector. For more information, refer Restoring the Parameters using the Helper Script.

To set the Big Data Protector configuration:

  1. Log in to the staging machine.

  2. Navigate to the directory where you executed configurator script and generated the installation files.

  3. To set the configurations using the helper script, run the following command:

    ./set_unset_bdp_config.sh

  4. Press ENTER.

    The prompt to enter the protocol for the Cloudera Manager server appears.

    Select the Cloudera Manager URL Protocol.
[ 1 ] : http://
[ 2 ] : https://
Enter the no.:

  5. To use https, type 2.

  6. Press ENTER.

    The prompt to enter the IP address of the Cloudera Manager server appears.

    Enter Cloudera Manager Server Node's Hostname/IP Address:

  7. Enter the IP address of the node where the Cloudera Manager Server is installed.

  8. Press ENTER.

    The prompt to enter the port number for the Cloudera Manager server appears.

    Enter Cloudera Manager Server's Port No. [7183]:

    Note: For https, the script will use 7183 as the default port and for http, the script will use 7180 as the default port.

  9. Press ENTER.

    The prompt to enter the name of the cluster appears.

    Enter Cluster's Name:

  10. Enter the name of the cluster.

  11. Press ENTER.

The prompt to enter the username to access Cloudera Manager appears.

```
Enter Cloudera Manager's Username:
```

  1. Enter the username.

  2. Press ENTER.

    The prompt to enter the password appears.

    Enter Cloudera Manager's Password:

  3. Enter the password.

  4. Press ENTER.

    The script verifies the cluster details and the prompt to set or remove the configuration appears.

    Cluster's existence verified.

Do you want to set or unset the BDP configs?
[ 1 ] : SET the BDP configs
[ 2 ] : UNSET the BDP configs
Enter the no.:

  5. To set the configuration for the Big Data Protector, type 1.

  6. Press ENTER.

    The script updates the configuration for the Big Data Protector.

    Checking existence of HBase service with name 'hbase'.
##O=#    #
Warning: Unable to check existence of HBase service 'hbase'. Skipping this service...
{
"message" : "Service 'hbase' not found in cluster <cluster_name>."
}

Checking existence of Hive on Tez service with name 'hive_on_tez'.
##O=#    #
Service 'hive_on_tez' exists.

Setting Hive on Tez's config...
##O=#    #
##O=#    #
############################################################################################################################## 100.0%
Hive on Tez Service wide configs ('HIVE_ON_TEZ_service_env_safety_valve' and 'hive_service_config_safety_valve') have been updated.
##O=#    #
##O=#    #
############################################################################################################################### 100.0%
Hive on Tez's 'hive_client_env_safety_valve' config for Role Group 'hive_on_tez-GATEWAY-BASE' has been updated.

Checking existence of Tez service with name 'tez'.
##O=#    #
Service 'tez' exists.

Setting Tez's config...
##O=#    #
############################################################################################################################### 100.0%
Tez Service wide config ('tez.cluster.additional.classpath.prefix') has been updated.

Checking existence of Impala service with name 'impala'.
##O=#    #
Warning: Unable to check existence of Impala service 'impala'. Skipping this service...
{
"message" : "Service 'impala' not found in cluster <cluster_name>."
}

Checking existence of Spark3 on Yarn service with name 'spark3_on_yarn'.
##O=#    #
Service 'spark3_on_yarn' exists.

Setting Spark3 on Yarn's config...
##O=#    #
############################################################################################################################# 100.0%
Spark3 on Yarn Service wide config ('spark3-conf/spark-env.sh_service_safety_valve') has been updated.

5.3 - Installing the UDFs using the Helper Script

The Big Data Protector build provides helper scripts to register the user-defined functions for the following components:

  • Hive
  • Impala
  • SparkSQL
  • PySpark

To use the helper scripts to drop the UDFs, refer Drop the UDFs using the Helper Script.

Registering the Termporary Hive UDFs

  1. Log in to the master node with a user account having permissions to create and drop UDFs.

  2. To navigate to the directory that contains the helper script, run the following command:

    cd /opt/cloudera/parcels/PTY_BDP/pephive/scripts

  3. To create the UDFs using the helper script, run the following command:

    beeline -f create_temp_hive_udfs.hql;

    Execute the command in beeline after establishing a connection.

  4. Press ENTER.

    The script creates all the temporary user-defined functions for Hive.

    Connected to: Apache Hive (version 3.1.3000.7.3.1.400-100)
Driver: Hive JDBC (version 3.1.3000.7.3.1.400-100)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyGetVersion AS 'com.protegrity.hive.udf.ptyGetVersion';
INFO  : Compiling command(queryId=hive_20250916121741_49e4a9e3-5322-45b1-bc74-b812ae853934): CREATE TEMPORARY FUNCTION ptyGetVersion AS 'com.protegrity.hive.udf.ptyGetVersion'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121741_49e4a9e3-5322-45b1-bc74-b812ae853934); Time taken: 0.088 seconds
INFO  : Executing command(queryId=hive_20250916121741_49e4a9e3-5322-45b1-bc74-b812ae853934): CREATE TEMPORARY FUNCTION ptyGetVersion AS 'com.protegrity.hive.udf.ptyGetVersion'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121741_49e4a9e3-5322-45b1-bc74-b812ae853934); Time taken: 0.003 seconds
INFO  : OK
No rows affected (0.154 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyGetVersionExtended AS 'com.protegrity.hive.udf.ptyGetVersionExtended';
INFO  : Compiling command(queryId=hive_20250916121741_2a7a85d4-b1b6-479c-8552-d26f1cce1d53): CREATE TEMPORARY FUNCTION ptyGetVersionExtended AS 'com.protegrity.hive.udf.ptyGetVersionExtended'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121741_2a7a85d4-b1b6-479c-8552-d26f1cce1d53); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121741_2a7a85d4-b1b6-479c-8552-d26f1cce1d53): CREATE TEMPORARY FUNCTION ptyGetVersionExtended AS 'com.protegrity.hive.udf.ptyGetVersionExtended'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121741_2a7a85d4-b1b6-479c-8552-d26f1cce1d53); Time taken: 0.002 seconds
INFO  : OK
No rows affected (0.052 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyWhoAmI AS 'com.protegrity.hive.udf.ptyWhoAmI';
INFO  : Compiling command(queryId=hive_20250916121741_ebe06a7c-c265-4705-ae7b-8181065d1c8e): CREATE TEMPORARY FUNCTION ptyWhoAmI AS 'com.protegrity.hive.udf.ptyWhoAmI'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121741_ebe06a7c-c265-4705-ae7b-8181065d1c8e); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121741_ebe06a7c-c265-4705-ae7b-8181065d1c8e): CREATE TEMPORARY FUNCTION ptyWhoAmI AS 'com.protegrity.hive.udf.ptyWhoAmI'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121741_ebe06a7c-c265-4705-ae7b-8181065d1c8e); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.05 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectStr AS 'com.protegrity.hive.udf.ptyProtectStr';
INFO  : Compiling command(queryId=hive_20250916121741_260aef88-ab60-4d0b-b502-ebf178bd9065): CREATE TEMPORARY FUNCTION ptyProtectStr AS 'com.protegrity.hive.udf.ptyProtectStr'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121741_260aef88-ab60-4d0b-b502-ebf178bd9065); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121741_260aef88-ab60-4d0b-b502-ebf178bd9065): CREATE TEMPORARY FUNCTION ptyProtectStr AS 'com.protegrity.hive.udf.ptyProtectStr'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121741_260aef88-ab60-4d0b-b502-ebf178bd9065); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.052 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectStr AS 'com.protegrity.hive.udf.ptyUnprotectStr';
INFO  : Compiling command(queryId=hive_20250916121741_feba856d-e516-4b08-805a-3ee89aeff524): CREATE TEMPORARY FUNCTION ptyUnprotectStr AS 'com.protegrity.hive.udf.ptyUnprotectStr'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121741_feba856d-e516-4b08-805a-3ee89aeff524); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121741_feba856d-e516-4b08-805a-3ee89aeff524): CREATE TEMPORARY FUNCTION ptyUnprotectStr AS 'com.protegrity.hive.udf.ptyUnprotectStr'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121741_feba856d-e516-4b08-805a-3ee89aeff524); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.053 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyReprotect AS 'com.protegrity.hive.udf.ptyReprotect';
INFO  : Compiling command(queryId=hive_20250916121741_cd516b0a-648a-4893-92f0-7486ee4525d9): CREATE TEMPORARY FUNCTION ptyReprotect AS 'com.protegrity.hive.udf.ptyReprotect'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121741_cd516b0a-648a-4893-92f0-7486ee4525d9); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121741_cd516b0a-648a-4893-92f0-7486ee4525d9): CREATE TEMPORARY FUNCTION ptyReprotect AS 'com.protegrity.hive.udf.ptyReprotect'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121741_cd516b0a-648a-4893-92f0-7486ee4525d9); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.049 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectUnicode AS 'com.protegrity.hive.udf.ptyProtectUnicode';
INFO  : Compiling command(queryId=hive_20250916121741_ab35e17d-4050-4ae9-a3eb-a796be96f739): CREATE TEMPORARY FUNCTION ptyProtectUnicode AS 'com.protegrity.hive.udf.ptyProtectUnicode'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121741_ab35e17d-4050-4ae9-a3eb-a796be96f739); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121741_ab35e17d-4050-4ae9-a3eb-a796be96f739): CREATE TEMPORARY FUNCTION ptyProtectUnicode AS 'com.protegrity.hive.udf.ptyProtectUnicode'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121741_ab35e17d-4050-4ae9-a3eb-a796be96f739); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.048 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectUnicode AS 'com.protegrity.hive.udf.ptyUnprotectUnicode';
INFO  : Compiling command(queryId=hive_20250916121741_10e20116-c909-45e3-be3d-870ce13ef54b): CREATE TEMPORARY FUNCTION ptyUnprotectUnicode AS 'com.protegrity.hive.udf.ptyUnprotectUnicode'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121741_10e20116-c909-45e3-be3d-870ce13ef54b); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121741_10e20116-c909-45e3-be3d-870ce13ef54b): CREATE TEMPORARY FUNCTION ptyUnprotectUnicode AS 'com.protegrity.hive.udf.ptyUnprotectUnicode'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121741_10e20116-c909-45e3-be3d-870ce13ef54b); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.048 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyReprotectUnicode AS 'com.protegrity.hive.udf.ptyReprotectUnicode';
INFO  : Compiling command(queryId=hive_20250916121741_7a142263-62c2-44b1-bbb0-05530e77daa7): CREATE TEMPORARY FUNCTION ptyReprotectUnicode AS 'com.protegrity.hive.udf.ptyReprotectUnicode'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121741_7a142263-62c2-44b1-bbb0-05530e77daa7); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121741_7a142263-62c2-44b1-bbb0-05530e77daa7): CREATE TEMPORARY FUNCTION ptyReprotectUnicode AS 'com.protegrity.hive.udf.ptyReprotectUnicode'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121741_7a142263-62c2-44b1-bbb0-05530e77daa7); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.056 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectShort AS 'com.protegrity.hive.udf.ptyProtectShort';
INFO  : Compiling command(queryId=hive_20250916121742_1b5414e1-175e-4889-bad3-e64ca6595fe0): CREATE TEMPORARY FUNCTION ptyProtectShort AS 'com.protegrity.hive.udf.ptyProtectShort'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_1b5414e1-175e-4889-bad3-e64ca6595fe0); Time taken: 0.022 seconds
INFO  : Executing command(queryId=hive_20250916121742_1b5414e1-175e-4889-bad3-e64ca6595fe0): CREATE TEMPORARY FUNCTION ptyProtectShort AS 'com.protegrity.hive.udf.ptyProtectShort'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_1b5414e1-175e-4889-bad3-e64ca6595fe0); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.053 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectShort AS 'com.protegrity.hive.udf.ptyUnprotectShort';
INFO  : Compiling command(queryId=hive_20250916121742_58b96d4d-25a2-4886-a4c5-ad1c69164016): CREATE TEMPORARY FUNCTION ptyUnprotectShort AS 'com.protegrity.hive.udf.ptyUnprotectShort'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_58b96d4d-25a2-4886-a4c5-ad1c69164016); Time taken: 0.026 seconds
INFO  : Executing command(queryId=hive_20250916121742_58b96d4d-25a2-4886-a4c5-ad1c69164016): CREATE TEMPORARY FUNCTION ptyUnprotectShort AS 'com.protegrity.hive.udf.ptyUnprotectShort'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_58b96d4d-25a2-4886-a4c5-ad1c69164016); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.058 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectInt AS 'com.protegrity.hive.udf.ptyProtectInt';
INFO  : Compiling command(queryId=hive_20250916121742_8e53692d-4094-4b1a-98fa-829d90b3dea3): CREATE TEMPORARY FUNCTION ptyProtectInt AS 'com.protegrity.hive.udf.ptyProtectInt'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_8e53692d-4094-4b1a-98fa-829d90b3dea3); Time taken: 0.023 seconds
INFO  : Executing command(queryId=hive_20250916121742_8e53692d-4094-4b1a-98fa-829d90b3dea3): CREATE TEMPORARY FUNCTION ptyProtectInt AS 'com.protegrity.hive.udf.ptyProtectInt'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_8e53692d-4094-4b1a-98fa-829d90b3dea3); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.058 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectInt AS 'com.protegrity.hive.udf.ptyUnprotectInt';
INFO  : Compiling command(queryId=hive_20250916121742_7762d5cd-1934-45d4-8297-d76236575744): CREATE TEMPORARY FUNCTION ptyUnprotectInt AS 'com.protegrity.hive.udf.ptyUnprotectInt'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_7762d5cd-1934-45d4-8297-d76236575744); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121742_7762d5cd-1934-45d4-8297-d76236575744): CREATE TEMPORARY FUNCTION ptyUnprotectInt AS 'com.protegrity.hive.udf.ptyUnprotectInt'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_7762d5cd-1934-45d4-8297-d76236575744); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.048 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectBigInt as 'com.protegrity.hive.udf.ptyProtectBigInt';
INFO  : Compiling command(queryId=hive_20250916121742_7fb43101-562a-441b-8acb-decbe8e216ea): CREATE TEMPORARY FUNCTION ptyProtectBigInt as 'com.protegrity.hive.udf.ptyProtectBigInt'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_7fb43101-562a-441b-8acb-decbe8e216ea); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121742_7fb43101-562a-441b-8acb-decbe8e216ea): CREATE TEMPORARY FUNCTION ptyProtectBigInt as 'com.protegrity.hive.udf.ptyProtectBigInt'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_7fb43101-562a-441b-8acb-decbe8e216ea); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectBigInt as 'com.protegrity.hive.udf.ptyUnprotectBigInt';
INFO  : Compiling command(queryId=hive_20250916121742_1f107ea2-c7b1-4d18-8f33-04442d84b80e): CREATE TEMPORARY FUNCTION ptyUnprotectBigInt as 'com.protegrity.hive.udf.ptyUnprotectBigInt'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_1f107ea2-c7b1-4d18-8f33-04442d84b80e); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121742_1f107ea2-c7b1-4d18-8f33-04442d84b80e): CREATE TEMPORARY FUNCTION ptyUnprotectBigInt as 'com.protegrity.hive.udf.ptyUnprotectBigInt'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_1f107ea2-c7b1-4d18-8f33-04442d84b80e); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.047 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectFloat as 'com.protegrity.hive.udf.ptyProtectFloat';
INFO  : Compiling command(queryId=hive_20250916121742_88d8ed50-72d1-4e4b-9c9e-ed105e099c7b): CREATE TEMPORARY FUNCTION ptyProtectFloat as 'com.protegrity.hive.udf.ptyProtectFloat'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_88d8ed50-72d1-4e4b-9c9e-ed105e099c7b); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121742_88d8ed50-72d1-4e4b-9c9e-ed105e099c7b): CREATE TEMPORARY FUNCTION ptyProtectFloat as 'com.protegrity.hive.udf.ptyProtectFloat'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_88d8ed50-72d1-4e4b-9c9e-ed105e099c7b); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.047 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectFloat as 'com.protegrity.hive.udf.ptyProtectFloat';
INFO  : Compiling command(queryId=hive_20250916121742_7455402f-bc13-425b-98b0-204c72b088c2): CREATE TEMPORARY FUNCTION ptyUnprotectFloat as 'com.protegrity.hive.udf.ptyProtectFloat'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_7455402f-bc13-425b-98b0-204c72b088c2); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121742_7455402f-bc13-425b-98b0-204c72b088c2): CREATE TEMPORARY FUNCTION ptyUnprotectFloat as 'com.protegrity.hive.udf.ptyProtectFloat'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_7455402f-bc13-425b-98b0-204c72b088c2); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectDouble as 'com.protegrity.hive.udf.ptyProtectDouble';
INFO  : Compiling command(queryId=hive_20250916121742_0f1a9810-1311-4dcd-a4c6-bfe52088cf1b): CREATE TEMPORARY FUNCTION ptyProtectDouble as 'com.protegrity.hive.udf.ptyProtectDouble'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_0f1a9810-1311-4dcd-a4c6-bfe52088cf1b); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121742_0f1a9810-1311-4dcd-a4c6-bfe52088cf1b): CREATE TEMPORARY FUNCTION ptyProtectDouble as 'com.protegrity.hive.udf.ptyProtectDouble'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_0f1a9810-1311-4dcd-a4c6-bfe52088cf1b); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.048 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectDouble as 'com.protegrity.hive.udf.ptyUnprotectDouble';
INFO  : Compiling command(queryId=hive_20250916121742_cab992d8-efa2-41e6-b83e-71074d444565): CREATE TEMPORARY FUNCTION ptyUnprotectDouble as 'com.protegrity.hive.udf.ptyUnprotectDouble'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_cab992d8-efa2-41e6-b83e-71074d444565); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121742_cab992d8-efa2-41e6-b83e-71074d444565): CREATE TEMPORARY FUNCTION ptyUnprotectDouble as 'com.protegrity.hive.udf.ptyUnprotectDouble'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_cab992d8-efa2-41e6-b83e-71074d444565); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectDec as 'com.protegrity.hive.udf.ptyProtectDec';
INFO  : Compiling command(queryId=hive_20250916121742_95374cee-053b-42b1-a7f9-39d4d0eb24c6): CREATE TEMPORARY FUNCTION ptyProtectDec as 'com.protegrity.hive.udf.ptyProtectDec'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_95374cee-053b-42b1-a7f9-39d4d0eb24c6); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121742_95374cee-053b-42b1-a7f9-39d4d0eb24c6): CREATE TEMPORARY FUNCTION ptyProtectDec as 'com.protegrity.hive.udf.ptyProtectDec'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_95374cee-053b-42b1-a7f9-39d4d0eb24c6); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectDec as 'com.protegrity.hive.udf.ptyUnprotectDec';
INFO  : Compiling command(queryId=hive_20250916121742_1d045781-69d8-4a55-a87b-0f43121b2a09): CREATE TEMPORARY FUNCTION ptyUnprotectDec as 'com.protegrity.hive.udf.ptyUnprotectDec'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_1d045781-69d8-4a55-a87b-0f43121b2a09); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121742_1d045781-69d8-4a55-a87b-0f43121b2a09): CREATE TEMPORARY FUNCTION ptyUnprotectDec as 'com.protegrity.hive.udf.ptyUnprotectDec'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_1d045781-69d8-4a55-a87b-0f43121b2a09); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectHiveDecimal as 'com.protegrity.hive.udf.ptyProtectHiveDecimal';
INFO  : Compiling command(queryId=hive_20250916121742_281198df-c918-470f-99a5-bca98a5936b7): CREATE TEMPORARY FUNCTION ptyProtectHiveDecimal as 'com.protegrity.hive.udf.ptyProtectHiveDecimal'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_281198df-c918-470f-99a5-bca98a5936b7); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121742_281198df-c918-470f-99a5-bca98a5936b7): CREATE TEMPORARY FUNCTION ptyProtectHiveDecimal as 'com.protegrity.hive.udf.ptyProtectHiveDecimal'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_281198df-c918-470f-99a5-bca98a5936b7); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectHiveDecimal as 'com.protegrity.hive.udf.ptyUnprotectHiveDecimal';
INFO  : Compiling command(queryId=hive_20250916121742_b3d564e5-22fa-4985-b988-eac4d188b3fa): CREATE TEMPORARY FUNCTION ptyUnprotectHiveDecimal as 'com.protegrity.hive.udf.ptyUnprotectHiveDecimal'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_b3d564e5-22fa-4985-b988-eac4d188b3fa); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121742_b3d564e5-22fa-4985-b988-eac4d188b3fa): CREATE TEMPORARY FUNCTION ptyUnprotectHiveDecimal as 'com.protegrity.hive.udf.ptyUnprotectHiveDecimal'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_b3d564e5-22fa-4985-b988-eac4d188b3fa); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectDate AS 'com.protegrity.hive.udf.ptyProtectDate';
INFO  : Compiling command(queryId=hive_20250916121742_05f34839-7ab3-47c9-bf7c-de32b44b1305): CREATE TEMPORARY FUNCTION ptyProtectDate AS 'com.protegrity.hive.udf.ptyProtectDate'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_05f34839-7ab3-47c9-bf7c-de32b44b1305); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121742_05f34839-7ab3-47c9-bf7c-de32b44b1305): CREATE TEMPORARY FUNCTION ptyProtectDate AS 'com.protegrity.hive.udf.ptyProtectDate'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_05f34839-7ab3-47c9-bf7c-de32b44b1305); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectDate AS 'com.protegrity.hive.udf.ptyUnprotectDate';
INFO  : Compiling command(queryId=hive_20250916121742_71ac75b0-da95-409f-a070-5e06b4451973): CREATE TEMPORARY FUNCTION ptyUnprotectDate AS 'com.protegrity.hive.udf.ptyUnprotectDate'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_71ac75b0-da95-409f-a070-5e06b4451973); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121742_71ac75b0-da95-409f-a070-5e06b4451973): CREATE TEMPORARY FUNCTION ptyUnprotectDate AS 'com.protegrity.hive.udf.ptyUnprotectDate'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_71ac75b0-da95-409f-a070-5e06b4451973); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectDateTime AS 'com.protegrity.hive.udf.ptyProtectDateTime';
INFO  : Compiling command(queryId=hive_20250916121742_96dfb779-c77b-4c8f-a5c0-6c3f99e5aa9c): CREATE TEMPORARY FUNCTION ptyProtectDateTime AS 'com.protegrity.hive.udf.ptyProtectDateTime'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121742_96dfb779-c77b-4c8f-a5c0-6c3f99e5aa9c); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121742_96dfb779-c77b-4c8f-a5c0-6c3f99e5aa9c): CREATE TEMPORARY FUNCTION ptyProtectDateTime AS 'com.protegrity.hive.udf.ptyProtectDateTime'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121742_96dfb779-c77b-4c8f-a5c0-6c3f99e5aa9c); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.044 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectDateTime AS 'com.protegrity.hive.udf.ptyUnprotectDateTime';
INFO  : Compiling command(queryId=hive_20250916121743_08af3475-e39c-4cf6-9762-77c42ca6efdf): CREATE TEMPORARY FUNCTION ptyUnprotectDateTime AS 'com.protegrity.hive.udf.ptyUnprotectDateTime'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121743_08af3475-e39c-4cf6-9762-77c42ca6efdf); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121743_08af3475-e39c-4cf6-9762-77c42ca6efdf): CREATE TEMPORARY FUNCTION ptyUnprotectDateTime AS 'com.protegrity.hive.udf.ptyUnprotectDateTime'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121743_08af3475-e39c-4cf6-9762-77c42ca6efdf); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.047 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyProtectChar AS 'com.protegrity.hive.udf.ptyProtectChar';
INFO  : Compiling command(queryId=hive_20250916121743_f2ce3155-6e16-41a2-a072-d9f59142b0ea): CREATE TEMPORARY FUNCTION ptyProtectChar AS 'com.protegrity.hive.udf.ptyProtectChar'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121743_f2ce3155-6e16-41a2-a072-d9f59142b0ea); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121743_f2ce3155-6e16-41a2-a072-d9f59142b0ea): CREATE TEMPORARY FUNCTION ptyProtectChar AS 'com.protegrity.hive.udf.ptyProtectChar'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121743_f2ce3155-6e16-41a2-a072-d9f59142b0ea); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyUnprotectChar AS 'com.protegrity.hive.udf.ptyUnprotectChar';
INFO  : Compiling command(queryId=hive_20250916121743_851371db-2ce1-4af0-93a0-11e83ace1b87): CREATE TEMPORARY FUNCTION ptyUnprotectChar AS 'com.protegrity.hive.udf.ptyUnprotectChar'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121743_851371db-2ce1-4af0-93a0-11e83ace1b87); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121743_851371db-2ce1-4af0-93a0-11e83ace1b87): CREATE TEMPORARY FUNCTION ptyUnprotectChar AS 'com.protegrity.hive.udf.ptyUnprotectChar'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121743_851371db-2ce1-4af0-93a0-11e83ace1b87); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.047 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyStringEnc as 'com.protegrity.hive.udf.ptyStringEnc';
INFO  : Compiling command(queryId=hive_20250916121743_f72cb4f2-b99f-43c8-8a3e-a8ab7370fe94): CREATE TEMPORARY FUNCTION ptyStringEnc as 'com.protegrity.hive.udf.ptyStringEnc'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121743_f72cb4f2-b99f-43c8-8a3e-a8ab7370fe94); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121743_f72cb4f2-b99f-43c8-8a3e-a8ab7370fe94): CREATE TEMPORARY FUNCTION ptyStringEnc as 'com.protegrity.hive.udf.ptyStringEnc'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121743_f72cb4f2-b99f-43c8-8a3e-a8ab7370fe94); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyStringDec as 'com.protegrity.hive.udf.ptyStringDec';
INFO  : Compiling command(queryId=hive_20250916121743_0cc281bd-6d0c-4937-aaae-5a94f6010197): CREATE TEMPORARY FUNCTION ptyStringDec as 'com.protegrity.hive.udf.ptyStringDec'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121743_0cc281bd-6d0c-4937-aaae-5a94f6010197); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121743_0cc281bd-6d0c-4937-aaae-5a94f6010197): CREATE TEMPORARY FUNCTION ptyStringDec as 'com.protegrity.hive.udf.ptyStringDec'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121743_0cc281bd-6d0c-4937-aaae-5a94f6010197); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>.> CREATE TEMPORARY FUNCTION ptyStringReEnc as 'com.protegrity.hive.udf.ptyStringReEnc';
INFO  : Compiling command(queryId=hive_20250916121743_ad7d4c07-9973-4f61-9afc-20df0ea9b34a): CREATE TEMPORARY FUNCTION ptyStringReEnc as 'com.protegrity.hive.udf.ptyStringReEnc'
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121743_ad7d4c07-9973-4f61-9afc-20df0ea9b34a); Time taken: 0.019 seconds
INFO  : Executing command(queryId=hive_20250916121743_ad7d4c07-9973-4f61-9afc-20df0ea9b34a): CREATE TEMPORARY FUNCTION ptyStringReEnc as 'com.protegrity.hive.udf.ptyStringReEnc'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121743_ad7d4c07-9973-4f61-9afc-20df0ea9b34a); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)

Registering the Permanent Hive UDFs

  1. Log in to the master node with a user account having permissions to create and drop UDFs.

  2. To navigate to the directory that contains the helper script, run the following command:

    cd /opt/cloudera/parcels/PTY_BDP/pephive/scripts

  3. To create the UDFs using the helper script, run the following command:

    beeline -f create_perm_hive_udfs.hql;

    Execute the command in beeline after establishing a connection.

  4. Press ENTER.

    The script creates all the permanent user-defined functions for Hive.

    Connected to: Apache Hive (version 3.1.3000.7.3.1.400-100)
Driver: Hive JDBC (version 3.1.3000.7.3.1.400-100)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyGetVersion AS 'com.protegrity.hive.udf.ptyGetVersion';
INFO  : Compiling command(queryId=hive_20250916112109_e285254e-4d3f-4485-bc15-8c97cb1fd704): CREATE FUNCTION ptyGetVersion AS 'com.protegrity.hive.udf.ptyGetVersion'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112109_e285254e-4d3f-4485-bc15-8c97cb1fd704); Time taken: 0.035 seconds
INFO  : Executing command(queryId=hive_20250916112109_e285254e-4d3f-4485-bc15-8c97cb1fd704): CREATE FUNCTION ptyGetVersion AS 'com.protegrity.hive.udf.ptyGetVersion'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112109_e285254e-4d3f-4485-bc15-8c97cb1fd704); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.128 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyGetVersionExtended AS 'com.protegrity.hive.udf.ptyGetVersionExtended';
INFO  : Compiling command(queryId=hive_20250916112109_318b5667-be58-48fc-a945-c44f4f3e6ce3): CREATE FUNCTION ptyGetVersionExtended AS 'com.protegrity.hive.udf.ptyGetVersionExtended'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112109_318b5667-be58-48fc-a945-c44f4f3e6ce3); Time taken: 0.028 seconds
INFO  : Executing command(queryId=hive_20250916112109_318b5667-be58-48fc-a945-c44f4f3e6ce3): CREATE FUNCTION ptyGetVersionExtended AS 'com.protegrity.hive.udf.ptyGetVersionExtended'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112109_318b5667-be58-48fc-a945-c44f4f3e6ce3); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.091 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyWhoAmI AS 'com.protegrity.hive.udf.ptyWhoAmI';
INFO  : Compiling command(queryId=hive_20250916112109_6f93df62-3990-42a9-9e85-aa4faf03a587): CREATE FUNCTION ptyWhoAmI AS 'com.protegrity.hive.udf.ptyWhoAmI'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112109_6f93df62-3990-42a9-9e85-aa4faf03a587); Time taken: 0.028 seconds
INFO  : Executing command(queryId=hive_20250916112109_6f93df62-3990-42a9-9e85-aa4faf03a587): CREATE FUNCTION ptyWhoAmI AS 'com.protegrity.hive.udf.ptyWhoAmI'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112109_6f93df62-3990-42a9-9e85-aa4faf03a587); Time taken: 0.024 seconds
INFO  : OK
No rows affected (0.096 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectStr AS 'com.protegrity.hive.udf.ptyProtectStr';
INFO  : Compiling command(queryId=hive_20250916112109_ec8d621c-be21-4a7a-88e4-b7ee9bf0c23f): CREATE FUNCTION ptyProtectStr AS 'com.protegrity.hive.udf.ptyProtectStr'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112109_ec8d621c-be21-4a7a-88e4-b7ee9bf0c23f); Time taken: 0.029 seconds
INFO  : Executing command(queryId=hive_20250916112109_ec8d621c-be21-4a7a-88e4-b7ee9bf0c23f): CREATE FUNCTION ptyProtectStr AS 'com.protegrity.hive.udf.ptyProtectStr'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112109_ec8d621c-be21-4a7a-88e4-b7ee9bf0c23f); Time taken: 0.018 seconds
INFO  : OK
No rows affected (0.094 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectStr AS 'com.protegrity.hive.udf.ptyUnprotectStr';
INFO  : Compiling command(queryId=hive_20250916112109_53e5e6e4-7253-4451-b88e-61fa84ea3a47): CREATE FUNCTION ptyUnprotectStr AS 'com.protegrity.hive.udf.ptyUnprotectStr'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112109_53e5e6e4-7253-4451-b88e-61fa84ea3a47); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112109_53e5e6e4-7253-4451-b88e-61fa84ea3a47): CREATE FUNCTION ptyUnprotectStr AS 'com.protegrity.hive.udf.ptyUnprotectStr'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112109_53e5e6e4-7253-4451-b88e-61fa84ea3a47); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.091 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyReprotect AS 'com.protegrity.hive.udf.ptyReprotect';
INFO  : Compiling command(queryId=hive_20250916112109_294034e4-947e-4ea9-8fa3-3f6e96227c98): CREATE FUNCTION ptyReprotect AS 'com.protegrity.hive.udf.ptyReprotect'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112109_294034e4-947e-4ea9-8fa3-3f6e96227c98); Time taken: 0.03 seconds
INFO  : Executing command(queryId=hive_20250916112109_294034e4-947e-4ea9-8fa3-3f6e96227c98): CREATE FUNCTION ptyReprotect AS 'com.protegrity.hive.udf.ptyReprotect'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112109_294034e4-947e-4ea9-8fa3-3f6e96227c98); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.095 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectUnicode AS 'com.protegrity.hive.udf.ptyProtectUnicode';
INFO  : Compiling command(queryId=hive_20250916112109_b79fdf6b-c51f-4956-ba9c-0cbf50b96060): CREATE FUNCTION ptyProtectUnicode AS 'com.protegrity.hive.udf.ptyProtectUnicode'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112109_b79fdf6b-c51f-4956-ba9c-0cbf50b96060); Time taken: 0.028 seconds
INFO  : Executing command(queryId=hive_20250916112109_b79fdf6b-c51f-4956-ba9c-0cbf50b96060): CREATE FUNCTION ptyProtectUnicode AS 'com.protegrity.hive.udf.ptyProtectUnicode'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112109_b79fdf6b-c51f-4956-ba9c-0cbf50b96060); Time taken: 0.024 seconds
INFO  : OK
No rows affected (0.098 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectUnicode AS 'com.protegrity.hive.udf.ptyUnprotectUnicode';
INFO  : Compiling command(queryId=hive_20250916112110_ead2098b-374f-42ac-822d-285c50cb865d): CREATE FUNCTION ptyUnprotectUnicode AS 'com.protegrity.hive.udf.ptyUnprotectUnicode'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112110_ead2098b-374f-42ac-822d-285c50cb865d); Time taken: 0.03 seconds
INFO  : Executing command(queryId=hive_20250916112110_ead2098b-374f-42ac-822d-285c50cb865d): CREATE FUNCTION ptyUnprotectUnicode AS 'com.protegrity.hive.udf.ptyUnprotectUnicode'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112110_ead2098b-374f-42ac-822d-285c50cb865d); Time taken: 0.021 seconds
INFO  : OK
No rows affected (0.112 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyReprotectUnicode AS 'com.protegrity.hive.udf.ptyReprotectUnicode';
INFO  : Compiling command(queryId=hive_20250916112110_4e3502b6-cbc4-4089-9420-cef27520de02): CREATE FUNCTION ptyReprotectUnicode AS 'com.protegrity.hive.udf.ptyReprotectUnicode'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112110_4e3502b6-cbc4-4089-9420-cef27520de02); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112110_4e3502b6-cbc4-4089-9420-cef27520de02): CREATE FUNCTION ptyReprotectUnicode AS 'com.protegrity.hive.udf.ptyReprotectUnicode'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112110_4e3502b6-cbc4-4089-9420-cef27520de02); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.089 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectShort AS 'com.protegrity.hive.udf.ptyProtectShort';
INFO  : Compiling command(queryId=hive_20250916112110_58e9c7fa-3cda-4e52-9d49-7e048e29f6ac): CREATE FUNCTION ptyProtectShort AS 'com.protegrity.hive.udf.ptyProtectShort'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112110_58e9c7fa-3cda-4e52-9d49-7e048e29f6ac); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112110_58e9c7fa-3cda-4e52-9d49-7e048e29f6ac): CREATE FUNCTION ptyProtectShort AS 'com.protegrity.hive.udf.ptyProtectShort'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112110_58e9c7fa-3cda-4e52-9d49-7e048e29f6ac); Time taken: 0.018 seconds
INFO  : OK
No rows affected (0.089 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectShort AS 'com.protegrity.hive.udf.ptyUnprotectShort';
INFO  : Compiling command(queryId=hive_20250916112110_3c257d7e-2032-43b7-808f-c4bcd4e74435): CREATE FUNCTION ptyUnprotectShort AS 'com.protegrity.hive.udf.ptyUnprotectShort'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112110_3c257d7e-2032-43b7-808f-c4bcd4e74435); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112110_3c257d7e-2032-43b7-808f-c4bcd4e74435): CREATE FUNCTION ptyUnprotectShort AS 'com.protegrity.hive.udf.ptyUnprotectShort'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112110_3c257d7e-2032-43b7-808f-c4bcd4e74435); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.091 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectInt AS 'com.protegrity.hive.udf.ptyProtectInt';
INFO  : Compiling command(queryId=hive_20250916112110_368c452c-c0ab-4887-ac46-5bb5ff1b487a): CREATE FUNCTION ptyProtectInt AS 'com.protegrity.hive.udf.ptyProtectInt'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112110_368c452c-c0ab-4887-ac46-5bb5ff1b487a); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112110_368c452c-c0ab-4887-ac46-5bb5ff1b487a): CREATE FUNCTION ptyProtectInt AS 'com.protegrity.hive.udf.ptyProtectInt'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112110_368c452c-c0ab-4887-ac46-5bb5ff1b487a); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.089 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectInt AS 'com.protegrity.hive.udf.ptyUnprotectInt';
INFO  : Compiling command(queryId=hive_20250916112110_68178046-266b-4e42-834c-9fc2261c6b47): CREATE FUNCTION ptyUnprotectInt AS 'com.protegrity.hive.udf.ptyUnprotectInt'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112110_68178046-266b-4e42-834c-9fc2261c6b47); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112110_68178046-266b-4e42-834c-9fc2261c6b47): CREATE FUNCTION ptyUnprotectInt AS 'com.protegrity.hive.udf.ptyUnprotectInt'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112110_68178046-266b-4e42-834c-9fc2261c6b47); Time taken: 0.024 seconds
INFO  : OK
No rows affected (0.097 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectBigInt as 'com.protegrity.hive.udf.ptyProtectBigInt';
INFO  : Compiling command(queryId=hive_20250916112110_14c69433-7fcd-4b48-b5e2-a091b28dd22a): CREATE FUNCTION ptyProtectBigInt as 'com.protegrity.hive.udf.ptyProtectBigInt'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112110_14c69433-7fcd-4b48-b5e2-a091b28dd22a); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112110_14c69433-7fcd-4b48-b5e2-a091b28dd22a): CREATE FUNCTION ptyProtectBigInt as 'com.protegrity.hive.udf.ptyProtectBigInt'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112110_14c69433-7fcd-4b48-b5e2-a091b28dd22a); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.091 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectBigInt as 'com.protegrity.hive.udf.ptyUnprotectBigInt';
INFO  : Compiling command(queryId=hive_20250916112110_0a3767a6-4db2-4645-81c4-c4f874c7015c): CREATE FUNCTION ptyUnprotectBigInt as 'com.protegrity.hive.udf.ptyUnprotectBigInt'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112110_0a3767a6-4db2-4645-81c4-c4f874c7015c); Time taken: 0.028 seconds
INFO  : Executing command(queryId=hive_20250916112110_0a3767a6-4db2-4645-81c4-c4f874c7015c): CREATE FUNCTION ptyUnprotectBigInt as 'com.protegrity.hive.udf.ptyUnprotectBigInt'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112110_0a3767a6-4db2-4645-81c4-c4f874c7015c); Time taken: 0.02 seconds
INFO  : OK
No rows affected (0.091 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectFloat as 'com.protegrity.hive.udf.ptyProtectFloat';
INFO  : Compiling command(queryId=hive_20250916112110_3f894aa3-4bb4-47f4-ac7e-c24021dbc7fb): CREATE FUNCTION ptyProtectFloat as 'com.protegrity.hive.udf.ptyProtectFloat'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112110_3f894aa3-4bb4-47f4-ac7e-c24021dbc7fb); Time taken: 0.028 seconds
INFO  : Executing command(queryId=hive_20250916112110_3f894aa3-4bb4-47f4-ac7e-c24021dbc7fb): CREATE FUNCTION ptyProtectFloat as 'com.protegrity.hive.udf.ptyProtectFloat'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112110_3f894aa3-4bb4-47f4-ac7e-c24021dbc7fb); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.091 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectFloat as 'com.protegrity.hive.udf.ptyProtectFloat';
INFO  : Compiling command(queryId=hive_20250916112110_7a9ee32b-f391-4eff-a54e-53577961dd33): CREATE FUNCTION ptyUnprotectFloat as 'com.protegrity.hive.udf.ptyProtectFloat'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112110_7a9ee32b-f391-4eff-a54e-53577961dd33); Time taken: 0.028 seconds
INFO  : Executing command(queryId=hive_20250916112110_7a9ee32b-f391-4eff-a54e-53577961dd33): CREATE FUNCTION ptyUnprotectFloat as 'com.protegrity.hive.udf.ptyProtectFloat'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112110_7a9ee32b-f391-4eff-a54e-53577961dd33); Time taken: 0.023 seconds
INFO  : OK
No rows affected (0.096 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectDouble as 'com.protegrity.hive.udf.ptyProtectDouble';
INFO  : Compiling command(queryId=hive_20250916112111_3f9d2bfd-9fda-4af6-be16-d620370ccffc): CREATE FUNCTION ptyProtectDouble as 'com.protegrity.hive.udf.ptyProtectDouble'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112111_3f9d2bfd-9fda-4af6-be16-d620370ccffc); Time taken: 0.026 seconds
INFO  : Executing command(queryId=hive_20250916112111_3f9d2bfd-9fda-4af6-be16-d620370ccffc): CREATE FUNCTION ptyProtectDouble as 'com.protegrity.hive.udf.ptyProtectDouble'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112111_3f9d2bfd-9fda-4af6-be16-d620370ccffc); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.093 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectDouble as 'com.protegrity.hive.udf.ptyUnprotectDouble';
INFO  : Compiling command(queryId=hive_20250916112111_d4a442ea-3e40-4096-a760-caa65aaea9c1): CREATE FUNCTION ptyUnprotectDouble as 'com.protegrity.hive.udf.ptyUnprotectDouble'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112111_d4a442ea-3e40-4096-a760-caa65aaea9c1); Time taken: 0.028 seconds
INFO  : Executing command(queryId=hive_20250916112111_d4a442ea-3e40-4096-a760-caa65aaea9c1): CREATE FUNCTION ptyUnprotectDouble as 'com.protegrity.hive.udf.ptyUnprotectDouble'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112111_d4a442ea-3e40-4096-a760-caa65aaea9c1); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.09 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectDec as 'com.protegrity.hive.udf.ptyProtectDec';
INFO  : Compiling command(queryId=hive_20250916112111_f024de0b-ba3f-44ef-b2d6-1faa94ee0732): CREATE FUNCTION ptyProtectDec as 'com.protegrity.hive.udf.ptyProtectDec'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112111_f024de0b-ba3f-44ef-b2d6-1faa94ee0732); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112111_f024de0b-ba3f-44ef-b2d6-1faa94ee0732): CREATE FUNCTION ptyProtectDec as 'com.protegrity.hive.udf.ptyProtectDec'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112111_f024de0b-ba3f-44ef-b2d6-1faa94ee0732); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.087 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectDec as 'com.protegrity.hive.udf.ptyUnprotectDec';
INFO  : Compiling command(queryId=hive_20250916112111_1e7c1d77-ecdf-4ab1-b6e2-e0421b34f29c): CREATE FUNCTION ptyUnprotectDec as 'com.protegrity.hive.udf.ptyUnprotectDec'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112111_1e7c1d77-ecdf-4ab1-b6e2-e0421b34f29c); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112111_1e7c1d77-ecdf-4ab1-b6e2-e0421b34f29c): CREATE FUNCTION ptyUnprotectDec as 'com.protegrity.hive.udf.ptyUnprotectDec'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112111_1e7c1d77-ecdf-4ab1-b6e2-e0421b34f29c); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.087 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectHiveDecimal as 'com.protegrity.hive.udf.ptyProtectHiveDecimal';
INFO  : Compiling command(queryId=hive_20250916112111_0edaca35-32d5-49d9-9a3a-d6e644de1afd): CREATE FUNCTION ptyProtectHiveDecimal as 'com.protegrity.hive.udf.ptyProtectHiveDecimal'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112111_0edaca35-32d5-49d9-9a3a-d6e644de1afd); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112111_0edaca35-32d5-49d9-9a3a-d6e644de1afd): CREATE FUNCTION ptyProtectHiveDecimal as 'com.protegrity.hive.udf.ptyProtectHiveDecimal'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112111_0edaca35-32d5-49d9-9a3a-d6e644de1afd); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.088 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectHiveDecimal as 'com.protegrity.hive.udf.ptyUnprotectHiveDecimal';
INFO  : Compiling command(queryId=hive_20250916112111_399c586f-a1eb-4d1c-b750-d704a13061b1): CREATE FUNCTION ptyUnprotectHiveDecimal as 'com.protegrity.hive.udf.ptyUnprotectHiveDecimal'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112111_399c586f-a1eb-4d1c-b750-d704a13061b1); Time taken: 0.026 seconds
INFO  : Executing command(queryId=hive_20250916112111_399c586f-a1eb-4d1c-b750-d704a13061b1): CREATE FUNCTION ptyUnprotectHiveDecimal as 'com.protegrity.hive.udf.ptyUnprotectHiveDecimal'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112111_399c586f-a1eb-4d1c-b750-d704a13061b1); Time taken: 0.023 seconds
INFO  : OK
No rows affected (0.094 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectDate AS 'com.protegrity.hive.udf.ptyProtectDate';
INFO  : Compiling command(queryId=hive_20250916112111_0343ff3f-90a3-4b51-9276-1f53a95aae27): CREATE FUNCTION ptyProtectDate AS 'com.protegrity.hive.udf.ptyProtectDate'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112111_0343ff3f-90a3-4b51-9276-1f53a95aae27); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112111_0343ff3f-90a3-4b51-9276-1f53a95aae27): CREATE FUNCTION ptyProtectDate AS 'com.protegrity.hive.udf.ptyProtectDate'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112111_0343ff3f-90a3-4b51-9276-1f53a95aae27); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.086 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectDate AS 'com.protegrity.hive.udf.ptyUnprotectDate';
INFO  : Compiling command(queryId=hive_20250916112111_de190f84-ba1c-41e4-bdb3-e0efd091eb48): CREATE FUNCTION ptyUnprotectDate AS 'com.protegrity.hive.udf.ptyUnprotectDate'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112111_de190f84-ba1c-41e4-bdb3-e0efd091eb48); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112111_de190f84-ba1c-41e4-bdb3-e0efd091eb48): CREATE FUNCTION ptyUnprotectDate AS 'com.protegrity.hive.udf.ptyUnprotectDate'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112111_de190f84-ba1c-41e4-bdb3-e0efd091eb48); Time taken: 0.018 seconds
INFO  : OK
No rows affected (0.09 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectDateTime AS 'com.protegrity.hive.udf.ptyProtectDateTime';
INFO  : Compiling command(queryId=hive_20250916112111_0189db86-a9dc-4934-a15f-08d08aa238d5): CREATE FUNCTION ptyProtectDateTime AS 'com.protegrity.hive.udf.ptyProtectDateTime'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112111_0189db86-a9dc-4934-a15f-08d08aa238d5); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112111_0189db86-a9dc-4934-a15f-08d08aa238d5): CREATE FUNCTION ptyProtectDateTime AS 'com.protegrity.hive.udf.ptyProtectDateTime'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112111_0189db86-a9dc-4934-a15f-08d08aa238d5); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.088 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectDateTime AS 'com.protegrity.hive.udf.ptyUnprotectDateTime';
INFO  : Compiling command(queryId=hive_20250916112111_c629e410-5d0a-40ef-bb26-8daa6e3546bb): CREATE FUNCTION ptyUnprotectDateTime AS 'com.protegrity.hive.udf.ptyUnprotectDateTime'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112111_c629e410-5d0a-40ef-bb26-8daa6e3546bb); Time taken: 0.028 seconds
INFO  : Executing command(queryId=hive_20250916112111_c629e410-5d0a-40ef-bb26-8daa6e3546bb): CREATE FUNCTION ptyUnprotectDateTime AS 'com.protegrity.hive.udf.ptyUnprotectDateTime'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112111_c629e410-5d0a-40ef-bb26-8daa6e3546bb); Time taken: 0.024 seconds
INFO  : OK
No rows affected (0.094 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyProtectChar AS 'com.protegrity.hive.udf.ptyProtectChar';
INFO  : Compiling command(queryId=hive_20250916112112_be9b293c-6336-45d8-825a-480e31c54700): CREATE FUNCTION ptyProtectChar AS 'com.protegrity.hive.udf.ptyProtectChar'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112112_be9b293c-6336-45d8-825a-480e31c54700); Time taken: 0.026 seconds
INFO  : Executing command(queryId=hive_20250916112112_be9b293c-6336-45d8-825a-480e31c54700): CREATE FUNCTION ptyProtectChar AS 'com.protegrity.hive.udf.ptyProtectChar'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112112_be9b293c-6336-45d8-825a-480e31c54700); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.086 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyUnprotectChar AS 'com.protegrity.hive.udf.ptyUnprotectChar';
INFO  : Compiling command(queryId=hive_20250916112112_386a3434-a990-4180-a23a-f684fcfe391c): CREATE FUNCTION ptyUnprotectChar AS 'com.protegrity.hive.udf.ptyUnprotectChar'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112112_386a3434-a990-4180-a23a-f684fcfe391c); Time taken: 0.026 seconds
INFO  : Executing command(queryId=hive_20250916112112_386a3434-a990-4180-a23a-f684fcfe391c): CREATE FUNCTION ptyUnprotectChar AS 'com.protegrity.hive.udf.ptyUnprotectChar'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112112_386a3434-a990-4180-a23a-f684fcfe391c); Time taken: 0.016 seconds
INFO  : OK
No rows affected (0.086 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyStringEnc as 'com.protegrity.hive.udf.ptyStringEnc';
INFO  : Compiling command(queryId=hive_20250916112112_7f32e8f6-4688-41b8-b025-b6eb9ddb45fd): CREATE FUNCTION ptyStringEnc as 'com.protegrity.hive.udf.ptyStringEnc'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112112_7f32e8f6-4688-41b8-b025-b6eb9ddb45fd); Time taken: 0.026 seconds
INFO  : Executing command(queryId=hive_20250916112112_7f32e8f6-4688-41b8-b025-b6eb9ddb45fd): CREATE FUNCTION ptyStringEnc as 'com.protegrity.hive.udf.ptyStringEnc'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112112_7f32e8f6-4688-41b8-b025-b6eb9ddb45fd); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.087 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyStringDec as 'com.protegrity.hive.udf.ptyStringDec';
INFO  : Compiling command(queryId=hive_20250916112112_c2be0c4b-5208-4799-bcc3-f591ae30718a): CREATE FUNCTION ptyStringDec as 'com.protegrity.hive.udf.ptyStringDec'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112112_c2be0c4b-5208-4799-bcc3-f591ae30718a); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112112_c2be0c4b-5208-4799-bcc3-f591ae30718a): CREATE FUNCTION ptyStringDec as 'com.protegrity.hive.udf.ptyStringDec'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112112_c2be0c4b-5208-4799-bcc3-f591ae30718a); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.085 seconds)
0: jdbc:hive2://<master_node_name>> CREATE FUNCTION ptyStringReEnc as 'com.protegrity.hive.udf.ptyStringReEnc';
INFO  : Compiling command(queryId=hive_20250916112112_99ffe441-5bf8-4ab9-9746-78d43e1306eb): CREATE FUNCTION ptyStringReEnc as 'com.protegrity.hive.udf.ptyStringReEnc'
WARN  : permanent functions created without USING  clause will not be replicated.
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916112112_99ffe441-5bf8-4ab9-9746-78d43e1306eb); Time taken: 0.027 seconds
INFO  : Executing command(queryId=hive_20250916112112_99ffe441-5bf8-4ab9-9746-78d43e1306eb): CREATE FUNCTION ptyStringReEnc as 'com.protegrity.hive.udf.ptyStringReEnc'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916112112_99ffe441-5bf8-4ab9-9746-78d43e1306eb); Time taken: 0.016 seconds
INFO  : OK
No rows affected (0.088 seconds)

Registering the Impala user-defined functions

  1. Navigate to the S3 bucket where the installation files are uploaded.

  2. Open the createobjects.sql file.

  3. Copy the contents of the createobjects.sql file.

  4. Log in to the master node with a user account having permissions to create and drop UDFs.

  5. To create the UDFs using the helper script, run the following command:

    impala-shell -i <IP_Address_of_node> -k

  6. Press ENTER.

    The script creates all the required user-defined functions for Impala.

    
CREATE FUNCTION pty_getversion() RETURNS STRING
>   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
>   SYMBOL = 'pty_getversion';

SYMBOL = ‘pty_getversionextended’;

Query: CREATE FUNCTION pty_getversion() RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_getversion'
CREATE FUNCTION pty_whoami() RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_whoami';


CREATE FUNCTION pty_stringenc(STRING, STRING) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_stringenc' prepare_fn='UdfPrepare' close_fn='UdfClose';


CREATE FUNCTION pty_stringdec(STRING, STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_stringdec' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_stringins(STRING,STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_stringins' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_stringsel(STRING, STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_stringsel' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_unicodestringins(STRING,STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_unicodestringins' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_unicodestringsel(STRING,STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_unicodestringsel' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_unicodestringfpeins(STRING,STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_unicodestringfpeins' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_unicodestringfpesel(STRING,STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_unicodestringfpesel' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_integerenc(INTEGER, STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_integerenc' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_integerdec(STRING, STRING ) RETURNS INTEGER
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_integerdec' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_integerins(INTEGER, STRING ) RETURNS INTEGER
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_integerins' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_integersel(INTEGER, STRING ) RETURNS INTEGER
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_integersel' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_doubleenc(double, STRING ) RETURNS string
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_doubleenc' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_doubledec(STRING, STRING ) RETURNS double
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_doubledec' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_doubleins(double, STRING ) RETURNS double
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_doubleins' prepare_fn='UdfPrepare' close_fn='UdfClose';

CREATE FUNCTION pty_doublesel(DOUBLE, STRING ) RETURNS DOUBL+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.67s
 default>
>
> CREATE FUNCTION pty_getversionextended() RETURNS STRING
>   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
>   SYMBOL = 'pty_getversionextended';
Query: CREATE FUNCTION pty_getversionextended() RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_getversionextended'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.21s
 default>
f default>
l default> CREATE FUNCTION pty_whoami() RETURNS STRING
>   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
>   SYMBOL = 'pty_whoami';
Query: CREATE FUNCTION pty_whoami() RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_whoami'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.20s
 default>
                                                                                >
                                                                                > CREATE FUNCTION pty_stringenc(STRING, STRING) RETURNS STRING
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_stringenc' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_stringenc(STRING, STRING) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_stringenc' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.25s
 default>
                                                                                >
d                                                                                  > CREATE FUNCTION pty_stringdec(STRING, STRING ) RETURNS STRING
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_stringdec' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_stringdec(STRING, STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_stringdec' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.25s
 default>
 default> CREATE FUNCTION pty_stringins(STRING,STRING ) RETURNS STRING
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_stringins' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_stringins(STRING,STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_stringins' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.24s
 default>
 default> CREATE FUNCTION pty_stringsel(STRING, STRING ) RETURNS STRING
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_stringsel' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_stringsel(STRING, STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_stringsel' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.28s
 default>
U default> CREATE FUNCTION pty_unicodestringins(STRING,STRING ) RETURNS STRING
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_unicodestringins' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_unicodestringins(STRING,STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_unicodestringins' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.25s
 default>
 default> CREATE FUNCTION pty_unicodestringsel(STRING,STRING ) RETURNS STRING
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_unicodestringsel' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_unicodestringsel(STRING,STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_unicodestringsel' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.24s
 default>
 default> CREATE FUNCTION pty_unicodestringfpeins(STRING,STRING ) RETURNS STRING
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_unicodestringfpeins' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_unicodestringfpeins(STRING,STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_unicodestringfpeins' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.24s
 default>
 default> CREATE FUNCTION pty_unicodestringfpesel(STRING,STRING ) RETURNS STRING
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_unicodestringfpesel' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_unicodestringfpesel(STRING,STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_unicodestringfpesel' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.27s
 default>
 default> CREATE FUNCTION pty_integerenc(INTEGER, STRING ) RETURNS STRING
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_integerenc' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_integerenc(INTEGER, STRING ) RETURNS STRING
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_integerenc' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.27s
 default>
I default> CREATE FUNCTION pty_integerdec(STRING, STRING ) RETURNS INTEGER
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_integerdec' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_integerdec(STRING, STRING ) RETURNS INTEGER
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_integerdec' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.26s
 default>
t default> CREATE FUNCTION pty_integerins(INTEGER, STRING ) RETURNS INTEGER
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_integerins' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_integerins(INTEGER, STRING ) RETURNS INTEGER
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_integerins' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.26s
 default>
e default> CREATE FUNCTION pty_integersel(INTEGER, STRING ) RETURNS INTEGER
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_integersel' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_integersel(INTEGER, STRING ) RETURNS INTEGER
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_integersel' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.28s
 default>
, default> CREATE FUNCTION pty_doubleenc(double, STRING ) RETURNS string
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_doubleenc' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_doubleenc(double, STRING ) RETURNS string
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_doubleenc' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.26s
 default>
 default> CREATE FUNCTION pty_doubledec(STRING, STRING ) RETURNS double
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_doubledec' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_doubledec(STRING, STRING ) RETURNS double
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_doubledec' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.25s
 default>
 default> CREATE FUNCTION pty_doubleins(double, STRING ) RETURNS double
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_doubleins' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_doubleins(double, STRING ) RETURNS double
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_doubleins' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.28s
 default>
 default> CREATE FUNCTION pty_doublesel(DOUBLE, STRING ) RETURNS DOUBLE
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_doublesel' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_doublesel(DOUBLE, STRING ) RETURNS DOUBLE
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_doublesel' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.23s
 default>
 default> CREATE FUNCTION pty_floatenc(float, STRING ) RETURNS string
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_floatenc' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_floatenc(float, STRING ) RETURNS string
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_floatenc' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.27s
 default>
 default> CREATE FUNCTION pty_floatdec(STRING, STRING ) RETURNS float
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_floatdec' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_floatdec(STRING, STRING ) RETURNS float
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_floatdec' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.24s
 default>
 default> CREATE FUNCTION pty_floatins(float, STRING ) RETURNS float
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_floatins' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_floatins(float, STRING ) RETURNS float
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_floatins' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.29s
 default>
 default> CREATE FUNCTION pty_floatsel(float, STRING ) RETURNS float
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_floatsel' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_floatsel(float, STRING ) RETURNS float
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_floatsel' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.23s
 default>
 default> CREATE FUNCTION pty_smallintenc(smallint, STRING ) RETURNS string
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_smallintenc'  prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_smallintenc(smallint, STRING ) RETURNS string
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_smallintenc'  prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.24s
 default>
 default> CREATE FUNCTION pty_smallintdec(STRING, STRING ) RETURNS smallint
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_smallintdec' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_smallintdec(STRING, STRING ) RETURNS smallint
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_smallintdec' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.28s
 default>
 default> CREATE FUNCTION pty_smallintins(smallint, STRING ) RETURNS smallint
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_smallintins' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_smallintins(smallint, STRING ) RETURNS smallint
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_smallintins' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.25s
 default>
 default> CREATE FUNCTION pty_smallintsel(smallint, STRING ) RETURNS smallint
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >    SYMBOL = 'pty_smallintsel' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_smallintsel(smallint, STRING ) RETURNS smallint
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_smallintsel' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.24s
 default>
 default> CREATE FUNCTION pty_bigintenc(bigint, STRING) RETURNS string
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_bigintenc' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_bigintenc(bigint, STRING) RETURNS string
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_bigintenc' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.24s
 default>
 default> CREATE FUNCTION pty_bigintdec(STRING, STRING) RETURNS bigint
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_bigintdec' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_bigintdec(STRING, STRING) RETURNS bigint
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_bigintdec' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.25s
 default>
 default> CREATE FUNCTION pty_bigintins(bigint, STRING) RETURNS bigint
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_bigintins' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_bigintins(bigint, STRING) RETURNS bigint
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_bigintins' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.24s
 default>
 default> CREATE FUNCTION pty_bigintsel(bigint, STRING) RETURNS bigint
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_bigintsel' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_bigintsel(bigint, STRING) RETURNS bigint
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_bigintsel' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.24s
 default>
 default> CREATE FUNCTION pty_dateenc(date, STRING ) RETURNS string
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_dateenc' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_dateenc(date, STRING ) RETURNS string
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_dateenc' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.27s
 default>
 default> CREATE FUNCTION pty_datedec(STRING, STRING ) RETURNS date
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_datedec' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_datedec(STRING, STRING ) RETURNS date
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_datedec' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.24s
 default>
 default> CREATE FUNCTION pty_dateins(date, STRING ) RETURNS date
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_dateins' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_dateins(date, STRING ) RETURNS date
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_dateins' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.22s
 default>
 default> CREATE FUNCTION pty_datesel(date, STRING ) RETURNS date
                                                                                >   LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
                                                                                >   SYMBOL = 'pty_datesel' prepare_fn='UdfPrepare' close_fn='UdfClose';
Query: CREATE FUNCTION pty_datesel(date, STRING ) RETURNS date
LOCATION 's3a://<bucket_name>/<directory_name>/pepimpala/pepimpala4_0_RHEL.so'
SYMBOL = 'pty_datesel' prepare_fn='UdfPrepare' close_fn='UdfClose'
+----------------------------+
| summary                    |
+----------------------------+
| Function has been created. |
+----------------------------+
Fetched 1 row(s) in 0.24s

```

Registering the SparkSQL user-defined functions

  1. Log in to the master node with a user account having permissions to create and drop UDFs.

  2. To navigate to the directory that contains the helper script, run the following command:

    cd /opt/cloudera/parcels/PTY_BDP/pepspark/scripts

  3. To create the UDFs using the helper script, run the following command in the spark-shell:

    :load /opt/cloudera/parcels/PTY_BDP/pepspark/scripts/create_spark_sql_udfs.scala

  4. Press ENTER.

    The script creates all the required user-defined functions for SparkSQL.

    Loading /opt/cloudera/parcels/PTY_BDP/pepspark/scripts/create_spark_sql_udfs.scala...
res0: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2557/1214243533@e9f28,StringType,List(),Some(class[value[0]: string]),Some(ptyGetVersion),true,true)
res1: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2603/321785376@684ad81c,StringType,List(),Some(class[value[0]: string]),Some(ptyGetVersionExtended),true,true)
res2: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2604/289080194@594bedf5,StringType,List(),Some(class[value[0]: string]),Some(ptyWhoAmI),true,true)
res3: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2605/430442099@6ec6adcc,StringType,List(Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: string]),Some(ptyProtectStr),true,true)
res4: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2612/1566019818@55b678dc,StringType,List(Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: string]),Some(ptyUnprotectStr),true,true)
res5: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2613/1992744664@2dff4ef9,StringType,List(Some(class[value[0]: string]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: string]),Some(ptyReprotectStr),true,true)
res6: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2621/2144907913@4d13970d,StringType,List(Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: string]),Some(ptyProtectUnicode),true,true)
res7: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2622/567181258@7c8d4a94,StringType,List(Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: string]),Some(ptyUnprotectUnicode),true,true)
res8: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2623/1248911890@590eb2c5,StringType,List(Some(class[value[0]: string]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: string]),Some(ptyReprotectUnicode),true,true)
res9: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2639/1206966491@4e3617fe,ShortType,List(Some(class[value[0]: smallint]), Some(class[value[0]: string])),Some(class[value[0]: smallint]),Some(ptyProtectShort),false,true)
res10: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2643/1430577369@5056f8d7,ShortType,List(Some(class[value[0]: smallint]), Some(class[value[0]: string])),Some(class[value[0]: smallint]),Some(ptyUnprotectShort),false,true)
res11: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2644/1959246940@3e7d458a,ShortType,List(Some(class[value[0]: smallint]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: smallint]),Some(ptyReprotectShort),false,true)
res12: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2646/468430240@6b874125,IntegerType,List(Some(class[value[0]: int]), Some(class[value[0]: string])),Some(class[value[0]: int]),Some(ptyProtectInt),false,true)
res13: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2648/1849024377@377b8c99,IntegerType,List(Some(class[value[0]: int]), Some(class[value[0]: string])),Some(class[value[0]: int]),Some(ptyUnprotectInt),false,true)
res14: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2649/1850050643@1ddbf1b0,IntegerType,List(Some(class[value[0]: int]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: int]),Some(ptyReprotectInt),false,true)
res15: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2650/1751709974@65f23702,LongType,List(Some(class[value[0]: bigint]), Some(class[value[0]: string])),Some(class[value[0]: bigint]),Some(ptyProtectLong),false,true)
res16: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2652/1397163963@5d98ac30,LongType,List(Some(class[value[0]: bigint]), Some(class[value[0]: string])),Some(class[value[0]: bigint]),Some(ptyUnprotectLong),false,true)
res17: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2653/231449448@5ce648c7,LongType,List(Some(class[value[0]: bigint]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: bigint]),Some(ptyReprotectLong),false,true)
res18: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2654/916221467@203dff48,FloatType,List(Some(class[value[0]: float]), Some(class[value[0]: string])),Some(class[value[0]: float]),Some(ptyProtectFloat),false,true)
res19: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2656/1642716671@2403ecd0,FloatType,List(Some(class[value[0]: float]), Some(class[value[0]: string])),Some(class[value[0]: float]),Some(ptyUnprotectFloat),false,true)
res20: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2657/449484397@780f6346,FloatType,List(Some(class[value[0]: float]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: float]),Some(ptyReprotectFloat),false,true)
res21: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2658/311232024@4718da4b,DoubleType,List(Some(class[value[0]: double]), Some(class[value[0]: string])),Some(class[value[0]: double]),Some(ptyProtectDouble),false,true)
res22: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2660/1882823613@136e7e2c,DoubleType,List(Some(class[value[0]: double]), Some(class[value[0]: string])),Some(class[value[0]: double]),Some(ptyUnprotectDouble),false,true)
res23: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2661/1574577816@2f4f900d,DoubleType,List(Some(class[value[0]: double]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: double]),Some(ptyReprotectDouble),false,true)
res24: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2662/701508258@404d6f2,DateType,List(Some(class[value[0]: date]), Some(class[value[0]: string])),Some(class[value[0]: date]),Some(ptyProtectDate),true,true)
res25: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2673/1441934479@512f3e71,DateType,List(Some(class[value[0]: date]), Some(class[value[0]: string])),Some(class[value[0]: date]),Some(ptyUnprotectDate),true,true)
res26: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2674/19354823@7bacb1b0,DateType,List(Some(class[value[0]: date]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: date]),Some(ptyReprotectDate),true,true)
res27: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2675/1203531300@31fe39d3,TimestampType,List(Some(class[value[0]: timestamp]), Some(class[value[0]: string])),Some(class[value[0]: timestamp]),Some(ptyProtectDateTime),true,true)
res28: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2676/1395761147@5d81b1ef,TimestampType,List(Some(class[value[0]: timestamp]), Some(class[value[0]: string])),Some(class[value[0]: timestamp]),Some(ptyUnprotectDateTime),true,true)
res29: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2677/971152222@1af59a5e,TimestampType,List(Some(class[value[0]: timestamp]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: timestamp]),Some(ptyReprotectDateTime),true,true)
res30: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2678/449445798@4f994c53,DecimalType(38,18),List(Some(class[value[0]: decimal(38,18)]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: decimal(38,18)]),Some(ptyProtectDecimal),true,true)
res31: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2687/375594857@7f5ae905,DecimalType(38,18),List(Some(class[value[0]: decimal(38,18)]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: decimal(38,18)]),Some(ptyUnprotectDecimal),true,true)
res32: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2688/2133807474@33f1f5a,DecimalType(38,18),List(Some(class[value[0]: decimal(38,18)]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: decimal(38,18)]),Some(ptyReprotectDecimal),true,true)
res33: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2691/1933809761@d57894d,BinaryType,List(Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: binary]),Some(ptyStringEnc),true,true)
res34: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2693/255369243@25ed9699,StringType,List(Some(class[value[0]: binary]), Some(class[value[0]: string])),Some(class[value[0]: string]),Some(ptyStringDec),true,true)
res35: org.apache.spark.sql.expressions.UserDefinedFunction = SparkUserDefinedFunction($Lambda$2694/542980564@7382cd26,BinaryType,List(Some(class[value[0]: binary]), Some(class[value[0]: string]), Some(class[value[0]: string])),Some(class[value[0]: binary]),Some(ptyStringReEnc),true,true)

Registering the PySpark Scala Wrapper user-defined functions

  1. Log in to the master node with a user account having permissions to create and drop UDFs.

  2. To navigate to the directory that contains the helper script, run the following command:

    cd /opt/cloudera/parcels/PTY_BDP/pepspark/scripts

  3. To create the UDFs using the helper script, run the following command in the pyspark shell:

    exec(open("/opt/cloudera/parcels/PTY_BDP/pepspark/scripts/create_scala_wrapper_udfs.py").read());

  4. Press ENTER.

    The script creates all the required Scala Wrapper user-defined functions.

5.4 - Updating the Configuration Parameters

The Big Data Protector provides the following files that contain different parameters to control the protector behavior:

  • config.ini - provides parameters to control the protector behavior.
  • rpagent.cfg - provides parameters to control the RPAgent behavior.

Updating the configuration parameters for the Log Forwarder:

  1. Using a browser, log in to the Cloudera Manager.

  2. Click BDP PEP. The BDP PEP page appears.

  3. Click the Configuration tab. The Configuration tab appears.

  4. In the Filters pane, under Scope, click PTY Log Forwarder. The options related to the Log Forwarder appear.

  5. Update the parameters, as per the descriptions, listed in the following table:

OptionDescription
Audit Store TypeSpecifies the type of Audit Store(s) where PTY LogForwarder sends logs to.
Protegrity Audit Store List
of Hostnames/IP Addresses and/or Ports		Is the comma-delimited List of Protegrity Audit Store appliances’ Hostnames/IP addresses and/or Ports where LogForwarder sends logs.

Allowed Syntax:
hostname[:port][,hostname[:port],hostname[:port]…]
(By default 9200 is set for empty ports)

Examples:
auditstore-a:9200,auditstore-b:9201,auditstore-c:9202
hostname-a
hostname-a,hostname-b,hostname-c
hostname-a:9201,hostname-b,hostname-c,hostname-d

When using only External Audit Store, set this to NA.
LogForwarder Log LevelSpecifies the LogForwarder logging verbosity level.
Enable Generation of a Log File
for Application Logs		Enables the logforwarder/data/config.d/out_applog_file.conf file to create an Application Log file locally on the Nodes.
Application Log File Directory PathSpecifies the directory Path on the Nodes to store Application Log File. This is set as value of ‘Path’ in out_applog_file.conf when ’enable_applog_file’ is true.
Application Log File NameSpecifies the name of the Application Log File. This is set as value of ‘File’ in out_applog_file.conf when ’enable_applog_file’ is true.

Updating the configuration parameters for the RPAgent:

  1. Using a browser, navigate to the Cloudera Manager screen. The Cloudera Manager Home page appears.

  2. Click BDP PEP. The BDP PEP page appears.

  3. Click the Configuration tab. The Configuration tab appears.

  4. In the Filters pane, under Scope, click PTY RPAgent. The options related to the RPAgent appear.

  5. Update the parameters, as per the descriptions, listed in the following table:

OptionDescription
RPA Sync Interval (Seconds)Specifies the frequency at which the RPAgent will fetch the policy from ESA. The minimum value is 1 second and the maximum value is 86400 seconds.
RPA Sync Hostname/IP AddressSpecifies the hostname/IP Address to the service that provides the resilient packages.
RPA Sync PortSpecifies the port to the service that provides the resilient packages.
RPA Sync CA Certificate PathSpecfies the path to the CA certificate to validate the server certificate. Note: Do not modify the value of this parameter.
RPA Sync Client Certificate PathSpecifies the path to the client certificate. Note: Do not modify the value of this parameter.
RPA Sync Client Certificate Key PathSpecifies the path to the client certificate key. Note: Do not modify the value of this parameter.
RPA Sync Client Certificate Key Secret File PathSpecifies the path to the secret file used to decrypt the client certificate key. Note: Do not modify the value of this parameter.
RPA Log HostSpecifies the LogForwarder Host/IP Address where logs will be forwarded from the RPA.
RPA Log ModeIn case that connection to LogForwarder is lost, set how logs are handled.
drop = (Default) Protector throws logs away if connection to the logforwarder is lost
error = Protector returns error without protecting/unprotecting data if connection to the logforwarder is lost.

To update the configuration parameters in the config.ini file:

  1. Using a browser, log in to the Cloudera Manager UI. The Cloudera Manager Home page appears.

  2. Click BDP PEP. The BDP PEP page appears.

  3. Click the Configuration tab. The Configuration tab appears.

  4. In the Filters pane, under Scope, click Gateway. The options related to the config.ini file appear.

  5. Update the parameters, as per the descriptions, listed in the following table:

ParameterDescription
Protector CadenceDetermines how often the protector’s sync thread will execute (in seconds). The default is 60 seconds. By default, every 60 seconds the protector attempts to fetch the policy updates. If the cadence is set to ‘0’, then the protector will get the policy only once (per process). The interval is reset when the previous sync is finished.
Minimum Value = 0 sec
Maximum Value = 86400 sec (i.e. 24 hours)
Log OutputDefines the output type for protections logs.
Accepted values are:
- tcp = (Default) Logs are sent to LogForwarder using tcp
- stdout = Logs are sent to stdout.
Log HostSpecifies the LogForwarder Host/IP Address where logs will be forwarded from the protector.
Log ModeDetermines the approach to handle logs when the connection to the LogForwarder is lost.
This setting is only for the protector logs and not application logs.
- drop = (Default) Protector throws logs away if connection to the logforwarder is lost.
- error = Protector returns error without protecting/unprotecting data if connection to the logforwarder is lost.
Deploy DirectorySpecifies the directory where the client configs will be deployed.
Note: The Gateway Role requires this parameter to stage the temporary files (like the config.ini.properties). The default value is set to /etc/protegrity-bdp/.
BDP PEP Client Advanced Configuration Snippet (Safety Valve)
for bdp-conf/config.ini.properties		For advanced use only, a string to be inserted into the client configuration for bdp-conf/config.ini.properties.
Log PortSpecifies the LogForwarder port where logs will be forwarded from the protector.

Note: If you add or modify any parameter in the config.ini file, then you must restart all the dependent services to reload the configuration changes.

To add a new configuration parameter in the config.ini file:

  1. Using a browser, log in to the Cloudera Manager UI. The Cloudera Manager Home page appears.

  2. Click BDP PEP. The BDP PEP page appears.

  3. Click the Configuration tab. The Configuration tab appears.

  4. In the Filters pane, under Scope, click Gateway. The options related to the config.ini file appear.

  5. To add a new parameter for the config.ini file, perform the following steps:

    1. Under the BDP PEP Client Advanced Configuration Snippet (Safety Valve) for bdp-conf/config.ini.properties box, enter the required parameter and the corresponding value in the group.key=value format. When you enter the parameter in the group.key=value format, Cloudera Manager appends the parameter in the config.ini file on all the nodes in the following format:
      [group]
key = value
    2. Click Save Changes (CTRL+S).

  6. To verify whether the parameter is added to the config.ini file, perform the following steps:

    1. Log in to the Master Node.
    2. To navigate to the /opt/cloudera/parcels/PTY_BDP/bdp/data/ directory, run the following command:
      cd /opt/cloudera/parcels/PTY_BDP/bdp/data/
    3. Press ENTER. The command changes the working directory to /opt/cloudera/parcels/PTY_BDP/bdp/data/.
    4. To view the contents of the config.ini file, run the following command:
      vim config.ini
    5. Press ENTER. The command displays the contents of the config.ini file.
      [log]
host=localhost
port=15780
output=tcp
mode=drop
[protector]
cadence=60
[core]
emptystring=empty

  7. Using a browser, login to the Cloudera Manager home page.

  8. Click BDP PEP. The BDP PEP page appears.

  9. To generate the config.ini file on the nodes where you have installed the Gateway Role, select Actions » Deploy Client Configuration. The prompt to confirm the action appears.

  10. Click Deploy Client Configuration.

    Cloudera Manager generates the config.ini file to all the nodes where the Gateway role is installed.

Note: If you add or modify any parameter in the config.ini file, then you must restart all the dependent services to reload the configuration changes.

5.5 - Updating the Certificates Parcel

If you changed ESA, with which the Big Data Protector is configured, then the Certificates parcel must be updated with the new certificates. The updated Certificates parcel must be utilized by all the nodes in the cluster.

To update the certificates for the PTY_CERT parcel:

  1. Log in to the staging machine.

  2. Nagivate to the directory where you extracted the installation files.

  3. To execute the configurator script, run the following command:

    ./BDPConfigurator_CDP-AWS-DataHub-7.3_<BDP_Version>.sh

  4. Press ENTER. The prompt to continue the installation appears.

    *******************************************************************************
            Welcome to the Big Data Protector Configurator Wizard
*******************************************************************************
This will setup the Big Data Protector Installation Files for CDP AWS Data Hub.

Do you want to continue? [yes or no]:

  5. To continue, type yes.

  6. Press ENTER. The prompt to select the type of installation files appears.

    Big Data Protector Configurator started...
Unpacking...
Extracting files...


Select the type of Installation files you want to generate.
[ 1: Create All ]      : Creates entire Big Data Protector CSDs, Parcels, Recipes and other files.
[ 2: Update PTY_CERT ] : Creates new PTY_CERT parcel with an incremented patch version.
                        Use this if you have updated the ESA certificates.
[ 3: Update PTY_LOGFORWARDER_CONF ]
                    : Creates new PTY_LOGFORWARDER_CONF parcel with an incremented patch version.
                        Use this if you want to set Custom LogForwarder configuration files to
                        forward logs to an External Audit Store.

[ 1, 2 or 3 ]:

  7. To update the PTY_CERT parcel, type 2.

  8. Press ENTER. The prompt to select the operating system version for the parcel appears.

    Select the OS version for Cloudera Manager Parcel.
This will be used as the OS Distro suffix in the Parcel name.

[ 1: el7 ]    :  RHEL 7 and clones (CentOS, Scientific Linux, etc)
[ 2: el8 ]    :  RHEL 8 and clones (CentOS, Scientific Linux, etc)
[ 3: el9 ]    :  RHEL 9 and clones (CentOS, Scientific Linux, etc)
[ 4: sles12 ] :  SuSE Linux Enterprise Server 12.x
[ 5: sles15 ] :  SuSE Linux Enterprise Server 15.x

Enter the no.:

  9. Depending on the requirements, type 1, 2, 3, 4 or 5 to select the operating system version for the Big Data Protector parcels.

  10. Press ENTER. The prompt to enter the S3 URI to upload the installation files appears.

    Enter the S3 URI where the BDP Installation files are to be uploaded.

(E.g. s3://examplebucket/folder):

  11. Enter the location of the S3 bucket to host the installation files.

  12. Press ENTER. The prompt to select the upload type appears.

    Choose one option among the following for BDP Installation files:
[ 1 ] : Upload files to 's3://<bucket_name>/<directory_name>/' S3 URI.
[ 2 ] : Generate files locally to current working directory. (You would have to manually upload the files to the specified S3 URI)

[ 1 or 2 ]:

  13. To upload the installation files to the S3 bucket, type 1.

  14. Press ENTER. The prompt to select the type of AWS access keys appears.

    Choose the Type of AWS Access Keys from the following options:
[ 1 ] : IAM User Access Keys (Permanent access key id & secret access key)
[ 2 ] : Temporary Security Credentials (Temporary access key id, secret access key & session token)

[ 1 or 2 ]:

  15. Depending upon the authentication option, the script will prompt for the following inputs:

    OptionDescription
    1Prompts to enter the following permanent IAM user access keys:
    AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY
    2Prompts to enter the following temporary security credentials:
    AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY
    AWS_SESSION_TOKEN

  16. Enter the required credentials.

  17. Press ENTER. The prompt to enter ESA hostname or IP address appears.

    Enter ESA Hostname or IP Address:

  18. Enter ESA IP address.

  19. Press ENTER. The prompt to enter ESA listening port appears.

    Enter ESA host listening port [8443]:

  20. Enter the listening port number.

  21. Press ENTER.

    The prompt to enter ESA JSON Web Token appears.

    If you have an existing ESA JSON Web Token (JWT) with Export Certificates role, enter it otherwise enter 'no':

    Note: The script silently reads the user input. Therefore, the user will be unable to see the entered JWT or no.

  22. Enter the JWT token. a. If you do not have an existing ESA JSON Web Token (JWT), type no.

    b. Press ENTER.
    The prompt to enter the user name with Export Certificates permission appears.

    ```
JWT was not provided. Script will now prompt for ESA username and password.
Enter ESA Username with Export Certificates role:
```

    c. Enter the username that has permissions to export the certificates.

    d. Press ENTER.
    The prompt to enter the password appears. e. Enter the password.

  23. Press ENTER. The script fetches the JWT token from ESA, generates the installation files and the prompt to enter the current version of the PTY_CERT parcel appears.

    Fetching JWT from ESA....

Fetching Certificates from ESA....

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100 11264  100 11264    0     0   233k      0 --:--:-- --:--:-- --:--:--  234k

-------------------------------------------------------------------------------


Generating Installation files...



NOTE:
You can verify the version of the activated PTY_CERT parcel from the parcel
name, such as PTY_CERT-x.x.x.x_CDPx.x.p<version>-<os>.parcel, where the
<version> parameter denotes the patch version of the PTY_CERT parcel.

For Example: If the current activated PTY_CERT parcel is
PTY_CERT-x.x.x.x_CDPx.x.p0-<os>.parcel, the patch version of the PTY_CERT
parcel will be 0. Do NOT include 'p' while specifying the version.

Enter the <version> of the current PTY_CERT Parcel as specified in the parcel name [0]:

  24. Enter the current activated patch version of the PTY_CERT parcel.

  25. Press ENTER. The script updates the PTY_CERT parcel and uploads them to the S3 bucket.

    ****************************************************************************************************************************************

Retrieving the S3 bucket's AWS Region via AWS S3 REST API...
Successfully retrieved S3 bucket's AWS region: <region_name>


Started uploading the updated PTY_CERT parcel to S3 bucket using REST API.

Uploading PTY_CERT-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel...
-> File uploaded to s3://<bucket_name>/<directory_name>/UpdatedCERTParcel/PTY_CERT-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel

Uploading PTY_CERT-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel.sha...
-> File uploaded to s3://<bucket_name>/<directory_name>/UpdatedCERTParcel/PTY_CERT-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel.sha

Successfully uploaded the updated PTY_CERT parcel to S3 URI: s3://<bucket_name>/<directory_name>

****************************************************************************************************************************************

* The updated PTY_CERT parcel 'PTY_CERT-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel' and sha1 checksum are locally generated in ./Installation_Files/ directory.

****************************************************************************************************************************************

Successfully configured the Updated PTY_CERT parcel for CDP AWS DataHub.

    If you use the option to locally generate the installation files, the script generates them under the ./Installation_Files/ directory.

    ****************************************************************************************************************************************

* The updated PTY_CERT parcel 'PTY_CERT-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel' and sha1 checksum are locally generated in ./Installation_Files/ directory.
-> Manually copy the PTY_CERT-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel and .sha files to Cloudera Manager Server's local parcel repository on the existing running Data Hub cluster.

****************************************************************************************************************************************

Successfully configured the Updated PTY_CERT parcel for CDP AWS DataHub.

5.6 - Updating the Log Forwarder Parcel

If you want to use a newer set of custom Log Forwarder configuration files to send the logs to an External Audit Store, then you must update, distribute, and activate the PTY_LOGFORWARDER_CONF parcel on all the nodes in the cluster.

To create the updated log forwarder parcel:

  1. Log in to the staging machine.

  2. Nagivate to the directory where you extracted the installation files.

  3. To execute the configurator script, run the following command:

    ./BDPConfigurator_CDP-AWS-DataHub-7.3_<BDP_Version>.sh

  4. Press ENTER. The prompt to continue the installation appears.

    *******************************************************************************
            Welcome to the Big Data Protector Configurator Wizard
*******************************************************************************
This will setup the Big Data Protector Installation Files for CDP AWS Data Hub.

Do you want to continue? [yes or no]:

  5. To continue, type yes.

  6. Press ENTER. The prompt to select the type of installation files appears.

    Big Data Protector Configurator started...
Unpacking...
Extracting files...


Select the type of Installation files you want to generate.
[ 1: Create All ]      : Creates entire Big Data Protector CSDs, Parcels, Recipes and other files.
[ 2: Update PTY_CERT ] : Creates new PTY_CERT parcel with an incremented patch version.
                        Use this if you have updated the ESA certificates.
[ 3: Update PTY_LOGFORWARDER_CONF ]
                    : Creates new PTY_LOGFORWARDER_CONF parcel with an incremented patch version.
                        Use this if you want to set Custom LogForwarder configuration files to
                        forward logs to an External Audit Store.

[ 1, 2 or 3 ]:

  7. To update the PTY_LOGFORWARDER_CONF parcel, type 3.

  8. Press ENTER. The prompt to select the operating system version for the parcel appears.

    Select the OS version for Cloudera Manager Parcel.
This will be used as the OS Distro suffix in the Parcel name.

[ 1: el7 ]    :  RHEL 7 and clones (CentOS, Scientific Linux, etc)
[ 2: el8 ]    :  RHEL 8 and clones (CentOS, Scientific Linux, etc)
[ 3: el9 ]    :  RHEL 9 and clones (CentOS, Scientific Linux, etc)
[ 4: sles12 ] :  SuSE Linux Enterprise Server 12.x
[ 5: sles15 ] :  SuSE Linux Enterprise Server 15.x

Enter the no.:

  9. Depending on the requirements, type 1, 2, 3, 4 or 5 to select the operating system version for the Big Data Protector parcels.

  10. Press ENTER. The prompt to enter the S3 URI to upload the installation files appears.

    Enter the S3 URI where the BDP Installation files are to be uploaded.

(E.g. s3://examplebucket/folder):

  11. Enter the location of the S3 bucket to host the installation files.

  12. Press ENTER. The prompt to select the upload type appears.

    Choose one option among the following for BDP Installation files:
[ 1 ] : Upload files to 's3://<bucket_name>/<directory_name>/' S3 URI.
[ 2 ] : Generate files locally to current working directory. (You would have to manually upload the files to the specified S3 URI)

[ 1 or 2 ]:

  13. To upload the installation files to the S3 bucket, type 1.

  14. Press ENTER. The prompt to select the type of AWS access keys appears.

    Choose the Type of AWS Access Keys from the following options:
[ 1 ] : IAM User Access Keys (Permanent access key id & secret access key)
[ 2 ] : Temporary Security Credentials (Temporary access key id, secret access key & session token)

[ 1 or 2 ]:

  15. Depending upon the authentication option, the script will prompt for the following inputs:

    OptionDescription
    1Prompts to enter the following permanent IAM user access keys:
    AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY
    2Prompts to enter the following temporary security credentials:
    AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY
    AWS_SESSION_TOKEN

  16. Enter the required credentials.

  17. Press ENTER.

    The prompt to enter local directory path that stores the LogForwarder configuration files for External Audit Store appears.

    Enter the local directory path on this machine that stores the LogForwarder configuration files for External Audit Store:

  18. Enter the location where the Log Forwarder configuration files are stored.

  19. Press ENTER. The script generates the installation files and the prompt to enter the current version of the PTY_LOGFORWARDER_CONF parcel appears.

    Generating Installation files...

NOTE:
You can verify the version of the activated PTY_LOGFORWARDER_CONF parcel from the parcel
name, such as PTY_LOGFORWARDER_CONF-x.x.x.x_CDPx.x.p<version>-<os>.parcel, where the
<version> parameter denotes the patch version of the PTY_LOGFORWARDER_CONF parcel.

For Example: If the current activated PTY_LOGFORWARDER_CONF parcel is
PTY_LOGFORWARDER_CONF-x.x.x.x_CDPx.x.p0-<os>.parcel, the patch version of the PTY_LOGFORWARDER_CONF
parcel will be 0. Do NOT include 'p' while specifying the version.

Enter the <version> of the current PTY_LOGFORWARDER_CONF Parcel as specified in the parcel name [0]:

  20. Enter the current activated patch version of the PTY_LOGFORWARDER_CONF parcel.

  21. Press ENTER. The script updates the PTY_LOGFORWARDER_CONF parcel and uploads them to the S3 bucket.

    ****************************************************************************************************************************************

Retrieving the S3 bucket's AWS Region via AWS S3 REST API...
Successfully retrieved S3 bucket's AWS region: <region_name>


Started uploading the updated PTY_LOGFORWARDER_CONF parcel to S3 bucket using REST API.

Uploading PTY_LOGFORWARDER_CONF-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel...
-> File uploaded to s3://<bucket_name>/<directory_name>/Updated_LOGFORWARDER_CONF_Parcel/PTY_LOGFORWARDER_CONF-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel

Uploading PTY_LOGFORWARDER_CONF-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel.sha...
-> File uploaded to s3://<bucket_name>/<directory_name>/Updated_LOGFORWARDER_CONF_Parcel/PTY_LOGFORWARDER_CONF-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel.sha

Successfully uploaded the updated PTY_LOGFORWARDER_CONF parcel to S3 URI: s3://<bucket_name>/<directory_name>

****************************************************************************************************************************************

* The updated PTY_CERT parcel 'PTY_LOGFORWARDER_CONF-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel' and sha1 checksum are locally generated in ./Installation_Files/ directory.

****************************************************************************************************************************************

Successfully configured the Updated PTY_LOGFORWARDER_CONF parcel for CDP AWS DataHub.

    If you use the option to locally generate the installation files, the script generates them under the ./Installation_Files/ directory.

    ****************************************************************************************************************************************

* The updated PTY_LOGFORWARDER_CONF parcel 'PTY_LOGFORWARDER_CONF-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel' and sha1 checksum are locally generated in ./Installation_Files/ directory.
-> Manually copy the PTY_LOGFORWARDER_CONF-<BDP_version>_CDP7.3.p<patch_version>-<operating_system_version>.parcel and .sha files to Cloudera Manager Server's local parcel repository on the existing running Data Hub cluster.

****************************************************************************************************************************************

Successfully configured the Updated PTY_LOGFORWARDER_CONF parcel for CDP AWS DataHub.

6 - Uninstalling the Big Data Protector

6.1 - Drop the UDFs using the Helper Script

The Big Data Protector build provides helper scripts to drop the user-defined functions for the following components:

  • Hive
    • Temporary UDFs
    • Permanent UDFs
  • Impala

To create the UDFs using the helper script, refer Installing the UDFs using the Helper Script.

Dropping the Temporary Hive user-defined functions

  1. Log in to the master node with a user account having permissions to create and drop UDFs.

  2. To navigate to the directory that contains the helper script, run the following command:

    cd /opt/cloudera/parcels/PTY_BDP/pephive/scripts

  3. To drop the UDFs using the helper script, run the following command:

    beeline -f drop_temp_hive_udfs.hql;

    Execute the command in beeline after establishing a connection.

  4. Press ENTER.

    The script drops all the temporary user-defined functions for Hive.

    Connected to: Apache Hive (version 3.1.3000.7.3.1.400-100)
Driver: Hive JDBC (version 3.1.3000.7.3.1.400-100)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyGetVersion;
INFO  : Compiling command(queryId=hive_20250916121826_101cd2f3-a216-4786-a37c-15ce02258a51): DROP TEMPORARY FUNCTION IF EXISTS ptyGetVersion
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121826_101cd2f3-a216-4786-a37c-15ce02258a51); Time taken: 0.026 seconds
INFO  : Executing command(queryId=hive_20250916121826_101cd2f3-a216-4786-a37c-15ce02258a51): DROP TEMPORARY FUNCTION IF EXISTS ptyGetVersion
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121826_101cd2f3-a216-4786-a37c-15ce02258a51); Time taken: 0.0 seconds
INFO  : OK
No rows affected (0.08 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyGetVersionExtended;
INFO  : Compiling command(queryId=hive_20250916121826_57bf7665-53ce-408d-b5f1-58a77d313de9): DROP TEMPORARY FUNCTION IF EXISTS ptyGetVersionExtended
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121826_57bf7665-53ce-408d-b5f1-58a77d313de9); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121826_57bf7665-53ce-408d-b5f1-58a77d313de9): DROP TEMPORARY FUNCTION IF EXISTS ptyGetVersionExtended
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121826_57bf7665-53ce-408d-b5f1-58a77d313de9); Time taken: 0.0 seconds
INFO  : OK
No rows affected (0.051 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyWhoAmI;
INFO  : Compiling command(queryId=hive_20250916121826_f143b8ad-6e53-47fe-bff2-0938a6fd2bc0): DROP TEMPORARY FUNCTION IF EXISTS ptyWhoAmI
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121826_f143b8ad-6e53-47fe-bff2-0938a6fd2bc0); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121826_f143b8ad-6e53-47fe-bff2-0938a6fd2bc0): DROP TEMPORARY FUNCTION IF EXISTS ptyWhoAmI
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121826_f143b8ad-6e53-47fe-bff2-0938a6fd2bc0); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.051 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectStr;
INFO  : Compiling command(queryId=hive_20250916121826_29c19376-89f5-4615-8a89-719ac9bbdeb3): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectStr
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121826_29c19376-89f5-4615-8a89-719ac9bbdeb3); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121826_29c19376-89f5-4615-8a89-719ac9bbdeb3): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectStr
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121826_29c19376-89f5-4615-8a89-719ac9bbdeb3); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.05 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectStr;
INFO  : Compiling command(queryId=hive_20250916121826_0aba5da5-4612-45be-83f0-d486be5c75ff): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectStr
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121826_0aba5da5-4612-45be-83f0-d486be5c75ff); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121826_0aba5da5-4612-45be-83f0-d486be5c75ff): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectStr
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121826_0aba5da5-4612-45be-83f0-d486be5c75ff); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.053 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyReprotect;
INFO  : Compiling command(queryId=hive_20250916121826_44bf7ea2-4b1a-4cf0-8399-df419470ec30): DROP TEMPORARY FUNCTION IF EXISTS ptyReprotect
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121826_44bf7ea2-4b1a-4cf0-8399-df419470ec30); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121826_44bf7ea2-4b1a-4cf0-8399-df419470ec30): DROP TEMPORARY FUNCTION IF EXISTS ptyReprotect
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121826_44bf7ea2-4b1a-4cf0-8399-df419470ec30); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.053 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectUnicode;
INFO  : Compiling command(queryId=hive_20250916121827_5a628700-978d-4851-8422-21e47c0886b8): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectUnicode
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_5a628700-978d-4851-8422-21e47c0886b8); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121827_5a628700-978d-4851-8422-21e47c0886b8): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectUnicode
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_5a628700-978d-4851-8422-21e47c0886b8); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.052 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectUnicode;
INFO  : Compiling command(queryId=hive_20250916121827_984f396a-376a-4b12-af37-9c0f66e3808c): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectUnicode
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_984f396a-376a-4b12-af37-9c0f66e3808c); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121827_984f396a-376a-4b12-af37-9c0f66e3808c): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectUnicode
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_984f396a-376a-4b12-af37-9c0f66e3808c); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.052 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyReprotectUnicode;
INFO  : Compiling command(queryId=hive_20250916121827_d5d8a4c4-4a06-4449-92a3-27e1544001be): DROP TEMPORARY FUNCTION IF EXISTS ptyReprotectUnicode
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_d5d8a4c4-4a06-4449-92a3-27e1544001be); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121827_d5d8a4c4-4a06-4449-92a3-27e1544001be): DROP TEMPORARY FUNCTION IF EXISTS ptyReprotectUnicode
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_d5d8a4c4-4a06-4449-92a3-27e1544001be); Time taken: 0.002 seconds
INFO  : OK
No rows affected (0.052 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectShort;
INFO  : Compiling command(queryId=hive_20250916121827_2425a29e-73a9-4f57-9db0-ff2d6569e6c2): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectShort
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_2425a29e-73a9-4f57-9db0-ff2d6569e6c2); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121827_2425a29e-73a9-4f57-9db0-ff2d6569e6c2): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectShort
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_2425a29e-73a9-4f57-9db0-ff2d6569e6c2); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.049 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectShort;
INFO  : Compiling command(queryId=hive_20250916121827_2f9f8568-a59b-4c52-8d71-8e387dab11ab): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectShort
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_2f9f8568-a59b-4c52-8d71-8e387dab11ab); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121827_2f9f8568-a59b-4c52-8d71-8e387dab11ab): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectShort
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_2f9f8568-a59b-4c52-8d71-8e387dab11ab); Time taken: 0.002 seconds
INFO  : OK
No rows affected (0.049 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectInt;
INFO  : Compiling command(queryId=hive_20250916121827_95d7f52f-8582-45da-91f0-a484a86414c7): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectInt
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_95d7f52f-8582-45da-91f0-a484a86414c7); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121827_95d7f52f-8582-45da-91f0-a484a86414c7): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectInt
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_95d7f52f-8582-45da-91f0-a484a86414c7); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.051 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectInt;
INFO  : Compiling command(queryId=hive_20250916121827_a53f2fb8-fa11-4cf3-8393-dac63edb3dc6): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectInt
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_a53f2fb8-fa11-4cf3-8393-dac63edb3dc6); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121827_a53f2fb8-fa11-4cf3-8393-dac63edb3dc6): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectInt
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_a53f2fb8-fa11-4cf3-8393-dac63edb3dc6); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.048 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectBigInt;
INFO  : Compiling command(queryId=hive_20250916121827_28c686fd-b6ac-4055-8d41-1ed029c83527): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectBigInt
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_28c686fd-b6ac-4055-8d41-1ed029c83527); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121827_28c686fd-b6ac-4055-8d41-1ed029c83527): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectBigInt
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_28c686fd-b6ac-4055-8d41-1ed029c83527); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.049 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectBigInt;
INFO  : Compiling command(queryId=hive_20250916121827_9ae22908-7c59-41f2-8673-1a20fbc7869e): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectBigInt
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_9ae22908-7c59-41f2-8673-1a20fbc7869e); Time taken: 0.022 seconds
INFO  : Executing command(queryId=hive_20250916121827_9ae22908-7c59-41f2-8673-1a20fbc7869e): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectBigInt
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_9ae22908-7c59-41f2-8673-1a20fbc7869e); Time taken: 0.0 seconds
INFO  : OK
No rows affected (0.049 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectFloat;
INFO  : Compiling command(queryId=hive_20250916121827_8079847b-adf7-4037-8081-62ae5ef4e58d): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectFloat
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_8079847b-adf7-4037-8081-62ae5ef4e58d); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121827_8079847b-adf7-4037-8081-62ae5ef4e58d): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectFloat
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_8079847b-adf7-4037-8081-62ae5ef4e58d); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.05 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectFloat;
INFO  : Compiling command(queryId=hive_20250916121827_eb8f298e-159d-4f65-8bd7-2ff7fdbef448): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectFloat
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_eb8f298e-159d-4f65-8bd7-2ff7fdbef448); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121827_eb8f298e-159d-4f65-8bd7-2ff7fdbef448): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectFloat
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_eb8f298e-159d-4f65-8bd7-2ff7fdbef448); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.05 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDouble;
INFO  : Compiling command(queryId=hive_20250916121827_648877a6-f5ca-486e-a9cb-89dd9ca3e1af): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDouble
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_648877a6-f5ca-486e-a9cb-89dd9ca3e1af); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121827_648877a6-f5ca-486e-a9cb-89dd9ca3e1af): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDouble
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_648877a6-f5ca-486e-a9cb-89dd9ca3e1af); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.051 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDouble;
INFO  : Compiling command(queryId=hive_20250916121827_45e35621-b0b2-4326-b2aa-628d36877f32): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDouble
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_45e35621-b0b2-4326-b2aa-628d36877f32); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121827_45e35621-b0b2-4326-b2aa-628d36877f32): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDouble
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_45e35621-b0b2-4326-b2aa-628d36877f32); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.048 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDec;
INFO  : Compiling command(queryId=hive_20250916121827_e7c905c2-f1dc-4b9e-8362-10dc8adae336): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDec
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_e7c905c2-f1dc-4b9e-8362-10dc8adae336); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121827_e7c905c2-f1dc-4b9e-8362-10dc8adae336): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDec
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_e7c905c2-f1dc-4b9e-8362-10dc8adae336); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.048 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDec;
INFO  : Compiling command(queryId=hive_20250916121827_11138a33-ee0c-4ec7-b20c-b8d4b607d835): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDec
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_11138a33-ee0c-4ec7-b20c-b8d4b607d835); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121827_11138a33-ee0c-4ec7-b20c-b8d4b607d835): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDec
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_11138a33-ee0c-4ec7-b20c-b8d4b607d835); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.049 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectHiveDecimal;
INFO  : Compiling command(queryId=hive_20250916121827_8f5dc74e-1b2d-4c10-be12-8afb8ef4fb40): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectHiveDecimal
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_8f5dc74e-1b2d-4c10-be12-8afb8ef4fb40); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121827_8f5dc74e-1b2d-4c10-be12-8afb8ef4fb40): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectHiveDecimal
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_8f5dc74e-1b2d-4c10-be12-8afb8ef4fb40); Time taken: 0.0 seconds
INFO  : OK
No rows affected (0.048 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectHiveDecimal;
INFO  : Compiling command(queryId=hive_20250916121827_bbd4e88a-dbf2-4aa0-9749-a124ffba6b64): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectHiveDecimal
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_bbd4e88a-dbf2-4aa0-9749-a124ffba6b64); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121827_bbd4e88a-dbf2-4aa0-9749-a124ffba6b64): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectHiveDecimal
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_bbd4e88a-dbf2-4aa0-9749-a124ffba6b64); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.05 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDate;
INFO  : Compiling command(queryId=hive_20250916121827_cb758dc8-020c-49b1-8d3f-e19d83cb204e): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDate
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121827_cb758dc8-020c-49b1-8d3f-e19d83cb204e); Time taken: 0.022 seconds
INFO  : Executing command(queryId=hive_20250916121827_cb758dc8-020c-49b1-8d3f-e19d83cb204e): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDate
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121827_cb758dc8-020c-49b1-8d3f-e19d83cb204e); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.05 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDate;
INFO  : Compiling command(queryId=hive_20250916121828_31a95bdd-b3c4-4f58-847e-f556fd289f66): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDate
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121828_31a95bdd-b3c4-4f58-847e-f556fd289f66); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121828_31a95bdd-b3c4-4f58-847e-f556fd289f66): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDate
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121828_31a95bdd-b3c4-4f58-847e-f556fd289f66); Time taken: 0.0 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDateTime;
INFO  : Compiling command(queryId=hive_20250916121828_bd36b2da-7078-4583-9597-7e9a4ff1ca50): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDateTime
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121828_bd36b2da-7078-4583-9597-7e9a4ff1ca50); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121828_bd36b2da-7078-4583-9597-7e9a4ff1ca50): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectDateTime
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121828_bd36b2da-7078-4583-9597-7e9a4ff1ca50); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.048 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDateTime;
INFO  : Compiling command(queryId=hive_20250916121828_096e0c55-f4ca-4a2f-937c-26cff1b80019): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDateTime
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121828_096e0c55-f4ca-4a2f-937c-26cff1b80019); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121828_096e0c55-f4ca-4a2f-937c-26cff1b80019): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectDateTime
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121828_096e0c55-f4ca-4a2f-937c-26cff1b80019); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.046 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyProtectChar;
INFO  : Compiling command(queryId=hive_20250916121828_4c654530-0c2e-44f4-a06b-c1e3ef976c04): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectChar
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121828_4c654530-0c2e-44f4-a06b-c1e3ef976c04); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121828_4c654530-0c2e-44f4-a06b-c1e3ef976c04): DROP TEMPORARY FUNCTION IF EXISTS ptyProtectChar
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121828_4c654530-0c2e-44f4-a06b-c1e3ef976c04); Time taken: 0.0 seconds
INFO  : OK
No rows affected (0.049 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectChar;
INFO  : Compiling command(queryId=hive_20250916121828_879d9212-e8c5-41c1-8f3d-565b4c372661): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectChar
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121828_879d9212-e8c5-41c1-8f3d-565b4c372661); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121828_879d9212-e8c5-41c1-8f3d-565b4c372661): DROP TEMPORARY FUNCTION IF EXISTS ptyUnprotectChar
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121828_879d9212-e8c5-41c1-8f3d-565b4c372661); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.048 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyStringEnc;
INFO  : Compiling command(queryId=hive_20250916121828_279e5622-8188-42df-8768-450706ace0d3): DROP TEMPORARY FUNCTION IF EXISTS ptyStringEnc
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121828_279e5622-8188-42df-8768-450706ace0d3); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121828_279e5622-8188-42df-8768-450706ace0d3): DROP TEMPORARY FUNCTION IF EXISTS ptyStringEnc
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121828_279e5622-8188-42df-8768-450706ace0d3); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.048 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyStringDec;
INFO  : Compiling command(queryId=hive_20250916121828_2ed2f7bb-1122-409a-80a7-4db99e4f33cf): DROP TEMPORARY FUNCTION IF EXISTS ptyStringDec
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121828_2ed2f7bb-1122-409a-80a7-4db99e4f33cf); Time taken: 0.02 seconds
INFO  : Executing command(queryId=hive_20250916121828_2ed2f7bb-1122-409a-80a7-4db99e4f33cf): DROP TEMPORARY FUNCTION IF EXISTS ptyStringDec
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121828_2ed2f7bb-1122-409a-80a7-4db99e4f33cf); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.049 seconds)
0: jdbc:hive2://<master_node_name>> DROP TEMPORARY FUNCTION IF EXISTS ptyStringReEnc;
INFO  : Compiling command(queryId=hive_20250916121828_a63059b5-fdc3-4647-9b1e-3dd8e4110793): DROP TEMPORARY FUNCTION IF EXISTS ptyStringReEnc
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916121828_a63059b5-fdc3-4647-9b1e-3dd8e4110793); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20250916121828_a63059b5-fdc3-4647-9b1e-3dd8e4110793): DROP TEMPORARY FUNCTION IF EXISTS ptyStringReEnc
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916121828_a63059b5-fdc3-4647-9b1e-3dd8e4110793); Time taken: 0.001 seconds
INFO  : OK
No rows affected (0.049 seconds)

Dropping the Permanent Hive user-defined functions

  1. Log in to the master node with a user account having permissions to create and drop UDFs.

  2. To navigate to the directory that contains the helper script, run the following command:

    cd /opt/cloudera/parcels/PTY_BDP/pephive/scripts

  3. To drop the UDFs using the helper script, run the following command:

    beeline -f drop_perm_hive_udfs.hql;

    Execute the command in beeline after establishing a connection.

  4. Press ENTER.

    The script drops all the temporary user-defined functions for Hive.

    Connected to: Apache Hive (version 3.1.3000.7.3.1.400-100)
Driver: Hive JDBC (version 3.1.3000.7.3.1.400-100)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyGetVersion;
INFO  : Compiling command(queryId=hive_20250916111817_b8a10f66-84a3-4ef8-97a9-b5510f4128a7): DROP FUNCTION IF EXISTS ptyGetVersion
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111817_b8a10f66-84a3-4ef8-97a9-b5510f4128a7); Time taken: 0.098 seconds
INFO  : Executing command(queryId=hive_20250916111817_b8a10f66-84a3-4ef8-97a9-b5510f4128a7): DROP FUNCTION IF EXISTS ptyGetVersion
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111817_b8a10f66-84a3-4ef8-97a9-b5510f4128a7); Time taken: 0.023 seconds
INFO  : OK
No rows affected (0.2 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyGetVersionExtended;
INFO  : Compiling command(queryId=hive_20250916111818_2e8451ab-8ff6-4513-a9fe-6e3589c353de): DROP FUNCTION IF EXISTS ptyGetVersionExtended
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111818_2e8451ab-8ff6-4513-a9fe-6e3589c353de); Time taken: 0.036 seconds
INFO  : Executing command(queryId=hive_20250916111818_2e8451ab-8ff6-4513-a9fe-6e3589c353de): DROP FUNCTION IF EXISTS ptyGetVersionExtended
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111818_2e8451ab-8ff6-4513-a9fe-6e3589c353de); Time taken: 0.021 seconds
INFO  : OK
No rows affected (0.109 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyWhoAmI;
INFO  : Compiling command(queryId=hive_20250916111818_a718bf41-376a-465e-92eb-361cbd720e03): DROP FUNCTION IF EXISTS ptyWhoAmI
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111818_a718bf41-376a-465e-92eb-361cbd720e03); Time taken: 0.037 seconds
INFO  : Executing command(queryId=hive_20250916111818_a718bf41-376a-465e-92eb-361cbd720e03): DROP FUNCTION IF EXISTS ptyWhoAmI
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111818_a718bf41-376a-465e-92eb-361cbd720e03); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.108 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectStr;
INFO  : Compiling command(queryId=hive_20250916111818_03046d2d-3661-465b-bd94-488d7d9340fa): DROP FUNCTION IF EXISTS ptyProtectStr
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111818_03046d2d-3661-465b-bd94-488d7d9340fa); Time taken: 0.035 seconds
INFO  : Executing command(queryId=hive_20250916111818_03046d2d-3661-465b-bd94-488d7d9340fa): DROP FUNCTION IF EXISTS ptyProtectStr
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111818_03046d2d-3661-465b-bd94-488d7d9340fa); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.1 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectStr;
INFO  : Compiling command(queryId=hive_20250916111818_ba5bdcd1-5aa6-4cdb-9fe4-9cfc7ffd7fa7): DROP FUNCTION IF EXISTS ptyUnprotectStr
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111818_ba5bdcd1-5aa6-4cdb-9fe4-9cfc7ffd7fa7); Time taken: 0.036 seconds
INFO  : Executing command(queryId=hive_20250916111818_ba5bdcd1-5aa6-4cdb-9fe4-9cfc7ffd7fa7): DROP FUNCTION IF EXISTS ptyUnprotectStr
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111818_ba5bdcd1-5aa6-4cdb-9fe4-9cfc7ffd7fa7); Time taken: 0.026 seconds
INFO  : OK
No rows affected (0.109 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyReprotect;
INFO  : Compiling command(queryId=hive_20250916111818_89549fd0-dad0-4571-9c59-e02ed8510d1e): DROP FUNCTION IF EXISTS ptyReprotect
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111818_89549fd0-dad0-4571-9c59-e02ed8510d1e); Time taken: 0.034 seconds
INFO  : Executing command(queryId=hive_20250916111818_89549fd0-dad0-4571-9c59-e02ed8510d1e): DROP FUNCTION IF EXISTS ptyReprotect
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111818_89549fd0-dad0-4571-9c59-e02ed8510d1e); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.098 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectUnicode;
INFO  : Compiling command(queryId=hive_20250916111818_4977c128-c3fc-495a-8ce7-b7893f513cbc): DROP FUNCTION IF EXISTS ptyProtectUnicode
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111818_4977c128-c3fc-495a-8ce7-b7893f513cbc); Time taken: 0.034 seconds
INFO  : Executing command(queryId=hive_20250916111818_4977c128-c3fc-495a-8ce7-b7893f513cbc): DROP FUNCTION IF EXISTS ptyProtectUnicode
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111818_4977c128-c3fc-495a-8ce7-b7893f513cbc); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.095 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectUnicode;
INFO  : Compiling command(queryId=hive_20250916111818_6840a7c6-c020-486e-b433-ba36f96e6f2b): DROP FUNCTION IF EXISTS ptyUnprotectUnicode
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111818_6840a7c6-c020-486e-b433-ba36f96e6f2b); Time taken: 0.034 seconds
INFO  : Executing command(queryId=hive_20250916111818_6840a7c6-c020-486e-b433-ba36f96e6f2b): DROP FUNCTION IF EXISTS ptyUnprotectUnicode
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111818_6840a7c6-c020-486e-b433-ba36f96e6f2b); Time taken: 0.018 seconds
INFO  : OK
No rows affected (0.097 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyReprotectUnicode;
INFO  : Compiling command(queryId=hive_20250916111818_f8812304-56e6-45f0-a110-196aba4ac5ba): DROP FUNCTION IF EXISTS ptyReprotectUnicode
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111818_f8812304-56e6-45f0-a110-196aba4ac5ba); Time taken: 0.033 seconds
INFO  : Executing command(queryId=hive_20250916111818_f8812304-56e6-45f0-a110-196aba4ac5ba): DROP FUNCTION IF EXISTS ptyReprotectUnicode
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111818_f8812304-56e6-45f0-a110-196aba4ac5ba); Time taken: 0.02 seconds
INFO  : OK
No rows affected (0.098 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectShort;
INFO  : Compiling command(queryId=hive_20250916111819_8a470cd1-fcd5-4a10-9ff3-4bef8c78c766): DROP FUNCTION IF EXISTS ptyProtectShort
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111819_8a470cd1-fcd5-4a10-9ff3-4bef8c78c766); Time taken: 0.034 seconds
INFO  : Executing command(queryId=hive_20250916111819_8a470cd1-fcd5-4a10-9ff3-4bef8c78c766): DROP FUNCTION IF EXISTS ptyProtectShort
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111819_8a470cd1-fcd5-4a10-9ff3-4bef8c78c766); Time taken: 0.018 seconds
INFO  : OK
No rows affected (0.095 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectShort;
INFO  : Compiling command(queryId=hive_20250916111819_806eb112-e38c-4b5d-b083-4826eb5b5912): DROP FUNCTION IF EXISTS ptyUnprotectShort
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111819_806eb112-e38c-4b5d-b083-4826eb5b5912); Time taken: 0.034 seconds
INFO  : Executing command(queryId=hive_20250916111819_806eb112-e38c-4b5d-b083-4826eb5b5912): DROP FUNCTION IF EXISTS ptyUnprotectShort
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111819_806eb112-e38c-4b5d-b083-4826eb5b5912); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.097 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectInt;
INFO  : Compiling command(queryId=hive_20250916111819_76429e06-e925-4848-84b0-ca327d46c0ca): DROP FUNCTION IF EXISTS ptyProtectInt
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111819_76429e06-e925-4848-84b0-ca327d46c0ca); Time taken: 0.033 seconds
INFO  : Executing command(queryId=hive_20250916111819_76429e06-e925-4848-84b0-ca327d46c0ca): DROP FUNCTION IF EXISTS ptyProtectInt
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111819_76429e06-e925-4848-84b0-ca327d46c0ca); Time taken: 0.02 seconds
INFO  : OK
No rows affected (0.098 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectInt;
INFO  : Compiling command(queryId=hive_20250916111819_1bd75b89-99ac-4736-8bba-dce41cf5fb6c): DROP FUNCTION IF EXISTS ptyUnprotectInt
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111819_1bd75b89-99ac-4736-8bba-dce41cf5fb6c); Time taken: 0.033 seconds
INFO  : Executing command(queryId=hive_20250916111819_1bd75b89-99ac-4736-8bba-dce41cf5fb6c): DROP FUNCTION IF EXISTS ptyUnprotectInt
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111819_1bd75b89-99ac-4736-8bba-dce41cf5fb6c); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.094 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectBigInt;
INFO  : Compiling command(queryId=hive_20250916111819_98cdfb4e-f6bc-4b85-869b-c612330f2d92): DROP FUNCTION IF EXISTS ptyProtectBigInt
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111819_98cdfb4e-f6bc-4b85-869b-c612330f2d92); Time taken: 0.033 seconds
INFO  : Executing command(queryId=hive_20250916111819_98cdfb4e-f6bc-4b85-869b-c612330f2d92): DROP FUNCTION IF EXISTS ptyProtectBigInt
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111819_98cdfb4e-f6bc-4b85-869b-c612330f2d92); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.097 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectBigInt;
INFO  : Compiling command(queryId=hive_20250916111819_30e6b595-f4fa-4de9-a766-3e25a043f225): DROP FUNCTION IF EXISTS ptyUnprotectBigInt
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111819_30e6b595-f4fa-4de9-a766-3e25a043f225); Time taken: 0.032 seconds
INFO  : Executing command(queryId=hive_20250916111819_30e6b595-f4fa-4de9-a766-3e25a043f225): DROP FUNCTION IF EXISTS ptyUnprotectBigInt
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111819_30e6b595-f4fa-4de9-a766-3e25a043f225); Time taken: 0.027 seconds
INFO  : OK
No rows affected (0.102 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectFloat;
INFO  : Compiling command(queryId=hive_20250916111819_67431a43-e4a1-4c26-8065-893e4f627698): DROP FUNCTION IF EXISTS ptyProtectFloat
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111819_67431a43-e4a1-4c26-8065-893e4f627698); Time taken: 0.034 seconds
INFO  : Executing command(queryId=hive_20250916111819_67431a43-e4a1-4c26-8065-893e4f627698): DROP FUNCTION IF EXISTS ptyProtectFloat
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111819_67431a43-e4a1-4c26-8065-893e4f627698); Time taken: 0.018 seconds
INFO  : OK
No rows affected (0.095 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectFloat;
INFO  : Compiling command(queryId=hive_20250916111819_a1764be9-0d63-4321-97ca-da0698546a1f): DROP FUNCTION IF EXISTS ptyUnprotectFloat
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111819_a1764be9-0d63-4321-97ca-da0698546a1f); Time taken: 0.033 seconds
INFO  : Executing command(queryId=hive_20250916111819_a1764be9-0d63-4321-97ca-da0698546a1f): DROP FUNCTION IF EXISTS ptyUnprotectFloat
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111819_a1764be9-0d63-4321-97ca-da0698546a1f); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.092 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectDouble;
INFO  : Compiling command(queryId=hive_20250916111819_4d9317c3-f0bd-4365-9aaf-dfc5a4b13a4d): DROP FUNCTION IF EXISTS ptyProtectDouble
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111819_4d9317c3-f0bd-4365-9aaf-dfc5a4b13a4d); Time taken: 0.034 seconds
INFO  : Executing command(queryId=hive_20250916111819_4d9317c3-f0bd-4365-9aaf-dfc5a4b13a4d): DROP FUNCTION IF EXISTS ptyProtectDouble
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111819_4d9317c3-f0bd-4365-9aaf-dfc5a4b13a4d); Time taken: 0.018 seconds
INFO  : OK
No rows affected (0.095 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectDouble;
INFO  : Compiling command(queryId=hive_20250916111820_0a7a7710-e708-4841-aa2b-97d1fbaebd7c): DROP FUNCTION IF EXISTS ptyUnprotectDouble
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111820_0a7a7710-e708-4841-aa2b-97d1fbaebd7c); Time taken: 0.106 seconds
INFO  : Executing command(queryId=hive_20250916111820_0a7a7710-e708-4841-aa2b-97d1fbaebd7c): DROP FUNCTION IF EXISTS ptyUnprotectDouble
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111820_0a7a7710-e708-4841-aa2b-97d1fbaebd7c); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.213 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectDec;
INFO  : Compiling command(queryId=hive_20250916111820_17fe182d-da8f-4dee-93af-b5c9b2f3ad96): DROP FUNCTION IF EXISTS ptyProtectDec
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111820_17fe182d-da8f-4dee-93af-b5c9b2f3ad96); Time taken: 0.034 seconds
INFO  : Executing command(queryId=hive_20250916111820_17fe182d-da8f-4dee-93af-b5c9b2f3ad96): DROP FUNCTION IF EXISTS ptyProtectDec
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111820_17fe182d-da8f-4dee-93af-b5c9b2f3ad96); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.094 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectDec;
INFO  : Compiling command(queryId=hive_20250916111820_90ec86dc-60ff-45d7-9487-d913cb9b20b3): DROP FUNCTION IF EXISTS ptyUnprotectDec
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111820_90ec86dc-60ff-45d7-9487-d913cb9b20b3); Time taken: 0.035 seconds
INFO  : Executing command(queryId=hive_20250916111820_90ec86dc-60ff-45d7-9487-d913cb9b20b3): DROP FUNCTION IF EXISTS ptyUnprotectDec
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111820_90ec86dc-60ff-45d7-9487-d913cb9b20b3); Time taken: 0.02 seconds
INFO  : OK
No rows affected (0.099 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectHiveDecimal;
INFO  : Compiling command(queryId=hive_20250916111820_12369750-83da-4c2b-a8f0-5e8c09c385a5): DROP FUNCTION IF EXISTS ptyProtectHiveDecimal
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111820_12369750-83da-4c2b-a8f0-5e8c09c385a5); Time taken: 0.034 seconds
INFO  : Executing command(queryId=hive_20250916111820_12369750-83da-4c2b-a8f0-5e8c09c385a5): DROP FUNCTION IF EXISTS ptyProtectHiveDecimal
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111820_12369750-83da-4c2b-a8f0-5e8c09c385a5); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.094 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectHiveDecimal;
INFO  : Compiling command(queryId=hive_20250916111820_7d44f289-86c6-4475-9cb5-58c9d947104f): DROP FUNCTION IF EXISTS ptyUnprotectHiveDecimal
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111820_7d44f289-86c6-4475-9cb5-58c9d947104f); Time taken: 0.033 seconds
INFO  : Executing command(queryId=hive_20250916111820_7d44f289-86c6-4475-9cb5-58c9d947104f): DROP FUNCTION IF EXISTS ptyUnprotectHiveDecimal
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111820_7d44f289-86c6-4475-9cb5-58c9d947104f); Time taken: 0.018 seconds
INFO  : OK
No rows affected (0.095 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectDate;
INFO  : Compiling command(queryId=hive_20250916111820_c8fde45f-0bb2-4a3e-a77f-7ebc1e4942f6): DROP FUNCTION IF EXISTS ptyProtectDate
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111820_c8fde45f-0bb2-4a3e-a77f-7ebc1e4942f6); Time taken: 0.033 seconds
INFO  : Executing command(queryId=hive_20250916111820_c8fde45f-0bb2-4a3e-a77f-7ebc1e4942f6): DROP FUNCTION IF EXISTS ptyProtectDate
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111820_c8fde45f-0bb2-4a3e-a77f-7ebc1e4942f6); Time taken: 0.017 seconds
INFO  : OK
No rows affected (0.092 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectDate;
INFO  : Compiling command(queryId=hive_20250916111820_b2d9505c-2dc8-4c6d-a1cd-5ced95f962a1): DROP FUNCTION IF EXISTS ptyUnprotectDate
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111820_b2d9505c-2dc8-4c6d-a1cd-5ced95f962a1); Time taken: 0.036 seconds
INFO  : Executing command(queryId=hive_20250916111820_b2d9505c-2dc8-4c6d-a1cd-5ced95f962a1): DROP FUNCTION IF EXISTS ptyUnprotectDate
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111820_b2d9505c-2dc8-4c6d-a1cd-5ced95f962a1); Time taken: 0.025 seconds
INFO  : OK
No rows affected (0.103 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectDateTime;
INFO  : Compiling command(queryId=hive_20250916111820_822e4610-ee0c-4ada-9c7e-f083c800747e): DROP FUNCTION IF EXISTS ptyProtectDateTime
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111820_822e4610-ee0c-4ada-9c7e-f083c800747e); Time taken: 0.033 seconds
INFO  : Executing command(queryId=hive_20250916111820_822e4610-ee0c-4ada-9c7e-f083c800747e): DROP FUNCTION IF EXISTS ptyProtectDateTime
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111820_822e4610-ee0c-4ada-9c7e-f083c800747e); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.094 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectDateTime;
INFO  : Compiling command(queryId=hive_20250916111820_43e080a3-85e8-4394-96b0-15929e45170d): DROP FUNCTION IF EXISTS ptyUnprotectDateTime
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111820_43e080a3-85e8-4394-96b0-15929e45170d); Time taken: 0.034 seconds
INFO  : Executing command(queryId=hive_20250916111820_43e080a3-85e8-4394-96b0-15929e45170d): DROP FUNCTION IF EXISTS ptyUnprotectDateTime
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111820_43e080a3-85e8-4394-96b0-15929e45170d); Time taken: 0.02 seconds
INFO  : OK
No rows affected (0.096 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyProtectChar;
INFO  : Compiling command(queryId=hive_20250916111821_052aef90-c79b-4ae1-be75-0f8a1598d226): DROP FUNCTION IF EXISTS ptyProtectChar
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111821_052aef90-c79b-4ae1-be75-0f8a1598d226); Time taken: 0.033 seconds
INFO  : Executing command(queryId=hive_20250916111821_052aef90-c79b-4ae1-be75-0f8a1598d226): DROP FUNCTION IF EXISTS ptyProtectChar
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111821_052aef90-c79b-4ae1-be75-0f8a1598d226); Time taken: 0.018 seconds
INFO  : OK
No rows affected (0.092 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyUnprotectChar;
INFO  : Compiling command(queryId=hive_20250916111821_4a231b4c-8186-4b6c-9146-baa00b16fb71): DROP FUNCTION IF EXISTS ptyUnprotectChar
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111821_4a231b4c-8186-4b6c-9146-baa00b16fb71); Time taken: 0.035 seconds
INFO  : Executing command(queryId=hive_20250916111821_4a231b4c-8186-4b6c-9146-baa00b16fb71): DROP FUNCTION IF EXISTS ptyUnprotectChar
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111821_4a231b4c-8186-4b6c-9146-baa00b16fb71); Time taken: 0.018 seconds
INFO  : OK
No rows affected (0.096 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyStringEnc;
INFO  : Compiling command(queryId=hive_20250916111821_8f1a9315-3ee0-4791-a3f0-b970ef004883): DROP FUNCTION IF EXISTS ptyStringEnc
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111821_8f1a9315-3ee0-4791-a3f0-b970ef004883); Time taken: 0.033 seconds
INFO  : Executing command(queryId=hive_20250916111821_8f1a9315-3ee0-4791-a3f0-b970ef004883): DROP FUNCTION IF EXISTS ptyStringEnc
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111821_8f1a9315-3ee0-4791-a3f0-b970ef004883); Time taken: 0.018 seconds
INFO  : OK
No rows affected (0.093 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyStringDec;
INFO  : Compiling command(queryId=hive_20250916111821_4da146eb-02e1-4534-b37f-9718594facaf): DROP FUNCTION IF EXISTS ptyStringDec
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111821_4da146eb-02e1-4534-b37f-9718594facaf); Time taken: 0.033 seconds
INFO  : Executing command(queryId=hive_20250916111821_4da146eb-02e1-4534-b37f-9718594facaf): DROP FUNCTION IF EXISTS ptyStringDec
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111821_4da146eb-02e1-4534-b37f-9718594facaf); Time taken: 0.019 seconds
INFO  : OK
No rows affected (0.095 seconds)
0: jdbc:hive2://<master_node_name>> DROP FUNCTION IF EXISTS ptyStringReEnc;
INFO  : Compiling command(queryId=hive_20250916111821_39f635f8-f1dc-42ae-8180-7f13cea6d8b9): DROP FUNCTION IF EXISTS ptyStringReEnc
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250916111821_39f635f8-f1dc-42ae-8180-7f13cea6d8b9); Time taken: 0.034 seconds
INFO  : Executing command(queryId=hive_20250916111821_39f635f8-f1dc-42ae-8180-7f13cea6d8b9): DROP FUNCTION IF EXISTS ptyStringReEnc
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hive_20250916111821_39f635f8-f1dc-42ae-8180-7f13cea6d8b9); Time taken: 0.02 seconds
INFO  : OK
No rows affected (0.096 seconds)

Dropping the Impala user-defined functions

  1. Log in to the master node with a user account having permissions to create and drop UDFs.

  2. To navigate to the directory that contains the helper script, run the following command:

    cd /opt/cloudera/parcels/PTY_BDP/pepimpala/sqlscripts

  3. To create the UDFs using the helper script, run the following command:

    impala-shell -i node1 -k -f dropobjects.sql

  4. Press ENTER.

    The script drops all the user-defined functions for Impala.

    default> DROP FUNCTION pty_getversion();
Query: DROP FUNCTION pty_getversion()
DROP FUNCTION pty_getversionextended();
DROP FUNCTION pty_whoami();

-- string UDFs ------
DROP FUNCTION pty_stringenc( STRING, STRING );
DROP FUNCTION pty_stringdec( STRING, STRING );
DROP FUNCTION pty_stringins( STRING, STRING );
DROP FUNCTION pty_unicodestringins( STRING, STRING );
DROP FUNCTION pty_unicodestringfpeins( STRING, STRING );
DROP FUNCTION pty_stringsel( STRING, STRING );
DROP FUNCTION pty_unicodestringsel( STRING, STRING );
DROP FUNCTION pty_unicodestringfpesel( STRING, STRING );

--- Integer Udfs -----------------------------
DROP FUNCTION pty_integerenc( INTEGER, STRING);
DROP FUNCTION pty_integerdec( STRING, STRING);
DROP FUNCTION pty_integerins( INTEGER, STRING);
DROP FUNCTION pty_integersel( INTEGER, STRING);

--------------double udfs ----------------------
DROP FUNCTION pty_doubleenc( double, string);
DROP FUNCTION pty_doubledec( string, string);
DROP FUNCTION pty_doubleins( double, string);
DROP FUNCTION pty_doublesel( double, string);

-------------float udfs -------------------------

DROP FUNCTION pty_floatenc( float, string);
DROP FUNCTION pty_floatdec( string, string);
DROP FUNCTION pty_floatins( float, string);
DROP FUNCTION pty_floatsel( float, string);

-------------bigint udfs ------------------------

DROP FUNCTION pty_bigintenc( bigint, string);
DROP FUNCTION pty_bigintdec( string, string);
DROP FUNCTION pty_bigintins( bigint, string);
DROP FUNCTION pty_bigintsel( bigint, string);

-------------date udfs --------------------------

DROP FUNCTION pty_dateenc( date, string);
DROP FUNCTION pty_datedec( string, string);
DROP FUNCTION pty_dateins( date, string);
DROP FUNCTION pty_datesel( date, string);

-------------smallint udfs ---------------------

DROP FUNCTION pty_smallintenc( smallint, string);
DROP FUNCTION pty_smallintdec( string, string);
DROP FUNCTION pty_smallintins( smallint, string);
DROP FUNCTION pty_smallintsel( smallint, string);+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.56s
 default> DROP FUNCTION pty_getversionextended();
Query: DROP FUNCTION pty_getversionextended()
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_whoami();
Query: DROP FUNCTION pty_whoami()
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default>
 default> -- string UDFs ------
> DROP FUNCTION pty_stringenc( STRING, STRING );
Query: -- string UDFs ------
DROP FUNCTION pty_stringenc( STRING, STRING )
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_stringdec( STRING, STRING );
Query: DROP FUNCTION pty_stringdec( STRING, STRING )
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_stringins( STRING, STRING );
Query: DROP FUNCTION pty_stringins( STRING, STRING )
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_unicodestringins( STRING, STRING );
Query: DROP FUNCTION pty_unicodestringins( STRING, STRING )
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_unicodestringfpeins( STRING, STRING );
Query: DROP FUNCTION pty_unicodestringfpeins( STRING, STRING )
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_stringsel( STRING, STRING );
Query: DROP FUNCTION pty_stringsel( STRING, STRING )
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_unicodestringsel( STRING, STRING );
Query: DROP FUNCTION pty_unicodestringsel( STRING, STRING )
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_unicodestringfpesel( STRING, STRING );
Query: DROP FUNCTION pty_unicodestringfpesel( STRING, STRING )
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default>
 default> --- Integer Udfs -----------------------------
> DROP FUNCTION pty_integerenc( INTEGER, STRING);
Query: --- Integer Udfs -----------------------------
DROP FUNCTION pty_integerenc( INTEGER, STRING)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_integerdec( STRING, STRING);
Query: DROP FUNCTION pty_integerdec( STRING, STRING)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_integerins( INTEGER, STRING);
Query: DROP FUNCTION pty_integerins( INTEGER, STRING)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_integersel( INTEGER, STRING);
Query: DROP FUNCTION pty_integersel( INTEGER, STRING)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default>
 default> --------------double udfs ----------------------
> DROP FUNCTION pty_doubleenc( double, string);
Query: --------------double udfs ----------------------
DROP FUNCTION pty_doubleenc( double, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_doubledec( string, string);
Query: DROP FUNCTION pty_doubledec( string, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_doubleins( double, string);
Query: DROP FUNCTION pty_doubleins( double, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_doublesel( double, string);
Query: DROP FUNCTION pty_doublesel( double, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default>
 default> -------------float udfs -------------------------
> DROP FUNCTION pty_floatenc( float, string);
Query: -------------float udfs -------------------------
DROP FUNCTION pty_floatenc( float, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_floatdec( string, string);
Query: DROP FUNCTION pty_floatdec( string, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_floatins( float, string);
Query: DROP FUNCTION pty_floatins( float, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_floatsel( float, string);
Query: DROP FUNCTION pty_floatsel( float, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default>
 default> -------------bigint udfs ------------------------
>
> DROP FUNCTION pty_bigintenc( bigint, string);
Query: -------------bigint udfs ------------------------
DROP FUNCTION pty_bigintenc( bigint, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_bigintdec( string, string);
Query: DROP FUNCTION pty_bigintdec( string, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_bigintins( bigint, string);
Query: DROP FUNCTION pty_bigintins( bigint, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_bigintsel( bigint, string);
Query: DROP FUNCTION pty_bigintsel( bigint, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default>
 default> -------------date udfs --------------------------
>
> DROP FUNCTION pty_dateenc( date, string);
Query: -------------date udfs --------------------------
DROP FUNCTION pty_dateenc( date, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_datedec( string, string);
Query: DROP FUNCTION pty_datedec( string, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_dateins( date, string);
Query: DROP FUNCTION pty_dateins( date, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_datesel( date, string);
Query: DROP FUNCTION pty_datesel( date, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default>
 default> -------------smallint udfs ---------------------
>
> DROP FUNCTION pty_smallintenc( smallint, string);
Query: -------------smallint udfs ---------------------
DROP FUNCTION pty_smallintenc( smallint, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_smallintdec( string, string);
Query: DROP FUNCTION pty_smallintdec( string, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_smallintins( smallint, string);
Query: DROP FUNCTION pty_smallintins( smallint, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s
 default> DROP FUNCTION pty_smallintsel( smallint, string);
Query: DROP FUNCTION pty_smallintsel( smallint, string)
+----------------------------+
| summary                    |
+----------------------------+
| Function has been dropped. |
+----------------------------+
Fetched 1 row(s) in 0.11s

6.2 - Restoring the Parameters using the Helper Script

Before uninstalling the Big Data Protector, restore the configuration parameters. These parameters will vary depending on the services in use. Protegrity now provides the set_unset_bdp_config.sh script to restore the configuration parameters for the required services.

To set the paramters using the helper script, refer Setting the Parameters using the Helper Script.

To restore the Big Data Protector configuration:

  1. Log in to the staging machine.

  2. Navigate to the directory where you executed configurator script and generated the installation files.

  3. To restore the configurations using the helper script, run the following command:

    ./set_unset_bdp_config.sh

  4. Press ENTER.

    The prompt to enter the protocol for the Cloudera Manager server appears.

    Select the Cloudera Manager URL Protocol.
[ 1 ] : http://
[ 2 ] : https://
Enter the no.:

  5. To use https, type 2.

  6. Press ENTER.

    The prompt to enter the IP address of the Cloudera Manager server appears.

    Enter Cloudera Manager Server Node's Hostname/IP Address:

  7. Enter the IP address of the node where the Cloudera Manager Server is installed.

  8. Press ENTER.

    The prompt to enter the port number for the Cloudera Manager server appears.

    Enter Cloudera Manager Server's Port No. [7183]:

    Note: For https, the script will use 7183 as the default port and for http, the script will use 7180 as the default port.

  9. Press ENTER.

    The prompt to enter the name of the cluster appears.

    Enter Cluster's Name:

  10. Enter the name of the cluster.

  11. Press ENTER.

The prompt to enter the username to access Cloudera Manager appears.

```
Enter Cloudera Manager's Username:
```

  1. Enter the username.

  2. Press ENTER.

    The prompt to enter the password appears.

    Enter Cloudera Manager's Password:

  3. Enter the password.

  4. Press ENTER.

    The script verifies the cluster details and the prompt to set or remove the configuration appears.

    Cluster's existence verified.

Do you want to set or unset the BDP configs?
[ 1 ] : SET the BDP configs
[ 2 ] : UNSET the BDP configs
Enter the no.:

  5. To restore the configuration for the Big Data Protector, type 2.

  6. Press ENTER.

    The script updates the configuration for the Big Data Protector.

    Checking existence of HBase service with name 'hbase'.
##O=#    #
Warning: Unable to check existence of HBase service 'hbase'. Skipping this service...
{
"message" : "Service 'hbase' not found in cluster <cluster_name>."
}

Checking existence of Hive on Tez service with name 'hive_on_tez'.
##O=#    #
Service 'hive_on_tez' exists.

Unsetting Hive on Tez's config...
##O=#    #
##O=#    #
############################################################################################################################## 100.0%
Hive on Tez Service wide configs ('HIVE_ON_TEZ_service_env_safety_valve' and 'hive_service_config_safety_valve') have been updated.
##O=#    #
##O=#    #
############################################################################################################################## 100.0%
Hive on Tez's 'hive_client_env_safety_valve' config for Role Group 'hive_on_tez-GATEWAY-BASE' has been updated.

Checking existence of Tez service with name 'tez'.
##O=#    #
Service 'tez' exists.

Unsetting Tez's config...
##O=#    #
############################################################################################################################## 100.0%
Tez Service wide config ('tez.cluster.additional.classpath.prefix') has been updated.

Checking existence of Impala service with name 'impala'.
##O=#    #
Warning: Unable to check existence of Impala service 'impala'. Skipping this service...
{
"message" : "Service 'impala' not found in cluster <cluster_name>."
}

Checking existence of Spark3 on Yarn service with name 'spark3_on_yarn'.
##O=#    #
Service 'spark3_on_yarn' exists.

Unsetting Spark3 on Yarn's config...
##O=#    #
############################################################################################################################## 100.0%
Spark3 on Yarn Service wide config ('spark3-conf/spark-env.sh_service_safety_valve') has been updated.