Installing the protector

• 1: Using the Bootstrap Installer
• 1.1: Creating a Cluster
• 1.2: Managing the Cluster Nodes
• 1.3: Verifying the Parameters
• 2: Using the Static Installer
• 2.1: Installing the Protector on all the Nodes
• 2.2: Installing the Protector on Specific Nodes
• 2.3: Verifying the Parameters

1 - Using the Bootstrap Installer

The Big Data Protector on Amazon EMR enables cluster creation using a bootstrap action. This action enables:

• configuration of cluster instances
• installation of custom and additional software
• setting up of the environment variables

Bootstrap actions are scripts that run on cluster instances after they are launched. These scripts installs the specified applications during cluster creation and before the cluster nodes start processing data. To create a bootstrap action, can specify the script when creating the cluster in any one of the following methods:

• Amazon EMR console - pass the location of the script in the Bootstrap actions section.
• AWS CLI - pass the location of the script to the --bootstrap-actions parameter.
• API

In this method of cluster creation, the nodes are automatically scaled depending on the workload. In case of instances where the workloads are minimal for a node, Amazon decomissions the node to balance the workload optimally.

1.1 - Creating a Cluster

The procedures mentioned in this section are applicable only for the Bootstrap approach to install the Big Data Protector.

Perform the following steps to create an EMR cluster on AWS and install Big Data Protector on all the nodes in the EMR cluster.

To install Big Data Protector on a New EMR Cluster:

1. On the AWS services screen, click EMR under the Analytics section.

   The Amazon EMR screen appears.

2. Click Create cluster.

   The Create Cluster - Quick Options screen appears.

3. Type the name of the cluster in the Cluster name box.

4. Depending on the requirements, enter the sum of the master and core nodes in the Number of instances box.

5. Click Create cluster.

   The Software and Steps tab on the Create Cluster - Advanced Options screen appears.

6. Depending on the requirements, select the components under the Software Configuration section.

7. Click Next.

   The Hardware tab on the Create Cluster - Advanced Options screen appears.

8. On the Hardware tab, if required, you can add or reduce the number of instances of the Master, Core, and Task nodes.

9. Click Next.

   The General Cluster Settings tab on the Create Cluster - Advanced Options screen appears.

10. Type the name of the cluster in the Cluster name box.

11. Under the Bootstrap Actions area, in the Add bootstrap action drop-down list, click Custom action.

    The Add Bootstrap Action dialog box appears.

12. Enter the name of the bootstrap action in the Name box.

13. To select the location of the bootstrap script, click the icon besides the Script location box.

    The Select S3 File dialog box appears.

14. Enter the path of the S3 bucket in the URL box.

    The contents of the S3 bucket appear.

15. Select the bdp_bootstrap_installer.sh file from the S3 bucket.

16. Click Select.

    The Big Data Protector bootstrap script file is selected and the Add Bootstrap Action dialog box appears.

17. To specify the directory in which the Big Data Protector needs to be installed on the nodes in the cluster, then provide the directory path in the Optional arguments box.

    If an installation directory for the Big Data Protector is not specified, then /opt/protegrity/ is considered as the default directory.

18. Click Add.

    The General Cluster Settings tab on the Create Cluster - Advanced Options screen appears and the Bootstrap actions are updated.

19. Click Next.

    The Security tab on the Create Cluster - Advanced Options screen appears.

20. Select the required EC2 key pair for the EMR cluster from the EC2 key pair drop-down list.

21. Click Create Cluster.

    The EMR cluster is created, Big Data Protector is installed on all the nodes in the cluster, and the required Big Data Protector parameters are configured.

22. You can also install create a new EMR cluster and install Big Data Protector on the nodes in the cluster using the CLI using the following command:

    aws emr create-cluster --auto-scaling-role EMR_AutoScaling_DefaultRole --termination-protected --applications Name=Hadoop Name=Hive Name=Pig Name=Hue Name=Spark Name=Tez Name=HBase --bootstrap-actions '[{"Path":"<S3_Path_For_BootstrapInstaller>","Name":"<Script_Name>"}]' --ec2-attributes '{"KeyName":"<KEY_NAME>","InstanceProfile":"EMR_EC2_DefaultRole","EmrManagedSlaveSecurityGroup":"sg-c8ef00de","EmrManagedMasterSecurityGroup":"sg-2deb043b"}' --service-role EMR_DefaultRole --enable-debugging --release-label emr-<EMR_Version> --log-uri 's3n://aws-logs-406396743807-us-east-1/elasticmapreduce/' --name '<Cluster_Name>' --instance-groups '[{"InstanceCount":2,"InstanceGroupType":"CORE","InstanceType":"m3.xlarge","Name":"Core - 2"},{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"m3.xlarge","Name":"Master - 1"}]' –
    scale-down-behavior TERMINATE_AT_INSTANCE_HOUR --region us-east-1
    

    where:

    • S3_Path_For_BootstrapInstaller: Specifies the S3 bucket path containing the Big Data Protector bootstrap installer script.
    • Script_Name: Specifies the name of the Big Data Protector installation script.
    • KEY_NAME: Specifies the Private Key file on the Master node in the EMR cluster, which is used to communicate with the other nodes in the cluster.
    • Cluster_Name: Specifies the name of the new EMR cluster.

1.2 - Managing the Cluster Nodes

The steps mentioned in this section are applicable only for the Bootstrap approach to install the Big Data Protector.

Depending on the workload on the EMR cluster, you can add or remove the Big Data Protector nodes. You can either set the cluster to automatically scale or manually add or remove nodes in the EMR cluster. You can add or remove nodes in the EMR cluster either while you create the cluster or after you have created the cluster. Before you add or remove the nodes from the cluster, ensure that you save all your data to S3, as standard practice, to avoid any data loss.

This section covers the procedure to add or remove nodes from an Amazon EMR cluster after you have created it.

To add or remove nodes from an Amazon EMR cluster:

1. On the AWS management console, expand Services and click Analytics.

   The sub-menu appears.

2. From the sub-menu, click EMR.

   The Amazon EMR page appears.

3. Click the required cluster.

   The Properties tab of the cluster appears.

4. Click the Instances tab.

5. To add an instance, perform the following steps:

   1. Under Instance groups, click Add task instance group. The Add task instance group page appears.
   2. In the Name box, enter the name to identify the node.
   3. From the Choose EC2 instance type list, select the required storage type.
   4. In the Instance group size box, enter the required number of instances.
   5. Click Add task instance group. The new instance is added to the node and appears on the Instances tab.

6. To resize an instance, perform the following steps:

   1. Under Instance groups, select the required instance that you want to resize.
   2. Click Resize instance group. The Resize page appears.
   3. In the Instance group size box, enter the required number of instances.
   4. Click Resize. The instance is resized as per the inputs and appears on the Instances tab.

1.3 - Verifying the Parameters

The content mentioned in this section is applicable only for the Bootstrap approach to install the Big Data Protector.

Before using Big Data Protector, configure the required Protegrity-related parameters in EMR. The Big Data Protector configuration parameters are set for the EMR cluster when it is installed on all the nodes in the cluster.

The following table provides the parameters that are set for the existing Amazon EMR cluster before using the Big Data Protector:

2 - Using the Static Installer

The static installer method of installation is applicable where the Big Data Protector must be installed on an existing EMR cluster. Using the Static Installer, users can enforce data protection policies at a granular level. This feature helps organizations to define specific rules for data protection based on sensitivity and usage.

The nodes in the cluster created using the static installer are do not have auto-scaling enabled. The nodes must be manually added or decommissioned depending upon the usage. The installation provides additional scripts to monitor and control the cluster behaviour. These scripts are available in the <installation_directory>/cluster_utils/ directory after installation.

2.1 - Installing the Protector on all the Nodes

The steps mentioned in this section are applicable only for the Static Installer approach to install the Big Data Protector.

1. Log in to the Master or Lead node of the EMR cluster.

2. Navigate to the directory that contains the BdpInstallx.x.x_Linux_<BDP_version>.sh script.

3. To run the installer, execute the following script:

   ./BdpInstallx.x.x_Linux_<BDP_version>.sh
   

4. Press ENTER.

   The prompt to continue the installation of the Big Data Protector appears.

   ************************************************************************************
              Welcome to the Hadoop Big Data Protector Setup Wizard
   ************************************************************************************
   This will install the Hadoop Big Data Protector on your system.
   
   This installation requires a Private Key file for communicating with other nodes in the cluster.
   
   Do you want to continue? [yes or no]:
   

5. To continue, type yes.

6. Press ENTER.

   The prompt to enter path of the Private Key file (.pem file) appears.

   Big Data Protector installation started
   Enter the path of the Private Key (.PEM) file:
   

7. Enter the path of the .PEM file.

8. Press ENTER.

   The prompt to enter ESA hostname or IP address appears.

   libhadoop.so located in directory '/usr/lib/hadoop/lib/native'
   Unpacking...
   Extracting files...
   
   Preparing for cluster deploy, Wait...
   
   Enter ESA Hostname or IP Address:
   

9. If you have installed a proxy, then enter the IP address of the proxy node. Alternatively, enter the IP Address of ESA.

10. Press ENTER.

    The prompt to enter the listening port for ESA appears.

    Enter ESA host listening port [8443]:
    

11. Enter the port for ESA.

12. Press ENTER.

    The prompt to enter the JWT token appears.

    If you have an existing ESA JSON Web Token (JWT) with Export Certificates role, enter it otherwise enter 'no':
    

13. Enter the JWT token.

14. Press ENTER.

    If you fail to provide a JWT token, the script will prompt to enter the username and password for ESA.

    JWT was not provided. Script will now prompt for ESA username and password.
    
    Enter ESA Username:
    

15. Enter the username for ESA.

16. Press ENTER.

    The prompt to enter the password appears.

    ************************************************************************************
                    Welcome to the RPAgent Setup Wizard.
    ************************************************************************************
    
    Unpacking...................
    Extracting files...
    Unpacked rpagent compressed file...
    RPAgent Installing in Lead Node...
    Please enter the password for downloading certificates[]:
    

17. Enter the password.

18. Press ENTER.

    The script retrieves the JWT token from ESA, installs the RPAgent, and the prompt to select the Audit Store type appears.

    Unpacking...
    Extracting files...
    Obtaining token from <ESA_IP_Address>:8443...
    Downloading certificates from <ESA_IP_Address>:8443...
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
    100 11264  100 11264    0     0  12124      0 --:--:-- --:--:-- --:--:-- 12111
    
    Extracting certificates...
    Certificates successfully downloaded and stored in /opt/protegrity/rpagent/data
    
    Protegrity RPAgent installed in /opt/protegrity/rpagent.
    
    
    RPAgent installed on Lead node at location /opt/protegrity/rpagent.
    
    Performing install on other nodes...
    
    RPAgent installed on other nodes at location /opt/protegrity/rpagent.
    
    Check the status in /opt/protegrity/logs/rpagent_setup.log
    
    
    Select the Audit Store type where Log Forwarder(s) should send logs to.
    
    [ 1 ] : Protegrity Audit Store
    [ 2 ] : External Audit Store
    [ 3 ] : Protegrity Audit Store + External Audit Store
    
    Enter the no.:
    

19. Depending on the Audit Store type, select any one of the following options:

    Option	Description
    1	To use the default setting using the Protegrity Audit Store appliance, type 1. If you enter 1, then the default Fluent Bit configuration files are used and Fluent Bit will forward the logs to the Protegrity Audit Store appliances.
    2	To use an external audit store, type 2. If you enter 2, then the default Fluent Bit configuration files used for the External Audit Store (out.conf and upstream.cfg in the /opt/protegrity/fluent-bit/data/config.d/ directory) are renamed (out.conf.bkp and upstream.cfg.bkp) so that they will not be used by Fluent Bit. Additionally, the custom Fluent Bit configuration files for the external audit store are copied to the /opt/protegrity/fluent-bit/data/config.d/ directory.
    3	To use a combination of the default setting with an external audit store, type 3. If you enter 3, then the default Fluent Bit configuration files used for the Protegrity Audit Store (out.conf and upstream.cfg in the /opt/protegrity/fluent-bit/data/config.d/ directory) are not renamed. However, the custom Fluent Bit configuration files for the external audit store are copied to the /opt/protegrity/fluent-bit/data/config.d/ directory.

20. Press ENTER.

    The prompt to enter the comma separated list of hostnames/IP addresses appears.

    Enter comma-separated list of Hostnames/IP Addresses and/or Ports of Protegrity Audit Store.
    Allowed Syntax: hostname[:port][,hostname[:port],hostname[:port]...] (Default Value - <ESA_IP_Address>:9200)
    Enter the list:
    

21. To use the default value, press ENTER.

    The prompt to enter the location of the Fluent Bit configuration file appears.

    Enter the local directory path on this node that stores the custom Fluent-Bit configuration files for External Audit Store:
    

    Note: The script will display this prompt only if you select option 2 in step 19. When you select option 2 in step 19, the custom configuration files are copied to the /<Installation directory>/fluent-bit/data/config.d/ directory on all the EMR nodes selected for installation.

22. Enter the path that contains the Fluent Bit configuration file.

23. Press ENTER.

    The prompt to save the RPAgent’s log in a file appears.

    Do you want RPAgent's log to be generated in a file? [yes or no]:
    

24. To generate the logs in a file, type yes.

25. Press ENTER.

    The script installs the protector on all the nodes in the cluster.

    RPAgent's log will be generated in a file.
    ************************************************************************************
                    Welcome to the LogForwarder Setup Wizard.
    ************************************************************************************
    
    Unpacking...................
    Extracting files...
    Unpacked logforwarder compressed file...
    Logforwarder Installing in Lead Node...
    Unpacking...
    Extracting files...
    
    Protegrity Log Forwarder installed in /opt/protegrity/logforwarder.
    
    
    LogForwarder installed on Lead node at location /opt/protegrity/logforwarder.
    
    Performing install on other nodes...
    
    Logforwarder installed on other nodes at location /opt/protegrity/logforwarder.
    
    Check the status in /opt/protegrity/logs/logforwarder_setup.log
    ************************************************************************************
                        Welcome to the JcoreLite Setup Wizard.
    ************************************************************************************
    
    Unpacking...................
    Extracting files...
    Unpacked jcorelite compressed file...
    Installing JcoreLite ....
    
    JcoreLite installed on lead node at location /opt/protegrity/bdp/lib.
    
    Performing install on other nodes...
    
    JcoreLite installed on other nodes at location /opt/protegrity/bdp/lib.
    
    Check the status in /opt/protegrity/logs/jcorelite_setup.log
    ************************************************************************************
                    Welcome to the Hive Protector Setup Wizard.
    ************************************************************************************
    
    Unpacking...................
    Extracting files...
    Unpacked pephive compressed file...
    
    Hive Big Data Protector installed on lead node at location /opt/protegrity/bdp/lib/ and /opt/protegrity/pephive/scripts/.
    
    Performing install on other nodes...
    
    Hive Big Data Protector installed on other nodes at location /opt/protegrity/bdp/lib/ and /opt/protegrity/pephive/scripts/.
    
    Check the status in /opt/protegrity/logs/pephive_setup.log
    ************************************************************************************
                        Welcome to the Pig Protector Setup Wizard.
    ************************************************************************************
    
    Unpacking...................
    Extracting files...
    Unpacked peppig compressed file...
    
    Pig Big Data Protector installed on lead node at location /opt/protegrity/bdp/lib/ and /opt/protegrity/peppig.
    
    Performing install on other nodes...
    
    Pig Big Data Protector installed on other nodes at location /opt/protegrity/bdp/lib/ and /opt/protegrity/peppig.
    
    Check the status in /opt/protegrity/logs/peppig_setup.log
    ************************************************************************************
                    Welcome to the MapReduce Protector Setup Wizard.
    ************************************************************************************
    
    Unpacking...................
    Extracting files...
    Unpacked pepmapreduce compressed file...
    
    Mapreduce Big Data Protector installed on lead node at location /opt/protegrity/bdp/lib/.
    
    Performing install on other nodes...
    
    Mapreduce Big Data Protector installed on other nodes at location /opt/protegrity/bdp/lib/.
    
    Check the status in /opt/protegrity/logs/pepmapreduce_setup.log
    ************************************************************************************
                        Welcome to the Hbase Protector Setup Wizard.
    ************************************************************************************
    
    Unpacking...................
    Extracting files...
    Unpacked pephbase compressed file...
    
    Hbase Big Data Protector installed on lead node at location /opt/protegrity/bdp/lib/.
    
    Performing install on other nodes...
    
    Hbase Big Data Protector installed on other nodes at location /opt/protegrity/bdp/lib/.
    
    Check the status in /opt/protegrity/logs/pephbase_setup.log
    ************************************************************************************
                    Welcome to the Spark Protector Setup Wizard.
    ************************************************************************************
    
    Unpacking...................
    Extracting files...
    Unpacked pepspark compressed file...
    
    Spark Big Data Protector installed on lead node at location /opt/protegrity/bdp/lib/ and /opt/protegrity/pepspark/scripts/.
    
    Performing install on other nodes...
    
    Spark Big Data Protector installed on other nodes at location /opt/protegrity/bdp/lib/ and /opt/protegrity/pepspark/scripts/.
    
    Check the status in /opt/protegrity/logs/pepspark_setup.log
    
    Starting Logforwarder on lead node...
    
    Starting Logforwarder on other nodes...
    
    Starting RPAgent on lead node...
    
    Starting RPAgent on other nodes...
    
    Hadoop Big Data Protector installed in /opt/protegrity.
    
    Generating Big Data Protector installation status report ...
    
    Clearing previous logs files ...
    
    Installation Status report generated in /opt/protegrity/cluster_utils/installation_report.txt
    

26. Restart the Hadoop, Hive, and HBase service daemon processes to start using the updated configuration.

2.2 - Installing the Protector on Specific Nodes

The steps mentioned in this section are applicable only for the Static Installer approach to install the Big Data Protector.

Protegrity provides the BdpInstallx.x.x_Linux_<arch>_<BDP_version>.sh script to install the Big Data Protector on the new nodes that you add to an existing EMR cluster.

Ensure to install the Big Data Protector from an account having full sudoer privileges.

1. Log in to the Lead Node on the EMR cluster.

2. Navigate to the <PROTEGRITY_DIR>/cluster_utils directory.

3. In the NEW_HOSTS_FILE file, add an additional entry for each new node in the EMR cluster, on which you want to install the Big Data Protector. The new nodes from the NEW_HOSTS_FILE file will be appended to the CLUSTERLIST_FILE.

4. To install the Big Data Protector on the new nodes, run the the following command:

   ./BdpInstallx.x.x_Linux_<arch>_<BDP_version>.sh –a <NEW_HOSTS_FILE>
   

5. Press ENTER.

   The prompt to enter the path of the Private Key file (.pem file) appears.

6. Enter the path of the Private Key file.

7. Press ENTER.

   The script installs the Big Data Protector on the new nodes in the EMR cluster.

2.3 - Verifying the Parameters

The content in this section is applicable only for the Static installer approach to install the Big Data Protector.

Before using the Big Data Protector, configure the required Protegrity-related parameters in EMR. The Big Data Protector configuration parameters are set for the EMR cluster when it is installed on all the nodes in the cluster.

The following table provides the parameters that are set for the existing Amazon EMR cluster before using the Big Data Protector: