Understanding the architecture

Mon, 01 Jan 0001 00:00:00 +0000

The architecture for the CDP-AWS-DataHub distribution of the Big Data Protector is depicted in the image below.

Component	Description
RPAgent	Is a daemon running on each node that downloads the package from ESA over a TLS channel using the installed Certificates.
Log Forwarder	Is a daemon running on each node that routes the audit logs and application logs to ESA/Audit Store.
config.ini	Is a file on each node containing the set of configuration parameters to modify the protector behavior.
BDP Layer	Contains the Big Data Protector UDFs and APIs executing in CDP service processes.
JcoreLite	Is the JNI library that provides a Java API layer to the Core libraries.
Core	Is the set of various libraries that provide the Protegrity Core functionality.

System Requirements

Mon, 01 Jan 0001 00:00:00 +0000

Ensure that the following prerequisites are met, before installing the Big Data Protector from the Cloudera Manager:

The Hadoop cluster is installed, configured, and running CDP-AWS-Datahub (Cloudera Runtime 7.3.1).
The ESA appliance, version 10.0.x or v10.1.x, is installed, configured, and running.
The ports that are configured on ESA and the nodes in the cluster, which will run the Big Data Protector, are listed in the following table:

Destination Port	Protocol	Source	Destination	Description
8443	TLS	RPAgent on the Big Data Protector cluster node	ESA	The RPAgent communicates with ESA through port 8443 to download a policy.
9200	TLS	Log Forwarder on the Big Data Protector Cluster node	Protegrity Audit Store appliance	The Log Forwarder sends all the logs to the Protegrity Audit Appliance through port 9200.
15780	TCP	Protector on the Big Data Protector cluster node	Log Forwarder on the Big Data Protector cluster node	The Big Data Protector writes Audit Logs to localhost through port 15780. The Application Logs are also written to localhost through port 15780. The Log Forwarder reads the logs from that socket.

The user, installing the Big Data Protector, has the requisite permissions to perform the following tasks:
- Copy the Big Data Protector parcels and CSDs to the Cloudera Manager repository directories
- Restart the Cloudera SCM Server
If you are installing the Big Data Protector on a cluster, then ensure that it is installed on all the nodes in the cluster.
The group ptyitusr and the user ptyitusr, responsible to manage the Big Data Protector-related services are managed by Cloudera Manager. The user and group are unavailable on the cluster nodes.

This build supports both Spark 2 and Spark 3 on the cluster using a single pepspark jar.
For more information about installing Spark3 on CDP AWS DataHub cluster, refer https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/cds-3/topics/spark-install-spark-3-parcel.html.