Audit Log Forwarder Installation

Audit Log Forwarder installation is done via Terraform scripts provided by Protegrity in the installation bundle.

ESA Audit Store Configuration

ESA server is required as the recipient of audit logs. Verify the information below to ensure ESA is accessible and configured properly.

1. ESA server running and accessible on TCP port 9200.

2. Audit Store service is configured and running on ESA. For information related to ESA Audit Store configuration, refer to Audit Store Guide.

Certificates on ESA

By default, ESA is configured with self-signed certificates, which can only be validated using self-signed CA certificate supplied in Log Forwarder configuration.

In case ESA is configured with publicly signed certificates, this section can be skipped since the Log Forwarder will use public CA to validate ESA certificates.

To obtain self-signed CA certificate from ESA:

1. Download ESA CA certificate from the /etc/ksa/certificates/plug directory of the ESA

2. After certificate is downloaded, open the PEM file in text editor and replace all new lines with escaped new line: \n.

   To escape new lines from command line, use one of the following commands depending on your operating system:

   Linux Bash:

   awk 'NF {printf "%s\\n",$0;}' CA.pem > output.txt
   

   Windows PowerShell:

   (Get-Content '.\CA.pem') -join '\n' | Set-Content 'output.txt'
   

3. Record the certificate content with new lines escaped.

   ESA CA Server Certificate (EsaCaCert): ___________________

   This value will be used to set pty_esa_ca_server_cert Terraform variable in installation section. Install Log Forwarder via Terraform

For more information about ESA certificate management refer to Certificate Management Guide in ESA documentation.

VPC configuration

Similar to Policy Agent Function, log forwarder function requires Google Cloud VPC to route traffic from the function to ESA. Review the VPC configuration steps for agent in section Identify or Create a new VPC. Same VPC connector as the policy agent can be used. Note down VPC connector name:

google_vpc_access_connector_name: ___________________

ESA Authentication

Audit Log Forwarder must authenticate with ESA using certificate-based authentication with client certificate and certificate key. Download the following certificates from the /etc/ksa/certificates/plug directory of the ESA:

Both certificate and certificate key must be converted to single-line values using code similar to the following examples.

Client certificate (client.pem):

• Powershell
• Bash

$folder = 'C:\Temp'
cd $folder
(Get-Content "$folder\client.pem") -join '\n' | Set-Content "$folder\one-liner-client.pem"
cat "$folder\one-liner-client.pem"

folder="/tmp"
cd "$folder"
awk 'NF {printf "%s\\n",$0}' "client.pem" > "one-liner-client.pem"
cat "one-liner-client.pem"

Client certificate key (client.key):

• Powershell
• Bash

$folder = 'C:\Temp'
cd $folder
(Get-Content "$folder\client.key") -join '\n' | Set-Content "$folder\one-liner-client.key"
cat "$folder\one-liner-client.key"

folder="/tmp"
cd "$folder"
awk 'NF {printf "%s\\n",$0}' "client.key" > "one-liner-client.key"
cat "one-liner-client.key"

While installing using Terraform template:

1. Provide single-line client certificate for pty_esa_client_cert
2. Provide ID of the GCP secret containing the single-line certificate key for pty_esa_client_cert_key_secret_id Secret is created in a later step

Configure ESA Secrets In GCP Secret Manager

Audit Log Forwarder Function uses GCP Secret Manager to store ESA Audit Store credentials used during authentication.

For information on how to configure basic and certificate authentication for Audit Store on ESA refer to Audit Store Guide.

1. Log in to Google Account and select project where Protegrity service will be installed.

2. Go to Security > Secret Manager.

3. Select CREATE SECRET.

4. Specify the Secret Value:

   {
     "username": "admin", 
     "password": "{esa_password}"
   }
   

5. Select Create Secret.

6. Once the secret is created, you should see the secret screen opened. If not click on the secret name to see a screen with secret versions.

7. Click on Actions, next to the secret version you just created.

8. Select Copy Resource ID and record the full secret version path, for example, projects/{project-id}/secrets/{secret name}/versions/2.

   ESA Log Forwarder Credentials Secret Name: _________________

9. Create another secret with single-line contents of ESA client certificate key file

   See Certificate Authentication for details on client certificate key

10. Record the full secret version path, for example, projects/{project-id}/secrets/{secret name}/versions/1.

    ESA Log Forwarder Client Certificate Key Secret Name: _________________

Create Log Forwarder Service Account

To create Log Forwarder Service Account:

1. Log in to Google Account and select the project where Protegrity service will be installed.

2. Navigate to IAM & Admin > Service Accounts.

3. Select CREATE SERVICE ACCOUNT.

4. Specify service account name and description.

5. Select Create and Continue.

6. In the next step, click Select Role. Then select the following roles:

   • Cloud KMS CryptoKey Decrypter
   • Pub/Sub Publisher
   • Secret Manager Secret Accessor

7. Click Done.

8. Once the service account is created, the screen should open on the service account. If the screen does not appear, refresh the page with the service account list and select the service account created.

9. Record the full email. For example, service-account-name@project-id.iam.gserviceaccount.com.

   Log Forwarder Function Service Account Email: ___________________

Create Service Account For Forwarder Pub/Sub

Pub/Sub service requires Cloud Run Invoker permissions in order to be able to send messages to the Forwarder function.

1. Log in to Google Account and select the project where Protegrity forwarder will be installed.

2. Navigate to IAM & Admin > Service Accounts.

3. Select CREATE SERVICE ACCOUNT.

4. Specify service account name and description.

5. Select Create and Continue.

6. In the next step, click Select Role. Then select Cloud Run Invoker.

7. Click Done.

8. Once the service account is created, the screen should open on the service account. If the screen does not appear, refresh the page with the service account list and select the service account created.

9. Record the full email. For example, service-account-name@project-id.iam.gserviceaccount.com.

   Pub/Sub Log Forwarder Service Account Email: ___________________

Preparation

1. Ensure that all the steps in Google Cloud Project are performed.

2. Log in to the Google Cloud account where Protegrity will be installed.

3. Select the project.

4. Ensure that you have access to shell command on your computer or Cloud Shell with Terraform CLI v0.14 or higher installed.

5. Ensure that the Terraform scripts provided by Protegrity are available on your local computer.

Install Log Forwarder Function via Terraform Scripts

Resources created with Terraform scripts include Audit Log Forwarder Cloud Functions Service and Pub/Sub topic. If you don’t specify the deployment bucket Terraform parameter, a new storage bucket will also be created. You can optionally choose to create a new service account with custom IAM role.

To install using Terraform:

1. From the command shell move to directory where you downloaded Protegrity installation bundle.

2. Unzip the bundle, then unzip the protegrity-gcp-bigquery-{version}.zip. Navigate to pty-log-forwarder-gcp/. Verify that the following files are available:

   • pty-log-forwarder-gcp/
   • main.tf
   • outputs.tf
   • protegrity-cloud-api-gcp-{version}.zip
   • README.md

3. Open the main.tf file and update Terraform backend information at the top of the file:

   terraform {
     backend "gcs" {
       bucket  = ""
       # The bucket/prefix combination must be unique for different deployments 
       # to avoid conflicting Terraform states and accidental resources destruction.
       # prefix = "protegrity-gcp-bigquery/forwarder/<deployment_id>/tf-state"
     }
   }
   

4. Set the bucket property to Terraform Backend Bucket Name recorded in Google Cloud Storage

5. Set the prefix property with value unique to your deployment.

6. In the same main.tf file, specify the following Terraform variables: All the values were recorded in Google Cloud Project.

   Warning

   Google Cloud Function 2nd Generation currently does not support CMEK.

   Parameter	Description
   project_id	The project id recorded in the pre-configuration step
   region	The Region recorded in the pre-configuration step.
   deployment_id	Specify short name to identify deployment. This id will be added to all resources deployed with Terraform.
   deployment_bucket	Use Deployment Bucket Name recorded in pre-configuration or leave empty to create new bucket.
   create_service_account	Leave this as false if you created service account in pre-configuration. Otherwise set to true.
   forwarder_function_service_account_email	Use Forwarder Function Service account recorded in pre-configuration or leave empty.
   pub_sub_log_forwarder_service_account_email	Service account of the audit log Pub/Sub trigger. The service account must be assigned Cloud Run Invoker (roles/run.invoker) role.
   create_vpc	If create_vpc flag is set, new vpc will be created together with vpc connector, NAT and external IP Use this flag if you have VPC admin permissions in your Google Account. If you set it to false, you can specify the existing VPC in the google_vpc_access_connector_name parameter.
   google_vpc_access_connector_name	Use existing VPC connector to associate with Log Forwarder Function. You can specify either the VPC connector name or the full resource name if vpc connector is in different region/project that the one specified for the deployment. You can alternatively set the use google_vpc_access_connector_full_resource_name. Both parameters are optional. Full resource name takes precedence over connector name.
   log_destination_esa_ip	Ip address of the ESA where Protector logs will be sent to.
   pty_esa_ca_server_cert	ESA self-signed CA certificate used by log forwarder function to ensure ESA is the trusted server. See documentation for more details.
   esa_credentials_secret_resource_id	GCP Secret Manager secret id where ESA Fluent Bit logger credentials are stored.
   pty_esa_client_cert	Single-line ESA client certificate content. See Certificate Authentication for details on client certificate
   pty_esa_client_cert_key_secret_id	GCP Secret Manager secret id where single-line ESA client certificate key content is stored. See Configure ESA Secrets In GCP Secret Manager for details on client certificate key secret
   min_log_level	Minimum log level for log forwarder function. Must be one of the following: [off,severe,warning,info,config,all].
   esa_tls_disable_cert_verify	Disable certificate verification when connecting to ESA. This is only for dev purposes, should not be used in production environment.
   esa_connect_timeout	Esa connection timeout in seconds.
   esa_virtual_host	ESA Virtual Host.
   audit_log_flush_interval	Time interval in seconds used to accumulate audit logs before sending to ESA. Default value: 10 Min value: 1 Max value: 900
   dlq_topic_message_retention_duration	Indicates the minimum duration to retain a message in dead letter queue topic in case log destination server is not available. Value must be decimal number, followed by the letter s (seconds). Cannot be more than 31 days or less than 10 minutes. Default value is 1 day
   audit_log_dead_letter_topic	This parameter is expected to be used in a separate deployment to replay dead letter queue messages.
   max_instance_count	GCP Cloud Functions advanced configuration
   available_memory_mb	GCP Cloud Functions advanced configuration
   timeout_seconds	GCP Cloud Functions advanced configuration
   gen2_available_cpu	2nd Gen Cloud Function advanced configuration
   gen2_container_concurrency	2nd Gen Cloud Function advanced configuration
   upgrade_step	Set this variable when upgrading to the latest version.
   labels	You can set this map to include labels for deployed resources. Pay attention to GCP label requirements. For more information, refer to the following link https://cloud.google.com/compute/docs/labeling-resources. For example, only use lowercase and maximum length of 63 characters.

7. From local command line or Cloud Shell, change directory to location of the main.tf, for example:

   pty-log-forwarder-gcp-{version}/pty-log-forwarder-gcp/
   

8. Run the following command.

   terraform init
   

9. Terraform will download necessary providers.

10. Run the following command to verify configuration and print out deployment plan.

    terraform plan
    

11. Run the following command to deploy resources to your account.

    terraform apply
    

12. Once deployment is complete Terraform will print output variables.

13. Record the following values:

    • forwarder_function_name: ____________________________
    • forwarder_function_url: ____________________________
    • forwarder_function_resource_name: __________________

Turn on Instance-based billing.

Both Protect and Log Forwarder functions must run for a short period of time after all requests are handled. In order for the GCP Cloud Run service to allow that, the Instance-based billing feature must be enabled for both function deployments.

To enable Instance-based billing:

1. Log in to Google Account and select the project where Protegrity Cloud Run Function was installed.

2. Navigate to Cloud Run.

3. Click on the Cloud Function name.

4. In Cloud Run revision view, select Edit & deploy new revision.

5. Scroll down to Billing.

6. Select Instance-based.

7. Click DEPLOY.

8. Repeat the steps for Log Forwarder function.

Test Log Forwarder Function Installation

Before continuing with next steps, you can verify whether Log Forwarder Function is installed correctly. This step is optional and can be skipped.

1. Below you can find example CURL command to test your function.

2. Before you can execute it, test if you can obtain temporary authentication token. Run the gcloud auth login and then gcloud auth print-identity-token commands. The logged in gcloud user must have the Cloud Run Invoker permissions. Continue to the next step if the command succeeds and prints the token.

3. Replace {forwarder_function_url}; with value recorded in previous step.

4. Run the following CURL command to test Function deployment.

   curl {forwarder_function_url} \
   -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
   -H "Content-Type: application/json" \
   -H "ce-id: 123451234512345" \
   -H "ce-specversion: 1.0" \
   -H "ce-time: 2020-01-02T12:34:56.789Z" \
   -H "ce-type: google.cloud.pubsub.topic.v1.messagePublished" \
   -H "ce-source: //pubsub.googleapis.com/projects/MY-PROJECT/topics/MY-TOPIC" \
   -d '{
       "message": { 
           "data": "'"$(echo '{"additional_info":{"description":"Data unprotect operation was successful.","query_id":"sf-query-id:k6-test-df51a612-4739-4cfb-9fe4-6ec548b70d23"},"client":{},"cnt":4000,"correlationid":"sf-query-id:k6-test-df51a612-4739-4cfb-9fe4-6ec548b70d23","level":"SUCCESS","logtype":"Protection","origin":{"hostname":"localhost","time_utc":1725558586},"process":{"id":1},"protection":{"audit_code":8,"dataelement":"alpha","datastore":"SAMPLE_POLICY","mask_setting":"","operation":"Unprotect","policy_user":"master_user"},"protector":{"core_version":"1.2.2+42.g01eb3.HEAD","family":"cp","pcc_version":"3.4.0.20","vendor":"gcp.snowflake","version":"3.1.0.158"},"signature":{"checksum":"7CE5FFCE9DBE570AAA72D1BB20CD083532EF8FAD3E96E38629EB92E837272D8E","key_id":"676c5178-756d-4363-9"}}' | base64 -w 0)"'",
           "attributes": {},  
           "messageId": "",  
           "publishTime": "2014-10-02T15:01:23Z",
           "orderingKey": ""
      }
   }'
   

5. In GCP Logs Explorer console verify that the following output appears in the logs:

   Request finished HTTP/1.1 POST http://pty-forwarder-31-smoke-jf-pfadh7riaq-uc.a.run.app/ - 200 0 - 75.6570ms
   

6. Warning

   Test steps will only succeed if the Policy Agent has not updated the Log Forwarder policy. Once updated, logs must be signed with your policy, and the sample data blob above will no longer pass the check, resulting in the error below: [/jenkins/workspace/iaplambda_release_3.1/src/iap/logging/log-aggregator.cpp:66] Failed to aggregate log entry at index 0
   .

Grant Pub/Sub Publisher Permission to the Protect Function Service Account

Protect function requires permissions to publish audit log messages to Pub/Sub.

1. Log in to Google Account and select the project where Protegrity service will be installed.

2. Navigate to IAM & Admin.

3. Search for protector function service account email recorded in protect service installation step.

4. Select Edit principal pencil icon.

5. Select ADD ANOTHER ROLE.

6. Select Pub/Sub Publisher.

7. Click Save.

Protect Function Pub/Sub Log Output

Protect function must be configured to output audit logs to Pub/Sub topic.

To configure Protect function audit log output:

1. Go to Protect function Terraform deployment.

2. Navigate to pty-protect-gcp/main.tf.

3. Set Terraform variable pty_log_output=“pub_sub”.

4. Set Terraform variable pty_pub_sub_topic to log forwarder Pub/Sub topic.

   Note

   You can obtain the topic resource name from Log Forwarder Terraform output: audit_log_topic.

5. Run terraform apply.

Troubleshooting

Configure additional logging:

1. Set min_log_level Terraform variable on both Protect function and Log Forwarder function to config.

2. In the GCP Logs Explorer, you can run the query below, replacing placeholders with your deployment id and project name.

   resource.type="cloud_run_revision"
   resource.labels.service_name=~"pty-(protect|forwarder)-<deploymentd-id>"
   severity=ERROR OR textPayload=~"\[error\]"
   -logName="projects/<gcp-project-id>/logs/run.googleapis.com%2Frequests"
   

3. Expand each log entry for more details. Check for jsonPayload > exception to see more detailed error.

Pub/Sub configuration error.

Failed to send x/y audit logs to GCP Pub/Sub.   

Audit log sink error: Unable to deliver all logs. 

opensearch.0: Dropped records: 1/1

[error] [output:opensearch:opensearch.0] HTTP status=401 URI=/_bulk