Audit Log Forwarder Installation

The following sections provide installation steps for the Log Forwarder component in Azure. The Log Forwarder deployment allows for the audit logs generated by Protect Service to be delivered to ESA for auditing and governance purposes. Log Forwarder component is optional and is not required for the Protect Service to work properly. See Audit Log Forwarder Architecture for more information. Some of the installation steps are not required for the operation of the software but recommended for establishing a secure environment. Contact Protegrity for further guidance on configuration alternatives in the cloud.

ESA Audit Store Configuration

ESA server is required as the recipient of audit logs. Verify the information below to ensure ESA is accessible and configured properly.

1. ESA server running and accessible on TCP port 9200 (Audit Store) or 24284 (td-agent).

2. Audit Store service is configured and running on ESA. Applies when audit logs are output to Audit Store directly or through td-agent. For information related to ESA Audit Store configuration, refer to Audit Store Guide.

3. (Optional) td-agent is configured for external input. For more information related to td-agent configuration, refer to ESA guide Sending logs to an external security information and event management (SIEM).

Certificates on ESA

By default, ESA is configured with self-signed certificates, which can only be validated using self-signed CA certificate supplied in Log Forwarder configuration.

In case ESA is configured with publicly signed certificates, this section can be skipped since the Log Forwarder will use public CA to validate ESA certificates.

To obtain self-signed CA certificate from ESA:

1. Download ESA CA certificate from the /etc/ksa/certificates/plug directory of the ESA

2. After certificate is downloaded, open the PEM file in text editor and replace all new lines with escaped new line: \n.

   To escape new lines from command line, use one of the following commands depending on your operating system:

   Linux Bash:

   awk 'NF {printf "%s\\n",$0;}' CA.pem > output.txt
   

   Windows PowerShell:

   (Get-Content '.\CA.pem') -join '\n' | Set-Content 'output.txt'
   

3. Record the certificate content with new lines escaped.

   ESA CA Server Certificate (EsaCaCert): ___________________

   This value will be used to set PTY_ESA_CA_SERVER_CERT Log Forwarder variable in section Install Log Forwarder via ARM template

For more information about ESA certificate management refer to Certificate Management Guide in ESA documentation.

ESA Authentication

Audit Log Forwarder must authenticate with ESA using certificate-based authentication with client certificate and certificate key. Download the following certificates from the /etc/ksa/certificates/plug directory of the ESA:

• client.key
• client.pem

Both certificate and certificate key must be converted to single-line values using code similar to the following examples.

Client certificate (client.pem):

• Powershell
• Bash

$folder = 'C:\Temp'
cd $folder
(Get-Content "$folder\client.pem") -join '\n' | Set-Content "$folder\one-liner-client.pem"
cat "$folder\one-liner-client.pem"

folder="/tmp"
cd "$folder"
awk 'NF {printf "%s\\n",$0}' "client.pem" > "one-liner-client.pem"
cat "one-liner-client.pem"

Client certificate key (client.key):

• Powershell
• Bash

$folder = 'C:\Temp'
cd $folder
(Get-Content "$folder\client.key") -join '\n' | Set-Content "$folder\one-liner-client.key"
cat "$folder\one-liner-client.key"

folder="/tmp"
cd "$folder"
awk 'NF {printf "%s\\n",$0}' "client.key" > "one-liner-client.key"
cat "one-liner-client.key"

There are two options to configure Log Forwarder for certificate authentication:

• While installing using ARM template
  1. Provide single-line client certificate for Esa Client Cert
  2. Provide Azure Key Vault secret name for Esa Client Cert Key Secret Name which stores single-line certificate key file.
• When re-configuring after installation using environment variables
  1. Provide single-line client certificate for ESA_CLIENT_CERT
  2. Provide Azure Key Vault secret name for ESA_CLIENT_CERT_KEY_SECRET_NAME which stores single-line certificate key file.

Create Key Vault Secrets

Log Forwarder uses Key Vault as a secure store for certificate key file.

Create secret in Key Vault for certificate key file:

1. Navigate to Key Vault.

2. Under Objects, select Secrets > Generate/import.

3. Select Manual, type in secret name and specify single-line certificate key file value for Secret value.

4. Select Create.

5. Record secret name:

   ESA Client Cert Key Secret Name (EsaClientCertKeySecretName): ___________________

Function User-Assigned Managed Identity

User-assigned Azure managed identities are optional. If a user-assigned identity is not provided, a system-assigned managed identity will be enabled the function. User-assigned managed identities offer less frequent updates to Azure resources and allow for configuration of permissions ahead of function creation.

1. In the search box, enter Managed Identities. Under Services, select Managed Identities

2. Select Create

3. For Subscription provide recorded value of AzureSubscriptionID

4. For Resource Group provide recorded value of ApiResourceGroup

5. For Region provide recorded value of ApiRegion

6. For Name provide a name of the new identity

7. Assign following roles to this identity:

   • Storage Blob Data Owner
   • Monitoring Metrics Publisher
   • Azure Event Hubs Data Receiver

8. Record Forward function user-assigned identity

   Forward Function User-Assigned Identity (ForwardFuncUserAssignedIdentity): ____________________

Install Log Forwarder via ARM template

Resources created with ARM template include Function App, App Service Plan and Application Insights service. Optionally, a new Event Hub namespace and Event Hub instance can be created.

To install Log Forwarder via ARM template:

1. From Azure Console, select Create a resource, search for template and then select Template deployment > Create.

2. Select Build your own template in editor.

3. Select Load File and upload pty_forward_arm_v2.json. Click Save.

4. Select Resource Group.

5. Specify Name for the resources (All resources will be prefixed with Protegrity-Forward).

6. For Location input specify Azure region name or leave default to deploy in the same region as resource group

7. For Storage Account Blob Service Url Optionally use the value recorded in Create Storage Account. If value is not given, it will be automatically derived from Forward Function Blob Url.

8. For Forward Function Blob Url use the value from Upload Files.

9. For Function Sku either EP1 or EP3 are recommended. Note that this will affect the running cost.

10. For Function Sku Count Minimum number of workers to keep active.

11. For WorkSpace Sku Azure Monitor log analytics pricing plan. See Azure Monitor Pricing tiers documentation for details: Azure Monitor Pricing

12. For Log Retention In Days The workSpace data retention in days. Allowed values are per pricing plan. See Azure Monitor Pricing tiers documentation for details: Azure Monitor Pricing

13. For Forward Logs to ESA select whether to collect audit logs from a new or an existing Event Hub. A new Event Hub namespace and new Event Hub instance will be created for ‘From new Event Hub’ option.

14. For Audit Log Output select whether to send logs directly to Audit Store or td-agent on ESA

15. For Event Hub Namespace enter Event Hub namespace name. Depending on previous option, a new namespace with this name will be created or an existing namespace with this name will be used.

16. For New Event Hub Namespace Sku Name select Event Hub namespace SKU name. Applicable only when ‘From new Event Hub’ is selected.

17. For New Event Hub Namespace Sku Tier select Event Hub namespace SKU Tier used for new Event Hub namespace. Applicable only when ‘From new Event Hub’ is selected.

18. For New Event Hub Namespace Sku Capacity enter a value of Event Hub throughput units for Basic or Standard tiers, where value should be 0 to 20 throughput units. The Event Hubs premium units for Premium tier, where value should be 0 to 10 premium units. Applicable only when ‘From new Event Hub’ is selected.

19. For Event Hub Name enter Event Hub instance name. A new Event Hub instance with this name will be created or an existing Event Hub instance with this name will be used.

20. For Event Hub Name DLQ enter Event Hub name for the dead-letter queue, where messages will be delivered to in case connection to ESA is lost. A new Event Hub instance with this name will be created or an existing Event Hub with this name will be used.

21. For New Event Hub Partition Count enter number of partitions to create in a new Event Hub. Allowed values are from 1 to 32 partitions. Applicable only when ‘From new Event Hub’ is selected.

22. For New Event Hub Audit Log Retention In Days enter number of days audit logs will be available in Event Hub. Applies to both primary Event Hub and dead-letter queue Event Hub. Applicable only when ‘From new Event Hub’ is selected.

23. For Log Destination Esa Ip enter ESA IP address.

24. For Esa Client Cert enter single-line ESA client certificate. See section Certificate Authentication for details.

25. For Esa Client Cert Key Secret Name enter secret name which stores ESA client certificate single-line private key. See section Certificate Authentication for details.

26. For Key Vault Uri enter URI of the Key Vault that stores ESA username/password secrets.

27. For Esa Tls Disable Cert Verify Set to ‘0’ to enable ESA certificate validation. Set to ‘1’ to disable ESA certificate verification. Disable only for initial setup and development purposes, do not disable in production environments.

28. If ESA is configured with self-signed certificate, set Pty Esa Ca Server Cert. Use the ESA CA Server Certificate escaped content recorded in Certificates on ESA.

    Note that for development and troubleshooting purposes, ESA certificate validation can be disabled by either redeploying this function with this ARM template where Esa Tls Disable Cert Verify option is set to ‘1’ or by directly setting PTY_ESA_DISABLE_TLS_CERT_VERIFY environment variable to ‘1’.

29. For Esa Connect Timeout set time in seconds to wait for the ESA connection response. Minimum value: 1. Default: 5.

30. For Esa Virtual Host provide ESA virtual hostname. This configuration is optional. It can be used when proxy server is present and supports TLS SNI extension.

31. For Min Log Level select minimum log level. Accepted values: off, severe, warning, info, config, all

32. Select Review + create then Create. Wait for all resources to deploy

After deployment is complete:

1. Go to Outputs and record:

   Forward Function Name (ForwardFuncName):__________________

2. Record:

   Event Hub Name (EventHubName):__________________

   Event Hub Namespace (EventHubNamespace):__________________

Function System-Assigned Managed Identity

System-assigned Azure managed identity is enabled if user-assigned managed identity is not used. User-assigned managed identities offer less frequent updates to Azure resources and allow for configuration of permissions ahead of function creation.

If you have not created a user-assigned managed identity at Function User-Assigned Managed Identity, setup following role assignments for system-assigned managed identity:

1. Navigate to the function

2. Select Settings, Identity.

3. Confirm Status of system-assigned identity is already On on System Assigned tab

4. Click on Azure role assignments button.

5. Assign following roles to this identity:

   • Storage Blob Data Owner
   • Monitoring Metrics Publisher
   • Azure Event Hubs Data Receiver

6. From Azure console, navigate to Function App and select audit log forwarder function deployed in previous section.

7. Select Overview and click Restart button. Wait until function restart completes.

Update Function Key Vault Access Policies

The Key vault must be updated to allow the Function App to decrypt the policy files. The Forwarder is using policy to confirm the authenticity of audit logs it receives from Event Hub and to digitally sign the aggregated logs that it sends to ESA. Update the Key vault access policies with function identity. To update the key vault access policies:

1. From Azure console navigate to Key Vaults, select the Key Vault created in Key Vault.
2. Select Access policies.
3. Click Create.
4. Select the following permissions in Permissions tab: a. Get under Key Management Operations. b. Unwrap Key under Cryptographic Operations. c. Get under Secret Permissions.
5. Proceed Next to Principal.
6. For Principal provide function identity a. For functions with user-assigned identity enter identity recorded in step Function User-Assigned Managed Identity b. For functions with system-assigned identity enter function name recorded in step Install Log Forwarder via ARM template
7. Proceed Next to Application and Next again to Review + Create.
8. Review permissions and Create.

Test Log Forwarder Installation

Follow the steps to validate Log Forwarder installation. Successful Log Forwarder installation will aggregate logs, connect to ESA and send audit log events.

Testing in this section validates the connectivity between Log Forwarder and ESA. The sample policy included with the initial installation and test event below are not based on your ESA policy. Any logs forwarded to ESA which are not signed with a policy generated by your ESA will not be added to the audit store.

Install Log Forwarder and configure according to previous sections. Log Forwarder configuration MinLogLevel must be at least info level.

1. In the following command, replace ‘forwarder-function-name’ with your function name

2. In the following command, replace ‘forwarder-function-key’ with your function key

3. Run the command in PowerShell:

   
   $forwarderFunctionName='forwarder-function-name'
   $forwarderFunctionKey='forwarder-function-key'
   
   Invoke-WebRequest -UseBasicParsing -Uri "https://$forwarderFunctionName.azurewebsites.net/admin/functions/auditlogforwarder" `
   -Method POST `
   -Headers @{
     "x-functions-key" = $forwarderFunctionKey
   } `
   -ContentType "application/json" `
   -Body "{`"input`":`"{\`"additional_info\`":{\`"description\`":\`"Data unprotect operation was successful.\`",\`"request_id\`":\`"f0ffbbf8-ab5b-42b7-90f4-51db7443af77\`"},\`"cnt\`": 1,\`"correlationid\`": \`"clfwrqgme0021nj329mijk52w\`",\`"logtype\`": \`"Protection\`",\`"level\`": \`"SUCCESS\`",\`"origin\`": {  \`"hostname\`": \`"169.254.197.189\`",  \`"ip\`": \`"169.254.197.189\`", \`"time_utc\`": 1722941687},\`"protection\`": {\`"dataelement\`": \`"alpha\`", \`"operation\`": \`"Unprotect\`",\`"audit_code\`": 8,\`"policy_user\`": \`"test_user\`",\`"datastore\`": \`"SAMPLE_POLICY\`"},\`"process\`": {  \`"name\`": \`"N/A\`",  \`"id\`": \`"15\`",\`"thread_id\`": \`"2243954624\`",\`"user\`": \`"sbx_user1051\`", \`"platform\`": \`"Linux_x32\`"},\`"client\`": {\`"username\`":\`"sbx_user1051\`",\`"ip\`":\`"169.254.197.189\`"},\`"protector\`": {\`"family\`": \`"IAP Lambda\`",\`"version\`": \`"3.1.0\`",\`"vendor\`": \`"Cloud Protect\`",\`"pcc_version\`": \`"3.5.0.1\`", \`"core_version\`": \`"2.0.1\`"},\`"signature\`": { \`"key_id\`":\`"5f143892-bbe4-4794-b1f4-ed28ca2a077e\`", \`"checksum\`": \`"90BC9BF39354869BD4BC5381820D201797DF4AF53B5A7F5F3AE01EC607C41A6E\`"}}`"}"
   

   Note

   For Cloud API on Azure Government Cloud use the following URL: https://$forwarderFunctionName.azurewebsites.us/admin/functions/auditlogforwarder

   Note

   The Body content is a sample audit log. See Audit Logging for detail on audit log contents.

4. Run following query to see your function logs, allow for a few minutes for Azure to deliver the logs

   
   traces
   | project timestamp, message
   | where timestamp > ago(5m)
   

5. Test is successful if the logs contain the following entry:

   opensearch.0: All logs successfully send to destination
   

   If the log entry is not present, please consult the Troubleshooting section for common errors and solutions.

Update Protect Service With Event Hub details

In this section, Event Hub details will be provided to the Protect Service installation.

1. Navigate to the Protect function environment variables.

2. Set EVENTHUB_NAME to the output value recorded in Install Log Forwarder via ARM template.

3. Set EventHub__fullyQualifiedNamespace to the output value recorded in Install Log Forwarder via ARM template.

4. Apply and Confirm to apply the changes.

Update Policy Agent With Log Forwarder Function Target

Log Forwarder requires a Protegrity policy which is in sync with the Protector Service. This section will describe the steps to update the Policy Agent to include updating the Log Forwarder.

1. Navigate to the Policy Agent function created in Install Agent via ARM template

2. Select Settings > Environment variables > PROTEGRITY_PROTECT_FUNCTION

3. Edit the value for environment variable PROTEGRITY_PROTECT_FUNCTION to include the Log Forwarder function’s name in the comma separated list of function names.

4. Select Apply > Apply > Confirm to save the changes

5. Test Policy Agent installation as described in Test Agent Function Installation

Test Full Log Forwarder Installation

1. Install and configure Protegrity Agent, Protect Service and Log Forwarder components.
2. Set EVENT_LEVEL environment variable on Protect Service function to Informational.
3. Set PTY_LOG_LEVEL environment variable on both Protect Service function and Log Forwarder function to config.

Test Installation

1. Make a protect operation using a data element or user which will result in audit log generation

2. Navigate to the Logs for the Protect Service function

3. Execute ’traces’ query

4. Expect to see a log similar to the below:

   
   Completed publishing events for Event Hub: audit-logs (Partition Id/Key: '0'), Operation Id: 'e17bacd6-91e6-4fb5-8281-2929788bef09'. Service Retry Count: 0; Duration: '0.02' seconds
   

5. Navigate to the Logs for the Log Forwarder function

6. Execute ’traces’ query

7. Expect to see a log similar to the below:

   
   opensearch.0: All logs successfully send to destination
   

Troubleshooting

Configure additional logging for functions:

1. Set EVENT_LEVEL environment variable on Protect function to Informational.
2. Set PTY_LOG_LEVEL environment variable on both Protect function and Log Forwarder function to config.

Unhandled exception. System.Exception: Failed to initialize 
function type,expecting environment variable 
'AzureWebJobs.AuditLogForwarder.Disabled' 
to be set to either 'true' or 'false'

[2024/08/08 10:00:00] [error] [tls] error: unexpected EOF

Failed to aggregate audit logs, 1 audit logs dropped, error: 
The user, group or application 'appid=;oid=;iss=' does not have 
secrets get permission on key vault ';location='...
Status: 403 (Forbidden)
ErrorCode: Forbidden

TCP connection timed out: 001.001.001.001:9200

fail: Protect.Aggregate[0] Failed to aggregate audit logs, 
1 audit logs dropped, error: Invalid URI: The format of the URI 
could not be determined. 
info: iap[0] Shared memory segment POLICY does not exist: The 
system cannot find the file specified.

[Error] Azure-Messaging-EventHubs: An exception occurred while 
publishing a batch of events for buffered producer instance with 
identifier 'x' to Event Hub: audit-logs, Partition Id: '0', 
Operation Id: 'x'. Error Message: 'Unauthorized access.