Redshift

• 1: Overview
• 2: Architecture
• 2.1:
• 3: Installation
• 3.1: Prerequisites
• 3.2: Pre-Configuration
• 3.3: Protect Service Installation
• 3.4: Policy Agent Installation
• 3.5: Audit Log Forwarder Installation
• 3.6:
• 3.7:
• 3.8:
• 4: Understanding Redshift Objects
• 4.1:
• 4.2:
• 4.3:
• 5: Performance
• 5.1:
• 5.2:
• 5.3:
• 5.4:
• 6: Audit Logging
• 7: No Access Behavior
• 8: Known Limitations
• 9: Upgrading To The Latest Version
• 10: Appendices
• 10.1: Installing the Policy Agent and Protector in Different AWS Accounts
• 10.2: Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)
• 10.3: Policy Agent - Custom VPC Endpoint Hostname Configuration
• 10.4: Redshift Cross-Account Configuration
• 10.5: Sample Redshift External Function
• 10.6: Configuring Regular Expression to Extract Policy Username
• 10.7: Associating ESA Data Store With Cloud Protect Agent

1 - Overview

Solution Overview

Amazon Redshift Protector is a cloud native, serverless product for fine-grained data protection with Redshift™, a managed Cloud data warehouse. This enables invocation of the Protegrity data protection cryptographic methods from the Redshift SQL execution context. The benefits of serverless include rapid auto-scaling, performance, low administrative overhead, and reduced infrastructure costs compared to a server-based solution.

This product provides data protection services invoked by External User Defined Functions (UDFs) within Amazon Redshift. The UDFs act as a client transmitting micro-batches of data to the serverless Protegrity Lambda function. User queries may generate hundreds or thousands of parallel requests to perform security operations. Protegrity’s serverless function is designed to scale and yield reliable query performance under such load.

Amazon Redshift Protector utilizes a data security policy maintained by an Enterprise Security Administrator (ESA), similar to other Protegrity products. Using regular SQL database queries or tools, such as, Tableau™, authorized users can perform de-identification (protect) and re-identification (unprotect) operations on data within the managed Cloud data warehouse. A user’s individual capabilities are subject to privileges and policies defined by the Enterprise Security Administrator.

The following data ingestion patterns are available with your managed Cloud data warehouse:

• Data protection at source applications: In this case, sensitive data is already de-identified (protected) across the enterprise wherever it resides, including the managed data warehouse. Protected data can be ingested directly into your managed Cloud data warehouse. Depending on usage patterns, this ensures that your managed data warehouse is not brought into scope for PCI, PII, GDPR, HIPPA, and other compliance policies.
• Data protection using the Extract-Transform-Load (ETL) pattern: In this case, sensitive data may be transformed with a Protegrity protector either on-premise or in the Cloud before it is ingested into Redshift.
• Data protection using the Extract-Load-Transform (ELT) pattern: In this case, sensitive data is protected after it lands into the target system typically through a temporary landing table. It uses the native data warehouse’s compute engine with Protegrity to protect incoming data at very high throughput rates. After the data is protected, the intermediate loading tables are dropped as part of the ingestion workflow.

Analytics on Protected Data

Protegrity’s format and length preserving tokenization scheme make it possible to perform analytics directly on protected data. Tokens are join-preserving so protected data can be joined across datasets. Often statistical analytics and machine learning training can be performed without the need to re-identify protected data. However, a user or service account with authorized security policy privileges may re-identify subsets of data using the Redshift Protector on AWS service.

Features

Amazon Redshift Protector incorporates Protegrity’s patent-pending vaultless tokenization capabilities into cloud-native serverless technology. Combined with an ESA security policy, Protegrity provides the following features:

• Fine grained field-level protection in the managed Cloud data warehouse
• Role-based access control (RBAC) to protect and unprotect (re-identify) data depending on the user privileges
• Policy enforcement features of other Protegrity application protectors.

For more information about the available protection options, such as, data types, Tokenization or Encryption types, or length preserving and non-preserving tokens, refer to Protection Methods Reference.

2 - Architecture

Protector Deployment Architecture

The Protegrity product should be deployed in the customer’s Cloud account within the same AWS region as the Redshift cluster. The product incorporates Protegrity’s vaultless tokenization engine within an AWS Lambda Function. The encrypted data security policy from an ESA is deployed periodically as a static resource through an AWS Lambda Layer. The policy is decrypted in memory at runtime within the Lambda. This architecture allows Protegrity to be highly available and scale very quickly without direct dependency on any other Protegrity services.

The product exposes a remote data protection service invoked from external User Defined Functions (UDFs), a native feature of specific Cloud databases. The UDFs can be invoked through direct SQL queries or database views. The external UDF makes parallel calls to the serverless Protegrity Lambda function to perform protect and unprotect data operations. Each network REST request includes a micro-batch of data to process and a secure context header generated by the database with the username and a Protegrity context header with the data element type, and operation to perform. The product applies the ESA security policy including user authorization and returns a corresponding response.

The security policy is synchronized through another serverless component called the Protegrity Policy Agent. The agent operates on a configurable schedule, fetches the policy from the ESA, performs additional envelope encryption using Amazon KMS, and deploys the policy into the Lambda Layer used by the serverless product. This solution can be configured to automatically provision the static policy or the final step can be performed on-demand by an administrator. The policy takes effect immediately for all new requests. There is no downtime for users during this process.

The following diagram shows the high-level architecture described above.

The following diagram shows a reference architecture for synchronizing the security policy from ESA.

The Protegrity Policy Agent requires network access to an Enterprise Security Administrator (ESA). Most organizations install the ESA on-premise. Therefore, it is recommended that the Policy Agent is installed into a private subnet with a Cloud VPC using a NAT Gateway to enable this communication through a corporate firewall.

The ESA is a soft appliance that must be pre-installed on a separate server. It is used to create and manage security policies.

For more information about installing the ESA, and creating and managing policies, refer the Policy Management Guide.

Log Forwarding Architecture

Audit logs are by default sent to CloudWatch as long as the function’s execution role has the necessary permissions. The Protegrity Product can also be configured to send audit logs to ESA. Such configuration requires deploying Log Forwarder component which is available as part of Protegrity Product deployment bundle. The diagram below shows additional resources deployed with Log Forwarder component.

The log forwarder component includes Amazon Kinesis data stream and the forwarder Lambda function. Amazon Kinesis stream is used to batch audit records before sending them to forwarder function, where similar audit logs are aggregated before sending to ESA. Aggregation rules are described in the Protegrity Log Forwarding guide. When the protector function is configured to send audit logs to log forwarder, audit logs are aggregated on the protector function before sending to Amazon Kinesis. Due to specifics of the Lambda runtime lifecycle, audit logs may take up to 15 minutes before being sent to Amazon Kinesis. Protector function exposes configuration to minimize this time which is described in the protector function installation section.

The security of audit logs is ensured by using HTTPS connection on each link of the communication between protector function and ESA. Integrity and authenticity of audit logs is additionally checked on log forwarder which verifies individual logs signature. The signature verification is done upon arrival from Amazon Kinesis before applying aggregation. If signature cannot be verified, the log is forwarded as is to ESA where additional signature verification can be configured. Log forwarder function uses client certificate to authenticate calls to ESA.

To learn more about individual audit log entry structure and purpose of audit logs, refer to Audit Logging section in this document. Installation instructions can be found in Audit Log Forwarder installation.

The audit log forwarding requires network access from the cloud to the ESA. Most organizations install the ESA on-premise. Therefore, it is recommended that the Log Forwarder Function is installed into a private subnet with a Cloud VPC using a NAT Gateway to enable this communication through a corporate firewall.

Redshift Connectivity

The AWS Redshift system is a cloud data warehouse Platform-as-a-Service (PaaS) from AWS. Unlike traditional (on-prem or customer-managed) data warehouses and RDBMS systems, Redshift is a fully managed, multi-tenant PaaS service, where the customers do not have the ability to install 3rd party software within the Redshift infrastructure.

Currently, the Redshift architecture does not allow installing Protegrity data protection UDFs and policy enforcement point (PEP) agents inside the Redshift infrastructure. Therefore, a remote-coupling integration mechanism where Redshift invokes RESTful APIs exposed by the Protegrity product is used.

Redshift supports remote invocations to the Protegrity infrastructure through External Functions, which in turn calls AWS Lambda.

Access and authorization between various AWS services involved in this architecture is achieved through Identity Access Management (IAM). For instance, Redshift is given an IAM role to call Lambda, Lambda is given an IAM role to call Secrets Manager, etc. Various IAM role configuration settings are shown in the appendices of this document.

The following figure illustrates the Protegrity Redshift Integration architecture.

2.1 -

Redshift Connectivity

The AWS Redshift system is a cloud data warehouse Platform-as-a-Service (PaaS) from AWS. Unlike traditional (on-prem or customer-managed) data warehouses and RDBMS systems, Redshift is a fully managed, multi-tenant PaaS service, where the customers do not have the ability to install 3rd party software within the Redshift infrastructure.

Currently, the Redshift architecture does not allow installing Protegrity data protection UDFs and policy enforcement point (PEP) agents inside the Redshift infrastructure. Therefore, a remote-coupling integration mechanism where Redshift invokes RESTful APIs exposed by the Protegrity product is used.

Redshift supports remote invocations to the Protegrity infrastructure through External Functions, which in turn calls AWS Lambda.

Access and authorization between various AWS services involved in this architecture is achieved through Identity Access Management (IAM). For instance, Redshift is given an IAM role to call Lambda, Lambda is given an IAM role to call Secrets Manager, etc. Various IAM role configuration settings are shown in the appendices of this document.

The following figure illustrates the Protegrity Redshift Integration architecture.

3 - Installation

3.1 - Prerequisites

AWS Services

The following table describes the AWS services that may be a part of your Protegrity installation.

ESA Version Requirements

The Protector and Log Forwarder functions require a security policy from a compatible ESA version.

The table below shows compatibility between different Protector and ESA versions.

Prerequisites

Required Skills and Abilities

What’s Next

• Pre-Configuration

3.2 - Pre-Configuration

Provide AWS sub-account

Identify or create an AWS account where the Protegrity solution will be installed. It is recommended that a new AWS sub-account be created. This can provide greater security controls and help avoid conflicts with other applications that might impact regional account limits. An individual with the Cloud Administrator role will be required for some subsequent installation steps.

AWS Account ID: ___________________

AWS Region (AwsRegion): ___________________

Determine AWS Region

Determine the AWS region where the Amazon Redshift cluster is running. This is the region in where the Protegrity solution must be installed.

AWS Region (AccountRegion): ___________________

Create S3 bucket for Installing Artifacts

This S3 bucket will be used for the artifacts required by the CloudFormation installation steps. This S3 bucket must be created in the region that is defined in Provide AWS sub-account

1. Sign in to the AWS Management Console and open the Amazon S3 console.

2. Change region to the one determined in Provide AWS sub-account

3. Click Create Bucket.

4. Enter a unique bucket name:

   For example, protegrity-install.us-west-2.example.com

5. Upload the installation artifacts to this bucket. Protegrity will provide the following three artifacts:

   • protegrity-protect-<version>.zip
   • protegrity-agent-<version>.zip
   • protegrity-external-extension-<version>.zip
   • protegrity-sample-policy-<version>.zip

     Important

     The deployment package you receive from Protegrity must be extracted to reveal the Protegrity artifacts. CloudFormation requires them in the provided .zip format. Do not extract the individual Protegrity artifacts. Upload these artifacts to the S3 bucket created.

   S3 Bucket name (ArtifactS3Bucket): ___________________

Create KMS Key

The Amazon Key Management Service (KMS) provides the ability for the Protegrity Serverless solution to encrypt and decrypt the Protegrity Security Policy.

To create KMS key:

1. In the AWS sub-account where the KMS key will reside, select the region.

2. Navigate to Key Management Service > Create Key.

   

3. Configure the key settings:

   • Key type: Asymmetric
   • Key usage: Encrypt and decrypt
   • Key spec: RSA_4096
   • Click Next

4. Create alias and optional description, such as, Protegrity-Serverless and click Next.

5. Define key administrative permissions, the IAM user who will administrate the key.

   Note

   It is recommended the administrator be different than the administrator of the Protegrity Serverless account

6. Click Next.

7. Define the key usage permissions.

8. In Other AWS accounts, enter the AWS account id used for the Protegrity Serverless installation.

9. Continue on to create the key. If there is a concern this permission is overly broad, then you can return later to restrict access to the role of two Protegrity Serverless Lambda as principals. Click to open the key in the list and record the ARN.

   KMS Key ARN (AWS_KMS_KEY_ID): ___________________

10. Download the public key from the KMS key. Navigate to the key in KMS console, select the Public key tab, and click Download. Save the PEM file. This public key will be added to the ESA data store as an export key. Refer to Exporting Keys to Datastore for instructions on adding the public key to the data store.

    Note

    This step is not applicable for ESA versions lower than 10.2.

    KMS Public Key PEM file: ___________________

Create IAM Account Role

An IAM role is used to authorize Redshift to access the future Protect Lambda resource.

To create IAM account role:

1. From the AWS console, access IAM, select Roles and then Create Role.

2. In Trusted entity type section, select AWS service

3. In Use case section, for Service or use case, select Redshift.

4. For Use case, select Redshift – Customizable.

5. Click Next to advance to Add Permissions step.

6. Click Next to skip Add Permissions step.

7. Enter a Role name, for example, RedshiftProtegrity.

8. Click Create role.

9. After the role is created, click on the role. Record the following information:

   • Role Name (DBRoleName): ____________________
   • Role ARN (DBRoleARN): ____________________

What’s Next

• Protect Service Installation

3.3 - Protect Service Installation

Preparation

1. Ensure that all the steps in Pre-Configuration are performed.

2. Login to the AWS account console where Amazon Redshift Protector will be installed.

3. Ensure that the required CloudFormation templates provided by Protegrity are available on your local computer.

Create Protect Lambda IAM Execution Policy

This task defines a policy used by the Protegrity Lambda function to write CloudWatch logs and access the KMS encryption key to decrypt the policy.

Perform the following steps to create the Lambda execution role and required policies:

1. From the AWS IAM console, select Policies > Create Policy.

2. Select the JSON tab and copy the following sample policy.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "CloudWatchWriteLogs",
         "Effect": "Allow",
         "Action": [
           "logs:CreateLogGroup",
           "logs:CreateLogStream",
           "logs:PutLogEvents"
         ],
         "Resource": "*"
       },
       {
         "Sid": "KmsDecrypt",
         "Effect": "Allow",
         "Action": [
           "kms:Decrypt"
         ],
         "Resource": [
           "arn:aws:kms:*:*:key/*"
         ]
       }
     ]
   }
   

3. For the KMS policy, replace the Resource with the ARN for the KMS key created in a previous step.

4. Select Next, type in a policy name, for example, ProtegrityProtectLambdaPolicy and Create Policy. Record the policy name:

   ProtectLambdaPolicyName:__________________

Create Protect Lambda IAM Role

The following steps create the role to utilize the policy defined in Create Protect Lambda IAM Execution Policy.

To create protect lambda IAM execution role:

1. From the AWS IAM console, select Roles > Create Role.

2. Select AWS Service > Lambda > Next.

3. In the list, search and select the policy created in Create Protect Lambda IAM Execution Policy.

4. Click Next

5. Type the role name, for example, ProtegrityProtectRole

6. Click Create role

7. Record the role ARN.

   Role ARN (LambdaExecutionRoleArn): ___________________

Install through CloudFormation

The following steps describe the deployment of the Lambda function.

To install through CloudFormation:

1. Access CloudFormation and select the target AWS Region in the console.

2. Click Create Stack and choose With new resources.

3. Specify the template.

4. Select Upload a template file.

5. Upload the Protegrity-provided CloudFormation template called pty_protect_cf.json and click Next.

6. Specify the stack details. Enter a stack name.

   Note

   The stack name will be appended to all the services created by the template.

7. Enter the required parameters. All the values were generated in the pre-configuration steps.

   Parameter	Description
   DBRoleName	Name of the account role created in the pre-configuration step
   ArtifactS3Bucket	Name of S3 bucket created in the pre-configuration step
   LambdaExecutionRoleArn	The ARN of Lambda role created in the prior step
   MinLogLevel	Minimum log level for protect function. Allowed Values: off, severe, warning, info, config, all
   UsernameRegex	If set, the effective policy user will be extracted from the user in the request. Note See Configuring Regular Expression to Extract Policy Username to learn how to extract username from the request

8. If you are not planning to deploy log forwarder you can skip this step. The log forwarder parameters can be provided later after log forwarder is deployed.

   Log Forwarder Parameters	Description
   KinesisLogStreamArn	The ARN of the AWS Kinesis stream where audit logs will be sent for aggregation
   AuditLogFlushInterval	Time interval in seconds used to accumulate audit logs before sending to Kinesis. Default value: 30. See Log Forwarder Performance section for more details.

9. Proceed to the last step of the Create Stack wizard with defaults and click Submit to create CloudFormation stack.

10. Select Outputs tab of the stack after stack is created.

11. Record the following values:

    ProtectFunctionName: __________________________

    ProtectFunctionProductionAlias: __________________________

    ProtectLayerName: _____________________________

Protect Lambda Configuration

After CloudFormation stack is deployed, the Protect Lambda default configuration can be changed using Lambda environment configuration. See below for list of available configuration and instructions how to update it.

• List of Protect Lambda Environment Variables

  Variable Name	Description	Notes
  LOG_REDSHIFT_CLUSTER_ARN	When enabled, Redshift cluster ARN is recorded in CloudWatch audit log.	Set LOG_REDSHIFT_CLUSTER_ARN = 1 to enable. See Audit Logging for audit log examples.

• Updating Lambda Configuration

  From your AWS console, navigate to Lambda and select the following Lambda:

  Protegrity_Protect_<STACK_NAME>

  Select Configuration tab and scroll down to the Environment variables section. Select Edit then Add environment variable. For the list of allowed variables and corresponding values refer to the table above.

Delete API Gateway (Optional)

CloudFormation created an API Gateway that makes this product compatible with Snowflake. However, this service is not required for Amazon Redshift so it may be optionally removed.

To delete API Gateway:

1. From Services, access API Gateway.

2. Select the API Gateway service that is created.

   Note

   Sort by Created date. The name will have the same extension as the stack name created above

3. Click Actions > Delete.

Add Lambda Protect IAM Permissions to Role

To add inline Lambda Protect IAM permissions to role:

1. Select the role created in section Create IAM Account Role (DBRoleName).

   Tip

   Sort descending by created date.

2. In the Permissions tab, click Add inline policy.

3. Select the JSON tab and copy the following sample policy.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "ProtegrityProtectInvokePermission",
         "Effect": "Allow",
         "Action": "lambda:InvokeFunction",
         "Resource": "arn:aws:lambda:*:*:function:*"
       }
     ]
   }
   

4. Edit the Resource value in the snippet with the ARN of the Protect Lambda function to further restrict privileges.

Attach IAM Account Role to Redshift

Perform the following steps to allow Redshift to invoke the Protect Lambda function.

Attach the IAM role (DBRoleName) created in Create IAM Account Role to the Redshift cluster.

To attach IAM account role to Redshift:

1. From the AWS console, access Amazon Redshift and click on the cluster.

2. Select Actions > Manage IAM Roles.

3. Under Available IAM roles, select Enter ARN.

4. Enter the Role ARN recorded in Create Protect Lambda IAM Role.

5. Click Add IAM role.

   Note

   After attaching the role, wait for the cluster to complete modification status (in the cluster status) before proceeding to test connectivity.

Test Connectivity

Perform the following steps to verify if Redshift is working correctly with the Protegrity product.

To test the connectivity:

1. Access the Redshift SQL console.

2. Copy and paste the following snippet into a worksheet.

   CREATE OR REPLACE EXTERNAL FUNCTION pty_unprotect_alpha(varchar) RETURNS varchar
   VOLATILE lambda '<replace_with_protect_function_name>:Production' iam_role 'arn:aws:iam::<you-awsaccount-
   number>:role/<role-name>
   

3. Replace the placeholder values with the lambda function name and role created in earlier steps.

4. Run the following command in the console:

   select pty_unprotect_alpha('UtfVk UHgcD!')
   

5. Verify that the string hello world! is returned.

Troubleshooting Tips

What’s Next

• Install Policy Agent

3.4 - Policy Agent Installation

The following sections will install the Policy Agent. The Policy Agent polls the ESA and deploys the policy to Protegrity Serverless as a static resource. Some of the installation steps are not required for the operation of the software but recommended for establishing a secure environment. Contact Protegrity Professional Services for further guidance on configuration alternatives in the Cloud.

ESA Server

Policy Agent Lambda requires ESA server running and accessible on TCP port 443.

Note down ESA IP address:

ESA IP Address (EsaIpAddress): ___________________

Certificates on ESA

Whether your ESA is configured with default self-signed certificate or your corporate CA certificate, Policy Agent can validate authenticity of ESA connection using CA certificate. The process for both scenarios is the same:

• Obtain CA certificate
• Convert CA certificate to a value accepted by Policy Agent
• Provide converted CA certificate value to Policy Agent

To obtain self-signed CA certificate from ESA:

1. Log in to ESA Web UI.

2. Select Settings > Network > Manage Certificates.

3. Hover over Server Certificate and click on download icon to download the CA certificate.

4. To convert downloaded CA certificate to a value accepted by Policy Agent, open the downloaded PEM file in text editor and replace all new lines with escaped new line: \n.

   To escape new lines from command line, use one of the following commands depending on your operating system:

   Linux Bash:

   awk 'NF {printf "%s\\n",$0;}' ProtegrityCA.pem > output.txt
   

   Windows PowerShell:

   (Get-Content '.\ProtegrityCA.pem') -join '\n' | Set-Content 'output.txt'
   

5. Record the certificate content with new lines escaped.

   ESA CA Server Certificate (EsaCaCert): ___________________

   This value will be used to set PTY_ESA_CA_SERVER_CERT or PTY_ESA_CA_SERVER_CERT_SECRET Lambda variable in section Policy Agent Lambda Configuration

For more information about ESA certificate management refer to Certificate Management Guide in ESA documentation.

Identify or Create a new VPC

Establish a VPC where the Policy Agent will be hosted. This VPC will need connectivity to the ESA. The VPC should be in the same account and region established in Pre-Configuration.

VPC name: ___________________

VPC Subnet Configuration

Identify or create a new subnet in the VPC where tha Lambda function will be connected to. It is recommended to use a private subnet.

Subnet name: ___________________

NAT Gateway For ESA Hosted Outside AWS Network

If ESA server is hosted outside of the AWS Cloud network, the VPC configured for Lambda function must ensure additional network configuration is available to allow connectivity with ESA. For instance if ESA has a public IP, the Lambda function VPC must have public subnet with a NAT server to allow routing traffic outside of the AWS network. A Routing Table and Network ACL may need to be configured for outbound access to the ESA as well.

VPC Endpoints Configuration

If an internal VPC was created, then add VPC Endpoints, which will be used by the Policy Agent to access AWS services. Policy Agent needs access to the following AWS services:

Identify or Create Security Groups

Policy Agent and cloud-based ESA appliance use AWS security groups to control traffic that is allowed to leave and reach them. Policy Agent runs on schedule and is mostly concerned with allowing traffic out of itself to ESA and AWS services it depends on. ESA runs most of the time and it must allow Policy Agent to connect to it.

Policy Agent security group must allow outbound traffic using rules described in the table below. To edit security group navigate:

From VPC > Security Groups > Policy Agent Security Group configuration.

Record Policy Agent security group ID:

Policy Agent Security Group Id: ___________________

Policy Agent will reach out to ESA on port 443. Create following inbound security group rule for cloud-based ESA appliance to allow connections from Policy Agent:

Creating ESA Credentials

Policy Agent Lambda requires ESA credentials to be provided as one of the three options.

• Option 1: Secrets Manager

• Option 2: KMS Encrypted Password

• Option 3: Custom AWS Lambda function

Option 1: Secrets Manager

Creating secrets manager secret with ESA username and password.

1. From the AWS Secrets Manager Console, select Store New Secret.

2. Select Other Type of Secrets.

3. Specify the username and password key value pair.

   

4. Select the encryption key or leave default AWS managed key.

5. Specify the Secret Name and record it.

   ESA Credentials Secret Name: __________________

Option 2: KMS Encrypted Password

ESA password is encrypted with AWS KMS symmetric key.

1. Create AWS KMS symmetric key which will be used to encrypt ESA password. See Create KMS Key for instructions on how to create KMS symmetric key using AWS console.

2. Record KMS Key ARN.

   ESA PASSWORD KMS KEY ARN: __________________

3. Run AWS CLI command to encrypt ESA password. Below you can find sample Linux aws cli command. Replace <key_arn> with KMS symmetric key ARN.

   aws kms encrypt --key-id <key_arn> --plaintext $(echo '<esa_password>' | base64 )
   

4. Sample output.

   {
     "CiphertextBlob": "esa_encrypted_password",
     "KeyId": "arn:aws:kms:region:aws_account:key/key_id ",
     "EncryptionAlgorithm": "SYMMETRIC_DEFAULT"
   }
   

5. Record ESA username and encrypted password.

   ESA USERNAME: __________________

   ESA ENCRYPTED PASSWORD: __________________

Option 3: Custom AWS Lambda function

With this option ESA username and password are returned by a custom AWS Lambda function. This method may be used to get the username and password from external vaults.

1. Create AWS Lambda in any AWS supported runtime.

   1. There is no input needed.

   2. The Lambda function must return the following response schema.

      response:
      type: object
        properties:
          username: string
          password: string
      

      For example,

      example output: {"username": "admin", "password": "Password1234"}
      

   3. Sample AWS Lambda function in Python:

      import json
      
      def lambda_handler(event, context):
      
          return {"username": "admin", "password": "password1234"}
      

      Warning

      Protegrity does not recommend hardcoding ESA username and password in the clear.

2. Record the Lambda name:

   Custom AWS lambda for ESA credentials: _______________

Create Agent Lambda IAM Policy

Follow the steps below to create Lambda execution policies.

Create Agent Lambda IAM policy

1. From AWS IAM console, select Policies > Create Policy.

2. Select JSON tab and copy the following snippet.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "EC2ModifyNetworkInterfaces",
         "Effect": "Allow",
         "Action": [
           "ec2:CreateNetworkInterface",
           "ec2:DescribeNetworkInterfaces",
           "ec2:DeleteNetworkInterface"
         ],
         "Resource": "*"
       },
       {
         "Sid": "CloudWatchWriteLogs",
         "Effect": "Allow",
         "Action": [
           "logs:CreateLogGroup",
           "logs:CreateLogStream",
           "logs:PutLogEvents"
         ],
         "Resource": "*"
       },
       {
         "Sid": "LambdaUpdateFunction",
         "Effect": "Allow",
         "Action": [
           "lambda:UpdateFunctionConfiguration"
         ],
         "Resource": [
           "arn:aws:lambda:*:*:function:*"
         ]
       },
       {
         "Sid": "LambdaReadLayerVersion",
         "Effect": "Allow",
         "Action": [
           "lambda:GetLayerVersion",
           "lambda:ListLayerVersions"
         ],
         "Resource": "*"
       },
       {
         "Sid": "LambdaDeleteLayerVersion",
         "Effect": "Allow",
         "Action": "lambda:DeleteLayerVersion",
         "Resource": "arn:aws:lambda:*:*:layer:*:*"
       },
       {
         "Sid": "LambdaPublishLayerVersion",
         "Effect": "Allow",
         "Action": "lambda:PublishLayerVersion",
         "Resource": "arn:aws:lambda:*:*:layer:*"
       },
       {
         "Sid": "S3GetObject",
         "Effect": "Allow",
         "Action": [
           "s3:GetObject"
         ],
         "Resource": "arn:aws:s3:::*/*"
       },
       {
         "Sid": "S3PutObject",
         "Effect": "Allow",
         "Action": [
           "s3:PutObject"
         ],
         "Resource": "arn:aws:s3:::*/*"
       },
       {
         "Sid": "KmsEncrypt",
         "Effect": "Allow",
         "Action": [
           "kms:GetPublicKey"
         ],
         "Resource": [
           "arn:aws:kms:*:*:key/*"
         ]
       },
       {
         "Sid": "SecretsManagerGetSecret",
         "Effect": "Allow",
         "Action": [
           "secretsmanager:GetSecretValue"
         ],
         "Resource": [
           "arn:aws:secretsmanager:*:*:secret:*"
         ]
       },
       {
         "Sid": "LambdaGetConfiguration",
         "Effect": "Allow",
         "Action": [
             "lambda:GetFunctionConfiguration"
         ],
         "Resource": [
             "arn:aws:lambda:*:*:function:*"
         ]
       }
     ]
   }
   

3. Replace wildcard * with the region, account, and resource name information where required.

4. This step is required if KMS is used to encrypt ESA password.

   Add policy entry below. Replace ESA PASSWORD KMS KEY ARN with the value recorded in Option 2: KMS Encrypted Password.

   
     {
       "Sid": "KmsDecryptEsaPassword",
       "Effect": "Allow",
       "Action": [
         "kms:Decrypt"
       ],
       "Resource": [
         "**ESA PASSWORD KMS KEY ARN**"
       ]
     }
   

5. Select Next type in the policy name and Create Policy. Record policy name:

   Policy Name: ___________________

Create Agent Lambda IAM Role

Perform the following steps to create Agent Lambda execution IAM role.

To create agent Lambda IAM role:

1. From AWS IAM console, select Roles > Create Role.

2. Select AWS Service > Lambda > Next.

3. Select the policy created in Create Agent Lambda IAM policy.

4. Proceed to Name, Review and Create.

5. Type the role name, for example, ProtegrityAgentRole and click Confirm.

6. Select Create role.

7. Record the role ARN.

   Agent Lambda IAM Execution Role Name: ___________________

Corporate Firewall Configuration

If an on-premise firewall is used, then the firewall must allow access from the NAT Gateway to an ESA. The firewall must allow access from the NAT Gateway IP to ESA via port 443 and 443.

CloudFormation Installation

Create the Policy Agent in the VPC using the CloudFormation script provided by Protegrity.

1. Access the CloudFormation service.

2. Select the target installation region.

3. Create a stack with new resources.

4. Upload the Policy Agent CloudFormation template (file name: pty_agent_cf.json).

5. Specify the following parameters for Cloud Formation:

   Parameter	Description	Note
   VPC	VPC where the Policy Agent will be hosted	Identify or Create a new VPC
   Subnet	Subnet where the Policy Agent will be hosted	VPC Subnet Configuration
   PolicyAgentSecurityGroupId	Security Group Id, which allows communication between the Policy Agent and the ESA	Identify or Create Security Groups
   LambdaExecutionRoleArn	Agent Lambda IAM execution role ARN allowing access to the S3 bucket, KMS encryption Key, Lambda and Lambda Layer	Create Agent Lambda IAM Role
   ArtifactS3Bucket	S3 bucket name with deployment package for the Policy Agent	Use S3 Bucket name recorded in Create S3 bucket for Installing Artifacts
   CreateCRONJob	Set to True to create a CloudWatch schedule for the agent to run.	Default: False

Policy Agent Lambda Configuration

After the CloudFormation stack is deployed, the Policy Agent Lambda must be configured with parameters recorded in earlier steps. From your AWS Console, navigate to lambda and select the following Lambda.

Protegrity_Agent<STACK_NAME>_

Select Configuration tab and scroll down to the Environment variables section. Select Editand replace all entries with the actual values.

{
  "PTY_ESA_CA_SERVER_CERT":"-----BEGIN CERTIFICATE----- MIIF..."
}

Test Installation

Open the Lambda and configure Test to execute the lambda and specify the default test event. Wait for around 20 seconds for the Lambda to complete. If policy is downloaded successfully, then a success message appears.

Navigate to the AWS_POLICY_S3_BUCKET bucket and verify that the AWS_POLICY_S3_FILENAME file was created.

Troubleshooting

2020-10-06T23:40:54.121Z 2dc84942-b5cc-4be9-aa4c-965f322307e4 Task timed out after 90.09 seconds

{
  "errorMessage": "Timeout! Unable to download policy in 20 seconds.",
  "errorType": "Exception",
  "stackTrace": [...]
}

{
  "errorMessage": "ESA connection error. Failed to download certificates. HTTP response code: 401.",
  "errorType": "ConnectionError",
  "stackTrace": [...]
}

xyz Access Denied: Exception
Traceback (most recent call last):
  … Exception: xyz Access Denied

Secrets Manager Access Denied: Exception
Traceback (most recent call last):
  … Exception: Secrets Manager Access Denied

[ERROR] PolicyAgentException: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

kms:Decrypt
kms:GenerateDatakey

An error occurred (AccessDeniedException) when calling the PublishLayerVersion operation: Your access has been denied by S3,
please make sure your request credentials have permission to GetObject for BUCKET_NAME/FILENAME. S3 Error Code: AccessDenied. S3 Error Message: Access Denied

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VpcRestrictions",
      "Effect": "Deny",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:*Object",
      "Resource": [
        "arn:aws:s3:::<s3_bucket_name>/*",
        "arn:aws:s3:::<s3_bucket_name>"
      ],
      "Condition": {
        "Bool": {
          "aws:ViaAWSService": "false"
        },
        "StringNotEquals": {
          "aws:sourceVpc": "<vpc_id>"
        }
      }
    }
  ]
}

Additional Configuration

Strengthen the KMS IAM policy by granting access only to the required Lambda function(s).

Finalize the IAM policy for the Lambda Execution Role. Ensure to replace wildcard * with the region, account, and resource name information where required.

For example,

"arn:aws:lambda:*:*:function:*" -> "arn:aws:lambda:us-east-1:account:function:function_name"

Policy Agent Schedule

If specified in CloudFormation Installation, the agent installation created a CloudWatch event rule, which checks for policy update on an hourly schedule. This schedule can be altered to the required frequency.

Under CloudWatch > Events > Rules, find Protegrity_Agent_{stack_name}. Click Action > Edit Set the cron expression. A cron expression can easily be defined using CronMaker, a free online tool. Refer to http://www.cronmaker.com.

What’s Next

• Audit Log Forwarder Installation

3.5 - Audit Log Forwarder Installation

The following sections show steps how to install Audit Log Forwarder component in the AWS Cloud. The Log Forwarder deployment allows for the audit logs generated by Protector to be delivered to ESA for auditing and governance purposes. Log Forwarder component is optional and is not required for the Protector Service to work properly. See Log Forwarding Architecture section in this document for more information. Some of the installation steps are not required for the operation of the software but recommended for establishing a secure environment. C ontact Protegrity for further guidance on configuration alternatives in the Cloud.

ESA Audit Store Configuration

ESA server is required as the recipient of audit logs. Verify the information below to ensure ESA is accessible and configured properly.

1. ESA server running and accessible on TCP port 9200 (Audit Store) or 24284 (td-agent).

2. Audit Store service is configured and running on ESA. Applies when audit logs are output to Audit Store directly or through td-agent. For information related to ESA Audit Store configuration, refer to Audit Store Guide.

3. (Optional) td-agent is configured for external input. For more information related to td-agent configuration, refer to ESA guide Sending logs to an external security information and event management (SIEM).

Certificates on ESA

By default, ESA is configured with self-signed certificates, which can optionally be validated using a self-signed CA certificate supplied in the Log Forwarder configuration. If no CA certificate is provided, the Log Forwarder will skip server certificate validation.

If ESA is configured with publicly signed certificates, this section can be skipped since the forwarder Lambda will use the public CA to validate ESA certificates.

To obtain the self-signed CA certificate from ESA:

1. Download ESA CA certificate from the /etc/ksa/certificates/plug directory of the ESA

2. After certificate is downloaded, open the PEM file in text editor and replace all new lines with escaped new line: \n.

   To escape new lines from command line, use one of the following commands depending on your operating system:

   Linux Bash:

   awk 'NF {printf "%s\\n",$0;}' ProtegrityCA.pem > output.txt
   

   Windows PowerShell:

   (Get-Content '.\ProtegrityCA.pem') -join '\n' | Set-Content 'output.txt'
   

3. Record the certificate content with new lines escaped.

   ESA CA Server Certificate (EsaCaCert): ___________________

   This value will be used to set PtyEsaCaServerCert cloudformation parameter in section Install through CloudFormation

For more information about ESA certificate management refer to Certificate Management Guide in ESA documentation.

AWS VPC Configuration

Log forwarder Lambda function requires network connectivity to ESA, similar to Policy Agent Lambda function. Therefore, it can be hosted in the same VPC as Policy Agent.

Separate VPC can be used, as long as it provides network connectivity to ESA.

VPC Name: ___________________

VPC Subnet Configuration

Log Forwarder can be connected to the same subnet as Policy Agent or separate one as long as it provides connectivity to ESA.

Subnet Name: ___________________

NAT Gateway For ESA Hosted Outside AWS Network

If ESA server is hosted outside of the AWS Cloud network, the VPC configured for Lambda function must ensure additional network configuration is available to allow connectivity with ESA. For instance if ESA has a public IP, the Lambda function VPC must have public subnet with a NAT server to allow routing traffic outside of the AWS network. A Routing Table and Network ACL may need to be configured for outbound access to the ESA as well.

VPC Endpoint Configuration

Log Forwarder Lambda function requires connectivity to Secrets Manager AWS service. If the VPC identified in the steps before has no connectivity to public internet through the NAT Gateway, then the following service endpoint must be configured:

• com.amazonaws.{REGION}.cloudwatch
• com.amazonaws.{REGION}.secretsmanager
• com.amazonaws.{REGION}.kms

Security Group Configuration

Security groups restrict communication between Log Forwarder Lambda function and the ESA appliance. The following rules must be in place for ESA and Log Forwarder Lambda function.

From VPC > Security Groups > Log Forwarder Security Group configuration.

Record the name of Log Forwarder security group name.

Log Forwarder Security Group Id: ___________________

The following port must be open for the ESA. If the ESA is running in the Cloud, then create the following security.

ESA Security Group configuration

Configure ESA Audit Store Credentials

Audit Log Forwarder can optionally authenticate with ESA using certificate-based authentication with a client certificate and certificate key. If used, both the certificate and certificate key will be stored in AWS Secrets Manager.

Download the following certificates from the /etc/ksa/certificates/plug directory of the ESA:

• client.key
• client.pem

After certificates are downloaded, open each PEM file in text editor and replace all new lines with escaped new line: \n. To escape new lines from command line, use one of the commands below depending on your operating system.

Linux Bash:

awk 'NF {printf "%s\\n",$0;}' client.key > private_key.txt
awk 'NF {printf "%s\\n",$0;}' client.pem > public_key.txt

Windows PowerShell:

(Get-Content '.\client.key') -join '\n' | Set-Content 'private_key.pem'
(Get-Content '.\client.pem') -join '\n' | Set-Content 'public_key.pem'

For more information on how to configure client certificate authentication for Audit Store on ESA refer to Audit Store Guide.

To create secret with ESA client certificate/key pair in AWS Secrets Manager.

1. From the AWS Secrets Manager Console, select Store New Secret.

2. Select Other Type of Secrets.

3. Specify the private_key and public_key value pair.

   

4. Select the encryption key or leave default AWS managed key.

5. Specify the Secret Name and record it below.

   ESA Client Certificate/Key Pair Secret Name: ___________________

   This value will be used to set PtyEsaClientCertificatesSecretId cloudformation parameter in section Install through CloudFormation

Create Audit Log Forwarder IAM Execution Policy

This task defines a policy used by the Protegrity Log Forwarder Lambda function to write CloudWatch logs, access the KMS encryption key to decrypt the policy and access Secrets Manager for log forwarder user credentials.

Perform the following steps to create the Lambda execution role and required policies:

1. From the AWS IAM console, select Policies > Create Policy.

2. Select the JSON tab and copy the following sample policy.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "EC2ModifyNetworkInterfaces",
         "Effect": "Allow",
         "Action": [
           "ec2:CreateNetworkInterface",
           "ec2:DescribeNetworkInterfaces",
           "ec2:DeleteNetworkInterface"
         ],
         "Resource": "*"
       },
       {
         "Sid": "CloudWatchWriteLogs",
         "Effect": "Allow",
         "Action": [
           "logs:CreateLogGroup",
           "logs:CreateLogStream",
           "logs:PutLogEvents"
         ],
         "Resource": "*"
       },
       {
         "Sid": "KmsDecrypt",
         "Effect": "Allow",
         "Action": [
           "kms:Decrypt"
         ],
         "Resource": [
           "arn:aws:kms:*:*:key/*"
         ]
       },
       {
           "Sid": "KinesisStreamRead",
           "Effect": "Allow",
           "Action": [
               "kinesis:GetRecords",
               "kinesis:GetShardIterator",
               "kinesis:DescribeStream",
               "kinesis:DescribeStreamSummary",
               "kinesis:ListShards",
               "kinesis:ListStreams"
           ],
           "Resource": "*"
       },
       {
         "Sid": "SecretsManagerGetSecret",
         "Effect": "Allow",
         "Action": [
           "secretsmanager:GetSecretValue"
         ],
         "Resource": [
           "arn:aws:secretsmanager:*:*:secret:*"
         ]
       }
     ]
   }
   

3. For the KMS policy, replace the Resource with the ARN for the KMS key created in a previous step.

4. Select Review policy, type in a policy name, for example, ProtegrityLogForwarderLambdaPolicy and Confirm. Record the policy name:

   LogForwarderLambdaPolicyName:__________________

Create Log Forwarder IAM Role

Perform the following steps to create Log Forwarder execution IAM role.

To create Log Forwarder IAM role:

1. From AWS IAM console, select Roles > Create Role.

2. Select AWS Service > Lambda > Next.

3. Select the policy created in Create Audit Log Forwarder IAM Execution Policy.

4. Proceed to Name, Review and Create.

5. Type the role name, for example, ProtegrityForwarderRole and click Confirm.

6. Record the role ARN.

   Log Forwarder IAM Execution Role Name: ___________________

Installation Artifacts

Audit Log Forwarder installation artifacts are part of the same deployment package as the one used for protect and policy agent services. Follow the steps below to ensure the right artifacts are available for log forwarder installation.

1. Verify that the Protegrity deployment package is available on your local system, if not, you can download it from the Protegrity portal.

   Note

   If you maintain multiple Protegrity Cloud Protectors, make sure that the deployment package downloaded for Audit Log Forwarder is the same as the one used for Protect service installation.

2. Extract the pty_log_forwarder_cf.json cloud formation file from the deployment package.

3. Check the S3 deployment bucket identified in section Create S3 bucket for Installing Artifacts. Make sure that all Protegrity deployment zip files are uploaded to the S3 bucket.

Install through CloudFormation

The following steps describe the deployment of the Audit Log Forwarder AWS cloud components.

1. Access CloudFormation and select the target AWS Region in the console.

2. Click Create Stack and choose With new resources.

3. Specify the template.

4. Select Upload a template file.

5. Upload the Protegrity-provided CloudFormation template called pty_log_forwarder_cf.json and click Next.

6. Specify the stack details. Enter a stack name.

   Note

   The stack name will be appended to all the services created by the template.

7. Enter the required parameters. All the values were generated in the pre-configuration steps.

8. Click Next with defaults to complete CloudFormation.

9. After CloudFormation is completed, select the Outputstab in the stack.

10. Record the following values

    KinesisLogStreamArn: ________________________________

Add Kinesis Put Record permission to the Protect Function IAM Role

1. Login to the AWS account that hosts the Protect Lambda Function.

2. Search for Protect Lambda Function IAM Execution Role Name created in Create Protect Lambda IAM role.

3. Under Permissions policies, select Add Permissions > Create inline policy.

4. In Specify permissions view, switch to JSON.

5. Copy the policy json from below replacing the placeholder value indicated in the following snippet as <Audit Log Kinesis Stream ARN> with the value recorded in the previous step.

   {
   	"Version": "2012-10-17",
   	"Statement": [
   		{
   			"Sid": "KinesisPutRecords",
   			"Effect": "Allow",
   			"Action": "kinesis:PutRecords",
   			"Resource": "<Audit Log Kinesis Stream ARN>"
   		}
   	]
   }
   

6. When you are finished, choose Next.

7. On the Review and create page, type a Name, then choose Create policy.

Test Log Forwarder Installation

Testing in this section validates the connectivity between Log Forwarder and ESA. The sample policy included with the initial installation and test event below are not based on your ESA policy. Any logs forwarded to ESA which are not signed with a policy generated by your ESA will not be added to the audit store.

Install Log Forwarder and configure according to previous sections. Log Forwarder configuration MinLogLevel must be at least info level.

1. Navigate to the log forwarder lambda function.

2. Select the Test tab.

3. Copy the json test event into the Event JSON pane.

   {
       "Records": [
           {
               "kinesis": {
                   "kinesisSchemaVersion": "1.0",
                   "partitionKey": "041e96d78c778677ce43f50076a8ae3e",
                   "sequenceNumber": "49620336010289430959432297775520367512250709822916263938",
                   "data": "eyJpbmdlc3RfdGltZV91dGMiOiIyMDI2LTAzLTI3VDEzOjIzOjEyLjkwNFoiLCJhZGRpdGlvbmFsX2luZm8iOnsiZGVzY3JpcHRpb24iOiJEYXRhIHVucHJvdGVjdCBvcGVyYXRpb24gd2FzIHN1Y2Nlc3NmdWwuIn0sImNsaWVudCI6e30sImNudCI6NCwiY29ycmVsYXRpb25pZCI6InJzLXF1ZXJ5LWlkOjEyMzQiLCJsZXZlbCI6IlNVQ0NFU1MiLCJsb2d0eXBlIjoiUHJvdGVjdGlvbiIsIm9yaWdpbiI6eyJob3N0bmFtZSI6IlBSTy1VUy1QRjNSSEdGOSIsImlwIjoiMTAuMTAuMTAuMTAiLCJ0aW1lX3V0YyI6MTcxMTU1OTEwMH0sInByb2Nlc3MiOnsiaWQiOjJ9LCJwcm90ZWN0aW9uIjp7ImF1ZGl0X2NvZGUiOjgsImRhdGFlbGVtZW50IjoiYWxwaGEiLCJkYXRhc3RvcmUiOiJTQU1QTEVfUE9MSUNZIiwibWFza19zZXR0aW5nIjoiIiwib2xkX2RhdGFlbGVtZW50IjpudWxsLCJvcGVyYXRpb24iOiJVbnByb3RlY3QiLCJwb2xpY3lfdXNlciI6IlBUWURFViJ9LCJwcm90ZWN0b3IiOnsiY29yZV92ZXJzaW9uIjoiMS4yLjErNTUuZzU5MGZlLkhFQUQiLCJmYW1pbHkiOiJjcCIsInBjY192ZXJzaW9uIjoiMy40LjAuMTQiLCJ2ZW5kb3IiOiJyZWRzaGlmdCIsInZlcnNpb24iOiIwLjAuMS1kZXYifSwic2lnbmF0dXJlIjp7ImNoZWNrc3VtIjoiN0VCMkEzN0FDNzQ5MDM1NjQwMzBBNUUxNENCMTA5QzE0OEJGODYwRjE3NEVCMjMxMTAyMEI3RkE1QTY4MjdGQyIsImtleV9pZCI6ImM0MjQ0MzZhLTExMmYtNGMzZi1iMmRiLTEwYmFhOGI1NjJhNyJ9fQ==",
                   "approximateArrivalTimestamp": 1626878559.213
               },
               "eventSource": "aws:kinesis",
               "eventVersion": "1.0",
               "eventID": "shardId-000000000000:49620336010289430959432297775520367512250709822916261234",
               "eventName": "aws:kinesis:record",
               "invokeIdentityArn": "arn:aws:iam::555555555555:role/service-role/TestRole",
               "awsRegion": "us-east-1",
               "eventSourceARN": "arn:aws:kinesis:us-east-1:555555555555:stream/CloudProtectEventStream"
           }
       ]
   }
   

   Note

   The data payload is the base64 encoded audit log. See Audit Logging for detail on audit log contents.

4. Select Test to execute the test event.

5. Test is successful if the Log Output of test results contains the following log:

   [INFO] [kinesis-log-aggregation-format.cpp:77] Aggregated 1 records into 0 aggregated, 1 forwarded and 0 failed records
   

   If the log is not present, please consult the Troubleshooting section for common errors and solutions.

Update Protector With Kinesis Log Stream

In this section, Kinesis log stream ARN will be provided to the Protect Function installation.

1. Navigate to the Protector CloudFormation stack created in the protector installation section.

2. Select Update.

3. Choose Use existing template > Next.

4. Set parameter KinesisLogStreamArn to the output value recorded in Install through CloudFormation.

5. Proceed with Next and Submit the changes.

6. Continue to the next section once stack status indicates UPDATE_COMPLETE.

Update Policy Agent With Log Forwarder Function Target

Log Forwarder Lambda function requires a policy layer which is in sync with the Protegrity Protector. This section will describe the steps to update the policy agent to include updating Log Forwarder Lambda function.

1. Navigate to the Policy Agent Function created in Policy Agent Installation

2. Select Configuration > Environment variables > Edit

3. Edit the value for environment variable AWS_PROTECT_FN_NAME to include the log forwarder function name/arn in the comma separated list of Lambda functions.

4. Save the changes and continue when update completes

5. Navigate to Test tab

6. Add an event {} and select Test to run the Policy Agent function

7. Verify Log forwarder function was updated to use the policy layer by inspecting the log output. Logs should include the following:

   
   [INFO] 2024-07-09 18:58:04,793.793Z 622d374b-1f73-4123-9a38-abc61973adef iap_agent.policy_deployer:Updating lambda [Protegrity_LogForwarder_<stack ID>] to use layer version [arn:aws:lambda:<aws region>:<aws account number>:layer:Protegrity_Layer_<layer name>:<layer version>]
   

Test Full Log Forwarder Installation

Install and configure Protegrity Agent, Protector, and Log Forwarder components.

1. Send a protect operation to the protector using a data element or user which will result in audit log generation

2. Navigate to the CloudWatch log group for the Protect function

3. Select the log stream for the test operation and scroll to the latest logs

4. Expect to see a log similar to the below:

   
   [2024-07-09T19:28:23.158] [INFO] [kinesis-external-sink.cpp:51] Sending 2 logs to Kinesis ...
   [2024-07-09T19:28:23.218] [INFO] [aws-utils.cpp:206] Kinesis send time: 0.060s
   

5. Navigate to the CloudWatch log group for the Log Forwarder function

6. Expect to see a new log stream - it may take several minutes for the stream to start

7. Select the new stream and scroll to the most recent logs in the stream

8. Expect to see a log similar to the below:

   
   [2024-07-09T19:32:31.648] [INFO] [kinesis-log-aggregation-format.cpp:77] Aggregated 1 records into 0 aggregated, 1 forwarded and 0 failed records
   

Troubleshooting

[SEVERE] User: <arn> is not authorized to perform: secretsmanager:
GetSecretValue on resource: <secret name> because no identity-based
policy allows the secretsmanager:GetSecretValue action

{
  "error_msg": "Failed to decrypt the policy. Please verify
    that the function has access to the key service and the key.",
  "success": false
}

The provided execution role does not have permissions to call
[CreateNetworkInterface|DescribeNetworkInterfaces|DeleteNetworkInterface]
on EC2 (Service: Lambda, Status Code: 400, Request ID: <request id>)"
(RequestToken: <request token>, HandlerErrorCode: InvalidRequest)

[SEVERE] Kinesis stream client returned 400 error with error message:
User: <function arn> is not authorized to perform: kinesis:PutRecords
on resource: <kinesis stream arn> because no identity-based policy
allows the kinesis:PutRecords action

[error] [tls] <error message>

3.6 -

Prerequisites

3.7 -

AWS Services

The following table describes the AWS services that may be a part of your Protegrity installation.

3.8 -

Required Skills and Abilities

4 - Understanding Redshift Objects

External Functions

Redshift provides an External Function capability that is used to call out to a process external to Redshift. In this solution, the external service is Protegrity Redshift Protector, an AWS Lambda for re-identification operations.

Function Naming Convention

The request payload header indicates the current user context making the Protegrity operation through an SQL request. Protegrity also requires the type of operation and the security policy element name. Protegrity Serverless provides a UDF function naming convention to provide this additional context.

The function name convention requires the prefix pty, the type of operation (protect or unprotect), and the ESA policy element name. The three tokens are separated by underscores. Additional underscores are interpreted as part of the element name. (e.g. pty_protect_tok_deSSN).

The UDF naming convention is as follows.

For example, the following UDF will perform an unprotect using the alpha element policy.

CREATE OR REPLACE EXTERNAL FUNCTION pty_unprotect_alpha(varchar)
RETURNS varchar VOLATILE lambda '<replace_with_protect_function_name>:Production' iam_role 'arn:aws:iam::<you-aws-account-number>:role/<role-name>'

Mapping File

Protegrity Serverless provides an additional method for mapping UDF function names to operations and security policy elements through a JSON mapping file. This method is recommended when either custom naming conventions are needed or element names do not conform to Redshift’s function naming validation rules. Here is an example.

The mapping file must be provided in the same S3 bucket as policy export: AWS_POLICY_S3_BUCKET

{
  "myudf_unp_city":
    {
      "Operation": "unprotect",
      "Element": "deCity”
    },
  "myudf_pro_dob": {
      "Operation": "protect",
      "Element": "deBirthdate"
    },
    ...
}

The example mapping above would cause Protegrity Serverless to perform an unprotect on the deCity security element for the requests made from the myudf_unp_city UDF function within Redshift.

4.1 -

External Functions

Redshift provides an External Function capability that is used to call out to a process external to Redshift. In this solution, the external service is Protegrity Redshift Protector, an AWS Lambda for re-identification operations.

4.2 -

Function Naming Convention

The request payload header indicates the current user context making the Protegrity operation through an SQL request. Protegrity also requires the type of operation and the security policy element name. Protegrity Serverless provides a UDF function naming convention to provide this additional context.

The function name convention requires the prefix pty, the type of operation (protect or unprotect), and the ESA policy element name. The three tokens are separated by underscores. Additional underscores are interpreted as part of the element name. (e.g. pty_protect_tok_deSSN).

The UDF naming convention is as follows.

For example, the following UDF will perform an unprotect using the alpha element policy.

CREATE OR REPLACE EXTERNAL FUNCTION pty_unprotect_alpha(varchar)
RETURNS varchar VOLATILE lambda '<replace_with_protect_function_name>:Production' iam_role 'arn:aws:iam::<you-aws-account-number>:role/<role-name>'

4.3 -

Mapping File

Protegrity Serverless provides an additional method for mapping UDF function names to operations and security policy elements through a JSON mapping file. This method is recommended when either custom naming conventions are needed or element names do not conform to Redshift’s function naming validation rules. Here is an example.

The mapping file must be provided in the same S3 bucket as policy export: AWS_POLICY_S3_BUCKET

{
  "myudf_unp_city":
    {
      "Operation": "unprotect",
      "Element": "deCity”
    },
  "myudf_pro_dob": {
      "Operation": "protect",
      "Element": "deBirthdate"
    },
    ...
}

The example mapping above would cause Protegrity Serverless to perform an unprotect on the deCity security element for the requests made from the myudf_unp_city UDF function within Redshift.

5 - Performance

Performance Considerations

The following factors may cause variation in real performance versus benchmarks:

• Cold startup: The Lambda spends additional time on the initial invocation to decrypt and load the policy into memory. This time can vary between 400 ms and 1200 ms depending on the policy size. Once the Lambda is initialized, subsequent “warm executions” should process quickly.
• Size of policy: The size of the policy impacts cold start performance. Larger policies take more time to initialize.
• Lambda memory: AWS provides more virtual cores based on the memory configuration. The initial configuration of 1728 MB provides a good tradeoff between performance and cost with the benchmarked policy. Memory can be increased to optimize for your individual cases.
• Cluster size: Cluster size may make a significant difference depending on the workload.
• Number of security operations (protect or unprotect).
• Lambda concurrency and burst quotas: AWS limits the number of concurrent executions and how quickly lambda can scale to meet demand. This is discussed in an upcoming section of the document.
• Size of data element: Operations on larger text consume time.

Sample Benchmarks

The following benchmarks were performed against different cluster sizes. These are median times of approximately five runs each. The query unprotected six columns per row (first_name, last_name, email, postal_code, ssn, iban):

Lambda Tuning

AWS maintains quotas for Lambda concurrent execution. Two of these quotas impact concurrency and compete with other Lambdas in the same account and region:

The Concurrent executions quota cap is the maximum number of Lambda instances that can serve requests for an account and region. The default AWS quota may be inadequate based on peak concurrency based on the table in the previous section. This quota can be increased with an AWS support ticket.

The Burst concurrency quota limits the rate at which Lambda will scale to accommodate demand. This quota is also per account and region. The burst quota cannot be adjusted. AWS will quickly scale until the burst limit is reached. After the burst limit is reached, functions will scale at a reduced rate per minute (e.g. 500). If no Lambda instances can serve a request, the request will fail with a 429 Too Many Requests response. Redshift will generally retry until all requests succeed but may abort if a high percentage of failed responses occur.

The burst limit is a fixed value and varies significantly by AWS region. The highest burst (3,000) is currently available in the following regions: US West (Oregon), US East (N.Virginia), and Europe (Ireland). Other regions can burst between 500 and 1,000. It is recommended to select a Redshift AWS region with the highest burst limits.

Concurrency Troubleshooting

Hitting up against quota limits may indicate that quota adjustments are required. Exceeding quota limits may cause a Redshift query to fail or reduce performance. In the worst case, significant throttling can impact the performance of all your Lambda services in the region.

Redshift is tolerant of a certain ratio of failed requests and automatically retries. If a high percentage of requests fail, then the query may abort and the last error code will display in the console. For example, 429 Too Many Requests.

CloudWatch Metrics can be manually enabled on the Lambda to reveal if quotas are being reached. Metrics aggregate errors into two buckets that are 4xx. CloudWatch logs can be used to access the actual error code.

Log Forwarder Performance

Log forwarder architecture is optimized to minimize the amount of connections and reduce the overall network bandwidth required to send audit logs to ESA. This is achieved with batching and aggregation taking place on two levels. The first level is in protect function instances, where audit logs from consecutive requests to an instance are batched and aggregated. The second level of batching takes place in Amazon Kinesis Stream where log records from different protect function instances are additionally batched and sent to log forwarder function where they are aggregated. This section shows how to configure the deployment to accommodate different patterns of anticipated audit log stream. It also shows how to monitor deployment resources to detect problems before audit records are lost.

• Protector Cloud Formation Parameters

  • AuditLogFlushInterval: Determines the minimum amount of time required for the audit log to be sent to Amazon Kinesis. Changing flush interval may affect the level of aggregation, which in turn may result in different number of connections and different data rates to Amazon Kinesis. Default value is 30 seconds.

    Increasing the flush interval may result in higher aggregation of audit logs, in fewer connections to Amazon Kinesis, in higher latency of audit logs arriving to ESA and in higher data throughput.

    Lowering the flush interval may result in lower aggregation of audit logs, in more connections to Amazon Kinesis, in lower latency of audit logs arriving to ESA and in lower data throughput.

    It is not recommended to reduce the flush interval from default value in production environment as it may overload the Amazon Kinesis service. However, it may be beneficial to reduce flush interval during testing to make audit records appear on ESA faster.

      

• Log Forwarder Cloud Formation Parameters

  • Amazon KinesisLogStreamShardCount: The number of shards represents the level of parallel streams in the Amazon Kinesis and it is proportional to the throughput capacity of the stream. If the number of shards is too low and the volume of audit logs is too high, Amazon Kinesis service may be overloaded and some audit records sent from protect function may be lost.

    Default value is 10, however you are advised to test with a production-like load to determine whether this is sufficient or not.

  • Amazon KinesisLogStreamRetentionPeriodHours: The time for the audit records to be retained in Amazon Kinesis log stream in cases where log forwarder function is unable to read records from the Kinesis stream or send records to ESA, for example due to a connectivity outage. Amazon Kinesis will retain failed audit records and retry periodically until connectivity with ESA is restored or retention period expires.

    Default value is 24 hours, however you are advised to review this value to align it with your Recovery Time Objective and Recovery Point Objective SLAs.

      

• Monitoring Log Forwarder Resources

  • Amazon Kinesis Stream Metrics: Any positive value in Amazon Kinesis PutRecords throttled records metric indicates that audit logs rate from protect function is too high. The recommended action is to increase the Amazon KinesisLogStreamShardCount or optionally increase the AuditLogFlushInterval.

  • Log Forwarder Function CloudWatch Logs: If log forwarder function is unable to send logs to ESA, it will log the following message:

    [SEVERE] Dropped records: x.
    

    Note

    When the error message above occurs, the dropped audit records will be preserved in the Amazon Kinesis data stream and retried again according to Amazon Kinesis retry schedule. Records will be retried until Amazon KinesisLogStreamRetentionPeriodHours expires.

  • Protect Function CloudWatch Logs: If protect function is unable to send logs to Amazon Kinesis, it will log the following message:

    [SEVERE] Amazon Kinesis error, retrying in x ms (retry: y/z) ..."
    

    Any dropped audit log records will be reported with the following log message:

    [SEVERE] Failed to send x/y audit logs to Amazon Kinesis.
    

5.1 -

Concurrency Troubleshooting

Hitting up against quota limits may indicate that quota adjustments are required. Exceeding quota limits may cause a Redshift query to fail or reduce performance. In the worst case, significant throttling can impact the performance of all your Lambda services in the region.

Redshift is tolerant of a certain ratio of failed requests and automatically retries. If a high percentage of requests fail, then the query may abort and the last error code will display in the console. For example, 429 Too Many Requests.

CloudWatch Metrics can be manually enabled on the Lambda to reveal if quotas are being reached. Metrics aggregate errors into two buckets that are 4xx. CloudWatch logs can be used to access the actual error code.

5.2 -

Lambda Tuning

AWS maintains quotas for Lambda concurrent execution. Two of these quotas impact concurrency and compete with other Lambdas in the same account and region:

The Concurrent executions quota cap is the maximum number of Lambda instances that can serve requests for an account and region. The default AWS quota may be inadequate based on peak concurrency based on the table in the previous section. This quota can be increased with an AWS support ticket.

The Burst concurrency quota limits the rate at which Lambda will scale to accommodate demand. This quota is also per account and region. The burst quota cannot be adjusted. AWS will quickly scale until the burst limit is reached. After the burst limit is reached, functions will scale at a reduced rate per minute (e.g. 500). If no Lambda instances can serve a request, the request will fail with a 429 Too Many Requests response. Redshift will generally retry until all requests succeed but may abort if a high percentage of failed responses occur.

The burst limit is a fixed value and varies significantly by AWS region. The highest burst (3,000) is currently available in the following regions: US West (Oregon), US East (N.Virginia), and Europe (Ireland). Other regions can burst between 500 and 1,000. It is recommended to select a Redshift AWS region with the highest burst limits.

5.3 -

Sample Benchmarks

The following benchmarks were performed against different cluster sizes. These are median times of approximately five runs each. The query unprotected six columns per row (first_name, last_name, email, postal_code, ssn, iban):

5.4 -

Performance Considerations

The following factors may cause variation in real performance versus benchmarks:

• Cold startup: The Lambda spends additional time on the initial invocation to decrypt and load the policy into memory. This time can vary between 400 ms and 1200 ms depending on the policy size. Once the Lambda is initialized, subsequent “warm executions” should process quickly.
• Size of policy: The size of the policy impacts cold start performance. Larger policies take more time to initialize.
• Lambda memory: AWS provides more virtual cores based on the memory configuration. The initial configuration of 1728 MB provides a good tradeoff between performance and cost with the benchmarked policy. Memory can be increased to optimize for your individual cases.
• Cluster size: Cluster size may make a significant difference depending on the workload.
• Number of security operations (protect or unprotect).
• Lambda concurrency and burst quotas: AWS limits the number of concurrent executions and how quickly lambda can scale to meet demand. This is discussed in an upcoming section of the document.
• Size of data element: Operations on larger text consume time.

6 - Audit Logging

Audit Logging

Audit records and application logs stream to Amazon CloudWatch Logs or optionally be sent to ESA. Cloud Protect uses a JSON format for audit records that is described in the following sections.

You can analyze and alert on audit records using Protegrity ESA or Amazon CloudWatch. Third-party solutions may be used if they are supported by Amazon Cloudwatch or AWS Lambda logging extensions. For more information about forwarding your audit records to ESA, contact Protegrity. For more information about Amazon CloudWatch, refer to the Amazon CloudWatch User Guide.

For more information about audit records, refer to the Protegrity Analytics Guide.

Audit record fields

The audit record format has been altered in version 3.1 of the protector to provide more information.

Example Audit Records

The following are sample audit messages:

Protect Success:

{
      "additional_info": {
        "deployment_id": "Protegrity-Protect-function-deployment-id",
        "description": "Data protect operation was successful.",
        "query_id": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
        "request_id": "8476a536-e9f4-11e8-9739-2dfe598c3fcd"
      },
      "cnt": 4000,
      "correlationid": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
      "logtype": "Protection",
      "level": "SUCESS",
      "origin": {
        "hostname": "localhost",
        "ip": "127.0.0.1",
        "time_utc": 1635363966
      },
      "protection": {
        "dataelement": "deAddress",
        "operation": "Protect",
        "audit_code": 6,
        "datastore": "SAMPLE_POLICY",
        "policy_user": "test_user"
      },
      process":{
        "name":"protect",
        "id":"13",
        "module":"coreprovider",
        "thread_id":"573580544",
        "user":"sbx_user1051",
        "platform":"\"Linux_x64\"",
        "version":"UNKNOWN"
      },
      "client": {
        "ip":"169.254.62.117"
      },
      "protector": {
        "family": "cp",
        "version": "4.0.0.102",
        "vendor": "aws.snowflake",
        "datastore":"SAMPLE_POLICY",
        "pcc_version": "4.0.0.9",
        "core_version": "2.1.4+0.g93016.2.1",
        "lambda_version":"4.0.1"
      },
      "signature": {
        "key_id": "95f5a194-b0a4-4351-a",
        "checksum": "B324AF7C56944D91C47847A77C0367C594C0B948E7E75654B889571BD4F60A71"
      }
    }

User permission denied:

{
      "additional_info": {
        "deployment_id": "Protegrity-Protect-function-deployment-id",
        "description": "The user does not have the appropriate permissions to perform the requested operation.",
        "query_id": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
        "request_id": "8476a536-e9f4-11e8-9739-2dfe598c3fcd"
      },
      "cnt": 4000,
      "correlationid": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
      "logtype": "Protection",
      "level": "ERROR",
      "origin": {
        "hostname": "localhost",
        "ip": "127.0.0.1",
        "time_utc": 1635363966
      },
      "protection": {
        "dataelement": "deAddress",
        "operation": "Protect",
        "audit_code": 3,
        "datastore": "SAMPLE_POLICY",
        "policy_user": "test_user"
      },
      process":{
        "name":"protect",
        "id":"13",
        "module":"coreprovider",
        "thread_id":"573580544",
        "user":"sbx_user1051",
        "platform":"\"Linux_x64\"",
        "version":"UNKNOWN"
      },
      "client": {
        "ip":"169.254.62.117"
      },
      "protector": {
        "family": "cp",
        "version": "4.0.0.102",
        "vendor": "aws.snowflake",
        "datastore":"SAMPLE_POLICY",
        "pcc_version": "4.0.0.9",
        "core_version": "2.1.4+0.g93016.2.1",
        "lambda_version":"4.0.1"
      },
      "signature": {
        "key_id": "95f5a194-b0a4-4351-a",
        "checksum": "A216797C56944D91C47847A77C0367C594C0B948E7E75654B889571BD4F60A71"
      }
    }

Data element not found:

{
      "additional_info": {
        "deployment_id": "Protegrity-Protect-function-deployment-id",
        "description": "The data element could not be found in the policy.",
        "query_id": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
        "request_id": "8476a536-e9f4-11e8-9739-2dfe598c3fcd"
      },
      "cnt": 4000,
      "correlationid": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
      "logtype": "Protection",
      "level": "ERROR",
      "origin": {
        "hostname": "localhost",
        "ip": "127.0.0.1",
        "time_utc": 1635363966
      },
      "protection": {
        "dataelement": "deAddress",
        "operation": "Protect",
        "audit_code": 2,
        "datastore": "SAMPLE_POLICY",
        "policy_user": "test_user"
      },
      process":{
        "name":"protect",
        "id":"13",
        "module":"coreprovider",
        "thread_id":"573580544",
        "user":"sbx_user1051",
        "platform":"\"Linux_x64\"",
        "version":"UNKNOWN"
      },
      "client": {
        "ip":"169.254.62.117"
      },
      "protector": {
        "family": "cp",
        "version": "4.0.0.102",
        "vendor": "aws.snowflake",
        "datastore":"SAMPLE_POLICY",
        "pcc_version": "4.0.0.9",
        "core_version": "2.1.4+0.g93016.2.1",
        "lambda_version":"4.0.1"
      },
      "signature": {
        "key_id": "95f5a194-b0a4-4351-a",
        "checksum": "AF09217C56944D91C47847A77C0367C594C0B948E7E75654B889571BD4F60A71"
      }
    }

7 - No Access Behavior

No Access Behavior

The security policy maintains a No Access Operation, configured in an ESA, which determines the response for unauthorized unprotect requests.

The following table describes the result returned in the response for the various no access unprotect permissions.

8 - Known Limitations

Known Limitations

Known product limitations:

• FPE is supported only for ASCII values.
• Only the protect and unprotect operations are supported. The reprotect operation is not currently supported.
• The BIGINT data types are not protected/unprotected accurately. Use String instead.
• The FLOAT8 (8-byte float) data type is not supported. Use String instead.
• The timestamp data type is not supported.

9 - Upgrading To The Latest Version

Download the Latest Version

You can download the latest version of the deployment package from https://my.protegrity.com. Navigate to Data Protection > Cloud Protect to download the latest version.

After downloading the deployment package from the Protegrity Portal, unzip the package to extract the artifact files. In the AWS Console, navigate to the S3 bucket that was previously created to upload deployment artifacts (see: Create S3 bucket for Installing Artifacts).

Upload the following artifacts to the S3 bucket:

• Snowflake
• Redshift
• Cloud API
• Athena

If the release version matches your existing deployment, you don’t need to upload it again. Save the following artifacts on your local system so that you have them available during the next steps:

• Snowflake
• Redshift
• Cloud API
• Athena

Perform the following steps to upgrade the Agent Lambda and Protect Lambda separately.

Disable Protegrity Agent Function CloudWatch Event Rule

Cloud Watch Event Rule is used to periodically run Protegrity Agent Function to synchronize policy from ESA. This functionality is optional when deploying Protegrity Serverless Solution. If the Event Rule is enabled, it must be disabled temporarily for the time of the upgrade process.

Follow the steps below to determine if your deployment uses Event Rule and disable it.

1. Go to AWS Cloud Formation and select existing Protegrity deployment stack.

2. Select Resources tab from the top portion of the screen.

3. Check if there is a resource with ScheduledRule LogicalID. If there is no such resource you can skip to Upgrading Policy Agent Lambda section. If the scheduled rule is there, continue with the next steps in this section.

4. Click on the Physical ID link in the ScheduledRule row. The link opens Policy Agent Event Rule configuration.

5. Select Disable from the top-right portion of the screen. This will disable the rule. You will re-enable it after the upgrade process is complete.

Upgrading Policy Agent Lambda

1. Go to AWS Lambda console and select existing Protegrity Agent Lambda.

2. Click Actions in top right portion of the screen. Select Publish new version. Click Publish. The version of Agent Lambda you just created will serve as restore point in the case you needed to rollback the upgrade.

3. Go to Lambda Configuration > Environment variables.

4. Record environment variables values. You will use them later to configure upgraded Lambda Function. You can use the aws cli command below to save the function variables into the local json file:

   aws lambda get-function-configuration --function-name \
   arn:aws:lambda:<aws_region>:<aws_account>:function:<function_name> \
   --query Environment > <function_name>_env_config.json
   

5. Go to AWS Cloud Formation and select existing Protegrity Agent deployment stack.

6. Select Update. Check Replace current template > Upload a template file.

7. Upload pty_agent_cf.json file and select Next.

8. Click Next until Review window and then select Update stack.

9. Wait for the Cloud Formation to complete.

10. Navigate back to Agent Lambda Function.

11. Go to Configuration > Environment variables. Replace placeholder values with values recorded in previous step. Alternatively, you can run the following aws cli command to update function configuration using json file saved in the previous steps:

    aws lambda update-function-configuration --function-name \
    arn:aws:lambda:<aws_region>:<aws_account>:function:<function_name> \
    --environment file://./<function_name>_env_config.json
    

    Note

    If your current agent installation version is lower than 3.0.12, make sure you set the following function configuration variables:

    • PTY_ADDIPADDRESSHEADER
    • PTY_ESA_CA_SERVER_CERT You can read more about these variables in section Policy Agent Lambda Configuration.

12. If you are upgrading from versions prior to v3.0, backup and remove existing policy from the bucket defined by AWS_POLICY_S3_BUCKET property, so that the policy can be re-downloaded and re-encrypted with new ‘key commitment’ feature.

13. If you are upgrading from version prior to 1.6.1 please follow the steps below, otherwise the upgrade process is completed.

14. From AWS Console, navigate to IAM > Policies

15. Search for the Agent Lambda IAM Policy created in Create Agent Lambda IAM policy

16. Click on the policy, then select Edit Policy. Select JSON tab.

17. Add the following statement to the list of policy statements.

    
    {
      "Sid": "LambdaGetConfiguration",
      "Effect": "Allow",
      "Action": [
          "lambda:GetFunctionConfiguration"
      ],
      "Resource": [
          "arn:aws:lambda:*:*:function:*"
      ]
    }
    

18. Click Review Policy, then Save Changes. Wait for the changes to save.

Upgrading Log Forwarder Lambda

• Publish Log Forwarder Lambda Version

  Publishing a version of the Log Forwarder Lambda allows to roll-back to pre-existing version if upgrade fails

  1. Go to AWS Lambda console and select existing Protegrity Log Forwarder Lambda.

  2. Click Actions in top right portion of the screen. Select Publish new version. Click Publish.

  3. Record the Lambda version number. It will be displayed at the top of the screen. You can also retrieve it from the Lambda function view, under Versions tab.

     Log Forwarder Lambda version number for roll-backs: ___________________

• Disable Kinesis Trigger

  Disabling Kinesis trigger ensures there are no unprocessed or re-processed events while function is upgraded.

  1. Go to AWS Lambda console and select existing Protegrity Log Forwarder Lambda.
  2. Select Configuration tab > Triggers
  3. Check Kinesis trigger and click Edit button
  4. Uncheck Activate trigger and click Save
  5. Wait for function to stop processing events by monitoring function in Monitor tab

• Upgrade Forwarder Lambda Version

  Upgrade Log Forwarder function with new code

  1. Go to AWS Cloud Formation and select existing Protegrity Log Forwarder deployment stack.
  2. Select Update Stack > Make a direct update.
  3. Select Replace existing template > Upload a template file.
  4. Upload pty_log_forwarder_cf file and select Next.
  5. Click Next until Review window and then select Update stack.
  6. Wait for the Cloud Formation to complete.

• Enable Kinesis Trigger

  1. Go to AWS Lambda console and select existing Protegrity Log Forwarder Lambda.
  2. Select Configuration tab > Triggers
  3. Check Kinesis trigger and click Edit button
  4. Check Activate trigger and click Save Log Forwarder function will now start processing events from where it left off when Kinesis trigger was disabled.

• Monitor and roll-back

  Monitor Log Forwarder function for errors in its CloudWatch logs and in Montior tab. To roll back function to the previous version if any errors occur follow these steps:

  1. Go to AWS Lambda console and select existing Protegrity Log Forwarder Lambda.

  2. Select Configuration tab > Triggers

  3. Expand Details section of Kinesis trigger and record UUID value

  4. Execute the following AWS CLI command to move Kinesis trigger to previous version of Log Forwarder Lambda that was created earlier and recorded as Log Forwarder Lambda version number for roll-backs. Substitute kinesis-mapping-uuid, log-forwarder-function-name, version-for-roll-backs with your values:

     
     aws lambda update-event-source-mapping --uuid <kinesis-mapping-uuid> --function-name <log-forwarder-function-name>:<version-for-roll-backs>
     

  5. Find Kinesis trigger attached to previous version of Log Forwarder Lambda by navigating Versions tab > Version number link in the Versions column Kinesis trigger is now moved to previous version of Log Forwarder Lambda function.

Upgrading Protect Lambda

Diagram below illustrates upgrade steps.

• Snowflake
• Redshift
• Cloud API
• Athena

• Publish Protect Lambda Version

  Publishing a version of the Protect Lambda allows updating it without interruptions to the existing traffic.

  1. Go to AWS Lambda console and select existing Protegrity Protect Lambda.

  2. Go to Lambda Configuration > Environment variables.

  3. Record environment variables values. You will use them later to configure upgraded Lambda Function. You can use the aws cli command below to save the function variables into the local json file:

     aws lambda get-function-configuration --function-name \
     arn:aws:lambda:<aws_region>:<aws_account>:function:<function_name> \
     --query Environment > <function_name>_env_config.json
     

  4. Click Actions in top right portion of the screen. Select Publish new version. Click Publish.

  5. Record the Lambda version number. It will be displayed at the top of the screen. You can also retrieve it from the Lambda function view, under Versions tab.

     Protect Lambda version number: ___________________

  6. If you are upgrading a Cloud Protect Redshift version 1.x to 2.x/3x, you must recreate your Redshift external function definitions with Protect Lambda Function version appended to the Lambda Function name. See example below.

     CREATE OR REPLACE EXTERNAL FUNCTION PTY_PROTECT_TEST ( val varchar )
         RETURNS varchar
         VOLATILE lambda
         'Protegrity_Protect_test:<protect_lambda_version_number>' iam_role
         'arn:aws:iam::123456789212:role/example_role';
     

• Run protect service upgrade

  In this step, the Protect service including Lambda $LATEST version will be updated using Cloud Formation template. The Lambda version created in previous step will be used to serve existing traffic during the upgrade process.

  1. Go to AWS Cloud Formation and select existing Protegrity deployment stack.

  2. Select Update. Check Replace current template > Upload a template file.

  3. Upload pty_protect_cf.json file and select Next.

  4. Update ProtectFunctionProductionVersion parameter with Protect Lambda version number recorded in step 3.

  5. Note

     If you are upgrading protector to one of these versions: [3.2.2, 3.2.3], set parameter KinesisLogStreamArn to the output value recorded in Install through CloudFormation for the newly deployed log forwarder.

  6. Click Next until Review window and then select Update stack.

  7. Wait for the Cloud Formation to complete.

  8. Go back to Lambda console and select Protect Lambda.

  9. Go to Configuration > Environment variables. Replace placeholder values with values recorded in previous step. Alternatively, you can run the following aws cli command to update function configuration using json file saved in the previous steps:

     aws lambda update-function-configuration --function-name \
     arn:aws:lambda:<aws_region>:<aws_account>:function:<function_name> \
     --environment file://./<function_name>_env_config.json
     

     Note

     If your current protect installation version is lower than 3.0.14, you can optionally set the following variable:

     • LOG_REDSHIFT_CLUSTER_ARN You can read more about this configuration in section Protect Lambda Configuration.

  10. Navigate to Aliases tab. Verify that Production alias points to the lambda version you specified in the cloud formation template.

  11. The upgraded Protect Lambda is configured with a sample policy. Run Agent Lambda Function before continuing with next steps.

• Finalize upgrade

  In this step, the Protect Lambda will be configured to serve traffic using $LATEST version upgraded in the previous step.

  1. Go back to Protegrity AWS Cloud Formation deployment stack.

  2. Select Update. Check Use Current template.

  3. Update ProtectFunctionProductionVersion parameter with the following value: $LATEST.

  4. Click Next until Review window and then select Update stack.

  5. Go back to Lambda console and select Protect Lambda.

  6. From the Lambda console, verify that Latest alias points to $LATEST version.

  7. Test your function to make sure it works as expected.

  8. If you are upgrading a Cloud Protect Redshift version 1.x to 2.x/3x, you must recreate your Redshift external function definitions with Protect Lambda Function version appended to the Lambda Function name. See example below.

     CREATE OR REPLACE EXTERNAL FUNCTION PTY_PROTECT_TEST ( val varchar )
         RETURNS varchar
         VOLATILE lambda
         'Protegrity_Protect_test:Production' iam_role
         'arn:aws:iam::123456789212:role/example_role';
     

  9. If you need to rollback to older version of Protect Lambda, you can re-run the cloud formation with ProtectFunctionProductionVersion parameter set to the previous version of Protect Lambda.

Re-enable Protegrity Agent Function CloudWatch Event Rule

If the Event Rule was disabled at the beginning of the upgrade process, you must re-enabled it. Follow the steps below to re-enable Policy Agent Event rule.

1. Go to the Protegrity Agent Cloud Formation Stack.

2. Select Resources tab from the top portion of the screen.

3. Click on the Physical ID link in the ScheduledRule row. The link opens Policy Agent Event Rule configuration.

4. Select Enable from the top-right portion of the screen. This will enable the rule. You will re-enable it after the upgrade process is complete.

10 - Appendices

10.1 - Installing the Policy Agent and Protector in Different AWS Accounts

The Policy Agent Lambda function and Protect Lambda functions can be installed in separate AWS accounts. However, additional configuration is required to authorize the Policy Agent to provision the security policy to a remote Protect Lambda function.

Create Agent Lambda IAM policy

1. Login to the AWS account that hosts the Protect Lambda function.

2. From the AWS IAM console, select Policies > Create Policy.

3. Select the JSON tab and copy the following snippet.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "LambdaUpdateFunction",
         "Effect": "Allow",
         "Action": [
           "lambda:UpdateFunctionConfiguration"
         ],
         "Resource": [
           "arn:aws:lambda:*:*:function:*"
         ]
       },
       {
         "Sid": "LambdaReadLayerVersion",
         "Effect": "Allow",
         "Action": [
           "lambda:GetLayerVersion",
           "lambda:ListLayerVersions"
         ],
         "Resource": "*"
       },
       {
         "Sid": "LambdaDeleteLayerVersion",
         "Effect": "Allow",
         "Action": "lambda:DeleteLayerVersion",
         "Resource": "arn:aws:lambda:*:*:layer:*:*"
       },
       {
         "Sid": "LambdaPublishLayerVersion",
         "Effect": "Allow",
         "Action": "lambda:PublishLayerVersion",
         "Resource": "arn:aws:lambda:*:*:layer:*"
       },
       {
         "Sid": "S3GetObject",
         "Effect": "Allow",
         "Action": [
           "s3:GetObject"
         ],
         "Resource": "arn:aws:s3:::*/*"
       },
       {
         "Sid": "S3PutObject",
         "Effect": "Allow",
         "Action": [
           "s3:PutObject"
         ],
         "Resource": "arn:aws:s3:::*/*"
       },
       {
         "Sid": "LambdaGetConfiguration",
         "Effect": "Allow",
         "Action": [
             "lambda:GetFunctionConfiguration"
         ],
         "Resource": [
             "arn:aws:lambda:*:*:function:*"
         ]
       }
     ]
   }
   

4. Replace the wildcards (*) with the region, account, and resource name information where required.

5. Select Review policy, type in the policy name, and confirm. Record policy name:

   Agent Lambda Cross Account Policy Name: ___________________

Create Policy Agent cross-account IAM Role

1. Login to the AWS account that hosts the Protect Lambda function.

2. From the AWS IAM console, select Roles > Create Role

3. Select AWS Service > Lambda . Proceed to Permissions.

4. Select Policy created in the step above. Proceed to Tags.

5. Specify Tag, proceed to the final screen. Type in policy name and confirm. Record the name.

   Policy Agent Cross Account IAM Role Name: ___________________

Allow the Policy Agent Cross-Account Role to be Assumed by the Policy Agent IAM Role

1. Login to the AWS account that hosts the Protect Lambda function.

2. Navigate to the previously created IAM Role (Agent Lambda Cross-Account IAM Role Name).

3. Navigate to Trust Relationships > Edit Trust Relationships.

4. Modify the Policy Document, replacing the placeholder value indicated in the following snippet as <Agent Lambda IAM Execution Role ARN> with ARN of Agent Lambda IAM Role that was created in Agent Installation.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
   
      "Principal": {
   
               "AWS": "<Agent Lambda IAM Execution Role Name>"
   
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

5. Click Update Trust Policy.

Add Assume Role to the Policy Agent Execution IAM Role

1. Login to the AWS account that hosts the Policy Agent.

2. Navigate to the Agent Lambda IAM Execution Role that was created in Agent Installation.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
   
      "Principal": {
   
               "AWS": "<Agent Lambda IAM Execution Role Name>"
   
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

3. Add Inline Policy.

4. Modify the Policy Document, replacing the placeholder value indicated in the following snippet as <Agent Lambda Cross-Account IAM ARN> with the value recorded in Create Policy Agent cross-account IAM Role.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "sts:AssumeRole"
         ],
         "Resource": "<Agent Lambda Cross-Account IAM  ARN>."
       }
     ]
   }
   

5. When you are finished, choose Review Policy.

6. On the Review policy page, type a Name, then choose Create Policy.

Update the Policy Agent Lambda Configuration

1. From the AWS console, navigate to Lambda, and select the Policy Agent Lambda function.

2. Select Configuration tab | Environment variables.

3. Select Edit and add the following environment variables with the value from Agent Lambda Cross-Account IAM ARN:

   Parameter	Value
   AWS_ASSUME_ROLE	Agent Lambda Cross-Account IAM ARN

4. Ensure the values in the Parameters AWS_POLICY_S3_BUCKET, AWS_PROTECT_FN_NAME and AWS_POLICY_LAYER_NAME are all in the Protect Lambda Function AWS Account.

5. In case custom VPC hostname configuration is used, you will need to set the ENDPOINT_URL. Refer to Policy Agent - Custom VPC Endpoint Hostname Configuration.

   AWS_VPC_ENDPOINT_URL

   <AWS_VPC_ENDPOINT>

6. Click Save and Run the Lambda. The Lambda will now assume the Role in Protect Lambda Function AWS Account and update the policy cross accounts.

10.2 - Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)

This guide describes how to configure the Protegrity Policy Agent and Log Forwarder to connect to a Protegrity Provisioned Cluster (PPC), highlighting the differences from connecting to ESA.

Key Differences: PPC vs ESA

Prerequisites

• Access to PPC and required credentials.
• Tools: curl, kubectl installed.

Policy Agent Setup with PPC

Follow these instructions as a guide for understanding specific inputs for Policy Agent integrating with PPC:

1. Obtain the Datastore Key Fingerprint

   To retrieve the fingerprint for your Policy Agent:

   curl -k -H "Authorization: Bearer ${TOKEN}" -X POST https://${HOST}/pty/v2/pim/datastores/1/export/keys  -H "Content-Type: application/json" --data '{
     "algorithm": "RSA-OAEP-256",
     "description": "example-key-from-kms",
     "pem": "-----BEGIN PUBLIC KEY-----\nABC123... ...890XYZ\n-----END PUBLIC KEY-----"
   }'
   

   Sample Output:

   {"uid":"1","algorithm":"RSA-OAEP-256","fingerprint":"4c:46:d8:05:35:2e:eb:39:4d:39:8e:6f:28:c3:ab:d3:bc:9e:7a:cb:95:cb:b1:8e:b5:90:21:0f:d3:2c:0b:27","description":"example-key-from-kms"}
   

   Record the fingerprint value and configure it as the PTY_DATASTORE_KEY for the Policy Agent.

2. Retrieve the PPC CA Certificate

   To obtain the CA certificate from PPC:

   kubectl -n api-gateway get secret ingress-certificate-secret -o jsonpath='{.data.ca\.crt}' | base64 -d > CA.pem
   

   Use the CA.pem that was returned as described in Policy Agent Installation.

3. Configure the PPC Address

   Use the PPC address in place of the ESA IP address wherever required in your configuration.

Log Forwarder Setup with PPC

• The Log Forwarder will proceed without certificates and will print a warning if PtyEsaCaServerCert and PtyEsaClientCertificatesSecretId are not provided.
• No additional certificate or CA configuration is needed for PPC.

Troubleshooting

Protector Lambda fails with “AWS KMS Decrypt failed”

Symptom:

After a successful Policy Agent run and layer update, the Protector Lambda returns:

{
  "body": "{\"error_msg\":\"Failed to open decoder: rpdecode decrypt failure: dek callback failed: AWS KMS Decrypt failed: \",\"success\":false}",
  "isBase64Encoded": false,
  "statusCode": 400
}

The Protector Lambda logs show:

[SEVERE] [utils.cpp:185] AWS KMS Decrypt failed:

Cause:

The public key configured in the PPC/ESA datastore does not match the KMS key pair used by the Policy Agent. The policy package is encrypted with the public key stored in the datastore. If that key does not correspond to the KMS key pair whose private key is used for decryption, the Protector Lambda will fail to decrypt the policy.

Resolution:

1. Identify the KMS key pair used by the Policy Agent (the key ARN configured during pre-configuration).
2. Export the public key from that KMS key pair.
3. In PPC/ESA, ensure the datastore’s export key is configured with the public key from that same KMS key pair. See Obtain the Datastore Key Fingerprint above.
4. Re-run the Policy Agent to generate a new policy package encrypted with the correct key.
5. Test the Protector Lambda again.

Additional Notes

• For troubleshooting, consult the Protegrity Documentation.

10.3 - Policy Agent - Custom VPC Endpoint Hostname Configuration

The Policy Agent uses default endpoint hostnames to communicate with other AWS services (for example, secretsmanager.amazonaws.com). This configuration will only work in VPCs where Amazon-provided DNS is available (default VPC configuration with private DNS option enabled for the endpoint). If your VPC uses custom DNS, follow the instructions below to configure the Policy Agent Lambda to use custom endpoint hostnames.

Identify DNS Hostnames

To identify DNS hostnames:

1. From AWS console, select VPC > Endpoints.

2. Select Secrets Manager endpoint from the list of endpoints.

3. Under Details > DNS Names, note the private endpoint DNS names adding https:// at the beginning of the endpoint name.

   For example, https://vpce-1234-4pzomrye.kms.us-west-1.vpce.amazonaws.com

4. Note down DNS names for the KMS and Lambda endpoints:

   AWS_SECRETSMANAGER_ENDPOINT: https://_________________

   AWS_KMS_ENDPOINT: https://_________________

   AWS_LAMBDA_ENDPOINT: https://_________________

Update the Policy Agent Lambda configuration

To update policy agent lambda configuration:

1. From the AWS console, navigate to Lambda, and select the Policy Agent Lambda function.

2. Select the Configuration section and choose Environment variables.

3. Select Edit and add the following environment variables with the corresponding endpoint URLs recorded in steps 3-4:

   Parameters	Value
   AWS_SECRETSMANAGER_ENDPOINT_URL	<AWS_SECRETS_ENDPOINT>
   AWS_KMS_ENDPOINT_URL	<AWS KMS ENDPOINT>
   AWS_LAMBDA_ENDPOINT_URL	<AWS LAMBDA ENDPOINT>

4. Click Save and Run the Lambda. The Lambda will now use endpoints you have just configured.

10.4 - Redshift Cross-Account Configuration

Cross-Account Configuration

The following figure illustrates the Protegrity Redshift Integration architecture when Protegrity Solution is installed on separate from Amazon Redshift Cluster Account.

Redshift Account IAM Configuration

This step creates Redshift IAM role with permissions to assume role in separate account.

Create Redshift IAM policy:

1. Login to the AWS account that hosts the Amazon Redshift cluster.

2. In the AWS console, access Services > IAM and click Policies.

3. Click Create Policy.

4. Select the JSON tab and paste the following JSON snippet:

   { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRole" ], "Resource": "<DBRoleARN>" } ] }
   

5. Replace the resource value with DBRoleARN recorded in Create IAM Account Role

6. Click Review policy to continue.

7. Enter a name for the policy.

8. Click Create Policy.

9. Record the policy name.

   Redshift IAM Policy Name: ___________________

Create Redshift IAM Role

This step creates Redshift IAM role with permissions to assume role in separate account.

Create Redshift IAM Role:

1. Login to the AWS account that hosts the Amazon Redshift cluster.

2. In the AWS console, access Services > IAM and click Roles.

3. Click Create Role.

4. Select AWS Service as the trusted entity type, then select Redshift use case from the list of services.

5. Select Redshift – Customizable from the use case section.

6. Continue by clicking Next:Permissions.

7. Filter the list and search for the policy recorded in the step above (Redshift IAM Policy Name).

8. Click the Next:Tags button to continue to the next step.

9. Click the Next:Review button to continue.

10. Enter a role name, such as Redshift2ProtegrityRole.

11. Click Create Role.

12. Record role ARN:

    RedshiftIAMRoleARN: ____________________

Attach IAM Role to Redshift Cluster

This step creates Redshift IAM role with permissions to assume role in separate account.

Attach IAM role to the Redshift cluster:

1. Login to AWS Console.

2. Access Amazon Redshift and select your cluster.

3. Select Properties > Cluster Permissions > Manage IAM Roles.

4. Select RedshiftIAMRoleARN configured in the step above and click Associate IAM role.

5. Save the changes.

6. After saving the changes it may take couple of minutes until the cluster IAM role is fully configured.

7. You can check configuration status by navigating back to the cluster IAM role settings.

8. The status field next to the IAM role will show in-sync once the role is configured.

Defining Redshift External Functions

The external function for cross-account differs slightly from the reference material in this document. The function requires two roles to be specified. The following is an example of the modified function definition.

CREATE OR REPLACE EXTERNAL FUNCTION demo_schema.pty_unprotect_deName(varchar)
RETURNS varchar
VOLATILE
LAMBDA 'ProtectFunctionProductionAlias'
IAM_ROLE '<RedshiftIAMRoleARN>,<DBRoleARN>;

Replace <ProtectFunctionProductionAlias> with the value recorded in Install through CloudFormation Replace <RedshiftIAMRoleARN> with the value recorded in Create Redshift IAM Role Replace <DBRoleARN> with the value recorded in Create IAM Account Role

10.5 - Sample Redshift External Function

Sample Redshift External Function

CREATE EXTERNAL FUNCTION PTY_PROTECT_ALPHA ( val varchar )
RETURNS varchar
VOLATILE lambda
'<replace_with_protect_function_name>:Production' iam_role 'arn:aws:iam::<REPLACE_WITH_YOUR_AWS_ACCOUNT>:role/<REPLACE_WITH_IAM_ROLE_NAME>';

SELECT PTY_PROTECT_ALPHA ('Hello World');

SELECT PTY_UNPROTECT_ALPHA('rfDtw sLMJK');

CREATE OR REPLACE EXTERNAL FUNCTION PTY_PROTECT_NUMERIC ( val int )
RETURNS int
VOLATILE lambda
'<replace_with_protect_function_name>:Production' iam_role 'arn:aws:iam::<REPLACE_WITH_YOUR_AWS_ACCOUNT>:role/<REPLACE_WITH_IAM_ROLE_NAME>';

SELECT PTY_PROTECT_NUMERIC (2147483647);

SELECT PTY_UNPROTECT_NUMERIC(-12344556564);

10.6 - Configuring Regular Expression to Extract Policy Username

Configuring Regular Expression to Extract Policy Username

Cloud Protect Lambda Function exposes USERNAME_REGEX configuration to allow extraction of policy username from user in the request.

• USERNAME_REGEX Lambda Environment configuration

  The USERNAME_REGEX configuration can be used to extract policy username from user in the request. The following are allowed values for USERNAME_REGEX:

  • 1 - Default build-in regular expression is used:

    ^arn:aws:(?:iam|sts)::[0-9]{12}:(?:role|user|group|assumed\-role|federated\-user)\/([\w\/+=,.\-]{1,1024}|[\w\/+=,.\-@]{1,1024})(?:@[a-zA-Z0-9\-]{1,320}(?:\.\w+)+)?$
    

  • ^User regex$ - Custom regex with one capturing group. This group is used to extract the username. Examples below show different regular expression values and the resulting policy user.

^arn:aws:(?:iam|sts)::[0-9]{12}:((?:role|user|group|assumed-role|federated-user).*)$

10.7 - Associating ESA Data Store With Cloud Protect Agent

Associating ESA Data Store With Cloud Protect Agent

ESA controls which policy is deployed to protector using concept of data store. A data store may contain a list of IP addresses identifying servers allowed to pull the policy associated with that specific data store. Data store may also be defined as default data store, which allows any server to pull the policy, provided it does not belong to any other data stores. Node registration occurs when the policy server (in this case the policy agent) makes a policy request to ESA, where the agent’s IP address is identified by ESA.

Policy agent lambda source IP address used for node registration on ESA depends on ESA hubcontroller configuration ASSIGN_DATASTORE_USING_NODE_IP and the PTY_ADDIPADDRESSHEADER configuration exposed by the agent lambda.

The Lambda service uses multiple network interfaces, internal network interface with ephemeral IP range of 169.254.x.x and external network interface with IP range of the VPC subnet the Lambda is associated with. By default, when agent lambda is contacting ESA to register node for policy download, ESA uses agent Lambda VPC IP address. This default behavior is caused by the default ESA hubcontroller configuration ASSIGN_DATASTORE_USING_NODE_IP=false and agent default configuration PTY_ADDIPADDRESSHEADER=yes.

In some cases, when there is a proxy server between the ESA and agent lambda, the desirable ESA configuration is ASSIGN_DATASTORE_USING_NODE_IP=true. and PTY_ADDIPADDRESSHEADER=no which will cause the ESA to use proxy server IP address.

The table below shows how the hubcontroller and agent settings will affect node IP registration on ESA.