Cloud Protect

• 1: AWS
• 1.1: Cloud API
• 1.1.1: Overview
• 1.1.2: Architecture
• 1.1.3: Installation
• 1.1.3.1: Prerequisites
• 1.1.3.2: Pre-Configuration
• 1.1.3.3: Protect Service Installation
• 1.1.3.4: Policy Agent Installation
• 1.1.3.5: Audit Log Forwarder Installation
• 1.1.3.6:
• 1.1.3.7:
• 1.1.3.8:
• 1.1.3.9:
• 1.1.4: REST API
• 1.1.4.1: Authorization
• 1.1.4.2: HTTP Status Codes
• 1.1.4.3: Payload Encoding
• 1.1.4.4: TLS Mutual Authentication
• 1.1.4.5: v1 Specification
• 1.1.4.6: Legacy Specification
• 1.1.5: Performance
• 1.1.6: Audit Logging
• 1.1.7: No Access Behavior
• 1.1.8: Known Limitations
• 1.1.9: Upgrading To The Latest Version
• 1.1.10: Appendices
• 1.1.10.1: ADFS Federation using AWS Cognito User Pools
• 1.1.10.2: Installing the Policy Agent and Protector in Different AWS Accounts
• 1.1.10.3: Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)
• 1.1.10.4: Invoke Lambda Directly
• 1.1.10.5: Policy Agent - Custom VPC Endpoint Hostname Configuration
• 1.1.10.6: Protection Methods
• 1.1.10.7: Configuring Regular Expression to Extract Policy Username
• 1.1.10.8: Associating ESA Data Store With Cloud Protect Agent
• 1.2: Athena
• 1.2.1: Overview
• 1.2.2: Architecture
• 1.2.3: Installation
• 1.2.3.1: Prerequisites
• 1.2.3.2: Pre-Configuration
• 1.2.3.2.1:
• 1.2.3.3: Protect Service Installation
• 1.2.3.4: Policy Agent Installation
• 1.2.3.5: Audit Log Forwarder Installation
• 1.2.3.6:
• 1.2.3.7:
• 1.2.3.8:
• 1.2.3.9:
• 1.2.3.10:
• 1.2.3.11:
• 1.2.3.12:
• 1.2.4: Understanding Athena Objects
• 1.2.4.1:
• 1.2.5: Performance
• 1.2.6: Audit Logging
• 1.2.7: No Access Behavior
• 1.2.8: Known Limitations
• 1.2.9: Upgrading To The Latest Version
• 1.2.10: Appendices
• 1.2.10.1: Sample Athena External Functions
• 1.2.10.2: Installing the Policy Agent and Protector in Different AWS Accounts
• 1.2.10.3: Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)
• 1.2.10.4: Security Recommendations
• 1.2.10.5: Policy Agent - Custom VPC Endpoint Hostname Configuration
• 1.2.10.6: Protection Methods
• 1.2.10.7: Configuring Regular Expression to Extract Policy Username
• 1.2.10.8: Associating ESA Data Store With Cloud Protect Agent
• 1.3: Redshift
• 1.3.1: Overview
• 1.3.2: Architecture
• 1.3.2.1:
• 1.3.3: Installation
• 1.3.3.1: Prerequisites
• 1.3.3.2: Pre-Configuration
• 1.3.3.3: Protect Service Installation
• 1.3.3.4: Policy Agent Installation
• 1.3.3.5: Audit Log Forwarder Installation
• 1.3.3.6:
• 1.3.3.7:
• 1.3.3.8:
• 1.3.4: Understanding Redshift Objects
• 1.3.4.1:
• 1.3.4.2:
• 1.3.4.3:
• 1.3.5: Performance
• 1.3.5.1:
• 1.3.5.2:
• 1.3.5.3:
• 1.3.5.4:
• 1.3.6: Audit Logging
• 1.3.7: No Access Behavior
• 1.3.8: Known Limitations
• 1.3.9: Upgrading To The Latest Version
• 1.3.10: Appendices
• 1.3.10.1: Installing the Policy Agent and Protector in Different AWS Accounts
• 1.3.10.2: Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)
• 1.3.10.3: Policy Agent - Custom VPC Endpoint Hostname Configuration
• 1.3.10.4: Redshift Cross-Account Configuration
• 1.3.10.5: Sample Redshift External Function
• 1.3.10.6: Configuring Regular Expression to Extract Policy Username
• 1.3.10.7: Associating ESA Data Store With Cloud Protect Agent
• 1.4: Snowflake
• 1.4.1: Overview
• 1.4.2: Installation
• 1.4.2.1: Pre-Configuration
• 1.4.2.2: Prerequisites
• 1.4.2.3: Protect Service Installation
• 1.4.2.4: Policy Agent Installation
• 1.4.2.5: Audit Log Forwarder Installation
• 1.4.2.6:
• 1.4.2.7:
• 1.4.2.8:
• 1.4.3: Understanding Snowflake Objects
• 1.4.3.1: External Functions
• 1.4.3.2: Snowflake Masking Policies
• 1.4.4: Performance
• 1.4.4.1: Performance Considerations
• 1.4.4.2: Sample Benchmarks
• 1.4.4.3: Concurrency
• 1.4.4.4: Concurrency Tuning
• 1.4.4.5: Log Forwarder Performance Tuning
• 1.4.4.6:
• 1.4.4.7:
• 1.4.4.8:
• 1.4.4.9:
• 1.4.5: Audit Logging
• 1.4.6: No Access Behavior
• 1.4.7: Upgrading To The Latest Version
• 1.4.8: Known Limitations
• 1.4.9: Appendices
• 1.4.9.1: Sample Snowflake External Function
• 1.4.9.2: Installing the Policy Agent and Protector in Different AWS Accounts
• 1.4.9.3: Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)
• 1.4.9.4: Policy Agent - Custom VPC Endpoint Hostname Configuration
• 1.4.9.5: Protection Methods
• 1.4.9.6: Configuring Regular Expression to Extract Policy Username
• 1.4.9.7: Associating ESA Data Store With Cloud Protect Agent
• 1.5: Amazon S3
• 1.5.1: Overview
• 1.5.2: Architecture
• 1.5.3: Installation
• 1.5.3.1: Prerequisites
• 1.5.3.2: Pre-Configuration
• 1.5.3.3: S3 Protector Service Installation
• 1.5.3.4:
• 1.5.3.5:
• 1.5.3.6:
• 1.5.4: Mapping File Configuration
• 1.5.4.1: Mapping File
• 1.5.4.2: Column Mapping Rules
• 1.5.5: Large File Processing
• 1.5.6: Performance
• 1.5.7: Known Limitations
• 1.5.8: Upgrading To The Latest Version
• 1.5.9: Appendices
• 1.5.9.1: Sample Configuration
• 1.5.9.2: Amazon S3 Security Best Practices Examples
• 1.6:
• 1.7:
• 1.8:
• 1.9:
• 1.10:
• 1.11:
• 1.12:
• 1.13:
• 1.14:
• 1.15:
• 2: Azure
• 2.1: Cloud API
• 2.1.1: Overview
• 2.1.2: Architecture
• 2.1.3: Installation
• 2.1.3.1: Prerequisites
• 2.1.3.2: Pre-Configuration
• 2.1.3.3: Protect Service Installation
• 2.1.3.4: Policy Agent Installation
• 2.1.3.5: Audit Log Forwarder Installation
• 2.1.3.6:
• 2.1.3.7:
• 2.1.3.8:
• 2.1.4: REST API
• 2.1.4.1: Export an OpenAPI Spec
• 2.1.4.2: Payload Encoding
• 2.1.4.3: v1 Specification
• 2.1.4.4: Legacy Specification
• 2.1.4.5: SSL Certificates
• 2.1.5: Performance
• 2.1.5.1: Function App Performance
• 2.1.5.2: Log Forwarder Performance
• 2.1.6: Audit Logging
• 2.1.7: No Access Behavior
• 2.1.8: Upgrading To The Latest Version
• 2.1.9: Known Limitations
• 2.1.10: Appendices
• 2.1.10.1: Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)
• 2.1.10.2: Configuring Regular Expression to Extract Policy Username
• 2.1.10.3: Getting JWT for Service Account in Azure Active Directory
• 2.1.10.4: Protection Methods
• 2.1.10.5: ARM Template Installation - Required Permissions
• 2.1.10.6: Associating ESA Data Store With Cloud Protect Agent
• 2.2: Snowflake
• 2.2.1: Overview
• 2.2.2: Architecture
• 2.2.3: Installation
• 2.2.3.1: Prerequisites
• 2.2.3.2: Pre-Configuration
• 2.2.3.3: Protect Service Installation
• 2.2.3.4: Snowflake Configuration
• 2.2.3.5: Policy Agent Installation
• 2.2.3.6: Audit Log Forwarder Installation
• 2.2.3.7:
• 2.2.3.8:
• 2.2.4: Understanding Snowflake Objects
• 2.2.4.1: External Functions
• 2.2.4.2: Snowflake Masking Policies
• 2.2.5: Performance
• 2.2.5.1: Function App Performance
• 2.2.5.2: Log Forwarder Performance
• 2.2.6: Audit Logging
• 2.2.7: No Access Behavior
• 2.2.8: Upgrading To The Latest Version
• 2.2.9: Known Limitations
• 2.2.10: Appendices
• 2.2.10.1: Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)
• 2.2.10.2: Configuring Regular Expression to Extract Policy Username
• 2.2.10.3: Sample Snowflake External Functions for Protegrity Integration
• 2.2.10.4: ARM Template Installation - Required Permissions
• 2.2.10.5: Associating ESA Data Store With Cloud Protect Agent
• 2.3:
• 2.4:
• 2.5:
• 2.6:
• 2.7:
• 2.8:
• 2.9:
• 2.10:
• 2.11:
• 2.12:
• 2.13:
• 2.14:
• 2.15:
• 2.16:
• 2.17:
• 2.18:
• 2.19:
• 2.20:
• 2.21:
• 2.22:
• 2.23:
• 2.24:
• 2.25:
• 2.26:
• 2.27:
• 2.28:
• 2.29:
• 2.30:
• 2.31:
• 2.32:
• 2.33:
• 2.34:
• 2.35:
• 2.36:
• 2.37:
• 2.38:
• 2.39:
• 2.40:
• 2.41:
• 2.42:
• 2.43:
• 2.44:
• 2.45:
• 2.46:
• 2.47:
• 2.48:
• 2.49:
• 2.50:
• 2.51:
• 2.52:
• 2.53:
• 2.54:
• 2.55:
• 2.56:
• 2.57:
• 2.58:
• 2.59:
• 2.60:
• 2.61:
• 2.62:
• 2.63:
• 2.64:
• 2.65:
• 2.66:
• 2.67:
• 2.68:
• 2.69:
• 2.70:
• 2.71:
• 2.72:
• 2.73:
• 3: GCP
• 3.1: Cloud API
• 3.1.1: Overview
• 3.1.2: Architecture
• 3.1.3: Installation
• 3.1.3.1: Prerequisites
• 3.1.3.2: Pre-Configuration
• 3.1.3.3: Protect Service Installation
• 3.1.3.4: Policy Agent Installation
• 3.1.3.5: Audit Log Forwarder Installation
• 3.1.3.6:
• 3.1.3.7:
• 3.1.4: REST API
• 3.1.4.1: Policy Users
• 3.1.4.2: JWT Verification
• 3.1.4.3: Authorization Header
• 3.1.4.4: Export an OpenAPI Spec
• 3.1.4.5: v1 Specification
• 3.1.4.6: Legacy Specification
• 3.1.4.7: HTTP Status Codes
• 3.1.4.8: SSL Certificates
• 3.1.4.9:
• 3.1.4.10:
• 3.1.4.11:
• 3.1.4.12:
• 3.1.4.13:
• 3.1.4.14:
• 3.1.5: Performance
• 3.1.5.1: Performance Considerations
• 3.1.5.2: Log Forwarder Performance
• 3.1.6: Audit Logging
• 3.1.7: No Access Behavior
• 3.1.8: Upgrading To The Latest Version
• 3.1.9: Known Limitations
• 3.1.10: Appendices
• 3.1.10.1: Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)
• 3.1.10.2: Protection Methods
• 3.1.10.3: Configuring Regular Expression to Extract Policy Username
• 3.1.10.4: Associating ESA Data Store With Cloud Protect Agent
• 3.1.10.5: Undeliverable Audit Log Recovery
• 3.2: BigQuery
• 3.2.1: Overview
• 3.2.2: Architecture
• 3.2.3: Installation
• 3.2.3.1: Prerequisites
• 3.2.3.2: Pre-Configuration
• 3.2.3.3: Protect Service Installation
• 3.2.3.4: Policy Agent Installation
• 3.2.3.5: Audit Log Forwarder Installation
• 3.2.3.6:
• 3.2.3.7:
• 3.2.4: BigQuery Configuration
• 3.2.5: Performance
• 3.2.5.1: Performance Considerations
• 3.2.5.2: Sample Benchmarks
• 3.2.5.3: Log Forwarder Performance
• 3.2.6: Audit Logging
• 3.2.7: No Access Behavior
• 3.2.8: Upgrading To The Latest Version
• 3.2.9: Known Limitations
• 3.2.10: Appendices
• 3.2.10.1: Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)
• 3.2.10.2: Sample BigQuery Remote Function
• 3.2.10.3: Configuring Regular Expression to Extract Policy Username
• 3.2.10.4: Associating ESA Data Store With Cloud Protect Agent
• 3.2.10.5: Undeliverable Audit Log Recovery
• 3.2.10.6:
• 3.2.10.7:
• 3.2.10.8:
• 3.2.10.9:
• 3.2.11:
• 3.3: Snowflake
• 3.3.1: Overview
• 3.3.2: Architecture
• 3.3.3: Installation
• 3.3.3.1: Prerequisites
• 3.3.3.2: Pre-Configuration
• 3.3.3.3: Protect Service Installation
• 3.3.3.4: Snowflake Configuration
• 3.3.3.5: Policy Agent Installation
• 3.3.3.6: Audit Log Forwarder Installation
• 3.3.3.7:
• 3.3.3.8:
• 3.3.4: Understanding Snowflake Objects
• 3.3.4.1: External Functions
• 3.3.4.2: Snowflake Masking Policies
• 3.3.5: Performance
• 3.3.5.1: Performance Considerations
• 3.3.5.2: Sample Benchmarks
• 3.3.5.3: Concurrency
• 3.3.5.4: Log Forwarder Performance
• 3.3.6: Audit Logging
• 3.3.7: No Access Behavior
• 3.3.8: Upgrading To The Latest Version
• 3.3.9: Known Limitations
• 3.3.10: Appendices
• 3.3.10.1: Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)
• 3.3.10.2: Sample Snowflake External Function
• 3.3.10.3: Configuring Regular Expression to Extract Policy Username
• 3.3.10.4: Associating ESA Data Store With Cloud Protect Agent
• 3.3.10.5: Undeliverable Audit Log Recovery
• 3.3.11:
• 3.4:
• 3.5:
• 3.6:
• 3.7:
• 3.8:
• 3.9:
• 3.10:
• 3.11:
• 3.12:
• 3.13:
• 3.14:
• 3.15:
• 3.16:
• 3.17:
• 3.18:
• 3.19:
• 3.20:
• 3.21:
• 3.22:
• 3.23:
• 3.24:
• 3.25:
• 3.26:
• 3.27:
• 3.28:
• 3.29:
• 3.30:
• 3.31:
• 3.32:
• 3.33:
• 3.34:
• 3.35:
• 3.36:
• 3.37:
• 3.38:
• 3.39:
• 3.40:
• 3.41:
• 3.42:
• 3.43:
• 3.44:
• 3.45:
• 3.46:
• 3.47:
• 3.48:
• 3.49:
• 3.50:
• 3.51:
• 3.52:
• 3.53:
• 3.54:
• 3.55:
• 3.56:
• 3.57:
• 3.58:
• 3.59:
• 3.60:
• 3.61:
• 3.62:
• 3.63:
• 3.64:
• 3.65:
• 4:
• 5:
• 6:
• 7:
• 8:
• 9:
• 10:
• 11:
• 12:
• 13:
• 14:
• 15:

1 - AWS

1.1 - Cloud API

This section describes the high-level architecture of Protegrity Cloud API, installation procedures, and performance guidance. This section focuses on Protegrity specific aspects and should be consumed in conjunction with the corresponding AWS documentation.

This guide may also be used with the Protegrity Enterprise Security Administrator Guide, which explains the mechanism for managing the data security policy.

1.1.1 - Overview

Solution Overview

Cloud API Protector on AWS is a cloud-native, serverless product for fine-grained data protection. This enables the invocation of Protegrity data protection cryptographic methods in cloud-native serverless technology. The benefits of serverless include rapid auto-scaling, performance, low administrative overhead, and reduced infrastructure costs compared to a server-based solution.

This product provides a data protection API end-point for clients. The product is designed to scale elastically and yield reliable query performance under extremely high concurrent loads. During idle use, the serverless product will scale completely down, providing significant savings in Cloud compute fees.

Protegrity utilizes a data security policy maintained by an Enterprise Security Administrator (ESA), similar to other Protegrity products. Using a simple REST API interface, authorized users can perform both de-identification (protect) and re-identification (unprotect) operations on data. A user’s individual capabilities are subject to privileges and policies defined by the Enterprise Security Administrator.

Analytics on Protected Data

Protegrity’s format and length preserving tokenization scheme make it possible to perform analytics directly on protected data. Tokens are join-preserving so protected data can be joined across datasets. Often statistical analytics and machine learning training can be performed without the need to re-identify protected data. However, a user or service account with authorized security policy privileges may re-identify subsets of data using the Cloud API Protector on AWS service.

Features

Cloud API Protector on AWS incorporates Protegrity’s patent-pending vaultless tokenization capabilities into cloud-native serverless technology. Combined with an ESA security policy, the protector provides the following features:

• Role-based access control (RBAC) to protect and unprotect (re-identify) data depending on the user privileges.
• Policy enforcement features of other Protegrity application protectors.

For more information about the available protection options, such as, data types, Tokenization or Encryption types, or length preserving and non-preserving tokens, refer to Protection Methods Reference.

1.1.2 - Architecture

Protector Deployment Architecture

The product will be deployed in the Customer’s AWS account. The product incorporates Protegrity’s vaultless tokenization engine within an AWS Lambda Function. The encrypted data security policy from an ESA is deployed as a static resource through an Amazon Lambda Layer. The policy is decrypted in memory at runtime within the Lambda. This architecture allows Protegrity Serverless to be highly available and scale very quickly without direct dependency on any other Protegrity services.

The product exposes a remote data protection service. Each REST request includes a micro-batch of data to process and the data element type. The function applies the data security policy including user authorization and returns a corresponding response.

When used with an Enterprise Security Administrator (ESA) application, the security policy is synchronized through another serverless component called the Protegrity Policy Agent. The agent operates on a configurable schedule, fetches the policy from the ESA, performs additional envelope encryption using Amazon KMS, and deploys the policy into the Lambda Layer used by the serverless product. This solution can be configured to automatically provision the static policy or the final step can be performed on-demand by an administrator. The policy takes effect immediately for all new requests. There is no downtime for users during this process.

The following diagram shows the high-level architecture described above.

The following diagram shows a reference architecture for synchronizing the security policy from ESA.

The Protegrity Policy Agent requires network access to an Enterprise Security Administrator (ESA). Most organizations install the ESA on-premise. Therefore, it is recommended that the Policy Agent is installed into a private subnet with a Cloud VPC using a NAT Gateway to enable this communication through a corporate firewall.

The ESA is a soft appliance that must be pre-installed on a separate server. It is used to create and manage security policies.

For more information about installing the ESA, and creating and managing policies, refer to the Policy Management Guide.

Log Forwarding Architecture

Audit logs are by default sent to CloudWatch as long as the function’s execution role has the necessary permissions. The Protegrity Product can also be configured to send audit logs to ESA. Such configuration requires deploying Log Forwarder component which is available as part of Protegrity Product deployment bundle. The diagram below shows additional resources deployed with Log Forwarder component.

The log forwarder component includes Amazon Kinesis data stream and the forwarder Lambda function. Amazon Kinesis stream is used to batch audit records before sending them to forwarder function, where similar audit logs are aggregated before sending to ESA. Aggregation rules are described in the Protegrity Log Forwarding guide. When the protector function is configured to send audit logs to log forwarder, audit logs are aggregated on the protector function before sending to Amazon Kinesis. Due to specifics of the Lambda runtime lifecycle, audit logs may take up to 15 minutes before being sent to Amazon Kinesis. Protector function exposes configuration to minimize this time which is described in the protector function installation section.

The security of audit logs is ensured by using HTTPS connection on each link of the communication between protector function and ESA. Integrity and authenticity of audit logs is additionally checked on log forwarder which verifies individual logs signature. The signature verification is done upon arrival from Amazon Kinesis before applying aggregation. If signature cannot be verified, the log is forwarded as is to ESA where additional signature verification can be configured. Log forwarder function uses client certificate to authenticate calls to ESA.

To learn more about individual audit log entry structure and purpose of audit logs, refer to Audit Logging section in this document. Installation instructions can be found in Audit Log Forwarder installation.

The audit log forwarding requires network access from the cloud to the ESA. Most organizations install the ESA on-premise. Therefore, it is recommended that the Log Forwarder Function is installed into a private subnet with a Cloud VPC using a NAT Gateway to enable this communication through a corporate firewall.

Access Control

The following mechanisms are available for controlling and restricting access to the endpoint:

• IAM policy: The IAM resource policy controls which IAM users or services may invoke Protect operations. IAM policies can be applied to the API Gateway and/or the Lambda directly depending on allowable access patterns and the client.
• JWT tokens: The Lambda can be configured to use JSON Web Tokens (JWT) with optional verification. JSON Web Tokens (JWT) are an open, industry-standard RFC 7519 method for representing claims securely between two parties. JWT provides a mechanism for implementing custom authentication or integrating with AWS Cognito.

AWS Resources

Access and authorization between various AWS services involved in this architecture are achieved through IAM resource policies. For instance, the Amazon Lambda resource-based policy can restrict requests to the Amazon API Gateway or optionally allow direct invocation to the Lambda function itself. The installation steps provide a default recommended configuration. Alternative IAM role configurations are shown in the appendices in this document.

REST API Authentication

AWS API Gateway supports multiple mechanisms for controlling and managing access to the product.

In standard solutions, AWS API Gateway will authorize access tokens generated in the identity provider. When setting up an AWS API Gateway method to require an authorization, customers can leverage AWS Signature Version 4 or Lambda authorizers to support their organization’s bearer token auth strategy.

https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html

REST API Authorization

Once the request is authenticated and authorized in the API Gateway, Protegrity Lambda Protector validates the user received in the authorization header of JWT, and the data element and security operations (protect or unprotect) from the payload with Protegrity Security Policy.

If the API Gateway is not used or configured to verify JWT tokens, the product can be configured to perform the JWT verification in the Lambda function itself.

1.1.3 - Installation

1.1.3.1 - Prerequisites

AWS Services

The following table describes the AWS services that may be a part of your Protegrity installation.

ESA Version Requirements

The Protector and Log Forwarder functions require a security policy from a compatible ESA version.

The table below shows compatibility between different Protector and ESA versions.

Prerequisites

Required Skills and Abilities

Cheat Sheet Recommendation

AWS Account ID: ___________________

AWS Region (AwsRegion): ___________________

S3 Bucket name (ArtifactS3Bucket): ___________________

KMS Key ARN (AWS_KMS_KEY_ID): ___________________

ProtectLambdaPolicyName: __________________

Role ARN (LambdaExecutionRoleArn): ___________________

ApiGatewayId: ________________________________

ProtectFunctionName: __________________________

ProtectLayerName: _____________________________

ESA IP address: ___________________

VPC name: ___________________

Subnet name: ___________________

Policy Agent Security Group Id: ___________________

ESA Credentials Secret Name: ___________________

Policy Name: ___________________

Agent Lambda IAM Execution Role Name: ___________________

What’s Next

• Pre-Configuration

1.1.3.2 - Pre-Configuration

Provide AWS sub-account

Identify or create an AWS account where the Protegrity solution will be installed. It is recommended that a new AWS sub-account be created. This can provide greater security controls and help avoid conflicts with other applications that might impact regional account limits. An individual with the Cloud Administrator role will be required for some subsequent installation steps.

AWS Account ID: ___________________

AWS Region (AwsRegion): ___________________

Create S3 bucket for Installing Artifacts

This S3 bucket will be used for the artifacts required by the CloudFormation installation steps. This S3 bucket must be created in the region that is defined in Provide AWS sub-account

1. Sign in to the AWS Management Console and open the Amazon S3 console.

2. Change region to the one determined in Provide AWS sub-account

3. Click Create Bucket.

4. Enter a unique bucket name:

   For example, protegrity-install.us-west-2.example.com

5. Upload the installation artifacts to this bucket. Protegrity will provide the following three artifacts:

   • protegrity-protect-<version>.zip
   • protegrity-agent-<version>.zip
   • protegrity-external-extension-<version>.zip
   • protegrity-sample-policy-<version>.zip

     Important

     The deployment package you receive from Protegrity must be extracted to reveal the Protegrity artifacts. CloudFormation requires them in the provided .zip format. Do not extract the individual Protegrity artifacts. Upload these artifacts to the S3 bucket created.

   S3 Bucket name (ArtifactS3Bucket): ___________________

Create KMS Key

The Amazon Key Management Service (KMS) provides the ability for the Protegrity Serverless solution to encrypt and decrypt the Protegrity Security Policy.

To create KMS key:

1. In the AWS sub-account where the KMS key will reside, select the region.

2. Navigate to Key Management Service > Create Key.

   

3. Configure the key settings:

   • Key type: Asymmetric
   • Key usage: Encrypt and decrypt
   • Key spec: RSA_4096
   • Click Next

4. Create alias and optional description, such as, Protegrity-Serverless and click Next.

5. Define key administrative permissions, the IAM user who will administrate the key.

   Note

   It is recommended the administrator be different than the administrator of the Protegrity Serverless account

6. Click Next.

7. Define the key usage permissions.

8. In Other AWS accounts, enter the AWS account id used for the Protegrity Serverless installation.

9. Continue on to create the key. If there is a concern this permission is overly broad, then you can return later to restrict access to the role of two Protegrity Serverless Lambda as principals. Click to open the key in the list and record the ARN.

   KMS Key ARN (AWS_KMS_KEY_ID): ___________________

10. Download the public key from the KMS key. Navigate to the key in KMS console, select the Public key tab, and click Download. Save the PEM file. This public key will be added to the ESA data store as an export key. Refer to Exporting Keys to Datastore for instructions on adding the public key to the data store.

    Note

    This step is not applicable for ESA versions lower than 10.2.

    KMS Public Key PEM file: ___________________

What’s Next

• Protect Service Installation

1.1.3.3 - Protect Service Installation

Protect Service Installation

The following sections install the Cloud Protect serverless Lambda function.

Preparation

Ensure that all the steps in Pre-Configuration are performed.

1. Login to the AWS sub-account console where Protegrity will be installed.

2. Ensure that the required CloudFormation templates provided by Protegrity are available on your local computer.

Create Protect Lambda IAM Execution Policy

This task defines a policy used by the Protegrity Lambda function to write CloudWatch logs and access the KMS encryption key to decrypt the policy.

Perform the following steps to create the Lambda execution role and required policies:

1. From the AWS IAM console, select Policies > Create Policy.

2. Select the JSON tab and copy the following sample policy.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "CloudWatchWriteLogs",
         "Effect": "Allow",
         "Action": [
           "logs:CreateLogGroup",
           "logs:CreateLogStream",
           "logs:PutLogEvents"
         ],
         "Resource": "*"
       },
       {
         "Sid": "KmsDecrypt",
         "Effect": "Allow",
         "Action": [
           "kms:Decrypt"
         ],
         "Resource": [
           "arn:aws:kms:*:*:key/*"
         ]
       }
     ]
   }
   

3. For the KMS policy, replace the Resource with the ARN for the KMS key created in a previous step.

4. Select Next, type in a policy name, for example, ProtegrityProtectLambdaPolicy and Create Policy. Record the policy name:

   ProtectLambdaPolicyName:__________________

Create Protect Lambda IAM Role

The following steps create the role to utilize the policy defined in Create Protect Lambda IAM Execution Policy.

To create protect lambda IAM execution role:

1. From the AWS IAM console, select Roles > Create Role.

2. Select AWS Service > Lambda > Next.

3. In the list, search and select the policy created in Create Protect Lambda IAM Execution Policy.

4. Click Next

5. Type the role name, for example, ProtegrityProtectRole

6. Click Create role

7. Record the role ARN.

   Role ARN (LambdaExecutionRoleArn): ___________________

Install through CloudFormation

The following steps describe the deployment of the Lambda function.

1. Access CloudFormation and select the target AWS Region in the console.

2. Click Create Stack and choose With new resources.

3. Specify the template.

4. Select Upload a template file.

5. Upload the Protegrity-provided CloudFormation template called pty_protect_api_cf.json and click Next.

6. Specify the stack details. Enter a stack name.

   Note

   The stack name will be appended to all the services created by the template.

7. Enter the required parameters. All the values were generated in the pre-configuration steps.

   Parameter	Description	Default Value
   ArtifactS3Bucket	The name of S3 bucket created during the pre-configuration steps
   LambdaExecutionRoleArn	The ARN of Lambda role created in the prior step
   MinLogLevel	The minimum log level for the protect function. Supported values: [off, severe, warning, info, config, all]	severe
   AllowAssumeUser (Optional)	If 0 is set, the user in the request body will be ignored and the REST API authenticated user will be the acting user. Supported values: [0, 1]	0
   LambdaAuthorization (Optional)	If “jwt” is set, then Authorization header for JWT will be required in the AWS Protect Lambda request. Any other value is ignored and effective policy user is taken from request payload. Supported values: [“jwt”, “”]	""
   jwtUsernameClaim (Optional)	The JWT claim with username. Common claims: name, preferred_username, cognito:username Also accepts ordered list of claim names in JSON array format, e.g. [“username”, “email”]	cognito:username
   jwtVerify (Optional)	If 1 is set, then jwtSecretBase64 is required. Only applicable when LambdaAuthorization is set to “jwt”. Supported JWT algorithms are: RS256, RS384, RS512. While algorithms HS256, HS384, HS512 are supported, they are not recommended for use.	0
   jwtSecretBase64 (Optional)	Required when jwtVerify is set to 1 and Authorization is set to “jwt”. The secret must be provided in base64 encoding. It is recommended to only use public key (asymmetric algorithm).	""

8. The log forwarder parameters can be provided later after log forwarder is deployed. If you are not planning to deploy log forwarder you can skip this step.

   Parameter	Description
   KinesisLogStreamArn	The ARN of the AWS Kinesis stream where audit logs will be sent for aggregation
   AuditLogFlushInterval	Time interval in seconds used to accumulate audit logs before sending to Kinesis. Default value: 30. See Log Forwarder Performance section for more details.

9. Click Next with defaults to complete CloudFormation.

10. After CloudFormation is completed, select the Outputstab in the stack.

11. Record the following values:

    ApiGatewayId: ________________________________

    ProtectFunctionName: __________________________

    ProtectFunctionProductionAlias: __________________________

    ProtectLayerName: _____________________________

Test Connectivity

Perform the following steps to verify if the API Gateway is working correctly with the Protegrity product.

1. 1. In the AWS console Access API Gateway.

2. Search for API (CloudFormation output ApiGatewayId value).

3. Select Resources > /v1/unprotect POST.

4. Click on Test tab.

5. Provide the following input in Headers.

   Authorization:
   

6. Provide the following input in Request body.

   {"user": "test_user", "data_element": "alpha", "data": ["UtfVk UHgcD!"]}
   

7. Click Test.

8. Verify the response Status is 200 and the Response Body is as follows:

   {"encoding": "utf8", "results": ["hello world!"], "success": true}
   

   For example:

Troubleshooting

{
    "body": "{\"user\":\"test_user\",\"data_element\": \"alpha\", \"data\": [\"UtfVk UHgcD!\"]}",
    "path": "/unprotect",
    "headers": {
        "Authorization": ""
    }
}

Setting up Authentication

This step describes how to setup an AWS API Gateway Authorization.

By default, the API Gateway is configured to use AWS_IAM Authorization.

1. From AWS console access API Gateway.
2. If you are using AWS API Gateway Authorizer, ensure that the Authorizer is configured in Authorizers.
3. Go to Resources > /v1/protect POST > Method Request tab > Click Edit button
4. Select from Authorization. dropdown.
5. Click Save button.
6. Click Deploy API button to deploy to pty stage.

Protect Lambda Configuration

After Cloud Formation stack is deployed, Protector Lambda can be reconfigured based on the authentication selected in previous stage. From your AWS console navigate to lambda and select following lambda: Protegrity_Protect_RESTAPI_<STACK_NAME>. Scroll down to Environment variables section, select Edit and replace the entries based on the following information.

What’s Next

• Install Policy Agent

1.1.3.4 - Policy Agent Installation

The following sections will install the Policy Agent. The Policy Agent polls the ESA and deploys the policy to Protegrity Serverless as a static resource. Some of the installation steps are not required for the operation of the software but recommended for establishing a secure environment. Contact Protegrity Professional Services for further guidance on configuration alternatives in the Cloud.

ESA Server

Policy Agent Lambda requires ESA server running and accessible on TCP port 443.

Note down ESA IP address:

ESA IP Address (EsaIpAddress): ___________________

Certificates on ESA

Whether your ESA is configured with default self-signed certificate or your corporate CA certificate, Policy Agent can validate authenticity of ESA connection using CA certificate. The process for both scenarios is the same:

• Obtain CA certificate
• Convert CA certificate to a value accepted by Policy Agent
• Provide converted CA certificate value to Policy Agent

To obtain self-signed CA certificate from ESA:

1. Log in to ESA Web UI.

2. Select Settings > Network > Manage Certificates.

3. Hover over Server Certificate and click on download icon to download the CA certificate.

4. To convert downloaded CA certificate to a value accepted by Policy Agent, open the downloaded PEM file in text editor and replace all new lines with escaped new line: \n.

   To escape new lines from command line, use one of the following commands depending on your operating system:

   Linux Bash:

   awk 'NF {printf "%s\\n",$0;}' ProtegrityCA.pem > output.txt
   

   Windows PowerShell:

   (Get-Content '.\ProtegrityCA.pem') -join '\n' | Set-Content 'output.txt'
   

5. Record the certificate content with new lines escaped.

   ESA CA Server Certificate (EsaCaCert): ___________________

   This value will be used to set PTY_ESA_CA_SERVER_CERT or PTY_ESA_CA_SERVER_CERT_SECRET Lambda variable in section Policy Agent Lambda Configuration

For more information about ESA certificate management refer to Certificate Management Guide in ESA documentation.

Identify or Create a new VPC

Establish a VPC where the Policy Agent will be hosted. This VPC will need connectivity to the ESA. The VPC should be in the same account and region established in Pre-Configuration.

VPC name: ___________________

VPC Subnet Configuration

Identify or create a new subnet in the VPC where tha Lambda function will be connected to. It is recommended to use a private subnet.

Subnet name: ___________________

NAT Gateway For ESA Hosted Outside AWS Network

If ESA server is hosted outside of the AWS Cloud network, the VPC configured for Lambda function must ensure additional network configuration is available to allow connectivity with ESA. For instance if ESA has a public IP, the Lambda function VPC must have public subnet with a NAT server to allow routing traffic outside of the AWS network. A Routing Table and Network ACL may need to be configured for outbound access to the ESA as well.

VPC Endpoints Configuration

If an internal VPC was created, then add VPC Endpoints, which will be used by the Policy Agent to access AWS services. Policy Agent needs access to the following AWS services:

Identify or Create Security Groups

Policy Agent and cloud-based ESA appliance use AWS security groups to control traffic that is allowed to leave and reach them. Policy Agent runs on schedule and is mostly concerned with allowing traffic out of itself to ESA and AWS services it depends on. ESA runs most of the time and it must allow Policy Agent to connect to it.

Policy Agent security group must allow outbound traffic using rules described in the table below. To edit security group navigate:

From VPC > Security Groups > Policy Agent Security Group configuration.

Record Policy Agent security group ID:

Policy Agent Security Group Id: ___________________

Policy Agent will reach out to ESA on port 443. Create following inbound security group rule for cloud-based ESA appliance to allow connections from Policy Agent:

Creating ESA Credentials

Policy Agent Lambda requires ESA credentials to be provided as one of the three options.

• Option 1: Secrets Manager

• Option 2: KMS Encrypted Password

• Option 3: Custom AWS Lambda function

Option 1: Secrets Manager

Creating secrets manager secret with ESA username and password.

1. From the AWS Secrets Manager Console, select Store New Secret.

2. Select Other Type of Secrets.

3. Specify the username and password key value pair.

   

4. Select the encryption key or leave default AWS managed key.

5. Specify the Secret Name and record it.

   ESA Credentials Secret Name: __________________

Option 2: KMS Encrypted Password

ESA password is encrypted with AWS KMS symmetric key.

1. Create AWS KMS symmetric key which will be used to encrypt ESA password. See Create KMS Key for instructions on how to create KMS symmetric key using AWS console.

2. Record KMS Key ARN.

   ESA PASSWORD KMS KEY ARN: __________________

3. Run AWS CLI command to encrypt ESA password. Below you can find sample Linux aws cli command. Replace <key_arn> with KMS symmetric key ARN.

   aws kms encrypt --key-id <key_arn> --plaintext $(echo '<esa_password>' | base64 )
   

4. Sample output.

   {
     "CiphertextBlob": "esa_encrypted_password",
     "KeyId": "arn:aws:kms:region:aws_account:key/key_id ",
     "EncryptionAlgorithm": "SYMMETRIC_DEFAULT"
   }
   

5. Record ESA username and encrypted password.

   ESA USERNAME: __________________

   ESA ENCRYPTED PASSWORD: __________________

Option 3: Custom AWS Lambda function

With this option ESA username and password are returned by a custom AWS Lambda function. This method may be used to get the username and password from external vaults.

1. Create AWS Lambda in any AWS supported runtime.

   1. There is no input needed.

   2. The Lambda function must return the following response schema.

      response:
      type: object
        properties:
          username: string
          password: string
      

      For example,

      example output: {"username": "admin", "password": "Password1234"}
      

   3. Sample AWS Lambda function in Python:

      import json
      
      def lambda_handler(event, context):
      
          return {"username": "admin", "password": "password1234"}
      

      Warning

      Protegrity does not recommend hardcoding ESA username and password in the clear.

2. Record the Lambda name:

   Custom AWS lambda for ESA credentials: _______________

Create Agent Lambda IAM Policy

Follow the steps below to create Lambda execution policies.

Create Agent Lambda IAM policy

1. From AWS IAM console, select Policies > Create Policy.

2. Select JSON tab and copy the following snippet.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "EC2ModifyNetworkInterfaces",
         "Effect": "Allow",
         "Action": [
           "ec2:CreateNetworkInterface",
           "ec2:DescribeNetworkInterfaces",
           "ec2:DeleteNetworkInterface"
         ],
         "Resource": "*"
       },
       {
         "Sid": "CloudWatchWriteLogs",
         "Effect": "Allow",
         "Action": [
           "logs:CreateLogGroup",
           "logs:CreateLogStream",
           "logs:PutLogEvents"
         ],
         "Resource": "*"
       },
       {
         "Sid": "LambdaUpdateFunction",
         "Effect": "Allow",
         "Action": [
           "lambda:UpdateFunctionConfiguration"
         ],
         "Resource": [
           "arn:aws:lambda:*:*:function:*"
         ]
       },
       {
         "Sid": "LambdaReadLayerVersion",
         "Effect": "Allow",
         "Action": [
           "lambda:GetLayerVersion",
           "lambda:ListLayerVersions"
         ],
         "Resource": "*"
       },
       {
         "Sid": "LambdaDeleteLayerVersion",
         "Effect": "Allow",
         "Action": "lambda:DeleteLayerVersion",
         "Resource": "arn:aws:lambda:*:*:layer:*:*"
       },
       {
         "Sid": "LambdaPublishLayerVersion",
         "Effect": "Allow",
         "Action": "lambda:PublishLayerVersion",
         "Resource": "arn:aws:lambda:*:*:layer:*"
       },
       {
         "Sid": "S3GetObject",
         "Effect": "Allow",
         "Action": [
           "s3:GetObject"
         ],
         "Resource": "arn:aws:s3:::*/*"
       },
       {
         "Sid": "S3PutObject",
         "Effect": "Allow",
         "Action": [
           "s3:PutObject"
         ],
         "Resource": "arn:aws:s3:::*/*"
       },
       {
         "Sid": "KmsEncrypt",
         "Effect": "Allow",
         "Action": [
           "kms:GetPublicKey"
         ],
         "Resource": [
           "arn:aws:kms:*:*:key/*"
         ]
       },
       {
         "Sid": "SecretsManagerGetSecret",
         "Effect": "Allow",
         "Action": [
           "secretsmanager:GetSecretValue"
         ],
         "Resource": [
           "arn:aws:secretsmanager:*:*:secret:*"
         ]
       },
       {
         "Sid": "LambdaGetConfiguration",
         "Effect": "Allow",
         "Action": [
             "lambda:GetFunctionConfiguration"
         ],
         "Resource": [
             "arn:aws:lambda:*:*:function:*"
         ]
       }
     ]
   }
   

3. Replace wildcard * with the region, account, and resource name information where required.

4. This step is required if KMS is used to encrypt ESA password.

   Add policy entry below. Replace ESA PASSWORD KMS KEY ARN with the value recorded in Option 2: KMS Encrypted Password.

   
     {
       "Sid": "KmsDecryptEsaPassword",
       "Effect": "Allow",
       "Action": [
         "kms:Decrypt"
       ],
       "Resource": [
         "**ESA PASSWORD KMS KEY ARN**"
       ]
     }
   

5. Select Next type in the policy name and Create Policy. Record policy name:

   Policy Name: ___________________

Create Agent Lambda IAM Role

Perform the following steps to create Agent Lambda execution IAM role.

To create agent Lambda IAM role:

1. From AWS IAM console, select Roles > Create Role.

2. Select AWS Service > Lambda > Next.

3. Select the policy created in Create Agent Lambda IAM policy.

4. Proceed to Name, Review and Create.

5. Type the role name, for example, ProtegrityAgentRole and click Confirm.

6. Select Create role.

7. Record the role ARN.

   Agent Lambda IAM Execution Role Name: ___________________

Corporate Firewall Configuration

If an on-premise firewall is used, then the firewall must allow access from the NAT Gateway to an ESA. The firewall must allow access from the NAT Gateway IP to ESA via port 443 and 443.

CloudFormation Installation

Create the Policy Agent in the VPC using the CloudFormation script provided by Protegrity.

1. Access the CloudFormation service.

2. Select the target installation region.

3. Create a stack with new resources.

4. Upload the Policy Agent CloudFormation template (file name: pty_agent_cf.json).

5. Specify the following parameters for Cloud Formation:

   Parameter	Description	Note
   VPC	VPC where the Policy Agent will be hosted	Identify or Create a new VPC
   Subnet	Subnet where the Policy Agent will be hosted	VPC Subnet Configuration
   PolicyAgentSecurityGroupId	Security Group Id, which allows communication between the Policy Agent and the ESA	Identify or Create Security Groups
   LambdaExecutionRoleArn	Agent Lambda IAM execution role ARN allowing access to the S3 bucket, KMS encryption Key, Lambda and Lambda Layer	Create Agent Lambda IAM Role
   ArtifactS3Bucket	S3 bucket name with deployment package for the Policy Agent	Use S3 Bucket name recorded in Create S3 bucket for Installing Artifacts
   CreateCRONJob	Set to True to create a CloudWatch schedule for the agent to run.	Default: False

Policy Agent Lambda Configuration

After the CloudFormation stack is deployed, the Policy Agent Lambda must be configured with parameters recorded in earlier steps. From your AWS Console, navigate to lambda and select the following Lambda.

Protegrity_Agent<STACK_NAME>_

Select Configuration tab and scroll down to the Environment variables section. Select Editand replace all entries with the actual values.

{
  "PTY_ESA_CA_SERVER_CERT":"-----BEGIN CERTIFICATE----- MIIF..."
}

Test Installation

Open the Lambda and configure Test to execute the lambda and specify the default test event. Wait for around 20 seconds for the Lambda to complete. If policy is downloaded successfully, then a success message appears.

Navigate to the AWS_POLICY_S3_BUCKET bucket and verify that the AWS_POLICY_S3_FILENAME file was created.

Troubleshooting

2020-10-06T23:40:54.121Z 2dc84942-b5cc-4be9-aa4c-965f322307e4 Task timed out after 90.09 seconds

{
  "errorMessage": "Timeout! Unable to download policy in 20 seconds.",
  "errorType": "Exception",
  "stackTrace": [...]
}

{
  "errorMessage": "ESA connection error. Failed to download certificates. HTTP response code: 401.",
  "errorType": "ConnectionError",
  "stackTrace": [...]
}

xyz Access Denied: Exception
Traceback (most recent call last):
  … Exception: xyz Access Denied

Secrets Manager Access Denied: Exception
Traceback (most recent call last):
  … Exception: Secrets Manager Access Denied

[ERROR] PolicyAgentException: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

kms:Decrypt
kms:GenerateDatakey

An error occurred (AccessDeniedException) when calling the PublishLayerVersion operation: Your access has been denied by S3,
please make sure your request credentials have permission to GetObject for BUCKET_NAME/FILENAME. S3 Error Code: AccessDenied. S3 Error Message: Access Denied

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VpcRestrictions",
      "Effect": "Deny",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:*Object",
      "Resource": [
        "arn:aws:s3:::<s3_bucket_name>/*",
        "arn:aws:s3:::<s3_bucket_name>"
      ],
      "Condition": {
        "Bool": {
          "aws:ViaAWSService": "false"
        },
        "StringNotEquals": {
          "aws:sourceVpc": "<vpc_id>"
        }
      }
    }
  ]
}

Additional Configuration

Strengthen the KMS IAM policy by granting access only to the required Lambda function(s).

Finalize the IAM policy for the Lambda Execution Role. Ensure to replace wildcard * with the region, account, and resource name information where required.

For example,

"arn:aws:lambda:*:*:function:*" -> "arn:aws:lambda:us-east-1:account:function:function_name"

Policy Agent Schedule

If specified in CloudFormation Installation, the agent installation created a CloudWatch event rule, which checks for policy update on an hourly schedule. This schedule can be altered to the required frequency.

Under CloudWatch > Events > Rules, find Protegrity_Agent_{stack_name}. Click Action > Edit Set the cron expression. A cron expression can easily be defined using CronMaker, a free online tool. Refer to http://www.cronmaker.com.

What’s Next

• Audit Log Forwarder Installation

1.1.3.5 - Audit Log Forwarder Installation

The following sections show steps how to install Audit Log Forwarder component in the AWS Cloud. The Log Forwarder deployment allows for the audit logs generated by Protector to be delivered to ESA for auditing and governance purposes. Log Forwarder component is optional and is not required for the Protector Service to work properly. See Log Forwarding Architecture section in this document for more information. Some of the installation steps are not required for the operation of the software but recommended for establishing a secure environment. C ontact Protegrity for further guidance on configuration alternatives in the Cloud.

ESA Audit Store Configuration

ESA server is required as the recipient of audit logs. Verify the information below to ensure ESA is accessible and configured properly.

1. ESA server running and accessible on TCP port 9200 (Audit Store) or 24284 (td-agent).

2. Audit Store service is configured and running on ESA. Applies when audit logs are output to Audit Store directly or through td-agent. For information related to ESA Audit Store configuration, refer to Audit Store Guide.

3. (Optional) td-agent is configured for external input. For more information related to td-agent configuration, refer to ESA guide Sending logs to an external security information and event management (SIEM).

Certificates on ESA

By default, ESA is configured with self-signed certificates, which can optionally be validated using a self-signed CA certificate supplied in the Log Forwarder configuration. If no CA certificate is provided, the Log Forwarder will skip server certificate validation.

If ESA is configured with publicly signed certificates, this section can be skipped since the forwarder Lambda will use the public CA to validate ESA certificates.

To obtain the self-signed CA certificate from ESA:

1. Download ESA CA certificate from the /etc/ksa/certificates/plug directory of the ESA

2. After certificate is downloaded, open the PEM file in text editor and replace all new lines with escaped new line: \n.

   To escape new lines from command line, use one of the following commands depending on your operating system:

   Linux Bash:

   awk 'NF {printf "%s\\n",$0;}' CA.pem > output.txt
   

   Windows PowerShell:

   (Get-Content '.\CA.pem') -join '\n' | Set-Content 'output.txt'
   

3. Record the certificate content with new lines escaped.

   ESA CA Server Certificate (EsaCaCert): ___________________

   This value will be used to set PtyEsaCaServerCert cloudformation parameter in section Install through CloudFormation

For more information about ESA certificate management refer to Certificate Management Guide in ESA documentation.

AWS VPC Configuration

Log forwarder Lambda function requires network connectivity to ESA, similar to Policy Agent Lambda function. Therefore, it can be hosted in the same VPC as Policy Agent.

Separate VPC can be used, as long as it provides network connectivity to ESA.

VPC Name: ___________________

VPC Subnet Configuration

Log Forwarder can be connected to the same subnet as Policy Agent or separate one as long as it provides connectivity to ESA.

Subnet Name: ___________________

NAT Gateway For ESA Hosted Outside AWS Network

If ESA server is hosted outside of the AWS Cloud network, the VPC configured for Lambda function must ensure additional network configuration is available to allow connectivity with ESA. For instance if ESA has a public IP, the Lambda function VPC must have public subnet with a NAT server to allow routing traffic outside of the AWS network. A Routing Table and Network ACL may need to be configured for outbound access to the ESA as well.

VPC Endpoint Configuration

Log Forwarder Lambda function requires connectivity to Secrets Manager AWS service. If the VPC identified in the steps before has no connectivity to public internet through the NAT Gateway, then the following service endpoint must be configured:

• com.amazonaws.{REGION}.cloudwatch
• com.amazonaws.{REGION}.secretsmanager
• com.amazonaws.{REGION}.kms

Security Group Configuration

Security groups restrict communication between Log Forwarder Lambda function and the ESA appliance. The following rules must be in place for ESA and Log Forwarder Lambda function.

From VPC > Security Groups > Log Forwarder Security Group configuration.

Record the name of Log Forwarder security group name.

Log Forwarder Security Group Id: ___________________

The following port must be open for the ESA. If the ESA is running in the Cloud, then create the following security.

ESA Security Group configuration

Configure ESA Audit Store Credentials

Audit Log Forwarder can optionally authenticate with ESA using certificate-based authentication with a client certificate and certificate key. If used, both the certificate and certificate key will be stored in AWS Secrets Manager.

Download the following certificates from the /etc/ksa/certificates/plug directory of the ESA:

• client.key
• client.pem

After certificates are downloaded, open each PEM file in text editor and replace all new lines with escaped new line: \n. To escape new lines from command line, use one of the commands below depending on your operating system.

Linux Bash:

awk 'NF {printf "%s\\n",$0;}' client.key > private_key.txt
awk 'NF {printf "%s\\n",$0;}' client.pem > public_key.txt

Windows PowerShell:

(Get-Content '.\client.key') -join '\n' | Set-Content 'private_key.pem'
(Get-Content '.\client.pem') -join '\n' | Set-Content 'public_key.pem'

For more information on how to configure client certificate authentication for Audit Store on ESA refer to Audit Store Guide.

To create secret with ESA client certificate/key pair in AWS Secrets Manager.

1. From the AWS Secrets Manager Console, select Store New Secret.

2. Select Other Type of Secrets.

3. Specify the private_key and public_key value pair.

   

4. Select the encryption key or leave default AWS managed key.

5. Specify the Secret Name and record it below.

   ESA Client Certificate/Key Pair Secret Name: ___________________

   This value will be used to set PtyEsaClientCertificatesSecretId cloudformation parameter in section Install through CloudFormation

Create Audit Log Forwarder IAM Execution Policy

This task defines a policy used by the Protegrity Log Forwarder Lambda function to write CloudWatch logs, access the KMS encryption key to decrypt the policy and access Secrets Manager for log forwarder user credentials.

Perform the following steps to create the Lambda execution role and required policies:

1. From the AWS IAM console, select Policies > Create Policy.

2. Select the JSON tab and copy the following sample policy.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "EC2ModifyNetworkInterfaces",
         "Effect": "Allow",
         "Action": [
           "ec2:CreateNetworkInterface",
           "ec2:DescribeNetworkInterfaces",
           "ec2:DeleteNetworkInterface"
         ],
         "Resource": "*"
       },
       {
         "Sid": "CloudWatchWriteLogs",
         "Effect": "Allow",
         "Action": [
           "logs:CreateLogGroup",
           "logs:CreateLogStream",
           "logs:PutLogEvents"
         ],
         "Resource": "*"
       },
       {
         "Sid": "KmsDecrypt",
         "Effect": "Allow",
         "Action": [
           "kms:Decrypt"
         ],
         "Resource": [
           "arn:aws:kms:*:*:key/*"
         ]
       },
       {
           "Sid": "KinesisStreamRead",
           "Effect": "Allow",
           "Action": [
               "kinesis:GetRecords",
               "kinesis:GetShardIterator",
               "kinesis:DescribeStream",
               "kinesis:DescribeStreamSummary",
               "kinesis:ListShards",
               "kinesis:ListStreams"
           ],
           "Resource": "*"
       },
       {
         "Sid": "SecretsManagerGetSecret",
         "Effect": "Allow",
         "Action": [
           "secretsmanager:GetSecretValue"
         ],
         "Resource": [
           "arn:aws:secretsmanager:*:*:secret:*"
         ]
       }
     ]
   }
   

3. For the KMS policy, replace the Resource with the ARN for the KMS key created in a previous step.

4. Select Review policy, type in a policy name, for example, ProtegrityLogForwarderLambdaPolicy and Confirm. Record the policy name:

   LogForwarderLambdaPolicyName:__________________

Create Log Forwarder IAM Role

Perform the following steps to create Log Forwarder execution IAM role.

To create Log Forwarder IAM role:

1. From AWS IAM console, select Roles > Create Role.

2. Select AWS Service > Lambda > Next.

3. Select the policy created in Create Audit Log Forwarder IAM Execution Policy.

4. Proceed to Name, Review and Create.

5. Type the role name, for example, ProtegrityForwarderRole and click Confirm.

6. Record the role ARN.

   Log Forwarder IAM Execution Role Name: ___________________

Installation Artifacts

Audit Log Forwarder installation artifacts are part of the same deployment package as the one used for protect and policy agent services. Follow the steps below to ensure the right artifacts are available for log forwarder installation.

1. Verify that the Protegrity deployment package is available on your local system, if not, you can download it from the Protegrity portal.

   Note

   If you maintain multiple Protegrity Cloud Protectors, make sure that the deployment package downloaded for Audit Log Forwarder is the same as the one used for Protect service installation.

2. Extract the pty_log_forwarder_cf.json cloud formation file from the deployment package.

3. Check the S3 deployment bucket identified in section Create S3 bucket for Installing Artifacts. Make sure that all Protegrity deployment zip files are uploaded to the S3 bucket.

Install through CloudFormation

The following steps describe the deployment of the Audit Log Forwarder AWS cloud components.

1. Access CloudFormation and select the target AWS Region in the console.

2. Click Create Stack and choose With new resources.

3. Specify the template.

4. Select Upload a template file.

5. Upload the Protegrity-provided CloudFormation template called pty_log_forwarder_cf.json and click Next.

6. Specify the stack details. Enter a stack name.

   Note

   The stack name will be appended to all the services created by the template.

7. Enter the required parameters. All the values were generated in the pre-configuration steps.

8. Click Next with defaults to complete CloudFormation.

9. After CloudFormation is completed, select the Outputstab in the stack.

10. Record the following values

    KinesisLogStreamArn: ________________________________

Add Kinesis Put Record permission to the Protect Function IAM Role

1. Login to the AWS account that hosts the Protect Lambda Function.

2. Search for Protect Lambda Function IAM Execution Role Name created in Create Protect Lambda IAM role.

3. Under Permissions policies, select Add Permissions > Create inline policy.

4. In Specify permissions view, switch to JSON.

5. Copy the policy json from below replacing the placeholder value indicated in the following snippet as <Audit Log Kinesis Stream ARN> with the value recorded in the previous step.

   {
   	"Version": "2012-10-17",
   	"Statement": [
   		{
   			"Sid": "KinesisPutRecords",
   			"Effect": "Allow",
   			"Action": "kinesis:PutRecords",
   			"Resource": "<Audit Log Kinesis Stream ARN>"
   		}
   	]
   }
   

6. When you are finished, choose Next.

7. On the Review and create page, type a Name, then choose Create policy.

Test Log Forwarder Installation

Testing in this section validates the connectivity between Log Forwarder and ESA. The sample policy included with the initial installation and test event below are not based on your ESA policy. Any logs forwarded to ESA which are not signed with a policy generated by your ESA will not be added to the audit store.

Install Log Forwarder and configure according to previous sections. Log Forwarder configuration MinLogLevel must be at least info level.

1. Navigate to the log forwarder lambda function.

2. Select the Test tab.

3. Copy the json test event into the Event JSON pane.

   {
       "Records": [
           {
               "kinesis": {
                   "kinesisSchemaVersion": "1.0",
                   "partitionKey": "041e96d78c778677ce43f50076a8ae3e",
                   "sequenceNumber": "49620336010289430959432297775520367512250709822916263938",
                   "data": "eyJpbmdlc3RfdGltZV91dGMiOiIyMDI2LTAzLTI3VDEzOjIzOjEyLjkwNFoiLCJhZGRpdGlvbmFsX2luZm8iOnsiZGVzY3JpcHRpb24iOiJEYXRhIHVucHJvdGVjdCBvcGVyYXRpb24gd2FzIHN1Y2Nlc3NmdWwuIn0sImNsaWVudCI6e30sImNudCI6NCwiY29ycmVsYXRpb25pZCI6InJzLXF1ZXJ5LWlkOjEyMzQiLCJsZXZlbCI6IlNVQ0NFU1MiLCJsb2d0eXBlIjoiUHJvdGVjdGlvbiIsIm9yaWdpbiI6eyJob3N0bmFtZSI6IlBSTy1VUy1QRjNSSEdGOSIsImlwIjoiMTAuMTAuMTAuMTAiLCJ0aW1lX3V0YyI6MTcxMTU1OTEwMH0sInByb2Nlc3MiOnsiaWQiOjJ9LCJwcm90ZWN0aW9uIjp7ImF1ZGl0X2NvZGUiOjgsImRhdGFlbGVtZW50IjoiYWxwaGEiLCJkYXRhc3RvcmUiOiJTQU1QTEVfUE9MSUNZIiwibWFza19zZXR0aW5nIjoiIiwib2xkX2RhdGFlbGVtZW50IjpudWxsLCJvcGVyYXRpb24iOiJVbnByb3RlY3QiLCJwb2xpY3lfdXNlciI6IlBUWURFViJ9LCJwcm90ZWN0b3IiOnsiY29yZV92ZXJzaW9uIjoiMS4yLjErNTUuZzU5MGZlLkhFQUQiLCJmYW1pbHkiOiJjcCIsInBjY192ZXJzaW9uIjoiMy40LjAuMTQiLCJ2ZW5kb3IiOiJyZWRzaGlmdCIsInZlcnNpb24iOiIwLjAuMS1kZXYifSwic2lnbmF0dXJlIjp7ImNoZWNrc3VtIjoiN0VCMkEzN0FDNzQ5MDM1NjQwMzBBNUUxNENCMTA5QzE0OEJGODYwRjE3NEVCMjMxMTAyMEI3RkE1QTY4MjdGQyIsImtleV9pZCI6ImM0MjQ0MzZhLTExMmYtNGMzZi1iMmRiLTEwYmFhOGI1NjJhNyJ9fQ==",
                   "approximateArrivalTimestamp": 1626878559.213
               },
               "eventSource": "aws:kinesis",
               "eventVersion": "1.0",
               "eventID": "shardId-000000000000:49620336010289430959432297775520367512250709822916261234",
               "eventName": "aws:kinesis:record",
               "invokeIdentityArn": "arn:aws:iam::555555555555:role/service-role/TestRole",
               "awsRegion": "us-east-1",
               "eventSourceARN": "arn:aws:kinesis:us-east-1:555555555555:stream/CloudProtectEventStream"
           }
       ]
   }
   

   Note

   The data payload is the base64 encoded audit log. See Audit Logging for detail on audit log contents.

4. Select Test to execute the test event.

5. Test is successful if the Log Output of test results contains the following log:

   [INFO] [kinesis-log-aggregation-format.cpp:77] Aggregated 1 records into 0 aggregated, 1 forwarded and 0 failed records
   

   If the log is not present, please consult the Troubleshooting section for common errors and solutions.

Update Protector With Kinesis Log Stream

In this section, Kinesis log stream ARN will be provided to the Protect Function installation.

1. Navigate to the Protector CloudFormation stack created in the protector installation section.

2. Select Update.

3. Choose Use existing template > Next.

4. Set parameter KinesisLogStreamArn to the output value recorded in Install through CloudFormation.

5. Proceed with Next and Submit the changes.

6. Continue to the next section once stack status indicates UPDATE_COMPLETE.

Update Policy Agent With Log Forwarder Function Target

Log Forwarder Lambda function requires a policy layer which is in sync with the Protegrity Protector. This section will describe the steps to update the policy agent to include updating Log Forwarder Lambda function.

1. Navigate to the Policy Agent Function created in Policy Agent Installation

2. Select Configuration > Environment variables > Edit

3. Edit the value for environment variable AWS_PROTECT_FN_NAME to include the log forwarder function name/arn in the comma separated list of Lambda functions.

4. Save the changes and continue when update completes

5. Navigate to Test tab

6. Add an event {} and select Test to run the Policy Agent function

7. Verify Log forwarder function was updated to use the policy layer by inspecting the log output. Logs should include the following:

   
   [INFO] 2024-07-09 18:58:04,793.793Z 622d374b-1f73-4123-9a38-abc61973adef iap_agent.policy_deployer:Updating lambda [Protegrity_LogForwarder_<stack ID>] to use layer version [arn:aws:lambda:<aws region>:<aws account number>:layer:Protegrity_Layer_<layer name>:<layer version>]
   

Test Full Log Forwarder Installation

Install and configure Protegrity Agent, Protector, and Log Forwarder components.

1. Send a protect operation to the protector using a data element or user which will result in audit log generation

2. Navigate to the CloudWatch log group for the Protect function

3. Select the log stream for the test operation and scroll to the latest logs

4. Expect to see a log similar to the below:

   
   [2024-07-09T19:28:23.158] [INFO] [kinesis-external-sink.cpp:51] Sending 2 logs to Kinesis ...
   [2024-07-09T19:28:23.218] [INFO] [aws-utils.cpp:206] Kinesis send time: 0.060s
   

5. Navigate to the CloudWatch log group for the Log Forwarder function

6. Expect to see a new log stream - it may take several minutes for the stream to start

7. Select the new stream and scroll to the most recent logs in the stream

8. Expect to see a log similar to the below:

   
   [2024-07-09T19:32:31.648] [INFO] [kinesis-log-aggregation-format.cpp:77] Aggregated 1 records into 0 aggregated, 1 forwarded and 0 failed records
   

Troubleshooting

[SEVERE] User: <arn> is not authorized to perform: secretsmanager:
GetSecretValue on resource: <secret name> because no identity-based
policy allows the secretsmanager:GetSecretValue action

{
  "error_msg": "Failed to decrypt the policy. Please verify
    that the function has access to the key service and the key.",
  "success": false
}

The provided execution role does not have permissions to call
[CreateNetworkInterface|DescribeNetworkInterfaces|DeleteNetworkInterface]
on EC2 (Service: Lambda, Status Code: 400, Request ID: <request id>)"
(RequestToken: <request token>, HandlerErrorCode: InvalidRequest)

[SEVERE] Kinesis stream client returned 400 error with error message:
User: <function arn> is not authorized to perform: kinesis:PutRecords
on resource: <kinesis stream arn> because no identity-based policy
allows the kinesis:PutRecords action

[error] [tls] <error message>

1.1.3.6 -

Cheat Sheet Recommendation

AWS Account ID: ___________________

AWS Region (AwsRegion): ___________________

S3 Bucket name (ArtifactS3Bucket): ___________________

KMS Key ARN (AWS_KMS_KEY_ID): ___________________

ProtectLambdaPolicyName: __________________

Role ARN (LambdaExecutionRoleArn): ___________________

ApiGatewayId: ________________________________

ProtectFunctionName: __________________________

ProtectLayerName: _____________________________

ESA IP address: ___________________

VPC name: ___________________

Subnet name: ___________________

Policy Agent Security Group Id: ___________________

ESA Credentials Secret Name: ___________________

Policy Name: ___________________

Agent Lambda IAM Execution Role Name: ___________________

1.1.3.7 -

Prerequisites

1.1.3.8 -

AWS Services

The following table describes the AWS services that may be a part of your Protegrity installation.

1.1.3.9 -

Required Skills and Abilities

1.1.4 - REST API

The following sections describe the key concepts in understanding the REST API.

Base URL:

https://{ApiGatewayId}.execute-api.{Region}.amazonaws.com/pty/v1

Once Protegrity Serverless REST API is installed, you can export the OpenAPI documentation file from API Gateway Export API, located in the AWS API Gateway Control Service.

https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-export-api.html

For testing the REST API, we recommend using tools, such as, Postman.

1.1.4.1 - Authorization

Policy Users

Protegrity Policy roles defines the unique data access privileges for every member. The Protegrity Lambda protects the data with the username sent in either the JWT-formatted authorization header or the request body.

The lambda behavior can be set in the Lambda environment variables as described in Protect Lambda Configuration

JWT Verification

To ensure the integrity of the user, the lambda protect can verify the JWT.

1. From your AWS console, navigate to lambda and select the following Lambda: Protegrity_Protect_RESTAPI_<STACK_NAME>
2. Scroll down to the Environment variables section, select Edit to replace the entries.

-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC4fkg/JYyN3Skr6RYLiAd/Yhl0
2TE3/HzHSNPnCaRdUakGp9og7oXBMcoadFDjnoSq1sz+gUHnpoO7s2fwkD5Q4OnC
BGD3oKP2A4PlOOWD2B2cVmMqX/vf1nAA/343496jsbfgkh1Q7LTzR0IXfdii0o1U
CbvrVCuaBoyiv4TxWQIDAQAB
-----END PUBLIC KEY-----

LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tL
S0KTUlHZk1BMEdDU3FHU0liM0RRRUJBUV
VBQTRHTkFEQ0JpUUtCZ1FDNGZrZy9KWXl
OM1NrcjZSWUxpQWQvWWhsMAoyVEUzL0h6
SFNOUG5DYVJkVWFrR3A5b2c3b1hCTWNvY
WRGRGpub1NxMXN6K2dVSG5wb083czJmd2
tENVE0T25DCkJHRDNvS1AyQTRQbE9PV0Q
yQjJjVm1NcVgvdmYxbkFBLzM0MzQ5Nmpz
YmZna2gxUTdMVHpSMElYZmRpaTBvMVUKQ
2J2clZDdWFCb3lpdjRUeFdRSURBUUFCCi
0tLS0tRU5EIFBVQkxJQyBLRVktLS0tLQ==

1.1.4.2 - HTTP Status Codes

The following table explains the different HTTP Status Codes with their corresponding response.

{"results":["<string>","<string>"],
 "success": true, "encoding":[hex|utf8]}

{
    "error_msg": "<string>",
    "success": false
}

Authorization header was not found.

{
    "error_msg": "token expired",
    "success": false
}

{
    "error_msg": "invalid signature",
    "success": false
}
{
    "error_msg": " unknown algorithm error",
    "success": false
}

{
    "error_msg": "Invalid json array in jwt_user_claim",
    "success": false
}

{    "error_msg": "jwt secret is missing ",
    "success": false
}

1.1.4.3 - Payload Encoding

Payload Encoding

The following encoding formats are supported in the REST API.

For every encoding, the resultant protected data is returned in the same encoding. For example, if request is hex-encoded, the response is also hex-encoded.

For more information about the encoding formats, refer to the Protection Methods Reference.

1.1.4.4 - TLS Mutual Authentication

By default, AWS API Gateway supports HTTPS endpoint and doesn’t allow HTTP protocol.

For an additional layer of security, you can configure AWS API Gateway TLS Mutual Authentication.

AWS API Gateway will ensure that the clients with a valid certificate only can call the REST API. Protegrity recommends to setup TLS Mutual Authentication in AWS API Gateway. For more information on how to set AWS API Gateway Mutual TLS go to the following link.

https://docs.aws.amazon.com/apigateway/latest/developerguide/rest-api-mutual-tls.html

1.1.4.5 - v1 Specification

AWS Lambda service limits maximum size of payload to 6 MB. Client applications of Protegrity Cloud API must ensure their payload size is within this limit. This applies to all types of requests described below.

Request

AWS Lambda service limits maximum size of payload to 6 MB. Client applications of Protegrity Cloud API must ensure their payload size is within this limit. This applies to all types of requests described below.

Performs a policy operation such as protect, unprotect, or reprotect.

• URI

  /v1/protect or /v1/unprotect or /v1/reprotect

• Method

  POST

• Parameters

  data: Input data to the policy operation.

  data_element: Data element to use for the policy operation.

  encoding: Optional, encoding of the data. One of: base64, base64_mime, base64_pem, base64_url, hex, or utf8. Defaults to hex for binary data elements, otherwise defaults to utf8.

  external_iv: Optional, external initialization vector.

  old_data_element: (reprotect) Data element for unprotecting the input.

  old_external_iv: (reprotect) Optional, external initialization vector for the input.

  query_id: Optional, identifier for the request.

  user: User performing the operation.

• Result

  Returns the output of the policy operation.

Example 1 - without external IV

{
  "encoding": "utf8",
  "query_id": "1",
  "user": "user1",
  "data_element": "Alphanum",
  "data":[<data1>,<data2>]
}

Example 2 - with external IV

{
  "encoding": "utf8",
  "query_id": "1",
  "user": "user1",
  "data_element": "Alphanum",
  "external_iv": "abc123",
  "data":[<data1>,<data2>]
}

Example 3 - reprotect

{
  "encoding": "utf8",
  "query_id": "1",
  "user": "user1",
  "data_element": "deName",
  "old_data_element": "Alphanum",
  "data":[<data1>,<data2>]
}

Performs policy operations using different data elements for each data set.

• URI

  /v1/protect or /v1/unprotect or /v1/reprotect

• Method

  POST

• Parameters

  encoding: Optional, encoding of the data. One of: base64, base64_mime, base64_pem, base64_url, hex, or utf8. Defaults to hex for binary data elements, otherwise defaults to utf8.

  query_id: Optional, prefix for the request identifier.

  user: User performing the operation.

  arguments[].data: Input data to the policy operation.

  arguments[].data_element: Data element to use for the policy operation.

  arguments[].external_iv: Optional, external initialization vector.

  arguments[].id: Optional, request identifier.

  arguments[].old_data_element: (reprotect) Data element for unprotecting the input.

  arguments[].old_external_iv: (reprotect) Optional, external initialization vector for the input.

Example 4 - multiple data elements support payload

{
  "encoding": "utf8",
  "query_id": "query1234", 
  "user": <policy_user>, 
  "arguments": [
    {
      "id": "1", 
      "external_iv": "<external iv>", 
      "data_element": "<data element name>", 
      "data":["<data1>","<data2>"]
    }, 
    {
      "data_element": <data element name>, 
      "data":["<data1>","<data2>"]
    }
  ]
}

• External IV

  The External IV feature provides an additional level of security by allowing different tokenized results across protectors for the same input data and token element, depending on the External IV set on each protector. External IVs must adhere to certain guardrails and not all data elements support External IV. To read more about the Tokenization model with External IV, refer to the External Initialization Vector (IV) chapter in the Protection Methods Reference.

Response

responsePayloadV1:
type: object
  properties:
    success:
      type: bool
    error_msg:
      type: string
      description: When success is false, error_msg is included
    results:
      type: array
      items:
        type: string

Example success:

{
  "encoding": "utf8", 
  "results":["str1","str2"], 
  "success": true
}

If the request was successful, the success flag will always be true.

Example failure:

{
  "error_msg": "token expired", 
  "success": false
}

If the request fails, the success flag will always be false.

Multi Data Elements Support Payload

responsePayloadV2:
type: object
  properties:
    success:
      type: boolean
    results:
      type: array
        items:
          type: object
          properties:
            success:
              type: bool
            error_msg:
              type: string
              description: When success is false, error_msg is included
            id:
              type: string
              description: When id is sent in the request payload, id is included in the response
            results:
              type: array
              items:
                type: string

Example success:

{
  "encoding": "utf8", 
  "results":[
    {
      "encoding": "utf8", 
      "results":["str1","str2"], 
      "success": true
    }, 
    {
      "encoding": "utf8", 
      "results":["str1","str2"], 
      "success": true
    }
  ], 
  "success": true
}

If the all the subrequests were successful, the success flag will be true.

Example failure:

{
  "results": [
    {
      "error_msg": 
      "Protect failed. Data element not found. Refer to audit log for details", 
      "success": false
    }, 
    {
      "encoding": "utf8", 
      "results":["str1","str2"], 
      "success": true
    }
  ], 
  "success": false
}

It is possible to have a mix of successful and unsuccessful results. In this case, the global success flag will be false.

1.1.4.6 - Legacy Specification

Protegrity has multiple products with REST API capabilities, such as Protection Server (out of support), DSG, and the latest product - IAP REST. Each one has its use case. To help you move to cloud-native implementation, Cloud Product REST API supports legacy payload.

• Request
• Response

Legacy documentation can be downloaded from AWS console, API, Models, prefix Legacy.

Request

Performs a policy operation such as protect, unprotect, or reprotect.

• Method

  POST

• Parameters

  dataelementname: (protect/unprotect) Data element to use for the policy operation.

  externaliv: (protect/unprotect) Optional, external initialization vector.

  newdataelementname: (reprotect) Data element to use for the output.

  newexternaliv: (reprotect) Optional, external initialization vector for the output.

  olddataelementname: (reprotect) Data element to use for the input.

  oldexternaliv: (reprotect) Optional, external initialization vector for the input.

  policyusername: User performing the operation.

  bulk.id: Optional, identifier for the request.

  bulk.data[].content: Input data to the policy operation.

• Result

  Returns the output of the policy operation.

Example 1 - protect without external IV

{
  "protect": {
    "policyusername": "user1",
    "dataelementname": "Alphanum",
    "bulk": {
      "id": "1",
      "data": [
        {
          "content": <Data encoded in base64>
        }
      ]
    }
  }
}

Example 2 - protect with external IV

{
  "protect": {
    "policyusername": "user1",
    "dataelementname": "Alphanum",
    "externaliv": "abc123",
    "bulk": {
      "id": "1",
      "data": [
        {
          "content": <Data encoded in base64>
        }
      ]
    }
  }
}

Example 3 - unprotect

{
  "unprotect": {
    "policyusername": "user1",
    "dataelementname": "Alphanum",
    "bulk": {
      "id": "1",
      "data": [
        {
          "content": <Data encoded in base64>
        }
      ]
    }
  }
}

Example 4 - reprotect

{
  "reprotect": {
    "policyusername": "user1",
    "newdataelementname": "deName",
    "olddataelementname": "Alphanum",
    "bulk": {
      "id": "1",
      "data": [
        {
          "content": <Data encoded in base64>
        }
      ]
    }
  }
}

Response

Example:

{"protect":{"bulk":{"returntype":"success","data":[{"returntype":"success","message":"Data
 protection was successful.","content":"RGZBUFR4ODAzejFwNjQ5TWg0TEFpcFNqbA=="},{"returntype":"success",
"message":"Data protection was successful.","content":"aHNnVVB5QWFDYw=="}]}}}

1.1.5 - Performance

Performance Considerations

The following factors may cause variation in real performance versus benchmarks:

1. Cold startup: The Lambda spends additional time on the initial invocation to decrypt and load the policy into memory. This time can vary depending on the policy size. Once the Lambda is initialized, subsequent “warm executions” should process quickly.
2. Noisy neighbors: There are many multi-tenant components in the Cloud. The same request may differ by 50% between runs regardless of Protegrity. A single execution may not be the best predictor of average performance.
3. Size of policy: The size of the policy impacts cold start performance. Larger policies take more time to initialize.
4. Lambda memory: AWS provides more virtual cores based on the memory configuration. The initial configuration of 1728MB provides a good tradeoff between performance and cost with the benchmarked policy. Memory can be increased to optimize for your individual case.
5. Number of protect, unprotect and reprotect security operations.
6. Lambda Concurrent and Burst quotas: AWS limits the number of concurrent executions and how quickly lambda can scale to meet demand. This is discussed in an upcoming section of the document.
7. Size of data element. Operations on larger text may take more time.
8. API Gateway Authentication. API Gateway support different authorizers and authentication method, which may differ.

Lambda Tuning

AWS maintains quotas for Lambda concurrent execution. Two of these quotas impact concurrency and compete with other Lambdas in the same account and region.

The concurrent executions quota cap is the maximum number of Lambda instances that can serve requests for an account and region. The default AWS quota may be inadequate based on peak concurrency based on the table in the previous section. This quota can be increased with an AWS support ticket.

The Burst concurrency quota limits the rate at which Lambda will scale to accommodate demand. This quota is also per account and region. The burst quota cannot be adjusted. AWS will quickly scale until the burst limit is reached. After the burst limit is reached, functions will scale at a reduced rate per minute (e.g. 500). If no Lambda instances can serve a request, the request will fail with a 429 Too Many Requests response.

The burst limit is a fixed value and varies significantly by AWS region. The highest burst (3,000) is currently available in the following regions: US West (Oregon), US East (N.Virginia), and Europe (Ireland). Other regions can burst between 500 and 1,000. It is recommended to select an AWS region with the highest burst limits.

API Gateway Tuning

AWS maintains a Throttle quota for the API Gateway. By default, API Gateway limits concurrent requests to 10,000 requests per second and throttles subsequent traffic with a 429 Too Many Requests error response. This quota applies across all APIs in an account and region.

The API Gateway default quota may need to be increased based on the Concurrency table. Keep in mind that hitting quota limits effectively throttles any other API services in the region.

The API Gateway also limits burst. Burst is the maximum number of concurrent requests that API Gateway can fulfill at any instant without returning 429 Too Many Requests error responses. This limit can be increased by AWS, but is not adjustable.

Enable CloudWatch metrics within the API Gateway to monitor max concurrency and investigate throttling errors. See the Concurrency Troubleshooting section on interpreting CloudWatch metrics.

Quotas adjustments are applied for region and account. Throttling is also enabled by default in the API Gateway stage used by the Protegrity Lambda function. The stage configuration throttling must be adjusted if the quota is modified. Stage throttling is shown in the following image.

For example, the REST Client makes over 20,000 requests/sec to execute the given query. Using API Gateway’s default settings, the requests exceeding 10,000 requests/sec will be throttled. Therefore, this query may fail intermittently due to a high number of throttling errors.

Concurrency Troubleshooting

Hitting up against quota limits may indicate that quota adjustments are required. Exceeding quota limits may cause a client query to fail or reduce performance. In the worst case, significant throttling can impact the performance of all your API Gateway or Lambda services in the region.

CloudWatch Metrics can be manually enabled on the API Gateway to reveal if quotas are being reached. Metrics aggregate errors into two buckets that are 4xx and 5xx. CloudWatch logs can be used to access the actual error code. The following table describes how to interpret the CloudWatch metrics.

The following screenshot shows an example of searching CloudWatch Logs using Log Insights:

Cold-Start Performance

Cold-start vs warm execution refers to the state of the Lambda when a request is received. A cold-start undergoes additional initialization, such as, loading the security policy. Warm execution applies to all subsequent requests served by the Lambda.

The following table shows an example how these states impact latency and performance:

Log Forwarder Performance

Log forwarder architecture is optimized to minimize the amount of connections and reduce the overall network bandwidth required to send audit logs to ESA. This is achieved with batching and aggregation taking place on two levels. The first level is in protect function instances, where audit logs from consecutive requests to an instance are batched and aggregated. The second level of batching takes place in Amazon Kinesis Stream where log records from different protect function instances are additionally batched and sent to log forwarder function where they are aggregated. This section shows how to configure the deployment to accommodate different patterns of anticipated audit log stream. It also shows how to monitor deployment resources to detect problems before audit records are lost.

• Protector Cloud Formation Parameters

  • AuditLogFlushInterval: Determines the minimum amount of time required for the audit log to be sent to Amazon Kinesis. Changing flush interval may affect the level of aggregation, which in turn may result in different number of connections and different data rates to Amazon Kinesis. Default value is 30 seconds.

    Increasing the flush interval may result in higher aggregation of audit logs, in fewer connections to Amazon Kinesis, in higher latency of audit logs arriving to ESA and in higher data throughput.

    Lowering the flush interval may result in lower aggregation of audit logs, in more connections to Amazon Kinesis, in lower latency of audit logs arriving to ESA and in lower data throughput.

    It is not recommended to reduce the flush interval from default value in production environment as it may overload the Amazon Kinesis service. However, it may be beneficial to reduce flush interval during testing to make audit records appear on ESA faster.

      

• Log Forwarder Cloud Formation Parameters

  • Amazon KinesisLogStreamShardCount: The number of shards represents the level of parallel streams in the Amazon Kinesis and it is proportional to the throughput capacity of the stream. If the number of shards is too low and the volume of audit logs is too high, Amazon Kinesis service may be overloaded and some audit records sent from protect function may be lost.

    Default value is 10, however you are advised to test with a production-like load to determine whether this is sufficient or not.

  • Amazon KinesisLogStreamRetentionPeriodHours: The time for the audit records to be retained in Amazon Kinesis log stream in cases where log forwarder function is unable to read records from the Kinesis stream or send records to ESA, for example due to a connectivity outage. Amazon Kinesis will retain failed audit records and retry periodically until connectivity with ESA is restored or retention period expires.

    Default value is 24 hours, however you are advised to review this value to align it with your Recovery Time Objective and Recovery Point Objective SLAs.

      

• Monitoring Log Forwarder Resources

  • Amazon Kinesis Stream Metrics: Any positive value in Amazon Kinesis PutRecords throttled records metric indicates that audit logs rate from protect function is too high. The recommended action is to increase the Amazon KinesisLogStreamShardCount or optionally increase the AuditLogFlushInterval.

  • Log Forwarder Function CloudWatch Logs: If log forwarder function is unable to send logs to ESA, it will log the following message:

    [SEVERE] Dropped records: x.
    

    Note

    When the error message above occurs, the dropped audit records will be preserved in the Amazon Kinesis data stream and retried again according to Amazon Kinesis retry schedule. Records will be retried until Amazon KinesisLogStreamRetentionPeriodHours expires.

  • Protect Function CloudWatch Logs: If protect function is unable to send logs to Amazon Kinesis, it will log the following message:

    [SEVERE] Amazon Kinesis error, retrying in x ms (retry: y/z) ..."
    

    Any dropped audit log records will be reported with the following log message:

    [SEVERE] Failed to send x/y audit logs to Amazon Kinesis.
    

1.1.6 - Audit Logging

Audit Logging

Audit records and application logs stream to Amazon CloudWatch Logs or optionally be sent to ESA. Cloud Protect uses a JSON format for audit records that is described in the following sections.

You can analyze and alert on audit records using Protegrity ESA or Amazon CloudWatch. Third-party solutions may be used if they are supported by Amazon Cloudwatch or AWS Lambda logging extensions. For more information about forwarding your audit records to ESA, contact Protegrity. For more information about Amazon CloudWatch, refer to the Amazon CloudWatch User Guide.

For more information about audit records, refer to the Protegrity Analytics Guide.

Audit record fields

The audit record format has been altered in version 3.1 of the protector to provide more information.

Example Audit Records

The following are sample audit messages:

Protect Success:

{
      "additional_info": {
        "deployment_id": "Protegrity-Protect-function-deployment-id",
        "description": "Data protect operation was successful.",
        "query_id": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
        "request_id": "8476a536-e9f4-11e8-9739-2dfe598c3fcd"
      },
      "cnt": 4000,
      "correlationid": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
      "logtype": "Protection",
      "level": "SUCESS",
      "origin": {
        "hostname": "localhost",
        "ip": "127.0.0.1",
        "time_utc": 1635363966
      },
      "protection": {
        "dataelement": "deAddress",
        "operation": "Protect",
        "audit_code": 6,
        "datastore": "SAMPLE_POLICY",
        "policy_user": "test_user"
      },
      process":{
        "name":"protect",
        "id":"13",
        "module":"coreprovider",
        "thread_id":"573580544",
        "user":"sbx_user1051",
        "platform":"\"Linux_x64\"",
        "version":"UNKNOWN"
      },
      "client": {
        "ip":"169.254.62.117"
      },
      "protector": {
        "family": "cp",
        "version": "4.0.0.102",
        "vendor": "aws.snowflake",
        "datastore":"SAMPLE_POLICY",
        "pcc_version": "4.0.0.9",
        "core_version": "2.1.4+0.g93016.2.1",
        "lambda_version":"4.0.1"
      },
      "signature": {
        "key_id": "95f5a194-b0a4-4351-a",
        "checksum": "B324AF7C56944D91C47847A77C0367C594C0B948E7E75654B889571BD4F60A71"
      }
    }

User permission denied:

{
      "additional_info": {
        "deployment_id": "Protegrity-Protect-function-deployment-id",
        "description": "The user does not have the appropriate permissions to perform the requested operation.",
        "query_id": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
        "request_id": "8476a536-e9f4-11e8-9739-2dfe598c3fcd"
      },
      "cnt": 4000,
      "correlationid": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
      "logtype": "Protection",
      "level": "ERROR",
      "origin": {
        "hostname": "localhost",
        "ip": "127.0.0.1",
        "time_utc": 1635363966
      },
      "protection": {
        "dataelement": "deAddress",
        "operation": "Protect",
        "audit_code": 3,
        "datastore": "SAMPLE_POLICY",
        "policy_user": "test_user"
      },
      process":{
        "name":"protect",
        "id":"13",
        "module":"coreprovider",
        "thread_id":"573580544",
        "user":"sbx_user1051",
        "platform":"\"Linux_x64\"",
        "version":"UNKNOWN"
      },
      "client": {
        "ip":"169.254.62.117"
      },
      "protector": {
        "family": "cp",
        "version": "4.0.0.102",
        "vendor": "aws.snowflake",
        "datastore":"SAMPLE_POLICY",
        "pcc_version": "4.0.0.9",
        "core_version": "2.1.4+0.g93016.2.1",
        "lambda_version":"4.0.1"
      },
      "signature": {
        "key_id": "95f5a194-b0a4-4351-a",
        "checksum": "A216797C56944D91C47847A77C0367C594C0B948E7E75654B889571BD4F60A71"
      }
    }

Data element not found:

{
      "additional_info": {
        "deployment_id": "Protegrity-Protect-function-deployment-id",
        "description": "The data element could not be found in the policy.",
        "query_id": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
        "request_id": "8476a536-e9f4-11e8-9739-2dfe598c3fcd"
      },
      "cnt": 4000,
      "correlationid": "sf-query-id:01978dbc-0582-d7e4-0000-002a3603a20d",
      "logtype": "Protection",
      "level": "ERROR",
      "origin": {
        "hostname": "localhost",
        "ip": "127.0.0.1",
        "time_utc": 1635363966
      },
      "protection": {
        "dataelement": "deAddress",
        "operation": "Protect",
        "audit_code": 2,
        "datastore": "SAMPLE_POLICY",
        "policy_user": "test_user"
      },
      process":{
        "name":"protect",
        "id":"13",
        "module":"coreprovider",
        "thread_id":"573580544",
        "user":"sbx_user1051",
        "platform":"\"Linux_x64\"",
        "version":"UNKNOWN"
      },
      "client": {
        "ip":"169.254.62.117"
      },
      "protector": {
        "family": "cp",
        "version": "4.0.0.102",
        "vendor": "aws.snowflake",
        "datastore":"SAMPLE_POLICY",
        "pcc_version": "4.0.0.9",
        "core_version": "2.1.4+0.g93016.2.1",
        "lambda_version":"4.0.1"
      },
      "signature": {
        "key_id": "95f5a194-b0a4-4351-a",
        "checksum": "AF09217C56944D91C47847A77C0367C594C0B948E7E75654B889571BD4F60A71"
      }
    }

1.1.7 - No Access Behavior

No Access Behavior

The security policy maintains a No Access Operation, configured in an ESA, which determines the response for unauthorized unprotect requests.

The following table describes the result returned in the response for the various no access unprotect permissions.

1.1.8 - Known Limitations

Known Limitations

Known product limitations:

• FPE is supported only for ASCII values.

1.1.9 - Upgrading To The Latest Version

Download the Latest Version

You can download the latest version of the deployment package from https://my.protegrity.com. Navigate to Data Protection > Cloud Protect to download the latest version.

After downloading the deployment package from the Protegrity Portal, unzip the package to extract the artifact files. In the AWS Console, navigate to the S3 bucket that was previously created to upload deployment artifacts (see: Create S3 bucket for Installing Artifacts).

Upload the following artifacts to the S3 bucket:

• Snowflake
• Redshift
• Cloud API
• Athena

If the release version matches your existing deployment, you don’t need to upload it again. Save the following artifacts on your local system so that you have them available during the next steps:

• Snowflake
• Redshift
• Cloud API
• Athena

Perform the following steps to upgrade the Agent Lambda and Protect Lambda separately.

Disable Protegrity Agent Function CloudWatch Event Rule

Cloud Watch Event Rule is used to periodically run Protegrity Agent Function to synchronize policy from ESA. This functionality is optional when deploying Protegrity Serverless Solution. If the Event Rule is enabled, it must be disabled temporarily for the time of the upgrade process.

Follow the steps below to determine if your deployment uses Event Rule and disable it.

1. Go to AWS Cloud Formation and select existing Protegrity deployment stack.

2. Select Resources tab from the top portion of the screen.

3. Check if there is a resource with ScheduledRule LogicalID. If there is no such resource you can skip to Upgrading Policy Agent Lambda section. If the scheduled rule is there, continue with the next steps in this section.

4. Click on the Physical ID link in the ScheduledRule row. The link opens Policy Agent Event Rule configuration.

5. Select Disable from the top-right portion of the screen. This will disable the rule. You will re-enable it after the upgrade process is complete.

Upgrading Policy Agent Lambda

1. Go to AWS Lambda console and select existing Protegrity Agent Lambda.

2. Click Actions in top right portion of the screen. Select Publish new version. Click Publish. The version of Agent Lambda you just created will serve as restore point in the case you needed to rollback the upgrade.

3. Go to Lambda Configuration > Environment variables.

4. Record environment variables values. You will use them later to configure upgraded Lambda Function. You can use the aws cli command below to save the function variables into the local json file:

   aws lambda get-function-configuration --function-name \
   arn:aws:lambda:<aws_region>:<aws_account>:function:<function_name> \
   --query Environment > <function_name>_env_config.json
   

5. Go to AWS Cloud Formation and select existing Protegrity Agent deployment stack.

6. Select Update. Check Replace current template > Upload a template file.

7. Upload pty_agent_cf.json file and select Next.

8. Click Next until Review window and then select Update stack.

9. Wait for the Cloud Formation to complete.

10. Navigate back to Agent Lambda Function.

11. Go to Configuration > Environment variables. Replace placeholder values with values recorded in previous step. Alternatively, you can run the following aws cli command to update function configuration using json file saved in the previous steps:

    aws lambda update-function-configuration --function-name \
    arn:aws:lambda:<aws_region>:<aws_account>:function:<function_name> \
    --environment file://./<function_name>_env_config.json
    

    Note

    If your current agent installation version is lower than 3.0.12, make sure you set the following function configuration variables:

    • PTY_ADDIPADDRESSHEADER
    • PTY_ESA_CA_SERVER_CERT You can read more about these variables in section Policy Agent Lambda Configuration.

12. If you are upgrading from versions prior to v3.0, backup and remove existing policy from the bucket defined by AWS_POLICY_S3_BUCKET property, so that the policy can be re-downloaded and re-encrypted with new ‘key commitment’ feature.

13. If you are upgrading from version prior to 1.6.1 please follow the steps below, otherwise the upgrade process is completed.

14. From AWS Console, navigate to IAM > Policies

15. Search for the Agent Lambda IAM Policy created in Create Agent Lambda IAM policy

16. Click on the policy, then select Edit Policy. Select JSON tab.

17. Add the following statement to the list of policy statements.

    
    {
      "Sid": "LambdaGetConfiguration",
      "Effect": "Allow",
      "Action": [
          "lambda:GetFunctionConfiguration"
      ],
      "Resource": [
          "arn:aws:lambda:*:*:function:*"
      ]
    }
    

18. Click Review Policy, then Save Changes. Wait for the changes to save.

Upgrading Log Forwarder Lambda

• Publish Log Forwarder Lambda Version

  Publishing a version of the Log Forwarder Lambda allows to roll-back to pre-existing version if upgrade fails

  1. Go to AWS Lambda console and select existing Protegrity Log Forwarder Lambda.

  2. Click Actions in top right portion of the screen. Select Publish new version. Click Publish.

  3. Record the Lambda version number. It will be displayed at the top of the screen. You can also retrieve it from the Lambda function view, under Versions tab.

     Log Forwarder Lambda version number for roll-backs: ___________________

• Disable Kinesis Trigger

  Disabling Kinesis trigger ensures there are no unprocessed or re-processed events while function is upgraded.

  1. Go to AWS Lambda console and select existing Protegrity Log Forwarder Lambda.
  2. Select Configuration tab > Triggers
  3. Check Kinesis trigger and click Edit button
  4. Uncheck Activate trigger and click Save
  5. Wait for function to stop processing events by monitoring function in Monitor tab

• Upgrade Forwarder Lambda Version

  Upgrade Log Forwarder function with new code

  1. Go to AWS Cloud Formation and select existing Protegrity Log Forwarder deployment stack.
  2. Select Update Stack > Make a direct update.
  3. Select Replace existing template > Upload a template file.
  4. Upload pty_log_forwarder_cf file and select Next.
  5. Click Next until Review window and then select Update stack.
  6. Wait for the Cloud Formation to complete.

• Enable Kinesis Trigger

  1. Go to AWS Lambda console and select existing Protegrity Log Forwarder Lambda.
  2. Select Configuration tab > Triggers
  3. Check Kinesis trigger and click Edit button
  4. Check Activate trigger and click Save Log Forwarder function will now start processing events from where it left off when Kinesis trigger was disabled.

• Monitor and roll-back

  Monitor Log Forwarder function for errors in its CloudWatch logs and in Montior tab. To roll back function to the previous version if any errors occur follow these steps:

  1. Go to AWS Lambda console and select existing Protegrity Log Forwarder Lambda.

  2. Select Configuration tab > Triggers

  3. Expand Details section of Kinesis trigger and record UUID value

  4. Execute the following AWS CLI command to move Kinesis trigger to previous version of Log Forwarder Lambda that was created earlier and recorded as Log Forwarder Lambda version number for roll-backs. Substitute kinesis-mapping-uuid, log-forwarder-function-name, version-for-roll-backs with your values:

     
     aws lambda update-event-source-mapping --uuid <kinesis-mapping-uuid> --function-name <log-forwarder-function-name>:<version-for-roll-backs>
     

  5. Find Kinesis trigger attached to previous version of Log Forwarder Lambda by navigating Versions tab > Version number link in the Versions column Kinesis trigger is now moved to previous version of Log Forwarder Lambda function.

Upgrading Protect Lambda

Diagram below illustrates upgrade steps.

• Snowflake
• Redshift
• Cloud API
• Athena

• Publish Protect Lambda Version

  Publishing a version of the Protect Lambda allows updating it without interruptions to the existing traffic.

  1. Go to AWS Lambda console and select existing Protegrity Protect Lambda.

  2. Go to Lambda Configuration > Environment variables.

  3. Record environment variables values. You will use them later to configure upgraded Lambda Function. You can use the aws cli command below to save the function variables into the local json file:

     aws lambda get-function-configuration --function-name \
     arn:aws:lambda:<aws_region>:<aws_account>:function:<function_name> \
     --query Environment > <function_name>_env_config.json
     

  4. Click Actions in top right portion of the screen. Select Publish new version. Click Publish.

  5. Record the Lambda version number. It will be displayed at the top of the screen. You can also retrieve it from the Lambda function view, under Versions tab.

     Protect Lambda version number: ___________________

  6. If you are upgrading a Cloud Protect Redshift version 1.x to 2.x/3x, you must recreate your Redshift external function definitions with Protect Lambda Function version appended to the Lambda Function name. See example below.

     CREATE OR REPLACE EXTERNAL FUNCTION PTY_PROTECT_TEST ( val varchar )
         RETURNS varchar
         VOLATILE lambda
         'Protegrity_Protect_test:<protect_lambda_version_number>' iam_role
         'arn:aws:iam::123456789212:role/example_role';
     

• Run protect service upgrade

  In this step, the Protect service including Lambda $LATEST version will be updated using Cloud Formation template. The Lambda version created in previous step will be used to serve existing traffic during the upgrade process.

  1. Go to AWS Cloud Formation and select existing Protegrity deployment stack.

  2. Select Update. Check Replace current template > Upload a template file.

  3. Upload pty_protect_cf.json file and select Next.

  4. Update ProtectFunctionProductionVersion parameter with Protect Lambda version number recorded in step 3.

  5. Note

     If you are upgrading protector to one of these versions: [3.2.2, 3.2.3], set parameter KinesisLogStreamArn to the output value recorded in Install through CloudFormation for the newly deployed log forwarder.

  6. Click Next until Review window and then select Update stack.

  7. Wait for the Cloud Formation to complete.

  8. Go back to Lambda console and select Protect Lambda.

  9. Go to Configuration > Environment variables. Replace placeholder values with values recorded in previous step. Alternatively, you can run the following aws cli command to update function configuration using json file saved in the previous steps:

     aws lambda update-function-configuration --function-name \
     arn:aws:lambda:<aws_region>:<aws_account>:function:<function_name> \
     --environment file://./<function_name>_env_config.json
     

     Note

     If your current protect installation version is lower than 3.0.14, you can optionally set the following variable:

     • LOG_REDSHIFT_CLUSTER_ARN You can read more about this configuration in section Protect Lambda Configuration.

  10. Navigate to Aliases tab. Verify that Production alias points to the lambda version you specified in the cloud formation template.

  11. The upgraded Protect Lambda is configured with a sample policy. Run Agent Lambda Function before continuing with next steps.

• Finalize upgrade

  In this step, the Protect Lambda will be configured to serve traffic using $LATEST version upgraded in the previous step.

  1. Go back to Protegrity AWS Cloud Formation deployment stack.

  2. Select Update. Check Use Current template.

  3. Update ProtectFunctionProductionVersion parameter with the following value: $LATEST.

  4. Click Next until Review window and then select Update stack.

  5. Go back to Lambda console and select Protect Lambda.

  6. From the Lambda console, verify that Latest alias points to $LATEST version.

  7. Test your function to make sure it works as expected.

  8. If you are upgrading a Cloud Protect Redshift version 1.x to 2.x/3x, you must recreate your Redshift external function definitions with Protect Lambda Function version appended to the Lambda Function name. See example below.

     CREATE OR REPLACE EXTERNAL FUNCTION PTY_PROTECT_TEST ( val varchar )
         RETURNS varchar
         VOLATILE lambda
         'Protegrity_Protect_test:Production' iam_role
         'arn:aws:iam::123456789212:role/example_role';
     

  9. If you need to rollback to older version of Protect Lambda, you can re-run the cloud formation with ProtectFunctionProductionVersion parameter set to the previous version of Protect Lambda.

Re-enable Protegrity Agent Function CloudWatch Event Rule

If the Event Rule was disabled at the beginning of the upgrade process, you must re-enabled it. Follow the steps below to re-enable Policy Agent Event rule.

1. Go to the Protegrity Agent Cloud Formation Stack.

2. Select Resources tab from the top portion of the screen.

3. Click on the Physical ID link in the ScheduledRule row. The link opens Policy Agent Event Rule configuration.

4. Select Enable from the top-right portion of the screen. This will enable the rule. You will re-enable it after the upgrade process is complete.

1.1.10 - Appendices

1.1.10.1 - ADFS Federation using AWS Cognito User Pools

ADFS Federation using AWS Cognito User Pools

Common use case for Protegrity Policy Member source integrates with customer’s Active Directory. The AWS API Gateway can integrate with ADFS using AWS Cognito. The following article describes the procedure.

https://aws.amazon.com/blogs/mobile/building-adfs-federation-for-your-web-app-using-amazon-cognito-user-pools/

1.1.10.2 - Installing the Policy Agent and Protector in Different AWS Accounts

The Policy Agent Lambda function and Protect Lambda functions can be installed in separate AWS accounts. However, additional configuration is required to authorize the Policy Agent to provision the security policy to a remote Protect Lambda function.

Create Agent Lambda IAM policy

1. Login to the AWS account that hosts the Protect Lambda function.

2. From the AWS IAM console, select Policies > Create Policy.

3. Select the JSON tab and copy the following snippet.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "LambdaUpdateFunction",
         "Effect": "Allow",
         "Action": [
           "lambda:UpdateFunctionConfiguration"
         ],
         "Resource": [
           "arn:aws:lambda:*:*:function:*"
         ]
       },
       {
         "Sid": "LambdaReadLayerVersion",
         "Effect": "Allow",
         "Action": [
           "lambda:GetLayerVersion",
           "lambda:ListLayerVersions"
         ],
         "Resource": "*"
       },
       {
         "Sid": "LambdaDeleteLayerVersion",
         "Effect": "Allow",
         "Action": "lambda:DeleteLayerVersion",
         "Resource": "arn:aws:lambda:*:*:layer:*:*"
       },
       {
         "Sid": "LambdaPublishLayerVersion",
         "Effect": "Allow",
         "Action": "lambda:PublishLayerVersion",
         "Resource": "arn:aws:lambda:*:*:layer:*"
       },
       {
         "Sid": "S3GetObject",
         "Effect": "Allow",
         "Action": [
           "s3:GetObject"
         ],
         "Resource": "arn:aws:s3:::*/*"
       },
       {
         "Sid": "S3PutObject",
         "Effect": "Allow",
         "Action": [
           "s3:PutObject"
         ],
         "Resource": "arn:aws:s3:::*/*"
       },
       {
         "Sid": "LambdaGetConfiguration",
         "Effect": "Allow",
         "Action": [
             "lambda:GetFunctionConfiguration"
         ],
         "Resource": [
             "arn:aws:lambda:*:*:function:*"
         ]
       }
     ]
   }
   

4. Replace the wildcards (*) with the region, account, and resource name information where required.

5. Select Review policy, type in the policy name, and confirm. Record policy name:

   Agent Lambda Cross Account Policy Name: ___________________

Create Policy Agent cross-account IAM Role

1. Login to the AWS account that hosts the Protect Lambda function.

2. From the AWS IAM console, select Roles > Create Role

3. Select AWS Service > Lambda . Proceed to Permissions.

4. Select Policy created in the step above. Proceed to Tags.

5. Specify Tag, proceed to the final screen. Type in policy name and confirm. Record the name.

   Policy Agent Cross Account IAM Role Name: ___________________

Allow the Policy Agent Cross-Account Role to be Assumed by the Policy Agent IAM Role

1. Login to the AWS account that hosts the Protect Lambda function.

2. Navigate to the previously created IAM Role (Agent Lambda Cross-Account IAM Role Name).

3. Navigate to Trust Relationships > Edit Trust Relationships.

4. Modify the Policy Document, replacing the placeholder value indicated in the following snippet as <Agent Lambda IAM Execution Role ARN> with ARN of Agent Lambda IAM Role that was created in Agent Installation.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
   
      "Principal": {
   
               "AWS": "<Agent Lambda IAM Execution Role Name>"
   
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

5. Click Update Trust Policy.

Add Assume Role to the Policy Agent Execution IAM Role

1. Login to the AWS account that hosts the Policy Agent.

2. Navigate to the Agent Lambda IAM Execution Role that was created in Agent Installation.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
   
      "Principal": {
   
               "AWS": "<Agent Lambda IAM Execution Role Name>"
   
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

3. Add Inline Policy.

4. Modify the Policy Document, replacing the placeholder value indicated in the following snippet as <Agent Lambda Cross-Account IAM ARN> with the value recorded in Create Policy Agent cross-account IAM Role.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "sts:AssumeRole"
         ],
         "Resource": "<Agent Lambda Cross-Account IAM  ARN>."
       }
     ]
   }
   

5. When you are finished, choose Review Policy.

6. On the Review policy page, type a Name, then choose Create Policy.

Update the Policy Agent Lambda Configuration

1. From the AWS console, navigate to Lambda, and select the Policy Agent Lambda function.

2. Select Configuration tab | Environment variables.

3. Select Edit and add the following environment variables with the value from Agent Lambda Cross-Account IAM ARN:

   Parameter	Value
   AWS_ASSUME_ROLE	Agent Lambda Cross-Account IAM ARN

4. Ensure the values in the Parameters AWS_POLICY_S3_BUCKET, AWS_PROTECT_FN_NAME and AWS_POLICY_LAYER_NAME are all in the Protect Lambda Function AWS Account.

5. In case custom VPC hostname configuration is used, you will need to set the ENDPOINT_URL. Refer to Policy Agent - Custom VPC Endpoint Hostname Configuration.

   AWS_VPC_ENDPOINT_URL

   <AWS_VPC_ENDPOINT>

6. Click Save and Run the Lambda. The Lambda will now assume the Role in Protect Lambda Function AWS Account and update the policy cross accounts.

1.1.10.3 - Integrating Cloud Protect with PPC (Protegrity Provisioned Cluster)

Key Differences: PPC vs ESA

Prerequisites

• Access to PPC and required credentials.
• Tools: curl, kubectl installed.

Policy Agent Setup with PPC

Follow these instructions as a guide for understanding specific inputs for Policy Agent integrating with PPC:

1. Obtain the Datastore Key Fingerprint

   To retrieve the fingerprint for your Policy Agent:

   1. Retrieve public key from the Cloud Provider Key Management service for the policy encryption key created in pre-configuration:

      • AWS
      • Azure
      • GCP
      1. Navigate to the Key Management Service in AWS console and open Customer Managed Keys
      2. Select the desired key
      3. Select the Public Key tab
      4. Select Download

      1. Navigate to the Key Vault in Azure console and open Objects>Keys
      2. Select the desired key
      3. Select the key indicated as CURRENT VERSION
      4. Select Download public key

      1. Navigate to Key Management in GCP console
      2. Select the desired key and open the Versions tab
      3. Select Get public key from the Actions column menu
      4. Select Download

   2. Escape the new line characters in the downloaded public key for use in the next step - for example:

      awk 'NF {printf "%s\\n",$0}' "<public_key_file>" > "new-line-escaped-public-key.pem"
      cat new-line-escaped-public-key.pem
      

   3. Export key fingerprint using the PPC API as indicated in the curl example below:

      curl -k -H "Authorization: Bearer ${TOKEN}" -X POST https://${HOST}/pty/v2/pim/datastores/1/export/keys  -H "Content-Type: application/json" --data '{
        "algorithm": "RSA-OAEP-256",
        "description": "example-key-from-key-management",
        "pem": "<value of new-line-escaped-public-key>"
      }'
      

      Sample Output:

      {"uid":"1","algorithm":"RSA-OAEP-256","fingerprint":"4c:46:d8:05:35:2e:eb:39:4d:39:8e:6f:28:c3:ab:d3:bc:9e:7a:cb:95:cb:b1:8e:b5:90:21:0f:d3:2c:0b:27","description":"example-key-from-kms"}
      

      Note

      Alternatively, set using the PPC CLI utility. See the export key example in Create Datastores Key

   4. Record the value for fingerprint and configure the Policy Agent:

      • AWS
      • Azure
      • GCP

      Set the environment variable PTY_DATASTORE_KEY in the Policy Agent Lambda function to the fingerprint value.

      Set the environment variable PTY_DATASTORE_KEY in the Policy Agent Function App to the fingerprint value.

      Set the variable in Policy Agent main.tf pty_datastore_key to the fingerprint value and apply the changes.

2. Retrieve the PPC CA Certificate

   To obtain the CA certificate from PPC:

   kubectl -n api-gateway get secret ingress-certificate-secret -o jsonpath='{.data.ca\.crt}' | base64 -d > CA.pem
   

   Use the ProtegrityCA.pem that was returned as described in Policy Agent Installation.

3. Configure the PPC Address

   Use the PPC address in place of the ESA IP address wherever required in your configuration.

   Note

   Use FQDN as described in the PPC Rest API documentation

Log Forwarder Setup with PPC

• The Log Forwarder will proceed without certificates and will print a warning if PTY_ESA_CA_SERVER_CERT is not provided.
• No additional certificate or CA configuration is needed for PPC.

1.1.10.4 - Invoke Lambda Directly

AWS Lambda can be invoked directly, such as from AWS SDK. This section contains information about request and response payloads with examples demonstrating direct invocation using AWS CLI and Python SDK (Boto3).

Request Payload

Lambda request payload for the direct invocation is defined as following

{
  "body": "<rest-api-request-payload>",
  "path": "/v1/<operation>",
  "headers": {}
}

• body - JSON string. Request schemas defined in Rest API Request.
• path - can be either ‘/v1/protect’ or ‘/v1/unprotect’.
• headers - can be used to pass authorization headers. See example below.

Example request:

{
  "body": "{\"query_id\": \"3\",\"user\": \"user1\",\"data_element\": \"deAlpha\",\"data\": [\"data1\", \"data2\"]}",
  "path": "/v1/protect",
  "headers": {}
}

Example Request with JWT authorization:

authorization="jwt"
jwt_verify=1
allow_assume_user=1
jwt_user_claim="username"
jwt_secret_base64="Must be set to public certificate"

{
  "body": "{\"query_id\": \"3\",\"user\": \"user1\",\"data_element\": \"deAlpha\",\"data\": [\"data1\", \"data2\"]}",
  "path": "/v1/protect",
  "headers": {
    "authorization": "bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2MTM4NjIzNzEsImlkIjoiYS1iLWMtZC1lLWYtMS0yLTMiLCJ1c2VybmFtZSI6IlBhdWwgQXRyZWlkZXMifQ.R1NcJ43540HKdhEBOK9WaMMpjBOYSJetckQKrcPQdz0z6sx1EDwHXYngBP9DtHgUM-6Vf1VNjtFh_Nqfeepp1BavmigIXoe3ZbrxRI3DFKi2UuLmgn--EYrSGlWsQjnmjaz5qUkID9iY2MtsRunKSuolSvG9UsD1G32kv0KZYX0"
  }
}

Response Payload

Lambda response payload has the following structure

{
  "body": "<rest-api-response-payload>"
  "isBase64Encoded": false,
  "statusCode": <http-status-code>,
}

• body - JSON string. Response schemas defined in Rest API Response.
• isBase64Encoded - always set to false.
• statusCode - HTTP status code integer.

Success Response Payload Example:

{
  "body": "{\"encoding\":\"utf8\",\"results\":[\"xcgd\", \"migs\"],\"success\":true}", 
  "isBase64Encoded": false, 
  "statusCode": 200
}

Error Response

Cloud API Lambda returns following error responses depending on the error type

Cloud API Protection Operation Error
      Returned when invalid data element is used or user has insufficient permissions to execute security operation.

{
  "body": "{\"error_msg\":\"Unprotect failed. Data element not found. Refer to audit log for details.\",\"success\":false}", 
  "isBase64Encoded": false, 
  "statusCode": 400
}

Cloud API Invalid Request Error
      Missing fields in the incoming request or malformed request JSON.

{
  "body": "Request format is not supported", 
  "isBase64Encoded": false, 
  "statusCode": 400
}

Cloud API Unexpected Lambda Exception Error
      Caused by Lambda runtime exception, for instance due to too short timeout or not enough memory.

{
  "errorMessage": "2023-01-18T16:42:19.593Z d0cf62d0-9eaf-427b-8ca5-1bdd8bd0b082 Task timed out after 10.25 seconds"
}

Examples

Prerequisites:

• AWS SDK or Command Line
• AWS Access Key ID and AWS Access Key

See Request Payload for request payload examples.

AWS CLI command to invoke Cloud API Lambda function:

aws lambda invoke --function-name Protegrity_Protect_RESTAPI_{stackname} --payload
    fileb://request_payload.json --log-type Tail output

Sample Python code demonstrating Cloud API Direct Lambda Calls

import json
import logging
import boto3
lambda_client = boto3.client("lambda")
logging.basicConfig(format="%(message)s")
logger = logging.getLogger('pty_cloud_api_sample')
logger.setLevel(logging.DEBUG)


class ProtectClient(object):
  """ 
    Sample client demonstrating how to invoke Protegrity Cloud API Lambda

      protect_fn: str - Name of the Cloud API Lambda (for example, Protegrity_Protect_RESTAPI_my_deployment)
  """

  def __init__(self, protect_fn):
    self.protect_fn = protect_fn

  def invoke_protect(self, values, data_element, operation, user, query_id,
              column_info=""):
    """ 
    Invokes Protegrity Cloud API Lambda to execute protect or unprotect operation

      values: list[str] - List of values to be protected/unprotected
      data_element: str - Name of the policy data element to use with protect/unprotect operation
      operation: str - Either 'protect' or 'unprotect'
      user: str - Policy user
      query_id: str - Query id will be present in the audit log
      column_info: - Used for troubleshooting, for instance, when protecting values/rows from multiple database columns
    """

    # Set authorization header here if JWT authorization is 
    # enabled in Cloud API Function configuration
    headers = {"Authorization": ""}
    request_body = {
      "user": user,
      "data_element": data_element,
      "data": values,
      "query_id": query_id
    }
    payload = json.dumps({"body": json.dumps(request_body), "path": f"/v1/{operation}",
                  "headers": headers})
    logger.debug(f"Request payload: {payload}")
    response = lambda_client.invoke(FunctionName=self.protect_fn, Payload=payload)
    lambda_response_payload = json.loads(response["Payload"].read().decode())
    logger.debug(f"Response payload: {lambda_response_payload}")
    response_status_code = lambda_response_payload.get("statusCode")
    response_body_string = lambda_response_payload.get("body")
    if response_status_code == None or response_body_string == None:
      raise Exception(f"Unexpected Cloud API Lambda error: [{lambda_response_payload}]")
    try:
      body_json = json.loads(response_body_string)
      if response_status_code == 200:
        return body_json.get("results", [])
      elif body_json.get("error_msg"):
        raise Exception(f"Cloud API Lambda error: [{response_status_code} - {body_json.get('error_msg')}]")
      raise Exception(f"Unexpected Cloud API Lambda error: [{lambda_response_payload}]")
    except json.decoder.JSONDecodeError:
      # Cloud API may return error in the response body
      # For example, {"statusCode": 400, "body":"Error message"}
      raise Exception(f"Cloud API Lambda error: [{response_status_code} - {response_body_string}]")

# Replace cloud-api-lambda-name with the name of the Cloud API Lambda
# For example, Protegrity_Protect_RESTAPI_my_deployment
protect_client = ProtectClient('cloud-api-lambda-name')
protected_data = ["UtfVk UHgcD!"]
logger.info(f"Protected data: {protected_data}")
unprotected_data = protect_client.invoke_protect(
  values=protected_data, 
  data_element='alpha', 
  operation='unprrotect', 
  user='test-user', 
  query_id='1234')
logger.info(f"Unprotected data: {unprotected_data}")

1.1.10.5 - Policy Agent - Custom VPC Endpoint Hostname Configuration

The Policy Agent uses default endpoint hostnames to communicate with other AWS services (for example, secretsmanager.amazonaws.com). This configuration will only work in VPCs where Amazon-provided DNS is available (default VPC configuration with private DNS option enabled for the endpoint). If your VPC uses custom DNS, follow the instructions below to configure the Policy Agent Lambda to use custom endpoint hostnames.

Identify DNS Hostnames

To identify DNS hostnames:

1. From AWS console, select VPC > Endpoints.

2. Select Secrets Manager endpoint from the list of endpoints.

3. Under Details > DNS Names, note the private endpoint DNS names adding https:// at the beginning of the endpoint name.

   For example, https://vpce-1234-4pzomrye.kms.us-west-1.vpce.amazonaws.com

4. Note down DNS names for the KMS and Lambda endpoints:

   AWS_SECRETSMANAGER_ENDPOINT: https://_________________

   AWS_KMS_ENDPOINT: https://_________________

   AWS_LAMBDA_ENDPOINT: https://_________________

Update the Policy Agent Lambda configuration

To update policy agent lambda configuration:

1. From the AWS console, navigate to Lambda, and select the Policy Agent Lambda function.

2. Select the Configuration section and choose Environment variables.

3. Select Edit and add the following environment variables with the corresponding endpoint URLs recorded in steps 3-4:

   Parameters	Value
   AWS_SECRETSMANAGER_ENDPOINT_URL	<AWS_SECRETS_ENDPOINT>
   AWS_KMS_ENDPOINT_URL	<AWS KMS ENDPOINT>
   AWS_LAMBDA_ENDPOINT_URL	<AWS LAMBDA ENDPOINT>

4. Click Save and Run the Lambda. The Lambda will now use endpoints you have just configured.

1.1.10.6 - Protection Methods

Protection Methods

For more information about the protection methods supported by Protegrity, refer to the Protection Methods Reference.

1.1.10.7 - Configuring Regular Expression to Extract Policy Username

Configuring Regular Expression to Extract Policy Username

Cloud Protect Lambda Function exposes USERNAME_REGEX configuration to allow extraction of policy username from user in the request.

• USERNAME_REGEX Lambda Environment configuration

  The USERNAME_REGEX configuration can be used to extract policy username from user in the request. The following are allowed values for USERNAME_REGEX:

  • 1 - Default build-in regular expression is used:

    ^arn:aws:(?:iam|sts)::[0-9]{12}:(?:role|user|group|assumed\-role|federated\-user)\/([\w\/+=,.\-]{1,1024}|[\w\/+=,.\-@]{1,1024})(?:@[a-zA-Z0-9\-]{1,320}(?:\.\w+)+)?$
    

  • ^User regex$ - Custom regex with one capturing group. This group is used to extract the username. Examples below show different regular expression values and the resulting policy user.

^arn:aws:(?:iam|sts)::[0-9]{12}:((?:role|user|group|assumed-role|federated-user).*)$

1.1.10.8 - Associating ESA Data Store With Cloud Protect Agent

Associating ESA Data Store With Cloud Protect Agent

ESA controls which policy is deployed to protector using concept of data store. A data store may contain a list of IP addresses identifying servers allowed to pull the policy associated with that specific data store. Data store may also be defined as default data store, which allows any server to pull the policy, provided it does not belong to any other data stores. Node registration occurs when the policy server (in this case the policy agent) makes a policy request to ESA, where the agent’s IP address is identified by ESA.

Policy agent lambda source IP address used for node registration on ESA depends on ESA hubcontroller configuration ASSIGN_DATASTORE_USING_NODE_IP and the PTY_ADDIPADDRESSHEADER configuration exposed by the agent lambda.

The Lambda service uses multiple network interfaces, internal network interface with ephemeral IP range of 169.254.x.x and external network interface with IP range of the VPC subnet the Lambda is associated with. By default, when agent lambda is contacting ESA to register node for policy download, ESA uses agent Lambda VPC IP address. This default behavior is caused by the default ESA hubcontroller configuration ASSIGN_DATASTORE_USING_NODE_IP=false and agent default configuration PTY_ADDIPADDRESSHEADER=yes.

In some cases, when there is a proxy server between the ESA and agent lambda, the desirable ESA configuration is ASSIGN_DATASTORE_USING_NODE_IP=true. and PTY_ADDIPADDRESSHEADER=no which will cause the ESA to use proxy server IP address.

The table below shows how the hubcontroller and agent settings will affect node IP registration on ESA.

1.2 - Athena

This section describes the high-level architecture of Athena, installation procedures, and performance guidance. This section focuses on Protegrity specific aspects and should be consumed in conjunction with the corresponding AWS documentation.

This guide may also be used with the Protegrity Enterprise Security Administrator Guide, which explains the mechanism for managing the data security policy.

1.2.1 - Overview

Solution Overview

Athena Protector on AWS is a cloud native, serverless product for fine-grained data protection with Amazon Athena. It enables invocation of Protegrity data protection operations from Amazon Athena SQL.

The product provides data protection services invoked by External User Defined Functions (UDFs) within Amazon Athena. The UDFs act as a client transmitting micro-batches of data to a remote Protegrity Athena Protector. The protector runs inside a serverless AWS Lambda function. User queries from Amazon Athena may generate hundreds or thousands of parallel requests to perform security operations. Protegrity’s serverless solution is designed to scale and yield reliable query performance under such load.

The Amazon Athena Protector utilizes a data security policy from an Enterprise Security Administrator (ESA), similar to other Protegrity products.

Analytics on Protected Data

Protegrity’s format and length preserving tokenization scheme make it possible to perform analytics directly on protected data. Tokens are join-preserving so protected data can be joined across datasets. Often statistical analytics and machine learning training can be performed without the need to re-identify protected data. However, a user or service account with authorized security policy privileges may re-identify subsets of data using the Athena Protector on AWS service.

Features

Athena Protector on AWS incorporates Protegrity’s patent-pending vaultless tokenization capabilities into cloud-native serverless technology. Combined with an ESA security policy, the protector provides the following features:

• Role-based access control (RBAC) to protect and unprotect (re-identify) data depending on the user privileges.
• Policy enforcement features of other Protegrity application protectors.

For more information about the available protection options, such as, data types, Tokenization or Encryption types, or length preserving and non-preserving tokens, refer to Protection Methods Reference.

1.2.2 - Architecture

Deployment Architecture

The Amazon Athena Protector should be deployed in the customer’s AWS account. The product incorporates Protegrity’s vaultless tokenizationengine within an AWS Lambda Function. The data security policy from an ESA is provisioned periodically as a static, encrypted resource through an AWS Lambda Layer. The policy is decrypted in memory at runtime within the Lambda. This architecture allows Protegrity to be highly available and scale very quickly without direct dependency on other Protegrity services.

The product exposes a remote data protection service invoked from external User Defined Functions (UDFs), a native feature of Amazon Athena.The UDFs is invoked by users through SQL queries. The Amazon Athena execution engine makes remote calls to the Protegrity product to perform protect and unprotect data operations. Each network request includes a micro-batch of data to process and a secure payload including the federated user (if available) and the data element type. The product applies the ESA security policy including user authorization and returns a corresponding response.

When used with an Enterprise Security Administrator (ESA) application, the security policy is synchronized through another serverless component called the Protegrity Policy Agent. The agent operates on a configurable schedule, fetches the policy from the ESA, performs additional envelope encryption using Amazon KMS, and deploys the policy into the Lambda Layer used by the serverless product. This solution can be configured to automatically provision the static policy or the final step can be performed on-demand by an administrator. The policy takes effect immediately for all new requests. There is no downtime for users during this process.

The following diagram shows the high-level architecture described above.

The following diagram shows a reference architecture for synchronizing the security policy from the ESA to the product.

The Protegrity Policy Agent requires network access to an Enterprise Security Administrator (ESA). Most organizations install the ESA on-premise. Therefore, it is recommended that the Policy Agent is installed into a private subnet with a Cloud VPC using a NAT Gateway to enable this communication through a corporate firewall.

The ESA is a soft appliance that must be pre-installed on a separate server. It is used to create and manage security policies.

For more information about installing the ESA, and creating and managing policies, refer to the Policy Management Guide.

Log Forwarding Architecture

Audit logs are by default sent to CloudWatch as long as the function’s execution role has the necessary permissions. The Protegrity Product can also be configured to send audit logs to ESA. Such configuration requires deploying Log Forwarder component which is available as part of Protegrity Product deployment bundle. The diagram below shows additional resources deployed with Log Forwarder component.

The log forwarder component includes Amazon Kinesis data stream and the forwarder Lambda function. Amazon Kinesis stream is used to batch audit records before sending them to forwarder function, where similar audit logs are aggregated before sending to ESA. Aggregation rules are described in the Protegrity Log Forwarding guide. When the protector function is configured to send audit logs to log forwarder, audit logs are aggregated on the protector function before sending to Amazon Kinesis. Due to specifics of the Lambda runtime lifecycle, audit logs may take up to 15 minutes before being sent to Amazon Kinesis. Protector function exposes configuration to minimize this time which is described in the protector function installation section.

The security of audit logs is ensured by using HTTPS connection on each link of the communication between protector function and ESA. Integrity and authenticity of audit logs is additionally checked on log forwarder which verifies individual logs signature. The signature verification is done upon arrival from Amazon Kinesis before applying aggregation. If signature cannot be verified, the log is forwarded as is to ESA where additional signature verification can be configured. Log forwarder function uses client certificate to authenticate calls to ESA.

To learn more about individual audit log entry structure and purpose of audit logs, refer to Audit Logging section in this document. Installation instructions can be found in Audit Log Forwarder installation.

The audit log forwarding requires network access from the cloud to the ESA. Most organizations install the ESA on-premise. Therefore, it is recommended that the Log Forwarder Function is installed into a private subnet with a Cloud VPC using a NAT Gateway to enable this communication through a corporate firewall.

Athena Connectivity

Amazon Athena is an interactive, serverless query service that makes it easy to analyze data in Amazo n S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

Athena’s external UDF makes remote invocations to the Protegrity protector, a serverless Lambda function incorporating Protegrity libraries and the Athena Query Federation SDK.

Access and authorization between various AWS services involved in this architecture is achieved throu gh Identity Access Management (IAM). For instance, the IAM principal running the query must be allowed to invoke the protect Lambda. Various IAM role configuration settings are shown in the appen dices of this document.

The following figure illustrates the Protegrity Athena Integration architecture.

1.2.3 - Installation

1.2.3.1 - Prerequisites

AWS Service Dependencies

The following table describes the AWS services that may be a part of your Protegrity installation.

ESA Version Requirements

The Protector and Log Forwarder functions require a security policy from a compatible ESA version.

The table below shows compatibility between different Protector and ESA versions.