S3 Protector Service Installation

Preparation

Ensure that all the steps in Pre-Configuration are performed.

1. Login to the AWS sub-account console where Protegrity will be installed.

2. Ensure that the required CloudFormation templates provided by Protegrity are available on your local computer.

Create S3 Protector Lambda IAM Execution Policy

The below steps create an IAM policy for use by the Protegrity Lambda function. The policy grants permissions to:

• Write logs to CloudWatch
• Read from input S3 bucket
• Write to output S3 bucket
• Invoke Cloud Protect API function

Steps

1. From the AWS IAM console, select Policies → Create Policy.

2. Select the JSON tab and copy the following sample policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CloudWatchWriteLogs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ReadS3In",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetObjectAcl",
        "s3:ListBucket",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::PLACEHOLDER_S3_IN_BUCKET_NAME",
        "arn:aws:s3:::PLACEHOLDER_S3_IN_BUCKET_NAME/*"
      ]
    },
    {
      "Sid": "WriteS3Out",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:ListBucket",
        "s3:PutObjectAcl",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::PLACEHOLDER_S3_OUT_BUCKET_NAME",
        "arn:aws:s3:::PLACEHOLDER_S3_OUT_BUCKET_NAME/*"
      ]
    },
    {
      "Sid": "InvokeCloudProtectApi",
      "Effect": "Allow",
      "Action": [
        "lambda:InvokeFunction"
      ],
      "Resource": [
        "PLACEHOLDER_CLOUD_PROTECT_API_ARN"
      ]
    }
  ]
}

3. Replace the PLACEHOLDER values with the values recorded in earlier steps:

   • Cloud Protect API prerequisites
   • S3 Data Buckets prerequisites

4. Select Review policy, type in a policy name (e.g., ProtegrityS3ProtectorLambdaPolicy) and Confirm.

5. Record the policy name.

   S3 Protector Function Policy Name: __________________

Create S3 Protector Lambda IAM Role

The following steps create the role to utilize the policy defined in the previous section.

Steps

1. From the AWS IAM console, select Role → Create Role.

2. Select AWS Service → Lambda → Permissions.

3. In the list, search and select the policy created in the previous step.

4. Proceed to Tags.

5. Proceed to final step of the wizard.

6. Type the role name (e.g., ProtegrityS3ProtectorLambdaRole) and click Confirm.

7. Record the role ARN.

   Protegrity S3 Protector Lambda Role ARN: ___________________

Install through CloudFormation

The following steps describe deployment of the S3 Protector Lambda Function using CloudFormation.

1. Access CloudFormation and select the target AWS Region in the console.

2. Click Create Stack and choose With new resources.

3. Specify the template.

4. Select Upload a template file.

5. Upload the Protegrity-provided CloudFormation template called pty_s3_protector_cf.json and click Next.

6. Specify the stack details. Enter a stack name.

   Note

   The stack name will be appended to all the services created by the template.

7. Enter the required parameters. All the values were generated in the pre-configuration steps.

CloudFormation Parameters

8. Click Next with defaults to complete CloudFormation.

9. After CloudFormation is completed, select the Outputs tab in the stack.

10. Record the S3 Protector Lambda Name and Arn.

    S3 Protector Lambda Name: __________________________

    S3 Protector Lambda Arn: ________________________________

Test S3 Protector Function Configuration

Perform the following steps to verify that S3 Protector Function can read files from input S3 bucket, call Cloud API protector and write data to output S3 bucket.

Before you begin:

1. Update S3 Protector Cloud Formation stack with temporary settings used for testing:
   • In AWS Cloud Formation console, go to Stacks
   • Select Cloud Formation stack deployed in the installation step
   • In the stack details pane, choose Update
   • Select Use existing template and then choose Next
   • Change the following parameters:

• Select Next and then Submit. Wait until the changes are deployed.

2. Upload sample data file to S3 input bucket.

data.csv:

first name,last name,email
tusqB,FrjKe,ebMgF.VoiDd@bqclblD.wOt
JXVVW,acg,BikPa.ufb@UmPxcTD.bLh
mDNJ,IZWCYkbnrAs,NWXD.GdrzMJwmwJG@fMZsuSE.Qlp
jIqColWOss,XKfz,NVabzoUSgx.XRHM@BQleCST.Mnb
muUxYvz,FLZxCHlca,eiNjzCm.UMRNYANwn@isvxpAV.PJk

3. Upload mapping.json to the input S3 bucket next to the input data file. Replace placeholders with data element names configured in your security policy. If your Cloud Protect API Function uses sample policy you can replace “protect” with “unprotect” for operation and use “alpha” as data element.

{
 "columns":{
    "first name":{
      "operation":"protect",
      "data_element":"<data_element_1_name>"
    },
    "last name":{
      "operation":"protect",
      "data_element":"<data_element_2_name>"
    },
    "email":{
      "operation":"protect",
      "data_element":"<data_element_3_name>"
    }
 }
}

Execute S3 Protector Function in AWS console:

With the input data file and mapping file uploaded, follow the steps below to trigger the S3 Protect Function.

1. Sign in to the AWS Management Console and go to Lambda console.

2. Select Lambda Function recorded in S3 Protector Lambda Name in Install through CloudFormation section.

3. On the S3 Protector Function page, choose Test tab.

4. Copy the json test event into the Event JSON pane - replace bucket name placeholder with your input bucket name.

   {
     "Records": [
       {
         "s3": {
           "bucket": {
             "name": "<PLACEHOLDER_S3_IN_BUCKET_NAME>"
           },
           "object": {
             "key": "data.csv"
           }
         }
       }
     ]
   }
   

5. Select Test to execute the test event.

Verify execution results:

1. Execution is successful if the output of test contains the following:

{
  "statusCode": 200,
  "body": {
    "target": "s3://<PLACEHOLDER_S3_OUT_BUCKET_NAME>/data.<timestamp>.csv"
  }
}

If the expected output is not present, please consult the Troubleshooting section for common errors and solutions.

2. Download the output file mentioned in the response body in the “target” field. Verify that it was processed according to your mapping.json. If sample policy was used with “unprotect” and “alpha” data element, the output file should contain values below:

first name,last name,email
Lorem,Ipsum,lorem.ipsum@example.com
Dolor,Sit,dolor.sit@example.com
Amet,Consectetur,amet.consectetur@example.com
Adipiscing,Elit,adipiscing.elit@example.com
Vivamus,Elementum,vivamus.elementum@example.com

Restore production configuration:

After S3 Protector Function configuration has been verified, make sure that the following configuration was restored for production environment:

• Cloud Formation configuration - restore values changed in pre-configuration steps at the beginning of this section.
• IAM permissions - remove any additional S3 read/write IAM permissions used to manually upload test datasets to S3.

Configure S3 Lambda Triggers

Follow the steps below to configure Amazon S3 event notification on the input bucket. This will allow Amazon S3 to send an event to S3 Protector Lambda function when an object is created or updated.

Steps to Add S3 Lambda trigger:

1. Sign in to the AWS Management Console and open the Amazon Lambda console.

2. Select Lambda Function recorded in S3 Protector Lambda Name in the installation section.

3. On the S3 Protector Function page, choose Aliases, then click on Production alias.

4. In the Function overview pane, choose Add trigger.

5. Select S3.

6. Under Bucket, select the bucket recorded in Input S3 Bucket Name in prerequisites section.

7. Under Event types, select All object create events.

8. Optionally enter a file prefix.

9. Enter a file suffix, e.g.: .csv. You can find the full list of supported file formats in the Features section.

10. Under Recursive invocation, select the check box to acknowledge that using the same Amazon S3 bucket for input and output is not recommended.

11. Choose Add.

12. Repeat these steps for additional file suffixes supported by S3 Protector.

Example Usage

This section describes typical usage of S3 Protector.

Prepare data for testing:

Sample datasets and mapping.json files are provided in appendix sections:

• CSV with no header
• CSV with pipe delimiter

Create a new folder in the input S3 bucket:

A new folder must be created in the S3 input bucket for each distinct file schema. Each folder can have a mapping.json file corresponding to the dataset type expected. It is recommended that input folders use S3 encryption:

• From the AWS S3 console, search and select the S3 input bucket created earlier for input files
• Click the Create folder button
• Provide a descriptive name for the type of dataset, e.g. sales orders
• In Server-side encryption, select Enable
• Use the default key type, Amazon S3 key (SSE-S3)
• Click Create folder

Upload the mapping.json and dataset to the folder:

The appropriate mapping.json file must be uploaded to the folder prior to uploading the dataset.

• Choose one of the sample dataset and mapping.json pairs from the appendix. Replace the data elements in mapping.json with similar data elements from your security policy
• From the AWS console, navigate to Amazon S3, search and select the S3 input bucket created earlier for incoming files
• Navigate to the desired folder
• Click the Upload button
• Click Add files
• Upload the mapping.json file
• Click the Upload button
• Now repeat the above step for the sample dataset

Verify output:

Verify the output file was created:

• From the AWS console, navigate to Amazon S3, search and select the S3 output or target bucket created earlier for writing processed files
• Navigate to the corresponding folder
• There should be a non-zero byte file with protected values
• Select the file
• From the menu select Actions | Query with S3 Select
• Click the Run SQL query
• Click the Formatted tab of the resultset
• Verify the data is protected

Troubleshooting / Logs:

Logs are written to CloudWatch. This could provide helpful information if the results are not as expected:

• From the AWS console, navigate to the Lambda service | Functions
• Select and open the Lambda we created for protecting S3 files
• At the top of function’s workspace, click the Monitoring tab
• Click the button View logs in CloudWatch
• Click the latest log stream
• Scroll to the bottom of the stream for the latest log entries

Troubleshooting

By default S3 Protector is set to log minimal information. It is beneficial to increase S3 Protector log level to either ‘config’ or ‘all’ while troubleshooting any error conditions. Use the CloudFormation installation steps to increase ‘MinLogLevel’ function configuration.

S3 Protector Error States

Restarting S3 Protector

If S3 Protector fails, it is possible to start S3 Protector for existing source file without re-uploading the file again by using AWS Lambda console. With the input data file and mapping file uploaded, follow the steps below to trigger the S3 Protect Function.

Steps

1. Sign in to the AWS Management Console and go to Lambda console.

2. Select Lambda Function recorded in S3 Protector Lambda Name in the CloudFormation installation section.

3. On the S3 Protector Function page, choose Test tab.

4. Copy the json test event into the Event JSON pane - replace bucket name placeholder with your input bucket name:

{
  "Records": [
    {
      "s3": {
        "bucket": {
          "name": "<PLACEHOLDER_S3_IN_BUCKET_NAME>"
        },
        "object": {
          "key": "data.csv"
        }
      }
    }
  ]
}

5. Select Test to execute the test event.