Performance

Performance Considerations

The following factors may affect S3 Protector performance:

1. Number of protected columns in a file: Affects the number of requests to Cloud API. The more protected columns, the longer it will take to process the file. Review Performance section in Cloud API on AWS documentation on how to monitor and configure Cloud API for best performance.

2. Maximum batch size: MAX_BATCH_SIZE determines the maximum number of rows to process in a single Cloud API invocation. This value is applied to all columns. The higher the batch size, the higher the Cloud API throughput. This value controls the size of a single request to Cloud API. The request size is limited by AWS Lambda infrastructure to 6 MB. Review AWS Lambda quotas and limitations for latest information.

3. Maximum parallel protect calls: MAX_PARALLEL_PROTECT_CALLS controls how many protect requests can be sent at the same time across the file being processed. The default is 1000. Lower values can reduce pressure on Cloud API; higher values can improve throughput if Cloud API has enough capacity.

4. File streaming read size: FILE_CHUNK_SIZE controls how many rows S3 Protector reads into memory at a time from source file. Lower values reduce peak memory usage. Higher values may improve throughput. This does not apply to JSON arrays. Applies to all other file types including JSON lines.

5. Large-file threshold: LARGE_FILE_DISPATCH_THRESHOLD_BYTES Files larger than this threshold are processed in parallel by multiple S3 Protector instances. Does not apply to XLS files.

6. Large-file chunk size: LARGE_FILE_CHUNK_SIZE_BYTES controls the target size of each chunk when large files are processed in parallel. For JSON array files, each chunk must fit into a single Lambda’s memory; this constraint does not apply to JSON lines files.

7. Function timeout: Default S3 Protector execution time is set to 15 minutes, the maximum value imposed by AWS Lambda infrastructure. If S3 Protector runs out of time when processing a file, ensure LARGE_FILE_DISPATCH_THRESHOLD_BYTES is smaller than the files size and LARGE_FILE_CHUNK_SIZE_BYTES is small enough to fit file processing into 15 minute timeout.

8. Cloud API performance: S3 Protector uses Cloud API to apply protect operations to data in the file. Performance of Cloud API directly affects the performance of S3 Protector. Review Performance section in Cloud API on AWS documentation.

Benchmarks

Single S3 Protector Instance Per File

The following table shows throughput and latency for three different files when large file processing is not enabled. Each file has 21 columns, 6 of which were protected by S3 Protector with tokenization data elements. The remaining 15 columns were configured to pass through without applying protection.

Two of the default S3 Protector settings were updated for this benchmark:

1. Default function timeout was increased to its maximum of 15 minutes
2. ‘MaxBatchSize’ was increased from default ‘25000’ to ‘50000’ (via CloudFormation template)

Performance Results

Multiple S3 Protector Instances Per File

The following results show wall-clock duration and throughput when large-file dispatch is enabled and each file is split across multiple parallel S3 Protector instances.

Test configuration:

• Lambda memory: 1700 MB
• LARGE_FILE_CHUNK_SIZE_BYTES: 109 MB per chunk
• FILE_CHUNK_SIZE: 150,000 rows
• MAX_BATCH_SIZE: 75,000
• Each file has 31 columns; 9 protected with tokenization data elements

Performance Results

Throughput scales with file size because larger files produce more parallel Processor Lambda invocations, increasing overall concurrency.