Mapping File

Overview

S3 Protector uses a mapping.json file to determine how columns in the source file are mapped to Protegrity data elements for protection. The Lambda resolves the mapping file location using the following precedence order (highest to lowest):

Using S3 Object Tags for Mapping File Resolution

Add a tag to the source S3 object to point to a specific mapping file. The tag key is always MAPPING_LOCATION. The tag value supports two formats:

IAM note: The Lambda execution role must have s3:GetObjectTagging on the source bucket when using tag-based resolution.

Configuration Structure

The mapping.json file must be formatted in valid JSON with the key-values configuration pairs described below:

{
  "ignored-columns": ["<ignored-col-name-1>", "<ignored-col-name-n>"],
  "columns": {
    "<col-name-1>": {
      "operation": "[protect|unprotect]",
      "data_element": "<data-element-name>"
    },
    "<col-name-2>": {
      "operation": "[protect|unprotect]",
      "data_element": "<data-element-name>"
    }
  },
  "input": {
    "format": "<file-format>",
    "spec": { "<reader-arg>": "<value>" }
  },
  "output": {
    "format": "<file-format>",
    "spec": { "<writer-arg>": "<value>" }
  }
}

Data Columns Transformation

Every source file column must appear in either ‘columns’ or ‘ignored-columns’.

1. “columns” (required) - Maps input data columns to Protegrity security operation such as ‘protect’ or ‘unprotect’. Each operation is applied using provided data element.

2. “ignored-columns” (optional) - Lists the names of input data columns which do not require any Protegrity security operations applied. Data for these columns will be left unprocessed and will be written to target file as is.

Input Data Configuration

The “input” optional configuration contains the following key-values pairs:

format

Specifies the format of the input data files. If format is not provided in the mapping json, the format will be inferred from the file extension.

spec

Provides additional configuration for input file processing. This allows processing of non-default file formats. For example, pipe delimited files, header-less files, and various JSON record structures.

The properties within the input spec block correspond with the Python Pandas reader functions arguments. For more information about supported format arguments refer to the Pandas documentation. Below you can find a list of links to Pandas official online documentation for each format supported by S3 Protector:

• CSV - read_csv

  Note

  The default configuration expects header record, comma-delimited fields, and double quotes for text-qualified fields.

• Excel - read_excel

• Parquet - read_parquet

  Note

  The default configuration reads Parquet files in batches to reduce memory usage. This ignores storage_options argument, which affects non-AWS S3 implementations such as MinIO and LocalStack. To load full file and enable storage_options, set "chunked": false in input.spec.

• JSON - read_json

  Note

  The default configuration expects the JSON input file to represent tabular data. Common supported layouts are a flat JSON array of records or JSON Lines. Each record becomes one row, and the keys become the column names. See the JSON appendix example and the Known Limitations section.

Output Data Configuration

The “output” optional configuration contains the following key-values pairs:

format

Specifies the format of the output data files. The format in the mapping json is only used when S3 Protector Function deployment parameter OutputFileFormat is set to use_mapping_spec. See the CloudFormation installation section for the full list of the output format configuration.

spec

Provides additional configuration for the output file processing.

The properties within the output spec block correspond with the Python Pandas DataFrame output function arguments. For more information about supported format arguments refer to the Pandas documentation. Below you can find a list of links to Pandas official online documentation for each format supported by S3 Protector:

• CSV - DataFrame.to_csv

  Note

  The default configuration writes header record based on the IncludeHeader deployment parameter.

• Excel - DataFrame.to_excel

• Parquet - DataFrame.to_parquet

  Note

  If a large Parquet file is being processed in chunks, only index and compression arguments are applied from the output spec. Other arguments are ignored.

• JSON - DataFrame.to_json