This product is currently in Tech Preview and is not available for General Availability (GA). It should not be used in production environments, as features and functionality may change before the final GA release.

Python Iceberg Protector Architecture on Databricks

Understand the Architecture to install the Python Iceberg Protector on Databricks.

The architecture of the Iceberg Protector using Databricks is depicted in the following diagram:

Python Iceberg and Parquet Modular Encryption (PME) Architecture

All of the following components, except Storage (S3), run inside the Databricks environment.

  1. Warehouse: Initiates the data write and exchanges table metadata with the catalog services (HMS, Delta, Unity, In-Memory, Glue, REST).

  2. Iceberg: Receives data from the Warehouse and also exchanges metadata with the catalog services to resolve table location, schema, and snapshot information.

  3. HMS / Delta / Unity / In-Memory / Glue / REST: The set of catalog providers that the Warehouse and Iceberg layers communicate with to manage table metadata. This catalog also receives encryption policy from the Column Encryption Config.

  4. Column Encryption Config: Supplies the column-level encryption policy (which columns to encrypt, key references, etc.) up to the catalog services so encryption is applied consistently.

  5. Arrow (Parquet PME): Receives data from Iceberg and performs Parquet Modular Encryption on the configured columns.

  6. Parquet File With Encrypted Columns: The output of Arrow/Parquet PME — a Parquet file where the sensitive columns are encrypted at the column level rather than the whole file.

  7. Storage (S3): The encrypted Parquet file is persisted to S3, outside the Databricks boundary, as the durable storage layer.

End-to-end flow: The Warehouse writes data and coordinates metadata with the catalog (HMS/Delta/Unity/In-Memory/Glue/REST) → data flows to Iceberg, which also coordinates with the same catalog → Iceberg hands data to Arrow (Parquet PME) → PME encrypts the configured columns, guided by policy that the Column Encryption Config supplies to the catalog → the result is written as a Parquet File With Encrypted Columns → the file is persisted to Storage (S3).

Last modified : October 06, 2026