Python Iceberg Protector on Databricks

The Pyhon Iceberg Protector on Databricks integrates Protegrity data protection with Apache Iceberg tables managed by the Databricks Lakehouse Platform. The protector is delivered as a PyIceberg-compatible extension that runs inside Databricks clusters and SQL warehouses, and applies column-level protection to Iceberg tables registered in Unity Catalog.

Protection is enforced through Parquet Modular Encryption (PME) using the Apache Arrow and PyIceberg engines. Columns identified in the Column Encryption Config are transformed at write time using Protegrity data elements like tokenization, encryption, masking, or hashing, while non-sensitive columns are written as standard Parquet. Protection is embedded in Parquet footers and column metadata. Protected data remains portable across engines like Snowflake, Trino, and Cloudera, and across storage such as S3, Azure Blob, and Ozone.

At runtime, the protector communicates with the Protegrity Data Security Platform to resolve policy decisions and obtain the keys required to protect or unprotect column values. Policy is evaluated per request against the caller’s identity, role, and other attributes. A single physical copy of an Iceberg table serves multiple entitlement levels, without separate views or datasets.

The Python Iceberg Protector on Databricks provides the following capabilities:

• Column-level protection of Iceberg tables during ingestion from Databricks notebooks, jobs, and Delta Live Tables pipelines that use the PyIceberg APIs.
• Reversible and irreversible protection methods like tokenization, encryption, masking, hashing, selected per column based on the configured data elements.
• Policy-driven unprotection at query time, enforced by the Protegrity policy engine and applied transparently to authorized readers.
• Interoperability with other Iceberg engines that consume the same Parquet files from cloud storage, subject to Protegrity policy and key access.
• Centralized policy management, key management, and audit logging through the Protegrity Enterprise Security Administrator (ESA).

The following sections describe the architecture, system requirements, environment preparation, and installation steps for the Iceberg Protector on Databricks.