Python Iceberg Protector

Iceberg Protector on Python

The Python Iceberg Protector enables secure, policy-driven protection of sensitive data processed through Python-native Apache Iceberg workflows. It extends data-centric protection capabilities to Python Iceberg-based pipelines, ensuring that sensitive data remains protected at every stage of the data lifecycle from ingestion and transformation to storage and analytics.

Python Iceberg Protector depends on PyArrow, which is backed by C++, for data operations. It allows applications to read, write, and manage Iceberg tables, while maintaining full compatibility with Iceberg’s table format and metadata model. It operates within a layered Iceberg architecture consisting of catalog, metadata, and storage layers, enabling scalable and ACID-compliant data operations.

The Python Iceberg Protector integrates seamlessly into this architecture by embedding protection directly into Python-based data operations, ensuring that:

• Sensitive data is protected before it is written to Iceberg tables.
• Protection persists at the data layer. For example, within Parquet files.
• Authorized clients can securely access and process protected data without exposing clear-text values unnecessarily.

The protector adopts a data-centric security model, where protection travels with the data regardless of where it is stored or processed. This aligns with modern Lakehouse security principles that enforce fine-grained encryption and policy-based access controls across distributed environments.

For more information about the Iceberg Protector on Python, refer to Python Iceberg Protector.

Python Iceberg Protector on Databricks

The Python Iceberg Protector on Databricks integrates Protegrity data protection with Apache Iceberg tables managed by the Databricks Lakehouse Platform. The protector is delivered as a PyIceberg-compatible extension that runs inside Databricks clusters and SQL warehouses, and applies column-level protection to Iceberg tables registered in Unity Catalog.

Protection is enforced through Parquet Modular Encryption (PME) using the Apache Arrow and PyIceberg engines. Columns identified in the Column Encryption Config are transformed at write time using Protegrity data elements like tokenization, encryption, masking, or hashing, while non-sensitive columns are written as standard Parquet. Protection is embedded in Parquet footers and column metadata. Protected data remains portable across engines like Snowflake, Trino, and Cloudera, and across storage such as S3, Azure Blob, and Ozone.

At runtime, the protector communicates with the Protegrity Data Security Platform to resolve policy decisions and obtain the keys required to protect or unprotect column values. Policy is evaluated per request against the caller’s identity, role, and other attributes. A single physical copy of an Iceberg table serves multiple entitlement levels, without separate views or datasets.

The Python Iceberg Protector on Databricks provides the following capabilities:

• Column-level protection of Iceberg tables during ingestion from Databricks notebooks, jobs, and Delta Live Tables pipelines that use the PyIceberg APIs.
• Reversible and irreversible protection methods like tokenization, encryption, masking, hashing, selected per column based on the configured data elements.
• Policy-driven unprotection at query time, enforced by the Protegrity policy engine and applied transparently to authorized readers.
• Interoperability with other Iceberg engines that consume the same Parquet files from cloud storage, subject to Protegrity policy and key access.
• Centralized policy management, key management, and audit logging through the Protegrity Enterprise Security Administrator (ESA).

For more information about the Python Iceberg Protector on Databricks, refer to Python Iceberg Protector on Databricks.

Python Iceberg Protector on Snowflake

The Protegrity Python Iceberg Protector on Snowflake delivers column-level data protection for Apache Iceberg tables. These tables are managed by the Snowflake REST Catalog (Polaris) and stored as Parquet in cloud object storage, such as AWS S3. It enables data engineers and analysts to read from and write to Iceberg tables from Python workloads while sensitive fields are transparently protected. Protection uses the same Protegrity policy that governs the rest of the enterprise data estate.

The protector is delivered as a Custom Runtime Environment (CRE) that runs inside Snowflake Snowpark Container Services (SPCS). The runtime image is built and published through a standard container pipeline like Docker and the Snowflake CLI and deployed to SPCS as a managed container. Inside the CRE, a Snowflake Notebook hosts user code that calls the Protegrity-instrumented Iceberg and Arrow libraries, PTYPyIceberg and PTYPyArrow. These libraries are drop-in replacements for the standard Python Iceberg and PyArrow APIs, so existing Iceberg workloads can adopt protection with minimal code changes.

Protection is enforced by the Application Protector for C (AP-C), which is co-located in the runtime and invoked by PTYPyIceberg and PTYPyArrow on the columns identified by policy. On write, protected column values are encrypted, tokenized, or masked before the Parquet files are persisted to S3. On read, the same operations are reversed in memory based on the caller’s entitlements. Because protection is applied in the client runtime, the Parquet objects that land in the Iceberg table are already protected at rest, independently of the storage layer’s own encryption.

AP-C obtains its policy and key material from the DevOps Policy and Remote Protection Agent (RPAgent) components that ship inside the CRE. Policy is authored and managed centrally on the Protegrity Data Security Platform (ESA) and distributed to the runtime. Data element definitions, protection methods, and role-based access rules remain consistent with the customer’s existing Protegrity deployment.

Access to Iceberg metadata and data is brokered by Snowflake. The runtime authenticates to the Snowflake REST Catalog (Polaris) using a Personal Access Token (PAT) to resolve namespaces, table locations, and snapshots. Polaris then vends short-lived, scoped credentials that PTYPyIceberg and PTYPyArrow use to read and write the underlying Parquet files in S3. This removes the need for long-lived storage credentials in the runtime.

For more information about the Python Iceberg Protector on Snowflake, refer to Python Iceberg Protector on Snowflake.