https://docs.protegrity.com/protectors/10.0/docs/bdp/pyiceberg_protector/pyiceberg_databricks/Recent content in Python Iceberg Protector on Databricks onHugoenhttps://docs.protegrity.com/protectors/10.0/docs/bdp/pyiceberg_protector/pyiceberg_databricks/pyiceberg_dbr_arch/Mon, 01 Jan 0001 00:00:00 +0000https://docs.protegrity.com/protectors/10.0/docs/bdp/pyiceberg_protector/pyiceberg_databricks/pyiceberg_dbr_arch/<p>The architecture of the Iceberg Protector using Databricks is depicted in the following diagram:</p> <p><img src="https://docs.protegrity.com/protectors/10.0/docs/images/bdp/py_iceberg/iceberg_databricks/pyicerberg_databricks_architecture.png" alt="" title="Python Iceberg Protector Architecture on Databricks"></p> <h2 id="python-iceberg-and-parquet-modular-encryption-pme-architecture">Python Iceberg and Parquet Modular Encryption (PME) Architecture</h2> <h3 id="write-path-in-databricks-environment">Write Path in Databricks environment</h3> <ol> <li> <p><strong>Warehouse</strong>: The data platform like Databricks initiates the data write and interacts with the <strong>Unified Catalog</strong> to register/manage table metadata.</p> </li> <li> <p><strong>Unified Catalog</strong>: Serves as the central metadata registry. It integrates with catalog providers such as <strong>HMS, Delta, Unity, Polaris, Horizon/Open/REST, and Glue</strong>, and receives encryption instructions from the <strong>Column Encryption Config</strong>.</p>https://docs.protegrity.com/protectors/10.0/docs/bdp/pyiceberg_protector/pyiceberg_databricks/pyiceberg_dbr_sys_req/Mon, 01 Jan 0001 00:00:00 +0000https://docs.protegrity.com/protectors/10.0/docs/bdp/pyiceberg_protector/pyiceberg_databricks/pyiceberg_dbr_sys_req/<p>Ensure the following prerequisites are met:</p> <ol> <li>Databricks Unity Catalog is available with: <ol> <li>A Terminated Dedicated or Standard Compute.</li> <li>A Unity Catalog Volume.</li> <li>Service Principal. Ensure that the Service Principal has: <ol> <li>USE CATALOG, USE SCHEMA, READ VOLUME, and WRITE VOLUME privileges on Unity Catalog Volume.</li> <li>MANAGE ALLOWLIST privilege on Unity Catalog Metastore.</li> <li>CAN MANAGE privilege on Compute.</li> </ol> </li> </ol> </li> </ol>