Policy Workbench on Microsoft Azure

• 1: Introduction
• 2: Installation
• 2.1: Prerequisites
• 2.2: Installing Policy Workbench
• 3: Upgrade
• 3.1: Upgrading Policy Workbench in Azure
• 4: Backup and Restore
• 4.1: Backing up the Policy Workbench
• 4.2: Restoring the Policy Workbench
• 5: Uninstallation

1 - Introduction

The Policy Workbench is the policy management component of AI Team Edition, designed to provide a modern, cloud-native environment for data protection policy management. Built to run on Kubernetes, Policy Workbench uses a containerized architecture and is deployed on a supported Protegrity Provisioned Cluster (PPC) running on Azure Kubernetes Service (AKS). OpenTofu modules and Helm charts help administrators prepare the required resources and deploy Policy Workbench with minimal manual intervention, ensuring consistency and reducing operational overhead.

2 - Installation

This section provides the steps to install the Policy Workbench on Azure Kubernetes Service (AKS).

2.1 - Prerequisites

The following prerequisites apply to both resource-management paths. Both paths require access to the OCI registry to install the Helm chart.

• Ensure that the target AKS cluster is accessible and has a supported PPC installation.

  For more information, refer to PPC installation.

• Ensure that the jumpbox can connect to the required registries. If not already authenticated, then log in to the required registry.

• For connecting and deploying from the Protegrity Container Registry (PCR), use the following command and the credentials obtained from the My.Protegrity portal during account creation:

helm registry login registry.protegrity.com:9443

• For connecting and deploying to the local registry, use your local credentials and local registry endpoint as required.

Required Tools

Permissions

The identity used to provision Workbench resources needs the following Azure permissions, irrespective of whether provisioning uses OpenTofu or the existing infrastructure process.

Helm deployment: The installer needs OCI registry access and Kubernetes RBAC to install or upgrade the chart. Helm access does not grant Azure or keystore access.

KMGW runtime: The KMGW pod uses the workbench-kms workload identity to access the configured keystore. Grant this identity only the keystore permissions required by the configured backend. These permissions are separate from those required by the OpenTofu or Helm operator.

2.2 - Installing Policy Workbench

Policy Workbench is installed on an existing, supported PPC installation running on AKS.Prepare the Azure and Kubernetes resources needs using one of two methods:

• Module-managed Workbench resources: Use the provided policy-workbench OpenTofu module to prepare the resources on the existing cluster. The OpenTofu installation procedure depends on whether a root module is available.

  For more information about OpenTofu modules and the root module, refer to Modules in the OpenTofu documentation.

• Self-managed Workbench resources: Prepare and manage those resources through the existing infrastructure process.

  For more information on deploying PPC onto an existing AKS cluster, refer to In-cluster deployment.

Both methods install Policy Workbench with Helm. The following sections describe the resource preparation and installation steps for each method.

Installing Policy Workbench

To install Policy Workbench, select an installation path:

• Module-managed Workbench resources
• Self-managed Workbench resources

Module-managed Workbench resources

Use this path when the policy-workbench OpenTofu module should prepare the Azure and Kubernetes resources required by Policy Workbench on the existing AKS/PPC cluster. After OpenTofu completes, install the Helm chart using the generated Helm values file, policy-workbench-values.yaml.

Module prerequisites

• Ensure that the PPC installed and accessible, before installing Policy Workbench on PPC. For more information about deploying the PPC cluster, refer to Automated script-based installation or Manual component-based installation.
• The policy-workbench OpenTofu module creates a federated identity credential bound to the AKS OIDC issuer URL and the workbench-kms ServiceAccount.

Configuring Credentials for the Policy Workbench

Configure credentials for the approved Protegrity Container Registry using the following instructions:

1. Run the following command to create the configuration directory.

   mkdir -p ~/.config/containers
   

2. Obtain the username and access token from the My.Protegrity portal. For more information about obtaining the credentials from the My.Protegrity portal, refer to the section Configuring Authentication for Protegrity AI Team Edition.

3. Generate base64 encoded string with padding for username:accesstoken obtained from the My.Protegrity portal.

   Ensure that you specify the username and access token within single quotes when generating the base64 encoded value. For example, 'username:accesstoken'.

   Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

4. Create a file named ~/.config/containers/auth.json with the following content.

   {
       "auths": {
           "<Container_Registry_Path>": {
               "auth": "<base64 generated string from step-3>"
           }
       }
   }
   

Follow the instructions for the scenario that matches whether a root module is available. Refer to the appropriate section using the following links:

• Root module is not available.
• Root module is available.

Installing Policy Workbench when root module is not available

Install the Policy Workbench using the following steps:

1. Run the following command to create the deployment directory:

   # must install from an empty directory
   mkdir policy-workbench && cd policy-workbench
   

2. Create a root module with a single main.tf file:

   terraform {
     required_version = "~> 1.10"
   
     required_providers {
       azurerm = {
         source  = "registry.opentofu.org/hashicorp/azurerm"
         version = "~> 5.7"
       }
       kubernetes = {
         source  = "registry.opentofu.org/hashicorp/kubernetes"
         version = "~> 2.35"
       }
     }
   }
   
   provider "azurerm" {
     features {}
     resource_provider_registrations = "none"
   }
   
   data "azurerm_kubernetes_cluster" "this" {
     name                = var.cluster_name
     resource_group_name = var.resource_group_name
   }
   
   provider "kubernetes" {
     host                   = data.azurerm_kubernetes_cluster.this.kube_config[0].host
     client_certificate     = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_certificate)
     client_key             = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_key)
     cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].cluster_ca_certificate)
   }
   
   module "policy_workbench" {
     source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/azure?tag=<version>"
   
     cluster_name                    = var.cluster_name
     oci_host                        = "<Container_Registry_Path>"
     resource_group_name             = var.resource_group_name
   
     azure_keystore = {
       type                = "key_vault"
       name                = var.key_vault_name
       resource_group_name = var.key_vault_resource_group_name
     }
   
     fips                            = var.fips
     arch                            = var.arch
     instance_types                  = var.instance_types
     # Defaults:
     # fips = true
     # arch = "arm64"
     #
     # Default ARM64 node type:
     # instance_types = ["Standard_D8ps_v5"]
     #
     # AMD64 deployment:
     # arch = "amd64"
     # instance_types = ["Standard_D8as_v5"]
   }
   
   variable "cluster_name"                    { type = string }
   variable "resource_group_name"             { type = string }
   variable "key_vault_name"                  { type = string }
   
   variable "key_vault_resource_group_name" {
     type    = string
     default = ""
   }
   
   variable "fips" {
     type        = bool
     description = "Use FIPS-enabled nodes for Karpenter. Set to false for standard mode."
     default     = true
   }
   
   variable "instance_types" {
     type    = list(string)
     default = ["Standard_D8ps_v5"]
   }
   
   variable "arch" {
     type        = string
     description = "CPU architecture for Karpenter nodes. Use 'amd64' or 'arm64'."
     default     = null
   
     validation {
       condition     = var.arch == null || contains(["amd64", "arm64"], var.arch)
       error_message = "arch must be 'amd64' or 'arm64'."
     }
   }
   
   output "helm_install" {
     value = module.policy_workbench.helm_install
   }
   

   To use an existing Managed HSM instead of a Standard or Premium Key Vault, replace the keystore block with the following code block:

   azure_keystore = {
     type                = "managed_hsm"
     name                = "<managed-hsm-name>"
     resource_group_name = "<hsm-resource-group-name>" # omit if same as resource_group_name
   }
   

In the main.tf file, specify the values of the following parameters:

3. Configure OCI registry credentials as described in Configuring Credentials for the Policy Workbench.

4. Run the following commands to plan and install the Policy Workbench OpenTofu module:

   tofu init
   
   # Default deployment (ARM64 + FIPS)
   tofu plan \
     -var="cluster_name=<cluster-name>" \
     -var="resource_group_name=<aks-resource_group_name>" \
     -var="key_vault_name=<key-vault-name>"
   
   tofu apply \
     -var="cluster_name=<cluster-name>" \
     -var="resource_group_name=<aks-resource_group_name>" \
     -var="key_vault_name=<key-vault-name>"
   

   To deploy on amd64 (x86) nodes instead, set arch and instance_types in both the tofu plan and tofu apply commands:

   # AMD64 deployment (add -var="fips=false" to disable FIPS)
   tofu plan \
     -var="cluster_name=<cluster-name>" \
     -var="resource_group_name=<aks-resource_group_name>" \
     -var="key_vault_name=<key-vault-name>" \
     -var="arch=amd64" \
     -var='instance_types=["Standard_D8as_v5"]'
   
   tofu apply \
     -var="cluster_name=<cluster-name>" \
     -var="resource_group_name=<aks-resource_group_name>" \
     -var="key_vault_name=<key-vault-name>" \
     -var="arch=amd64" \
     -var='instance_types=["Standard_D8as_v5"]'
   

   If the Key Vault is in a different resource group, include the following variable:

   -var="key_vault_resource_group_name=<key-vault-resource-group-name>"
   

   To override the default Karpenter instance type, include the following variable:

   -var='instance_types=["<instance-type>"]'
   

   To override the default secure mode, include the following variable:

   -var="fips=false"
   

   For the values of cluster_name, resource_group_name, key_vault_name, fips, arch, instance_types, and key_vault_resource_group_name, refer to the parameter table in step 2.

   OpenTofu prints the plan and prompts for confirmation. Enter yes to proceed. To skip the prompt, add the -auto-approve option to the commands.

After completing this procedure, refer to Deploy Policy Workbench with Helm to continue with the installation.

Installing Policy Workbench when root module is available

Install the Policy Workbench using the following steps:

Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

1. Add the following module block to the existing root module:

   module "policy_workbench" {
     source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/azure?tag=<version>"
   
     cluster_name                    = "<aks-cluster-name>"
     oci_host                        = "<Container_Registry_Path>"
     resource_group_name             = "<resource-group>"
   
     azure_keystore = {
       type                = "key_vault"
       name                = "<key-vault-name>"
       resource_group_name = "<key-vault-resource-group>" # omit if same as resource_group_name
     }
     # Optional overrides; set these module arguments before planning:
     # fips = false
     # arch = "amd64"
     # instance_types = ["Standard_D8as_v5"]
   }
   

   To use an existing Managed HSM instead of a Standard or Premium Key Vault, replace the keystore block with the following code block:

   azure_keystore = {
     type                = "managed_hsm"
     name                = "<managed-hsm-name>"
     resource_group_name = "<hsm-resource-group-name>" # omit if same as resource_group_name
   }
   

   For more information about adding a module to an existing root module, refer to Module Blocks in the OpenTofu documentation.

   Set the value of version in the main.tf file as specified in the parameters table.

2. If the root module does not include the following provider versions, add them to the terraform {} block:

   • hashicorp/azurerm version ~> 5.7
   • hashicorp/kubernetes version ~> 2.35

   required_providers {
     azurerm = {
       source  = "registry.opentofu.org/hashicorp/azurerm"
       version = "~> 5.7"
     }
     kubernetes = {
       source  = "registry.opentofu.org/hashicorp/kubernetes"
       version = "~> 2.35"
     }
   }
   

3. If the root module does not configure the Azure provider with resource_provider_registrations = "none", add the following code block to avoid requiring subscription-level registration permissions:

   provider "azurerm" {
     features {}
     resource_provider_registrations = "none"
   }
   

4. Configure the Kubernetes provider to connect to the target AKS cluster by adding the following code block:

   data "azurerm_kubernetes_cluster" "this" {
     name                = "<aks-cluster-name>"
     resource_group_name = "<resource-group>"
   }
   
   provider "kubernetes" {
     host                   = data.azurerm_kubernetes_cluster.this.kube_config[0].host
     client_certificate     = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_certificate)
     client_key             = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_key)
     cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].cluster_ca_certificate)
   }
   

5. Configure OCI registry credentials as described in Configuring Credentials for the Policy Workbench.

6. Navigate to the directory containing the root module.

7. Replace the cluster, resource group, and keystore placeholders in the data and module blocks. To change architecture, VM size, or FIPS mode, edit the corresponding module arguments before planning.

8. Run the following commands from the root-module directory:

   tofu init
   tofu plan
   tofu apply
   

   OpenTofu prints the plan and prompts for confirmation. Enter yes to proceed. To skip the prompt, add the -auto-approve option to the commands.

After completing this procedure, refer to Deploy Policy Workbench with Helm to continue with the installation.

Self-managed Workbench resources

Use this path when the existing infrastructure process manages Workbench resources instead of the policy-workbench OpenTofu module. This path does not require OpenTofu commands. After completing the prerequisites and configuration below, install the chart using the shared Helm procedure.

Self-managed prerequisites

• Ensure that the target AKS cluster has a supported PPC installation and is accessible. For more information on deploying PPC onto an existing AKS cluster, refer to In-cluster PPC deployment.
• Ensure that Azure CLI access and permissions are available to configure a user-assigned managed identity, federated identity credential, and Key Vault or Managed HSM role assignment.
• Identify the Key Vault or Managed HSM to use and the StorageClass and node scheduling requirements for the Workbench workloads.

Self-managed node resources and scheduling

If nodes are provisioned manually instead of using the provided Karpenter modules, size them to cover the Policy Workbench footprint.

Where Karpenter is configured, it can provision additional nodes as workloads need them. Otherwise, ensure the cluster has enough capacity to run Policy Workbench and other workloads.

Azure KMS Backend Resources for KMGW

The following resources must be provisioned manually:

• User-assigned managed identity.
• Federated identity credential.
• Key Vault or Managed HSM role assignment.

The following table lists the required role assignment for each keystore type:

The Helm chart expects kmgw to run as the ServiceAccount configured by keystore.serviceAccount.name.

Configuring Persistent Storage

Policy Workbench uses the default StorageClass of the cluster unless pvc.storageClassName is set. Verify the StorageClasses available in the target AKS cluster before installation.

1. Run the following command to list the StorageClasses in the cluster:

   kubectl get storageclass
   

2. If the cluster has no default StorageClass, or the default is not suitable for Policy Workbench, set the StorageClass name in the Helm values file, policy-workbench-values.yaml.

   pvc:
     storageClassName: "<storage-class-name>"
   

   For example, PPC normally creates a StorageClass named pty-sc.

   Note: Without a suitable default StorageClass, the Policy Workbench PVCs remain unbound. Due to which the StatefulSet pods cannot be scheduled. Retain this setting in the policy-workbench-values.yaml file used for installation and future upgrades.

Configuring Node Selection for Policy Workbench Pods

By default, Kubernetes can schedule Policy Workbench pods on any available node. To pin Policy Workbench pods to specific nodes in the target AKS cluster, perform the following steps:

1. Run the following command to apply the policy-workbench label to each node where Policy Workbench pods will be scheduled:

   kubectl label node <node-name> nodepool.protegrity.com/name=policy-workbench
   

   Replace <node-name> with the name of the target node. Repeat the command for each node.

2. In the policy-workbench-values.yaml file, add a nodeSelector entry under scheduling that matches the label applied in the previous step:

   scheduling:
     nodeSelector:
       nodepool.protegrity.com/name: policy-workbench
   

3. If the target nodes have taints applied, apply a PreferNoSchedule taint and a matching toleration.

   Important: Use PreferNoSchedule, not NoSchedule. Velero backup and restore pods do not inherit the tolerations set for Policy Workbench. A NoSchedule taint on the workbench nodes blocks Velero restores.

   a. Run the following command to apply the taint on each target node:

   kubectl taint node <node-name> workload=policy-workbench:PreferNoSchedule
   

   b. Add the matching toleration under scheduling.tolerations in the policy-workbench-values.yaml file. The key, value, and effect must match the taint applied to the node:

   scheduling:
     tolerations:
       - key: "workload"
         operator: "Equal"
         value: "policy-workbench"
         effect: "PreferNoSchedule"
   

Applying Scheduling to Individual Services

To apply a different scheduling policy to a specific service instead of all services, add a scheduling block under the key for that service.

For example:

rpproxy:
  replicas: 3
  scheduling:
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: rpproxy

Here, the Resilient Package Proxy or rpproxy, which delivers packages to Protectors, is the only service whose replica count can be scaled. By default, rpproxy runs with one replica. This example increases the rpproxy replica count from one to three. Add a topologySpreadConstraint to spread the replicas across nodes, so that a single node failure does not take down every replica:

Note: When increasing the replica count, ensure the cluster has sufficient capacity to meet the higher minimum resource requirements.

The whenUnsatisfiable field accepts one of the following values:

• ScheduleAnyway: Spreads pods on a best-effort basis without blocking scheduling if the constraint cannot be satisfied.
• DoNotSchedule: Leaves a pod in Pending state rather than placing it on a node that violates the spread constraint.

Configuring the AKS Backend and Helm Values

On the jump box, perform the following steps to set up the KMGW managed identity and configure the policy-workbench-values.yamlfile:

1. Set the environment variables used by the following commands:

   export RESOURCE_GROUP="<resource-group>"
   export CLUSTER_NAME="<cluster-name>"
   export LOCATION="<region>"           # for example, eastus
   export IDENTITY_NAME="aks-${CLUSTER_NAME}-policy-workbench-kmgw-identity"
   export NAMESPACE="policy-workbench"
   export SERVICE_ACCOUNT="workbench-kms"
   export KEY_VAULT_NAME="<key-vault-name>" # set for the Key Vault branch
   export HSM_NAME="<managed-hsm-name>"      # set for the Managed HSM branch
   

2. If OIDC Issuer and Workload Identity are not enabled on the AKS cluster, run the following command to enable them:

   az aks update \
     --resource-group "$RESOURCE_GROUP" \
     --name "$CLUSTER_NAME" \
     --enable-oidc-issuer \
     --enable-workload-identity
   

   Warning: Enabling OIDC Issuer and Workload Identity triggers an AKS cluster update. Once enabled, these features cannot be disabled.

3. Run the following command to create the user-assigned managed identity and capture its client and principal IDs:

   az identity create \
     --name "$IDENTITY_NAME" \
     --resource-group "$RESOURCE_GROUP" \
     --location "$LOCATION"
   
   export IDENTITY_CLIENT_ID=$(az identity show -g "$RESOURCE_GROUP" -n "$IDENTITY_NAME" --query clientId -o tsv)
   export IDENTITY_PRINCIPAL_ID=$(az identity show -g "$RESOURCE_GROUP" -n "$IDENTITY_NAME" --query principalId -o tsv)
   

4. Run the following command to retrieve the AKS OIDC issuer URL:

   export OIDC_ISSUER=$(az aks show -g "$RESOURCE_GROUP" -n "$CLUSTER_NAME" --query oidcIssuerProfile.issuerUrl -o tsv)
   

5. Run the following command to bind the federated identity credential to the Helm release namespace and ServiceAccount:

   az identity federated-credential create \
     --name "aks-${CLUSTER_NAME}-policy-workbench-kmgw-federated" \
     --identity-name "$IDENTITY_NAME" \
     --resource-group "$RESOURCE_GROUP" \
     --issuer "$OIDC_ISSUER" \
     --subject "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT}" \
     --audience api://AzureADTokenExchange
   

   The --subject and --audience values must remain exactly as shown.

6. Choose the role-assignment branch for the configured keystore.

   Azure Key Vault Standard or Premium

   Run the following commands to assign the managed identity the Key Vault Crypto Officer role and retrieve the vault URI:

   export KEY_VAULT_ID=$(az keyvault show -g "$RESOURCE_GROUP" -n "$KEY_VAULT_NAME" --query id -o tsv)
   
   az role assignment create \
     --assignee-object-id "$IDENTITY_PRINCIPAL_ID" \
     --assignee-principal-type ServicePrincipal \
     --role "Key Vault Crypto Officer" \
     --scope "$KEY_VAULT_ID"
   
   export KEYSTORE_URI=$(az keyvault show -g "$RESOURCE_GROUP" -n "$KEY_VAULT_NAME" --query properties.vaultUri -o tsv)
   
   echo "Managed identity client ID: $IDENTITY_CLIENT_ID"
   echo "Key Vault URI: $KEYSTORE_URI"
   

   Azure Managed HSM

   Run the following commands to assign the managed identity the Managed HSM Crypto User role and retrieve the HSM URI:

   az keyvault role assignment create \
     --hsm-name "$HSM_NAME" \
     --assignee-object-id "$IDENTITY_PRINCIPAL_ID" \
     --role "Managed HSM Crypto User" \
     --scope /
   
   export KEYSTORE_URI=$(az keyvault show --hsm-name "$HSM_NAME" --query properties.hsmUri -o tsv)
   
   echo "Managed identity client ID: $IDENTITY_CLIENT_ID"
   echo "Managed HSM URI: $KEYSTORE_URI"
   

7. Add the keystore settings to policy-workbench-values.yaml using the IDENTITY_CLIENT_ID and KEYSTORE_URI captured in the selected branch:

   keystore:
     backend: azure
     serviceAccount:
       name: workbench-kms
     azure:
       vaultUri: <key-vault-or-managed-hsm-uri>
       clientId: <managed-identity-client-id>
   

   For Azure Key Vault Standard or Premium only, enable software-generated random bytes by adding softwareRandomEnabled: true under keystore.azure:

   keystore:
     backend: azure
     serviceAccount:
       name: workbench-kms
     azure:
       vaultUri: <key-vault-uri>
       clientId: <managed-identity-client-id>
       softwareRandomEnabled: true
   

   Warning: Azure Key Vault Standard and Premium do not provide random bytes from a KMS or HSM backend. When using Key Vault, kmgw uses software-generated random bytes. Use Managed HSM if random bytes must come from an HSM-backed service.

Deploy Policy Workbench with Helm

Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

After completing either resource-preparation path, install Policy Workbench with Helm.

• Self-managed resources: Configure the policy-workbench-values.yaml file with the storage, scheduling, and keystore settings described in Self-managed workbench resources.
• Module-managed resources: The OpenTofu module generates the policy-workbench-values.yaml file with the required resource settings. The tofu output -raw helm_install command prints a ready-to-run Helm command with the version and namespace values filled in.

Set <version> to the Policy Workbench version being installed. If the namespace was overridden in the OpenTofu module, use the same namespace in the Helm command. Retain the values file for future upgrades.

1. Run the following command to complete the installation of the Policy Workbench using Helm:

   helm upgrade --install policy-workbench \
     oci://<Container_Registry_Path>/policy-workbench/helm/policy-workbench \
     --version <version> \
     --namespace policy-workbench \
     --create-namespace \
     --values policy-workbench-values.yaml
   

   In the command, specify the value of the following variable.

   Variable Name	Description	Value
   <version>	Tag version of the Policy Workbench, as specified in the product part number. Obtain the product part number from the Policy Workbench Readme.	1.13.1

2. Run the following command to view the pods created in the policy-workbench namespace:

   kubectl get pods -n policy-workbench
   

   The following output appears:

   NAME                        READY   STATUS    RESTARTS   AGE
   bootstrap-bffb4b5d9-v6ww4   1/1     Running   0          13m
   cert-7b88dcd84-zx7cv        1/1     Running   0          13m
   devops-75755d87d4-qw9n6     1/1     Running   0          13m
   hubcontroller-0             1/1     Running   0          13m
   kmgw-0                      1/1     Running   0          13m
   mbs-6b7dc765dd-brrfk        1/1     Running   0          13m
   repository-0                1/1     Running   0          13m
   rpproxy-79fc498d8-qp4fz     1/1     Running   0          13m
   rps-8d79b7d98-svhdw         1/1     Running   0          13m
   

   The number of rpproxy pods matches rpproxy.replicas if that value is overridden. Immediately after installation, the short-lived hubcontroller-init-* init job and kmgw-create-keystore-* pod may also appear.

Validating the deployment

Before validating the deployment of the Policy Workbench, ensure that the kubectl context is set to the target PPC cluster.

Note: The curl examples below assume that the server’s TLS certificate can be successfully verified. Do not use -k or --insecure in production, as these options disable certificate verification. For services using a private or self-signed CA, distribute the CA certificate to the client through a separate trusted channel and either install it in the client’s trusted certificate store or specify the CA certificate directly using curl --cacert.

Run kubectl config current-context to verify the current context. Run kubectl config use-context <context-name> to switch the context.

After installation, validate the Policy Workbench deployment using the following steps.

1. Run the following commands to retrieve the gateway host details and generate an admin JWT token:

Note: The admin user must change the default password on first login. Replace <password> with the new admin password.

export GW_HOST="$(kubectl get gateway pty-main -n api-gateway -o jsonpath='{.status.addresses[0].value}')"

TOKEN=$(curl --silent --show-error --dump-header - --output /dev/null "https://$GW_HOST/pty/v1/auth/login/token" \
  -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'loginname=admin' \
  -d 'password=<password>' | grep -i 'pty_access_jwt_token:' | sed 's/pty_access_jwt_token: //' | tr -d '\r')

2. Create a workbench user using the following command:

curl --silent --show-error -X POST "https://$GW_HOST/pty/v1/auth/users" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "username": "workbench",
    "password": "<password>",
    "roles": [
      "workbench_administrator",
      "security_administrator"
    ]
  }'

Note: The password policy requires a minimum of 14 characters, including at least one digit, one uppercase letter, one lowercase letter, and one special character.

3. Run the following command to get a token for the workbench user:

TOKEN=$(curl --silent --show-error --dump-header - --output /dev/null "https://$GW_HOST/pty/v1/auth/login/token" \
  -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'loginname=workbench' \
  -d 'password=<password>' | grep -i 'pty_access_jwt_token:' | sed 's/pty_access_jwt_token: //' | tr -d '\r')

4. Run the Policy Management REST API to get datastores:

curl --silent --show-error "https://$GW_HOST/pty/v2/pim/datastores" \
  -H "Authorization: Bearer $TOKEN"

The expected output is []. This indicates that the Policy Workbench is initialized but the datastore is not yet created.

For more information about the workbench_administrator permissions, refer to Workbench Roles and Permissions.

3 - Upgrade

This section describes how to upgrade Policy Workbench on a Protegrity Provisioned Cluster (PPC) running on Azure Kubernetes Service (AKS) using OpenTofu and Helm.

3.1 - Upgrading Policy Workbench in Azure

Before you begin

• Back up the existing Policy Workbench data.

  For more information, refer to Backing up the Policy Workbench for Azure.

• Before upgrading, ensure the existing managed identity, federated identity credential, and Key Vault role assignment are tracked in OpenTofu state.

  • If any of these resources are untracked, import them with tofu import before running tofu apply. This prevents recreation conflicts.

Update to Configure credentials for the Policy Workbench

Perform the following steps to configure the credentials to access the Policy Workbench from the OCI registry.

1. Run the following command to create the configuration directory.

   mkdir -p ~/.config/containers
   

2. Obtain the username and access token from the My.Protegrity portal. For more information about obtaining the credentials from the My.Protegrity portal, refer to the section Configuring Authentication for Protegrity AI Team Edition.

3. Generate base64 encoded string with padding for username:accesstoken obtained from the My.Protegrity portal.

   Ensure that you specify the username and access token within single quotes when generating the base64 encoded value. For example, 'username:accesstoken'.

   Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

4. Create a file named ~/.config/containers/auth.json with the following content.

   {
       "auths": {
           "<Container_Registry_Path>": {
               "auth": "<base64 generated string from step-3>"
           }
       }
   }
   

Upgrade the Policy Workbench deployment

To upgrade the Policy Workbench, refer to the appropriate section using the following links:

• Root module was not available during installation
• Root module was available during installation

Root module was not available during installation

If no root module was available during installation, use a standalone root module for the upgrade. Start with the main.tf configuration using the following steps:

1. Navigate to the deployment directory.

2. Copy the main.tf contents from step 2 of Installing Policy Workbench when root module is not available.

   Note: The new main.tf includes a kubernetes provider and additional module arguments, including oci_host, fips, arch, and instance_types, that were not present in earlier versions.

3. Replace the contents of the existing main.tf with the copied content.

4. In main.tf, set the following parameter values:

   Parameter	Description	Value
   <version>	Tag version of the Policy Workbench, as listed in the Policy Workbench README.	1.13.1
   <cluster_name>	The name of the PPC cluster specified during deployment.	The value used for the cluster name when deploying PPC.
   resource_group_name	The name of the resource group containing the AKS cluster.	The resource group used when deploying PPC.
   azure_keystore.type	Type of keystore used.	Specify one of the following values: key_vault - Use Standard or Premium Azure Key Vault. Protegrity supports 3072-bit RSA software-backed Master Key type. The Master Key uses the RSA-OAEP-256 algorithm to encrypt the Data Encryption Key (DEK). managed_hsm - Use Azure Key Vault Managed HSM. Protegrity supports 256-bit AES oct-HSM HSM-backed Master Key type. The Master Key uses the AES-256-CBC algorithm to encrypt the Data Encryption Key (DEK). For more information about the Azure Key Vault, refer to About Azure Key Vault in the Azure documentation. For more information about the Azure Key Vault Managed HSM, refer to What is Azure Key Vault Managed HSM? in the Azure documentation. Note: Protegrity supports only software-protected keys for the Premium Key Vault. Protegrity does not support HSM-protected keys for the Premium Vault. For more information about software-protected and HSM-protected keys, refer to the Azure documentation. Important: You cannot migrate among the Standard Key Vault, Premium Key Vault, and Managed HSM.
   azure_keystore.name	The name of the existing Standard or Premium Key Vault or Managed HSM.	Specify one of the following values: var.key_vault_name - Specify this variable if a Standard or Premium Key Vault is used. Set the value of this variable when running the tofu apply command in the Apply the Configuration section. The name of the existing Managed HSM.
   azure_keystore.resource_group_name	Resource group of the Standard or Premium Key Vault or Managed HSM.	Specify one of the following values: var.key_vault_resource_group_name - Specify this value only if a Standard or Premium Key Vault is used. Set the value of this variable when running the tofu apply command in the Apply the Configuration section. Name of the resource group for the existing Managed HSM. Specify this parameter only if the name of the resource group for the Managed HSM differs from the value of the resource_group_name variable.
   fips	Use a secure VM image for Karpenter nodes.	The default value is true. Set the value to false for standard nodes.
   arch	CPU architecture for Karpenter nodes.	Uses the CPU architecture from installation.
   instance_types	Azure VM sizes for Karpenter nodes.	Uses the VM sizes from installation.
   key_vault_name	Name of the Azure Key Vault used to store the keystore.	The Key Vault name specified during deployment.
   key_vault_resource_group_name	Resource group containing the Key Vault, if different from resource_group_name.	Defaults to "", which uses resource_group_name.

For more information about cluster_name, refer to Deploying PPC.

4. Ensure that the credentials to install the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to Configuring credentials for the Policy Workbench.

Root module was available during installation

If a root module was available during installation, follow these steps to update its policy_workbench module block and ensure that the required provider and Kubernetes cluster configuration are present in the root module:

1. Navigate to the root module directory.

2. Locate the policy_workbench module block in the .tf files.

3. Update the module block with the following:

   Note: The oci_host, fips, arch, and instance_types arguments are new in this version. If the existing module block does not include them, add them along with the updated source value.

   module "policy_workbench" {
      source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/azure?tag=<version>"
   
      cluster_name        = var.cluster_name
      oci_host            = "<Container_Registry_Path>"
      resource_group_name = var.resource_group_name
   
      azure_keystore = {
        type                = "key_vault"
        name                = var.key_vault_name
        resource_group_name = var.key_vault_resource_group_name
     }
   
     fips           = var.fips
     arch           = var.arch
     instance_types = var.instance_types
   }
   

Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

1. In the policy_workbench module block, set the parameter values as specified in the parameter table.

2. If the root module does not include the following provider versions, add them to the terraform {} block:

   • hashicorp/azurerm version ~> 4.20
   • hashicorp/kubernetes version ~> 2.35

   required_providers {
     azurerm = {
       source  = "registry.opentofu.org/hashicorp/azurerm"
       version = "~> 4.20"
     }
     kubernetes = {
       source  = "registry.opentofu.org/hashicorp/kubernetes"
       version = "~> 2.35"
     }
   }
   

3. Configure the kubernetes provider to connect to the target AKS cluster by adding the following to the root module:

   data "azurerm_kubernetes_cluster" "this" {
    name                = var.cluster_name
    resource_group_name = var.resource_group_name
   }
   
   provider "kubernetes" {
     host                   = data.azurerm_kubernetes_cluster.this.kube_config[0].host
     client_certificate     = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_certificate)
     client_key             = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_key)
     cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].cluster_ca_certificate)
   }
   

4. Ensure that the credentials to update the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to Configuring credentials for the Policy Workbench.

Apply the configuration

Navigate to the directory containing the configured main.tf file. Run the following steps from that directory to initialize the OpenTofu providers and apply the configuration:

1. Run the following command to initialize the OpenTofu providers:

   tofu init -upgrade
   

2. Run the following commands to preview and apply the OpenTofu changes:

   tofu plan \
     -var="cluster_name=<cluster-name>" \
     -var="resource_group_name=<resource_group_name>" \
     -var="key_vault_name=<key_vault_name>"
   
   tofu apply \
     -var="cluster_name=<cluster-name>" \
     -var="resource_group_name=<resource_group_name>" \
     -var="key_vault_name=<key_vault_name>"
   

   OpenTofu displays the plan and prompts for confirmation before applying. Enter yes to proceed, or add the -auto-approve flag to the tofu apply command to skip the prompt.

Deploy the Upgrade with Helm

Run the following command to upgrade Policy Workbench:

helm upgrade --install policy-workbench \
  oci://<Container_Registry_Path>/policy-workbench/helm/policy-workbench \
  --version <version> \
  --namespace policy-workbench \
  --values policy-workbench-values.yaml

Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint. Set the value of <version> as specified in the parameter table.

The policy-workbench-values.yaml file is generated by the OpenTofu module in the working directory during the previous step. If the file is missing, re-run the tofu apply command from the Apply the Configuration step, with the same -var values, before proceeding.

If the upgrade succeeds, Helm displays the following output:

NOTES:
Protegrity policy-workbench 1.13.1 installed

The version number reflects the upgraded version.

Verify the upgrade

Confirm that the Helm release and pods are running correctly.

1. Run the following command to verify the Helm release status:

   helm list -n policy-workbench
   

   The output appears as follows:

   NAME                NAMESPACE           REVISION    UPDATED                                     STATUS      CHART                       APP VERSION
   policy-workbench    policy-workbench    4           2026-09-08 05:50:45.367488574 +0000 UTC     deployed    policy-workbench-1.13.0    1.13.1
   

   The REVISION number increments with each upgrade.

2. Run the following command to verify the Policy Workbench pods are running:

   kubectl get pods -n policy-workbench
   

   The output appears as follows:

   NAME                         READY   STATUS    RESTARTS   AGE
   bootstrap-7dcff9dd5f-v6p8s   1/1     Running   0          16h
   cert-84f768f744-6vzqd        1/1     Running   0          16h
   devops-6dc846566-crglt       1/1     Running   0          16h
   hubcontroller-0              1/1     Running   0          17h
   hubcontroller-init-fcrww     1/1     Running   0          16h
   kmgw-0                       1/1     Running   0          17h
   kmgw-create-keystore-9c27h   1/1     Running   0          16h
   mbs-d88d59965-lcwsh          1/1     Running   0          16h
   repository-0                 1/1     Running   0          16h
   rpproxy-5c8ddbff69-hncbx     1/1     Running   0          16h
   

   All pods must show 1/1 under READY and Running under STATUS. Pod names and ages may differ across environments.

If the Helm release status is deployed and all pods meet these criteria, the Policy Workbench upgrade is complete.

Validate the Deployment

Confirm that the Policy Workbench API is accessible after the upgrade.

1. Ensure kubectl is configured to use the target AKS cluster:

   az aks get-credentials --resource-group <resource_group_name> --name <cluster-name> --overwrite-existing
   

2. Run the following command to retrieve the gateway address:

   export GW_HOST="$(kubectl get gateway pty-main -n api-gateway -o jsonpath='{.status.addresses[0].value}')"
   

3. Run the following command to obtain an authentication token for the workbench user:

   export TOKEN=$(curl -k -s https://$GW_HOST/pty/v1/auth/login/token \
     -X POST \
     -H 'Content-Type: application/x-www-form-urlencoded' \
     -d 'loginname=workbench' \
     -d 'password=<workbench-password>' \
     -D - -o /dev/null 2>&1 | grep -i 'pty_access_jwt_token:' | sed 's/pty_access_jwt_token: //' | tr -d '\r')
   

   Set <workbench-password> to the workbench user password specified during deployment. Confirm that $TOKEN is not empty before proceeding.

4. Run the following command to verify access to the Policy Workbench datastores API:

   curl -k -v https://$GW_HOST/pty/v2/pim/datastores -H "Authorization: Bearer $TOKEN"
   

The response should list any existing datastores. If no datastores have been created, the response should be an empty array [].

4 - Backup and Restore

This section describes how to back up Policy Workbench data on Azure and restore it from an existing backup.

4.1 - Backing up the Policy Workbench

By default, the Policy Workbench data is backed up on a daily basis using a scheduled backup, after the Policy Workbench has been installed. The backed-up data includes the Kubernetes object state and the persistent volume data. The backed-up data is automatically stored in the encrypted Azure Blob Storage container that you created when you deployed PPC.

For more information about the Azure Blob Storage container, refer to Creating Azure Storage Account and Container.

You can also choose to manually back up the data to the Azure Blob Storage container using Velero.

Important: Before you manually back up the data, ensure that Velero CLI version 1.18 or later is installed.

To manually back up the data:

1. Run the following command on the jump box:

   velero backup create --from-schedule workbench-backup-schedule -n <Namespace where data is backed up>
   

   For example:

   velero backup create --from-schedule workbench-backup-schedule -n pty-backup-recovery
   

   The following output appears:

   INFO[0000] No Schedule.template.metadata.labels set - using Schedule.labels for backup object  backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 labels="map[app.kubernetes.io/managed-by:Helm deployment:policy-workbench]"
   Creating backup from schedule, all other filters are ignored.
   Backup request "workbench-backup-schedule-20260702112543" submitted successfully.
   Run `velero backup describe workbench-backup-schedule-20260702112543` or `velero backup logs workbench-backup-schedule-20260702112543` for more details.
   

   For more information about the velero backup command, refer to Backup Reference in the Velero documentation.

2. Run the following commands to monitor the backup status.

   1. Run the following command to retrieve the list of existing backups:

      velero backup get -n pty-backup-recovery
      

      The following output appears:

      NAME                                               STATUS                       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
      workbench-backup-schedule-20260702112543           InProgress                    0        0          2026-07-02 11:25:43 +0000 UTC   59d       default            <none>
      

   2. Run the following command to obtain details of a specific backup:

      velero backup describe <backup-name> -n pty-backup-recovery
      

      The following code block shows a snippet of the output:

      Name:         workbench-backup-schedule-20260702112543
      Namespace:    pty-backup-recovery
      Labels:       app.kubernetes.io/managed-by=Helm
                  deployment=policy-workbench
                  velero.io/schedule-name=workbench-backup-schedule
                  velero.io/storage-location=default
      Annotations:  meta.helm.sh/release-name=policy-workbench
                  meta.helm.sh/release-namespace=policy-workbench
                  velero.io/resource-timeout=10m0s
                  velero.io/source-cluster-k8s-gitversion=v1.35.5
                  velero.io/source-cluster-k8s-major-version=1
                  velero.io/source-cluster-k8s-minor-version=35
      
      Phase:  InProgress
      

   3. Run the following command to obtain the log details for a specific backup:

      velero backup logs <backup-name> -n pty-backup-recovery
      

      The following code block shows a snippet of the output:

      time="2026-03-31T09:47:38Z" level=info msg="Setting up backup temp file" backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 logSource="pkg/controller/backup_controller.go:690"
      time="2026-03-31T09:47:38Z" level=info msg="Setting up plugin manager" backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 logSource="pkg/controller/backup_controller.go:697"
      time="2026-03-31T09:47:38Z" level=info msg="Getting backup item actions" backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 logSource="pkg/controller/backup_controller.go:701"
      

      Note: Verify that the log output contains no entries with level=error. Entries with level=warning are acceptable and do not indicate a backup failure.

Validating the backup

1. Run the following command to validate the backup status:

   velero backup get -n pty-backup-recovery
   

   The following output appears:

   NAME                                               STATUS                       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
   workbench-backup-schedule-20260702112543           Completed                    0        0          2026-07-02 11:25:43 +0000 UTC   59d       default            <none>
   

4.2 - Restoring the Policy Workbench

Before you begin

Before starting a restore, ensure that the following prerequisites are met:

• An existing backup is available. Backups are taken automatically as part of the default installation of the Policy Workbench using scheduled backup mechanisms. The backups are available in the encrypted Azure Blob Storage container that is created by the PPC installation. Backups can also be created manually.

  For more information about the Azure Blob Storage container, refer to Creating Azure Storage Account and Container.

  For more information about manually backing up the data, refer to Backing up the Policy Workbench for Azure.

• A restored PPC cluster is required for both module-managed and self-managed Workbench resources. For information about restoring the PPC, refer to Restoring the PPC on Azure.

Important: Before you restore the data, ensure that Velero CLI version 1.18 or later is installed.

Ensure that the workbench-restore-resource-modifier ConfigMap exists in the pty-backup-recovery namespace.

Restore the Policy Workbench

To restore Policy Workbench, refer to the appropriate section using the following links:

• Module-managed Workbench resources
• Self-managed Workbench resources

Do not install the Helm release before the Velero restore. Velero restores the backed-up Workbench resources.

Module-managed Workbench resources

1. Ensure that the main.tf file in the root module, which is the working directory for executing the OpenTofu commands, contains the following code block. If the root module is not available, create a root module with the main.tf file.

   module "policy_workbench" {
     source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/azure?tag=<version>"
   
     cluster_name = var.cluster_name
   }
   
   variable "cluster_name" {
     type        = string
     description = "AKS cluster name."
     nullable    = false
   
     validation {
       condition     = length(trimspace(var.cluster_name)) > 0
       error_message = "cluster_name must be provided and cannot be empty."
     }
   }
   

   Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

   This code block adds the Policy Workbench OpenTofu module.

2. Run the following commands on the jump box to provision the Workbench resources on the destination PPC cluster:

   tofu init
   tofu plan -var="cluster_name=<Restored-cluster-name>"
   tofu apply -var="cluster_name=<Restored-cluster-name>"
   

   Set cluster_name to the name of the restored PPC cluster.

After completing these steps, continue to Trigger and monitor the restore.

Self-managed Workbench resources

Self-managed prerequisites

• The destination AKS cluster and required Azure resources are provisioned outside OpenTofu.
• The destination managed identity exists and has the required Key Vault or Managed HSM permissions.
• The identity has a federated credential with subject system:serviceaccount:policy-workbench:workbench-kms and an issuer matching the destination cluster’s OIDC issuer URL.

Configure the Azure restore modifier

A Velero ResourceModifier is a set of rules stored in a ConfigMap that Velero applies to Kubernetes resources during restore. During restore, it updates the workbench-kms ServiceAccount annotation with the destination managed identity client ID and removes stale selected-node annotations from restored PVCs, allowing them to be scheduled on the destination cluster.

Create the workbench-restore-resource-modifier ConfigMap before starting the restore.

1. Run the following commands to set the destination cluster and identity details:

   export CLUSTER_NAME="<destination-aks-cluster-name>"
   export RESOURCE_GROUP="<resource-group>"
   export WORKBENCH_NS="policy-workbench"
   export VELERO_NS="pty-backup-recovery"
   export IDENTITY_NAME="aks-${CLUSTER_NAME}-policy-workbench-kmgw-identity"
   

2. Run the following command to retrieve the client ID of the destination managed identity:

   export CLIENT_ID=$(az identity show \
     --name "$IDENTITY_NAME" \
     --resource-group "$RESOURCE_GROUP" \
     --query clientId -o tsv)
   
   echo "Destination client ID: $CLIENT_ID"
   

   If no client ID is returned, create the managed identity before continuing.

3. Run the following commands to verify the federated identity credential matches the destination cluster:

   az identity federated-credential list \
     --identity-name "$IDENTITY_NAME" \
     --resource-group "$RESOURCE_GROUP" \
     --query "[].{issuer:issuer,subject:subject}" -o json
   
   az aks show --name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" \
     --query oidcIssuerProfile.issuerUrl -o tsv
   

   Confirm the subject is system:serviceaccount:policy-workbench:workbench-kms and the credential issuer matches the OIDC issuer URL for the destination cluster.

4. Run the following command to create the ResourceModifier ConfigMap with the destination client ID:

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: workbench-restore-resource-modifier
     namespace: ${VELERO_NS}
     labels:
       velero.io/resource-modifier-config: "true"
       deployment: policy-workbench
   data:
     resource-modifier.yaml: |
       version: v1
       resourceModifierRules:
       - conditions:
           groupResource: serviceaccounts
           resourceNameRegex: "^workbench-kms\\$"
           namespaces:
           - ${WORKBENCH_NS}
         mergePatches:
         - patchData: |
             {"metadata":{"annotations":{"azure.workload.identity/client-id":"${CLIENT_ID}"}}}
       - conditions:
           groupResource: persistentvolumeclaims
           namespaces:
           - ${WORKBENCH_NS}
         mergePatches:
         - patchData: |
             {"metadata":{"annotations":{"volume.kubernetes.io/selected-node":null}}}
   EOF
   

5. Run the following command to verify that the ConfigMap contains the destination client ID:

   kubectl get configmap workbench-restore-resource-modifier -n "$VELERO_NS" \
     -o jsonpath='{.data.resource-modifier\\.yaml}{"\\n"}'
   

After completing these steps, continue to Trigger and monitor the restore.

Trigger and monitor the restore

After completing the destination-cluster setup and verifying the ResourceModifier ConfigMap, run these steps on the jump box:

1. Run the following command to start the Velero restore and wait for completion:

   velero restore create workbench-restore-$(date +%Y%m%d-%H%M%S) \
     --from-backup <backup-name> \
     --resource-modifier-configmap workbench-restore-resource-modifier \
     -n pty-backup-recovery --wait
   

   The following output appears if Velero accepts the restore request:

   Restore request "workbench-restore-20260702112543" submitted successfully.
   Waiting for restore to complete. You may safely press Ctrl+C to stop waiting, but the restore will continue to run in the background.
   

2. Run the following command to check the restore status:

   velero restore get -n pty-backup-recovery
   

   WaitingForPluginOperations means the restore is still in progress. Wait until the status is Completed.

3. Run the following command to remove the last-handled configuration annotation and trigger the controller to process the restored configuration:

   kubectl annotate productconfiguration workbench -n pty-admin kopf.zalando.org/last-handled-configuration- --overwrite
   

   The following output appears when the annotation is removed successfully:

   productconfiguration.pty.com/workbench annotated
   

4. Run the following command to view details of the restore:

   velero restore describe workbench-restore-<timestamp> -n pty-backup-recovery
   

5. Run the following command to view the restore logs:

   velero restore logs workbench-restore-<timestamp> -n pty-backup-recovery
   

Check the logs for entries with level=error. Warnings alone do not indicate restore failure. For more information, refer to Velero Restore Reference.

The restore is successful when Velero reports Completed, the restore logs contain no level=error entries, and the annotation command succeeds.

5 - Uninstallation

To uninstall the Policy Workbench:

1. Run the following command to uninstall the Policy Workbench:

   helm uninstall policy-workbench -n policy-workbench
   

2. Run the following command to clean up the AWS resources:

   tofu destroy -var='cluster_name=<cluster name>'