This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Backup and Restore

Back up and restore Policy Workbench data on Azure.

1: Backing up the Policy Workbench
2: Restoring the Policy Workbench

This section describes how to back up Policy Workbench data on Azure and restore it from an existing backup.

1 - Backing up the Policy Workbench

Back up Policy Workbench data for Azure to an encrypted Azure Blob Storage container using a scheduled or manual Velero backup.

By default, the Policy Workbench data is backed up on a daily basis using a scheduled backup, after the Policy Workbench has been installed. The backed-up data includes the Kubernetes object state and the persistent volume data. The backed-up data is automatically stored in the encrypted Azure Blob Storage container that you created when you deployed PPC.

For more information about the Azure Blob Storage container, refer to Creating Azure Storage Account and Container.

You can also choose to manually back up the data to the Azure Blob Storage container using Velero.

Important: Before you manually back up the data, ensure that Velero CLI version 1.18 or later is installed.

To manually back up the data:

Run the following command on the jump box:

velero backup create --from-schedule workbench-backup-schedule -n <Namespace where data is backed up>

For example:

velero backup create --from-schedule workbench-backup-schedule -n pty-backup-recovery

The following output appears:

INFO[0000] No Schedule.template.metadata.labels set - using Schedule.labels for backup object  backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 labels="map[app.kubernetes.io/managed-by:Helm deployment:policy-workbench]"
Creating backup from schedule, all other filters are ignored.
Backup request "workbench-backup-schedule-20260702112543" submitted successfully.
Run `velero backup describe workbench-backup-schedule-20260702112543` or `velero backup logs workbench-backup-schedule-20260702112543` for more details.

For more information about the velero backup command, refer to Backup Reference in the Velero documentation.

Run the following commands to monitor the backup status.

Run the following command to retrieve the list of existing backups:

velero backup get -n pty-backup-recovery

The following output appears:

NAME                                               STATUS                       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
workbench-backup-schedule-20260702112543           InProgress                    0        0          2026-07-02 11:25:43 +0000 UTC   59d       default            <none>

Run the following command to obtain details of a specific backup:

velero backup describe <backup-name> -n pty-backup-recovery

The following code block shows a snippet of the output:

Name:         workbench-backup-schedule-20260702112543
Namespace:    pty-backup-recovery
Labels:       app.kubernetes.io/managed-by=Helm
            deployment=policy-workbench
            velero.io/schedule-name=workbench-backup-schedule
            velero.io/storage-location=default
Annotations:  meta.helm.sh/release-name=policy-workbench
            meta.helm.sh/release-namespace=policy-workbench
            velero.io/resource-timeout=10m0s
            velero.io/source-cluster-k8s-gitversion=v1.35.5
            velero.io/source-cluster-k8s-major-version=1
            velero.io/source-cluster-k8s-minor-version=35

Phase:  InProgress

Run the following command to obtain the log details for a specific backup:

velero backup logs <backup-name> -n pty-backup-recovery

The following code block shows a snippet of the output:

time="2026-03-31T09:47:38Z" level=info msg="Setting up backup temp file" backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 logSource="pkg/controller/backup_controller.go:690"
time="2026-03-31T09:47:38Z" level=info msg="Setting up plugin manager" backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 logSource="pkg/controller/backup_controller.go:697"
time="2026-03-31T09:47:38Z" level=info msg="Getting backup item actions" backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 logSource="pkg/controller/backup_controller.go:701"

Note: Verify that the log output contains no entries with level=error. Entries with level=warning are acceptable and do not indicate a backup failure.

Validating the backup

Run the following command to validate the backup status:

velero backup get -n pty-backup-recovery

The following output appears:

NAME                                               STATUS                       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
workbench-backup-schedule-20260702112543           Completed                    0        0          2026-07-02 11:25:43 +0000 UTC   59d       default            <none>

2 - Restoring the Policy Workbench

Restore the Policy Workbench data for Azure using an existing backup.

Before you begin

Before starting a restore, ensure that the following prerequisites are met:

An existing backup is available. Backups are taken automatically as part of the default installation of the Policy Workbench using scheduled backup mechanisms. The backups are available in the encrypted Azure Blob Storage container that is created by the PPC installation. Backups can also be created manually.
For more information about the Azure Blob Storage container, refer to Creating Azure Storage Account and Container.
For more information about manually backing up the data, refer to Backing up the Policy Workbench for Azure.
A restored PPC cluster is required for both module-managed and self-managed Workbench resources. For information about restoring the PPC, refer to Restoring the PPC on Azure.

Important: Before you restore the data, ensure that Velero CLI version 1.18 or later is installed.

Ensure that the workbench-restore-resource-modifier ConfigMap exists in the pty-backup-recovery namespace.

Restore the Policy Workbench

To restore Policy Workbench, refer to the appropriate section using the following links:

Do not install the Helm release before the Velero restore. Velero restores the backed-up Workbench resources.

Module-managed Workbench resources

Ensure that the main.tf file in the root module, which is the working directory for executing the OpenTofu commands, contains the following code block. If the root module is not available, create a root module with the main.tf file.

module "policy_workbench" {
  source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/azure?tag=<version>"

  cluster_name = var.cluster_name
}

variable "cluster_name" {
  type        = string
  description = "AKS cluster name."
  nullable    = false

  validation {
    condition     = length(trimspace(var.cluster_name)) > 0
    error_message = "cluster_name must be provided and cannot be empty."
  }
}

Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

This code block adds the Policy Workbench OpenTofu module.

Run the following commands on the jump box to provision the Workbench resources on the destination PPC cluster:
```
tofu init
tofu plan -var="cluster_name=<Restored-cluster-name>"
tofu apply -var="cluster_name=<Restored-cluster-name>"
```
Set cluster_name to the name of the restored PPC cluster.

After completing these steps, continue to Trigger and monitor the restore.

Self-managed Workbench resources

Self-managed prerequisites

The destination AKS cluster and required Azure resources are provisioned outside OpenTofu.
The destination managed identity exists and has the required Key Vault or Managed HSM permissions.
The identity has a federated credential with subject system:serviceaccount:policy-workbench:workbench-kms and an issuer matching the destination cluster’s OIDC issuer URL.

Configure the Azure restore modifier

A Velero ResourceModifier is a set of rules stored in a ConfigMap that Velero applies to Kubernetes resources during restore. During restore, it updates the workbench-kms ServiceAccount annotation with the destination managed identity client ID and removes stale selected-node annotations from restored PVCs, allowing them to be scheduled on the destination cluster.

Create the workbench-restore-resource-modifier ConfigMap before starting the restore.

Run the following commands to set the destination cluster and identity details:

export CLUSTER_NAME="<destination-aks-cluster-name>"
export RESOURCE_GROUP="<resource-group>"
export WORKBENCH_NS="policy-workbench"
export VELERO_NS="pty-backup-recovery"
export IDENTITY_NAME="aks-${CLUSTER_NAME}-policy-workbench-kmgw-identity"

Run the following command to retrieve the client ID of the destination managed identity:

export CLIENT_ID=$(az identity show \
  --name "$IDENTITY_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query clientId -o tsv)

echo "Destination client ID: $CLIENT_ID"

If no client ID is returned, create the managed identity before continuing.

Run the following commands to verify the federated identity credential matches the destination cluster:

az identity federated-credential list \
  --identity-name "$IDENTITY_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query "[].{issuer:issuer,subject:subject}" -o json

az aks show --name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" \
  --query oidcIssuerProfile.issuerUrl -o tsv

Confirm the subject is system:serviceaccount:policy-workbench:workbench-kms and the credential issuer matches the OIDC issuer URL for the destination cluster.

Run the following command to create the ResourceModifier ConfigMap with the destination client ID:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: workbench-restore-resource-modifier
  namespace: ${VELERO_NS}
  labels:
    velero.io/resource-modifier-config: "true"
    deployment: policy-workbench
data:
  resource-modifier.yaml: |
    version: v1
    resourceModifierRules:
    - conditions:
        groupResource: serviceaccounts
        resourceNameRegex: "^workbench-kms\\$"
        namespaces:
        - ${WORKBENCH_NS}
      mergePatches:
      - patchData: |
          {"metadata":{"annotations":{"azure.workload.identity/client-id":"${CLIENT_ID}"}}}
    - conditions:
        groupResource: persistentvolumeclaims
        namespaces:
        - ${WORKBENCH_NS}
      mergePatches:
      - patchData: |
          {"metadata":{"annotations":{"volume.kubernetes.io/selected-node":null}}}
EOF

Run the following command to verify that the ConfigMap contains the destination client ID:

kubectl get configmap workbench-restore-resource-modifier -n "$VELERO_NS" \
  -o jsonpath='{.data.resource-modifier\\.yaml}{"\\n"}'

After completing these steps, continue to Trigger and monitor the restore.

Trigger and monitor the restore

After completing the destination-cluster setup and verifying the ResourceModifier ConfigMap, run these steps on the jump box:

Run the following command to start the Velero restore and wait for completion:

velero restore create workbench-restore-$(date +%Y%m%d-%H%M%S) \
  --from-backup <backup-name> \
  --resource-modifier-configmap workbench-restore-resource-modifier \
  -n pty-backup-recovery --wait

The following output appears if Velero accepts the restore request:

Restore request "workbench-restore-20260702112543" submitted successfully.
Waiting for restore to complete. You may safely press Ctrl+C to stop waiting, but the restore will continue to run in the background.

Run the following command to check the restore status:
```
velero restore get -n pty-backup-recovery
```
WaitingForPluginOperations means the restore is still in progress. Wait until the status is Completed.
Run the following command to remove the last-handled configuration annotation and trigger the controller to process the restored configuration:
```
kubectl annotate productconfiguration workbench -n pty-admin kopf.zalando.org/last-handled-configuration- --overwrite
```
The following output appears when the annotation is removed successfully:
```
productconfiguration.pty.com/workbench annotated
```

Run the following command to view details of the restore:

velero restore describe workbench-restore-<timestamp> -n pty-backup-recovery

Run the following command to view the restore logs:

velero restore logs workbench-restore-<timestamp> -n pty-backup-recovery

Check the logs for entries with level=error. Warnings alone do not indicate restore failure. For more information, refer to Velero Restore Reference.

The restore is successful when Velero reports Completed, the restore logs contain no level=error entries, and the annotation command succeeds.