Back Up and Restore Policy

Use this procedure as the platform operator and database maintenance owner. Policy keeps PostgreSQL inside its stack. A backup protects Policy’s database, not changes made in other products. Agree on acceptable data loss and a maintenance window before recovery.

Note: Earlier Amazon Web Services (AWS) restore drills used one shared database identity. They do not qualify the current release’s three database identities. Azure evidence covers mocked chart checks only. Engineering or Quality Assurance must qualify these command sequences on your target platform before production use.

Before You Begin

• Use Bash, Helm, kubectl, and jq with access to the intended cluster.
• Install the Velero command-line interface when using Velero snapshots.
• Obtain permission to manage Policy Jobs, workloads, and backup resources.
• Preserve the original maintenance, runtime, and backup usernames and credential Secrets.
• Keep the original chart version, configuration, signing keys, and dependency credentials in protected recovery storage.
• Coordinate other maintenance sessions. Suspending a CronJob does not stop existing Jobs or prevent another operator from creating one.

Run the examples in one dedicated Bash shell. The defaults below match a standard installation; adjust names for custom deployments. Set POLICY_REGISTRY_HOST to the chart registry used during installation. Use the chart version from the installation or recovery record, not a newer release.

set -euo pipefail
set +x
umask 077
CTX="$(kubectl config current-context)"
POLICY_NAMESPACE='pty-policy'
VELERO_NAMESPACE='pty-backup-recovery'
VELERO_SCHEDULE='policy-db-backup'
BACKUP_PVC='policy-db-backup-pvc'
CHART="oci://${POLICY_REGISTRY_HOST:?Set the chart registry host}/policy/helm/policy"
CHART_VERSION='1.1.0'
kubectl --context "$CTX" cluster-info
helm --kube-context "$CTX" list --namespace "$POLICY_NAMESPACE" --filter '^policy$'

Confirm the displayed cluster and installed chart match your recovery plan. Do not continue against another cluster. For disaster recovery, use the protected configuration and credentials from the source installation.

Understand the Backup Boundary

The chart writes a logical Structured Query Language (SQL) dump to a separate persistent volume claim (PVC). The default location is policy-db-backup-pvc, containing pg_dump.sql. It does not contain PostgreSQL’s live data directory. Each successful dump replaces the previous file only after staging succeeds. A failed dump leaves the previous file intact, so file presence does not establish freshness.

Velero snapshots this dump claim, not the complete application or dependency environment. It does not include database credential Secrets or external product state. An older snapshot contains the dump captured then, not an arbitrary point in database history. Without Velero or another approved off-node copy, a retained local dump does not protect against storage loss.

Backup Settings

Record nondefault settings in your protected customer values file. The following settings control dump and snapshot scheduling:

These settings select or configure recovery storage and replay:

Use the protected configuration again when resuming normal operation. The snapshot offset does not guarantee that a dump has finished. Choose schedules, capacity, and retention for your deployment. Changing a storage-class value does not migrate an existing claim. The claim’s Helm retention annotation does not protect against explicit deletion or storage loss.

Check Automatic Backups

Inspect the configured schedules and recent dump Jobs:

kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
  get cronjob policy-db-backup-cronjob
kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" get jobs,pvc
kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" get jobs \
  -l app.kubernetes.io/component=db-backup --sort-by=.metadata.creationTimestamp

In normal mode, expect SUSPEND to show False, the dump claim to show Bound, and recent scheduled Jobs to complete. Select the exact recent Job name from that output:

DUMP_JOB='replace-with-the-recent-job-name'
kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
  get job "$DUMP_JOB" -o json |
  jq '{name: .metadata.name, start: .status.startTime,
       completed: .status.completionTime, conditions: .status.conditions}'
kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" logs job/"$DUMP_JOB" --all-containers=true

Expect a Complete condition with status: "True" and a recent completion time. Investigate a failed Job or an unexpectedly old completion time. An older successful Job does not prove that the latest scheduled dump succeeded.

When Velero is enabled, inspect its Schedule and completed backups:

kubectl --context "$CTX" --namespace "$VELERO_NAMESPACE" \
  get schedules.velero.io "$VELERO_SCHEDULE"
kubectl --context "$CTX" --namespace "$VELERO_NAMESPACE" get backups.velero.io \
  -l "velero.io/schedule-name=$VELERO_SCHEDULE" -o json |
  jq -r '.items | sort_by(.status.completionTimestamp // "")[] |
    [.metadata.name, .status.phase, (.status.errors // 0),
     (.status.completionTimestamp // "pending")] | @tsv'

Expect the Schedule to remain unpaused and recent backups to report Completed with zero errors. Correlate the selected snapshot with the preceding successful dump. Do not assume the thirty-minute schedule offset establishes that relationship.

Take a Backup Now

This sequence creates a new logical dump using the chart’s existing CronJob template. It does not create database credentials or run the destructive development test scripts. Run it only in normal mode, with backups enabled and no other maintenance session.

1. Save normal-mode values privately before changing scheduling.

   RECOVERY_DIR="$(mktemp -d "$HOME/.policy-recovery.XXXXXX")"
   NORMAL_VALUES="$RECOVERY_DIR/normal-values.json"
   helm --kube-context "$CTX" get values policy \
     --namespace "$POLICY_NAMESPACE" --all --output json > "$NORMAL_VALUES"
   jq -e '.global.restore == false and .backup.enabled == true' "$NORMAL_VALUES"
   USE_VELERO="$(jq -er '.backup.veleroEnabled | tostring' "$NORMAL_VALUES")"
   

   Expect true from the configuration check. The private values file can contain sensitive configuration. Helm accepts this JSON file through its --values option. Retain it and the exact chart version in approved recovery storage outside the cluster. Do not include it in support logs or source control.

2. Confirm both schedules currently run normally, then pause them.

   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     get cronjob policy-db-backup-cronjob -o json |
     jq -e '(.spec.suspend // false) == false'
   if [[ "$USE_VELERO" == true ]]; then
     kubectl --context "$CTX" --namespace "$VELERO_NAMESPACE" \
       get schedules.velero.io "$VELERO_SCHEDULE" -o json |
       jq -e '(.spec.paused // false) == false'
   fi
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     patch cronjob policy-db-backup-cronjob --type=merge \
     -p '{"spec":{"suspend":true}}'
   if [[ "$USE_VELERO" == true ]]; then
     kubectl --context "$CTX" --namespace "$VELERO_NAMESPACE" \
       patch schedules.velero.io "$VELERO_SCHEDULE" --type=merge \
       -p '{"spec":{"paused":true}}'
   fi
   

   If a schedule was already paused, stop and identify its maintenance owner. Pausing does not cancel an active dump or snapshot.

3. Wait for existing dump Jobs and Velero backups to finish. Check for nonterminal dump Jobs:

   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" get jobs \
     -l app.kubernetes.io/component=db-backup -o json |
     jq -e '[.items[] | select(
       any(.status.conditions[]?;
         (.type == "Complete" or .type == "Failed") and .status == "True") | not
     )] | length == 0'
   

   Expect true before starting another dump. If false, inspect the listed Jobs and wait for them to finish. Also wait for any active Velero backup shown in the previous section. Do not run overlapping dump writers against the shared staging filename.

4. Create and wait for a uniquely named dump Job.

   DUMP_JOB="policy-db-manual-$(date -u +%Y%m%d%H%M%S)"
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" create job "$DUMP_JOB" \
     --from=cronjob/policy-db-backup-cronjob
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     wait --for=condition=complete job/"$DUMP_JOB" --timeout=10m
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     logs job/"$DUMP_JOB" -c policy-db-backup
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" get job "$DUMP_JOB" \
     -o jsonpath='{.metadata.name}{"\t"}{.status.startTime}{"\t"}{.status.completionTime}{"\n"}'
   

   Expect this new Job to complete after its recorded start time. The inherited command replaces pg_dump.sql only after a successful dump. Completion of this exact Job therefore confirms a fresh dump write. Stop if it fails or times out; do not treat the previous file as the requested backup.

5. If Velero is enabled, capture this dump in a named recovery snapshot. Keep dump scheduling paused until the snapshot finishes.

   if [[ "$USE_VELERO" == true ]]; then
     BACKUP_NAME="policy-db-manual-$(date -u +%Y%m%d%H%M%S)"
     velero --kubecontext "$CTX" --namespace "$VELERO_NAMESPACE" \
       backup create "$BACKUP_NAME" --from-schedule "$VELERO_SCHEDULE" --wait
     kubectl --context "$CTX" --namespace "$VELERO_NAMESPACE" \
       get backups.velero.io "$BACKUP_NAME" -o json |
       jq -e '.status.phase == "Completed" and (.status.errors // 0) == 0'
     velero --kubecontext "$CTX" --namespace "$VELERO_NAMESPACE" \
       backup describe "$BACKUP_NAME" --details
   fi
   

   Expect true and review any warnings before accepting the checkpoint. Record the exact backup name and completion time, not just the Schedule name. Without Velero, this procedure creates only the dump on the existing claim.

6. Resume the schedules after the backup succeeds.

   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     patch cronjob policy-db-backup-cronjob --type=merge \
     -p '{"spec":{"suspend":false}}'
   if [[ "$USE_VELERO" == true ]]; then
     kubectl --context "$CTX" --namespace "$VELERO_NAMESPACE" \
       patch schedules.velero.io "$VELERO_SCHEDULE" --type=merge \
       -p '{"spec":{"paused":false}}'
   fi
   

   Recheck both schedules. If any step fails, inspect the exact Job or backup and notify the maintenance owner. The schedules remain paused until that owner resolves the failure and explicitly resumes them.

Qualify a Backup Checkpoint

Record the dump Job, its completion time, the exact snapshot name, and the chart version. Retain the original values and credential references separately. A completed snapshot alone does not establish application-level recoverability. Rehearse recovery with representative policy revisions and history. Include signing-key recovery and external-product reconciliation in that rehearsal.

Policy signs ownership markers on Policy Integration Manager (PIM) objects. If signing keys changed after the backup, coordinate with the PIM administrator. For information about signing keys, refer to signing keys.

Restore Prerequisites and Stop Conditions

Restore replaces database contents and can discard changes after the selected dump. Use an approved maintenance window and preserve the current database when possible. Keep the original three usernames and matching credentials available. Use the same chart and application versions as the approved recovery record. Do not use database restore to repair an ordinary connectivity or publication failure.

Stop when the dump’s provenance, storage attachment, credentials, or version compatibility is uncertain. Exclude other database clients, including sessions using the maintenance username. The chart’s session check and advisory lock do not exclude every competing maintenance client.

Restore Sequence and Chart Gates

Choose one recovery source:

• Existing dump volume: replay the current file on the retained backup claim.
• Velero backup: recover a named earlier snapshot into a prepared recovery target.

Both paths keep application processing stopped during replay. The chart omits ordinary bootstrap, suspends scheduled dumps, and pauses Velero scheduling in restore mode. It uses the maintenance identity for one transactional SQL replay.

Set NORMAL_VALUES to the protected original JSON values file before choosing either path. Check that it describes normal operation, then read the original Velero setting:

test -s "${NORMAL_VALUES:?Set the protected original values file}"
jq -e '.global.restore == false and .backup.enabled == true' "$NORMAL_VALUES"
USE_VELERO="$(jq -er '.backup.veleroEnabled | tostring' "$NORMAL_VALUES")"

Expect true from the configuration check. Do not use a values file captured after entering restore mode.

Restore from the Existing Dump Volume

Use the NORMAL_VALUES file saved before recovery overrides. For another shell, set it to the full path of that protected file. Confirm the existing claim contains the intended dump; this path cannot select an older file.

1. Stop scheduled dumps and active dump Jobs before stopping the application programming interface (API). Deleting backup Jobs here terminates in-flight dumps, not stored database contents.

   test -s "${NORMAL_VALUES:?Set the protected original values file}"
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     patch cronjob policy-db-backup-cronjob --type=merge \
     -p '{"spec":{"suspend":true}}'
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" delete jobs \
     -l app.kubernetes.io/component=db-backup --cascade=foreground --wait=true
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     scale deployment/policy-api --replicas=0
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     wait --for=delete pod -l app.kubernetes.io/component=api --timeout=5m
   

   Confirm other maintenance clients have stopped. Do not continue if a dump or database client can still write.

2. Enter restore mode using the original values, disabling Velero for this retained-volume path.

   helm --kube-context "$CTX" upgrade policy "$CHART" --version "$CHART_VERSION" \
     --namespace "$POLICY_NAMESPACE" --reset-values --values "$NORMAL_VALUES" \
     --set global.restore=true --set backup.enabled=true --set backup.veleroEnabled=false \
     --wait --wait-for-jobs --timeout 6h
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     logs job/policy-db-restore-job -c policy-db-restore
   

   Expect the restore Job to complete successfully. The six-hour timeout matches the default replay deadline. If Helm fails, inspect the Job and its logs; keep the API stopped. Do not enable automatic rollback or return to normal mode to hide the failure.

3. Continue with verification and resumption.

Recover a Velero Backup

Use an approved recovery target with the same Policy namespace and no existing dump claim. Prepare its platform dependencies, original database Secrets, and protected normal-mode values first. Recovering storage does not recover those credentials. Do not delete an existing claim or uninstall a healthy release to follow this path. If the target contains retained storage, use the retained-storage handoff.

1. Select the exact completed backup from the source cluster’s available recovery storage. Use the list in Check automatic backups. Select the newest completed snapshot for latest recovery, or an approved older snapshot.

   RESTORE_BACKUP='replace-with-the-approved-backup-name'
   kubectl --context "$CTX" --namespace "$VELERO_NAMESPACE" \
     get backups.velero.io "$RESTORE_BACKUP" -o json |
     jq -e '.status.phase == "Completed" and (.status.errors // 0) == 0'
   test -s "${NORMAL_VALUES:?Set the protected original values file}"
   

   Expect true. Record the selected backup and its captured dump time. Do not use a failed backup or assume its name establishes freshness.

2. Create the recovery release with the source values and selected backup.

   helm --kube-context "$CTX" upgrade --install policy "$CHART" --version "$CHART_VERSION" \
     --namespace "$POLICY_NAMESPACE" --create-namespace \
     --reset-values --values "$NORMAL_VALUES" \
     --set global.restore=true --set backup.enabled=true --set backup.veleroEnabled=true \
     --set-string backup.backupName="$RESTORE_BACKUP"
   

   Do not add --wait-for-jobs in this phase. The SQL replay Job intentionally stays suspended until Velero completes recovery. An existing failed or differently selected policy-db-restore needs operator investigation; do not delete it blindly.

3. Wait for Velero, then check its exact source and error count.

   kubectl --context "$CTX" --namespace "$VELERO_NAMESPACE" \
     wait --for=jsonpath='{.status.phase}'=Completed \
     restores.velero.io/policy-db-restore --timeout=4h
   kubectl --context "$CTX" --namespace "$VELERO_NAMESPACE" \
     get restores.velero.io policy-db-restore -o json |
     jq -e --arg backup "$RESTORE_BACKUP" \
       '.spec.backupName == $backup and .status.phase == "Completed" and (.status.errors // 0) == 0'
   

   Expect true. A failed, partial, or mismatched recovery is not safe for SQL replay.

4. Verify the recovered claim and its ownership.

   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     wait --for=jsonpath='{.status.phase}'=Bound pvc/"$BACKUP_PVC" --timeout=10m
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" get pvc "$BACKUP_PVC" -o json |
     jq -e --arg backup "$RESTORE_BACKUP" --arg ns "$POLICY_NAMESPACE" \
       '.status.phase == "Bound"
        and .metadata.labels["velero.io/backup-name"] == $backup
        and .metadata.labels["velero.io/restore-name"] == "policy-db-restore"
        and .metadata.labels["app.kubernetes.io/backup"] == "policy-db"
        and .metadata.labels["app.kubernetes.io/managed-by"] == "Helm"
        and .metadata.annotations["meta.helm.sh/release-name"] == "policy"
        and .metadata.annotations["meta.helm.sh/release-namespace"] == $ns'
   

   Expect true before allowing replay. For custom backup labels, use the original backup.label settings in this check.

5. Reapply restore mode to release the SQL Job.

   helm --kube-context "$CTX" upgrade policy "$CHART" --version "$CHART_VERSION" \
     --namespace "$POLICY_NAMESPACE" --reuse-values --set global.restore=true \
     --wait --wait-for-jobs --timeout 6h
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     logs job/policy-db-restore-job -c policy-db-restore
   

   Helm checks Velero’s state when rendering; it does not watch for completion automatically. Continue only after SQL replay succeeds.

Verify and Resume Normal Operation

1. Confirm replay completion and inspect restored records before starting the API.

   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     wait --for=condition=complete job/policy-db-restore-job --timeout=10m
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     exec statefulset/policy-database -c policy-database -- sh -ec \
     'PGHOST=127.0.0.1 PGUSER="$PMX_BACKUP_USER" PGPASSWORD="$PMX_BACKUP_PASSWORD" \
      psql -X --no-password -v ON_ERROR_STOP=1 -d "$POSTGRES_DB" \
      -c "SELECT version_num FROM public.alembic_version;" \
      -c "SELECT count(*) AS policies FROM public.policies;" \
      -c "SELECT count(*) AS revisions FROM public.policy_revisions;" \
      -c "SELECT count(*) AS history_events FROM public.policy_lifecycle_events;"'
   

   Compare the schema and known policy/history records with the recovery record. Counts alone do not prove data fidelity. Keep the API stopped if the result differs unexpectedly.

2. Have the PIM, Traverse, and Guardrails owners reconcile changes made after the backup. Restoring SQL cannot reverse those external effects. Do not resume processing when a restored pending request could repeat or reverse an uncertain external change.

3. Clear manual dump suspension, then restore the original normal-mode configuration.

   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     patch cronjob policy-db-backup-cronjob --type=merge \
     -p '{"spec":{"suspend":false}}'
   helm --kube-context "$CTX" upgrade policy "$CHART" --version "$CHART_VERSION" \
     --namespace "$POLICY_NAMESPACE" --reset-values --values "$NORMAL_VALUES" \
     --set global.restore=false --wait --wait-for-jobs --timeout 15m
   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     rollout status deployment/policy-api --timeout=5m
   

   The original values restore the original Velero setting. Do not resume with temporary backup.veleroEnabled=false overrides or blindly reuse recovery values. Normal mode reruns bootstrap and runtime checks before admitting the API.

4. Confirm backup scheduling resumed.

   kubectl --context "$CTX" --namespace "$POLICY_NAMESPACE" \
     get cronjob policy-db-backup-cronjob -o json |
     jq -e '.spec.suspend == false'
   if [[ "$USE_VELERO" == true ]]; then
     kubectl --context "$CTX" --namespace "$VELERO_NAMESPACE" \
       get schedules.velero.io "$VELERO_SCHEDULE" -o json |
       jq -e '.spec.paused == false'
   fi
   

   Expect true for each enabled schedule.

5. Sign in with your own Policy account and open Policy Library. Compare known approved versions, drafts, and history with the selected backup. Then repeat Take a backup now to confirm a new dump and, when enabled, a new snapshot complete.

If recovery fails, collect the exact Job or Restore status and logs. Keep restore mode enabled and use the diagnostic handoff. Do not change credentials, delete storage, or restart ordinary processing merely to obtain healthy pods.

Recovery Limits

SQL replay cannot reverse PIM, Traverse, or Guardrails effects after the backup. A restored publication receipt does not prove current runtime protection. Use reconciliation and recovery with the integration owners. Do not infer a supported downgrade or recovery-time guarantee from a successful replay.

Recover after Uninstall

Use this path when uninstallation removed workloads but retained database storage. This is not a fresh installation or a routine release change. A generic reinstall against retained storage is not a qualified customer procedure. Contact your platform/database recovery owner or Protegrity Support before reattaching retained claims. Provide claim identities, original chart and image versions, protected configuration references, and the selected recovery checkpoint. Do not send credential values.

Healthy retained live data may not need SQL replay. Replaying an older dump can discard newer state. Do not run a fresh installation against retained storage or regenerate its credentials. If the shared identity-provider Secret needs recovery, follow the backfill guidance.

After an approved recovery, repeat installation verification. For normal setup, return to Installing Policy.