Backup and Restore

• 1: Backing up the PPC
• 2: Restoring the PPC
• 2.1: Prerequisites for restoring the PPC deployment
• 2.2: Restoring the PPC using the bootstrap script
• 2.3: Restoring the PPC onto an existing AKS cluster
• 3: Restoring the SSH keys and PCT from backed up Terraform state file

1 - Backing up the PPC

Protegrity Provisioned Cluster (PPC) supports backup of Insight indexes and cluster configurations. Use the procedures below to update the scheduled backups of your PPC deployment.

Backing up Velero backup schedule

To take a backup at an interval of 10 minutes, the cron job must be executed before the schedule. This ensures that the latest configurations are included and are available in the Storage Account.
By default, the cron job is executed after every 3 hours, and schedule is executed after every 3.5 hours.

For taking the backups, the frequency of schedule and cron job can be changed, as required.

Perform the following steps to update the frequency.

1. To update the frequency of the cron job, run the following command:

   $ kubectl patch cronjob authnz-postgresql-backup-recovery-cron-job \
   -n api-gateway \'
   --type=merge -p '{"spec":{"schedule":"<required frequency>"}}'
   

2. To update the frequency of the schedule, run the following command:

   $ kubectl patch schedule authnz-postgresql-schedule-backup \   
   -n pty-backup-recovery \
   --type=merge -p '{"spec":{"schedule":"<required frequency>"}}' 
   

For instance, to update the frequency of the backup schedule to 10 minutes, perform the following steps.

1. To update the cron job to every 8 minutes, run the following command:

   $ kubectl patch cronjob authnz-postgresql-backup-recovery-cron-job \
    -n api-gateway \
    --type=merge -p '{"spec":{"schedule":"8-58/10 * * * *"}}'
   

2. To update the schedule to every 10 minutes, run the following command:

   $ kubectl patch schedule authnz-postgresql-schedule-backup \
    -n pty-backup-recovery \
    --type=merge -p '{"spec":{"schedule":"*/10 * * * *"}}'
   

Backing up indexes

Insight indexes are backed up automatically using a scheduled job. To manually back up the indexes, complete the following steps:

1. Log in to the Insight Dashboard.
2. Select the main menu.
3. Navigate to Management > Snapshot Management > Snapshot policies.
4. Click the daily-insight-snapshots policy.
5. Click Edit.
6. Update the following parameters:
   • Snapshot schedule
     • Frequency: Custom (Cron expression)
     • Cron expression: */15 * * * *
7. Click Update.
8. Wait for the next scheduled snapshot to complete. Verify that the snapshot is created from the snapshot status.

Note: The snapshot schedule is set to run every 15 minutes for creating the manual backup. Restore the default settings after the manual backup is complete. Before upgrading, disable the scheduler job after the backup is created. Ensure that the default parameters are restored and the job is re-enabled after the upgrade.

2 - Restoring the PPC

This section provides the steps to restore a Protegrity Provisioned Cluster (PPC) deployment from an existing backup in Microsoft Azure. Before you begin, ensure that the backup is available in the original Azure Storage Account and that you have access to the resources required by the selected restore method.

The PPC deployment can be restored using one of the following methods:

• Restore using the bootstrap script: Use this method to restore PPC while creating a new AKS cluster. The bootstrap script provisions the cluster and restores PPC from the selected backup.

• Restore onto an existing AKS cluster: Use this method when the target AKS cluster and its supporting resources are already provisioned. The restore procedure deploys PPC components onto the existing cluster without creating a new cluster.

Choose the restore method that matches your target environment, then complete the prerequisites and follow the procedure for that method.

2.1 - Prerequisites for restoring the PPC deployment

Before you begin

Before starting a restore, ensure the following conditions are met:

• An existing backup is available. Backups are taken automatically as part of the default installation using scheduled backup mechanisms. These backups are stored in a Storage Account configured during the original installation.

• Access to the original backup Storage Account. During restore, the same Storage Account that was used during the original installation must be specified.

• Permissions to read from the Storage Account. The user performing the restore must have sufficient permissions to access the backup data stored in the account.

• Use the restore procedure that matches your target environment. Restoring the PPC using the bootstrap script restores as part of a fresh AKS cluster installation. Restoring the PPC onto an existing AKS cluster restores PPC onto an existing AKS cluster. The in-cluster procedure does not create the AKS cluster.

• Ensure that the storage_class_name in iac/azure/c luster/modules/05_helm_releases/terraform.tfvars file is the same for the targeted restore cluster and the source PPC cluster. The backup is created on the source PPC cluster. If this value differs, the restore operation fails.

• Do not perform Create, Read, Update, or Delete (CRUD) operations on the source cluster during the backup. This ensures backup consistency and prevents data corruption during restore.

• Before restoring to a new cluster, if the source cluster is accessible, disable the backup operations. Do this by setting the backup storage location to read‑only. This ensures that no additional backup data is written during the restore process.

  To disable the backup operation on the source cluster, run the following command:

  kubectl patch backupstoragelocation default -n pty-backup-recovery --type merge -p '{"spec":{"accessMode":"ReadOnly"}}'
  

  If the source cluster is not accessible, this step can be skipped.

Required tools on the jump box

The restore bootstrap script installs or updates the following tools on the jump box. If you install tools manually, ensure these are available before running the script:

• Azure CLI - Required to communicate with your Microsoft Azure account.
• OpenTofu - Required to manage infrastructure as code.
• kubectl - Required to communicate with the Kubernetes cluster.
• Helm - Required to manage Kubernetes packages.
• jq - Required to parse JSON.
• skopeo - Required to copy container images between registries.
• bc - Required for version comparisons and resource calculations.
• ORAS - Required to pull OCI artifacts such as deployment packages.
• ssh-keygen - Required to secure SSH access to deployment environments.
• cmctl - Required to validate and manage TLS certificates.

During Disaster management, the backup data is used to restore the cluster and the OpenSearch indexes using snapshots. However, Insight restores OpenSearch data only from the most recent snapshot created by the daily-insight-snapshots policy.
For more information, refer to Backing up and restoring indexes.

Warning: Do not install or manage multiple clusters from the same working directory. Each cluster deployment maintains its own Terraform/OpenTofu state. Reusing a directory can overwrite state files, causing loss of cluster tracking and unintended cleanup behavior.
Use a dedicated directory and jump box per cluster, where possible. Always verify the active kubectl context before running cleanup commands such as ./bootstrap.sh --cloud azure destroy.

2.2 - Restoring the PPC using the bootstrap script

Before you begin

The restore bootstrap script checks that you have the required Azure permissions. It installs or updates the tools listed in Restore prerequisites. It also creates the AKS cluster and deploys PPC services as part of the restore.

Note: Before running the bootstrap as the root user on RHEL, ensure that /usr/local/bin is included in the $PATH. Also include the Azure CLI binary path, if applicable. Alternatively, run the script using a non-root user. The default PATH for a non-root user already includes /usr/local/bin.

Architecture Selection

• amd64 (x86-64-based environments such as Standard_D8as_v5)
• arm64 (ARM-based environments such as Standard_D8ps_v5)

The bootstrap script prompts you to select the node architecture interactively during deployment (step 1b).

Restoring the PPC

Ensure that the PCT is downloaded and extracted on the jump box. For more information, refer to Preparing for PPC deployment.

Run the following command to initiate restore using an existing backup:

./bootstrap.sh --cloud azure --restore

The script prompts for the following variables.

1. Enter AKS Cluster Name [opencluster-v1]

   The following characters are allowed:

   • Lowercase letters: a-z
   • Numbers: 0-9
   • Hyphens: -

   The following characters are not allowed:

   • Uppercase letters: A-Z
   • Underscores: _
   • Spaces
   • Any special characters such as: / ? * + % ! @ # $ ^ & ( ) = [ ] { } : ; , .
   • Leading or trailing hyphens
   • Names must only contain up to 10 characters

   While creating a cluster, the cluster name must:

   • Start with a lowercase letter.
   • End with a letter or number.
   • Not contain consecutive hyphens.

   Note: Ensure that the cluster name does not exceed 10 characters. Cluster names longer than this limit can cause the bootstrap script to fail in subsequent installation steps.
   

   After the cluster name is accepted, the script prompts you to select the node architecture.

   [1b] Select Node Architecture

   Select the required architecture.

   [1b] Select Node Architecture
     1) amd64 (x86_64) — e.g. Standard_D8as_v5
     2) arm64 (ARM/Ampere) — e.g. Standard_D8ps_v5
   

   Enter 1 to select amd64 or 2 to select arm64.

   Note: Select amd64 for Standard_D8as_v5 environments. Select arm64 for ARM-based environments such as Standard_D8ps_v5.

2. Querying for available Resource Groups

   The script queries for the available Resource Groups.

   Enter the Resource Group name from the table []

   The script then automatically detects the location and subscription ID of the resource group.

3. Enter UAMI Resource ID

   Provide the complete Azure resource ID for the UAMI used by AKS in the following format:

   /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<identity-name>
   

   The UAMI client ID is detected automatically.

4. Enter AKS Subnet Resource ID

   Provide the complete resource ID of the pre-existing subnet used for AKS nodes in the following format:

   /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/<vnet-name>/subnets/<subnet-name>
   

5. Enter Private DNS Zone Resource ID

   Provide the Private DNS zone ID used by the AKS private cluster in the following format:

   /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Network/privateDnsZones/privatelink.<region>.azmk8s.io
   

   The script attempts to automatically detect network settings:

   • Virtual network address space
   • Service CIDR
   • Pod CIDR
   • DNS service IP

6. Enter FQDN [name.domain.com]

   This is the Fully Qualified Domain Name for the ingress.

   While entering the FQDN, ensure that:

   • contains at least two labels separated by a dot, such as app.example.com
   • does not start or end with a dot or hyphen
   • does not contain consecutive dot character
   • label length must be between 1–63 characters
   • each label must start and end with a letter or digit
   • the last label must start only with a letter

   Warning: Ensure that the FQDN does not exceed 253 characters and only the following characters are used:

   • Lowercase letters: a-z
   • Numbers: 0-9
   • Special characters: - .

7. Storage Account and Key Vault provisioning

   During restore, the script prompts for existing Storage Account and Key Vault configuration. Provide the existing values in this step.

   Storage Account & Key Vault provisioning
   
   Restore mode requires existing Storage Account and Key Vault configuration.
   Skipping provisioning choice; provide existing values in this step.
   
   Enter existing Storage Account Name:
   <bucket-name>
   Storage account '<bucket-name>' found in resource group '<resource-group>'.
   Using container: velero
   
   Enter existing Key Vault Name:
   <key-vault-name>
   

   During restore, the script prompts to manually select a backup from the available backups stored in the Storage Account. User input is required to either restore from the latest backup or choose a specific backup from the list.

   [restore] Backup selection — schedule: authnz-postgresql-schedule-backup
   Restore from latest backup? [Y/n]
   

   • Enter Y to restore from the most recent backup.
   • Enter n to manually select a backup.

   If n is selected, then the script displays a list of available backups (latest first) and prompts to select one by number:

   Available backups (latest first):
   [1] <timestamp> authnz-postgresql-schedule-backup-xxxxxxx
   [2] <timestamp> authnz-postgresql-schedule-backup-xxxxxxx
   Select a backup number (1-2):
   

   After entering the backup number, the chosen backup is used for the restore, and the installation continues.

   Enter Insight backup

   During restore, the script prompts to manually select the required insight backup from the available backups stored in the Storage Account. User input is required to either restore from the latest backup or choose a specific backup from the list.

   [restore] Insight backup selection — schedule: insight-configmaps-secrets
   Restore from latest backup? [Y/n]
   

   • Enter Y to restore from the most recent backup.
   • Enter n to manually select a backup.

   If n is selected, then the script displays a list of available backups (latest first) and prompts to select one by number:

   Available backups (latest first):
   [1] <timestamp>  insight-configmaps-secrets-xxxxxxxxx
   [2] <timestamp>  insight-configmaps-secrets-xxxxxxxxx
   Select a backup number (1-2):
   

   After entering the backup number, the chosen backup is used for the restore, and the installation continues.

8. Enter Velero UAMI Resource ID

   Enter the resource ID of the dedicated Velero User-Assigned Managed Identity (UAMI) in the format:

   /subscriptions/<subscription-id>/resourceGroups/<velero-resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<name>
   

   The script automatically validates the UAMI and detects the UAMI Client ID.

9. Enter OpenSearch UAMI Resource ID

   Enter the resource ID of the OpenSearch Velero User-Assigned Managed Identity (UAMI) in the format:

   /subscriptions/<subscription-id>/resourceGroups/<opensearch-resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<name>
   

   The script automatically validates the UAMI and detects the UAMI Client ID.

10. Enter Image Registry Endpoint

Enter <Container_Registry_Path>, the container registry endpoint from where the container images are retrieved.

Note: If deploying from the Protegrity Container Registry, use registry.protegrity.com:9443. Specify the local registry endpoint if a local registry is used.

Expected format: <hostname>[:port].

Do not include ‘https://’

11. Enter Registry Username

Enter the username for the registry mentioned in the previous step. Leave this entry blank if the registry does not require authentication.

12. Enter Registry Password or Access Token

Enter Password or Access Token for the registry.

Input is masked with * characters.

Leave this entry blank if the registry does not require authentication.

13. Enter Owner Email (for cluster tagging)

Enter email address in the format name@domain.tld. This email is used for the cluster ‘Owner’ tag and the platform owner contact.

Note: The cluster creation process can take approximately 15-20 minutes.

If the session is terminated during installation, the installation stops. Causes include network issues, a power outage, and similar events. To restart the installation, run the script again:

# Navigate to deployment directory and run the following command
./bootstrap.sh --cloud azure destroy

# Restore the cluster
./bootstrap.sh --cloud azure --restore

After the bootstrap script is completed, verify the cluster and workloads using the following commands:

# Confirm nodes are Ready
kubectl get nodes

# Confirm NFA workloads are Running
kubectl get pods -A

2.3 - Restoring the PPC onto an existing AKS cluster

Before you begin

Use this procedure to restore PPC from an existing backup onto an existing AKS cluster. The cluster and its supporting resources must already be provisioned; this procedure does not create an AKS cluster. Confirm that the backup is available in the original Azure Storage Account and that the target cluster can access it. Also confirm that storage_class_name matches the storage class used on the source cluster. Review the in-cluster deployment prerequisites before proceeding.

Finding the backup names in Azure Storage

Before editing the restore settings, find the backup names in the Azure portal:

1. Sign in to the Azure portal with an account that can access the backup Storage Account.
2. Open the Azure Storage Account that contains the backups.
3. Go to Data storage > Containers > velero, then open the Velero/backups folder.

   Note: velero is the default storage container name. If a different container name was configured when the source cluster was created, use that name in the Azure portal. Set storage_container_name to the same value in the restore configuration.

4. Choose the Velero backup for the restore point you want and record its full name, including the timestamp. The name follows this pattern: authnz-postgresql-schedule-backup-<timestamp>.
5. Record the corresponding Insight backup name if you are restoring Insight data. It follows this pattern: insight-configmaps-secrets-<timestamp>. The timestamp varies by backup; copy the exact name shown in the portal.

Use the recorded names for backup_name and insight_backup_name in the terraform.tfvars restore configuration. Replace the placeholders with the exact names from the portal, including their timestamps.

Restoring PPC using in-cluster deployment

Complete the following steps in order. Follow the in-cluster deployment links for Steps 1 through 7, and 9 and 10. For Step 8, use the restore-specific instructions on this page below the table instead of navigating to the in-cluster deployment instructions.

Updating terraform.tfvars for restore

To update the terraform.tfvars file, perform the following steps:

1. Navigate to the Helm releases module directory.

cd iac/azure/cluster/modules/05_helm_releases

2. Set the following variables in the terraform.tfvars file.

Note: Registry credentials are passed as environment variables (TF_VAR_username and TF_VAR_password) and are not set directly in terraform.tfvars.

3. Add the following variables to terraform.tfvars before allowing Stage 05 to apply.

run_readiness_check = true
check_prev_stage    = false

These variables ensure that PPC performs a readiness check on the existing AKS environment. They skip the previous automated stage validation, since those resources were created outside the installer.

4. Set the restore values in the same file.

restore             = true
backup_name         = "authnz-postgresql-schedule-backup-<timestamp>"
insight_backup_name = "insight-configmaps-secrets-<timestamp>"

5. Save the file and exit the editor.

6. Continue with Step 9: Running the readiness check and Step 10: Applying the OpenTofu configuration. Review the plan and confirm that it targets the intended cluster and backup before applying it.

3 - Restoring the SSH keys and PCT from backed up Terraform state file

This section describes the procedure to rebuild a usable PPC v1.1.0 deployment workspace on a jump box from an existing cluster and a backed up Terraform state file.

Before you Begin

Ensure the following requirements are met.

• The state file from all the following modules must be available on the Storage Account.

  • 00_backup_bucket
  • 01_iam_roles
  • 02_eks_cluster
  • 03_node_group
  • 04_eks_addons
  • 05_helm_releases

• Access to the same Microsoft Azure account and target AKS cluster.

• Azure CLI credentials with required permissions.

Perform the following steps to recover PPC

1. Create a workspace.

   To create a working directory for PCT, run the following command:

   mkdir -p deployment
   cd deployment
   

2. Download and extract the PCT

   Download the PCT package into the deployment directory and extract it.
   For more information about downloading and extracting the PCT, refer to Preparing for PPC deployment.

   After successfully extracting the template, this directory behaves as recovered installation workspace.

3. Login to Microsoft Azure as a Service Principal

   To login to Microsoft Azure as a Service Principal, run the following command

   az login -u <username> -p <password> -t <tenant-id>
   

4. Install required dev tools

   Run the following script:

   cd deployment/bootstrap-scripts
   ./setup-devtools-azure.sh
   

5. Initialize OpenTofu

   To initialize OpenTofu for the cluster module, run the following command,

   cd deployment/iac/azure/cluster/modules/05_helm_releases
   tofu init
   

6. Generate the SSH key for CLI access

   To generate the SSH key for CLI access

   tofu output -raw user_svc_private_key > ~/.ssh/user_svc.pem
   chmod 600 ~/.ssh/user_svc.pem
   

7. Configure kubectl to communicate with the AKS cluster

   To configure kubectl to communicate with the AKS cluster, run the following command:

   az aks get-credentials \
   --resource-group aks-ppc \
   --name           base-test
   

8. Verify the nodes and pods status

   To verify the nodes and pods status, run the following command:

   kubectl get nodes
   kubectl get pods -A