Backup and Restore

• 1: Backing up the PPC
• 2: Restoring the PPC
• 2.1: Prerequisites for restoring the PPC deployment
• 2.2: Restoring the PPC using the bootstrap script
• 2.3: Restoring the PPC using in-cluster deployment
• 3: Restoring the SSH keys and PCT from backed up Terraform state file

1 - Backing up the PPC

Protegrity Provisioned Cluster (PPC) supports backup of Insight indexes and cluster configurations. Use the procedures below to update the scheduled backups of your PPC deployment.

Backing up Velero backup schedule

To take a backup at an interval of 10 minutes, the cron job must be executed before the schedule. This ensures that the latest configurations are included and are available in the Storage Account.
By default, the cron job is executed after every 3 hours and schedule is executed after every 3.5 hours.

For taking the backups, the frequency of schedule and cron job can be changed, as required.

Perform the following steps to update the frequency.

1. To update the frequency of the cron job, run the following command:

   $ kubectl patch cronjob authnz-postgresql-backup-recovery-cron-job \
   -n api-gateway \'
   --type=merge -p '{"spec":{"schedule":"<required frequency>"}}'
   

2. To update the frequency of the schedule, run the following command:

   $ kubectl patch schedule authnz-postgresql-schedule-backup \   
   -n pty-backup-recovery \
   --type=merge -p '{"spec":{"schedule":"<required frequency>"}}' 
   

For instance, to update the frequency of the backup schedule to 10 minutes, perform the following steps.

1. To update the cron job to every 8 minutes, run the following command:

   $ kubectl patch cronjob authnz-postgresql-backup-recovery-cron-job \
    -n api-gateway \
    --type=merge -p '{"spec":{"schedule":"8-58/10 * * * *"}}'
   

2. To update the schedule to every 10 minutes, run the following command:

   $ kubectl patch schedule authnz-postgresql-schedule-backup \
    -n pty-backup-recovery \
    --type=merge -p '{"spec":{"schedule":"*/10 * * * *"}}'
   

Backing up indexes

Insight indexes are backed up automatically using a scheduled job. To manually back up the indexes, perform the following steps:

1. Log in to the Insight Dashboard.
2. Select the main menu.
3. Navigate to Management > Snapshot Management > Snapshot policies.
4. Click the daily-insight-snapshots policy.
5. Click Edit.
6. Update the following parameters:
   • Snapshot schedule
     • Frequency: Custom (Cron expression)
     • Cron expression: */15 * * * *
7. Click Update.
8. Wait for the next scheduled snapshot to complete. Verify that the snapshot is created from the snapshot status.

Note: The snapshot schedule is set to run every 15 minutes for creating the manual backup. Restore the default settings after the manual backup is complete. Before upgrading, disable the scheduler job after the backup is created. Ensure that the default parameters are restored and the job is re-enabled after the upgrade.

2 - Restoring the PPC

The Protegrity Provisioned Cluster PPC is the core framework that forms the AI Team Edition. It is designed to deliver a modern, cloud-native experience for data security and governance. Built on Kubernetes, PPC uses a containerized architecture that simplifies deployment and scaling. Using OpenTofu scripts and Helm charts, administrators can create clusters with minimal manual intervention, ensuring consistency and reducing operational overhead.

2.1 - Prerequisites for restoring the PPC deployment

Before starting a restore, ensure the following conditions are met:

• A Kubernetes cluster is created. Restore is performed as part of creating a cluster, not on an existing one. Restore is only supported during a fresh installation flow.

• An existing backup is available. Backups are taken automatically as part of the default installation using scheduled backup mechanisms. These backups are stored in an AWS S3 bucket configured during the original installation.

• Access to the original backup AWS S3 bucket. During restore, the same S3 bucket that was used during the original installation must be specified.

• Permissions to read from the S3 bucket. The user performing the restore must have sufficient permissions to access the backup data stored in the bucket.

• While the backup is taken from the source cluster, do not perform Create, Read, Update, or Delete (CRUD) operations on the source cluster. This ensures backup consistency and prevents data corruption during restore.

• Before initiating the restore, review, and update the KMS key policy to reflect the restore cluster name. Even if the policy was already configured for the source cluster, it must be updated for the new restore cluster. If the policy continues to reference the source cluster name, the IAM role created during restore cannot decrypt the backup data, causing the restore to fail.

• Ensure that the storage_class_name in iac/aws/cluster/modules/05_helm_releases/terraform.tfvars file is same for the targeted restore cluster and the source PPC cluster where the backup is created. If this value differs, the restore operation fails.

• Before restoring to a new cluster, if the source cluster is accessible, disable the backup operations on the source cluster by setting the backup storage location to read‑only. This ensures that no additional backup data is written during the restore process.

  To disable the backup operation on the source cluster, run the following command:

  kubectl patch backupstoragelocation default -n pty-backup-recovery --type merge -p '{"spec":{"accessMode":"ReadOnly"}}'
  

  If the source cluster is not accessible, this step can be skipped.

During Disaster management, the backup data is used to restore the cluster and the OpenSearch indexes using snapshots. However, Insight restores OpenSearch data only from the most recent snapshot created by the daily-insight-snapshots policy.
For more information, refer to Backing up and restoring indexes.

Warning: Do not install or manage multiple clusters from the same working directory. Each cluster deployment maintains its own Terraform or OpenTofu state, and reusing a directory can overwrite state files, causing loss of cluster tracking and unintended cleanup behavior.
Use a dedicated directory, and jump box, where possible, per cluster, and always verify the active kubectl context before running cleanup commands such as ./bootstrap.sh --cloud aws destroy.

The repository provides a bootstrap script that automatically installs or updates the following software on the jump box:

• AWS CLI - Required to communicate with your AWS account.
• bc - Required for version comparisons and resource calculations.
• cmctl - Required to validate and manage TLS certificates.
• Helm - Required to manage Kubernetes packages.
• jq - Required to parse JSON.
• kubectl - Required to communicate with the Kubernetes cluster.
• OpenTofu - Required to manage infrastructure as code.
• ORAS - Required to pull OCI artifacts such as deployment packages.
• skopeo - Required to copy container images between registries.
• ssh-keygen - Required to secure SSH access to deployment environments.

The bootstrap script also checks if you have the required permissions on AWS. It then sets up the EKS cluster and installs the microservices required for deploying the PCS on a PPC.

Note: Before running the bootstrap as the root user on RHEL, ensure that /usr/local/bin (and the AWS CLI binary path, if applicable) is included in the $PATH. Alternatively, run the script using a non-root user (such as ec2-user) where /usr/local/bin is already part of the default PATH.

2.2 - Restoring the PPC using the bootstrap script

Architecture Selection

• amd64 (x86-64-based environments such as standard Intel or AMD servers)
• arm64 (ARM-based environments such as AWS Graviton)

The bootstrap script prompts you to select the node architecture interactively during deployment (step 1b).

Restoring the PPC

Ensure that the PCT is downloaded and extracted on the jump box. For more information, refer to Preparing for PPC deployment.

Run the following command to initiate restore using an existing backup:

./bootstrap.sh --cloud aws --restore

To deploy PPC on ARM-based infrastructure, specify the architecture explicitly:

The bootstrap script asks for variables to be set to complete the deployment. Follow the instructions on the screen.

The --restore command enables the restore mode for the installation. It initiates restoration of data from the configured AWS S3 backup bucket. This process must be followed on a fresh jump box.

The script prompts for the following variables.

1. Enter Cluster Name

   • Ensure that the cluster name does not match the name of the source cluster. Reusing an existing cluster name during restore can lead to discrepancies during cluster installation.
     
   • This same cluster name must already be updated in the KMS key policy. If this update is not performed, the restore process fails because the new cluster cannot decrypt the backup data.
     
   • Ensure that the cluster name does not exceed 31 characters. Cluster names longer than this limit can cause the bootstrap script to fail in subsequent installation steps.
     If the installation fails because the cluster name exceeds the 31-character limit, correct the name and re-run the script.
     
     • Correction: Choose a cluster name with 31 characters or fewer.
       
     • Retry: Execute the installation command again with the updated name. The script will automatically handle the update and proceed with the bootstrap process.

   The following characters are allowed:

   • Lowercase letters: a-z
   • Numbers: 0-9
   • Hyphens: -

   The following characters are not allowed:

   • Uppercase letters: A-Z
   • Underscores: _
   • Spaces
   • Any special characters such as: / ? * + % ! @ # $ ^ & ( ) = [ ] { } : ; , .
   • Leading or trailing hyphens
   • More than 31 characters

   After the cluster name is accepted, the script prompts you to select the node architecture.

   [1b/10] Select Node Architecture

   [1b/10] Select Node Architecture
     1) amd64 (x86_64) — e.g. t3, m5, c5 instances
     2) arm64 (Graviton) — e.g. t4g, m7g, c7g instances
   

   Enter 1 to select amd64 or 2 to select arm64.

   Note: Select amd64 for standard Intel/AMD 64-bit environments. Select arm64 for ARM-based environments such as AWS Graviton instances.

2. Enter a VPC ID from the table

   The script automatically retrieves the available VPCs. Enter the VPC ID where the cluster must be created.

3. Querying for subnets in VPC…

   The script queries for the available VPC subnets and prompts to enter two private subnet IDs. Specify two private subnet IDs from different availability zones.
   The script then automatically updates the VPC CIDR block based on the VPC details.

4. Enter FQDN

   This is the Fully Qualified Domain Name (FQDN) for the ingress.

   While entering the FQDN, ensure that:

   • contains at least two labels separated by a dot, such as app.example.com
   • does not start or end with a dot or hyphen
   • does not contain consecutive dot character
   • label length must be between 1–63 characters
   • each label must start and end with a letter or digit
   • the last label must start only with a letter

   Warning: Ensure that the FQDN does not exceed 50 characters and only the following characters are used:

   • Lowercase letters: a-z
   • Numbers: 0-9
   • Special characters: - .

5. Restore mode requires an existing S3 bucket. Proceeding to existing bucket validation in Step 6.

6. Select S3 Backup Bucket Option

   An AWS S3 bucket encrypted with SSE‑KMS containing backup artifacts used during the restore process.

   Note: Use a dedicated S3 bucket per cluster for backup and restore operations to ensure data and encryption isolation. Sharing a bucket across clusters increases the risk of cross-cluster data access or decryption due to IAM misconfiguration. Dedicated buckets with unique IAM policies eliminate this risk.

   Select backup

   During restore, the script prompts to manually select a backup from the available backups stored in the S3 bucket. User input is required to either restore from the latest backup or choose a specific backup from the list.

   [restore] Backup selection — schedule: authnz-postgresql-schedule-backup
   Restore from latest backup? [Y/n]
   

   • Enter Y to restore from the most recent backup.
   • Enter n to manually select a backup.

   If n is selected, then the script displays a list of available backups (latest first) and prompts to select one by number:

   Available backups (latest first):
   [1] <timestamp> authnz-postgresql-schedule-backup-xxxxxxx
   [2] <timestamp> authnz-postgresql-schedule-backup-xxxxxxx
   
   Select a backup number (1-2):
   

   After entering the backup number, the chosen backup is used for the restore, and the installation continues.

   Enter Insight backup

   During restore, the script prompts to manually select the required insight backup from the available backups stored in the S3 bucket. User input is required to either restore from the latest backup or choose a specific backup from the list.

   [restore] Insight backup selection — schedule: insight-configmaps-secrets
   Restore from latest backup? [Y/n]
   

   • Enter Y to restore from the most recent backup.
   • Enter n to manually select a backup.

   If n is selected, then the script displays a list of available backups (latest first) and prompts to select one by number:

   Available backups (latest first):
   [1] <timestamp>  insight-configmaps-secrets-xxxxxxxxx
   [2] <timestamp>  insight-configmaps-secrets-xxxxxxxxx
   
   Select a backup number (1-2):
   

   After entering the backup number, the chosen backup is used for the restore, and the installation continues.

7. Enter Image Registry Endpoint

   Enter <Container_Registry_Path>, the container registry endpoint from where the container images are retrieved.

   Note: If deploying from the Protegrity Container Registry, use registry.protegrity.com:9443. Specify the local registry endpoint if a local registry is used.

   Expected format: <hostname>[:port].
   Do not include ‘https://’

   Note: The container registry endpoint must be a FQDN (Fully Qualified Domain Name). Sub-paths like, <Container_Registry_Path>/v2/path, are not supported by the OCI distribution specification.

8. Enter Registry Username

   Enter the username for the registry mentioned in the previous step. Leave this entry blank if the registry does not require authentication.

9. Enter Registry Password or Access Token

   Enter Password or Access Token for the registry. Input is masked with * characters. Press Enter to keep the current value.

   Leave this entry blank if the registry does not require authentication.

10. Enter Owner Email (for cluster tagging)

Enter email address in the format name@domain.tld. This email is used for the cluster 'Owner' tag and the platform owner contact. 

Note: The cluster creation process can take approximately 15-20 minutes.

If the session is terminated during restore due to network issues, power outage, and so on, then the restore stops. To restart the process, run the following commands:

# Navigate to deployment directory and run the following command
./bootstrap.sh --cloud aws destroy

# Restore the cluster
./bootstrap.sh --cloud aws --restore

Warning: Do not install or manage multiple clusters from the same working directory. Each cluster deployment maintains its own Terraform or OpenTofu state, and reusing a directory can overwrite state files, causing loss of cluster tracking and unintended cleanup behavior.
Use a dedicated directory, and jump box, where possible, per cluster, and always verify the active kubectl context before running cleanup commands such as ./bootstrap.sh --cloud aws destroy.
To check the active kubectl context, run the following command:
kubectl config current-context

After the restore to the new cluster is completed successfully and all required validation and migration activities are finished, the source cluster can be deleted.

2.3 - Restoring the PPC using in-cluster deployment

Before you begin

Use this procedure to restore PPC from an existing backup onto an existing AWS EKS cluster. The cluster and its supporting resources must already be provisioned; this procedure does not create an EKS cluster. The backup must be available in the original AWS S3 Account. The target cluster must be able to access it. Also confirm that storage_class_name matches the storage class used on the source cluster. Review the in-cluster deployment prerequisites before proceeding.

For more information about backing up the PPC, refer to Backing up the PPC.

Listing the backups created on the source cluster

To obtain a list of available PPC backups, run the following command:

kubectl get backups -n pty-backup-recovery \
  -l velero.io/schedule-name=authnz-postgresql-schedule-backup \
  -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,COMPLETED:.status.completionTimestamp \
  --sort-by=.status.completionTimestamp

To obtain a list of available Insight backups, run the following command:

kubectl get backups -n pty-backup-recovery \
  -l velero.io/schedule-name=insight-configmaps-secrets \
  -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,COMPLETED:.status.completionTimestamp \
  --sort-by=.status.completionTimestamp

Alternatively, the backup names can be obtained from the AWS S3 bucket.

Note: Note the name of the backup, which is required to perform the restore option. This information is required in the terraform.tfvars file.

Restoring PPC using in-cluster deployment

Complete the following steps in order as mentioned. Follow the below mentioned steps for in-cluster deployment instructions.

Updating terraform.tfvars for restore

To restore the terraform.tfvars file, perform the following steps:

1. Navigate to the Helm releases module directory.

   cd iac/aws/cluster/modules/05_helm_releases
   

2. Set the following variables in the terraform.tfvars file.

   Variable	Description
   architecture	Worker node architecture: amd64 or arm64.
   aws_region	AWS region where the EKS cluster is deployed.
   backup_bucket	S3 bucket used for PPC backups.
   backup_bucket_kms_arn	KMS key ARN used to encrypt the backup bucket.
   backup_storage_class_name	StorageClass used for backup PVCs and snapshots.
   eks_cluster_name	Name of the existing EKS cluster.
   global_image_reg	Container image registry endpoint: <Container_Registry_Path>. Note: If deploying from the Protegrity Container Registry, use registry.protegrity.com:9443. Specify the local registry endpoint if a local registry is used.
   harbor_secret	Kubernetes secret used for an authenticated private registry. Set to harbor-secret, or leave empty for an unauthenticated registry.
   ingress_fqdn	Fully qualified domain name for the PPC ingress.
   node_os	Worker node OS image string, such as BOTTLEROCKET_x86_64_FIPS.
   owner_email	Email address of the cluster owner.
   private_subnet_ids	IDs of the private subnets used by the cluster, one per Availability Zone.
   pty-bkp	Backup StorageClass used by Velero for snapshot storage.
   storage_class_name	Existing storage class used for PPC workloads.
   vpc_cidr_block	CIDR block of the VPC containing the EKS cluster.

   For more information about the configuration of variables, refer to In-cluster deployment.

   Note: Registry credentials are passed as environment variables (TF_VAR_username and TF_VAR_password), they are not set directly in terraform.tfvars file.

3. Add the following variables to terraform.tfvars. Set each to the value shown before allowing Stage 05 to apply.

   run_readiness_check = true
   check_prev_stage    = false
   

   These variables ensure that PPC performs a readiness check on the existing EKS environment. They skip the previous automated stage validation. That validation is not required because those resources were created outside the installer.

4. Set the restore values in the same file.

   restore             = true                                # Set to true to restore from a previous backup
   
   backup_name         = "authnz-postgresql-schedule-backup" # Velero backup name used when restore = true
   
   insight_backup_name = ""                                  # Insight Velero backup name (K8s resources only); leave empty to auto-resolve latest
   

   Note: The backup_name must be a real Velero backup name, not the schedule name. The shipped default backup_name = "authnz-postgresql-schedule-backup" is a schedule name, not an actual backup. Tofu passes it verbatim into keycloak.dr.backupName, so with the default value the restore silently no-ops. The actual backup name must be listed, such as, aws s3 ls s3:///velero/backups/. Additionally, set the exact timestamped name, such as, authnz-postgresql-schedule-backup-20260928100004.

5. Save the file and exit the editor.

6. Continue with Step 9: Running the readiness check and Step 10: Applying the OpenTofu configuration. Review the plan and confirm that it targets the intended cluster and backup before applying it.

3 - Restoring the SSH keys and PCT from backed up Terraform state file

This section describes the procedure to rebuild a usable PPC v1.1.0 deployment workspace on a jump box from an existing cluster and a backed up Terraform state file.

Before you Begin

Ensure the following requirements are met.

• The state file from all the following modules must be available on the AWS S3 backup bucket.

  • 00_backup_bucket
  • 01_iam_roles
  • 02_eks_cluster
  • 03_node_group
  • 04_eks_addons
  • 05_helm_releases

• Access to the same AWS account and target EKS cluster.

• AWS CLI credentials with required permissions.

Perform the following steps to recover PPC

1. Create a workspace.

   To create a working directory for PCT, run the following command:

   mkdir -p deployment
   cd deployment
   

2. Download and extract the PCT

   Download the PCT package into the deployment directory and extract it.
   For more information about downloading and extracting the PCT, refer to Preparing for PPC deployment.

   After successfully extracting the template, this directory behaves as recovered installation workspace.

3. Install required dev tools

   Run the following script:

   cd bootstrap-scripts/
   ./setup-devtools-aws.sh
   

4. Set kubectl to the target EKS cluster

   To set kubectl to the target EKS cluster, run the following command:

   aws eks update-kubeconfig --region <aws-region> --name <cluster-name>
   

   Example:

   aws eks update-kubeconfig --region us-east-1 --name <cluster-name>
   

   Verify the current cluster

   Before proceeding, verify the context and cluster connectivity. Run the following command:

   kubectl config current-context
   

5. Initialize OpenTofu

   To initialize OpenTofu for the cluster module, run the following command,

   cd deployment/iac/aws/cluster/modules/05_helm_releases
   tofu init
   

6. Generate the SSH key for CLI access

   To generate the SSH key for CLI access

   tofu output -raw user_svc_private_key > ~/.ssh/user_svc.pem
   chmod 600 ~/.ssh/user_svc.pem