Policy Workbench on AWS

• 1: Introduction
• 2: Installation
• 2.1: Prerequisites
• 2.2: Installing Policy Workbench
• 3: Upgrade
• 3.1: Upgrading Policy Workbench in AWS
• 4: Backup and Restore
• 4.1: Backing up the Policy Workbench
• 4.2: Restoring the Policy Workbench
• 5: Uninstallation

1 - Introduction

The Policy Workbench is the policy management component of AI Team Edition, designed to provide a modern, cloud-native environment for data protection policy management. Built to run on Kubernetes, Policy Workbench uses a containerized architecture and is deployed on a supported Protegrity Provisioned Cluster (PPC) running on Amazon Elastic Kubernetes Service (EKS). OpenTofu modules and Helm charts help administrators prepare the required resources and deploy Policy Workbench with minimal manual intervention, ensuring consistency and reducing operational overhead.

2 - Installation

This section covers preparing the environment, installing the Policy Workbench on AWS, and validating the deployment.

2.1 - Prerequisites

The following prerequisites apply to both resource-management paths. Both paths require access to the OCI registry to install the Helm chart.

• Ensure that the target EKS cluster is accessible and has a supported PPC installation.

  For more information, refer to PPC installation.

• Ensure that the jumpbox can connect to the required registries. If not already authenticated, then log in to the required registry.

• For connecting and deploying from the Protegrity Container Registry (PCR), use the following command and the credentials obtained from the My.Protegrity portal during account creation:

helm registry login registry.protegrity.com:9443

• For connecting and deploying to the local registry, use your local credentials and local registry endpoint as required.

Required Tools

Ensure that the following tools are available on the jump box on which Policy Workbench is installed.

Permissions

This section lists the permissions that are required to install Policy Workbench.

IAM Permissions

The following IAM permissions are required to provision AWS resources for Policy Workbench. For module-managed Workbench resources, these permissions are required by the AWS identity used to run the OpenTofu module. For self-managed Workbench resources, they are required by the identity used to provision the resources through your infrastructure automation or CLI commands.

EKS Permissions

The following EKS permissions are required to install Policy Workbench. In the OpenTofu-based installation, these permissions are used automatically by the module. In the in-cluster installation, the platform team must have these permissions to provision the resources manually.

Other Permissions

Helm deployment: The installer needs OCI registry access and Kubernetes RBAC to install or upgrade the chart. Helm access does not grant AWS or keystore access.

KMGW runtime: The KMGW pod uses the workbench-kms service account with a pod identity association to access the configured keystore. Grant this service account only the keystore permissions required by the configured backend. These permissions are separate from those required by the OpenTofu or Helm operator.

2.2 - Installing Policy Workbench

Policy Workbench is installed on an existing, supported PPC installation running on AWS EKS. Prepare the AWS and Kubernetes resources needs using one of two methods:

• Module-managed Workbench resources: Use the provided policy-workbench OpenTofu module to prepare the resources on the existing cluster.The OpenTofu installation procedure depends on whether a root module is available.
  For more information about OpenTofu modules and the root module, refer to Modules in the OpenTofu documentation.
• Self-managed Workbench resources: Prepare and manage those resources through the existing infrastructure process.
  For more information on deploying PPC onto an existing AWS EKS cluster, refer to In-cluster deployment.

Both methods install Policy Workbench with Helm. The following sections describe the prerequisites and steps for each method.

Installing Policy Workbench

To install Policy Workbench, select an installation path:

• Module-managed Workbench resources
• Self-managed Workbench resources

Module-managed Workbench resources

Use this path when the policy-workbench OpenTofu module should prepare the AWS and Kubernetes resources required by Policy Workbench on the existing EKS/PPC cluster. After OpenTofu completes, install the Helm chart using the generated Helm values file, policy-workbench-values.yaml.

Module prerequisites

• Ensure that the PPC installed and accessible, before installing Policy Workbench on PPC.

For more information about deploying the PPC cluster, refer to Automated script-based installation or Manual component-based installation.

• Before installing Policy Workbench, verify that the PPC worker security group’s AWS Name tag is set to <cluster-name>_SG, where <cluster-name> is the exact EKS cluster name used during PPC installation and SG must be in uppercase.
  Policy Workbench’s Karpenter EC2NodeClass discovers the worker security group by matching the tag Name = <cluster-name>_SG. This match is case-sensitive. If the tag value does not match exactly, Karpenter-provisioned nodes launch without the PPC worker security group attached. Those nodes cannot reach CoreDNS on the PPC system nodes, and Policy Workbench pods, such as, hubcontroller, will fail with DNS timeouts and enter CrashLoopBackOff.
  
  • In the Automated script-based installation, the bootstrap.sh script sets this tag automatically using the cluster name provided.

Configuring Credentials for the Policy Workbench

Perform the following steps to configure the credentials to access the Policy Workbench from the OCI registry.

1. Run the following command to create the configuration directory.

   mkdir -p ~/.config/containers
   

2. Obtain the username and access token from the My.Protegrity portal. For more information about obtaining the credentials from the My.Protegrity portal, refer to the section Configuring Authentication for Protegrity AI Team Edition.

3. Generate base64 encoded string with padding for username:accesstoken obtained from the My.Protegrity portal.

   Ensure that you specify the username and access token within single quotes when generating the base64 encoded value. For example, 'username:accesstoken'.

   Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

4. Create a file named ~/.config/containers/auth.json with the following content.

   {
       "auths": {
           "<Container_Registry_Path>": {
               "auth": "<base64 generated string from step-3>"
           }
       }
   }
   

Follow the instructions for the scenario that matches whether a root module is available. Refer to the appropriate section using the following links:

• Root module is not available.
• Root module is available.

Installing Policy Workbench when root module is not available

Install the Policy Workbench using the following steps:

1. Run the following command to create the deployment directory:

# must install from an empty directory
mkdir policy-workbench && cd policy-workbench

2. Create a root module with a single main.tf file.

terraform {
  required_version = "~> 1.10"

  required_providers {
    aws = {
      source  = "registry.opentofu.org/hashicorp/aws"
      version = "~> 6.0"
    }
    kubernetes = {
      source  = "registry.opentofu.org/hashicorp/kubernetes"
      version = "~> 2.35"
    }
  }
}

variable "cluster_name" {
  type        = string
  description = "EKS cluster name."
  nullable    = false

  validation {
    condition     = length(trimspace(var.cluster_name)) > 0
    error_message = "cluster_name must be provided and cannot be empty."
  }
}

variable "fips" {
  type        = bool
  description = "Use FIPS Bottlerocket AMI for Karpenter nodes."
  default     = true
}

variable "ami_id" {
  type        = string
  description = "Explicit AMI ID for Karpenter nodes. When set, overrides ami_family/fips-based AMI selection."
  default     = ""
}

variable "arch" {
  type        = string
  description = "CPU architecture for Karpenter nodes. Use 'amd64' or 'arm64'."
  default     = null

  validation {
    condition     = var.arch == null || contains(["amd64", "arm64"], var.arch)
    error_message = "arch must be 'amd64' or 'arm64'."
  }
}

data "aws_eks_cluster" "this" {
  name = var.cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.this.token
}

module "policy_workbench" {
  source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/aws?tag=<version>"

  cluster_name = var.cluster_name
  oci_host     = "<Container_Registry_Path>"
  fips         = var.fips
  ami_id       = var.ami_id
  arch         = var.arch
}

output "helm_install" {
  value = module.policy_workbench.helm_install
}

In the main.tf file, specify the value of the following variable.

Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

3. Ensure that the credentials to install the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to the section Configuring Credentials for the Policy Workbench.

4. Run the following commands to plan and install the Policy Workbench OpenTofu module:

# init, plan, and install
tofu init
tofu plan -var="cluster_name=<cluster_name>"
tofu apply -var="cluster_name=<cluster_name>"

In the cluster_name field, specify the name of the PPC cluster that you have specified in step 1 while deploying the PPC.

For more information about deploying the PPC, refer to the section Deploying PPC.

By default, the Policy Workbench is deployed on arm64-based environments such as AWS Graviton.

To deploy the Policy Workbench on amd64-based environments such as standard Intel or AMD servers, run the following tofu apply command:

tofu apply -var="cluster_name=<ppc-cluster-name>" -var="arch=amd64"

In this command, specify the value of the arch variable as amd64 to deploy the Policy Workbench on amd64-based environments.

To use a standard AMI, run the following tofu apply command:

tofu apply -var="cluster_name=<ppc-cluster-name>" -var="fips=false"

In this command, specify the value of the fips variable as false to use a standard AMI for Karpenter nodes.

To use a specific AMI, run the following tofu apply command:

tofu apply -var="cluster_name=<ppc-cluster-name>" -var="ami_id=<your AMI id>"

In this command, specify the value of the ami_id variable as the specific ID of the AMI for Karpenter nodes. If you specify this value, then it overrides the default secure AMI selection.

OpenTofu prints the plan and prompts for confirmation. Enter yes to proceed. To skip the prompt, add the -auto-approve option to the commands.

After completing this procedure, refer to Deploy Policy Workbench with Helm to continue with the installation.

Installing Policy Workbench when root module is available

Install the Policy Workbench using the following steps:

1. Add the policy-workbench OpenTofu module by adding the following code block to an existing root module:

module "policy_workbench" {
  source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/aws?tag=<version>"

  cluster_name = "<cluster_name>"
  oci_host     = "<Container_Registry_Path>"
  # fips       = true                # set to false for non-FIPS AMI
  # ami_id     = "<your AMI id>"     # explicit AMI override
}

For more information about adding a module to an existing root module, refer to the section Module Blocks in the OpenTofu documentation.

In the root module, specify the values of the following variables.

In the cluster_name field, specify the name of the PPC cluster that you have specified in step 1 while deploying the PPC.

For more information about deploying the PPC, refer to the section Deploying PPC.

1. If the root module does not include the hashicorp/aws provider version >= 6.0 and hashicorp/kubernetes provider version >= 2.35, then add the following code block to the terraform {} block. Else navigate to the next step:

required_providers {
  aws = {
    source  = "registry.opentofu.org/hashicorp/aws"
    version = "~> 6.0"
  }
  kubernetes = {
    source  = "registry.opentofu.org/hashicorp/kubernetes"
    version = "~> 2.35"
  }
}

The kubernetes provider must be configured to connect to the target EKS cluster:

data "aws_eks_cluster" "this" {
  name = "<cluster_name>"
}
data "aws_eks_cluster_auth" "this" {
  name = "<cluster_name>"
}
provider "kubernetes" {
host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.this.token
}

For more information about including the hashicorp/aws provider in the root module, refer to the OpenTofu Registry documentation.

For more information about including the hashicorp/kubernetes provider in the root module, refer to the OpenTofu Registry documentation.

3. Ensure that the credentials to install the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to the section Configuring Credentials for the Policy Workbench.

4. Navigate to the directory containing the root module.

5. Run the following commands to plan and install the Policy Workbench OpenTofu module:

# init, plan, and install
tofu init
tofu plan
tofu apply

OpenTofu prints the plan and prompts for confirmation. Enter yes to proceed. To skip the prompt, add the -auto-approve option to the commands.

After completing this procedure, refer to Deploy Policy Workbench with Helm to continue with the installation.

Self-managed Workbench resources

Use this path when the existing infrastructure process manages Workbench resources instead of the policy-workbench OpenTofu module. This path does not require OpenTofu commands. After completing the prerequisites and configuration below, install the Helm chart using the shared Helm procedure.

Self-managed prerequisites

Ensure that the PPC installed and accessible, before installing Policy Workbench on PPC.

For more information about deploying PPC on to an existing AWS EKS cluster, refer to the section In-cluster deployment.

Self-managed Node Resources and Scheduling

If nodes are provisioned manually instead of using the provided Karpenter modules, size them to cover the Policy Workbench footprint.

AWS KMS Backend Resources for KMGW

The following resources must be provisioned manually:

• IAM policy.
• IAM role.
• EKS Pod Identity association.

The following table lists the actions required for the IAM policy.

Application tags help KMGW find and identify keys. The tags do not control IAM authorization.

Configuring Persistent Storage

Policy Workbench uses the default StorageClass of the cluster unless pvc.storageClassName is set. Verify the StorageClasses available in the target EKS cluster before installation.

1. Run the following command to list the StorageClasses in the cluster:

   kubectl get storageclass
   

2. If the cluster has no default StorageClass, or the default is not suitable for Policy Workbench, set the StorageClass name in the Helm values file, policy-workbench-values.yaml:

   pvc:
     storageClassName: "<storage-class-name>"
   

   For example, PPC creates a StorageClass named pty-sc.

   Note: Without a suitable default StorageClass or this setting, the Policy Workbench PVCs remain unbound and the StatefulSet pods cannot be scheduled. Retain this setting in the policy-workbench-values.yaml file used for installation and future upgrades.

Configuring Node Selection for Policy Workbench Pods

By default, Kubernetes can schedule Policy Workbench pods on any available node. To pin Policy Workbench pods to specific nodes in your EKS cluster, perform the following steps:

1. Run the following command to apply the policy-workbench label to each node where Policy Workbench pods are scheduled:

   kubectl label node <node-name> nodepool.protegrity.com/name=policy-workbench
   

   Replace <node-name> with the name of the target node. Repeat the command for each node.

2. In the policy-workbench-values.yaml file, add a nodeSelector entry under scheduling that matches the label applied in step 1:

   scheduling:
     nodeSelector:
       nodepool.protegrity.com/name: policy-workbench
   

3. If the target nodes have taints applied, apply a PreferNoSchedule taint and a matching toleration.

   Important: Use PreferNoSchedule, not NoSchedule. Velero backup and restore pods do not inherit the tolerations set for Policy Workbench. A NoSchedule taint on the workbench nodes blocks Velero restores.

   a. Run the following command to apply the taint on each target node:

   kubectl taint node <node-name> workload=policy-workbench:PreferNoSchedule
   

   b. Add the matching toleration under scheduling.tolerations in the policy-workbench-values.yaml file. The key, value, and effect must match the taint applied to the node:

   scheduling:
     tolerations:
       - key: "workload"
         operator: "Equal"
         value: "policy-workbench"
         effect: "PreferNoSchedule"
   

Applying Scheduling to Individual Services

To apply a different scheduling policy to a specific service instead of all services, add a scheduling block under that service’s key.

For example:

rpproxy:
  replicas: 3
  scheduling:
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: rpproxy

Here, the Resilient Package Proxy or rpproxy, which delivers packages to Protectors, is the only service whose replica count can be scaled. When scaling rpproxy, add a topologySpreadConstraint to spread its pods across nodes, so that a single node failure does not take down every replica. In this example, the rpproxy replica has been increased from the default 1 to 3. If you change the value to 3, the minimum Workbench requirements will change.

The whenUnsatisfiable field accepts one of the following values:

• ScheduleAnyway : Spreads pods on a best-effort basis without blocking scheduling if the constraint cannot be satisfied.
• DoNotSchedule : Leaves a pod in Pending state rather than placing it on a node that violates the spread constraint.

Configuring the AWS KMS Backend and Helm Values

Complete the following steps if you use AWS KMS as the KMGW backend and configure the policy-workbench-values.yaml file:

1. Set the environment variables used by the following commands:

   export CLUSTER_NAME="<cluster-name>"
   export AWS_REGION="us-east-1"
   export POLICY_NAME="eks-${CLUSTER_NAME}-policy-workbench-kms-policy"
   export ROLE_NAME="eks-${CLUSTER_NAME}-policy-workbench-kms-role"
   export NAMESPACE="policy-workbench"
   export SERVICE_ACCOUNT="workbench-kms"
   

These example values use the policy-workbench namespace and the workbench-kms ServiceAccount.

2. Run the following command to check whether the eks-pod-identity-agent add-on is installed and active:

   aws eks describe-addon \
     --cluster-name "$CLUSTER_NAME" \
     --addon-name eks-pod-identity-agent \
     --query 'addon.{status:status,version:addonVersion}' --output table
   

   If AWS reports that the add-on is missing, install it using the following command:

   aws eks create-addon \
     --cluster-name "$CLUSTER_NAME" \
     --addon-name eks-pod-identity-agent
   

3. Create a file named kms-access-policy.json with the following policy:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "DiscoverKmgwKeys",
         "Effect": "Allow",
         "Action": ["tag:GetResources"],
         "Resource": "*"
       },
       {
         "Sid": "GenerateRandom",
         "Effect": "Allow",
         "Action": ["kms:GenerateRandom"],
         "Resource": "*"
       },
       {
         "Sid": "CreateKmgwKeys",
         "Effect": "Allow",
         "Action": ["kms:CreateKey"],
         "Resource": "*"
       },
       {
         "Sid": "UseKmgwKeys",
         "Effect": "Allow",
         "Action": [
           "kms:Decrypt",
           "kms:Encrypt",
           "kms:DescribeKey",
           "kms:TagResource"
         ],
         "Resource": "*"
       }
     ]
   }
   

4. Run the following command to create the IAM policy:

   export POLICY_ARN=$(aws iam create-policy \
     --policy-name "$POLICY_NAME" \
     --policy-document file://kms-access-policy.json \
     --query Policy.Arn --output text)
   

   This example uses "Resource": "*" for all statements. KMS does not support resource-level scoping for kms:GenerateRandom or kms:CreateKey. The tag:GetResources action applies across the AWS account.

5. Create a file named pod-identity-trust.json with the following trust policy:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": { "Service": "pods.eks.amazonaws.com" },
         "Action": ["sts:AssumeRole", "sts:TagSession"]
       }
     ]
   }
   

6. Run the following command to create the IAM role and attach the policy:

   export ROLE_ARN=$(aws iam create-role \
     --role-name "$ROLE_NAME" \
     --assume-role-policy-document file://pod-identity-trust.json \
     --query Role.Arn --output text)
   
   aws iam attach-role-policy \
     --role-name "$ROLE_NAME" \
     --policy-arn "$POLICY_ARN"
   

7. Run the following command to associate the role with the KMGW ServiceAccount:

   aws eks create-pod-identity-association \
     --cluster-name "$CLUSTER_NAME" \
     --namespace "$NAMESPACE" \
     --service-account "$SERVICE_ACCOUNT" \
     --role-arn "$ROLE_ARN"
   

8. Run the following command to verify that the association between the IAM role and KMGW ServiceAccount exists:

   aws eks list-pod-identity-associations \
     --cluster-name "$CLUSTER_NAME" \
     --namespace "$NAMESPACE" --output table
   

9. Set the following AWS backend values in the policy-workbench-values.yaml file:

   keystore:
     backend: aws
     serviceAccount:
       name: workbench-kms
     aws:
       region: <aws-region>
   

   Set <aws-region> to the AWS region used for AWS_REGION.

After completing this procedure, refer to Deploy Policy Workbench with Helm to continue with the installation.

Deploy Policy Workbench with Helm

Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

After completing either resource-preparation path, install Policy Workbench with Helm.

• Self-managed resources: Configure the policy-workbench-values.yaml file with the storage, scheduling, and keystore settings described in Self-managed workbench resources.
• Module-managed resources: The OpenTofu module generates the policy-workbench-values.yaml file with the required resource settings. The tofu output -raw helm_install command prints a ready-to-run Helm command with the version and namespace values filled in.

Set <version> to the Policy Workbench version being installed. If the namespace was overridden in the OpenTofu module, use the same namespace in the Helm command. Retain the values file for future upgrades.

1. Run the following command to complete the installation of the Policy Workbench using Helm:

helm upgrade --install policy-workbench \
  oci://<Container_Registry_Path>/policy-workbench/helm/policy-workbench \
  --version <version> \
  --namespace policy-workbench \
  --create-namespace \
  --values policy-workbench-values.yaml

In the command, specify the value of the following variable.

2. Run the following command to view the pods created in the policy-workbench namespace:

kubectl get pods -n policy-workbench

The following output appears:

NAME                        READY   STATUS    RESTARTS   AGE
bootstrap-bffb4b5d9-v6ww4   1/1     Running   0          13m
cert-7b88dcd84-zx7cv        1/1     Running   0          13m
devops-75755d87d4-qw9n6     1/1     Running   0          13m
hubcontroller-0             1/1     Running   0          13m
kmgw-0                      1/1     Running   0          13m
mbs-6b7dc765dd-brrfk        1/1     Running   0          13m
repository-0                1/1     Running   0          13m
rpproxy-79fc498d8-qp4fz     1/1     Running   0          13m
rps-8d79b7d98-svhdw         1/1     Running   0          13m

The number of rpproxy pods matches rpproxy.replicas if that value is overridden. Immediately after installation, the short-lived hubcontroller-init-* init job and kmgw-create-keystore-* pod may also appear.

Validating the deployment

Before validating the deployment of the Policy Workbench, ensure that the kubectl context is set to the target PPC cluster.

Note: The curl examples below assume that the server’s TLS certificate can be successfully verified. Do not use -k or --insecure in production, as these options disable certificate verification. For services using a private or self-signed CA, distribute the CA certificate to the client through a separate trusted channel and either install it in the client’s trusted certificate store or specify the CA certificate directly using curl --cacert.

Run kubectl config current-context to verify the current context. Run kubectl config use-context <context-name> to switch the context.

After installation, validate the Policy Workbench deployment using the following steps.

1. Run the following commands to retrieve the gateway host details and generate an admin JWT token:

Note: The admin user must change the default password on first login. Replace <password> with the new admin password.

export GW_HOST="$(kubectl get gateway pty-main -n api-gateway -o jsonpath='{.status.addresses[0].value}')"

TOKEN=$(curl --silent --show-error --dump-header - --output /dev/null "https://$GW_HOST/pty/v1/auth/login/token" \
  -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'loginname=admin' \
  -d 'password=<password>' | grep -i 'pty_access_jwt_token:' | sed 's/pty_access_jwt_token: //' | tr -d '\r')

2. Create a workbench user using the following command: Due to separation of duties, the datastores API requires a user with workbench roles:

curl --silent --show-error -X POST "https://$GW_HOST/pty/v1/auth/users" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "username": "workbench",
    "password": "<password>",
    "roles": [
      "workbench_administrator",
      "security_administrator"
    ]
  }'

Note: The password policy requires a minimum of 14 characters, including at least one digit, one uppercase letter, one lowercase letter, and one special character.

The following output appears:

{"user_id":"397beecc-87bb-404e-85bb-f8a6d83984d6","username":"workbench"}

3. Run the following command to get a token for the workbench user:

TOKEN=$(curl --silent --show-error --dump-header - --output /dev/null "https://$GW_HOST/pty/v1/auth/login/token" \
  -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'loginname=workbench' \
  -d 'password=<password>' | grep -i 'pty_access_jwt_token:' | sed 's/pty_access_jwt_token: //' | tr -d '\r')

4. Run the Policy Management REST API to get datastores. Use the JWT token generated in step 4.

curl --silent --show-error "https://$GW_HOST/pty/v2/pim/datastores" \
  -H "Authorization: Bearer $TOKEN"

The expected output is []. This indicates that the Policy Workbench is initialized but the datastore is not yet created.

For more information about the workbench_administrator permissions, refer to Workbench Roles and Permissions.

3 - Upgrade

This section describes how to upgrade Policy Workbench on a Protegrity Provisioned Cluster (PPC) running on Elastic Kubernetes Service (EKS) using OpenTofu and Helm.

3.1 - Upgrading Policy Workbench in AWS

Before you begin

• Back up the existing Policy Workbench data.

  For more information, refer to Backing up the Policy Workbench for AWS.

Configuring Credentials for the Policy Workbench

Perform the following steps to configure the credentials to access the Policy Workbench from the OCI registry.

1. Run the following command to create the configuration directory.

   mkdir -p ~/.config/containers
   

2. Obtain the username and access token from the My.Protegrity portal. For more information about obtaining the credentials from the My.Protegrity portal, refer to the section Configuring Authentication for Protegrity AI Team Edition.

3. Generate base64 encoded string with padding for username:accesstoken obtained from the My.Protegrity portal.

   Ensure that you specify the username and access token within single quotes when generating the base64 encoded value. For example, 'username:accesstoken'.

   Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

4. Create a file named ~/.config/containers/auth.json with the following content.

   {
       "auths": {
           "<Container_Registry_Path>": {
               "auth": "<base64 generated string from step-3>"
           }
       }
   }
   

Upgrade the Policy Workbench Deployment

Depending on the Policy Workbench installation type, follow the relevant steps:

Root module was not available during installation

1. Navigate to the deployment directory.

2. Copy the main.tf contents from step 2 of Installing Policy Workbench when root module is not available.

   Note: The new main.tf includes a kubernetes provider and additional module arguments (oci_host, fips, ami_id) that were not present in earlier versions.

3. Replace the contents of the existing main.tf with the copied content.

4. In main.tf, set the following parameter values.

   Parameter	Description	Value
   <version>	Tag version of the Policy Workbench, as listed in the Policy Workbench README.	1.12.0
   <cluster_name>	Name of the PPC cluster specified during deployment.	The value used for the cluster name when deploying PPC.
   fips	Use secure Bottlerocket AMI for Karpenter nodes.	The default value is true. Set the value to false for standard AMI.
   ami_id	Explicit Amazon Machine Image (AMI) ID for Karpenter nodes. When this value is set, it overrides the standard AMI selection.	<ami-id>
   arch	CPU architecture for Karpenter nodes. Set to amd64 for x86 nodes.	The default value is arm64.

   For more information about the <cluster_name>, refer to Deploying PPC.

5. Ensure that the credentials to install the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to the section Configuring Credentials for the Policy Workbench.

Root module was available during installation

1. Navigate to the root module directory.

2. Locate the policy_workbench module block in the .tf files.

3. Update the module block with the following:

   Note: The oci_host, fips, ami_id, and arch arguments are new in this version. If the existing module block does not include them, add them along with the updated source value.

   module "policy_workbench" {
    source       = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/aws?tag=<version>"
     cluster_name = var.cluster_name
    oci_host     = "<Container_Registry_Path>"
     fips         = var.fips
     ami_id       = var.ami_id
     arch         = var.arch
   }
   

   Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

4. In the policy_workbench module block, set the parameter values as specified in the parameter table.

5. If the root module does not include the following provider versions, add them to the terraform {} block:

   • hashicorp/aws version ~> 6.0
   • hashicorp/kubernetes version ~> 2.35

   required_providers {
     aws = {
       source  = "registry.opentofu.org/hashicorp/aws"
       version = "~> 6.0"
     }
     kubernetes = {
       source  = "registry.opentofu.org/hashicorp/kubernetes"
       version = "~> 2.35"
     }
   }
   

6. Configure the kubernetes provider to connect to the target EKS cluster by adding the following to the root module:

   data "aws_eks_cluster" "this" {
     name = "<cluster_name>"
   }
   
   data "aws_eks_cluster_auth" "this" {
     name = "<cluster_name>"
   }
   
   provider "kubernetes" {
     host                   = data.aws_eks_cluster.this.endpoint
     cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
     token                  = data.aws_eks_cluster_auth.this.token
   }
   

7. Ensure that the credentials to update the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to the section Configuring Credentials for the Policy Workbench.

Apply the Configuration

From the configured directory, complete the following steps:

1. Run the following command to initialize the OpenTofu providers:

   tofu init -upgrade
   

2. Run the following commands to import existing resources into the OpenTofu state:

   MANAGED_BY=$(kubectl get nodepool policy-workbench -o jsonpath='{.metadata.labels.app\.kubernetes\.io/managed-by}')
   
   if [ "$MANAGED_BY" = "Helm" ]; then
     tofu import -var="cluster_name=<cluster_name>" \
       'module.policy_workbench.kubernetes_manifest.node_pool[0]' \
       'apiVersion=karpenter.sh/v1,kind=NodePool,name=policy-workbench'
   
     tofu import -var="cluster_name=<cluster_name>" \
       'module.policy_workbench.kubernetes_manifest.ec2_node_class[0]' \
       'apiVersion=karpenter.k8s.aws/v1,kind=EC2NodeClass,name=policy-workbench'
   fi
   

3. Run the following commands to preview and apply the OpenTofu changes:

   tofu plan -var="cluster_name=<cluster_name>"
   tofu apply -var="cluster_name=<cluster_name>"
   

   OpenTofu displays the plan and prompts for confirmation before applying. Enter yes to proceed, or add the -auto-approve flag to the tofu apply command to skip the prompt.

Deploy the Upgrade with Helm

Run the following command to upgrade Policy Workbench:

helm upgrade --install policy-workbench \
  oci://<Container_Registry_Path>/policy-workbench/helm/policy-workbench \
  --version <version> \
  --namespace policy-workbench \
  --values policy-workbench-values.yaml

Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

Set the value of <version> as specified in the parameter table.

Note: The policy-workbench-values.yaml file is generated by the OpenTofu module in the working directory during the previous step. If the file is missing, re-run tofu apply before proceeding.

If the upgrade is successful, then Helm displays the following output:

NOTES:
Protegrity policy-workbench 1.12.0 installed

The version number reflects the upgraded version.

Verify the Upgrade

Confirm that the Helm release and pods are running correctly.

1. Run the following command to verify the Helm release status:

   helm list -n policy-workbench
   

   The output appears as follows:

   NAME              NAMESPACE         REVISION  STATUS    CHART                    APP VERSION
   policy-workbench  policy-workbench  2         deployed  policy-workbench-1.12.0  1.12.0
   

   The REVISION number increments with each upgrade.

2. Run the following command to verify the Policy Workbench pods are running:

   kubectl get pods -n policy-workbench
   

   The output appears as follows:

   NAME                        READY   STATUS    RESTARTS   AGE
   bootstrap-bffb4b5d9-v6ww4   1/1     Running   0          13m
   cert-7b88dcd84-zx7cv        1/1     Running   0          13m
   devops-75755d87d4-qw9n6     1/1     Running   0          13m
   hubcontroller-0             1/1     Running   0          13m
   kmgw-0                      1/1     Running   0          13m
   mbs-6b7dc765dd-brrfk        1/1     Running   0          13m
   repository-0                1/1     Running   0          13m
   rpproxy-79fc498d8-qp4fz     1/1     Running   0          13m
   rpproxy-79fc498d8-s9k5p     1/1     Running   0          13m
   rpproxy-79fc498d8-tbdtb     1/1     Running   0          13m
   rps-8d79b7d98-svhdw         1/1     Running   0          13m
   

   All pods must show 1/1 under READY and Running under STATUS. Pod names and ages may differ across environments.

Validate the Deployment

Confirm that the Policy Workbench API is accessible after the upgrade.

1. Run the following command to retrieve the gateway address:

   export GW_HOST="$(kubectl get gateway pty-main -n api-gateway -o jsonpath='{.status.addresses[0].value}')"
   

2. Run the following command to obtain an authentication token for the workbench user:

   export TOKEN=$(curl -k -s https://$GW_HOST/pty/v1/auth/login/token \
     -X POST \
     -H 'Content-Type: application/x-www-form-urlencoded' \
     -d 'loginname=workbench' \
     -d 'password=<workbench-password>' \
     -D - -o /dev/null 2>&1 | grep -i 'pty_access_jwt_token:' | sed 's/pty_access_jwt_token: //' | tr -d '\r')
   

   Set <workbench-password> to the workbench user password specified during deployment.

3. Run the following command to verify access to the Policy Workbench datastores API:

   curl -k -v https://$GW_HOST/pty/v2/pim/datastores -H "Authorization: Bearer $TOKEN"
   

The response should list any existing datastores. If no datastores have been created, the response should be an empty array [].

4 - Backup and Restore

This section describes how to back up Policy Workbench data on AWS and restore it from an existing backup.

4.1 - Backing up the Policy Workbench

By default, the Policy Workbench data is backed up on a daily basis using a scheduled backup, after the Policy Workbench has been installed. The backed-up data includes the Kubernetes object state and the persistent volume data. The backed-up data is automatically stored in the encrypted AWS S3 bucket that you created when you deployed PPC.

For more information about the AWS S3 bucket, refer to Creating AWS KMS Key and S3 Bucket.

You can also choose to manually back up the data to the AWS S3 bucket using Velero.

Important: Before you manually back up the data, ensure that Velero CLI version 1.18 or later is installed.

To manually back up the data:

1. Run the following command on the jump box:

   velero backup create --from-schedule workbench-backup-schedule -n <Namespace where data is backed up>
   

   For example:

   velero backup create --from-schedule workbench-backup-schedule -n pty-backup-recovery
   

   The following output appears:

   INFO[0001] No Schedule.template.metadata.labels set - using Schedule.labels for backup object  backup=pty-backup-recovery/workbench-backup-schedule-20260331094735 labels="map[app.kubernetes.io/managed-by:Helm deployment:policy-workbench]"
   Creating backup from schedule, all other filters are ignored.
   Backup request "workbench-backup-schedule-20260331094735" submitted successfully.
   Run `velero backup describe workbench-backup-schedule-20260331094735` or `velero backup logs workbench-backup-schedule-20260331094735` for more details.
   

   For more information about the velero backup command, refer to Backup Reference in the Velero documentation.

2. Run the following commands to monitor the backup status.

   1. Run the following command to retrieve the list of existing backups:

      velero backup get -n pty-backup-recovery
      

      The following output appears:

      NAME                                               STATUS                       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
      authnz-postgresql-schedule-backup-20260331093017   Completed                    0        0          2026-03-31 09:30:18 +0000 UTC   59d       default            app.kubernetes.io/persistence=enabled
      workbench-backup-schedule-20260331094735           WaitingForPluginOperations   0        0          2026-03-31 09:47:38 +0000 UTC   59d       default            <none>
      workbench-backup-schedule-20260331094704           WaitingForPluginOperations   0        0          2026-03-31 09:47:04 +0000 UTC   59d
      

   2. Run the following command to obtain details of a specific backup:

      velero backup describe <backup-name> -n pty-backup-recovery
      

      The following code block shows a snippet of the output:

      Name:         workbench-backup-schedule-20260331094735
      Namespace:    pty-backup-recovery
      Labels:       app.kubernetes.io/managed-by=Helm
                  deployment=policy-workbench
                  velero.io/schedule-name=workbench-backup-schedule
                  velero.io/storage-location=default
      Annotations:  meta.helm.sh/release-name=policy-workbench
                  meta.helm.sh/release-namespace=policy-workbench
                  velero.io/resource-timeout=10m0s
                  velero.io/source-cluster-k8s-gitversion=v1.35.2-eks-f69f56f
                  velero.io/source-cluster-k8s-major-version=1
                  velero.io/source-cluster-k8s-minor-version=35
      
      Phase:  WaitingForPluginOperations
      

   3. Run the following command to obtain the log details for a specific backup:

      velero backup logs <backup-name> -n pty-backup-recovery
      

      The following code block shows a snippet of the output:

      time="2026-03-31T09:47:38Z" level=info msg="Setting up backup temp file" backup=pty-backup-recovery/workbench-backup-schedule-20260331094735 logSource="pkg/controller/backup_controller.go:690"
      time="2026-03-31T09:47:38Z" level=info msg="Setting up plugin manager" backup=pty-backup-recovery/workbench-backup-schedule-20260331094735 logSource="pkg/controller/backup_controller.go:697"
      time="2026-03-31T09:47:38Z" level=info msg="Getting backup item actions" backup=pty-backup-recovery/workbench-backup-schedule-20260331094735 logSource="pkg/controller/backup_controller.go:701"
      

4.2 - Restoring the Policy Workbench

Before you begin

Before starting a restore, ensure that the following prerequisites are met:

• An existing backup is available. Backups are taken automatically as part of the default installation of the Policy Workbench using scheduled backup mechanisms. The backups are available in the encrypted AWS S3 bucket that you created when you deployed Policy Workbench. You can also manually back up the data.

  For more information about the AWS S3 bucket, refer to Creating AWS KMS Key and S3 Bucket.

  For more information about manually backing up the data, refer to Backing up the Policy Workbench for AWS.

• A restored PPC cluster is required for both module-managed and self-managed Workbench resources. For information about restoring the PPC, refer to Restoring the PPC for AWS.

  Note: You can restore the Policy Workbench without first restoring the PPC if only the Policy Workbench is corrupted.

  Important: Before you restore the data, ensure that Velero CLI version 1.18 or later is installed.

Restore the Policy Workbench data

To restore the data:

1. Ensure that the main.tf file in the root module, which is the working directory for executing the OpenTofu commands, contains the following code block. If the root module is not available, create a root module with the main.tf file:

   module "policy_workbench" {
     source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/<cloud>?tag=<version>"
   
     cluster_name = var.cluster_name
   }
   
   variable "cluster_name" {
     type        = string
     description = "EKS cluster name."
     nullable    = false
   
     validation {
       condition     = length(trimspace(var.cluster_name)) > 0
       error_message = "cluster_name must be provided and cannot be empty."
     }
   }
   

   Note: For the Protegrity Container Registry, set <Container_Registry_Path> to registry.protegrity.com:9443. For a local registry, set it to the local registry endpoint.

   This code block adds the Policy Workbench OpenTofu module.

2. Run the following commands on the jump box.

   Note: You do not need to run these commands if you only uninstall Policy Workbench and delete the namespace. These commands are required if you destroy the Policy Workbench cluster using tofu destroy. They are also required if you are restoring to a new PPC cluster where Policy Workbench was not previously installed.

   tofu init​
   tofu plan -var="cluster_name=<Restored-PPC-cluster-name>"​
   tofu apply -var="cluster_name=<Restored-PPC-cluster-name>" 
   

   Specify the name of the restored PPC cluster as the value of the cluster_name variable.

   For information about restoring the PPC, refer to Restoring the PPC for AWS.

3. Run the following command on the jump box:

   velero restore create workbench-restore-$(date +%Y%m%d-%H%M%S) \
    --from-backup <backup-name> \
    -n pty-backup-recovery --wait
   

   For example:

   velero restore create workbench-restore-$(date +%Y%m%d-%H%M%S) \
    --from-backup workbench-backup-schedule-20260331094735 \
    -n pty-backup-recovery --wait
   

   For more information about the velero restore command, refer to Restore Reference in the Velero documentation.

4. Run the following command to list all the restore operations in the specific namespace:

   velero restore get -n pty-backup-recovery
   

   Ensure that the status of the restore operation is WaitingForPluginOperations.

5. Run the following command to annotate the Kubernetes resources:

   kubectl annotate productconfiguration workbench -n pty-admin kopf.zalando.org/last-handled-configuration- --overwrite
   

   The following output appears if the annotation is added successfully:

   productconfiguration.pty.com/workbench annotated
   

6. Run the following commands to monitor the restore status.

   1. Run the following command to retrieve the list of existing restores:

      velero restore get -n pty-backup-recovery
      

   2. Run the following command to obtain details of a specific restore:

      velero restore describe workbench-restore-<timestamp> -n pty-backup-recovery
      

   3. Run the following command to obtain the log details for a specific restore:

      velero restore logs workbench-restore-<timestamp> -n pty-backup-recovery
      

      Note: Verify that the log output contains no entries with level=error. Entries with level=warning are acceptable and do not indicate a restore failure.

Uninstall Policy Workbench before restoring on the same cluster

Note: Perform these steps only if you are restoring the Policy Workbench on the same cluster where it was previously installed. If you are restoring to a new cluster, skip this section.

1. Uninstall the policy-workbench module from the existing PPC cluster by running the following command on the jump box:

   helm uninstall policy-workbench -n <namespace>
   

   For example:

   helm uninstall policy-workbench -n policy-workbench
   

   The following output appears if the module is uninstalled successfully:

   release "policy-workbench" uninstalled
   

2. Delete the namespace where the Policy Workbench is installed by running the following command on the jump box:

   kubectl delete namespace <namespace>
   

   For example:

   kubectl delete namespace policy-workbench
   

   The following output appears if the namespace is deleted successfully:

   namespace "policy-workbench" deleted
   

5 - Uninstallation

To uninstall the Policy Workbench:

1. Run the following command to uninstall the Policy Workbench:

   helm uninstall policy-workbench -n policy-workbench
   

2. Run the following command to clean up the AWS resources:

   tofu destroy -var='cluster_name=<cluster name>'