Protegrity Policy Manager

• 1: Installing Policy Workbench
• 1.1: Installing Policy Workbench on PPC in Amazon Elastic Kubernetes Service (EKS)
• 1.2: Installing Policy Workbench on PPC in Azure Kubernetes Service (AKS)
• 2: Uninstalling the Protegrity Policy Manager
• 3: Backing up the Policy Workbench
• 3.1: Backing up the Policy Workbench for AWS
• 3.2: Backing up the Policy Workbench for Azure
• 4: Restoring the Policy Workbench
• 4.1: Restoring the Policy Workbench for AWS
• 4.2: Restoring the Policy Workbench for Azure
• 5: Workbench Roles and Permissions
• 6: Troubleshooting the Protegrity Policy Manager
• 7: Upgrading Policy Workbench in AWS

1 - Installing Policy Workbench

This section provides the steps to install the Policy Workbench on Amazon Elastic Kubernetes Service (EKS) or Azure Kubernetes Service (AKS).

1.1 - Installing Policy Workbench on PPC in Amazon Elastic Kubernetes Service (EKS)

Before installing Policy Workbench, ensure that the prerequisites are met. For more information about the prerequisites, refer to the section Prerequisites.

To install Policy Workbench, first provision the AWS resources using the policy-workbench OpenTofu module and then deploy the Policy Workbench using Helm. The policy-workbench OpenTofu module is published to the Protegrity Container Registry and must be consumed from a root module. A root module is the working directory for executing the OpenTofu commands.

For more information about OpenTofu modules and the root module, refer to the section Modules in the OpenTofu documentation.

The Policy Workbench is installed depending on one of the following scenarios:

• Root module is not available.
• Root module is available.

Prerequisites

Ensure that the jumpbox can connect to the required registries. If not already authenticated, then log in to the required registry.

• For connecting and deploying from the Protegrity Container Registry (PCR), use the following command and the credentials obtained from the My.Protegrity portal during account creation:

helm registry login registry.protegrity.com:9443

• For connecting and deploying to the local registry, use your local credentials and local registry endpoint as required.

Ensure that the PPC Cluster is installed and accessible, before installing Policy Workbench on PPC.

For more information about installing PPC, refer to the section Installing PPC.

Required Tools

Ensure that the following tools are available on the jump box on which Policy Workbench is installed.

IAM Permissions

The following IAM permissions are automatically created by the OpenTofu script.

EKS Permissions

The following EKS permissions are automatically created by the OpenTofu script.

Configuring Credentials for the Policy Workbench

Perform the following steps to configure the credentials to access the Policy Workbench from the OCI registry.

1. Run the following command to create the configuration directory.

   mkdir -p ~/.config/containers
   

2. Obtain the username and access token from the My.Protegrity portal. For more information about obtaining the credentials from the My.Protegrity portal, refer to the section Configuring Authentication for Protegrity AI Team Edition.

3. Generate base64 encoded string with padding for username:accesstoken obtained from the My.Protegrity portal.

   Ensure that you specify the username and access token within single quotes when generating the base64 encoded value. For example, 'username:accesstoken'.

4. Create a file named ~/.config/containers/auth.json with the following content.

    {
        "auths": {
                "registry.protegrity.com:9443": {
                        "auth": "<base64 generated string from step-3>"
                }
        }
}

Installing Policy Workbench when root module is not available

Install the Policy Workbench using the following commands:

1. Run the following command to create the deployment directory.

# must install from an empty directory
mkdir policy-workbench && cd policy-workbench

2. Create a root module with a single main.tf file.

terraform {
  required_version = "~> 1.10"

  required_providers {
    aws = {
      source  = "registry.opentofu.org/hashicorp/aws"
      version = "~> 6.0"
    }
    kubernetes = {
      source  = "registry.opentofu.org/hashicorp/kubernetes"
      version = "~> 2.35"
    }
  }
}

variable "cluster_name" {
  type        = string
  description = "EKS cluster name."
  nullable    = false

  validation {
    condition     = length(trimspace(var.cluster_name)) > 0
    error_message = "cluster_name must be provided and cannot be empty."
  }
}

variable "fips" {
  type        = bool
  description = "Use FIPS Bottlerocket AMI for Karpenter nodes."
  default     = true
}

variable "ami_id" {
  type        = string
  description = "Explicit AMI ID for Karpenter nodes. When set, overrides ami_family/fips-based AMI selection."
  default     = ""
}

data "aws_eks_cluster" "this" {
  name = var.cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.this.token
}

module "policy_workbench" {
  source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/aws?tag=<version>"

  cluster_name = var.cluster_name
  oci_host     = "<Container_Registry_Hostname>"
  fips         = var.fips
  ami_id       = var.ami_id
}

output "helm_install" {
  value = module.policy_workbench.helm_install
}

In the main.tf file, specify the values of the following variables.

3. Ensure that the credentials to install the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to the section Configuring Credentials for the Policy Workbench.

4. Run the following command to navigate to the deployment directory.

   cd policy-workbench
   

5. Run the following commands to plan and install the Policy Workbench OpenTofu module.

# init, plan, and install
tofu init
tofu plan -var="cluster_name=<PPC-cluster-name>"
tofu apply -var="cluster_name=<PPC-cluster-name>"

To use a non-FIPS AMI, run the following tofu apply command:

tofu apply -var="cluster_name=<your-cluster-name>" -var="fips=false"

In this command, specify the value of the fips variable as false to use a non-FIPS AMI for Karpenter nodes.

To use a specific AMI, run the following tofu apply command:

tofu apply -var="cluster_name=<your-cluster-name>" -var="ami_id=<your AMI id>"

In this command, specify the value of the ami_id variable as the specific ID of the AMI for Karpenter nodes. If you specify this value, then it overrides the default FIPS-based AMI selection.

In the cluster_name field, specify the name of the PPC cluster that you have specified in step 1 while deploying the PPC.

For more information about deploying the PPC, refer to the section Deploying PPC.

OpenTofu prints the plan and prompts for confirmation. Enter yes to proceed. To skip the prompt, add the -auto-approve option to the commands.

6. Run the following command to complete the installation of the Policy Workbench using Helm.

helm upgrade --install policy-workbench \
  oci://<Container_Registry_Path>/policy-workbench/helm/policy-workbench \
  --version <version> \
  --namespace policy-workbench \
  --create-namespace \
  --values policy-workbench-values.yaml

In the command, specify the values of the following variables.

The OpenTofu module writes a policy-workbench-values.yaml file in your working directory with cloud-specific keystore settings. Pass it to Helm with the --values parameter.

Note: The tofu output -raw helm_install command prints a ready-to-run Helm command with the correct version and namespace values already filled in.

Important: You need to pass the <ami-id> in the command only if you are deploying the feature in regions other than us-east-1.

Option A (Recommended): Run the following AWS CLI command to retrieve the AMI ID dynamically.

aws ssm get-parameter \
  --name /aws/service/bottlerocket/aws-k8s-1.34/x86_64/latest/image_id \
  --region <region> \
  --query "Parameter.Value" \
  --output text

Alternatively, refer to these example AMI IDs.

Option B: The following table provides the list of AMI IDs.

7. Run the following command to view the pods created in the policy-workbench namespace.

kubectl get pods -n policy-workbench

The following output appears.

NAME                        READY   STATUS    RESTARTS   AGE
bootstrap-bffb4b5d9-v6ww4   1/1     Running   0          13m
cert-7b88dcd84-zx7cv        1/1     Running   0          13m
devops-75755d87d4-qw9n6     1/1     Running   0          13m
hubcontroller-0             1/1     Running   0          13m
kmgw-0                      1/1     Running   0          13m
mbs-6b7dc765dd-brrfk        1/1     Running   0          13m
repository-0                1/1     Running   0          13m
rpproxy-79fc498d8-qp4fz     1/1     Running   0          13m
rpproxy-79fc498d8-s9k5p     1/1     Running   0          13m
rpproxy-79fc498d8-tbdtb     1/1     Running   0          13m
rps-8d79b7d98-svhdw         1/1     Running   0          13m

Installing Policy Workbench when root module is available

Install the Policy Workbench using the following commands:

1. Add the policy-workbench OpenTofu module by adding the following code block to an existing root module.

module "policy_workbench" {
  source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/aws?tag=<version>"

  cluster_name = "<PPC-cluster-name>"
  oci_host     = "<Container_Registry_Hostname>"
  # fips       = true                # set to false for non-FIPS AMI
  # ami_id     = "<your AMI id>"     # explicit AMI override
}

For more information about adding a module to an existing root module, refer to the section Module Blocks in the OpenTofu documentation.

In the root module, specify the values of the following variables.

In the cluster_name field, specify the name of the PPC cluster that you have specified in step 1 while deploying the PPC.

For more information about deploying the PPC, refer to the section Deploying PPC.

2. If the root module does not include the hashicorp/aws provider version >= 6.0 and hashicorp/kubernetes provider version >= 2.35, then add the following code block to the terraform {} block. Else navigate to the next step.

required_providers {
  aws = {
    source  = "registry.opentofu.org/hashicorp/aws"
    version = "~> 6.0"
  }
  kubernetes = {
    source  = "registry.opentofu.org/hashicorp/kubernetes"
    version = "~> 2.35"
  }
}

The kubernetes provider must be configured to connect to the target EKS cluster:

data "aws_eks_cluster" "this" {
  name = "<PPC-cluster-name>"
}
data "aws_eks_cluster_auth" "this" {
  name = "<PPC-cluster-name>"
}
provider "kubernetes" {
host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.this.token
}

For more information about including the hashicorp/aws provider in the root module, refer to the OpenTofu Registry documentation.

For more information about including the hashicorp/kubernetes provider in the root module, refer to the OpenTofu Registry documentation.

3. Ensure that the credentials to install the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to the section Configuring Credentials for the Policy Workbench.

4. Navigate to the directory containing the root module.

5. Run the following commands to plan and install the Policy Workbench OpenTofu module.

# init, plan, and install
tofu init
tofu plan -var="cluster_name=<PPC-cluster-name>"
tofu apply -var="cluster_name=<PPC-cluster-name>"

OpenTofu prints the plan and prompts for confirmation. Enter yes to proceed. To skip the prompt, add the -auto-approve option to the commands.

In the cluster_name field, specify the name of the PPC cluster that you have specified in step 1 while deploying the PPC.

For more information about deploying the PPC, refer to the section Deploying PPC.

6. Run the following command to complete the installation of the Policy Workbench using Helm.

helm upgrade --install policy-workbench \
  oci://<Container_Registry_Path>/policy-workbench/helm/policy-workbench \
  --version <version> \
  --namespace policy-workbench \
  --create-namespace \
  --values policy-workbench-values.yaml

In the command, specify the values of the following variables.

The OpenTofu module writes a policy-workbench-values.yaml file in your working directory with cloud-specific keystore settings. Pass it to Helm with the --values parameter.

Note: The tofu output -raw helm_install command prints a ready-to-run Helm command with the correct version and namespace values already filled in.

Important: You need to pass the <ami-id> in the command only if you are deploying the feature in regions other than us-east-1.

Option A (Recommended): Run the following AWS CLI command to retrieve the AMI ID dynamically.

aws ssm get-parameter \
  --name /aws/service/bottlerocket/aws-k8s-1.34/x86_64/latest/image_id \
  --region <region> \
  --query "Parameter.Value" \
  --output text

Alternatively, refer to these example AMI IDs.

Option B: The following table provides the list of AMI IDs.

7. Run the following command to view the pods created in the policy-workbench namespace.

kubectl get pods -n policy-workbench

The following output appears.

NAME                        READY   STATUS    RESTARTS   AGE
bootstrap-bffb4b5d9-v6ww4   1/1     Running   0          13m
cert-7b88dcd84-zx7cv        1/1     Running   0          13m
devops-75755d87d4-qw9n6     1/1     Running   0          13m
hubcontroller-0             1/1     Running   0          13m
kmgw-0                      1/1     Running   0          13m
mbs-6b7dc765dd-brrfk        1/1     Running   0          13m
repository-0                1/1     Running   0          13m
rpproxy-79fc498d8-qp4fz     1/1     Running   0          13m
rpproxy-79fc498d8-s9k5p     1/1     Running   0          13m
rpproxy-79fc498d8-tbdtb     1/1     Running   0          13m
rps-8d79b7d98-svhdw         1/1     Running   0          13m

Validating the deployment

Note: Before validating the deployment of the Policy Workbench, ensure that the kubectl context is set to the target PPC cluster. Run kubectl config current-context to verify the current context. Run kubectl config use-context <context-name> to switch the context.

After installation, validate the Policy Workbench deployment using the following steps. The desired outcome of these steps is to get a [] response from the datastores API call using a dedicated workbench user.

1. Run the following command to retrieve the gateway host details.

export GW_HOST="$(kubectl get gateway pty-main -n api-gateway -o jsonpath='{.status.addresses[0].value}')"

2. Run the following command to generate the JWT token.

TOKEN=$(curl -k -s "https://$GW_HOST/api/v1/auth/login/token" \
  -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d "loginname=admin" \
  -d "password=Protegrity123!" \
  -D - -o /dev/null 2>&1 \
  | grep -i 'pty_access_jwt_token:' \
  | sed 's/pty_access_jwt_token: //' \
  | tr -d '\r') && echo "${TOKEN:0:10}"

3. Create a workbench user using the following command. Due to separation of duties, the datastores API requires a user with workbench roles.

curl -sk -X POST "https://$GW_HOST/pty/v1/auth/users" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "username": "workbench",
    "password": "<Password>",
    "roles": [
      "workbench_administrator"
    ]
  }'

Note: The password policy requires a minimum of 14 characters, including at least one digit, one uppercase letter, one lowercase letter, and one special character.

The following output appears.

{"user_id":"397beecc-87bb-404e-85bb-f8a6d83984d6","username":"workbench"}

Use the JWT token generated in step 2.

4. Ensure that the user with the workbench_administrator roles has the following permissions:

• workbench_management_policy_write
• workbench_deployment_immutablepackage_export
• workbench_deployment_certificate_export
• cli_access
• can_create_token

To ensure the required permissions, run the following command:

curl -sk -X PUT "https://$GW_HOST/pty/v1/auth/roles" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "workbench_administrator",
    "permissions": [
      "workbench_management_policy_write",
      "workbench_deployment_immutablepackage_export",
      "workbench_deployment_certificate_export",
      "cli_access",
      "can_create_token"
    ]
  }'

The following output appears.

{"role_name":"workbench_administrator","status":"updated"}

For more information about the workbench_administrator permissions, refer to the section Workbench Roles and Permissions.

For more information about the cli_access and can_create_token permissions, refer to the section Roles and Permissions.

5. Run the following command to get a token for the workbench user.

export TOKEN=$(curl -k -s https://$GW_HOST/pty/v1/auth/login/token \
  -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'loginname=workbench' \
  -d 'password=<Password>' \
-D - -o /dev/null 2>&1 | grep -i 'pty_access_jwt_token:' | sed 's/pty_access_jwt_token: //' | tr -d '\r')

6. Run the Policy Management REST API to get datastores. Use the JWT token generated in step 4.

curl -k -v https://$GW_HOST/pty/v2/pim/datastores -H "Authorization: Bearer $TOKEN"

The expected output is []. This indicates that the Policy Workbench is initialized but the datastore is not yet created.

Post-Installation steps

After installing PPW, rotate the certificates to ensure that the certificates on the PPW and PPC are sychronized.

Perform the following steps to rotate the certificates.

1. Run the following command to rotate all the certificates.

# Renew all certificates using eclipse-issuer
kubectl get certificates --all-namespaces -o json | \
jq -r '.items[] | select(.spec.issuerRef.name=="eclipse-issuer") | "\(.metadata.namespace) \(.metadata.name)"' | \
while read -r ns cert_name; do
  echo "Renewing certificate: $cert_name in namespace: $ns"
  cmctl renew "$cert_name" -n "$ns"
done

2. Run the following command to verify that the certificates have been rotated.

kubectl get certificates --all-namespaces

The following output appears.

NAMESPACE          NAME                           READY   SECRET                                AGE
api-gateway        ingress-certificate-secret     True    ingress-certificate-secret            5h46m
cert-manager       eclipse-ca                     True    eclipse-ca                            5h46m
policy-workbench   bootstrap-certificate          True    bootstrap-certificate-secret          17m
policy-workbench   cert-certificate               True    cert-certificate-secret               17m
policy-workbench   devops-certificate             True    devops-certificate-secret             17m
policy-workbench   hubcontroller-certificate      True    hubcontroller-certificate-secret      17m
policy-workbench   kmgw-certificate               True    kmgw-certificate-secret               17m
policy-workbench   mbs-certificate                True    mbs-certificate-secret                17m
policy-workbench   protector-certificate          True    protector-certificate-secret          17m
policy-workbench   pty-main-backend-client-cert   True    pty-main-backend-client-certificate   17m
policy-workbench   repository-certificate         True    repository-certificate-secret         17m
policy-workbench   rpproxy-certificate            True    rpproxy-certificate-secret            17m
policy-workbench   rps-certificate                True    rps-certificate-secret                17m
pty-insight        hubcontoller-client-secret     True    hubcontoller-client-secret            5h46m
pty-insight        tls-for-insight                True    tls-for-insight-key-pair              5h46m
pty-loginui        ingress-certificate-secret     True    ingress-certificate-secret            5h46m

The Ready column displays the value True for all the certificates indicating that they have been rotated.

1.2 - Installing Policy Workbench on PPC in Azure Kubernetes Service (AKS)

Before installing Policy Workbench, ensure that the prerequisites are met. For more information about the prerequisites, refer to the section Prerequisites.

To install Policy Workbench on PPC in AKS, first provision the Azure resources using the policy-workbench OpenTofu module and then deploy the Policy Workbench using Helm. The policy-workbench OpenTofu module is published to OCI and must be consumed from a root module. A root module is the working directory for executing the OpenTofu commands.

For more information about OpenTofu modules and the root module, refer to the section Modules in the OpenTofu documentation.

The Policy Workbench is installed depending on one of the following scenarios:

• Root module is not available.
• Root module is available.

Prerequisites

Ensure that the jumpbox can connect to the required registries. If not already authenticated, then log in to the required registry.

• For connecting and deploying from the Protegrity Container Registry (PCR), use the following command and the credentials obtained from the My.Protegrity portal during account creation:

helm registry login registry.protegrity.com:9443

• For connecting and deploying to the local registry, use your local credentials and local registry endpoint as required.

Ensure that the PPC Cluster is installed and accessible, before installing Policy Workbench on PPC.

For more information about installing PPC, refer to the section Installing PPC.

Required Tools

Ensure that the following tools are available on the jump box on which Policy Workbench is installed.

Azure Permissions

The following Azure permissions are automatically created by the OpenTofu script.

Configuring Credentials for the Policy Workbench

Perform the following steps to configure the credentials to access the Policy Workbench from the OCI registry.

1. Run the following command to create the configuration directory.

   mkdir -p ~/.config/containers
   

2. Obtain the username and access token from the My.Protegrity portal. For more information about obtaining the credentials from the My.Protegrity portal, refer to the section Configuring Authentication for Protegrity AI Team Edition.

3. Generate base64 encoded string with padding for username:accesstoken obtained from the My.Protegrity portal.

   Ensure that you specify the username and access token within single quotes when generating the base64 encoded value. For example, 'username:accesstoken'.

4. Create a file named ~/.config/containers/auth.json with the following content.

    {
        "auths": {
                "registry.protegrity.com:9443": {
                        "auth": "<base64 generated string from step-3>"
                }
        }
}

Installing Policy Workbench when root module is not available

Install the Policy Workbench using the following commands:

1. Run the following command to create the deployment directory.

# must install from an empty directory
mkdir policy-workbench && cd policy-workbench

2. Create a root module with a single main.tf file.

terraform {
  required_version = "~> 1.10"

  required_providers {
    azurerm = {
      source  = "registry.opentofu.org/hashicorp/azurerm"
      version = "~> 4.20"
    }
    kubernetes = {
      source  = "registry.opentofu.org/hashicorp/kubernetes"
      version = "~> 2.35"
    }
  }
}

provider "azurerm" {
  features {}
  resource_provider_registrations = "none"
}

data "azurerm_kubernetes_cluster" "this" {
  name                = var.cluster_name
  resource_group_name = var.resource_group_name
}

provider "kubernetes" {
  host                   = data.azurerm_kubernetes_cluster.this.kube_config[0].host
  client_certificate     = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_certificate)
  client_key             = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_key)
  cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].cluster_ca_certificate)
}

module "policy_workbench" {
  source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/azure?tag=<version>"

  cluster_name                    = var.cluster_name
  oci_host                        = "<Container_Registry_Hostname>"
  resource_group_name             = var.resource_group_name

  azure_keystore = {
    type                = "key_vault"
    name                = var.key_vault_name
    resource_group_name = var.key_vault_resource_group_name
  }

  fips                            = var.fips
  instance_types                  = var.instance_types
}

variable "cluster_name"                    { type = string }
variable "resource_group_name"             { type = string }
variable "key_vault_name"                  { type = string }

variable "key_vault_resource_group_name" {
  type    = string
  default = ""
}

variable "fips" {
  type        = bool
  description = "Use FIPS-enabled nodes for Karpenter. Set to false for non-FIPS."
  default     = true
}

variable "instance_types" {
  type    = list(string)
  default = ["Standard_D8as_v5"]
}

output "helm_install" {
  value = module.policy_workbench.helm_install
}

To use an existing Managed HSM instead of a Standard or Premium Key Vault, replace the keystore block in the main.tf file with the following code block:

azure_keystore = {
  type                = "managed_hsm"
  name                = "<managed-hsm-name>"
  resource_group_name = "<hsm-resource-group>" # omit if same as resource_group_name
}

In the main.tf file, specify the values of the following parameters.

3. Ensure that the credentials to install the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to the section Configuring Credentials for the Policy Workbench.

4. Run the following command to plan and install the Policy Workbench OpenTofu module.

# init, plan, and install
tofu init
tofu plan \
  -var="cluster_name=<PPC-cluster-name>" \
  -var="resource_group_name=<aks-resource-group-name>" \
  -var="key_vault_name=<key-vault-name>"
tofu apply \
  -var="cluster_name=<PPC-cluster-name>" \
  -var="resource_group_name=<aks-resource-group-name>" \
  -var="key_vault_name=<key-vault-name>"

If the Key Vault is in a different resource group, include the following variable:

-var="key_vault_resource_group_name=<key-vault-resource-group-name>"

To override the default Karpenter instance type, include the following variable:

-var='instance_types=["<Instance_type>"]'

To override the default FIPS-enabled mode, include the following variable:

-var='fips=["false"]'

In the command, specify the values of the following variables.

OpenTofu prints the plan and prompts for confirmation. Enter yes to proceed. To skip the prompt, add the -auto-approve option to the commands.

5. Run the following command to complete the installation of the Policy Workbench using Helm.

helm upgrade --install policy-workbench \
  oci://<Container_Registry_Path>/policy-workbench/helm/policy-workbench \
  --version <version> \
  --namespace policy-workbench \
  --create-namespace \
  --values policy-workbench-values.yaml

In the command, specify the values of the following variables.

The OpenTofu module writes a policy-workbench-values.yaml file in your working directory with cloud-specific keystore settings. Pass it to Helm with the --values parameter.

Note: The tofu output -raw helm_install command prints a ready-to-run Helm command with the correct version and namespace values already filled in.

6. Run the following command to view the pods created in the policy-workbench namespace.

kubectl get pods -n policy-workbench

Installing Policy Workbench when root module is available

Install the Policy Workbench using the following commands:

1. Add the policy-workbench OpenTofu module by adding the following code block to an existing root module.

module "policy_workbench" {
  source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/azure?tag=<version>"

  cluster_name                    = "<PPC-cluster-name>"
  oci_host                        = "<Container_Registry_Hostname>"
  resource_group_name             = "<resource-group>"

  azure_keystore = {
    type                = "key_vault"
    name                = "<key-vault-name>"
    resource_group_name = "<key-vault-resource-group>" # omit if same as resource_group_name
  }

  # fips     = true                  # set to false for non-FIPS nodes
  # instance_types = ["Standard_D8as_v5"]
}

To use an existing Managed HSM instead of a Standard or Premium Key Vault, replace the keystore block in the main.tf file with the following code block:

azure_keystore = {
  type                = "managed_hsm"
  name                = "<managed-hsm-name>"
  resource_group_name = "<hsm-resource-group>" # omit if same as resource_group_name
}

For more information about adding a module to an existing root module, refer to the section Module Blocks in the OpenTofu documentation.

In the main.tf file, specify the values of the following parameters.

2. If the root module does not include the hashicorp/azurerm provider version >= 4.20 and hashicorp/kubernetes provider version >= 2.35, then add the following code block to the terraform {} block. Else navigate to the next step.

required_providers {
  azurerm = {
    source  = "registry.opentofu.org/hashicorp/azurerm"
    version = ">= 4.20"
  }
  kubernetes = {
    source  = "registry.opentofu.org/hashicorp/kubernetes"
    version = ">= 2.35"
  }
}

3. If the root module does not configure the Azure provider with resource_provider_registrations = "none", add the following code block to avoid requiring subscription-level registration permissions.

provider "azurerm" {
  features {}
  resource_provider_registrations = "none"
}

4. If the root module does not configure the Kubernetes provider for the target AKS cluster, add the following code block.

data "azurerm_kubernetes_cluster" "this" {
  name                = "<PPC-cluster-name>"
  resource_group_name = "<aks-resource-group-name>"
}

provider "kubernetes" {
  host                   = data.azurerm_kubernetes_cluster.this.kube_config[0].host
  client_certificate     = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_certificate)
  client_key             = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_key)
  cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].cluster_ca_certificate)
}

5. Ensure that the credentials to install the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to the section Configuring Credentials for the Policy Workbench.

6. Navigate to the directory containing the root module.

7. Run the following commands to plan and install the Policy Workbench OpenTofu module.

# init, plan, and install
tofu init
tofu plan \
  -var="cluster_name=<PPC-cluster-name>" \
  -var="resource_group_name=<aks-resource-group-name>" \
  -var="key_vault_name=<key-vault-name>"
tofu apply \
  -var="cluster_name=<PPC-cluster-name>" \
  -var="resource_group_name=<aks-resource-group-name>" \
  -var="key_vault_name=<key-vault-name>"

If the Key Vault is in a different resource group, include the following variable:

-var="key_vault_resource_group_name=<key-vault-resource-group-name>"

To override the default Karpenter instance type, include the following variable:

-var='instance_types=["<Instance_type>"]'

To override the default FIPS-enabled mode, include the following variable:

-var='fips=["false"]'

In the command, specify the values of the following variables.

OpenTofu prints the plan and prompts for confirmation. Enter yes to proceed. To skip the prompt, add the -auto-approve option to the commands.

In the cluster_name field, specify the name of the PPC cluster that you have specified in step 1 while deploying the PPC.

For more information about deploying the PPC, refer to the section Deploying PPC.

8. Run the following command to complete the installation of the Policy Workbench using Helm.

helm upgrade --install policy-workbench \
  oci://<Container_Registry_Path>/policy-workbench/helm/policy-workbench \
  --version <version> \
  --namespace policy-workbench \
  --create-namespace \
  --values policy-workbench-values.yaml

In the command, specify the values of the following variables.

The OpenTofu module writes a policy-workbench-values.yaml file in your working directory with cloud-specific keystore settings. Pass it to Helm with the --values parameter.

Note: The tofu output -raw helm_install command prints a ready-to-run Helm command with the correct version and namespace values already filled in.

9. Run the following command to view the pods created in the policy-workbench namespace.

kubectl get pods -n policy-workbench

Validating the deployment

Note: Before validating the deployment of the Policy Workbench, ensure that the kubectl context is set to the target PPC cluster. Run kubectl config current-context to verify the current context. Run kubectl config use-context <context-name> to switch the context.

After installation, validate the Policy Workbench deployment using the following steps. The desired outcome of these steps is to get a [] response from the datastores API call using a dedicated workbench user.

1. Run the following commands to retrieve the gateway host details and generate an admin JWT token.

export GW_HOST="$(kubectl get gateway pty-main -n api-gateway -o jsonpath='{.status.addresses[0].value}')"

TOKEN=$(curl -k -s https://$GW_HOST/pty/v1/auth/login/token \
  -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'loginname=admin' \
  -d 'password=Protegrity123!' \
-D - -o /dev/null 2>&1 | grep -i 'pty_access_jwt_token:' | sed 's/pty_access_jwt_token: //' | tr -d '\r') && echo "${TOKEN:0:10}"

2. Create a workbench user using the following command. Due to separation of duties, the datastores API requires a user with workbench roles.

curl -sk -X POST "https://$GW_HOST/pty/v1/auth/users" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "username": "workbench",
    "password": "<Password>",
    "roles": [
      "workbench_administrator",
      "security_administrator"
    ]
  }'

Note: The password policy requires a minimum of 14 characters, including at least one digit, one uppercase letter, one lowercase letter, and one special character.

3. Run the following command to get a token for the workbench user.

export TOKEN=$(curl -k -s https://$GW_HOST/pty/v1/auth/login/token \
  -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'loginname=workbench' \
  -d 'password=<Password>' \
-D - -o /dev/null 2>&1 | grep -i 'pty_access_jwt_token:' | sed 's/pty_access_jwt_token: //' | tr -d '\r')

4. Run the Policy Management REST API to get datastores.

curl -k -v https://$GW_HOST/pty/v2/pim/datastores \
  -H "Authorization: Bearer $TOKEN"

The expected output is []. This indicates that the Policy Workbench is initialized but the datastore is not yet created.

For more information about the workbench_administrator permissions, refer to the section Workbench Roles and Permissions.

Post-Installation steps

After installing PPW, rotate the certificates to ensure that the certificates on the PPW and PPC are sychronized.

Perform the following steps to rotate the certificates.

1. Run the following command to rotate all the certificates.

# Renew all certificates using eclipse-issuer
kubectl get certificates --all-namespaces -o json | \
jq -r '.items[] | select(.spec.issuerRef.name=="eclipse-issuer") | "\(.metadata.namespace) \(.metadata.name)"' | \
while read -r ns cert_name; do
  echo "Renewing certificate: $cert_name in namespace: $ns"
  cmctl renew "$cert_name" -n "$ns"
done

2. Run the following command to verify that the certificates have been rotated.

kubectl get certificates --all-namespaces

The following output appears.

NAMESPACE          NAME                           READY   SECRET                                AGE
api-gateway        ingress-certificate-secret     True    ingress-certificate-secret            5h46m
cert-manager       eclipse-ca                     True    eclipse-ca                            5h46m
policy-workbench   bootstrap-certificate          True    bootstrap-certificate-secret          17m
policy-workbench   cert-certificate               True    cert-certificate-secret               17m
policy-workbench   devops-certificate             True    devops-certificate-secret             17m
policy-workbench   hubcontroller-certificate      True    hubcontroller-certificate-secret      17m
policy-workbench   kmgw-certificate               True    kmgw-certificate-secret               17m
policy-workbench   mbs-certificate                True    mbs-certificate-secret                17m
policy-workbench   protector-certificate          True    protector-certificate-secret          17m
policy-workbench   pty-main-backend-client-cert   True    pty-main-backend-client-certificate   17m
policy-workbench   repository-certificate         True    repository-certificate-secret         17m
policy-workbench   rpproxy-certificate            True    rpproxy-certificate-secret            17m
policy-workbench   rps-certificate                True    rps-certificate-secret                17m
pty-insight        hubcontoller-client-secret     True    hubcontoller-client-secret            5h46m
pty-insight        tls-for-insight                True    tls-for-insight-key-pair              5h46m
pty-loginui        ingress-certificate-secret     True    ingress-certificate-secret            5h46m

The Ready column displays the value True for all the certificates indicating that they have been rotated.

2 - Uninstalling the Protegrity Policy Manager

To uninstall the deployment:

1. Run the following command to uninstall the Policy Workbench.

helm uninstall policy-workbench -n policy-workbench

2. Run the following command to clean up the AWS resources.

tofu destroy -var='cluster_name=<PPC cluster name>'

3 - Backing up the Policy Workbench

The Policy Workbench supports daily scheduled backups and manual backups using Velero. This section describes the backup procedures for AWS and Azure deployments.

3.1 - Backing up the Policy Workbench for AWS

By default, the Policy Workbench data is backed up on a daily basis using a scheduled backup, after the Policy Workbench has been installed. The backed-up data includes the Kubernetes object state and the persistent volume data. The backed-up data is automatically stored in the encrypted AWS S3 bucket that you created when you deployed PPC.

For more information about the AWS S3 bucket, refer to the section Creating AWS KMS Key and S3 Bucket.

You can also choose to manually back up the data to the AWS S3 bucket using Velero.

Important: Before you manually back up the data, ensure that Velero CLI version 1.18 or later is installed.

To manually back up the data:

1. Run the following command on the jump box.

   velero backup create --from-schedule workbench-backup-schedule -n <Namespace where data is backed up>
   

   For example:

   velero backup create --from-schedule workbench-backup-schedule -n pty-backup-recovery
   

   The following output appears.

   INFO[0001] No Schedule.template.metadata.labels set - using Schedule.labels for backup object  backup=pty-backup-recovery/workbench-backup-schedule-20260331094735 labels="map[app.kubernetes.io/managed-by:Helm deployment:policy-workbench]"
   Creating backup from schedule, all other filters are ignored.
   Backup request "workbench-backup-schedule-20260331094735" submitted successfully.
   Run `velero backup describe workbench-backup-schedule-20260331094735` or `velero backup logs workbench-backup-schedule-20260331094735` for more details.
   

   For more information about the velero backup command, refer to the section Backup Reference in the Velero documentation.

2. Run the following commands to monitor the backup status.

   1. Run the following command to retrieve the list of existing backups.

      velero backup get -n pty-backup-recovery
      

      The following output appears.

      NAME                                               STATUS                       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
      authnz-postgresql-schedule-backup-20260331093017   Completed                    0        0          2026-03-31 09:30:18 +0000 UTC   59d       default            app.kubernetes.io/persistence=enabled
      workbench-backup-schedule-20260331094735           WaitingForPluginOperations   0        0          2026-03-31 09:47:38 +0000 UTC   59d       default            <none>
      workbench-backup-schedule-20260331094704           WaitingForPluginOperations   0        0          2026-03-31 09:47:04 +0000 UTC   59d
      

   2. Run the following command to obtain details of a specific backup.

      velero backup describe <backup-name> -n pty-backup-recovery
      

      The following code block shows a snippet of the output.

      Name:         workbench-backup-schedule-20260331094735
      Namespace:    pty-backup-recovery
      Labels:       app.kubernetes.io/managed-by=Helm
                  deployment=policy-workbench
                  velero.io/schedule-name=workbench-backup-schedule
                  velero.io/storage-location=default
      Annotations:  meta.helm.sh/release-name=policy-workbench
                  meta.helm.sh/release-namespace=policy-workbench
                  velero.io/resource-timeout=10m0s
                  velero.io/source-cluster-k8s-gitversion=v1.35.2-eks-f69f56f
                  velero.io/source-cluster-k8s-major-version=1
                  velero.io/source-cluster-k8s-minor-version=35
      
      Phase:  WaitingForPluginOperations
      

   3. Run the following command to obtain the log details for a specific backup.

      velero backup logs <backup-name> -n pty-backup-recovery
      

      The following code block shows a snippet of the output.

      time="2026-03-31T09:47:38Z" level=info msg="Setting up backup temp file" backup=pty-backup-recovery/workbench-backup-schedule-20260331094735 logSource="pkg/controller/backup_controller.go:690"
      time="2026-03-31T09:47:38Z" level=info msg="Setting up plugin manager" backup=pty-backup-recovery/workbench-backup-schedule-20260331094735 logSource="pkg/controller/backup_controller.go:697"
      time="2026-03-31T09:47:38Z" level=info msg="Getting backup item actions" backup=pty-backup-recovery/workbench-backup-schedule-20260331094735 logSource="pkg/controller/backup_controller.go:701"
      

3.2 - Backing up the Policy Workbench for Azure

By default, the Policy Workbench data is backed up on a daily basis using a scheduled backup, after the Policy Workbench has been installed. The backed-up data includes the Kubernetes object state and the persistent volume data. The backed-up data is automatically stored in the encrypted Azure Blob Storage container that you created when you deployed PPC.

For more information about the Azure Blob Storage container, refer to the section Creating Azure Storage Account and Container.

You can also choose to manually back up the data to the Azure Blob Storage container using Velero.

Important: Before you manually back up the data, ensure that Velero CLI version 1.18 or later is installed.

To manually back up the data:

1. Run the following command on the jump box.

   velero backup create --from-schedule workbench-backup-schedule -n <Namespace where data is backed up>
   

   For example:

   velero backup create --from-schedule workbench-backup-schedule -n pty-backup-recovery
   

   The following output appears.

   INFO[0000] No Schedule.template.metadata.labels set - using Schedule.labels for backup object  backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 labels="map[app.kubernetes.io/managed-by:Helm deployment:policy-workbench]"
   Creating backup from schedule, all other filters are ignored.
   Backup request "workbench-backup-schedule-20260702112543" submitted successfully.
   Run `velero backup describe workbench-backup-schedule-20260702112543` or `velero backup logs workbench-backup-schedule-20260702112543` for more details.
   

   For more information about the velero backup command, refer to the section Backup Reference in the Velero documentation.

2. Run the following commands to monitor the backup status.

   1. Run the following command to retrieve the list of existing backups.

      velero backup get -n pty-backup-recovery
      

      The following output appears.

      NAME                                               STATUS                       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
      workbench-backup-schedule-20260702112543           InProgress                    0        0          2026-07-02 11:25:43 +0000 UTC   59d       default            <none>
      

   2. Run the following command to obtain details of a specific backup.

      velero backup describe <backup-name> -n pty-backup-recovery
      

      The following code block shows a snippet of the output.

      Name:         workbench-backup-schedule-20260702112543
      Namespace:    pty-backup-recovery
      Labels:       app.kubernetes.io/managed-by=Helm
                  deployment=policy-workbench
                  velero.io/schedule-name=workbench-backup-schedule
                  velero.io/storage-location=default
      Annotations:  meta.helm.sh/release-name=policy-workbench
                  meta.helm.sh/release-namespace=policy-workbench
                  velero.io/resource-timeout=10m0s
                  velero.io/source-cluster-k8s-gitversion=v1.35.5
                  velero.io/source-cluster-k8s-major-version=1
                  velero.io/source-cluster-k8s-minor-version=35
      
      Phase:  InProgress
      

   3. Run the following command to obtain the log details for a specific backup.

      velero backup logs <backup-name> -n pty-backup-recovery
      

      The following code block shows a snippet of the output.

      time="2026-03-31T09:47:38Z" level=info msg="Setting up backup temp file" backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 logSource="pkg/controller/backup_controller.go:690"
      time="2026-03-31T09:47:38Z" level=info msg="Setting up plugin manager" backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 logSource="pkg/controller/backup_controller.go:697"
      time="2026-03-31T09:47:38Z" level=info msg="Getting backup item actions" backup=pty-backup-recovery/workbench-backup-schedule-20260702112543 logSource="pkg/controller/backup_controller.go:701"
      

      Note: Verify that the log output contains no entries with level=error. Entries with level=warning are acceptable and do not indicate a backup failure.

Validating the backup

1. Run the following command to validate the backup status.

   velero backup get -n pty-backup-recovery
   

   The following output appears.

   NAME                                               STATUS                       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
   workbench-backup-schedule-20260702112543           Completed                    0        0          2026-07-02 11:25:43 +0000 UTC   59d       default            <none>
   

4 - Restoring the Policy Workbench

The Policy Workbench supports restoring data from daily scheduled backups or manual backups created using Velero. This section describes how to restore the Policy Workbench from a backup on AWS and Azure deployments.

4.1 - Restoring the Policy Workbench for AWS

Before you begin

Before starting a restore, ensure that the following prerequisites are met:

• An existing backup is available. Backups are taken automatically as part of the default installation of the Policy Workbench using scheduled backup mechanisms. The backups are available in the encrypted AWS S3 bucket that you created when you deployed PPW. You can also manually back up the data.

  For more information about the AWS S3 bucket, refer to the section Creating AWS KMS Key and S3 Bucket.

  For more information about manually backing up the data, refer to the section Backing up the Policy Workbench for AWS.

• A restored PPC cluster is available. The Policy Workbench is restored on a restored PPC cluster. For information about restoring the PPC, refer to the section Restoring the PPC for AWS.

  Note: You can restore the Policy Workbench without first restoring the PPC if only the Policy Workbench is corrupted.

  Important: Before you restore the data, ensure that Velero CLI version 1.18 or later is installed.

Restore the Policy Workbench data

To restore the data:

1. Ensure that the main.tf file in the root module, which is the working directory for executing the OpenTofu commands, contains the following code block. If the root module is not available, create a root module with the main.tf file.

   module "policy_workbench" {
     source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/<cloud>?tag=<version>"
   
     cluster_name = var.cluster_name
   }
   
   variable "cluster_name" {
     type        = string
     description = "EKS cluster name."
     nullable    = false
   
     validation {
       condition     = length(trimspace(var.cluster_name)) > 0
       error_message = "cluster_name must be provided and cannot be empty."
     }
   }
   

   This code block adds the Policy Workbench OpenTofu module.

2. Run the following commands on the jump box.

   Note: You do not need to run these commands if you only uninstall PPW and delete the namespace. These commands are required if you destroy the PPW cluster using tofu destroy. They are also required if you are restoring to a new PPC cluster where PPW was not previously installed.

   tofu init​
   tofu plan -var="cluster_name=<Restored-PPC-cluster-name>"​
   tofu apply -var="cluster_name=<Restored-PPC-cluster-name>" 
   

   Specify the name of the restored PPC cluster as the value of the cluster_name variable.

   For information about restoring the PPC, refer to the section Restoring the PPC for AWS.

3. Run the following command on the jump box.

   velero restore create workbench-restore-$(date +%Y%m%d-%H%M%S) \
    --from-backup <backup-name> \
    -n pty-backup-recovery --wait
   

   For example:

   velero restore create workbench-restore-$(date +%Y%m%d-%H%M%S) \
    --from-backup workbench-backup-schedule-20260331094735 \
    -n pty-backup-recovery --wait
   

   For more information about the velero restore command, refer to the section Restore Reference in the Velero documentation.

4. Run the following command to list all the restore operations in the specific namespace.

   velero restore get -n pty-backup-recovery
   

   Ensure that the status of the restore operation is WaitingForPluginOperations.

5. Run the following command to annotate the Kubernetes resources.

   kubectl annotate productconfiguration workbench -n pty-admin kopf.zalando.org/last-handled-configuration- --overwrite
   

   The following output appears if the annotation is added successfully.

   productconfiguration.pty.com/workbench annotated
   

6. Run the following commands to monitor the restore status.

   1. Run the following command to retrieve the list of existing restores.

      velero restore get -n pty-backup-recovery
      

   2. Run the following command to obtain details of a specific restore.

      velero restore describe workbench-restore-<timestamp> -n pty-backup-recovery
      

   3. Run the following command to obtain the log details for a specific restore.

      velero restore logs workbench-restore-<timestamp> -n pty-backup-recovery
      

      Note: Verify that the log output contains no entries with level=error. Entries with level=warning are acceptable and do not indicate a restore failure.

Uninstall Policy Workbench before restoring on the same cluster

Note: Perform these steps only if you are restoring the Policy Workbench on the same cluster where it was previously installed. If you are restoring to a new cluster, skip this section.

1. Uninstall the policy-workbench module from the existing PPC cluster by running the following command on the jump box:

   helm uninstall policy-workbench -n <namespace>
   

   For example:

   helm uninstall policy-workbench -n policy-workbench
   

   The following output appears if the module is uninstalled successfully:

   release "policy-workbench" uninstalled
   

2. Delete the namespace where the Policy Workbench is installed by running the following command on the jump box:

   kubectl delete namespace <namespace>
   

   For example:

   kubectl delete namespace policy-workbench
   

   The following output appears if the namespace is deleted successfully:

   namespace "policy-workbench" deleted
   

4.2 - Restoring the Policy Workbench for Azure

Before you begin

Before starting a restore, ensure that the following prerequisites are met:

• An existing backup is available. Backups are taken automatically as part of the default installation of the Policy Workbench using scheduled backup mechanisms. The backups are available in the encrypted Azure Blob Storage container that is created by the PPC installation. You can also manually back up the data.

  For more information about the Azure Blob Storage container, refer to the section Creating Azure Storage Account and Container.

  For more information about manually backing up the data, refer to the section Backing up the Policy Workbench for Azure.

• A restored PPC cluster is available. The Policy Workbench is restored on a restored PPC cluster. For information about restoring the PPC, refer to Restoring the PPC on Azure.

  Note: You can restore the Policy Workbench without first restoring the PPC if only the Policy Workbench is corrupted.

  Important: Before you restore the data, ensure that Velero CLI version 1.18 or later is installed.

Restore the Policy Workbench data

To restore the data:

1. Ensure that the main.tf file in the root module, which is the working directory for executing the OpenTofu commands, contains the following code block. If the root module is not available, create a root module with the main.tf file.

   module "policy_workbench" {
     source = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/<cloud>?tag=<version>"
   
     cluster_name = var.cluster_name
   }
   
   variable "cluster_name" {
     type        = string
     description = "AKS cluster name."
     nullable    = false
   
     validation {
       condition     = length(trimspace(var.cluster_name)) > 0
       error_message = "cluster_name must be provided and cannot be empty."
     }
   }
   

   This code block adds the Policy Workbench OpenTofu module.

2. Run the following commands on the jump box.

   Note: You do not need to run these commands if you only uninstall PPW and delete the namespace. These commands are required if you destroy the PPW cluster using tofu destroy. They are also required if you are restoring to a new PPC cluster where PPW was not previously installed.

   tofu init​
   tofu plan -var="cluster_name=<Restored-cluster-name>"​
   tofu apply -var="cluster_name=<Restored-cluster-name>" 
   

   Specify the name of the restored PPC cluster as the value of the cluster_name variable.

   For information about restoring the PPC, refer to the section Restoring the PPC on Azure.

3. Run the following command on the jump box.

   velero restore create workbench-restore-$(date +%Y%m%d-%H%M%S) \
   --from-backup <backup-name> \
   --resource-modifier-configmap workbench-restore-resource-modifier \
   -n <Namespace where data is backed up> --wait
   

   For example:

   velero restore create workbench-restore-$(date +%Y%m%d-%H%M%S) \
   --from-backup workbench-backup-schedule-20260702112543 \
   --resource-modifier-configmap workbench-restore-resource-modifier \
   -n pty-backup-recovery --wait
   

   The following output appears if the restore operation is completed successfully:

   Restore request "workbench-restore-20260702112543" submitted successfully.
   Waiting for restore to complete. You may safely press Ctrl+C to stop waiting, but the restore will continue to run in the background.
   

   After the restore operation is completed, the following output appears:

   Restore completed with status: Completed. You may check for more information using the commands `velero restore describe workbench-restore-20260702-112955` and `velero restore logs workbench-restore-20260702-112955`.
   

   For more information about the velero restore command, refer to the section Restore Reference in the Velero documentation.

4. Run the following command to list all the restore operations in the specific namespace.

   velero restore get -n pty-backup-recovery
   

   Ensure that the status of the restore operation is WaitingForPluginOperations.

5. Run the following command to annotate the Kubernetes resources.

   kubectl annotate productconfiguration workbench -n pty-admin \
   kopf.zalando.org/last-handled-configuration- --overwrite
   

   The following output appears if the annotation is added successfully.

   productconfiguration.pty.com/workbench annotated
   

6. Run the following commands to monitor the restore status.

   1. Run the following command to retrieve the list of existing restores.

      velero restore get -n pty-backup-recovery
      

      The following output appears if the restore operation is completed successfully.

      NAME                                BACKUP                                      STATUS      STARTED                         COMPLETED                       WARNINGS    ERRORS   CREATED                         SELECTOR
      workbench-restore-20260702-112955   workbench-backup-schedule-20260702112543    Completed   2026-07-02 11:29:55 +0000 UTC   2026-07-02 11:30:55 +0000 UTC   0           11       2026-07-02 11:29:55 +0000 UTC   <none>
      

   2. Run the following command to obtain details of a specific restore.

      velero restore describe workbench-restore-<timestamp> -n pty-backup-recovery
      

      For example:

      velero restore describe workbench-restore-20260702-112955 -n pty-backup-recovery
      

      The following output appears if the restore operation is completed successfully.

      Name:         workbench-restore-20260702-112955
      Namespace:    pty-backup-recovery
      Labels:       app.kubernetes.io/managed-by=Helm
                    deployment=policy-workbench
                    velero.io/schedule-name=workbench-backup-schedule
                    velero.io/storage-location=default
      Annotations:  <none>
      
        Phase:  Completed
      

   3. Run the following command to obtain the log details for a specific restore.

      velero restore logs workbench-restore-<timestamp> -n pty-backup-recovery
      

      For example:

      velero restore logs workbench-restore-20260702-112955 -n pty-backup-recovery
      

      The following code block shows a snippet of the output.

      time="2026-07-02T11:30:01Z" level=warning msg="No annotations found in policy-workbench/sh.helm.release.v1.policy-workbench.v1, using restore spec setting: false" groupResource=secrets logSource="pkg/restore/restore.go:2630" namespace=policy-workbench original name=sh.helm.release.v1.policy-workbench.v1 restore=pty-backup-recovery/workbench-restore-20260702-112955
      

      Note: Verify that the log output contains no entries with level=error. Entries with level=warning are acceptable and do not indicate a restore failure.

Uninstall Policy Workbench before restoring on the same cluster

Note: Perform these steps only if you are restoring the Policy Workbench on the same cluster where it was previously installed. If you are restoring to a new cluster, skip this section.

1. Uninstall the policy-workbench module from the existing PPC cluster by running the following command on the jump box:

   helm uninstall policy-workbench -n <namespace>
   

   For example:

   helm uninstall policy-workbench -n policy-workbench
   

   The following output appears if the module is uninstalled successfully:

   release "policy-workbench" uninstalled
   

2. Delete the namespace where the Policy Workbench is installed by running the following command on the jump box:

   kubectl delete namespace <namespace>
   

   For example:

   kubectl delete namespace policy-workbench
   

   The following output appears if the namespace is deleted successfully:

   namespace "policy-workbench" deleted
   

5 - Workbench Roles and Permissions

Roles are templates that include permissions and users can be assigned to one or more roles. All users in the appliance must be associated with a role.

The roles packaged with Policy Workbench are as follows:

The capabilities of a role are defined by the permissions attached to the role. Though roles can be created, modified, or deleted from the appliance, permissions cannot be edited. The permissions that are available to map with a user and packaged with Policy Workbench as default permissions are as follows:

6 - Troubleshooting the Protegrity Policy Manager

Helm upgrade fails due to existing Kubernetes jobs

Issue: Helm upgrade fails because existing jobs, such as hubcontroller-init and kmgw-create-keystore, cannot be patched.

Description: Helm upgrade cannot modify or replace existing Kubernetes jobs if fields such as image registry, environment variables, args, and volumes are changed. This happens because the pod template of a job is immutable. So, the existing pods cannot be replaced when their template changes. As a result, the Helm upgrade fails.

Workaround:

Delete the existing jobs manually and then run the Helm upgrade command.

To manually delete the jobs, run the following commands:

kubectl delete job hubcontroller-init -n policy-workbench
kubectl delete job kmgw-create-keystore -n policy-workbench

Upgrading Policy Workbench from v1.11 to v1.12 fails for restored deployments

Issue: Upgrading Policy Workbench from version 1.11 to 1.12 may fail for deployments that were restored using Velero.

Description: During backup and restore, Velero adds additional labels to Kubernetes resources, causing the live resources to differ from the state managed by OpenTofu. OpenTofu detects this drift and reports the affected resources as modified, which causes the upgrade to fail.

Non-graceful node shutdown prevents StatefulSet pod rescheduling

Issue: Policy Workbench pods do not recover after a non-graceful node termination.

Description: After a non-graceful termination, for example, hardware failure, OS crash, or abrupt shutdown, policy-workbench nodes transition to NotReady state with reason NodeStatusUnknown, and statefulsets remain bound to the node without being rescheduled.

Workaround:

1. Restart the affected nodes if possible. If they return to Ready state, no further action is required.
2. If the issue persists, run the following command on each affected node to force pod rescheduling on healthy nodes:

kubectl taint node <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute

3. Verify that pods are rescheduled.

For more information, refer to Non-Graceful Node Shutdown.

7 - Upgrading Policy Workbench in AWS

Before you begin

• Back up the existing Policy Workbench data.

  For more information, refer to Backing up the Policy Workbench for AWS.

Configuring Credentials for the Policy Workbench

Perform the following steps to configure the credentials to access the Policy Workbench from the OCI registry.

1. Run the following command to create the configuration directory.

   mkdir -p ~/.config/containers
   

2. Obtain the username and access token from the My.Protegrity portal. For more information about obtaining the credentials from the My.Protegrity portal, refer to the section Configuring Authentication for Protegrity AI Team Edition.

3. Generate base64 encoded string with padding for username:accesstoken obtained from the My.Protegrity portal.

   Ensure that you specify the username and access token within single quotes when generating the base64 encoded value. For example, 'username:accesstoken'.

4. Create a file named ~/.config/containers/auth.json with the following content.

    {
        "auths": {
                "registry.protegrity.com:9443": {
                        "auth": "<base64 generated string from step-3>"
                }
        }
}

Upgrade the Policy Workbench Deployment

Depending on the Policy Workbench installation type, follow the relevant steps:

Root module was not available during installation

1. Navigate to the deployment directory.

2. Copy the main.tf contents from step 2 of Installing Policy Workbench when root module is not available.

   Note: The new main.tf includes a kubernetes provider and additional module arguments (oci_host, fips, ami_id) that were not present in earlier versions.

3. Replace the contents of the existing main.tf with the copied content.

4. In main.tf, set the following parameter values.

   Parameter	Description	Value
   <Container_Registry_Path>	Location of the Protegrity Container Registry or the local registry where the policy-workbench OpenTofu module is published.	registry.protegrity.com:9443 if Protegrity Container Registry is used. local registry endpoint if a local registry is used.
   <Container_Registry_Hostname>	OCI registry hostname used for oci_host.	registry.protegrity.com
   <version>	Tag version of the Protegrity Policy Manager, as listed in the Policy Manager README.	1.12.0
   <PPC-cluster-name>	Name of the PPC cluster specified during deployment.	The value used for the cluster name when deploying PPC.
   fips	Use FIPS Bottlerocket AMI for Karpenter nodes.	The default value is true. Set the value to false for non-FIPS.
   ami_id	Explicit Amazon Machine Image (AMI) ID for Karpenter nodes. When this value is set, it overrides the FIPS-based AMI selection.	<ami-id>

   For more information about the <PPC-cluster-name>, refer to Deploying PPC.

5. Ensure that the credentials to install the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to the section Configuring Credentials for the Policy Workbench.

Root module was available during installation

1. Navigate to the root module directory.

2. Locate the policy_workbench module block in the .tf files.

3. Update the module block with the following:

   Note: The oci_host, fips, and ami_id arguments are new in this version. If the existing module block does not include them, add them along with the updated source value.

   module "policy_workbench" {
     source       = "oci://<Container_Registry_Path>/policy-workbench/opentofu/modules/policy-workbench/aws?tag=<version>"
     cluster_name = var.cluster_name
     oci_host     = "<Container_Registry_Hostname>"
     fips         = var.fips
     ami_id       = var.ami_id
   }
   

4. In the policy_workbench module block, set the parameter values as specified in the parameter table.

5. If the root module does not include the following provider versions, add them to the terraform {} block:

   • hashicorp/aws version ~> 6.0
   • hashicorp/kubernetes version ~> 2.35

   required_providers {
     aws = {
       source  = "registry.opentofu.org/hashicorp/aws"
       version = "~> 6.0"
     }
     kubernetes = {
       source  = "registry.opentofu.org/hashicorp/kubernetes"
       version = "~> 2.35"
     }
   }
   

6. Configure the kubernetes provider to connect to the target EKS cluster by adding the following to the root module:

   data "aws_eks_cluster" "this" {
     name = "<PPC-cluster-name>"
   }
   
   data "aws_eks_cluster_auth" "this" {
     name = "<PPC-cluster-name>"
   }
   
   provider "kubernetes" {
     host                   = data.aws_eks_cluster.this.endpoint
     cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
     token                  = data.aws_eks_cluster_auth.this.token
   }
   

7. Ensure that the credentials to update the Policy Workbench from the Protegrity Container Registry have been configured.

   For more information about configuring the credentials, refer to the section Configuring Credentials for the Policy Workbench.

Apply the Configuration

From the configured directory, complete the following steps:

1. Run the following command to initialize the OpenTofu providers.

   tofu init -upgrade
   

2. Run the following commands to import existing resources into the OpenTofu state.

   MANAGED_BY=$(kubectl get nodepool policy-workbench -o jsonpath='{.metadata.labels.app\.kubernetes\.io/managed-by}')
   
   if [ "$MANAGED_BY" = "Helm" ]; then
     tofu import -var="cluster_name=<PPC-cluster-name>" \
       'module.policy_workbench.kubernetes_manifest.node_pool[0]' \
       'apiVersion=karpenter.sh/v1,kind=NodePool,name=policy-workbench'
   
     tofu import -var="cluster_name=<PPC-cluster-name>" \
       'module.policy_workbench.kubernetes_manifest.ec2_node_class[0]' \
       'apiVersion=karpenter.k8s.aws/v1,kind=EC2NodeClass,name=policy-workbench'
   fi
   

3. Run the following commands to preview and apply the OpenTofu changes.

   tofu plan -var="cluster_name=<PPC-cluster-name>"
   tofu apply -var="cluster_name=<PPC-cluster-name>"
   

   OpenTofu displays the plan and prompts for confirmation before applying. Enter yes to proceed, or add the -auto-approve flag to the tofu apply command to skip the prompt.

Deploy the Upgrade with Helm

Run the following command to upgrade Policy Workbench.

helm upgrade --install policy-workbench \
  oci://<Container_Registry_Path>/policy-workbench/helm/policy-workbench \
  --version <version> \
  --namespace policy-workbench \
  --values policy-workbench-values.yaml

Set the values of <Container_Registry_Path> and <version> as specified in the parameter table.

Note: The policy-workbench-values.yaml file is generated by the OpenTofu module in the working directory during the previous step. If the file is missing, re-run tofu apply before proceeding.

If the upgrade is successful, then Helm displays the following output.

NOTES:
Protegrity policy-workbench 1.12.0 installed

The version number reflects the upgraded version.

Verify the Upgrade

Confirm that the Helm release and pods are running correctly.

1. Run the following command to verify the Helm release status.

   helm list -n policy-workbench
   

   The output appears as follows.

   NAME              NAMESPACE         REVISION  STATUS    CHART                    APP VERSION
   policy-workbench  policy-workbench  2         deployed  policy-workbench-1.12.0  1.12.0
   

   The REVISION number increments with each upgrade.

2. Run the following command to verify the Policy Workbench pods are running.

   kubectl get pods -n policy-workbench
   

   The output appears as follows.

   NAME                        READY   STATUS    RESTARTS   AGE
   bootstrap-bffb4b5d9-v6ww4   1/1     Running   0          13m
   cert-7b88dcd84-zx7cv        1/1     Running   0          13m
   devops-75755d87d4-qw9n6     1/1     Running   0          13m
   hubcontroller-0             1/1     Running   0          13m
   kmgw-0                      1/1     Running   0          13m
   mbs-6b7dc765dd-brrfk        1/1     Running   0          13m
   repository-0                1/1     Running   0          13m
   rpproxy-79fc498d8-qp4fz     1/1     Running   0          13m
   rpproxy-79fc498d8-s9k5p     1/1     Running   0          13m
   rpproxy-79fc498d8-tbdtb     1/1     Running   0          13m
   rps-8d79b7d98-svhdw         1/1     Running   0          13m
   

   All pods must show 1/1 under READY and Running under STATUS. Pod names and ages may differ across environments.

Validate the Deployment

Confirm that the Policy Workbench API is accessible after the upgrade.

1. Run the following command to retrieve the gateway address.

   export GW_HOST="$(kubectl get gateway pty-main -n api-gateway -o jsonpath='{.status.addresses[0].value}')"
   

2. Run the following command to obtain an authentication token for the workbench user.

   export TOKEN=$(curl -k -s https://$GW_HOST/pty/v1/auth/login/token \
     -X POST \
     -H 'Content-Type: application/x-www-form-urlencoded' \
     -d 'loginname=workbench' \
     -d 'password=<workbench-password>' \
     -D - -o /dev/null 2>&1 | grep -i 'pty_access_jwt_token:' | sed 's/pty_access_jwt_token: //' | tr -d '\r')
   

   Set <workbench-password> to the workbench user password specified during deployment.

3. Run the following command to verify access to the Policy Workbench datastores API.

   curl -k -v https://$GW_HOST/pty/v2/pim/datastores -H "Authorization: Bearer $TOKEN"
   

An empty array [] or a list of datastore objects returned by the command confirms that the upgrade is complete.