Troubleshooting the Protegrity Policy Manager

Helm upgrade fails due to existing Kubernetes jobs

Issue: Helm upgrade fails because existing jobs, such as hubcontroller-init and kmgw-create-keystore, cannot be patched.

Description: Helm upgrade cannot modify or replace existing Kubernetes jobs if fields such as image registry, environment variables, args, and volumes are changed. This happens because the pod template of a job is immutable. So, the existing pods cannot be replaced when their template changes. As a result, the Helm upgrade fails.

Workaround:

Delete the existing jobs manually and then run the Helm upgrade command.

To manually delete the jobs, run the following commands:

kubectl delete job hubcontroller-init -n policy-workbench
kubectl delete job kmgw-create-keystore -n policy-workbench

Upgrading Policy Workbench from v1.11 to v1.12 fails for restored deployments

Issue: Upgrading Policy Workbench from version 1.11 to 1.12 may fail for deployments that were restored using Velero.

Description: During backup and restore, Velero adds additional labels to Kubernetes resources, causing the live resources to differ from the state managed by OpenTofu. OpenTofu detects this drift and reports the affected resources as modified, which causes the upgrade to fail.

Non-graceful node shutdown prevents StatefulSet pod rescheduling

Issue: Policy Workbench pods do not recover after a non-graceful node termination.

Description: After a non-graceful termination, for example, hardware failure, OS crash, or abrupt shutdown, policy-workbench nodes transition to NotReady state with reason NodeStatusUnknown, and statefulsets remain bound to the node without being rescheduled.

Workaround:

1. Restart the affected nodes if possible. If they return to Ready state, no further action is required.
2. If the issue persists, run the following command on each affected node to force pod rescheduling on healthy nodes:

kubectl taint node <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute

3. Verify that pods are rescheduled.

For more information, refer to Non-Graceful Node Shutdown.