Updating Registry Credentials for AKS/EKS Clusters

Protegrity Provisioned Cluster (PPC) uses image pull secrets to authenticate against the container registry, such as Harbor, when pulling images. When registry passwords are rotated, credentials expire, or the registry endpoint changes, the pull secrets must be updated across all namespaces in the cluster.

This procedure updates the harbor-secret image pull secret across all required namespaces using OpenTofu (tofu) -replace flags. This process does not require a reinstall of the full cluster.

Note: Existing running pods are unaffected by this operation. Only new image pulls use the rotated secret, for example, new pods, scaling events, and node replacements. If existing pods need to use the new credentials immediately, restart the affected deployments after applying the settings.

When to Use This Procedure

The following table outlines common scenarios and the frequency for updating registry credentials in PPC:

Prerequisites

Before you begin, ensure that the following requirements are met:

• You have access to the PPC Kubernetes cluster, kubectl is configured.
• OpenTofu (tofu) is installed.
• You have the new registry username and password.
• You have permission to manage secrets across namespaces in the cluster.
• The PPC directory is available at /eclipse-init/iac/.

Target Namespaces

The harbor-secret is managed in the following namespaces:

Procedure

This procedure must be run once per cloud, that is, AWS for EKS and Azure for AKS. Complete all steps for one cloud before proceeding to the next.

Follow these steps for each cloud environment:

1. Export the credentials for the new registry as environment variables.

   export TF_VAR_username='<new-registry-username>'
   export TF_VAR_password='<new-registry-password>'
   

2. Verify that the credentials are set.

   [[ -n "$TF_VAR_username" && -n "$TF_VAR_password" ]] && echo "creds set" || echo "MISSING"
   

3. Set the Kubernetes context to prevent accidentally writing secrets to the wrong cluster:

   CLOUD=aws                              # or: azure
   KCTX=<your-cluster-context-name>
   
   kubectl config use-context "$KCTX"
   kubectl config current-context         # must match the intended cluster
   

   Note: Operating on the wrong cluster will update secrets in the incorrect environment.

4. Navigate to the iac directory.

   cd iac/azure/cluster/modules/05_helm_releases
   

5. Plan the credential update that targets only the harbor-secret resources across all namespaces.

   NAMESPACES=(pty-admin api-gateway pty-loginui cert-manager karpenter default \
               pty-backup-recovery email-service cli reloader pty-insight)
   
   REPLACE_ARGS=()
   for ns in "${NAMESPACES[@]}"; do
     REPLACE_ARGS+=(-replace="module.helm_releases.module.common.kubernetes_secret_v1.harbor_secret[\"$ns\"]")
   done
   
   tofu plan "${REPLACE_ARGS[@]}" -out="rotate-${CLOUD}.tfplan"
   

6. Review the plan. Expected output:

   Plan: 11 to add, 0 to change, 11 to destroy.
   

   Or equivalently, 11 to replace.

   The plan must affect only harbor_secret resources. If any other resources appear in the plan, stop immediately and delete the plan file.

   rm "rotate-${CLOUD}.tfplan"
   

   Note: Investigate the unexpected changes before retrying.

7. Apply the plan.

   tofu apply "rotate-${CLOUD}.tfplan"
   

   After successfully applying the plan, clean up the plan file. Use the command below:

   rm "rotate-${CLOUD}.tfplan"
   

   Note: The lifecycle.ignore_changes = [data] directive on the secret resource is bypassed by -replace because the resource is destroyed and recreated, not updated in place.

8. Verify the update. Confirm that all harbor-secret resources were recreated with current timestamps.

   for ns in pty-admin api-gateway pty-loginui cert-manager karpenter default \
             pty-backup-recovery email-service cli reloader pty-insight; do
     kubectl -n "$ns" get secret harbor-secret \
       -o jsonpath='{.metadata.namespace}{"\t"}{.metadata.creationTimestamp}{"\n"}'
   done
   

   All timestamps must reflect the time of the apply operation.

9. Reset the context variables, then repeat from step 4 for the next cloud.

   CLOUD=azure
   KCTX=<your-azure-context-name>
   

Optional Post-Update Step: Rolling Deployments

If the running pods need to use the new credentials immediately, rather than waiting for the next pod restart, roll the affected deployments.

kubectl rollout restart deployment -n <namespace> <deployment-name>

Troubleshooting