Additional Information

• 1: Replacing the default CA with a Custom CA in PPC
• 2: Updating Registry Credentials for AKS/EKS Clusters

1 - Replacing the default CA with a Custom CA in PPC

In a PPC deployment, Envoy and other internal components rely on a CA to establish trusted TLS communication.

By default, PPC uses an internally generated CA. PPC supports replacing the default CA with a custom CA that can be preserved and reused across cluster restore or migration.

Prerequisites

Before you begin, ensure that:

• You have access to the PPC Kubernetes cluster (kubectl configured).
• Openssl is installed.
• Cert-manager is installed.
• The eclipse-issuer ClusterIssuer exists.
• You have permission to create secrets in the cert-manager namespace.

Perform the following steps:

1. Custom CA certificates are available.

   a. Users already have existing CA certificates.

   To retrieve custom CA certificate and key from custom-ca-secret secret, run the following commands:

    ```bash
    kubectl -n cert-manager get secret custom-ca-secret   -o jsonpath='{.data.tls\.crt}' | base64 -d > <your-ca>.crt
   
    kubectl -n cert-manager get secret custom-ca-secret   -o jsonpath='{.data.tls\.key}' | base64 -d > <your-ca>.key
    ```
   

   b. Users generate custom CA certificates.

   For more information about generating New Certificate and Key with OpenSSL, refer to Openssl documentation.

2. Copy the CA certificate to the jumpbox.

3. To create a TLS secret containing the CA certificate and key, navigate to the folder where certificates are available and run the following command:

kubectl create secret tls custom-ca-secret \
  --cert=<your-ca>.crt \
  --key=<your-ca>.key \
  -n cert-manager

4. To verify the secret was created, run the following command:

kubectl get secret custom-ca-secret -n cert-manager
NAME               TYPE                DATA   AGE
custom-ca-secret   kubernetes.io/tls   2      5s

5. To patch the eclipse-issuer ClusterIssuer to point to the new CA secret, run the following command:

kubectl patch clusterissuer eclipse-issuer \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/ca/secretName","value":"custom-ca-secret"}]'

Note: This changes cert-manager to issue all new certificates using the custom CA.

6. After patching the ClusterIssuer, existing certificates must be re-issued using the new CA. Use one of the following approaches:

Approach 1: Trigger renewal via cmctl (Recommended)

cmctl is the official cert-manager CLI and provides the most reliable way to trigger certificate renewal. The script below checks if cmctl is installed and downloads it automatically if not.

#!/bin/bash

# Install cmctl if not present
if ! command -v cmctl &>/dev/null; then
  echo "cmctl not found, downloading..."
  curl -L https://github.com/cert-manager/cmctl/releases/latest/download/cmctl_linux_amd64 \
    -o /usr/local/bin/cmctl
  chmod +x /usr/local/bin/cmctl
  echo "cmctl installed successfully"
fi

# Renew all certificates using eclipse-issuer
kubectl get certificates --all-namespaces -o json | \
jq -r '.items[] | select(.spec.issuerRef.name=="eclipse-issuer") | "\(.metadata.namespace) \(.metadata.name)"' | \
while read -r ns cert_name; do
  echo "Renewing certificate: $cert_name in namespace: $ns"
  cmctl renew "$cert_name" -n "$ns"
done

Approach 2: Trigger renewal via kubectl (status patch)

Use this approach if cmctl cannot be installed. Requires kubectl 1.24+.

#!/bin/bash

kubectl get certificates --all-namespaces -o json | \
jq -r '.items[] | select(.spec.issuerRef.name=="eclipse-issuer") | "\(.metadata.namespace) \(.metadata.name)"' | \
while read -r ns cert_name; do
  echo "Triggering renewal for certificate: $cert_name in namespace: $ns"
  kubectl patch certificate "$cert_name" -n "$ns" \
    --subresource=status \
    --type=merge \
    -p '{
      "status": {
        "conditions": [{
          "type": "Issuing",
          "status": "True",
          "reason": "ManuallyTriggered",
          "message": "Certificate renewal manually triggered",
          "lastTransitionTime": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'"
        }]
      }
    }'
done

Note: Due to cert-manager’s reconcile loop, some certificates may not renew on the first attempt. Re-run the script if any certificates remain unrenewed.

2 - Updating Registry Credentials for AKS/EKS Clusters

Protegrity Provisioned Cluster (PPC) uses image pull secrets to authenticate against the container registry, such as Harbor, when pulling images. When registry passwords are rotated, credentials expire, or the registry endpoint changes, the pull secrets must be updated across all namespaces in the cluster.

This procedure updates the harbor-secret image pull secret across all required namespaces using OpenTofu (tofu) -replace flags. This process does not require a reinstall of the full cluster.

Note: Existing running pods are unaffected by this operation. Only new image pulls use the rotated secret, for example, new pods, scaling events, and node replacements. If existing pods need to use the new credentials immediately, restart the affected deployments after applying the settings.

When to Use This Procedure

The following table outlines common scenarios and the frequency for updating registry credentials in PPC:

Prerequisites

Before you begin, ensure that the following requirements are met:

• You have access to the PPC Kubernetes cluster, kubectl is configured.
• OpenTofu (tofu) is installed.
• You have the new registry username and password.
• You have permission to manage secrets across namespaces in the cluster.
• The PPC directory is available at /eclipse-init/iac/.

Target Namespaces

The harbor-secret is managed in the following namespaces:

Procedure

This procedure must be run once per cloud, that is, AWS for Elastic Kubernetes Service (EKS) and Azure for Azure Kubernetes Service (AKS). Complete all steps for one cloud before proceeding to the next.

Follow these steps for each cloud environment:

1. Export the credentials for the new registry as environment variables.

   export TF_VAR_username='<new-registry-username>'
   export TF_VAR_password='<new-registry-password>'
   

2. Verify that the credentials are set.

   [[ -n "$TF_VAR_username" && -n "$TF_VAR_password" ]] && echo "creds set" || echo "MISSING"
   

3. Set the Kubernetes context to prevent accidentally writing secrets to the wrong cluster:

   CLOUD=aws                              # or: azure
   KCTX=<your-cluster-context-name>
   
   kubectl config use-context "$KCTX"
   kubectl config current-context         # must match the intended cluster
   

   Note: Operating on the wrong cluster will update secrets in the incorrect environment.

4. Navigate to the iac directory.

   cd iac/azure/cluster/modules/05_helm_releases
   

5. Plan the credential update that targets only the harbor-secret resources across all namespaces.

   NAMESPACES=(pty-admin api-gateway pty-loginui cert-manager karpenter default \
               pty-backup-recovery email-service cli reloader pty-insight)
   
   REPLACE_ARGS=()
   for ns in "${NAMESPACES[@]}"; do
     REPLACE_ARGS+=(-replace="module.helm_releases.module.common.kubernetes_secret_v1.harbor_secret[\"$ns\"]")
   done
   
   tofu plan "${REPLACE_ARGS[@]}" -out="rotate-${CLOUD}.tfplan"
   

6. Review the plan. Expected output:

   Plan: 11 to add, 0 to change, 11 to destroy.
   

   Or equivalently, 11 to replace.

   The plan must affect only harbor_secret resources. If any other resources appear in the plan, stop immediately and delete the plan file.

   rm "rotate-${CLOUD}.tfplan"
   

   Note: Investigate the unexpected changes before retrying.

7. Apply the plan.

   tofu apply "rotate-${CLOUD}.tfplan"
   

   After successfully applying the plan, clean up the plan file. Use the command below:

   rm "rotate-${CLOUD}.tfplan"
   

   Note: The lifecycle.ignore_changes = [data] directive on the secret resource is bypassed by -replace because the resource is destroyed and recreated, not updated in place.

8. Verify the update. Confirm that all harbor-secret resources were recreated with current timestamps.

   for ns in pty-admin api-gateway pty-loginui cert-manager karpenter default \
             pty-backup-recovery email-service cli reloader pty-insight; do
     kubectl -n "$ns" get secret harbor-secret \
       -o jsonpath='{.metadata.namespace}{"\t"}{.metadata.creationTimestamp}{"\n"}'
   done
   

   All timestamps must reflect the time of the apply operation.

9. Reset the context variables, then repeat from step 4 for the next cloud.

   CLOUD=azure
   KCTX=<your-azure-context-name>
   

Optional Post-Update Step: Rolling Deployments

If the running pods need to use the new credentials immediately, rather than waiting for the next pod restart, roll the affected deployments.

kubectl rollout restart deployment -n <namespace> <deployment-name>

Troubleshooting