Upgrade

• 1: Automated script-based upgrade
• 2: Manual component-based upgrade

1 - Automated script-based upgrade

Before you begin

Before starting an upgrade, ensure the following conditions are met:

• Access to the jump box associated with the PPC v1.1.0 cluster is required. This must be the same jump box used during the PPC v1.1.0 installation.

• Create a Velero backup and an OpenSearch snapshot to ensure data recovery is possible, in case of an upgrade failure.

• Ensure that the context is set to the cluster where the upgrade operation is to be performed. Run the following command:

  kubectl config current-context
  

  The cluster name in the output must match the cluster installed with PPC v1.1.0.

• Before running the bootstrap scripts as the root user on RHEL, ensure that /usr/local/bin, and the Azure CLI binary path, if applicable, is included in the $PATH. Alternatively, run the script using a non-root user, such as azureuser, where /usr/local/bin is already part of the default $PATH.

• Do not install or manage multiple clusters from the same working directory. Each cluster deployment maintains its own Terraform or OpenTofu state, and reusing a directory can overwrite state files, causing loss of cluster tracking and unintended cleanup behavior.
  Use a dedicated directory, per cluster.

Required Permissions

The Azure principal performing the upgrade must have the following RBAC roles assigned.

Required container registry Secrets

Export the container registry credentials as environment variables.

export TF_VAR_username='<registry-username>'
export TF_VAR_password='<registry-password>'

Values to collect before starting

Gather the following values before running the upgrade command.

Upgrading PPC

The upgrade process downloads the latest images from the repository and deploys the updated containers to the target environment.

To upgrade the PPC from v1.1.0 to v1.2.0, perform the following steps:

1. Create a upgrade_120 directory on the jump box and navigate to it.

   mkdir upgrade_120 && cd upgrade_120
   

2. Download and extract the PCT to the upgrade_120 directory.

3. Authenticate the Azure CLI and select the target subscription. Terraform and the Azure CLI operations invoked by the upgrade script rely on an active az session.

   If the session has expired or you are not logged in, run the following commands to authenticate and confirm the active account:

   az login
   az account show
   

   Note: CLI login using Single Sign-On (SSO) is not supported. Use az login --use-device-code or authenticate as a Service Principal.

   Set the active subscription to the one that hosts the AKS cluster.

   az account set --subscription <subscription-ID>
   

   Replace the placeholder with the value collected in Values to collect before starting. Ensure the target subscription, resource group, managed identities, subnet, and private DNS zone are accessible before continuing.

4. Verify the active context points to the correct cluster using the following command:

    kubectl config current-context
   

5. Configure the container registry pull secret based on your registry type:

• If using a private registry with authentication: The upgrade must recreate the Harbor pull secret. Navigate to /iac/azure/cluster/modules/05_helm_releases/ under the deployment_120 directory. Edit the terraform.tfvars file to update the value of harbor_secret parameter with harbor-secret.

• If the registry supports anonymous image pulls: No action is required. Ensure the harbor_secret value in the terraform.tfvars file is empty.

Note: Ensure that the value of the harbor_secret parameter in the 05_helm_releases module is the same in both installation and upgrade terraform.tfvars files.

6. Initiate the upgrade from the upgrade_120 directory using the following command:

   ./upgrade.sh --cloud azure --resource-group <resource-group> --storage-account <storage-account-name> --subscription-id <subscription-ID> --cluster <cluster-name>
   

   Replace the placeholders with the environment-specific values.

After the upgrade is successful, a Upgrade complete for all stages message is displayed.

The upgrade typically takes 10–15 minutes approximately, but duration may vary.

Post Upgrade Steps

After the upgrade completes, perform the following verification steps to confirm the cluster is healthy.

1. Verify all nodes are Ready

To verify if all nodes are in the ready state, run the following command:

kubectl get nodes -o wide

2. Verify all pods are Running

To verify if all pods are in the running state, run the following command:

kubectl get pods -A

3. Verify Helm releases

To verify if the helm releases are in the deployed state, run the following command:

helm list -A

4. Directory Maintenance after Upgrade

During upgrade, both the previous and the new PPC installation directories must be retained.

• The original directory contains Terraform state and deployment metadata.
• The new directory contains the updated configuration and binaries.

Both directories are required for proper management of the deployment.

Reverting to a previous version of PPC

To revert to a previous version of PPC, restore the cluster using the steps in Restoring the PPC.

2 - Manual component-based upgrade

Before you begin

Before starting an upgrade, ensure the following conditions are met:

• Access to the jump box associated with the PPC v1.1.0 cluster is required. This must be the same jump box used during the PPC v1.1.0 installation.

• Create a Velero backup and an OpenSearch snapshot to ensure data recovery is possible, in case of an upgrade failure.

• The context is set to the cluster where the upgrade operation is to be performed. Run the following command:

  kubectl config current-context
  

  The cluster name in the output must match the cluster installed with PPC v1.1.0.

• Do not install or manage multiple clusters from the same working directory. Each cluster deployment maintains its own Terraform or OpenTofu state, and reusing a directory can overwrite state files, causing loss of cluster tracking and unintended cleanup behavior.
  Use a dedicated directory, per cluster.

Jump box requirement

Ensure that the entire process is performed using the same jump box used for the original installation, with all tools already installed.

Required permissions

Required roles

The Azure principal must have AKS cluster admin access.

The required Azure RBAC Roles are listed in the following table:

Required container registry secrets

Export the container registry credentials as environment variables.

export TF_VAR_username='<registry-username>'
export TF_VAR_password='<registry-password>'

Values to collect before starting

Gather the following values before starting the upgrade.

Upgrading PPC using component-based approach

To upgrade the PPC from v1.1.0 to v1.2.0 using the component-based approach, perform the following steps:

Upgrading the PPC

To deploy the PPC using component-based installation, perform the following steps:

1. Authenticating with Azure CLI

If you are already authenticated, run the following command to verify your session:

az account show

If the session has expired or you are not logged in, run the following command to authenticate:

az login
az account show

Note: CLI login using Single Sign-On (SSO) is not supported. Use az login --use-device-code or authenticate as a Service Principal.

Ensure the target subscription, resource group, managed identities, subnet, and private DNS zone are accessible before continuing.

2. Downloading and extracting the PCT

Create a dedicated directory for the new release and extract the PCT.

1. Create a working directory for the upgrade and navigate to it.

mkdir upgrade_120 && cd upgrade_120

2. Download and extract the archive to the upgrade_120 directory.

3. Obtaining terraform.tfvars from blob storage

Each module’s terraform.tfvars is backed up to Azure Blob Storage during the original installation. These files are available on the Azure portal under the backup storage account in the infrastructure-manifest container.

4. Configuring container registry secrets

Configure the container registry pull secret based on your registry type:

• If using a private registry with authentication: The upgrade must recreate the Harbor pull secret. Navigate to /iac/azure/cluster/modules/05_helm_releases/ under the deployment_120 directory. Edit the terraform.tfvars file to update the value of harbor_secret parameter with harbor-secret.

• If the registry supports anonymous image pulls: No action is required. Ensure the harbor_secret value in the terraform.tfvars file is empty.

Note: Ensure that the value of the harbor_secret parameter in the 05_helm_releases module is the same in both installation and upgrade terraform.tfvars files.

5. Modifying and applying terraform.tfvars files

For each module, copy the terraform.tfvars from blob storage and save to the upgrade directory, then apply the Terraform configuration before proceeding to the next module.

To navigate to the modules directory, run the following command:

cd upgrade_120/iac/azure/cluster/modules

00_backup_storage

a. Navigate to the 00_backup_storage directory.

cd 00_backup_storage

b. Copy the terraform.tfvars from blob storage and save to the upgrade 00_backup_storage/terraform.tfvars.

az storage blob download \
 --account-name <storage-account-name> \
 --container-name <container-name> \
 --name infrastructure-manifest/<cluster-name>/tfvars/00_backup_storage/terraform.tfvars \
 --file iac/azure/cluster/modules/00_backup_storage/terraform.tfvars \
 --auth-mode login

c. Restore the existing state file for 00_backup_storage to prevent tofu apply from attempting to recreate existing resources.

az storage blob download \
  --account-name <storage-account-name> \
  --container-name <container-name> \
  --name infrastructure-manifest/<cluster-name>/states/00_backup_storage.tfstate \
  --file terraform.tfstate \
  --auth-mode login

d. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

e. Synchronize the state file from 00_backup_bucket to the Azure storage account.

bootstrap-scripts/sync-backup-storage-state-to-azure.sh

01_identity

a. Navigate to the 01_identity directory.

cd 01_identity

b. Copy the terraform.tfvars from blob storage and save to the upgrade 01_identity/terraform.tfvars.

az storage blob download \
 --account-name <storage-account-name> \
 --container-name <container-name> \
 --name infrastructure-manifest/<cluster-name>/tfvars/01_identity/terraform.tfvars \
 --file iac/azure/cluster/modules/01_identity/terraform.tfvars \
 --auth-mode login

c. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

02_aks_cluster

a. Navigate to the 02_aks_cluster directory.

cd 02_aks_cluster

b. Copy the terraform.tfvars from blob storage and save to the upgrade 02_aks_cluster/terraform.tfvars.

az storage blob download \
 --account-name <storage-account-name> \
 --container-name <container-name> \
 --name infrastructure-manifest/<cluster-name>/tfvars/02_aks_cluster/terraform.tfvars \
 --file iac/azure/cluster/modules/02_aks_cluster/terraform.tfvars \
 --auth-mode login

c. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

03_node_pool

a. Navigate to the 03_node_pool directory.

cd 03_node_pool

b. Copy the terraform.tfvars from blob storage and save to the upgrade 03_node_pool/terraform.tfvars.

az storage blob download \
 --account-name <storage-account-name> \
 --container-name <container-name> \
 --name infrastructure-manifest/<cluster-name>/tfvars/03_node_pool/terraform.tfvars \
 --file iac/azure/cluster/modules/03_node_pool/terraform.tfvars \
 --auth-mode login

c. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

04_aks_addons

a. Navigate to the 04_aks_addons directory.

cd 04_aks_addons

b. Copy the terraform.tfvars from blob storage and save to the upgrade 04_aks_addons/terraform.tfvars.

az storage blob download \
 --account-name <storage-account-name> \
 --container-name <container-name> \
 --name infrastructure-manifest/<cluster-name>/tfvars/04_aks_addons/terraform.tfvars \
 --file iac/azure/cluster/modules/04_aks_addons/terraform.tfvars \
 --auth-mode login

c. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

05_helm_releases

a. Navigate to the 05_helm_releases directory.

cd 05_helm_releases

b. Copy the terraform.tfvars from blob storage and save to the upgrade 05_helm_releases/terraform.tfvars.

az storage blob download \
 --account-name <storage-account-name> \
 --container-name <container-name> \
 --name infrastructure-manifest/<cluster-name>/tfvars/05_helm_releases/terraform.tfvars \
 --file iac/azure/cluster/modules/05_helm_releases/terraform.tfvars \
 --auth-mode login

c. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

6. Verifying the upgrade

After the upgrade completes, perform the following verification steps to confirm the cluster is healthy.

Verify all nodes are Ready

To verify if all nodes are in the Ready state, run the following command:

kubectl get nodes -o wide

Verify all pods are Running

To verify if all pods are in the Running state, run the following command:

kubectl get pods -A

Verify Helm releases

To verify if the helm releases are in the Deployed state, run the following command:

helm list -A

All nodes must be in the Ready state, all pods must be in the Running or Completed state, and the Velero backup storage location must show Available.

Reverting to a previous version of PPC

To revert to a previous version of PPC, restore the cluster using the steps in Restoring the PPC.