Automated script-based upgrade

Before you begin

Before starting an upgrade, ensure the following conditions are met:

• Access to the jump box associated with the PPC v1.1.0 cluster is required. This must be the same jump box used during the PPC v1.1.0 installation.

• Create a Velero backup and an OpenSearch snapshot to ensure data recovery is possible.

• Ensure that the context is set to the cluster where the upgrade operation is to be performed. Run the following command:

  kubectl config current-context
  

  The cluster name in the output must match the cluster installed with PPC v1.1.0.

• Before running the bootstrap scripts as the root user on RHEL, ensure that /usr/local/bin, and the AWS CLI binary path, if applicable, is included in the $PATH. Alternatively, run the script using a non-root user, such as ec2-user where /usr/local/bin is already part of the default $PATH.

• Do not install or manage multiple clusters from the same working directory. Each cluster deployment maintains its own Terraform or OpenTofu state, and reusing a directory can overwrite state files, causing loss of cluster tracking and unintended cleanup behavior.
  Use a dedicated directory, per cluster.

Required Permissions

The IAM user performing the upgrade must have the same permissions as those required for installation.

For the full list of required IAM, S3, KMS, EC2, and EKS permissions, refer to Prerequisites.

Required container registry Secrets

Export the container registry credentials as environment variables.

export TF_VAR_username='<registry-username>'
export TF_VAR_password='<registry-password>'

Values to collect before starting

Gather the following values before running the upgrade command.

Using Standard Bottlerocket AMI Deployment

Follow this section only if the upgraded cluster must run on standard Bottlerocket AMIs. Common reasons include:

• The AWS region does not offer the required endpoints or the secure Bottlerocket AMI. For the current list of endpoints by region, refer to the AWS documentation.
• The SSM parameter /aws/service/bottlerocket/aws-k8s-<eks-version>-fips/<arch>/latest/image_id does not resolve in the target region. Verify with aws ssm get-parameter --region <region> --name "/aws/service/bottlerocket/aws-k8s-<eks-version>-fips/<arch>/latest/image_id"; a ParameterNotFound response indicates that the secure AMI is not published in that region.
• The original PPC v1.1.0 cluster was installed with the standard configuration and the upgrade must preserve that state.

If none of the above applies, skip this section and proceed to Upgrading PPC with the default secure Bottlerocket AMI configuration.

Upgrading PPC

To upgrade the PPC from v1.1.0 to v1.2.0, perform the following steps:

1. Create a deployment_120 directory on the jump box and navigate to it.

   mkdir deployment_120 && cd deployment_120
   

2. Download and extract PCT to the deployment_120 directory.

3. Configure the container registry pull secret based on your registry type:

   • If using a private registry with authentication: The upgrade must recreate the Harbor pull secret. Navigate to /iac/aws/cluster/modules/05_helm_releases/ under the deployment_120 directory. Edit the terraform.tfvars file to update the value of harbor_secret parameter with harbor-secret.

   • If the registry supports anonymous image pulls: No action is required. Ensure the harbor_secret value in the terraform.tfvars file is empty.

   Note: Ensure that the value of the harbor_secret parameter in the 05_helm_releases module is the same in both installation and upgrade terraform.tfvars files.

4. Verify the active context points to the correct cluster using the following command:

    kubectl config current-context
   

5. Export the AWS credentials required by Terraform and the AWS CLI operations that the upgrade script invokes.

   Note: While using temporary credentials, such as, using AWS STS, ensure the session token is included and that the credentials have not expired before proceeding.

   To export the AWS credentials, run the following commands:

   export AWS_ACCESS_KEY_ID=<your-access-key-id>
   export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
   export AWS_SESSION_TOKEN=<your-session-token>
   

6. Initiate the upgrade from the deployment_120 directory using the following command:

   ./upgrade.sh --cloud aws --bucket <s3-bucket> --region <region> --cluster <cluster-name>
   

   Ensure that you replace the placeholders with your environment-specific values:

   Placeholder	Description	Where to find it
   <s3-bucket>	Name of the AWS S3 bucket used to store Terraform state.	Backup bucket configured during PPC v1.1.0 installation.
   <region>	Name of the AWS region where the EKS cluster is deployed.	AWS Management Console or your installation configuration.
   <cluster-name>	Name of the EKS cluster to upgrade.	Confirmed in step 4 from the kubectl config current-context output.

After the upgrade is successful, a Upgrade complete for all stages message is displayed.

The upgrade typically takes 10–15 minutes, but duration may vary.

Post Upgrade Steps

1. Verify all nodes are Ready

To verify if all nodes are in the ready state, run the following command:

kubectl get nodes -o wide

2. Verify all pods are Running

To verify if all pods are in the running state, run the following command:

kubectl get pods -A

3. Verify Helm releases

To verify if the helm releases are in the deployed state, run the following command:

helm list -A

4. Directory Maintenance after Upgrade

During upgrade, both the previous and the new PPC installation directories must be retained.

• The original directory contains Terraform state and deployment metadata.
• The new directory contains the updated configuration and binaries.

Both directories are required for proper management of the deployment.

Recovering from a Failed Upgrade

If the session is terminated during upgrade due to network issues, power outage, or any such issues, then the upgrade process stops. The ./upgrade.sh script resumes the upgrade.

To restart the process, run the following command:

./upgrade.sh --cloud aws --bucket <s3-bucket> --region <region> --cluster <cluster-name>

Use the same <s3-bucket>, <region>, and <cluster-name> values as in step 6.

Warning: Do not install or manage multiple clusters from the same working directory. Each cluster deployment maintains its own Terraform or OpenTofu state, and reusing a directory can overwrite state files, causing loss of cluster tracking and unintended cleanup behavior.
Use a dedicated directory, and jump box, where possible, per cluster, and always verify the active kubectl context before running cleanup commands such as ./bootstrap.sh --cloud aws destroy.
To check the active kubectl context, run the following command:
kubectl config current-context

Reverting to a previous version of PPC

To revert to a previous version of PPC, restore the cluster using the steps in Restoring the PPC.

Configuring for a Standard Deployment

Note: The following instructions are optional and only required for configuring a standard PPC deployment on AWS. Skip these steps if you have already installed the secure Bottlerocket AMIs.

To deploy a standard cluster, the following Terraform configuration files must be modified to use the standard Bottlerocket AMI types and SSM parameter paths before performing the upgrade process.

For a standard PPC upgrade on AWS EKS, perform the following steps:

1. Update the node group AMI type map in the /03_node_group/variables.tf file.

   a. Navigate to /03_node_group directory, using the following command:

   cd iac/aws/cluster/modules/03_node_group/
   

   b. Edit the variables.tf file to make the following changes.

   Change from (secure Bottlerocket AMI defaults):

   variable "node_group_ami_type_map" {
   description = "Fallback EKS AMI type per architecture, used when node_group_ami_type is empty."
   type        = map(string)
   default = {
      amd64 = "BOTTLEROCKET_x86_64_FIPS"
      arm64 = "BOTTLEROCKET_ARM_64_FIPS"
   }
   }
   

   Change to (standard Bottlerocket AMI values):

   variable "node_group_ami_type_map" {
   description = "Fallback EKS AMI type per architecture, used when node_group_ami_type is empty."
   type        = map(string)
   default = {
      amd64 = "BOTTLEROCKET_x86_64"
      arm64 = "BOTTLEROCKET_ARM_64"
   }
   }
   

2. Update the Karpenter AMI SSM parameter path in the /05_helm_releases/main.tf file.

   The karpenter_fips_ami_ssm_parameter value specifies the AWS Systems Manager (SSM) parameter path that Karpenter uses to resolve the correct Bottlerocket AMI for worker nodes. The default path targets the secure Bottlerocket AMI. For a standard cluster, remove the -fips segment from the EKS version portion of the path.

   a. Navigate to /05_helm_releases directory, using the following command:

   cd iac/aws/cluster/modules/05_helm_releases/
   

   b. Edit the main.tf file to make the following changes.

   Change from (secure Bottlerocket AMI SSM path):

   karpenter_fips_ami_ssm_parameter = "/aws/service/bottlerocket/aws-k8s-${data.aws_eks_cluster.this.version}-fips/${local.karpenter_ami_arch}/latest/<image_id>"
   

   Change to (standard Bottlerocket AMI SSM path):

   karpenter_fips_ami_ssm_parameter = "/aws/service/bottlerocket/aws-k8s-${data.aws_eks_cluster.this.version}/${local.karpenter_ami_arch}/latest/<image_id>"