Upgrade

• 1: Automated script-based upgrade
• 2: Manual component-based upgrade

1 - Automated script-based upgrade

Before you begin

Before starting an upgrade, ensure the following conditions are met:

• Access to the jump box associated with the PPC v1.1.0 cluster is required. This must be the same jump box used during the PPC v1.1.0 installation.

• Create a Velero backup and an OpenSearch snapshot to ensure data recovery is possible.

• Ensure that the context is set to the cluster where the upgrade operation is to be performed. Run the following command:

  kubectl config current-context
  

  The cluster name in the output must match the cluster installed with PPC v1.1.0.

• Before running the bootstrap scripts as the root user on RHEL, ensure that /usr/local/bin, and the AWS CLI binary path, if applicable, is included in the $PATH. Alternatively, run the script using a non-root user, such as ec2-user where /usr/local/bin is already part of the default $PATH.

• Do not install or manage multiple clusters from the same working directory. Each cluster deployment maintains its own Terraform or OpenTofu state, and reusing a directory can overwrite state files, causing loss of cluster tracking and unintended cleanup behavior.
  Use a dedicated directory, per cluster.

Required Permissions

The IAM user performing the upgrade must have the same permissions as those required for installation.

For the full list of required IAM, S3, KMS, EC2, and EKS permissions, refer to Prerequisites.

Required container registry Secrets

Export the container registry credentials as environment variables.

export TF_VAR_username='<registry-username>'
export TF_VAR_password='<registry-password>'

Values to collect before starting

Gather the following values before running the upgrade command.

Using Standard Bottlerocket AMI Deployment

Follow this section only if the upgraded cluster must run on standard Bottlerocket AMIs. Common reasons include:

• The AWS region does not offer the required endpoints or the secure Bottlerocket AMI. For the current list of endpoints by region, refer to the AWS documentation.
• The SSM parameter /aws/service/bottlerocket/aws-k8s-<eks-version>-fips/<arch>/latest/image_id does not resolve in the target region. Verify with aws ssm get-parameter --region <region> --name "/aws/service/bottlerocket/aws-k8s-<eks-version>-fips/<arch>/latest/image_id"; a ParameterNotFound response indicates that the secure AMI is not published in that region.
• The original PPC v1.1.0 cluster was installed with the standard configuration and the upgrade must preserve that state.

If none of the above applies, skip this section and proceed to Upgrading PPC with the default secure Bottlerocket AMI configuration.

Upgrading PPC

To upgrade the PPC from v1.1.0 to v1.2.0, perform the following steps:

1. Create a deployment_120 directory on the jump box and navigate to it.

   mkdir deployment_120 && cd deployment_120
   

2. Download and extract PCT to the deployment_120 directory.

3. Configure the container registry pull secret based on your registry type:

   • If using a private registry with authentication: The upgrade must recreate the Harbor pull secret. Navigate to /iac/aws/cluster/modules/05_helm_releases/ under the deployment_120 directory. Edit the terraform.tfvars file to update the value of harbor_secret parameter with harbor-secret.

   • If the registry supports anonymous image pulls: No action is required. Ensure the harbor_secret value in the terraform.tfvars file is empty.

   Note: Ensure that the value of the harbor_secret parameter in the 05_helm_releases module is the same in both installation and upgrade terraform.tfvars files.

4. Verify the active context points to the correct cluster using the following command:

    kubectl config current-context
   

5. Export the AWS credentials required by Terraform and the AWS CLI operations that the upgrade script invokes.

   Note: While using temporary credentials, such as, using AWS STS, ensure the session token is included and that the credentials have not expired before proceeding.

   To export the AWS credentials, run the following commands:

   export AWS_ACCESS_KEY_ID=<your-access-key-id>
   export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
   export AWS_SESSION_TOKEN=<your-session-token>
   

6. Initiate the upgrade from the deployment_120 directory using the following command:

   ./upgrade.sh --cloud aws --bucket <s3-bucket> --region <region> --cluster <cluster-name>
   

   Ensure that you replace the placeholders with your environment-specific values:

   Placeholder	Description	Where to find it
   <s3-bucket>	Name of the AWS S3 bucket used to store Terraform state.	Backup bucket configured during PPC v1.1.0 installation.
   <region>	Name of the AWS region where the EKS cluster is deployed.	AWS Management Console or your installation configuration.
   <cluster-name>	Name of the EKS cluster to upgrade.	Confirmed in step 4 from the kubectl config current-context output.

After the upgrade is successful, a Upgrade complete for all stages message is displayed.

The upgrade typically takes 10–15 minutes, but duration may vary.

Post Upgrade Steps

1. Verify all nodes are Ready

To verify if all nodes are in the ready state, run the following command:

kubectl get nodes -o wide

2. Verify all pods are Running

To verify if all pods are in the running state, run the following command:

kubectl get pods -A

3. Verify Helm releases

To verify if the helm releases are in the deployed state, run the following command:

helm list -A

4. Directory Maintenance after Upgrade

During upgrade, both the previous and the new PPC installation directories must be retained.

• The original directory contains Terraform state and deployment metadata.
• The new directory contains the updated configuration and binaries.

Both directories are required for proper management of the deployment.

Recovering from a Failed Upgrade

If the session is terminated during upgrade due to network issues, power outage, or any such issues, then the upgrade process stops. The ./upgrade.sh script resumes the upgrade.

To restart the process, run the following command:

./upgrade.sh --cloud aws --bucket <s3-bucket> --region <region> --cluster <cluster-name>

Use the same <s3-bucket>, <region>, and <cluster-name> values as in step 6.

Warning: Do not install or manage multiple clusters from the same working directory. Each cluster deployment maintains its own Terraform or OpenTofu state, and reusing a directory can overwrite state files, causing loss of cluster tracking and unintended cleanup behavior.
Use a dedicated directory, and jump box, where possible, per cluster, and always verify the active kubectl context before running cleanup commands such as ./bootstrap.sh --cloud aws destroy.
To check the active kubectl context, run the following command:
kubectl config current-context

Reverting to a previous version of PPC

To revert to a previous version of PPC, restore the cluster using the steps in Restoring the PPC.

Configuring for a Standard Deployment

Note: The following instructions are optional and only required for configuring a standard PPC deployment on AWS. Skip these steps if you have already installed the secure Bottlerocket AMIs.

To deploy a standard cluster, the following Terraform configuration files must be modified to use the standard Bottlerocket AMI types and SSM parameter paths before performing the upgrade process.

For a standard PPC upgrade on AWS EKS, perform the following steps:

1. Update the node group AMI type map in the /03_node_group/variables.tf file.

   a. Navigate to /03_node_group directory, using the following command:

   cd iac/aws/cluster/modules/03_node_group/
   

   b. Edit the variables.tf file to make the following changes.

   Change from (secure Bottlerocket AMI defaults):

   variable "node_group_ami_type_map" {
   description = "Fallback EKS AMI type per architecture, used when node_group_ami_type is empty."
   type        = map(string)
   default = {
      amd64 = "BOTTLEROCKET_x86_64_FIPS"
      arm64 = "BOTTLEROCKET_ARM_64_FIPS"
   }
   }
   

   Change to (standard Bottlerocket AMI values):

   variable "node_group_ami_type_map" {
   description = "Fallback EKS AMI type per architecture, used when node_group_ami_type is empty."
   type        = map(string)
   default = {
      amd64 = "BOTTLEROCKET_x86_64"
      arm64 = "BOTTLEROCKET_ARM_64"
   }
   }
   

2. Update the Karpenter AMI SSM parameter path in the /05_helm_releases/main.tf file.

   The karpenter_fips_ami_ssm_parameter value specifies the AWS Systems Manager (SSM) parameter path that Karpenter uses to resolve the correct Bottlerocket AMI for worker nodes. The default path targets the secure Bottlerocket AMI. For a standard cluster, remove the -fips segment from the EKS version portion of the path.

   a. Navigate to /05_helm_releases directory, using the following command:

   cd iac/aws/cluster/modules/05_helm_releases/
   

   b. Edit the main.tf file to make the following changes.

   Change from (secure Bottlerocket AMI SSM path):

   karpenter_fips_ami_ssm_parameter = "/aws/service/bottlerocket/aws-k8s-${data.aws_eks_cluster.this.version}-fips/${local.karpenter_ami_arch}/latest/<image_id>"
   

   Change to (standard Bottlerocket AMI SSM path):

   karpenter_fips_ami_ssm_parameter = "/aws/service/bottlerocket/aws-k8s-${data.aws_eks_cluster.this.version}/${local.karpenter_ami_arch}/latest/<image_id>"
   

2 - Manual component-based upgrade

Before you begin

Before starting an upgrade, ensure the following conditions are met:

• Access to the jump box associated with the PPC v1.1.0 cluster is required. This must be the same jump box used during the PPC v1.1.0 installation.

• Create a Velero backup and an OpenSearch snapshot to ensure data recovery is possible.

• Ensure that the context is set to the cluster where the upgrade operation is to be performed. Run the following command:

  kubectl config current-context
  

  The cluster name in the output must match the cluster installed with PPC v1.1.0.

• Before running the bootstrap as the root user on RHEL, ensure that /usr/local/bin, and the AWS CLI binary path, if applicable, is included in the $PATH. Alternatively, run the script using a non-root user, such as ec2-user where /usr/local/bin is already part of the default $PATH.

• Do not install or manage multiple clusters from the same working directory. Each cluster deployment maintains its own Terraform or OpenTofu state, and reusing a directory can overwrite state files, causing loss of cluster tracking and unintended cleanup behavior.
  Use a dedicated directory, per cluster.

Required permissions

The IAM user performing the upgrade must have the same permissions as those required for installation.

For the full list of required IAM, S3, KMS, EC2, and EKS permissions, refer to Prerequisites.

Jump box requirement

Ensure that the entire process is performed using the same jump box.

Required container registry secrets

Export the container registry credentials as environment variables.

export TF_VAR_username='<registry-username>'
export TF_VAR_password='<registry-password>'

Parameter values

The following values are required before starting the upgrade. Configuration such as VPC, subnets, FQDN, and so on, are already stored in the terraform.tfvars files in S3 and does not need to be re-entered.

Access to terraform.tfvars file from PPC v1.1.0

The terraform.tfvars files for each module are stored in the AWS S3 backup bucket from the PPC v1.1.0 installation.

In the AWS Management Console, navigate to Amazon S3 > Buckets.
Select the S3 bucket used for the installation of PPC v1.1.0, then navigate to infrastructure-manifest > cluster-name > tfvars.
For each module, open the corresponding folder and select the terraform.tfvars file. Copy the AWS S3 URI shown on the file’s detail page. This is the URI required for the aws s3 cp commands in Modifying and applying terraform.tfvars files.

Upgrading PPC using component-based approach

To upgrade the PPC from v1.1.0 to v1.2.0 using the component-based approach, perform the following steps:

1. Exporting AWS credentials

The installation process uses Terraform to provision and manage AWS resources, such as VPCs, EKS clusters, IAM roles, and S3 buckets. The AWS credentials must be exported as environment variables to authenticate these operations.

Note: While using temporary credentials, such as, using AWS STS, ensure the session token is included and that the credentials have not expired before proceeding.
CLI login using AWS SSO (IAM Identity Center) is not supported. Export IAM user credentials or temporary STS credentials as environment variables.

If the components are installed by different users, ensure that each user must export their AWS credentials. This ensures that all the required access and permissions are available.

To export the AWS credentials, run the following commands:

export AWS_ACCESS_KEY_ID=<your-access-key-id>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
export AWS_SESSION_TOKEN=<your-session-token>

2. Downloading and extracting the PCT

a. Create a deployment_120 directory on the jump box and navigate to it.

mkdir deployment_120 && cd deployment_120

b. Download and extract the PPC-K8S-ALL_1.2.0.x.tar PCT to the deployment_120 directory.

Note: For more information about the PCT, refer to My.Protegrity or the release notes.

3. Configuring container registry secrets

Configure the container registry pull secret based on your registry type:

• If using a private registry with authentication: The upgrade must recreate the Harbor pull secret. Navigate to /iac/aws/cluster/modules/05_helm_releases/ under the deployment_120 directory. Edit the terraform.tfvars file to update the value of harbor_secret parameter with harbor-secret.

• If the registry supports anonymous image pulls: No action is required. Ensure the harbor_secret value in the terraform.tfvars file is empty.

Note: Ensure that the value of the harbor_secret parameter in the 05_helm_releases module is the same in both installation and upgrade terraform.tfvars files.

4. Modifying and applying terraform.tfvars files

For each module, copy the terraform.tfvars from AWS S3 and save to the upgrade directory, then apply the Terraform configuration before proceeding to the next module.

To navigate to the modules directory, run the following command:

cd deployment_120/iac/aws/cluster/modules

00_backup_bucket

This module reconciles the existing S3 backup bucket state with the new working directory.

a. Navigate to the 00_backup_bucket directory.

cd 00_backup_bucket

b. Copy the terraform.tfvars from S3 and save to the upgrade 00_backup_bucket/terraform.tfvars.

aws s3 cp s3://<s3-bucket>/infrastructure-manifest/<cluster-name>/tfvars/00_backup_bucket/terraform.tfvars terraform.tfvars

c. Copy the terraform.tfstate from S3 to preserve the existing resource state inside 00_backup_bucket.

aws s3 cp s3://<s3-bucket>/infrastructure-manifest/<cluster-name>/states/00_backup_bucket.tfstate terraform.tfstate

d. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

e. Navigate to the bootstrap-scripts directory.

cd deployment_120/bootstrap-scripts

f. Run the following script to copy the state files to the S3 bucket:

./sync-backup-bucket-state-to-s3.sh

01_iam_roles

This module applies IAM role configuration for the EKS cluster, node group, and service accounts.

a. Navigate to the 01_iam_roles directory.

cd 01_iam_roles

b. Copy the terraform.tfvars from S3 and save to the upgrade 01_iam_roles/terraform.tfvars.

aws s3 cp s3://<s3-bucket>/infrastructure-manifest/<cluster-name>/tfvars/01_iam_roles/terraform.tfvars terraform.tfvars

c. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

02_eks_cluster

This module applies the EKS cluster, VPC security group, and networking add-ons configuration.

a. Navigate to the 02_eks_cluster directory.

cd 02_eks_cluster

b. Copy the terraform.tfvars from S3 and save to the upgrade 02_eks_cluster/terraform.tfvars.

aws s3 cp s3://<s3-bucket>/infrastructure-manifest/<cluster-name>/tfvars/02_eks_cluster/terraform.tfvars terraform.tfvars

c. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

03_node_group

This module applies the EKS worker node group configuration.

a. Navigate to the 03_node_group directory.

cd 03_node_group

b. Copy the terraform.tfvars from S3 and save to the upgrade 03_node_group/terraform.tfvars.

aws s3 cp s3://<s3-bucket>/infrastructure-manifest/<cluster-name>/tfvars/03_node_group/terraform.tfvars terraform.tfvars

c. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

04_eks_addons

This module applies EKS add-ons configuration including CoreDNS, Metrics Server, EBS CSI driver, and snapshot controller.

a. Navigate to the 04_eks_addons directory.

cd 04_eks_addons

b. Copy the terraform.tfvars from S3 and save to the upgrade 04_eks_addons/terraform.tfvars.

aws s3 cp s3://<s3-bucket>/infrastructure-manifest/<cluster-name>/tfvars/04_eks_addons/terraform.tfvars terraform.tfvars

c. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

05_helm_releases

This module applies all PPC platform Helm chart configurations.

a. Navigate to the 05_helm_releases directory.

cd 05_helm_releases

b. Copy the terraform.tfvars from S3 and save to the upgrade 05_helm_releases/terraform.tfvars.

aws s3 cp s3://<s3-bucket>/infrastructure-manifest/<cluster-name>/tfvars/05_helm_releases/terraform.tfvars terraform.tfvars

c. To apply the terraform configuration, run the following command:

tofu init
tofu plan   # review the plan that will be created
tofu apply  # when prompted with "Enter a value:", type "yes"

Post Upgrade Steps

1. Verify all nodes are Ready

To verify if all nodes are in the ready state, run the following command:

kubectl get nodes -o wide

2. Verify all pods are Running

To verify if all pods are in the running state, run the following command:

kubectl get pods -A

3. Verify Helm releases

To verify if the helm releases are in the deployed state, run the following command:

helm list -A

4. Directory Maintenance after Upgrade

During upgrade, both the previous and the new PPC installation directories must be retained.

• The original directory contains Terraform state and deployment metadata.
• The new directory contains the updated configuration and binaries.

Both directories are required for proper management of the deployment.

Reverting to a previous version of PPC

To revert to a previous version of PPC, restore the cluster using the steps in Restoring the PPC.