Deploying PPC

• 1: Automated script-based installation
• 2: Manual component-based installation
• 3: In-cluster deployment

1 - Automated script-based installation

Before you begin

• Before running the bootstrap as the root user on RHEL, ensure that /usr/local/bin, and the AWS CLI binary path, if applicable, is included in the $PATH. Alternatively, run the script using a non-root user, such as ec2-user, where /usr/local/bin is already part of the default PATH.

The repository provides a bootstrap script that automatically installs or updates the following software on the jump box:

• AWS CLI - Required to communicate with the AWS account.
• OpenTofu - Required to manage infrastructure as code.
• kubectl - Required to communicate with the Kubernetes cluster.
• Helm - Required to manage Kubernetes packages.
• skopeo - Required to copy and inspect container images across registries without requiring a daemon.
• jq - Required to parse JSON.

The bootstrap script also checks if you have the required permissions on AWS. It then sets up the EKS cluster and installs the microservices required for deploying the PPC.

Using Standard Bottlerocket AMI Deployment

Verify whether the secure Bottlerocket AMI is available before deciding to use the standard Bottlerocket AMI. Run the following AWS CLI command against the target region, replacing <region>, <eks-version>, and <arch> (x86_64 or arm64) with your values:

aws ssm get-parameter \
--region <region> \
--name "/aws/service/bottlerocket/aws-k8s-<eks-version>-fips/<arch>/latest/image_id"

If the command returns ParameterNotFound, then the secure Bottlerocket AMI is not published in that region, and you must complete the additional requirements. For the additional requirements, refer to Configuring for a Standard Deployment.

Architecture Selection

The step [1b] Select Node Architecture in the bootstrap script prompts to select the node architecture interactively during deployment.

• amd64 (x86-64-based environments such as standard Intel or AMD servers)

• arm64 (ARM-based environments such as AWS Graviton)

Deploying the PPC

The bootstrap script prompts for variables to be set to complete your deployment. Follow the instructions on the screen:

./bootstrap.sh --cloud aws

The script prompts for the following variables.

1. Enter Cluster Name

   The following characters are allowed:

   • Lowercase letters: a-z
   • Numbers: 0-9
   • Hyphens: -

   The following characters are not allowed:

   • Uppercase letters: A-Z
   • Underscores: _
   • Spaces
   • Any special characters such as: / ? * + % ! @ # $ ^ & ( ) = [ ] { } : ; , .
   • Leading or trailing hyphens
   • Names longer than 31 characters (the cluster name must be 31 characters or fewer)

   Note: Ensure that the cluster name does not exceed 31 characters. Cluster names longer than this limit can cause the bootstrap script to fail in subsequent installation steps. If the installation fails, specify a cluster name that is 31 characters or fewer and re-run the script. The script automatically applies the updated value and continues the bootstrap process.

   After the cluster name is accepted, the script prompts you to select the node architecture.

   [1b] Select Node Architecture

   [1b] Select Node Architecture
     1) amd64 (x86_64) — e.g. t3, m5, c5 instances
     2) arm64 (Graviton) — e.g. t4g, m7g, c7g instances
   

   Enter 1 to select amd64 or 2 to select arm64.

   Note: Select amd64 for standard Intel/AMD 64-bit environments. Select arm64 for ARM-based environments such as AWS Graviton instances.

2. Enter a VPC ID from the table

   The script automatically retrieves the available VPCs. Enter the VPC ID where the cluster must be created.

3. Querying for subnets in VPC…

   The script queries for the available VPC subnets and prompts to enter two private subnet IDs. Specify two private subnet IDs from different availability zones.
   The script automatically updates the VPC CIDR block based on the VPC details.

4. Enter FQDN

   This is the Fully Qualified Domain Name (FQDN) for the ingress.

   Warning: Ensure that the FQDN does not exceed 50 characters and only the following characters are used:

   • Lowercase letters: a-z
   • Numbers: 0-9
   • Special characters: - .

5. Select S3 Backup Bucket Option

   The script prompts to choose how to provide the AWS S3 bucket encrypted with SSE‑KMS used for storing cluster backups for disaster recovery.

   Do you want to use an existing S3 bucket, or provision a new one (with KMS key)?
     1) Use existing
     2) Provision new via Tofu (00_backup_bucket)
   

6. Enter S3 bucket name

   • Option 1 – Use existing: Enter the name of an existing S3 bucket in your AWS account. The bucket must already be encrypted with SSE‑KMS.
   • Option 2 – Provision new via Tofu: The script uses OpenTofu (00_backup_bucket) to create the S3 bucket and the corresponding KMS key on your behalf. Provide a globally unique bucket name when prompted:
     • Lowercase letters, digits, and hyphens only
     • Between 3 and 63 characters
     • Must not already exist

   Note: Use a dedicated S3 bucket per cluster for backup and restore operations to ensure data and encryption isolation. Sharing a bucket across clusters increases the risk of cross-cluster data access or decryption due to IAM misconfiguration. Dedicated buckets with unique IAM policies eliminate this risk.

   Note: During disaster management, OpenSearch restores only those snapshots that are created using the daily-insight-snapshots policy. For more information, refer to Backing up and restoring indexes.

7. Enter Image Registry Endpoint

   Enter <Container_Registry_Path>, the container registry endpoint from where the container images are retrieved.

   Note: If deploying from the Protegrity Container Registry, use registry.protegrity.com:9443. Specify the local registry endpoint if a local registry is used.

   Expected format: <hostname>[:port].
   Do not include ‘https://’

   Note: The container registry endpoint must be a Fully Qualified Domain Name (FQDN). Sub-paths like, <Container_Registry_Path>/v2/path, are not supported by the Open Container Initiative (OCI) distribution specification.

8. Enter Registry Username

   Enter the username for the registry mentioned in the previous step. Leave this entry blank if the registry does not require authentication.

9. Enter Registry Password or Access Token

   Enter Password or Access Token for the registry. Input is masked with * characters. Press Enter to keep the current value. Leave this entry blank if the registry does not require authentication.

10. Enter Owner Email (for cluster tagging)

    Enter email address in the format name@domain.tld. This email is used for the cluster ‘Owner’ tag and the platform owner contact.

Note: The cluster creation process can take 10-15 minutes.

If the session is terminated during installation due to network issues, power outage, and so on, then the installation stops. To restart the installation, run the script again:

# Navigate to setup directory 
./bootstrap.sh --cloud aws

Important Information

• Do not install or manage multiple clusters from the same working directory. Each cluster deployment maintains its own Terraform/OpenTofu state, and reusing a directory can overwrite state files, causing loss of cluster tracking and unintended cleanup behavior.

• Use a dedicated directory, and jump box, where possible, per cluster, and always verify the active kubectl context before running cleanup commands such as ./bootstrap.sh --cloud aws destroy.
  To check the active kubectl context, run the following command:
  kubectl config current-context

• Do not delete the original cluster deployment directory, as it contains the Terraform state files required for managing the deployment.

Configuring for a Standard Deployment

Note: The following instructions are optional and only required for configuring a standard PPC deployment on AWS. Skip these steps if you have already installed the secure Bottlerocket AMIs.

By default, the PCT is configured to deploy a cluster using secure Bottlerocket AMIs. To deploy a standard cluster, the Terraform configuration files must be modified to use the standard Bottlerocket AMI types and SSM parameter paths before running the bootstrap script or the component-based installation.

For a standard PPC deployment on AWS EKS, perform the following steps:

1. Ensure that you have downloaded and extracted the PCT. For more information, refer to Preparing for PPC deployment.

2. Update the node group AMI type map in the /03_node_group/variables.tf file.

The node_group_ami_type_map variable defines the fallback EKS AMI type used per node architecture when node_group_ami_type is not explicitly set. The secure Bottlerocket AMI defaults must be replaced with the corresponding standard Bottlerocket AMI types.

a. Navigate to /03_node_group directory.

cd iac/aws/cluster/modules/03_node_group/

b. Edit the variables.tf file to make the following changes.

Change from (secure defaults):

variable "node_group_ami_type_map" {
  description = "Fallback EKS AMI type per architecture, used when node_group_ami_type is empty."
  type        = map(string)
  default = {
    amd64 = "BOTTLEROCKET_x86_64_FIPS"
    arm64 = "BOTTLEROCKET_ARM_64_FIPS"
  }
}

Change to (standard values):

variable "node_group_ami_type_map" {
  description = "Fallback EKS AMI type per architecture, used when node_group_ami_type is empty."
  type        = map(string)
  default = {
    amd64 = "BOTTLEROCKET_x86_64"
    arm64 = "BOTTLEROCKET_ARM_64"
  }
}

3. Update the Karpenter AMI SSM parameter path in the /05_helm_releases/main.tf file.

The karpenter_fips_ami_ssm_parameter value specifies the AWS Systems Manager (SSM) parameter path that Karpenter uses to resolve the correct Bottlerocket AMI for worker nodes. The default path targets the secure Bottlerocket AMI. For a standard cluster, remove the -fips segment from the EKS version portion of the path.

a. Navigate to /05_helm_releases directory.

cd iac/aws/cluster/modules/05_helm_releases/

b. Edit the main.tf file to make the following changes.

Change from (secure SSM path):

karpenter_fips_ami_ssm_parameter = "/aws/service/bottlerocket/aws-k8s-${data.aws_eks_cluster.this.version}-fips/${local.karpenter_ami_arch}/latest/image_id"

Change to (standard SSM path):

karpenter_fips_ami_ssm_parameter = "/aws/service/bottlerocket/aws-k8s-${data.aws_eks_cluster.this.version}/${local.karpenter_ami_arch}/latest/image_id"

4. After completing all these configuration changes, proceed to deploy the cluster.

Note: The additional requirements update the Terraform configuration files for the bootstrap script. Do not modify the 03_node_group/variables.tf or 05_helm_releases/main.tf files in the preceding installation steps. Reverting those changes will result in a secure cluster being deployed instead of a standard cluster.

2 - Manual component-based installation

This section is intended for enterprise customers who need full control over each stage of the EKS cluster deployment. Use this path when infrastructure stages must be reviewed or approved separately. Use it also when integration with an existing CI/CD pipeline is required. Use it when corporate change-management controls prevent running an all-in-one automated script. All stages must be performed from the same jump box.

Review the Deploying PPC overview and AWS prerequisites before running tofu apply.

Note: If you are performing a standard deployment, complete Configuring for a Standard Deployment before proceeding. Do not add or override node_group_ami_type in the 03_node_group module’s terraform.tfvars file, as this will reintroduce secure Bottlerocket AMIs.

Deploying PPC using component-based installation

To deploy PPC using the component-based installation, perform the following steps

Before you begin

Ensure to have the following prerequisites before proceeding with the installation.

Values decided by user

Using Standard Bottlerocket AMI Deployment

By default, PPC deploys using the secure (FIPS) Bottlerocket AMI. Before choosing the standard Bottlerocket AMI, verify whether the secure AMI is available in your target region.

Run the following AWS CLI command, replacing <region>, <eks-version>, and <arch> with your values:

Architecture value: Use x86_64 for amd64 nodes, or arm64 for arm64 (Graviton) nodes. Do not use amd64, amd, or arm. The SSM parameter path must match exactly, or the lookup will fail.

EKS version value: Use the Kubernetes version with which the cluster will be created. This is the eks_cluster_version value in the 02_eks_cluster module in variables.tf file, and the default value is 1.35.

aws ssm get-parameter \
  --region <region> \
  --name "/aws/service/bottlerocket/aws-k8s-<eks-version>-fips/<arch>/latest/image_id"

For example, amd64 nodes, EKS 1.35, and us-east-1:

aws ssm get-parameter \
  --region us-east-1 \
  --name "/aws/service/bottlerocket/aws-k8s-1.35-fips/x86_64/latest/image_id"

Interpret the result as follows:

• Returns an image ID: The secure Bottlerocket AMI is published in that region. Proceed with the default (secure) deployment. No additional steps are required.

• Returns ParameterNotFound: Before concluding the AMI is unavailable, verify the following:

  • <arch> is exactly x86_64 or arm64.
  • <eks-version> matches the eks_cluster_version value in the 02_eks_cluster module (default 1.35). For example, aws-k8s-1.35-fips.

  Optionally, confirm which EKS versions AWS currently offers in the region. This works before any cluster exists.

  aws eks describe-cluster-versions --region <region> --query "clusterVersions[].clusterVersion" --output table
  

If the command returns ParameterNotFound, then the secure Bottlerocket AMI is not published in that region, and you must complete the additional requirements. For the additional requirements, refer to Configuring for a Standard Deployment.

Deploying PPC

To deploy PPC using component-based installation, perform the following steps:

1. Preparing a jump box

Create a jump box or use an existing one that has network access to the EKS cluster. The jump box must be able to access the Kubernetes API server endpoint.

2. Exporting AWS credentials

The installation process uses Terraform to provision and manage AWS resources. These resources include VPCs, EKS clusters, IAM roles, and S3 buckets. The AWS credentials must be exported as environment variables to authenticate these operations.

Note: While using temporary credentials, such as AWS STS credentials, ensure the session token is included. Also confirm that the credentials have not expired before proceeding.
CLI login using AWS SSO (IAM Identity Center) is not supported. Export IAM user credentials or temporary STS credentials as environment variables.

If the components are installed by different users, ensure that each user must export their AWS credentials. This ensures that all the required access and permissions are available.

To export the AWS credentials, run the following commands:

export AWS_ACCESS_KEY_ID=<your-access-key-id>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
export AWS_SESSION_TOKEN=<your-session-token>

3. Downloading and extracting the release PCT

To download and extract the PCT, refer to Preparing for PPC deployment.

4. Installing the tools

The following tools and minimum versions are required for the component-based installation. Ensure they are installed on your jump box before proceeding, or use the bootstrap.sh install-tools command to install them automatically.

After downloading and extracting the PPC-K8S-ALL_1.2.0.x.tar as described in Preparing for PPC deployment, run the following command to install the required tools:

Note: For more information about the PCT, refer to My.Protegrity or the release notes.

./bootstrap.sh install-tools --cloud aws

Alternatively, the following tools must be installed and available on the jump box before starting.

5. Provisioning the S3 bucket

This step is required if a new S3 bucket is to be provisioned for storing the backup data. If an existing S3 bucket is to be used, note the bucket name and KMS key ARN. These values are required for the installation process. Then skip to the next step.

a. To navigate to 00_backup_bucket directory, run the following command:

export PPC_DEPLOY_DIR=<path to PPC deployment directory>
cd "$PPC_DEPLOY_DIR/iac/aws/cluster/modules/00_backup_bucket"

b. Edit the terraform.tfvars file to provide the following information:

cluster_name                = "<cluster-name>" 
s3_bucket_name              = "<globally-unique-bucket-name>" 
kms_key_alias_name          = "alias/pty-backup-<cluster-name>-key" 
aws_region                  = "<region>" 

c. To apply the terraform configuration, run the following command:

tofu init 
tofu plan          # review the proposed changes before applying
tofu apply         # when prompted with "Enter a value:", type "yes"

After the S3 bucket is provisioned successfully, note the bucket name and KMS key ARN. These values are required during the installation process.

d. To sync the Terraform state file to the S3 bucket, run the following command:

# Navigate to the bootstrap-scripts directory
cd <package-root>/bootstrap-scripts

# Run the script to sync the backup data to the S3 bucket
./sync-backup-bucket-state-to-s3.sh

e. Verify the S3 bucket configuration and confirm the Terraform configuration and state files were uploaded. These checks confirm that the bucket is ready for backups. They also confirm that the KMS key is enabled with rotation. They further confirm that the artifacts needed to manage the bucket are stored in S3. Run the checks from the package root.

# --- Stage 00 parameters ---
export CLUSTER_NAME="<cluster-name>"      # e.g. prod-eks-cluster
export REGION="<region>"                  # e.g. us-east-1
export BUCKET_NAME="<s3-bucket-name>"     # e.g. modular-bucket
export KMS_ALIAS="<kms-key-alias>"        # e.g. alias/pty-backup-<cluster-name>-key

# a. Bucket exists
aws s3api head-bucket --bucket "$BUCKET_NAME" --region "$REGION" && echo "Bucket exists"
 
# b. Versioning enabled
aws s3api get-bucket-versioning --bucket "$BUCKET_NAME" --region "$REGION" --output table
 
# c. Default encryption is KMS
aws s3api get-bucket-encryption --bucket "$BUCKET_NAME" --region "$REGION" \
  --query "ServerSideEncryptionConfiguration.Rules[].ApplyServerSideEncryptionByDefault" --output table
 
# d. KMS key enabled + rotation
aws kms describe-key --key-id "$KMS_ALIAS" --region "$REGION" \
  --query "KeyMetadata.{KeyId:KeyId, Enabled:Enabled, State:KeyState}" --output table
KMS_KEY_ID=$(aws kms describe-key --key-id "$KMS_ALIAS" --region "$REGION" --query "KeyMetadata.KeyId" --output text)
aws kms get-key-rotation-status --key-id "$KMS_KEY_ID" --region "$REGION" --output table
 
# e. tfvars + state uploaded
aws s3 ls "s3://$BUCKET_NAME/infrastructure-manifest/$CLUSTER_NAME/tfvars/00_backup_bucket/" --region "$REGION"
aws s3 ls "s3://$BUCKET_NAME/infrastructure-manifest/$CLUSTER_NAME/states/" --region "$REGION"
 
# f. Record outputs
tofu output

The verification is complete when the bucket exists, versioning is Enabled, and encryption is aws:kms. The KMS key must be enabled with rotation. The tfvars and state files must be present in S3. Record s3_bucket_name and kms_key_arn.

6. Modifying and applying terraform.tfvars files

The iac/aws/cluster/modules/ directory contains the Terraform modules for the PPC installation. To enter details in the terraform.tfvars file, navigate to the appropriate module directory and edit the file accordingly.

After modifying the terraform.tfvars file, apply the changes.

01_iam_roles module

a. To navigate to 01_iam_roles directory, run the following command:

export PPC_DEPLOY_DIR=<path to PPC deployment directory>
cd "$PPC_DEPLOY_DIR/iac/aws/cluster/modules/01_iam_roles"

b. Edit the terraform.tfvars file to provide the following information:

aws_region            = "<region>"          
eks_cluster_name      = "<cluster-name>"
backup_bucket         = "<s3-bucket-name>"
backup_bucket_kms_arn = "<kms-key-arn>"    

c. To apply the terraform configuration, run the following command:

tofu init 
tofu plan          # review the proposed changes before applying
tofu apply         # when prompted with "Enter a value:", type "yes"

d. Verify the IAM roles, customer-managed policies, role policy attachments, and node instance profile created by this module. The commands also confirm the module’s terraform.tfvars backup is in S3. They also display the role ARNs needed by later modules. Run them from the 01_iam_roles module directory:

# --- Stage 01 parameters ---
export CLUSTER_NAME="<cluster-name>"
export REGION="<region>"
export BUCKET_NAME="<s3-bucket-name>"

# a. Four IAM roles
aws iam list-roles --query "Roles[?contains(RoleName, '${CLUSTER_NAME}')].[RoleName, Arn]" --output table
 
# b. Customer-managed policies
aws iam list-policies --scope Local \
  --query "Policies[?contains(PolicyName, '${CLUSTER_NAME}')].[PolicyName, Arn]" --output table
 
# c. Node role managed policies
aws iam list-attached-role-policies --role-name "eks-${CLUSTER_NAME}-node-role" \
  --query "AttachedPolicies[].PolicyName" --output table
 
# d. Cluster role managed policies
aws iam list-attached-role-policies --role-name "eks-${CLUSTER_NAME}-cluster-role" \
  --query "AttachedPolicies[].PolicyName" --output table
 
# e. Node instance profile
aws iam get-instance-profile --instance-profile-name "eks-${CLUSTER_NAME}-node-profile" \
  --query "InstanceProfile.{Name:InstanceProfileName, Role:Roles[0].RoleName}" --output table
 
# f. tfvars backed up
aws s3 ls "s3://${BUCKET_NAME}/infrastructure-manifest/${CLUSTER_NAME}/tfvars/01_iam_roles/" --region "$REGION"
 
# g. Record role ARNs
tofu output

The verification is complete when all four roles (cluster-role, node-role, karpenter-role, and backup-recovery-utility-role) and four local policies exist. The node and cluster managed policies must be attached. The node instance profile must be linked. The terraform.tfvars file must be present in S3.

02_eks_cluster module

a. To navigate to 02_eks_cluster directory, run the following command:

export PPC_DEPLOY_DIR=<path to PPC deployment directory>
cd "$PPC_DEPLOY_DIR/iac/aws/cluster/modules/02_eks_cluster"

b. Edit the terraform.tfvars file to provide the following information.

Note: The security group is created in AWS using the name specified by the security_group_name value in this file.

Important: The security group name must follow the exact format <cluster-name>_SG. The SG must be in uppercase. This value is used as the AWS Name tag on the worker security group. Protegrity components discover the security group by matching this tag value exactly. Tag values are case-sensitive. Any other format, for example, lowercase _sg or a custom name, will cause those components to fail. The security group name must also not start with SG.

aws_region          = "<region>"
eks_cluster_name    = "<cluster-name>"
backup_bucket       = "<s3-bucket-name>" 

# Important: The security group name must follow the exact format <cluster-name>_SG.
# The SG must be in uppercase. This value is used as the AWS Name tag on the worker security group.
# Protegrity components discover the security group by matching this tag value exactly.
# Tag values are case-sensitive. Any other format, for example, lowercase _sg or a custom name, will cause those components to fail.
# The security group name must also not start with SG.

# --- Networking ---
vpc_id              = "<vpc-id>" 
security_group_name = "<cluster-name>_SG"      # Must match this format exactly
private_subnet_ids  = [
  "<subnet-a>",
  "<subnet-b>"
]

# --- Tagging ---
owner_email = "<owner_email>"

# --- Access Entries (optional) ---
# Add IAM users/roles that need kubectl / tofu access beyond the original installer.
additional_access_entries = [
  # {
  #   principal_arn = "<principal-arn>"
  # },
]

c. To apply the terraform configuration, run the following command:

tofu init 
tofu plan          # review the proposed changes before applying
tofu apply         # when prompted with "Enter a value:", type "yes"

d. After applying the configuration, verify the setup. Check the EKS cluster, security group, add-ons, subnet tags, and kubeconfig context by running the following commands:

# --- Stage 02 parameters ---
export CLUSTER_NAME="<cluster-name>"
export REGION="<region>"
export BUCKET_NAME="<s3-bucket-name>"
export SG_NAME="${CLUSTER_NAME}_SG"       # worker SG name (uppercase SG)

# a. Cluster ACTIVE
aws eks describe-cluster --name "$CLUSTER_NAME" --region "$REGION" \
  --query "cluster.{Name:name, Status:status, Version:version, Endpoint:endpoint}" --output table
 
# b. No nodes yet (expected)
kubectl get nodes   # expected: No resources found (node group is Stage 03)
 
# c. Worker security group exists (exact name)
aws ec2 describe-security-groups --filters "Name=tag:Name,Values=${SG_NAME}" --region "$REGION" \
  --query "SecurityGroups[].{Name:Tags[?Key=='Name']|[0].Value, ID:GroupId, VPC:VpcId}" --output table
 
# d. VPC CNI + kube-proxy ACTIVE
aws eks list-addons --cluster-name "$CLUSTER_NAME" --region "$REGION" --output table
for addon in vpc-cni kube-proxy; do
  aws eks describe-addon --cluster-name "$CLUSTER_NAME" --addon-name "$addon" --region "$REGION" \
    --query "addon.{Name:addonName, Status:status, Version:addonVersion}" --output table
done
 
# e. Subnet tags (cluster + Karpenter discovery)
aws ec2 describe-subnets --region "$REGION" \
  --filters "Name=tag:kubernetes.io/cluster/${CLUSTER_NAME},Values=shared" \
  --query "Subnets[].{Subnet:SubnetId, AZ:AvailabilityZone}" --output table
aws ec2 describe-subnets --region "$REGION" \
  --filters "Name=tag:karpenter.sh/discovery/${CLUSTER_NAME},Values=true" \
  --query "Subnets[].SubnetId" --output table
 
# f. kubeconfig context added
kubectl config get-contexts | grep "$CLUSTER_NAME"
 
# g. tfvars backed up
aws s3 ls "s3://${BUCKET_NAME}/infrastructure-manifest/${CLUSTER_NAME}/tfvars/02_eks_cluster/" --region "$REGION"
 
# h. Record outputs
tofu output

The verification is complete when the cluster is ACTIVE and kubectl get nodes returns No resources found. The <cluster>_SG security group must exist, and both add-ons must be ACTIVE. The required subnet tags must be present. The kubeconfig context must be added, and the terraform.tfvars file must be present in S3. Record private_subnet_ids, vpc_id, vpc_cidr_block, and worker_security_group_id.

03_node_group module

a. To navigate to 03_node_group directory, run the following command:

export PPC_DEPLOY_DIR=<path to PPC deployment directory>
cd "$PPC_DEPLOY_DIR/iac/aws/cluster/modules/03_node_group"

b. Edit the terraform.tfvars file to provide the following information.

Note: Use the private subnet IDs from the VPC where the EKS cluster is provisioned while installing the 02_eks_cluster module.

aws_region       = "<region>" 
eks_cluster_name = "<cluster-name>" 
backup_bucket    = "<s3-bucket-name>" 

# --- System Node Group ---
node_group_name    = "<node-group-name>" 
private_subnet_ids = [
  "<subnet-a>",
  "<subnet-b>"
]
architecture            = "<amd64/arm64>"
node_group_desired_size = 2      
node_group_max_size     = 2
node_group_min_size     = 1

Note for standard deployments: If you completed the steps in Configuring for a Standard Deployment, the node_group_ami_type_map default values in variables.tf file have already been changed. The defaults are now BOTTLEROCKET_x86_64 (amd64) or BOTTLEROCKET_ARM_64 (arm64). Do not add or override node_group_ami_type in the terraform.tfvars file. Doing so reintroduces secure Bottlerocket AMIs.

c. To apply the terraform configuration, run the following command:

tofu init 
tofu plan          # review the proposed changes before applying
tofu apply         # when prompted with "Enter a value:", type "yes"

d. After applying the configuration, verify that the node group and worker nodes are healthy, the expected labels and taint are present, the launch template and EC2 instances exist, and the terraform.tfvars backup is in S3. Run these commands from the 03_node_group module directory:

# --- Stage 03 parameters ---
export CLUSTER_NAME="<cluster-name>"
export REGION="<region>"
export BUCKET_NAME="<s3-bucket-name>"
export NODE_GROUP_NAME="<node-group-name>"  # e.g. modular-nodegroup_NG

# a. Node group ACTIVE
aws eks describe-nodegroup --cluster-name "$CLUSTER_NAME" --nodegroup-name "$NODE_GROUP_NAME" --region "$REGION" \
  --query "nodegroup.{Name:nodegroupName, Status:status, AMIType:amiType, Instances:instanceTypes, Scaling:scalingConfig}" --output table
 
# b. Nodes Ready
kubectl get nodes -o wide
 
# c. Labels + taint
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.role}{"\t"}{.metadata.labels.nodepool-owner}{"\n"}{end}'
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'
 
# d. Launch template exists (note: --launch-template-ids, NOT a filter)
aws ec2 describe-launch-templates --region "$REGION" \
  --launch-template-ids "$(tofu output -raw launch_template_id)" \
  --query "LaunchTemplates[].{Name:LaunchTemplateName, ID:LaunchTemplateId, Default:DefaultVersionNumber}" --output table
#   Alternative by name:
#   aws ec2 describe-launch-templates --region "$REGION" \
#     --filters "Name=launch-template-name,Values=${NODE_GROUP_NAME}-*" \
#     --query "LaunchTemplates[].{Name:LaunchTemplateName, ID:LaunchTemplateId, Default:DefaultVersionNumber}" --output table
 
# e. Worker EC2 instances running
aws ec2 describe-instances --region "$REGION" \
  --filters "Name=tag:eks:nodegroup-name,Values=${NODE_GROUP_NAME}" "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[].{ID:InstanceId, Type:InstanceType, AZ:Placement.AvailabilityZone, State:State.Name}" --output table
 
# f. tfvars backed up
aws s3 ls "s3://${BUCKET_NAME}/infrastructure-manifest/${CLUSTER_NAME}/tfvars/03_node_group/" --region "$REGION"
 
# g. Record outputs + node OS details for Stage 05
tofu output
kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.osImage}'; echo

The verification is complete when the node group is ACTIVE and the desired number of nodes are Ready. Each node must have the role=system and nodepool-owner=ppc labels and the CriticalAddonsOnly taint. The launch template and running instances must exist. The terraform.tfvars file must be present in S3. Record node_group_name, the node architecture, and the node OS enum for Stage 05. The osImage output is informational. Do not copy it directly into node_os. For example, Bottlerocket OS 1.66.0 (aws-k8s-1.35-fips) on amd64 maps to architecture = "amd64". The same node maps to node_os = "BOTTLEROCKET_x86_64_FIPS".

04_eks_addons module

a. To navigate to 04_eks_addons directory, run the following command:

export PPC_DEPLOY_DIR=<path to PPC deployment directory>
cd "$PPC_DEPLOY_DIR/iac/aws/cluster/modules/04_eks_addons"

b. Edit the terraform.tfvars file to provide the following information:

aws_region       = "<region>"
eks_cluster_name = "<cluster-name>"
backup_bucket    = "<s3-bucket-name>" # 

# --- Storage classes ---
storage_class_name = "pty-sc" # default (general-purpose) StorageClass; must match Stage 05

# --- Upgrade mode only ---
# Set to "~/.kube/config" when upgrading an existing cluster;
# leave empty ("") for a fresh install.
kubeconfig_path  = ""

c. To apply the terraform configuration, run the following command:

tofu init 
tofu plan          # review the proposed changes before applying
tofu apply         # when prompted with "Enter a value:", type "yes"

Note: This stage waits for all nodes from Stage 03 to reach Ready status before installing add-ons.

d. After applying the configuration, verify the add-ons and related resources. Confirm that all EKS add-ons are active and that the default StorageClass settings are correct. Confirm that CoreDNS, Metrics Server, and the EBS CSI controller are running. Confirm that the Pod Identity associations exist and that the terraform.tfvars backup is in S3. Run these commands from the 04_eks_addons module directory:

# --- Stage 04 parameters ---
export CLUSTER_NAME="<cluster-name>"
export REGION="<region>"
export BUCKET_NAME="<s3-bucket-name>"
export STORAGE_CLASS="<storage-class-name>"  # default pty-sc

# a. All add-ons ACTIVE
aws eks list-addons --cluster-name "$CLUSTER_NAME" --region "$REGION" --output table
for addon in coredns metrics-server aws-ebs-csi-driver snapshot-controller eks-pod-identity-agent; do
  aws eks describe-addon --cluster-name "$CLUSTER_NAME" --addon-name "$addon" --region "$REGION" \
    --query "addon.{Name:addonName, Status:status, Version:addonVersion}" --output table
done
 
# b. Default StorageClass exists
kubectl get storageclass
 
# c. StorageClass details
kubectl get storageclass "$STORAGE_CLASS" \
  -o jsonpath='{.provisioner}{"\t"}{.reclaimPolicy}{"\t"}{.volumeBindingMode}{"\n"}'
 
# d. CoreDNS + Metrics Server running
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl get pods -n kube-system -l app.kubernetes.io/name=metrics-server
 
# e. EBS CSI controller running
kubectl get deployment ebs-csi-controller -n kube-system \
  -o jsonpath='{.status.readyReplicas}/{.status.replicas} ready{"\n"}'
kubectl get pods -n kube-system -l app=ebs-csi-controller
 
# f. Pod Identity associations
aws eks list-pod-identity-associations --cluster-name "$CLUSTER_NAME" --region "$REGION" \
  --query "associations[].{Namespace:namespace, ServiceAccount:serviceAccount}" --output table
 
# g. tfvars backed up
aws s3 ls "s3://${BUCKET_NAME}/infrastructure-manifest/${CLUSTER_NAME}/tfvars/04_eks_addons/" --region "$REGION"
 
# h. Record outputs
tofu output

The verification is complete when all five add-ons are ACTIVE. ${STORAGE_CLASS} must be the default StorageClass with the ebs.csi.aws.com provisioner, a Retain reclaim policy, and WaitForFirstConsumer volume binding mode. CoreDNS, Metrics Server, and the EBS CSI controller must be running. Both Pod Identity associations (karpenter and backup-recovery-utility) must exist. The terraform.tfvars file must be present in S3.

05_helm_releases module

a. To navigate to 05_helm_releases directory, run the following command:

export PPC_DEPLOY_DIR=<path to PPC deployment directory>
cd "$PPC_DEPLOY_DIR/iac/aws/cluster/modules/05_helm_releases"

b. Edit the terraform.tfvars file to provide the following information.

The default node OS is BOTTLEROCKET_x86_64_FIPS. Set node_os to the platform enum, not the raw osImage string returned by kubectl. For FIPS Bottlerocket nodes, use BOTTLEROCKET_x86_64_FIPS for amd64 or BOTTLEROCKET_ARM_64_FIPS for arm64.

The following fields must be updated with environment-specific values:

The following fields have default values and must be left unchanged for a fresh installation:

aws_region            = "<region>"
eks_cluster_name      = "<cluster-name>"
backup_bucket         = "<s3-bucket-name>" 
backup_bucket_kms_arn = "<kms-key-arn>"
storage_class_name        = "<storage-class-name>"  
backup_storage_class_name = "<backup-storage-class-name>" 

# --- Readiness ---
architecture          = "amd64"
node_os               = "BOTTLEROCKET_x86_64_FIPS"

# --- Karpenter instance families (must be org SCP-approved, p-6aysnndi) ---
karpenter_instance_families = ["t3", "t4g"] # amd64: m5, m6i, c5, r5, t3, t3a; arm64: see arm64_instance_families below
arm64_instance_families     = ["t4g"]       # subset of karpenter_instance_families classified as arm64 (Graviton)

# --- Registry ---
global_image_reg = "<Container_Registry_Path>"
harbor_secret    = "harbor-secret" 

# --- Ingress ---
ingress_fqdn = "<fqdn>" 

# --- Owner ---
owner_email = "<owner-email>" 

# --- CLI users (enrolled via bootstrap.sh access add-cli-user) ---
cli_user_public_keys = {} # e.g. { "jane.doe" = "ssh-rsa AAAA... jane.doe@host" } - on FIPS clusters only RSA (>=2048 bits) and ECDSA keys are accepted, not Ed25519/DSA

# cli_user_public_keys is an optional map of additional CLI users
# who should be granted SSH access to the cluster. Add entries
# as username-to-public-key pairs, for example: {"jane.doe" = "ssh-rsa AAAA... jane.doe@host"}

# This is used to enroll additional users through bootstrap.sh access add-cli-user.
# On FIPS-enabled clusters, only RSA keys with a minimum length of 2048 bits
# and ECDSA keys are accepted. Ed25519 and DSA keys are not supported.

# --- Networking (used to derive internal proxy CIDRs) ---
vpc_cidr_block = "<cidr>" 

# --- Envoy Gateway (LoadBalancer subnet discovery) ---
private_subnet_ids = []

# --- Restore ---
restore              = false                               
backup_name          = "authnz-postgresql-schedule-backup" 
insight_backup_name  = ""                                  

# --- Upgrade mode only ---
# Set to "~/.kube/config" when upgrading an existing cluster;
# leave empty ("") for a fresh install.
kubeconfig_path = ""
is_upgrade      = false
skip_validations = false

c. Save the file and exit the editor.

d. This step must be performed only if registry needs an authentication. Do not put them in the tfvars file.

To enter registry credentials using the environment variables, run the following commands:

export TF_VAR_username="<registry-username>"
export TF_VAR_password="<registry-password>"

e. To apply the terraform configuration, run the following command:

tofu init 
tofu plan          # review the proposed changes before applying
tofu apply         # when prompted with "Enter a value:", type "yes"

Note: This process may take 15–20 minutes to get completed. Karpenter nodes are provisioned on demand after the NodePool is created.

f. After applying the configuration, verify that all Helm releases are deployed. Confirm that the backup StorageClass exists and that Karpenter and Velero are running. Confirm that Envoy Gateway has provisioned a LoadBalancer and that Insight is healthy. Confirm that its bootstrap job has succeeded and that the terraform.tfvars backup is in S3. Run these commands from the 05_helm_releases module directory:

# --- Stage 05 parameters ---
export CLUSTER_NAME="<cluster-name>"
export REGION="<region>"
export BUCKET_NAME="<s3-bucket-name>"
export INSIGHT_NS=pty-insight

# a. All Helm releases deployed
helm list -A
 
# b. Backup StorageClass auto-created by this stage
kubectl get storageclass   # expect pty-bkp in addition to pty-sc
 
# c. Karpenter running + NodePool/EC2NodeClass present
kubectl get pods -n karpenter
kubectl get nodepools
kubectl get ec2nodeclasses
 
# d. Velero running
kubectl get pods -n pty-backup-recovery
 
# e. Envoy Gateway running + LoadBalancer provisioned
kubectl get pods -n api-gateway
kubectl get svc -n api-gateway \
  -o jsonpath='{range .items[?(@.spec.type=="LoadBalancer")]}{.metadata.name}{"\t"}{.status.loadBalancer.ingress[0].hostname}{"\n"}{end}'
 
# f. Insight pods healthy (incl. OpenSearch + Dashboards)
kubectl get pods -n "$INSIGHT_NS"
kubectl get pods -n "$INSIGHT_NS" -l app=opensearch-dashboards
 
# g. Insight bootstrap job succeeded (post-install hook)
kubectl get job -n "$INSIGHT_NS" initialize-opensearch-repository \
  -o jsonpath='{.status.succeeded} succeeded / {.status.failed} failed{"\n"}'
 
# h. Cluster-wide: no unhealthy pods
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
 
# i. tfvars backed up
aws s3 ls "s3://${BUCKET_NAME}/infrastructure-manifest/${CLUSTER_NAME}/tfvars/05_helm_releases/" --region "$REGION"

If the bootstrap job fails with a 403 anonymous/proxy-auth error (opendistro_security_anonymous), it indicates an OpenSearch Dashboards readiness or identity issue. It is not a node_os issue. Confirm the Dashboards pod is fully Ready, then rerun tofu apply from the 05_helm_releases module. This applies the corrected node_os and triggers a fresh post-install bootstrap attempt.

The verification is complete when all Helm releases are deployed and the pty-bkp backup StorageClass is present. Karpenter is running, and its NodePool and EC2NodeClass exist. Velero is running. Envoy Gateway is running with an external LoadBalancer address. The Insight pods, including OpenSearch Dashboards, are healthy. The initialize-opensearch-repository job has succeeded. The terraform.tfvars file is present in S3. No pods remain in a phase other than Running or Succeeded.

7. Verifying the installation

To verify the installation, run the following command:

# --- parameters ---
export CLUSTER_NAME="<cluster-name>"
export REGION="<region>"

kubectl get nodes          # all Ready
kubectl get pods -A        # all Running / Completed

All the nodes should be in the Ready state, and all the pods should be in the Running state.

Configuring for a Standard Deployment

Note: The following instructions are optional and only required for configuring a standard PPC deployment on AWS. Skip these steps if you have already installed the secure Bottlerocket AMIs.

By default, the PCT is configured to deploy a cluster using secure Bottlerocket AMIs. To deploy a standard cluster, the Terraform configuration files must be modified to use the standard Bottlerocket AMI types and SSM parameter paths before running the bootstrap script or the component-based installation.

For a standard PPC deployment on AWS EKS, perform the following steps:

1. Ensure that you have downloaded and extracted the PCT. For more information, refer to Preparing for PPC deployment.

2. Update the node group AMI type map in the /03_node_group/variables.tf file.

The node_group_ami_type_map variable defines the fallback EKS AMI type used per node architecture when node_group_ami_type is not explicitly set. The secure Bottlerocket AMI defaults must be replaced with the corresponding standard Bottlerocket AMI types.

a. Navigate to /03_node_group directory.

cd iac/aws/cluster/modules/03_node_group/

b. Edit the variables.tf file to make the following changes.

Change from (secure defaults):

variable "node_group_ami_type_map" {
  description = "Fallback EKS AMI type per architecture, used when node_group_ami_type is empty."
  type        = map(string)
  default = {
    amd64 = "BOTTLEROCKET_x86_64_FIPS"
    arm64 = "BOTTLEROCKET_ARM_64_FIPS"
  }
}

Change to (standard values):

variable "node_group_ami_type_map" {
  description = "Fallback EKS AMI type per architecture, used when node_group_ami_type is empty."
  type        = map(string)
  default = {
    amd64 = "BOTTLEROCKET_x86_64"
    arm64 = "BOTTLEROCKET_ARM_64"
  }
}

3. Update the Karpenter AMI SSM parameter path in the /05_helm_releases/main.tf file.

The karpenter_fips_ami_ssm_parameter value specifies the AWS Systems Manager (SSM) parameter path that Karpenter uses to resolve the correct Bottlerocket AMI for worker nodes. The default path targets the secure Bottlerocket AMI. For a standard cluster, remove the -fips segment from the EKS version portion of the path.

a. Navigate to /05_helm_releases directory.

cd iac/aws/cluster/modules/05_helm_releases/

b. Edit the main.tf file to make the following changes.

Change from (secure SSM path):

karpenter_fips_ami_ssm_parameter = "/aws/service/bottlerocket/aws-k8s-${data.aws_eks_cluster.this.version}-fips/${local.karpenter_ami_arch}/latest/image_id"

Change to (standard SSM path):

karpenter_fips_ami_ssm_parameter = "/aws/service/bottlerocket/aws-k8s-${data.aws_eks_cluster.this.version}/${local.karpenter_ami_arch}/latest/image_id"

4. After completing all these configuration changes, proceed to deploy the cluster.

Note: The additional requirements update the Terraform configuration files for the bootstrap script. Do not modify the 03_node_group/variables.tf or 05_helm_releases/main.tf files in the preceding installation steps. Reverting those changes will result in a secure cluster being deployed instead of a standard cluster.

3 - In-cluster deployment

This section is intended for users who already have an EKS cluster provisioned and managed outside of the PPC installer. In a standard PPC installation, the installer provisions the EKS cluster and the required AWS resources. In an in-cluster deployment, however, the user creates the cluster. It already exists before the PPC installation begins. Cluster information must be provided to deploy PPC onto the existing EKS cluster. Unlike the automated script-based installation or the manual component-based installation, the installer does not create these AWS resources. They must already exist before you begin.

Complete the infrastructure prerequisites outside the PPC installer first. Then use the PPC installer to deploy the PPC application Helm charts to the existing EKS cluster.

Deploying PPC using In-Cluster Deployment for EKS

To install PPC, complete the following steps in order. Each step is described in detail in the sections below.

Before you begin

Ensure the following prerequisites are met before starting the installation.

Access and permissions

The following access and permissions are required for deploying PPC.

Values needed for terraform.tfvars

The values required to populate terraform.tfvars in are collected from two sources:

Values decided by user

These are existing or user-determined values that are available before prerequisite creation begins:

Values captured from created resources

These values are generated when creating the resources listed in the Cluster requirements for PPC deployment section. Record each value as you create the corresponding resource:

Cluster requirements for PPC deployment

Your EKS cluster is already in place. Before running tofu apply in 05_helm_releases, use this checklist. It confirms that the cluster meets all the requirements PPC deployment expects.

Note: If you are unsure whether a requirement is met, work through each item. Verify each item against your cluster configuration before running tofu apply.

IAM - roles and policies

EKS cluster and access

Compute: system node group

Networking

Add-on scheduling (required after tainted system node group exists)

Storage and backup

System node group taint value

Ensure that the CriticalAddonsOnly=true:NoSchedule taint is applied when creating the system node group. Apply it together with the role=system label. This taint prevents application workloads from being scheduled on the system nodes. It is required for the Add-on scheduling configuration.

Deploying PPC

To deploy PPC onto the existing EKS cluster, perform the following steps:

1. Preparing a jump box

Create a jump box or use an existing one that has network access to the EKS cluster. The jump box must be able to access the Kubernetes API server endpoint.

2. Downloading and extracting the release archive

To download and extract the PCT, refer to Preparing for PPC deployment.

3. Installing required tools

To install the required tools from the extracted release bundle, run the following command:

./bootstrap-scripts/setup-devtools-aws.sh

The script installs the following tools:

4. Configuring AWS credentials

Configure your AWS credentials so that CLI commands authenticate as your deployer identity. Either run aws configure to set up a named profile, or export the credentials directly.

Note: AWS SSO web-based login is not supported. However, AWS IAM Identity Center temporary credentials can be exported as environment variables.

Export IAM user credentials, temporary STS credentials, or AWS IAM Identity Center temporary credentials. Include the session token, as shown below:

export AWS_ACCESS_KEY_ID="<ACCESS_KEY_ID>"
export AWS_SECRET_ACCESS_KEY="<SECRET_ACCESS_KEY>"
export AWS_SESSION_TOKEN="<SESSION_TOKEN>"
export AWS_DEFAULT_REGION="<REGION>"

Note: The AWS_SESSION_TOKEN variable is required only when using temporary credentials, such as the STS or AWS IAM Identity Center. Omit it when using long-term IAM user credentials.

To verify that the credentials are working, run the following command:

aws sts get-caller-identity

5. Configuring EKS access

To retrieve the EKS credentials and configure kubectl to communicate with the cluster, run the following command:

aws eks update-kubeconfig --name <CLUSTER_NAME> --region <REGION>

6. Verifying cluster connectivity

To confirm that the jump box can reach the cluster API, run the following commands:

kubectl get pods -A
kubectl get nodes

All nodes should be listed and in Ready state before proceeding.

7. Gathering values for terraform.tfvars

Complete each of the following sub-steps to populate terraform.tfvars with the required cluster-specific values.

7.1 Storage class

Record the name of the storage class to use and update storage_class_name in terraform.tfvars.

kubectl get sc -A

Record the name of the storage class. Update only storage_class_name in terraform.tfvars with this value. Leave backup_storage_class_name set to its default value, pty-bkp, which is created by Stage 05.

Note: Create a custom storage class named pty-sc. Use the following properties: gp3 volume type, Retain reclaim policy, and WaitForFirstConsumer binding mode. Then set storage_class_name=pty-sc in terraform.tfvars. For a quick bring-up, the console’s default gp2 storage class may be used instead. Do not modify backup_storage_class_name, leave it as pty-bkp.

7.2 Registry credentials

When using an authenticated private registry, set harbor_secret to harbor-secret in terraform.tfvars. Also export the container registry credentials before running tofu apply. Otherwise, the apply fails while pulling PPC images and charts. Leave harbor_secret empty for an unauthenticated registry.

export TF_VAR_username="<REGISTRY-USERNAME>"
export TF_VAR_password="<REGISTRY-PASSWORD>"

Note: Registry credentials TF_VAR_username and TF_VAR_password are passed as environment variables and are not set directly in terraform.tfvars.

7.3 Add-on scheduling configuration

After the tainted system node group is active, apply the scheduling configuration to the four Deployment-based add-ons. This lets their pods land on system nodes. Set CLUSTER_NAME and REGION, then run the following commands:

export CLUSTER_NAME=<CLUSTER-NAME>
export REGION=<REGION>

# CoreDNS
aws eks update-addon \
  --cluster-name "$CLUSTER_NAME" \
  --addon-name coredns \
  --region "$REGION" \
  --resolve-conflicts OVERWRITE \
  --configuration-values '{"tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists","effect":"NoSchedule"}],"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"role","operator":"In","values":["system"]}]}]}}}}'

# Amazon EBS CSI Driver (controller only)
aws eks update-addon \
  --cluster-name "$CLUSTER_NAME" \
  --addon-name aws-ebs-csi-driver \
  --region "$REGION" \
  --resolve-conflicts OVERWRITE \
  --configuration-values '{"controller":{"tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists","effect":"NoSchedule"}],"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"role","operator":"In","values":["system"]}]}]}}}}}'

# metrics-server
aws eks update-addon \
  --cluster-name "$CLUSTER_NAME" \
  --addon-name metrics-server \
  --region "$REGION" \
  --resolve-conflicts OVERWRITE \
  --configuration-values '{"tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists","effect":"NoSchedule"}],"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"role","operator":"In","values":["system"]}]}]}}}}'

# snapshot-controller
aws eks update-addon \
  --cluster-name "$CLUSTER_NAME" \
  --addon-name snapshot-controller \
  --region "$REGION" \
  --resolve-conflicts OVERWRITE \
  --configuration-values '{"tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists","effect":"NoSchedule"}],"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"role","operator":"In","values":["system"]}]}]}}}}'

To verify the add-on status after the updates complete, run the following commands:

for addon in coredns aws-ebs-csi-driver metrics-server snapshot-controller; do
  aws eks describe-addon \
    --cluster-name "$CLUSTER_NAME" \
    --addon-name "$addon" \
    --region "$REGION" \
    --query 'addon.[addonName,status]' \
    --output text
done

Expected: all four add-ons report ACTIVE. If an update is still in progress, wait for it to complete. Then run the verification again.

8. Updating terraform.tfvars

To update the terraform.tfvars file, perform the following steps:

1. Navigate to the Helm releases module directory.

cd iac/aws/cluster/modules/05_helm_releases

2. After completing Steps 7.1 through 7.3, set the following variables in 05_helm_releases/terraform.tfvars.

Note: Registry credentials are passed as environment variables (TF_VAR_username and TF_VAR_password). They are configured in Step 7.2. They are not set directly in terraform.tfvars. Leave backup_storage_class_name at its default value, pty-bkp, unless the deployment requires a different value.

3. Add the following variables to terraform.tfvars. Set each to the value shown before allowing Stage 05 to apply.

run_readiness_check = true
check_prev_stage    = false

These variables ensure that PPC performs a readiness check on the existing EKS environment. They skip the previous automated stage validation. That validation is not required because those resources were created outside the installer.

4. Save the file and exit the editor.

9. Running the readiness check

The readiness check verifies that the cluster is ready for the Stage 05 (Helm Releases) tofu apply. The same check runs automatically as a guard at the start of the Stage 05 apply. It runs because run_readiness_check is enabled in the terraform.tfvars file. If required prerequisites are missing, the apply stops before deploying any components.

Run the check independently now to identify and resolve any issues before starting the apply.

To run the readiness check, run the following command:

./bootstrap-scripts/readiness-helm-aws.sh

The script validates that all required EKS add-ons are ACTIVE. It also checks that the worker security group and subnet tags are present. It checks that Pod Identity associations exist, and that the backup bucket and KMS key are accessible.

Note: The readiness check is read-only and does not modify the cluster. Run it as often as needed, including after correcting reported errors. Repeat it until it completes successfully before proceeding to apply the OpenTofu configuration.

10. Applying the OpenTofu configuration

To deploy all PPC Helm charts onto the cluster, initialize and apply the OpenTofu configuration. Perform the following steps:

1. Navigate to the Helm releases module directory.

cd iac/aws/cluster/modules/05_helm_releases

Important: Before running tofu plan or tofu apply, review all values in terraform.tfvars. Confirm that they match your target environment.

2. To initialize, plan, and apply the OpenTofu configuration, run the following commands. They deploy all PPC Helm charts onto the cluster.

tofu init
tofu plan          # Review the planned resources that will be created.
tofu apply         # When prompted with "Enter a value:", type "yes".

3. To verify the resources created in this stage, run the following commands:

# Confirm nodes are Ready
kubectl get nodes

# Confirm pods are Running
kubectl get pods -A

11. Common misconfigurations in In-Cluster Deployment for EKS

When Stages 00–04 are created manually using the console or CLI, this repo’s OpenTofu does not create them. Every IAM role, policy, tag, and rule must match the exact names and permissions Stage 05 expects. These are the issues that break the install: