Migration

Procedures for migrating Policy Workbench nodes between CPU architectures on AWS and Azure.

This section provides the steps to migrate Policy Workbench nodes between arm64 and amd64 on Amazon Elastic Kubernetes Service (EKS) and Azure Kubernetes Service (AKS).

This optional procedure is needed only when changing the node architecture for an existing installation, including after an upgrade.

Note: These migration steps apply only to module-managed Workbench resources. They do not apply to self-managed Workbench resources.

Use the existing OpenTofu root module and state that manage the Workbench node pool. Before applying the migration, run tofu plan and review the proposed changes. Do not create a new root module or state for the migration.

Persist the target arch and any intended FIPS or ami_id settings in that root module’s configuration.

For root-module setup instructions, refer to Installing Policy Workbench without an existing root module or Installing Policy Workbench with an existing root module.

Persist the target arch and compatible instance_types in that root module’s configuration, either as module arguments or as declared and wired root-module inputs.

For root-module setup instructions, refer to Installing Policy Workbench without an existing root module or Installing Policy Workbench with an existing root module.

This updates the Karpenter-managed node-pool configuration but does not replace existing nodes, so cordon and drain the source nodes to move workloads to the new architecture.

Migrate the Nodes from amd64 to arm64

Run the following commands to migrate the Karpenter nodes from amd64 to arm64:

Run the following command to check node status in the workbench pool:
```
kubectl get nodes -l karpenter.sh/nodepool=policy-workbench
```
Confirm that each node is Ready before proceeding.
In the existing root-module configuration, set the target arch and compatible node settings for your platform, then run a plan and review its changes before applying:
Set arch to arm64 and set any required FIPS or AMI override values.
Set arch to arm64 and instance_types to Standard_D8ps_v5.
The instance_types value must be compatible with arch. OpenTofu updates the AKSNodeClass and NodePool so new nodes use kubernetes.io/arch=arm64 and Standard_D8ps_v5.
```
tofu plan
tofu apply
```

Run the following command to cordon the source amd64 nodes:

kubectl get nodes -l karpenter.sh/nodepool=policy-workbench,kubernetes.io/arch=amd64 -o name \
  | xargs -r -n1 kubectl cordon

The output indicates that the nodes are cordoned.

Run the following command to drain the source amd64 nodes:

kubectl get nodes -l karpenter.sh/nodepool=policy-workbench,kubernetes.io/arch=amd64 -o name \
  | xargs -r -n1 kubectl drain --ignore-daemonsets --delete-emptydir-data --timeout=5m

This triggers rescheduling onto nodes that satisfy the updated architecture requirements.

Run the following command to retrieve the status of the workbench pods:
```
kubectl get pods -n policy-workbench
```
Confirm that the required pods are Running and Ready before proceeding further.
Run the following command to retrieve the status of the nodes:
```
kubectl get nodes -l karpenter.sh/nodepool=policy-workbench,kubernetes.io/arch=amd64
```
Wait for the source nodes to terminate and disappear.

Migration is complete when no amd64 nodes remain in the workbench node pool.

Migrate the Nodes from arm64 to amd64

Run the following commands to migrate the Karpenter nodes from arm64 to amd64:

Run the following command to check node status in the workbench pool:
```
kubectl get nodes -l karpenter.sh/nodepool=policy-workbench
```
Confirm that each node is Ready before proceeding.
In the existing root-module configuration, set the target arch and compatible node settings for your platform, then run a plan and review its changes before applying:
Set arch to amd64 and set any required FIPS or AMI override values.
Set arch to amd64 and instance_types to Standard_D8as_v5.
The instance_types value must be compatible with arch. Karpenter provisions new Standard_D8as_v5 nodes for the evicted pods.
```
tofu plan
tofu apply
```

Run the following command to cordon the source arm64 nodes:

kubectl get nodes -l karpenter.sh/nodepool=policy-workbench,kubernetes.io/arch=arm64 -o name \
  | xargs -r -n1 kubectl cordon

The output indicates that the nodes are cordoned.

Run the following command to drain the source arm64 nodes:

kubectl get nodes -l karpenter.sh/nodepool=policy-workbench,kubernetes.io/arch=arm64 -o name \
  | xargs -r -n1 kubectl drain --ignore-daemonsets --delete-emptydir-data --timeout=5m

This triggers rescheduling onto nodes that satisfy the updated architecture requirements.

Run the following command to retrieve the status of the workbench pods:
```
kubectl get pods -n policy-workbench
```
Confirm that the required pods are Running and Ready before proceeding further.
Run the following command to retrieve the status of the nodes:
```
kubectl get nodes -l karpenter.sh/nodepool=policy-workbench,kubernetes.io/arch=arm64
```
Wait for the source nodes to terminate and disappear.

Migration is complete when no arm64 nodes remain in the workbench node pool.