Managing Cluster Upgrades
1. Planning Cluster Upgrades
| Step | Detail |
|---|
| Read release notes | Deprecations, removed APIs |
| Skew policy | kubelet ≤ 3 minors behind API server |
| API audit | kubectl convert, pluto, kube-no-trouble |
| Backup | etcd snapshot + Velero |
2. Upgrading Control Plane
sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.32.0-1.1 && sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.32.0
# other control-plane nodes
sudo kubeadm upgrade node
3. Upgrading Worker Nodes
kubectl drain NODE --ignore-daemonsets --delete-emptydir-data
ssh NODE
sudo apt-get install -y kubelet=1.32.0-1.1 kubectl=1.32.0-1.1
sudo systemctl daemon-reload && sudo systemctl restart kubelet
kubectl uncordon NODE
4. Upgrading kubectl
brew upgrade kubernetes-cli # macOS
curl -LO "https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl" # linux
5. Upgrading Managed Clusters
| Provider | Command |
|---|
| EKS | eksctl upgrade cluster --name N --version 1.32 |
| GKE | gcloud container clusters upgrade N --cluster-version=1.32 |
| AKS | az aks upgrade -g G -n N --kubernetes-version 1.32.0 |
6. Using Blue-Green Strategy
Blue/Green Cluster Upgrade
[blue cluster v1.31] -- traffic --→
↘
[green cluster v1.32] (replica) ←-- switch DNS / LB --
| Step | Detail |
|---|
| Per-node | drain → upgrade → uncordon |
| Parallelism | Limit to 1-2 nodes; respect PDBs |
8. Testing Before Upgrade
| Tool | Use |
|---|
| kube-no-trouble (kubent) | Detect deprecated APIs in cluster |
| pluto | Audit Helm/YAML for removed APIs |
| Sonobuoy | Conformance + e2e tests |
9. Rolling Back Upgrades
| Layer | Method |
|---|
| Control plane | Restore etcd snapshot |
| Workers | Downgrade kubelet package (skew permitting) |
| Managed | Most providers don't allow version rollback |
10. Monitoring Upgrade Progress
kubectl get nodes -o wide
kubectl get componentstatuses
kubectl version
kubectl get pods -A | grep -v Running