Working with Taints and Tolerations
1. Adding Taints to Nodes
kubectl taint nodes gpu-1 gpu=true:NoSchedule
kubectl taint nodes node-3 dedicated=team-a:NoExecute
| Format | Detail |
|---|
key=value:effect | Standard taint syntax |
key:effect | Value optional |
2. Removing Taints from Nodes
kubectl taint nodes gpu-1 gpu=true:NoSchedule-
kubectl taint nodes gpu-1 gpu- # remove all with key
3. Listing Node Taints
kubectl describe node gpu-1 | grep Taints
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
4. Understanding Taint Effects
| Effect | Behavior |
|---|
| NoSchedule | Reject new pods without toleration |
| PreferNoSchedule | Best-effort avoidance |
| NoExecute | Evict existing pods without toleration |
5. Configuring Tolerations
tolerations:
- key: gpu
operator: Equal
value: "true"
effect: NoSchedule
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 300
6. Using Toleration Operators
| operator | Match |
|---|
| Equal | key & value & effect match |
| Exists | key & effect match (any value) |
7. Setting Toleration Seconds
| Field | Effect |
|---|
| tolerationSeconds | How long pod stays before eviction (NoExecute) |
| Unset | Tolerate indefinitely |
8. Understanding Default Tolerations
| Built-in Taint | Auto Tolerated |
|---|
node.kubernetes.io/not-ready | 300s by default |
node.kubernetes.io/unreachable | 300s by default |
node.kubernetes.io/memory-pressure | No tolerance (eviction) |
node.kubernetes.io/disk-pressure | No tolerance |
node.kubernetes.io/pid-pressure | No tolerance |
9. Using Taints for Dedicated Nodes
Example: GPU node pool isolation
kubectl taint nodes gpu-* dedicated=ml:NoSchedule
kubectl label nodes gpu-* workload=ml
spec:
nodeSelector: { workload: ml }
tolerations:
- { key: dedicated, operator: Equal, value: ml, effect: NoSchedule }
10. Combining Taints with Node Affinity
Note: Taints repel pods without toleration. Node affinity attracts pods to specific nodes. Use both for strict dedicated-node patterns.
| Goal | Mechanism |
|---|
| Only GPU pods on GPU nodes | Taint + toleration (repel non-GPU) |
| GPU pods MUST be on GPU nodes | + nodeAffinity / nodeSelector |