Kubernetes Requests, Limits, Autoscaling and Scheduling
Set CPU and memory requests and limits, resize pods in place, add LimitRanges and quotas, autoscale with HPA, and drain nodes safely with taints and PDBs.
TLDR: Requests are what the scheduler reserves for a container; limits are the most it may use. A container over its memory limit is killed (OOMKilled); one over its CPU limit is slowed down. Set requests on everything, a memory limit on everything, and think twice about CPU limits. LimitRanges fill in defaults and ResourceQuotas cap a namespace. The HorizontalPodAutoscaler adds pods when they get busy, and since Kubernetes 1.35 you can change a running pod's CPU and memory in place. Labels, taints, affinity, topology spread and PodDisruptionBudgets decide where pods run and how safely nodes can be drained.
In this part you install metrics-server, give podinfo a resource budget, watch a container get OOMKilled, resize a pod without restarting it, add a quota and see what happens when a Deployment hits it, load-test podinfo until the autoscaler adds pods, and drain a node.
Prerequisites
- The app as it stands after Part 7: podinfo, its Service and the cache StatefulSet.
kubectl topneeds metrics-server, which you install first.
Install metrics-server
metrics-server collects CPU and memory usage from every kubelet and serves it through the metrics.k8s.io API, which kubectl top and the HorizontalPodAutoscaler read. That API became stable (v1) in Kubernetes 1.37. Most managed services install metrics-server for you. On kind, the kubelets use self-signed certificates, so metrics-server needs --kubelet-insecure-tls there (and only there):
After about a minute, the first numbers arrive:
podinfo uses about a millicore of CPU and under 20 MiB of memory while idle. These numbers are the starting point for sizing it.
Requests and Limits
resources:
requests: # reserved on the node; used for scheduling
cpu: 100m # 100 millicores = 0.1 of a CPU core
memory: 64Mi # 64 mebibytes
limits: # the ceiling at run time
memory: 128Mi # exceed it and the container is killed
- CPU is measured in cores:
1is a full core,100ma tenth of one. CPU is compressible: a container that wants more than its limit is throttled, not killed. - Memory is measured in bytes with binary suffixes (
Ki,Mi,Gi).128M(decimal) and128Mi(binary) are different numbers. Memory is not compressible: a container that goes over its limit is killed by the kernel's out-of-memory killer and restarted. - The scheduler only looks at requests. It places a pod on a node whose free allocatable resources cover the pod's requests, whatever the pod really uses.
A common, defensible policy: set CPU and memory requests close to what the app uses under normal load, set a memory limit (often equal to the request, so the pod cannot use memory it did not reserve), and leave the CPU limit off, so the app can use idle CPU on the node instead of being throttled. If you share nodes with teams you do not trust, add CPU limits or quotas. The Part 8 Deployment follows that policy, and spreads its pods over nodes:
apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo
labels:
app: podinfo
spec:
replicas: 3
revisionHistoryLimit: 5
progressDeadlineSeconds: 120
minReadySeconds: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: podinfo
template:
metadata:
labels:
app: podinfo
spec:
containers:
- name: podinfo
image: ghcr.io/stefanprodan/podinfo:6.15.0
command: ['./podinfo']
args:
- --config-path=/podinfo/config # the ConfigMap file
- --secure-port=8443 # HTTPS, with the key pair from the Secret
- --cert-path=/podinfo/tls
- --cache-server=tcp://cache:6379
env:
# Non-secret settings, one key from the ConfigMap each
- name: PODINFO_UI_MESSAGE
valueFrom:
configMapKeyRef:
name: podinfo-config
key: PODINFO_UI_MESSAGE
- name: PODINFO_LEVEL
valueFrom:
configMapKeyRef:
name: podinfo-config
key: PODINFO_LEVEL
ports:
- name: http
containerPort: 9898
- name: https
containerPort: 8443
volumeMounts:
- name: config
mountPath: /podinfo/config
readOnly: true
- name: tls # the Secret, as files: tls.crt and tls.key
mountPath: /podinfo/tls
readOnly: true
readinessProbe:
httpGet:
path: /readyz
port: http
periodSeconds: 5
livenessProbe:
httpGet:
path: /healthz
port: http
periodSeconds: 10
resources:
requests:
cpu: 100m # used for scheduling and as the HPA's 100 percent
memory: 64Mi
limits:
memory: 128Mi # no CPU limit: the container may use idle CPU on the node
# Prefer to put the replicas on different nodes
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: podinfo
volumes:
- name: config
configMap:
name: podinfo-config
items: # only this key becomes a file
- key: features.yaml
path: features.yaml
- name: tls
secret:
secretName: podinfo-tls
Quality of Service Classes
Kubernetes puts every pod into a QoS class from its requests and limits:
| Class | When |
|---|---|
Guaranteed |
Every container has CPU and memory requests equal to its limits |
Burstable |
At least one request or limit is set, but not Guaranteed |
BestEffort |
No requests or limits at all |
When a node runs low on memory, the kubelet evicts pods to protect itself. It goes first for pods that use more memory than they requested, then considers pod priority, then how far over their requests they are. BestEffort pods request nothing, so any memory they use counts as over their requests; Guaranteed pods cannot use more than they request, so they usually come last, but they are not immune when the node itself runs out. Priority can still move a pod ahead of another within the same group. The class also tunes the kernel's out-of-memory killer in the same order. podinfo and the cache are Burstable: they have requests, a memory limit and no CPU limit, the right class for most apps. Use Guaranteed for the workloads you most want to keep under memory pressure, and avoid BestEffort outside experiments.
OOMKilled
This pod asks podinfo to load 300 MB into memory with a limit of 128 MiB:
apiVersion: v1
kind: Pod
metadata:
name: memory-hog
spec:
containers:
- name: podinfo
image: ghcr.io/stefanprodan/podinfo:6.15.0
command: ['./podinfo']
args: ['--stress-memory=300'] # load 300 MB into memory at startup
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
memory: 128Mi
Here the container went over its own 128 MiB limit. In general, reason: OOMKilled means the kernel's out-of-memory killer ended the process, either because the container hit its limit or because the whole node ran out of memory; compare the container's usage with its limit, and check the node for memory pressure, before you raise anything. Exit code 137 (128 plus signal 9) alone only says the process got SIGKILL, which also happens when a container ignores SIGTERM until its grace period runs out. The kubelet restarts the container and backs off a little longer each time, so the pod soon shows CrashLoopBackOff. When the limit is the cause, the fix is a higher limit or an app that uses less memory, for example a JVM or Node.js heap size that fits inside the limit.
Resize a Running Pod
Changing the resources in a Deployment's template replaces its pods. Since Kubernetes 1.35, in-place pod resize is stable: you change a running pod's requests and limits through its resize subresource, and the kubelet applies the change without recreating the pod. resizePolicy decides per resource whether the container must restart:
apiVersion: v1
kind: Pod
metadata:
name: resize-demo
spec:
containers:
- name: podinfo
image: ghcr.io/stefanprodan/podinfo:6.15.0
resizePolicy:
- resourceName: cpu
restartPolicy: NotRequired # the default: apply a new CPU size without a restart
- resourceName: memory
restartPolicy: RestartContainer # restart the container to apply a new memory size
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
The patch only changes the spec; the kubectl wait steps wait until the kubelet reports the new values in the pod's status. The CPU change applied with 0 restarts. The memory change restarted the container once, as its RestartContainer policy asked. A resize that no node could ever satisfy is rejected right away:
A resize that does fit the node but not its current free capacity stays pending, with a PodResizePending condition on the pod, until room frees up. Resizing pods that a Deployment owns works too, but the next rollout creates pods from the template again, so the Vertical Pod Autoscaler (below) is the usual way to use this for Deployments.
LimitRanges and ResourceQuotas
A LimitRange sets per-container defaults and bounds in a namespace. A ResourceQuota caps the total of the whole namespace. Together they keep one team or one bad manifest from using a shared cluster up:
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
spec:
limits:
- type: Container
defaultRequest: # used when a container sets no request
cpu: 50m
memory: 64Mi
default: # used when a container sets no limit
memory: 256Mi
max: # reject containers that ask for more
memory: 1Gi
apiVersion: v1
kind: ResourceQuota
metadata:
name: demo-quota
spec:
hard:
requests.cpu: '2' # sum of all CPU requests in the namespace
requests.memory: 2Gi
limits.memory: 4Gi
pods: '20'
persistentvolumeclaims: '5'
services.loadbalancers: '1'
The LimitRange has no CPU limit default and no CPU maximum on purpose: a max for a resource forces every container to set a limit for it, which would contradict the no-CPU-limit policy above.
The defaults pod set no resources, and the LimitRange filled them in. Because the quota limits requests.cpu and limits.memory, every new pod in the namespace must have those values, which the LimitRange guarantees.
Now ask for more replicas than the quota allows:
The Deployment stopped short of 25, and nothing failed loudly: kubectl scale succeeded, and the reason only shows up in the ReplicaSet's FailedCreate events. When pods are missing in a namespace with a quota, look there first.
Horizontal Pod Autoscaling
The HorizontalPodAutoscaler (HPA, API autoscaling/v2) changes a workload's replica count to keep a metric near a target. With CPU, the target is a percentage of the pods' CPU requests, which is one more reason to set requests that reflect real use:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: podinfo
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
minReplicas: 3
maxReplicas: 8
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60 # percent of the CPU request
behavior:
scaleUp:
stabilizationWindowSeconds: 0
tolerance: 0.05 # react to a 5 percent difference (the default is 10 percent)
scaleDown:
stabilizationWindowSeconds: 60 # the default is 300 seconds
policies:
- type: Pods
value: 2 # remove at most two pods
periodSeconds: 30 # every 30 seconds
behavior controls how fast it reacts. Scaling up right away and scaling down slowly, at most two pods every 30 seconds after a minute of low load, avoids flapping. tolerance per HPA has been on by default since Kubernetes 1.35 (beta) and is stable in 1.37; without it, the HPA uses the cluster-wide 10 percent dead band.
For a one-off you can also run kubectl autoscale deployment podinfo --min=3 --max=8 --cpu=60%; the older --cpu-percent flag is deprecated.
Start a load generator. fortio sends requests from 16 connections as fast as it can for three minutes:
kubectl run loadgen --image=fortio/fortio:1.76.0 --restart=Never -- load -qps 0 -c 16 -t 3m -nocatchup http://podinfo/
pod/loadgen created
Then watch the autoscaler. This run was stopped after seven minutes, enough for the load to end and the HPA to scale back down:
kubectl get hpa podinfo --watch
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
podinfo Deployment/podinfo cpu: 1%/60% 3 8 3 0s
podinfo Deployment/podinfo cpu: 121%/60% 3 8 3 15s
podinfo Deployment/podinfo cpu: 810%/60% 3 8 7 30s
podinfo Deployment/podinfo cpu: 406%/60% 3 8 8 45s
podinfo Deployment/podinfo cpu: 368%/60% 3 8 8 60s
podinfo Deployment/podinfo cpu: 328%/60% 3 8 8 75s
podinfo Deployment/podinfo cpu: 332%/60% 3 8 8 90s
podinfo Deployment/podinfo cpu: 326%/60% 3 8 8 105s
podinfo Deployment/podinfo cpu: 329%/60% 3 8 8 2m
podinfo Deployment/podinfo cpu: 331%/60% 3 8 8 2m15s
podinfo Deployment/podinfo cpu: 333%/60% 3 8 8 2m30s
podinfo Deployment/podinfo cpu: 332%/60% 3 8 8 2m45s
podinfo Deployment/podinfo cpu: 293%/60% 3 8 8 3m15s
podinfo Deployment/podinfo cpu: 148%/60% 3 8 8 3m30s
podinfo Deployment/podinfo cpu: 1%/60% 3 8 8 3m45s
podinfo Deployment/podinfo cpu: 1%/60% 3 8 8 4m30s
podinfo Deployment/podinfo cpu: 1%/60% 3 8 6 4m45s
podinfo Deployment/podinfo cpu: 1%/60% 3 8 6 5m
podinfo Deployment/podinfo cpu: 1%/60% 3 8 4 5m15s
podinfo Deployment/podinfo cpu: 1%/60% 3 8 4 5m30s
podinfo Deployment/podinfo cpu: 1%/60% 3 8 3 5m45s
Within a minute of the load starting, CPU use passed 600 percent of the requests, and the HPA went from 3 to 7 replicas and then to its maximum of 8. When fortio stopped after three minutes, CPU dropped to 1 percent, and about a minute later the HPA started removing pods, never more than two in 30 seconds, until it was back at 3. The HPA's events record each decision:
fortio's summary shows the load the HPA reacted to: every request succeeded, at an average of under a millisecond. CPU is a good first metric. Queue workers scale better on queue length, and HTTP services on requests per second; those need a custom or external metrics adapter, such as the Prometheus Adapter or KEDA. Since Kubernetes 1.37, an HPA with object or external metrics can also scale a workload to zero (beta, on by default).
Vertical and Cluster Autoscaling
- The Vertical Pod Autoscaler (VPA, a separate project) recommends requests from real usage and can apply them. VPA 1.8 supports
Off(recommendations only),Initial,RecreateandInPlaceOrRecreate, which uses in-place resize;Autois deprecated. Do not let the VPA and an HPA act on the same metric. An example that only manages memory, leaving CPU to the HPA:
# Needs the Vertical Pod Autoscaler (kubernetes/autoscaler, vertical-pod-autoscaler 1.8)
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: podinfo
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
updatePolicy:
updateMode: 'Off' # recommendations only; InPlaceOrRecreate applies them
resourcePolicy:
containerPolicies:
- containerName: podinfo
controlledResources: ['memory'] # leave CPU to the HPA
minAllowed:
memory: 64Mi
maxAllowed:
memory: 512Mi
- The Cluster Autoscaler or Karpenter add nodes when pods cannot be scheduled for lack of capacity, and remove nodes that stay underused. They react to pending pods and requests, not to CPU usage, so again: set requests. On managed services you turn this on per node pool. On DigitalOcean Kubernetes, for example, this lets the node pool from Part 2 grow to five nodes (the new nodes are billed Droplets; if you sign up through our referral link, see Part 2 for the credit terms):
doctl kubernetes cluster node-pool update k8s-guide workers --auto-scale --min-nodes 2 --max-nodes 5
Scheduling: Where Pods Run
The scheduler places a pod on a node that has room for its requests and meets its constraints. You influence it with:
nodeSelectorand node affinity: run only on nodes with certain labels (disktype=ssd, a zone, an instance type).required...rules are hard,preferred...rules are hints.- Taints and tolerations: a taint on a node keeps away every pod that does not tolerate it. Use it to reserve nodes for special work such as GPUs or batch jobs. Control plane nodes carry a taint, which is why your pods never run there.
- Pod affinity and anti-affinity: run near, or away from, pods with certain labels.
- Topology spread constraints: spread replicas evenly over nodes or zones. The Part 8 Deployment uses one with
whenUnsatisfiable: ScheduleAnyway, a preference rather than a hard rule. - PriorityClasses: let important pods preempt less important ones when the cluster is full.
Taints and Tolerations
Reserve one worker for batch work. To set up the drain at the end of this part, pick the worker that does not hold the cache's volume from Part 7 as the batch node:
NODE=$(kubectl get pod cache-0 -o jsonpath='{.spec.nodeName}')
if [ "$NODE" = k8s-guide-worker ]; then OTHER=k8s-guide-worker2; else OTHER=k8s-guide-worker; fi
echo "cache-0 runs on $NODE, the batch node is $OTHER"
cache-0 runs on k8s-guide-worker2, the batch node is k8s-guide-worker
Taint and label the batch node, and restart podinfo so the scheduler places its pods again:
All podinfo pods now run on $NODE, next to cache-0, even though the spread constraint prefers two nodes: a taint is a hard rule, a ScheduleAnyway spread only a preference. Pods that were already running on a node when it was tainted stay there, because NoSchedule only affects new placements (NoExecute would evict them). A pod that tolerates the taint and requires the label lands on the batch node:
apiVersion: v1
kind: Pod
metadata:
name: batch-task
spec:
# Allowed onto nodes with the dedicated=batch:NoSchedule taint ...
tolerations:
- key: dedicated
operator: Equal
value: batch
effect: NoSchedule
# ... and required to run on a node labeled workload=batch
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: workload
operator: In
values: ['batch']
containers:
- name: task
image: busybox:1.38.0
command: ['sleep', '3600']
The toleration alone only allows the pod onto the tainted node; the node affinity is what sends it there. You need both to dedicate nodes. The trailing - removes the taint again. The podinfo pods stay where they are until something moves them, which is what the next step does.
Drain a Node Safely
Before node maintenance or an upgrade, you drain the node: kubectl cordons it (no new pods) and evicts its pods, which their controllers recreate elsewhere. A PodDisruptionBudget (PDB) limits how many pods of an app such voluntary evictions may take down at once:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: podinfo
spec:
minAvailable: 2 # voluntary evictions must leave at least two ready pods
selector:
matchLabels:
app: podinfo
# Pods that are not ready can be evicted anyway, so they cannot block a drain
unhealthyPodEvictionPolicy: AlwaysAllow
$NODE now runs all three podinfo pods and the cache. Drain it:
Read the drain output. The PDB allowed one podinfo eviction at a time and refused the others ("Cannot evict pod as it would violate the pod's disruption budget") until each replacement was ready on the other node. kubectl retried every 5 seconds, and the app never dropped below two ready pods.
The cache did not come back, though. cache-0 is Pending:
Its PersistentVolume is a directory on the drained node (didn't match PersistentVolume's node affinity), and that node is cordoned (were unschedulable), so there is nowhere it may run. After kubectl uncordon, it starts again with its data. With network storage such as a cloud disk, the volume would follow the pod to another node in the same zone. This is the real cost of node-local storage, and the reason stateful apps need replicas, PDBs and storage that can move.
Common Mistakes
- No requests. The scheduler then packs pods onto nodes as if they used nothing, and the HPA cannot compute a CPU percentage at all.
- Memory limits far above requests. Many pods can then burst at once and push the node into memory pressure; keep them close, or equal.
- Tight CPU limits on latency-sensitive apps. The CPU throttling shows up as slow responses, not errors.
- Requests from guesses. Measure with
kubectl top, Prometheus or the VPA's recommendations, and adjust. - No PDB on apps with several replicas. A node drain can then take all of them down at once. And no PDB with
minAvailableequal to the replica count: that blocks drains forever.
Summary
- Requests drive scheduling and HPA percentages; memory limits cause OOMKills, CPU limits cause throttling.
- Under memory pressure the kubelet evicts pods that use more than they requested first, then weighs priority; QoS classes follow from requests and limits, and
Burstablewith a memory limit fits most apps. - In-place resize (stable since 1.35) changes a running pod's resources, with or without a restart per
resizePolicy. - LimitRanges fill in defaults, ResourceQuotas cap namespaces, and quota failures show up as
FailedCreateevents. - The HPA scales replicas on metrics; the VPA sizes requests; the Cluster Autoscaler or Karpenter adds nodes.
- Taints, affinity and spread constraints steer the scheduler; PDBs keep drains safe; node-local volumes pin pods to nodes.
Keep Learning
- Right-sizing Kubernetes resources with VPA and Karpenter
- Checking pod CPU and memory utilization
- Kubernetes HPA lab
- Kubernetes scheduler simulator
Next, in Part 9, you debug broken workloads and set up metrics and alerts.
Tags
Found an issue?