Kubernetes Pod Priority

Pod Priority enables explicit prioritization of workloads through PriorityClass resources. Unlike Quality of Service classes, which are automatically derived from resource profiles, priority is explicitly assigned and controls preemption behavior.

PriorityClass

PriorityClass is a cluster-wide resource that defines priority levels:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "Critical production services"

Pods reference a PriorityClass to inherit its priority value:

apiVersion: v1
kind: Pod
metadata:
  name: critical-app
spec:
  priorityClassName: high-priority
  containers:
  - name: app
    image: myapp:latest

Preemption Mechanism

When a high-priority Pod cannot be scheduled due to insufficient resources, Kubernetes may preempt (evict) lower-priority Pods to make room.

Preemption Process:

1. Scheduler identifies that a Pod can’t be scheduled
2. Searches for nodes where evicting lower-priority Pods would free enough resources
3. Selects victim Pods with priority lower than the pending Pod
4. Evicts victims gracefully (respecting terminationGracePeriodSeconds)
5. Schedules the high-priority Pod once resources are freed

Preemption Selection: Among potential victims, Kubernetes prefers to evict:

• Lowest priority Pods first
• Pods with the shortest runtime (newer Pods)
• Pods using the least amount of resources beyond requests

Priority vs QoS

Priority and QoS class serve different but complementary purposes:

Priority:

• Explicit via PriorityClass
• Controls scheduling order and preemption
• Determines which Pods get scheduled first
• Enables lower-priority eviction for higher-priority scheduling

QoS:

• Implicit from resource profiles
• Controls eviction under node resource pressure
• Determines which Pods get killed when nodes run out of memory
• No impact on scheduling decisions

Orthogonality: A Pod can be high-priority BestEffort or low-priority Guaranteed. During scheduling, priority wins. During node resource pressure, QoS wins.

Common Priority Patterns

System Critical - Highest priority for core infrastructure (priority: 2000000000)

• CNI plugins
• DNS
• Metrics collectors

Production Services - High priority for customer-facing workloads (priority: 1000000)

• API servers
• Web frontends
• Core business logic

Background Jobs - Low priority for batch processing (priority: 0 or negative)

• Batch analytics
• Backup jobs
• Non-urgent processing

Development - Lowest priority for test environments (priority: -1000)

• CI/CD pipelines
• Development deployments
• Experimental services

Priority and Resource Quotas

ResourceQuotas can limit priority usage in Namespaces:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: priority-quota
spec:
  hard:
    pods: "10"
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values: ["high-priority"]

This prevents abuse of high-priority classes and ensures fair resource distribution across teams.

Preemption Caveats

Grace Periods: Preempted Pods receive graceful shutdown signals. If they don’t terminate within terminationGracePeriodSeconds, they’re force-killed.

PodDisruptionBudgets (PDBs): Preemption respects PDBs when possible, but may violate them temporarily if necessary to schedule critical workloads.

Resource Fragmentation: Preemption doesn’t guarantee scheduling - it may evict Pods from multiple nodes without finding a single node with enough resources.