Kubernetes SIGTERM

SIGTERM (Signal 15) is Kubernetes’ graceful shutdown signal sent to container processes when a Pod is being terminated. Unlike SIGKILL which forcefully kills processes, SIGTERM is a polite request asking the application to shutdown cleanly.

Shutdown Sequence

When Kubernetes terminates a Pod, SIGTERM initiates the graceful shutdown sequence:

1. Service Deregistration - Pod removed from Service endpoint lists
2. PreStop Hook - PreStop hook executes if configured (blocking call)
3. SIGTERM Delivery - Signal sent to PID 1 in each container
4. Grace Period - Application has .spec.terminationGracePeriodSeconds (default 30) to shutdown
5. Forced Termination - If process hasn’t exited, SIGKILL terminates it

Applications must handle SIGTERM to implement managed lifecycle properly.

Expected Application Behavior

When receiving SIGTERM, applications should:

Complete In-Flight Requests - Finish processing active requests rather than immediately exiting. For web servers, complete HTTP requests in progress. For workers, finish current job processing.

Stop Accepting New Work - Close listening sockets, stop polling queues, reject new connections. This prevents accepting work that won’t have time to complete.

Release Resources Gracefully - Close database connections properly, flush buffers to disk, commit or rollback transactions, release locks, and clean up temporary files.

Exit with Appropriate Code - Exit with status 0 after successful cleanup, non-zero if cleanup failed. Exit codes help operators understand shutdown success.

Implementation Patterns

Signal Handler Registration

Most languages require explicit signal handler registration:

Go Example:

sigterm := make(chan os.Signal, 1)
signal.Notify(sigterm, syscall.SIGTERM)
 
go func() {
    <-sigterm
    log.Println("Received SIGTERM, starting graceful shutdown")
 
    // Stop accepting new requests
    if err := server.Shutdown(context.Background()); err != nil {
        log.Printf("Shutdown error: %v", err)
    }
 
    // Wait for in-flight work
    wg.Wait()
 
    // Cleanup resources
    db.Close()
 
    os.Exit(0)
}()

Python Example (with signal module):

import signal
import sys
 
def handle_sigterm(signum, frame):
    print("Received SIGTERM, shutting down gracefully")
 
    # Stop accepting new work
    server.stop_accepting()
 
    # Complete in-flight requests
    server.wait_for_completion(timeout=25)
 
    # Cleanup
    db.close()
 
    sys.exit(0)
 
signal.signal(signal.SIGTERM, handle_sigterm)

Container PID 1 Consideration - The process running as PID 1 in a container receives signals. If using shell scripts as entrypoints, ensure they propagate signals correctly, or use exec to replace the shell with the application process.

Timeout Management

Applications must complete shutdown within the grace period:

Default Grace Period - 30 seconds provides reasonable time for most applications. Configure terminationGracePeriodSeconds if you need more:

spec:
  terminationGracePeriodSeconds: 60  # For long-running transactions
  containers:
  - name: app
    image: myapp:v1

Internal Timeout - Applications should use a timeout shorter than the grace period to ensure cleanup completes:

// Grace period is 30 seconds, use 25 for safety
ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
 
if err := server.Shutdown(ctx); err != nil {
    log.Printf("Forced shutdown after timeout: %v", err)
}

This buffer ensures cleanup logic completes before SIGKILL arrives.

Integration with Deployment Strategies

SIGTERM handling is critical for zero-downtime deployments:

Rolling Deployment - During rollout, old Pods receive SIGTERM. Proper handling ensures in-flight requests complete before Pod termination.

Connection Draining - Even though Kubernetes removes Pods from Service endpoints before SIGTERM, some clients may have cached endpoint lists or persistent connections. Applications must drain these connections during the grace period.

Health Check Coordination - Applications should fail readiness probes after receiving SIGTERM to signal they’re shutting down:

isShuttingDown := false
 
http.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
    if isShuttingDown {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
})
 
// In SIGTERM handler
isShuttingDown = true

This ensures Kubernetes doesn’t send new traffic during shutdown.

Common Pitfalls

Ignoring SIGTERM - Applications that don’t handle SIGTERM will be forcefully killed after the grace period, causing dropped requests and corrupted data.

Shell Script Entrypoints - Shell scripts don’t propagate signals to child processes by default:

# Bad - script receives SIGTERM, app doesn't
#!/bin/bash
/usr/local/bin/myapp
 
# Good - exec replaces shell with app
#!/bin/bash
exec /usr/local/bin/myapp

Use exec to ensure the application receives signals directly.

PID 1 Signal Handling - Processes running as PID 1 have special signal handling in Linux. Some signals may be ignored. Use tini or dumb-init as lightweight init systems if experiencing signal handling issues.

Insufficient Grace Period - Long-running operations (batch processing, large file uploads) may need extended grace periods. Monitor shutdown times and adjust terminationGracePeriodSeconds accordingly.

Relationship to Lifecycle Hooks

SIGTERM complements PreStop hooks:

PreStop Hook - Executes before SIGTERM, useful for cleanup tasks or graceful shutdown preparation for applications that can’t handle signals directly.

SIGTERM - Primary shutdown mechanism. Applications should handle signals directly when possible rather than relying on hooks.

The recommended pattern is:

1. Handle SIGTERM in application code for graceful shutdown
2. Use PreStop hooks only for supplementary tasks like deregistering from external service discovery or notifying monitoring systems

Managed lifecycle requires proper signal handling to enable automated operations.

Observability

Log SIGTERM receipt to aid debugging:

log.Printf("Received SIGTERM at %v, initiating shutdown", time.Now())
 
// Track shutdown phases
log.Println("Stopped accepting new requests")
log.Printf("Draining %d in-flight requests", inflightCount)
log.Println("Closed database connections")
log.Printf("Shutdown completed in %v", shutdownDuration)

Metrics help tune grace periods:

shutdownDuration.Observe(elapsed.Seconds())
inflightRequestsOnShutdown.Observe(float64(count))

This observability reveals whether grace periods are sufficient and how applications behave during termination.

Best Practices

Always Handle Signals - Every containerized application should handle SIGTERM explicitly. Don’t assume frameworks or runtime environments handle it correctly.

Use PreStop for Preparation - PreStop hooks are useful for tasks like removing load balancer backends or flushing metrics. Use them to supplement, not replace, SIGTERM handling.

Test Shutdown Logic - Regularly test graceful shutdown in non-production environments. Send SIGTERM manually and verify clean shutdown:

kubectl exec -it pod-name -- kill -TERM 1

Monitor Grace Period Usage - If containers consistently hit the grace period limit, either optimize shutdown logic or increase terminationGracePeriodSeconds.

Fail Fast When Needed - If critical cleanup fails, exit with non-zero status and log errors. This helps operators identify and fix shutdown issues.

SIGTERM handling transforms containers from brittle processes into cooperative cloud-native citizens that work harmoniously with managed lifecycle patterns.