Gradual Notes

Recent Writing

Recent Notes

Kubernetes Commandlet Pattern

9 min read

The Commandlet pattern replaces a container’s main entrypoint with a generic wrapper that handles lifecycle events, providing fine-grained control over startup, shutdown, and signal handling without modifying application code. It’s an advanced managed lifecycle technique that offers stronger guarantees and greater reusability than hooks alone.

Core Concept

Entrypoint Replacement - Instead of running the application binary directly, the container executes a wrapper (the “commandlet”) that:

  1. Performs initialization tasks
  2. Starts the actual application as a child process
  3. Intercepts and handles lifecycle signals
  4. Performs cleanup on shutdown
  5. Ensures proper process management

The application runs unchanged while the wrapper provides lifecycle intelligence.

Generic Wrapper - The commandlet is a reusable component, not application-specific. The same wrapper can manage lifecycle for different applications, often configured through environment variables or configuration files.

Injection Pattern - Commandlets are typically injected into containers by Init Containers rather than being built into application images. This separates lifecycle concerns from application packaging.

Implementation Approaches

Basic Shell Wrapper

A simple shell script wrapper handling SIGTERM:

#!/bin/sh
# commandlet-wrapper.sh
 
# Trap SIGTERM and forward to child
trap 'kill -TERM $APP_PID' TERM
 
# Startup initialization
echo "Starting application at $(date)"
/pre-startup-tasks.sh
 
# Start the application in background
"$@" &
APP_PID=$!
 
# Wait for application to exit
wait $APP_PID
EXIT_CODE=$?
 
# Cleanup
echo "Application exited with code $EXIT_CODE"
/post-shutdown-tasks.sh
 
exit $EXIT_CODE

Container configuration:

containers:
  - name: app
    image: myapp:v1
    command: ["/wrapper/commandlet-wrapper.sh"]
    args: ["/usr/local/bin/myapp", "--config", "/etc/app/config.yaml"]

The wrapper receives the application command as arguments via "$@".

Advanced Go Commandlet

A more sophisticated implementation in Go:

package main
 
import (
    "os"
    "os/exec"
    "os/signal"
    "syscall"
    "time"
)
 
func main() {
    if len(os.Args) < 2 {
        log.Fatal("Usage: commandlet <command> [args...]")
    }
 
    // Pre-startup tasks
    if err := runPreStartup(); err != nil {
        log.Fatalf("Pre-startup failed: %v", err)
    }
 
    // Start application
    cmd := exec.Command(os.Args[1], os.Args[2:]...)
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    cmd.Stdin = os.Stdin
 
    if err := cmd.Start(); err != nil {
        log.Fatalf("Failed to start application: %v", err)
    }
 
    // Handle signals
    sigterm := make(chan os.Signal, 1)
    signal.Notify(sigterm, syscall.SIGTERM, syscall.SIGINT)
 
    go func() {
        <-sigterm
        log.Println("Received SIGTERM, forwarding to application")
 
        // Give application time to shutdown gracefully
        cmd.Process.Signal(syscall.SIGTERM)
 
        // Wait with timeout
        done := make(chan error, 1)
        go func() {
            done <- cmd.Wait()
        }()
 
        select {
        case <-done:
            log.Println("Application exited gracefully")
        case <-time.After(25 * time.Second):
            log.Println("Application didn't exit, sending SIGKILL")
            cmd.Process.Kill()
        }
 
        runPostShutdown()
        os.Exit(0)
    }()
 
    // Wait for application to exit normally
    if err := cmd.Wait(); err != nil {
        log.Printf("Application exited with error: %v", err)
        runPostShutdown()
        os.Exit(1)
    }
 
    runPostShutdown()
    os.Exit(0)
}
 
func runPreStartup() error {
    // Wait for dependencies
    // Load configuration
    // Register with service discovery
    return nil
}
 
func runPostShutdown() {
    // Deregister from service discovery
    // Flush metrics
    // Cleanup resources
}

This commandlet handles graceful shutdown with timeouts, ensuring the application receives SIGTERM and has time to respond before SIGKILL.

Injection via Init Container

Commandlets are typically injected using Init Containers:

apiVersion: v1
kind: Pod
metadata:
  name: app-with-commandlet
spec:
  volumes:
    - name: commandlet-volume
      emptyDir: {}
 
  initContainers:
    - name: inject-commandlet
      image: commandlet-wrapper:v1
      command:
        - sh
        - -c
        - |
          cp /commandlet /shared/commandlet
          chmod +x /shared/commandlet
      volumeMounts:
        - name: commandlet-volume
          mountPath: /shared
 
  containers:
    - name: app
      image: myapp:v1
      command: ["/shared/commandlet"]
      args: ["/usr/local/bin/myapp"]
      volumeMounts:
        - name: commandlet-volume
          mountPath: /shared

The Init Container places the wrapper binary in a shared volume; the application container uses it as its entrypoint.

Benefits Over Hooks

The Commandlet pattern provides advantages over PostStart and PreStop hooks:

Guaranteed Ordering - The wrapper controls exactly when the application starts. No race conditions like PostStart hooks.

Signal Control - The wrapper can intercept, transform, or enhance signals before forwarding to the application. Applications that don’t handle SIGTERM can be wrapped to provide graceful shutdown.

Timeout Management - The wrapper implements its own timeouts independent of Kubernetes grace periods, ensuring shutdown logic completes within time budgets.

Reusability - The same wrapper works for multiple applications. Lifecycle logic is centralized and versioned separately from application code.

Observability - The wrapper logs lifecycle events, providing visibility into startup and shutdown behavior without modifying application code.

Process Management - The wrapper ensures proper PID 1 signal handling, zombie process reaping, and clean process tree termination.

Use Cases

Legacy Application Lifecycle

Provide managed lifecycle compliance for applications that can’t be modified:

#!/bin/sh
# Wrapper for legacy app that ignores SIGTERM
 
trap 'legacy_shutdown' TERM
 
legacy_shutdown() {
    # Use application-specific shutdown API
    curl -X POST http://localhost:8080/admin/shutdown
    sleep 5
    kill -9 $APP_PID
}
 
/legacy-app &
APP_PID=$!
wait $APP_PID

Startup Dependency Ordering

Ensure dependencies are available before application starts:

func runPreStartup() error {
    // Wait for database
    if err := waitForService("postgres-service:5432", 60*time.Second); err != nil {
        return fmt.Errorf("database not available: %w", err)
    }
 
    // Wait for cache
    if err := waitForService("redis-service:6379", 60*time.Second); err != nil {
        return fmt.Errorf("cache not available: %w", err)
    }
 
    // Run migrations
    if err := runMigrations(); err != nil {
        return fmt.Errorf("migrations failed: %w", err)
    }
 
    return nil
}

This provides stronger guarantees than Init Containers because it runs in the same container with the same network namespace.

Service Registry Integration

Automatically register/deregister with service discovery:

func runPreStartup() error {
    podIP := os.Getenv("POD_IP")
    serviceName := os.Getenv("SERVICE_NAME")
 
    // Register with Consul
    client, _ := consulapi.NewClient(consulapi.DefaultConfig())
    registration := &consulapi.AgentServiceRegistration{
        ID:      fmt.Sprintf("%s-%s", serviceName, podIP),
        Name:    serviceName,
        Address: podIP,
        Port:    8080,
    }
    return client.Agent().ServiceRegister(registration)
}
 
func runPostShutdown() {
    podIP := os.Getenv("POD_IP")
    serviceName := os.Getenv("SERVICE_NAME")
 
    client, _ := consulapi.NewClient(consulapi.DefaultConfig())
    client.Agent().ServiceDeregister(
        fmt.Sprintf("%s-%s", serviceName, podIP),
    )
}

Connection Draining

Implement sophisticated connection draining:

func handleShutdown(cmd *exec.Cmd) {
    log.Println("Shutdown initiated, draining connections")
 
    // Mark unhealthy
    healthCheckFile := "/tmp/healthy"
    os.Remove(healthCheckFile)
 
    // Allow connections to drain
    time.Sleep(15 * time.Second)
 
    // Send SIGTERM to application
    cmd.Process.Signal(syscall.SIGTERM)
 
    // Wait for graceful shutdown
    done := make(chan error, 1)
    go func() {
        done <- cmd.Wait()
    }()
 
    select {
    case <-done:
        log.Println("Application exited gracefully")
    case <-time.After(20 * time.Second):
        log.Println("Forcing shutdown")
        cmd.Process.Kill()
    }
}

The wrapper coordinates connection draining, health check status, and graceful shutdown timing.

Integration with Deployments

The Commandlet pattern enhances deployment reliability:

Rolling Deployment - Commandlets ensure new Pods don’t receive traffic until fully initialized and old Pods complete all work before termination:

spec:
  strategy:
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    spec:
      initContainers:
        - name: inject-commandlet
          image: lifecycle-wrapper:v1
          # ... injection logic
 
      containers:
        - name: app
          command: ["/shared/wrapper"]
          args: ["/app"]
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080

The wrapper coordinates with readiness probes to prevent premature traffic routing.

Zero-Downtime Deployments - Commandlets implement precise connection draining and startup coordination, enabling truly zero-downtime deployments even for applications that don’t natively support graceful lifecycle management.

Common Pitfalls

PID 1 Complexity - Running as PID 1 requires proper signal handling and zombie reaping:

// Bad - doesn't reap zombies
cmd := exec.Command(appBinary)
cmd.Start()
cmd.Wait()
 
// Good - reap zombie processes
func reapZombies() {
    for {
        var status syscall.WaitStatus
        pid, _ := syscall.Wait4(-1, &status, syscall.WNOHANG, nil)
        if pid <= 0 {
            break
        }
    }
}
 
go func() {
    ticker := time.NewTicker(1 * time.Second)
    for range ticker.C {
        reapZombies()
    }
}()

Consider using tini or dumb-init as the wrapper if full init system functionality is needed.

Signal Propagation - Ensure signals reach the application process:

# Bad - shell doesn't propagate signals
/app &
wait
 
# Good - trap and forward
trap 'kill -TERM $APP_PID' TERM
/app &
APP_PID=$!
wait $APP_PID

Wrapper Failures - Wrapper bugs affect all Pods using it. Test thoroughly and version carefully:

initContainers:
  - name: inject-commandlet
    image: commandlet-wrapper:v1.2.3 # Pin version

Resource Overhead - The wrapper consumes additional CPU and memory, though typically minimal. Account for this in resource profiles.

Observability

Commandlets enhance observability by logging lifecycle events:

func main() {
    log.Printf("Commandlet version %s starting", version)
    log.Printf("Application command: %v", os.Args[1:])
 
    startTime := time.Now()
    runPreStartup()
    log.Printf("Pre-startup completed in %v", time.Since(startTime))
 
    cmd := exec.Command(os.Args[1], os.Args[2:]...)
    cmd.Start()
    log.Printf("Application started with PID %d", cmd.Process.Pid)
 
    // ... signal handling
 
    shutdownStart := time.Now()
    handleShutdown(cmd)
    log.Printf("Shutdown completed in %v", time.Since(shutdownStart))
}

These logs appear in container logs, providing insight into lifecycle timing without instrumenting the application.

Best Practices

Version Commandlets - Treat wrappers as infrastructure code. Version, test, and deploy them deliberately:

initContainers:
  - name: inject-commandlet
    image: my-registry/commandlet-wrapper:v2.1.0

Keep Wrappers Simple - Commandlets should handle lifecycle concerns, not business logic. Complex wrappers become maintenance burdens.

Test Signal Handling - Verify the wrapper correctly handles and forwards signals:

# Test locally
docker run --rm -it wrapper-test /wrapper /app &
sleep 5
kill -TERM $!  # Verify graceful shutdown

Provide Configuration - Make wrappers configurable via environment variables:

shutdownTimeout := getEnvDuration("SHUTDOWN_TIMEOUT", 30*time.Second)
drainDuration := getEnvDuration("DRAIN_DURATION", 15*time.Second)

Document Behavior - Clearly document what the wrapper does, especially signal handling and timeout behavior.

Consider Alternatives - For simple cases, PostStart/PreStop hooks may suffice. For complex multi-stage initialization, Init Containers may be clearer.

Use Existing Implementations - Before building custom wrappers, evaluate existing tools:

  • tini - Lightweight init system
  • dumb-init - Simple process supervisor
  • s6-overlay - Full-featured init system
  • Specialized commandlets from your ecosystem

Real-World Examples

Kubernetes Pause Container Pattern

Kubernetes itself uses a commandlet-like pattern with pause containers in Pods. The pause container runs as PID 1, managing namespaces while application containers run as children.

Istio Sidecar Injection

Istio injects Envoy proxies using Init Containers to set up networking, similar to commandlet injection. The proxy intercepts traffic without application code changes.

Application Performance Monitoring

APM tools (New Relic, DataDog) often use commandlet-like injection to instrument applications:

initContainers:
  - name: inject-apm
    image: apm-agent:v1
    command: ["cp", "/agent", "/shared/agent"]
    volumeMounts:
      - name: agent
        mountPath: /shared
 
containers:
  - name: app
    command: ["/shared/agent"]
    args: ["exec", "/app"]
    volumeMounts:
      - name: agent
        mountPath: /shared

The Commandlet pattern provides fine-grained managed lifecycle control, enabling sophisticated startup and shutdown behavior without modifying application code. It represents the most advanced lifecycle management technique, offering stronger guarantees and greater reusability than PostStart hooks, PreStop hooks, or Init Containers alone.

Recent Writing

Recent Notes

Graph View

Backlinks