Cloud Infrastructure · 10 min read
Kubernetes in Production: What Most Teams Miss Before Going Live
Running Kubernetes in a lab and running it in production are two different disciplines. This is what separates clusters that hold up from clusters that surprise you at the worst time.
The Gap Between Working and Production-Ready
A Kubernetes cluster that runs your application without errors in a staging environment is not a production-ready cluster. It is a cluster that has not been tested under the conditions that production imposes: sustained load, partial failures, node recycling, cluster upgrades, and the operational urgency of a real incident.
Most of the problems we encounter when taking over Kubernetes infrastructure from other teams fall into a predictable set of categories. Not because the original engineers were careless — but because running Kubernetes correctly requires operating knowledge that goes well beyond the official documentation.
Here is the checklist we work through before declaring any cluster production-ready.
Resource Requests and Limits
Every pod must have resource requests and limits defined for CPU and memory. Without requests, the Kubernetes scheduler cannot make informed placement decisions — pods end up on nodes that cannot support them, leading to OOM kills and unpredictable evictions during resource pressure.
Without limits, a misbehaving pod can consume the entire node's resources, causing neighboring pods to fail. This is the most common cause of cascading failure we see in new Kubernetes deployments.
The subtlety is that limits should not be set too conservatively. A pod with a memory limit 20% above its average usage will trigger OOM kills under legitimate load spikes. Memory sizing requires measuring actual usage under realistic load, not guessing.
CPU limits in particular require care. CPU throttling happens when a pod exceeds its limit — it does not get killed, it gets slowed down. This can manifest as latency spikes that are extremely hard to diagnose without the right metrics.
Health Probes: Liveness, Readiness, and Startup
Liveness probes tell Kubernetes when to restart a container. Readiness probes tell Kubernetes when a container is ready to receive traffic. Startup probes give slow-starting containers time to initialize before liveness probes begin.
Getting these wrong produces some of the most frustrating production behaviors: pods that restart themselves during a slow database query (because the liveness probe timed out), traffic routed to pods that are still warming up (because the readiness probe passed too early), or rolling deployments that cannot complete because no pod ever passes readiness.
The rule: readiness probes should verify that the application can actually serve traffic — not just that it is running. Liveness probes should only trigger a restart if the container is genuinely unrecoverable. The failure thresholds and periods must be tuned based on observed startup and response times under load.
Pod Disruption Budgets
Rolling upgrades, node drains, and cluster autoscaler scale-down events all involve voluntarily removing pods from nodes. Without a PodDisruptionBudget, Kubernetes will remove as many pods as necessary to complete the operation, potentially leaving your service with zero healthy replicas.
A PDB with minAvailable: 1 is the minimum for any stateless service. For services with strict availability requirements, define minAvailable or maxUnavailable based on the minimum replica count you need to serve production traffic.
Horizontal Pod Autoscaling — Calibrated, Not Default
HPA based on default CPU thresholds (50%) does not work well for most real applications. Some services are CPU-efficient but memory-intensive. Others are IO-bound and stay at low CPU even under high request volume.
Effective autoscaling requires custom metrics: request rate per pod (from Prometheus), queue depth (from Kafka or SQS), or application-specific metrics exposed via the custom metrics API. KEDA is the standard choice for event-driven autoscaling — it supports Kafka consumer lag, HTTP request rate, and many other sources natively.
Scale-down speed matters. Default scale-down behavior is conservative (5-minute stabilization window). Under traffic spikes that resolve quickly, aggressive scale-down can lead to oscillation. Tune the stabilization windows based on your actual traffic patterns.
Secrets Management
kubectl create secret and YAML-encoded secrets in your git repository are not a secrets management strategy. Base64 encoding is not encryption.
In production, secrets should be injected from an external secrets store: HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault. The External Secrets Operator is the standard Kubernetes-native way to synchronize external secrets into Kubernetes secrets objects, with automatic rotation support.
Never commit Kubernetes secret YAML to a git repository, even if it contains only non-production values. The habit is dangerous.
Observability: The Three Signals You Cannot Skip
Metrics: Prometheus with Kubernetes kube-state-metrics and node-exporter gives you cluster-level visibility. Per-service metrics require instrumentation in the application or a sidecar. Grafana dashboards should include request rate, error rate, and latency (RED) for each service, plus resource saturation metrics.
Logging: Structured logs in JSON format, collected by a DaemonSet log shipper (Fluentd, Fluent Bit, or Promtail), aggregated in Elasticsearch, OpenSearch, or Loki. Logs without structured fields are difficult to query at scale. Set retention policies and index lifecycle management — unbounded log growth is an operational incident waiting to happen.
Tracing: Distributed tracing is the only reliable way to diagnose latency issues that span multiple services. OpenTelemetry is the standard for instrumentation. Jaeger or Tempo for trace storage. Without tracing, a latency regression in one service will take hours to attribute correctly in a multi-service system.
Cluster Upgrade Strategy
Kubernetes minor versions have a one-year support window on most managed services (EKS, GKE, AKS). Falling behind by two versions means you are running software with unpatched CVEs and no access to updated node images.
Cluster upgrades must be tested in a non-production environment first. Control plane upgrades come before node upgrades. Node upgrades should be executed with node drains and PDBs in place — not by terminating nodes directly.
Build upgrade runbooks before you need them. A cluster upgrade at 2am under time pressure, without a documented procedure, is where mistakes happen.
Production Kubernetes is an operational discipline, not a configuration exercise. The checklist above is not exhaustive, but it covers the areas where we most frequently find the gap between "it works" and "it works reliably under the conditions that production actually imposes." That gap is where incidents live.