Kubernetes¶

Kubernetes reconciles declared resources toward desired state across a cluster. It is valuable for operating many containerized workloads with shared scheduling and platform controls. It also introduces a distributed control plane, networking, storage, policy, and upgrade burden; use a simpler runtime when those capabilities are unnecessary.

Mental Model¶

Clients submit resource objects to the API server. Controllers observe desired and current state and take actions. The scheduler assigns unscheduled Pods to nodes; each node's kubelet and container runtime start and monitor containers. Cluster state is stored in etcd.

Reconciliation is asynchronous. An accepted object is not proof that the workload is ready, reachable, or healthy.

Core Resources¶

Pod: smallest deployable unit; one or more tightly coupled containers sharing network and selected storage.
Deployment: manages replicated, replaceable Pods and rolling updates.
StatefulSet: stable identities and ordered lifecycle for workloads that need them.
DaemonSet: places a Pod on selected nodes.
Job/CronJob: finite or scheduled work.
Service: stable discovery and virtual access to selected endpoints.
Ingress/Gateway: routes external traffic through an implementation.
ConfigMap/Secret: configuration objects; a Secret is not automatically encrypted or safely exposed.
PersistentVolumeClaim: requests storage through the cluster's storage system.
Namespace: naming and policy scope, not a complete security boundary.

Use controllers rather than creating standalone Pods. The current Kubernetes concepts documentation is authoritative for resource behavior.

Minimal Deployment and Service¶

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels: {app: web}
  template:
    metadata:
      labels: {app: web}
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: web
          image: registry.example.com/web@sha256:REPLACE_ME
          ports: [{name: http, containerPort: 8080}]
          resources:
            requests: {cpu: 100m, memory: 128Mi}
            limits: {memory: 256Mi}
          readinessProbe:
            httpGet: {path: /ready, port: http}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: ["ALL"]}
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector: {app: web}
  ports: [{port: 80, targetPort: http}]

Replace the image digest and tune every resource and probe from measurement. Validate manifests against the cluster version before deployment.

Pods and Lifecycle¶

Containers in a Pod share an IP address and port space. Put containers together only when they need the same lifecycle and close local communication. Sidecars add resource, startup, shutdown, and observability coupling.

On termination, Kubernetes marks the Pod for deletion, runs configured lifecycle behavior, sends termination signals, and later force-kills remaining processes after the grace period. Applications must stop accepting work, drain, and exit within that budget. Test rollouts with real long-lived requests and jobs.

Health Probes¶

startup probe: allows slow initialization before other probes run;
readiness probe: controls whether the Pod should receive Service traffic;
liveness probe: restarts a process that cannot recover itself.

Do not make liveness depend on every external service; an upstream outage could restart the whole fleet and worsen the incident. Keep probes cheap and set thresholds from observed startup and failure behavior. See the official probe guidance.

Requests, Limits, and Scheduling¶

The scheduler uses resource requests for placement. CPU limits throttle; memory limits can lead to termination. Missing or unrealistic requests cause contention and poor bin packing.

Use affinity, topology spread, taints, tolerations, and priority only for explicit placement or availability requirements. Replicas on one node or failure zone are not resilient to that domain's loss. Pod disruption budgets limit voluntary disruption but do not guarantee availability during node failure.

Networking¶

Each Pod normally receives a routable cluster IP. Services select Pods by labels and provide stable discovery. NetworkPolicy can restrict allowed traffic only when the installed network implementation enforces it.

Define ingress and egress intentionally, secure traffic at trust boundaries, and preserve application authorization. A Service does not wait for database readiness, provide end-to-end retries, or authenticate callers.

Configuration and Secrets¶

Separate deploy-time configuration from images. ConfigMap and Secret updates do not produce identical application behavior across environment variables, mounted files, and reload mechanisms; design and test the chosen rollout.

Enable encryption at rest for sensitive cluster data, restrict API access with RBAC, avoid leaking secrets into command arguments or logs, and prefer short-lived external identity or secret-store integration where available. Base64 encoding is not encryption.

Storage and Stateful Workloads¶

Persistent volumes outlive individual Pods according to storage and reclaim policy. Understand access mode, topology, snapshots, restore, expansion, and failure behavior of the actual driver.

Kubernetes can restart a database process; it does not automatically provide application-consistent backup, replication correctness, or tested recovery. Prefer a managed data service unless operating the stateful system is an intentional competency.

Security Baseline¶

use least-privilege service accounts and RBAC;
enforce Pod Security Standards appropriate to each namespace;
run as non-root, prevent privilege escalation, drop capabilities, and use seccomp;
avoid privileged Pods, host namespaces, host paths, and broad node access;
restrict network traffic and image registries;
pin and verify images, scan continuously, and patch nodes and the control plane;
isolate untrusted tenants with stronger boundaries than namespaces alone;
audit API access and protect administrative credentials.

Admission policy can reject unsafe resources, but policy must be tested during upgrades and emergency operations.

Rollouts and Availability¶

A rolling update temporarily runs old and new versions together. Maintain API, event, and schema compatibility across that interval. Set rollout surge/unavailable values according to capacity and availability needs.

kubectl rollout status deployment/web
kubectl rollout history deployment/web
kubectl rollout undo deployment/web

Rollback cannot reverse an incompatible data migration or external side effect. Prefer expand–migrate–contract changes and progressive traffic exposure for high-risk releases.

Autoscaling¶

Horizontal Pod Autoscaling changes replica count from observed metrics. Cluster autoscaling supplies or removes nodes; these loops operate at different speeds. Scaling cannot fix a saturated database, hot key, unbounded queue, or poor request limit.

Choose a metric tied to work, give the application realistic requests, preserve headroom, and load-test scale-up delay. Scale consumers using backlog age or work rate when CPU is not the constraint.

Observability and Debugging¶

Start from the user symptom and declared resource:

kubectl get deployment,pods,service
kubectl describe pod <pod>
kubectl logs <pod> -c <container> --previous
kubectl get events --sort-by=.metadata.creationTimestamp
kubectl get endpointslices -l kubernetes.io/service-name=web
kubectl auth can-i get secrets --as=<identity>
kubectl rollout status deployment/web

Inspect status conditions, events, previous container logs, exit reason, resource use, probes, endpoints, policy, DNS, and node health. Do not treat interactive edits as a durable fix; change versioned configuration and redeploy.

Monitor control-plane health, node and Pod saturation, scheduling delay, restart reasons, rollout progress, request outcomes, storage, DNS, and certificate expiry. Correlate application telemetry with namespace, workload, Pod, node, zone, and version without creating unbounded label cardinality.

Backup, Upgrade, and Ownership¶

Back up and test restoration of cluster state and application data independently. Managed control planes do not back up every workload's data. Document cluster, node, network, storage, policy, and add-on upgrade order and supported version skew.

Every controller, custom resource, webhook, and platform add-on becomes software the team must secure, observe, upgrade, and recover. Add one only when its automation repays that lifecycle cost.

Checklist¶

Is Kubernetes justified over a simpler deployment platform?
Are workloads controller-managed and images immutable?
Are requests, limits, probes, and shutdown behavior measured?
Can old and new versions coexist safely during rollout?
Are identity, secrets, network, and runtime privileges constrained?
Are state, backups, and restoration owned and tested?
Can overload and failure be diagnosed from status and telemetry?
Are policies, controllers, and add-ons upgradeable and recoverable?