Debugging playbook

Pod issue: describelogsexec

CrashLoopBackOff: logs → probes → command / args

Service issue: selector → endpoints → Pod labels

Pending: describe → Events → scheduling / resources / PVC

Drill-down order

  1. kubectl get po: READY, STATUS, RESTARTS
  2. kubectl describe po: Events section
  3. kubectl get events --sort-by=.metadata.creationTimestamp
  4. kubectl logs / logs -p (previous instance)
  5. kubectl exec -it / -c <container>
  6. kubectl debug (ephemeral container for distroless)
  7. kubectl port-forward po/<n> 8080:80
  8. kubectl top po / kubectl top nodes (metrics-server)

Symptom → cause → debug command

SymptomLikely CauseDebug Command
CrashLoopBackOffApp crashes on startkubectl logs <pod> --previous
ImagePullBackOffWrong image name/tag or missing registry secretkubectl describe pod <name>
PendingNo node fits (resources/taints/affinity)kubectl describe pod <name> → Events
OOMKilledContainer exceeded memory limitkubectl describe pod → 'OOMKilled'
Init:0/1Init container not completingkubectl logs <pod> -c <init-container>
0/1 RunningreadinessProbe failingkubectl describe pod → probe events

Container logs

TaskCommand
Logs from pod (single container)kubectl logs <pod>
Logs from specific containerkubectl logs <pod> -c <container>
Logs from crashed/previous containerkubectl logs <pod> --previous
Stream logs livekubectl logs <pod> -f
Last 50 lineskubectl logs <pod> --tail=50
Logs since 1 hour agokubectl logs <pod> --since=1h
Logs from all pods with labelkubectl logs -l app=webapp --all-containers=true
Logs from init containerkubectl logs <pod> -c <init-container-name>

Monitoring & CLI tools

TaskCommand
CPU/memory for podskubectl top pod
CPU/memory for pods in namespacekubectl top pod -n <ns>
CPU/memory for nodeskubectl top node
Sort pods by CPUkubectl top pod --sort-by=cpu
Sort pods by memorykubectl top pod --sort-by=memory
Recent events (sorted)kubectl get events --sort-by='.lastTimestamp'
Events for specific namespacekubectl get events -n <ns>
Watch events livekubectl get events -w
Pod with node + IPkubectl get pod -o wide
Full pod YAMLkubectl get pod <name> -o yaml
Watch pod statuskubectl get pod -w
JSONPath: get pod IPkubectl get pod <name> -o jsonpath='{.status.podIP}'
JSONPath: all container imageskubectl get pod <name> -o jsonpath='{.spec.containers[*].image}'
Custom columnskubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Debugging in Kubernetes

TaskCommand
Shell into running containerkubectl exec -it <pod> -- /bin/sh
Shell into specific containerkubectl exec -it <pod> -c <container> -- bash
Run one-off commandkubectl exec <pod> -- cat /etc/config/app.conf
Debug with ephemeral containerkubectl debug -it <pod> --image=busybox --target=<container>
Copy of pod + debug containerkubectl debug <pod> -it --image=busybox --copy-to=debug-pod
Copy file from podkubectl cp <pod>:/etc/config/app.conf ./local-app.conf
Copy file to podkubectl cp ./local.conf <pod>:/etc/config/local.conf
Port forward to podkubectl port-forward pod/<name> 8080:80
Port forward to servicekubectl port-forward svc/<name> 8080:80

Multi-container / distroless

kubectl logs <pod> -c sidecar
kubectl exec <pod> -c app -- /bin/sh
kubectl debug pod/mypod -it --image=busybox:1.36.1 --target=app
kubectl debug pod/mypod -it --image=ubuntu --copy-to=debug-pod --share-processes

Exam gotchas

  • Events section at the bottom of kubectl describe pod: pull errors, scheduling failures, probe failures
  • CrashLoopBackOff: --previous to see what killed the container
  • kubectl top needs Metrics Server; if "Metrics API not available" → server not installed
  • Running ≠ healthy: check readiness + logs
  • Multi-container: specify -c

Practice scenario: Troubleshooting

Broken state: frontend 0 ready; backend empty Endpoints; CronJob no Jobs

Solution

Workflow:

  1. get podescribelogs / logs -p
  2. Empty Endpoints → describe svc selector vs get po --show-labels
  3. CronJob → describe cronjob (schedule, suspend, concurrency)

Typical fixes:

kubectl set image deployment/frontend nginx=nginx:1.25.3
# Fix Service selector to match Pod labels
# Fix CronJob schedule or re-apply manifest