Debugging playbook
Pod issue: describe → logs → exec
CrashLoopBackOff: logs → probes → command / args
Service issue: selector → endpoints → Pod labels → throwaway kubectl run debug-pod + wget / nslookup from same namespace
Pending: describe → Events → scheduling / resources / PVC
Drill-down order
kubectl get po: READY, STATUS, RESTARTSkubectl describe po: Events sectionkubectl get events --sort-by=.metadata.creationTimestampkubectl logs/logs -p(previous instance)kubectl exec -it/-c <container>kubectl debug(ephemeral container for distroless)kubectl port-forward po/<n> 8080:80kubectl top po/kubectl top nodes(metrics-server)
Fix patterns
| Problem | Checks | Typical fix |
|---|---|---|
| Deployment CrashLoop | logs -p, probes, command | Fix command/image; increase probe delay |
| Service 0 endpoints | get endpoints, selector, labels | Edit Service selector or relabel Pods |
| Ingress 503 | Service endpoints, backend port | Use Service port, fix selector |
| RBAC forbidden | SA, RoleBinding, Role rules | Add missing verb/resource or namespace |
| CronJob no Jobs | describe cronjob, suspend, schedule | Fix schedule or unsuspend |
| PVC Pending | describe pvc, StorageClass | Use existing storageClassName or matching PV |
Common statuses
| STATUS | Cause | Fix |
|---|---|---|
ImagePullBackOff / ErrImagePull | Bad image name/tag, auth, registry | Verify image, pull secret |
CrashLoopBackOff | App/command crashes | logs, logs -p, fix command |
CreateContainerConfigError | Missing ConfigMap/Secret, runAsNonRoot + root image | describe events |
Pending | Insufficient resources, PVC, taints | describe, check requests/quota |
ContainerCreating | Volume mount fail | describe: FailedMount |
Logs
kubectl logs <pod> -f
kubectl logs <pod> -c <container>
kubectl logs <pod> -p # before last restart
kubectl logs <pod> --previous
Networking quick check
kubectl get svc -o wide
kubectl get endpoints <svc>
kubectl get po --show-labels
kubectl run net-debug --image=busybox:1.36.1 --rm -it --restart=Never -- sh
Multi-container / distroless
kubectl logs <pod> -c sidecar
kubectl exec <pod> -c app -- /bin/sh
See kubectl debug and kubectl cp for ephemeral debug containers (--target), --copy-to debug pods, and kubectl cp.
kubectl cp (copy files to/from pod)
kubectl cp mypod:/etc/config/app.conf ./app.conf # pod → local
kubectl cp ./app.conf mypod:/tmp/app.conf # local → pod
kubectl cp mypod:/var/log/app.log ./app.log -c sidecar # specify container
kubectl top
kubectl top pods # all pods, default NS
kubectl top pods -n kube-system
kubectl top pods --sort-by=cpu
kubectl top pods --sort-by=memory
kubectl top nodes
- Requires metrics-server installed;
<unknown>if missing
Exam tips
Running≠ healthy: check readiness + logsdescribeEvents often has exact root cause (e.g.secret "x" not found)- Multi-container: always specify
-c kubectl debugneeds--target=<container>to share process namespace; without it the ephemeral container runs isolateddescribetells you what Kubernetes tried;logstells you what the app did- Fix the smallest broken link first: Pod Ready → Service Endpoints → Ingress route