Debugging playbook
Pod issue: describe → logs → exec
CrashLoopBackOff: logs → probes → command / args
Service issue: selector → endpoints → Pod labels
Pending: describe → Events → scheduling / resources / PVC
Drill-down order
kubectl get po: READY, STATUS, RESTARTSkubectl describe po: Events sectionkubectl get events --sort-by=.metadata.creationTimestampkubectl logs/logs -p(previous instance)kubectl exec -it/-c <container>kubectl debug(ephemeral container for distroless)kubectl port-forward po/<n> 8080:80kubectl top po/kubectl top nodes(metrics-server)
Symptom → cause → debug command
| Symptom | Likely Cause | Debug Command |
|---|---|---|
| CrashLoopBackOff | App crashes on start | kubectl logs <pod> --previous |
| ImagePullBackOff | Wrong image name/tag or missing registry secret | kubectl describe pod <name> |
| Pending | No node fits (resources/taints/affinity) | kubectl describe pod <name> → Events |
| OOMKilled | Container exceeded memory limit | kubectl describe pod → 'OOMKilled' |
| Init:0/1 | Init container not completing | kubectl logs <pod> -c <init-container> |
| 0/1 Running | readinessProbe failing | kubectl describe pod → probe events |
Container logs
| Task | Command |
|---|---|
| Logs from pod (single container) | kubectl logs <pod> |
| Logs from specific container | kubectl logs <pod> -c <container> |
| Logs from crashed/previous container | kubectl logs <pod> --previous |
| Stream logs live | kubectl logs <pod> -f |
| Last 50 lines | kubectl logs <pod> --tail=50 |
| Logs since 1 hour ago | kubectl logs <pod> --since=1h |
| Logs from all pods with label | kubectl logs -l app=webapp --all-containers=true |
| Logs from init container | kubectl logs <pod> -c <init-container-name> |
Monitoring & CLI tools
| Task | Command |
|---|---|
| CPU/memory for pods | kubectl top pod |
| CPU/memory for pods in namespace | kubectl top pod -n <ns> |
| CPU/memory for nodes | kubectl top node |
| Sort pods by CPU | kubectl top pod --sort-by=cpu |
| Sort pods by memory | kubectl top pod --sort-by=memory |
| Recent events (sorted) | kubectl get events --sort-by='.lastTimestamp' |
| Events for specific namespace | kubectl get events -n <ns> |
| Watch events live | kubectl get events -w |
| Pod with node + IP | kubectl get pod -o wide |
| Full pod YAML | kubectl get pod <name> -o yaml |
| Watch pod status | kubectl get pod -w |
| JSONPath: get pod IP | kubectl get pod <name> -o jsonpath='{.status.podIP}' |
| JSONPath: all container images | kubectl get pod <name> -o jsonpath='{.spec.containers[*].image}' |
| Custom columns | kubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName |
Debugging in Kubernetes
| Task | Command |
|---|---|
| Shell into running container | kubectl exec -it <pod> -- /bin/sh |
| Shell into specific container | kubectl exec -it <pod> -c <container> -- bash |
| Run one-off command | kubectl exec <pod> -- cat /etc/config/app.conf |
| Debug with ephemeral container | kubectl debug -it <pod> --image=busybox --target=<container> |
| Copy of pod + debug container | kubectl debug <pod> -it --image=busybox --copy-to=debug-pod |
| Copy file from pod | kubectl cp <pod>:/etc/config/app.conf ./local-app.conf |
| Copy file to pod | kubectl cp ./local.conf <pod>:/etc/config/local.conf |
| Port forward to pod | kubectl port-forward pod/<name> 8080:80 |
| Port forward to service | kubectl port-forward svc/<name> 8080:80 |
Multi-container / distroless
kubectl logs <pod> -c sidecar
kubectl exec <pod> -c app -- /bin/sh
kubectl debug pod/mypod -it --image=busybox:1.36.1 --target=app
kubectl debug pod/mypod -it --image=ubuntu --copy-to=debug-pod --share-processes
Exam gotchas
- Events section at the bottom of
kubectl describe pod: pull errors, scheduling failures, probe failures - CrashLoopBackOff:
--previousto see what killed the container kubectl topneeds Metrics Server; if "Metrics API not available" → server not installedRunning≠ healthy: check readiness + logs- Multi-container: specify
-c
Practice scenario: Troubleshooting
Broken state: frontend 0 ready; backend empty Endpoints; CronJob no Jobs
Solution
Workflow:
get po→describe→logs/logs -p- Empty Endpoints →
describe svcselector vsget po --show-labels - CronJob →
describe cronjob(schedule, suspend, concurrency)
Typical fixes:
kubectl set image deployment/frontend nginx=nginx:1.25.3
# Fix Service selector to match Pod labels
# Fix CronJob schedule or re-apply manifest