Skip to content
Rahul Shishodiaon GitHub LinkedIn profile

Troubleshooting Pods and Containers

Debugging playbook

Pod issue: describelogsexec

CrashLoopBackOff: logs → probes → command / args

Service issue: selector → endpoints → Pod labels → throwaway kubectl run debug-pod + wget / nslookup from same namespace

Pending: describe → Events → scheduling / resources / PVC

Drill-down order

  1. kubectl get po: READY, STATUS, RESTARTS
  2. kubectl describe po: Events section
  3. kubectl get events --sort-by=.metadata.creationTimestamp
  4. kubectl logs / logs -p (previous instance)
  5. kubectl exec -it / -c <container>
  6. kubectl debug (ephemeral container for distroless)
  7. kubectl port-forward po/<n> 8080:80
  8. kubectl top po / kubectl top nodes (metrics-server)

Fix patterns

ProblemChecksTypical fix
Deployment CrashLooplogs -p, probes, commandFix command/image; increase probe delay
Service 0 endpointsget endpoints, selector, labelsEdit Service selector or relabel Pods
Ingress 503Service endpoints, backend portUse Service port, fix selector
RBAC forbiddenSA, RoleBinding, Role rulesAdd missing verb/resource or namespace
CronJob no Jobsdescribe cronjob, suspend, scheduleFix schedule or unsuspend
PVC Pendingdescribe pvc, StorageClassUse existing storageClassName or matching PV

Common statuses

STATUSCauseFix
ImagePullBackOff / ErrImagePullBad image name/tag, auth, registryVerify image, pull secret
CrashLoopBackOffApp/command crasheslogs, logs -p, fix command
CreateContainerConfigErrorMissing ConfigMap/Secret, runAsNonRoot + root imagedescribe events
PendingInsufficient resources, PVC, taintsdescribe, check requests/quota
ContainerCreatingVolume mount faildescribe: FailedMount

Logs

kubectl logs <pod> -f
kubectl logs <pod> -c <container>
kubectl logs <pod> -p              # before last restart
kubectl logs <pod> --previous

Networking quick check

kubectl get svc -o wide
kubectl get endpoints <svc>
kubectl get po --show-labels
kubectl run net-debug --image=busybox:1.36.1 --rm -it --restart=Never -- sh

Multi-container / distroless

kubectl logs <pod> -c sidecar
kubectl exec <pod> -c app -- /bin/sh

See kubectl debug and kubectl cp for ephemeral debug containers (--target), --copy-to debug pods, and kubectl cp.

kubectl cp (copy files to/from pod)

kubectl cp mypod:/etc/config/app.conf ./app.conf          # pod → local
kubectl cp ./app.conf mypod:/tmp/app.conf                 # local → pod
kubectl cp mypod:/var/log/app.log ./app.log -c sidecar    # specify container

kubectl top

kubectl top pods                        # all pods, default NS
kubectl top pods -n kube-system
kubectl top pods --sort-by=cpu
kubectl top pods --sort-by=memory
kubectl top nodes
  • Requires metrics-server installed; <unknown> if missing

Exam tips

  • Running ≠ healthy: check readiness + logs
  • describe Events often has exact root cause (e.g. secret "x" not found)
  • Multi-container: always specify -c
  • kubectl debug needs --target=<container> to share process namespace; without it the ephemeral container runs isolated
  • describe tells you what Kubernetes tried; logs tells you what the app did
  • Fix the smallest broken link first: Pod Ready → Service Endpoints → Ingress route