Debug Kubernetes Container

Launch an ephemeral debug container attached to a running pod for interactive debugging.

When to Use

•Pod is failing but you need to inspect it live
•Need debugging tools (curl, netstat, tcpdump) not available in production container
•Want to debug without modifying production container image
•Investigating network, process, or filesystem issues in running pod

Pre-flight Checks

Authentication and Context

bash

# Check kubectl context
kubectl config current-context || {
  echo "No kubectl context. Run: kubectl config use-context <context>"
  exit 1
}

# Show current context
echo "K8s Context: $(kubectl config current-context)"

Workflow

1. Launch Debug Container

Basic debug container:

bash

# Attach ephemeral debug container to running pod
kubectl debug -it <pod-name> \
  --image=nicolaka/netshoot \
  --namespace=<namespace>

With specific container target (for multi-container pods):

bash

# Target specific container in pod
kubectl debug -it <pod-name> \
  --image=nicolaka/netshoot \
  --target=<container-name> \
  --namespace=<namespace>

With custom image:

bash

# Use custom debug image
kubectl debug -it <pod-name> \
  --image=ubuntu:latest \
  --namespace=<namespace>

2. Common Debug Commands Inside Container

Once inside the debug container, use these commands:

Network debugging:

bash

# Test HTTP endpoints
curl http://localhost:8080/health
curl -v http://database-service:5432

# Check listening ports
netstat -tulpn

# Capture network traffic
tcpdump -i any port 8080

# DNS resolution
nslookup database-service
dig +short database-service.default.svc.cluster.local

Process inspection:

bash

# List processes
ps aux

# Monitor processes
top

# Check process details
ps aux | grep api-gateway

File system inspection:

bash

# Check application directory
ls -la /app

# View config files
cat /app/config.yaml

# Check environment variables
env | grep API

# Check mounted volumes
df -h
mount | grep /app

Container metadata:

bash

# Check container env
printenv

# View pod labels (if tools available)
cat /etc/podinfo/labels

3. Document and Exit

After debugging:

•Document findings in investigation notes or Linear ticket
•Exit debug container: exit or Ctrl+D
•Debug container is automatically removed

Recommended Debug Images

nicolaka/netshoot (Recommended for network debugging):

•Includes: curl, dig, netstat, tcpdump, wget, nc, nslookup
•Best for: Network connectivity, DNS, HTTP debugging
•Size: ~300MB

busybox:

•Includes: Basic Unix utilities
•Best for: Minimal debugging, file system inspection
•Size: ~5MB

ubuntu:latest:

•Includes: Full package manager (apt)
•Best for: Installing custom tools on-the-fly
•Size: ~70MB

alpine:latest:

•Includes: Package manager (apk)
•Best for: Lightweight debugging with custom tools
•Size: ~7MB

Tips

•Use --target when debugging multi-container pods
•Debug containers share process namespace with target container
•Debug containers have access to target container's filesystem
•Use --copy-to=<new-pod-name> to create a copy of the pod for debugging
•Debug containers are ephemeral and removed on exit
•Check pod status before debugging: kubectl get pod <pod-name> -n <namespace>

Limitations

•Requires Kubernetes 1.18+ (ephemeral containers feature)
•Some clusters may have PodSecurityPolicy restricting debug containers
•Shared PID namespace may not be available in all environments