A process that has stopped responding is a specific kind of frustrating, because it isn't giving you anything. There's no stack trace, no error line, no exit code. Just a command that hasn't printed since 14:02, and a growing suspicion that restarting it will make the problem go away until it happens again next Tuesday.
I'm Ajay, I build PandaStack — a platform where a lot of the running processes are other people's builds and agent-generated code, so 'why is this stuck' is a question I've had to get systematic about. This is the order of operations that works, from cheapest to most invasive. It applies to any Linux environment; the sandbox-specific parts are called out where they differ.
Step 1: What state is it in?
The Linux process state is the single highest-information character available, and it splits your investigation into three completely different paths. Get it first:
# STAT is the column that matters. WCHAN says what the kernel is waiting on.
ps -eo pid,ppid,stat,wchan:24,pcpu,etime,cmd --sort=-pcpu | head -20- R (running) with high CPU — it isn't hung, it's busy. You have an infinite loop or genuinely slow work. Go to step 2.
- S (interruptible sleep) with 0% CPU — it's waiting on something: a socket, a lock, a pipe, a child. This is the most common case. Go to step 3.
- D (uninterruptible sleep) — it's blocked in the kernel, almost always on I/O. Go to the D-state section; nothing you send it will help.
- Z (zombie) — it already exited and its parent never reaped it. The bug is in the parent.
- T (stopped) — something sent it SIGSTOP. Usually a debugger that went away.
The WCHAN column tells you the kernel function it's sleeping in, which is often the answer by itself. futex_wait means a userspace lock. pipe_read means it's waiting for a pipe nobody is writing to. wait_on_page_bit means it's waiting on disk.
Step 2: If it's burning CPU, get a stack
A process at 100% CPU is executing something. You want to know what, and the cheapest way is to sample where it is a few times and look for the line that keeps appearing.
Most runtimes have a built-in way to dump a stack on a signal, and it's much friendlier than a native debugger:
# Python 3.3+: faulthandler dumps tracebacks for every thread on SIGABRT
# if the process enabled it. Otherwise attach with py-spy, which needs
# no cooperation from the process at all.
py-spy dump --pid 1234
# Go: SIGQUIT prints goroutine stacks for every goroutine and exits.
kill -QUIT 1234
# Java: thread dump without stopping the process.
jstack 1234
# Node: send SIGUSR1 to open the inspector, then attach a debugger.
kill -USR1 1234py-spy is worth special mention because it reads the target process's memory from outside — no import, no signal handler, no restart. For a Python process that's stuck right now and can't be modified, it is the single most useful tool available.
If none of those apply, gdb gives you a native backtrace for any process:
# One-shot backtrace of every thread, then detach and leave it running.
gdb -p 1234 -batch -ex "thread apply all bt" 2>/dev/null | head -60Step 3: If it's asleep, find out what it's waiting for
A process in S state with no CPU is blocked on a syscall. strace attaches and shows you which one — and critically, whether it's making any at all.
# Attach to a running process and every thread of it.
strace -f -p 1234
# Nothing at all after a few seconds means it is blocked in a single
# syscall. The last line printed before the silence IS your answer.Read the silence, not the noise. These are the patterns worth recognising immediately:
- futex(...) with no return — waiting on a mutex or condition variable. A deadlock, or a lock held by a thread that's itself blocked.
- read(0, ...) — reading stdin. This is the single most common cause of a stuck build: an installer asked an interactive question and nobody is there to answer it.
- connect(...) or recvfrom(...) — waiting on a network peer that isn't answering. Check whether it's DNS by looking at the socket's destination port.
- wait4(...) — waiting for a child that never exits. Move your attention to the child.
- flock(...) or fcntl(..., F_SETLKW, ...) — waiting on a file lock, usually a package manager lock file held by a dead process.
If strace isn't available, the kernel exposes the same information without attaching:
# Which syscall it's in right now (first field is the syscall number).
cat /proc/1234/syscall
# Kernel stack -- the WCHAN column with more detail.
cat /proc/1234/stack
# For Python/Go/Rust binaries, the userspace stack sometimes shows here.
cat /proc/1234/wchan; echoStep 4: Look at what it has open
File descriptors tell you the shape of the problem when the syscall alone is ambiguous. A hung connect is unhelpful; a hung connect to port 5432 on a host that doesn't resolve is a diagnosis.
# Sockets, files and pipes this process holds open.
ls -l /proc/1234/fd | head -40
# Socket state -- SYN-SENT means the peer never answered the handshake.
ss -tanp | grep 1234
# Is it out of file descriptors? A process at its limit blocks in
# accept() or open() in a way that looks exactly like a hang.
cat /proc/1234/limits | grep 'open files'
ls /proc/1234/fd | wc -lThe file-descriptor exhaustion case is worth checking early because it's common and the symptom is so misleading. A server that leaks sockets doesn't crash — it stops accepting connections and looks frozen.
The D-state case
Uninterruptible sleep means the process is blocked inside the kernel and cannot be signalled. kill -9 will not work, and this surprises people. The process is not ignoring your signal; the kernel will not deliver it until the operation completes.
In practice D state means storage. A slow or stalled disk, a network filesystem whose server went away, or an overloaded host where I/O is queued behind other work. Check whether it's you or the machine:
# Everything currently in D state -- if it's several processes,
# the problem is the device, not your program.
ps -eo pid,stat,wchan:32,cmd | awk '$2 ~ /D/'
# Per-process I/O counters. If these are climbing, it's working,
# just slowly. If they're frozen, something below it is stuck.
cat /proc/1234/ioThere's no clever fix from userspace. Either the I/O completes or you restart the machine. What you can do is stop it recurring: put a timeout on the operation, and if it's a network filesystem, mount it with an interruptible option so a stalled server produces an error rather than an unkillable process.
Doing this inside a sandbox
The tooling above needs two things that constrained runtimes often don't give you: a real /proc, and the ptrace capability that strace and gdb depend on. In a Firecracker microVM you get both, because it's a real kernel with a real userspace rather than a restricted process container. On PandaStack that means you can exec into a live sandbox and run the same commands you'd run on any Linux box:
# Find the stuck process from outside.
pandastack exec $SANDBOX_ID -- \
ps -eo pid,stat,wchan:24,pcpu,etime,cmd --sort=-pcpu
# Attach strace to it. Nothing to enable, no privileged mode needed.
pandastack exec $SANDBOX_ID -- timeout 10 strace -f -p 1234Two practical notes. Wrap strace in timeout so an investigation command doesn't itself hang your terminal. And if you need to look at a hung state repeatedly — comparing before and after a change, or handing it to someone else — snapshot the sandbox first. A snapshot captures the memory and disk of the VM as it is, so you can fork the hung state and poke at a copy while the original sits untouched.
Making it not happen again
- Put a timeout on every external call. Not a generous one — a specific one. The default in most HTTP clients is no timeout at all, which is how a single unresponsive dependency freezes an entire service.
- Redirect stdin from /dev/null in any automated context, so an unexpected prompt fails immediately instead of waiting forever.
- Give long-running jobs an overall deadline, so a hang becomes a failed job with logs rather than a slot occupied indefinitely.
- Log a heartbeat from inside long operations. 'Processing item 400 of 10,000' converts a mystery hang into a known position.
- Alert on duration, not just on errors. A job whose p95 runtime is creeping toward its interval is the early warning that a hang is coming.
The summary
Get the process state first — it splits the problem three ways and costs nothing. Burning CPU means take a stack; py-spy, SIGQUIT and jstack are friendlier than gdb. Asleep means attach strace and read what it's blocked on, remembering that the silence after the last line is the signal. D state means storage, and no signal will help. Check file descriptors when the syscall is ambiguous. And resist the restart until you've looked, because the hung process is a free reproduction of a bug you'd otherwise have to chase.