Reproducing Deadlock While Using Popen.wait()

From the docs using Popen.wait() may: deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting fo

Solution 1:

It's easy.

Create a process which outputs a lot of text and don't read the output:

p = subprocess.Popen(["ls","-R"],stdout=subprocess.PIPE)
p.wait()

after a while the standard output pipe is full and process is blocked.

It's a deadlock situation because the subprocess cannot write anymore to the output until it's consumed (which is: never), and the python process waits for the subprocess to finish.

To avoid the deadlock, you could use a read line loop:

p = subprocess.Popen(["ls","-R"],stdout=subprocess.PIPE)
for line in p.stdout:
    # do something with the line
p.wait()

communicate also fixes that but also fixes the much trickier case where both output and error streams are redirected to separate streams (in that case, the naive loop above could still deadlock).

Let's suppose you have a compilation process

p = subprocess.Popen(["gcc","-c"]+mega_list_of_files,stdout=subprocess.PIPE,stderr=subprocess.PIPE)

Now you want to get the output from this one, so you do:

output = p.stdout.read()

unfortunately, a lot of errors pop up instead, blocking the error stream while you're reading the output stream: deadlock again.

Try to read error stream instead, and the exact opposite could occur: lots of stdout output blocking your process.

communicate uses multithreading to be able to process output & error streams at the same time and keep them separated, without risk of blocking. Only caveat is that you cannot control the process output line by line / print program output in real time:

p = subprocess.Popen(["gcc","-c"]+mega_list_of_files,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
output,error = p.communicate()
return_code = p.wait()