2 years ago

#58368

test-img

olamundo

What could be causing a multithreaded/multiprocessed program to deadlock when there are no cyclic locks?

I have a piece of python code that uses multiprocessing. It runs for a few hours and then gets stuck, in what I assume is a deadlock. The thing is, I can't see how could I have a deadlock as my process is sequential and there's no resource competition:

  • main launches all subprocesses and threads
  • a threading.Threadreader (spawned from main) runs in a loop, reads data and puts in a synced queue (mp.Queue) q_in
  • many multiprocessing.Processors run in a loop, calling q_in.get(), when they get some data they start processing it, and when each of them is done, they put the processed data in q_out
  • another threading.Thread spawned from main runs in a loop, calls q_out.get() and when receives processed data writes it to disk.
  • Main joins all processes/threads and waits for them to finish.

When the reader has no more data to read, it puts a special termination flag in the queue, and for any other process/thread, when they read that flag from an incoming queue they put it in an outgoing one and exit.

As you can see, the locking of queues goes in one direction and is not cyclic, so it doesn't seem like there can be a deadlock. However no error or exception is reported, and I am certain this has to do with the asynchrony of operations. What am I missing? I am communicating solely via the queues, which are synchronous.

python

multithreading

deadlock

0 Answers

Your Answer

Accepted video resources