2 years ago
#58368
olamundo
What could be causing a multithreaded/multiprocessed program to deadlock when there are no cyclic locks?
I have a piece of python code that uses multiprocessing. It runs for a few hours and then gets stuck, in what I assume is a deadlock. The thing is, I can't see how could I have a deadlock as my process is sequential and there's no resource competition:
- main launches all subprocesses and threads
- a
threading.Thread
reader (spawned from main) runs in a loop, reads data and puts in a synced queue (mp.Queue
)q_in
- many
multiprocessing.Processor
s run in a loop, callingq_in.get()
, when they get some data they start processing it, and when each of them is done, they put the processed data inq_out
- another
threading.Thread
spawned from main runs in a loop, callsq_out.get()
and when receives processed data writes it to disk. - Main
joins
all processes/threads and waits for them to finish.
When the reader has no more data to read, it puts a special termination flag in the queue, and for any other process/thread, when they read that flag from an incoming queue they put it in an outgoing one and exit.
As you can see, the locking of queues goes in one direction and is not cyclic, so it doesn't seem like there can be a deadlock. However no error or exception is reported, and I am certain this has to do with the asynchrony of operations. What am I missing? I am communicating solely via the queues, which are synchronous.
python
multithreading
deadlock
0 Answers
Your Answer