Deadlock In Python Threads

I am trying to implement a simpley portscanner with Python. It works by creating a number of worker threads which scan ports that are provided in a queue. They save the results in

Solution 1:

I don't see anything obviously wrong with your code, but as it stands the break will never be hit - self.to_scan.get() will wait forever rather than raising Queue.Empty. Given that you're loading up the queue with ports to scan before starting the threads, you can change that to self.to_scan.get(False) to have the worker threads exit correctly when all the ports have been claimed.

Combined with the fact that you have non-daemon threads (threads that will keep the process alive after the main thread finishes), that could be the cause of the hang. Try printing something after the to_scan.join() to see whether it's stopped there, or at process exit.

As Ray says, if an exception other than socket.error is raised between self.to_scan.get() and self.to_scan.task_done(), then the join call will hang. It could help to change that code to use a try/finally to be sure:

defrun(self):
    whileTrue:
        try:
            host, port = self.to_scan.get(False)
        except Queue.Empty:
            breaktry:
            s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            try:
                s.connect((host, port))
                s.close()
                self.scanned.put((host, port, 'open'))
            except socket.error:
                self.scanned.put((host, port, 'closed'))
        finally:
            self.to_scan.task_done()

In general, debugging multithreaded processes is tricky. I try to avoid anything blocking indefinitely - it's better to have something crash noisily because a timeout was too short than to have it just stop forever waiting for an item that will never appear. So I'd specify timeouts for your self.to_scan.get, socket.connect and to_scan.join calls.

Use logging to work out the order events are occurring - printing can get interleaved from different threads, but loggers are thread-safe.

Also, something like this recipe can be handy for dumping the current stack trace for each thread.

I haven't used any debuggers with support for debugging multiple threads in Python, but there are some listed here.

Solution 2:

It is likely that not all items on the to_scan queue are consumed and that you're not calling the task_done method enough times to unblock ConnectScanner.

Could it be that an exception is thrown during the runtime of ConnectScan.run that you're not catching and your threads prematurely terminate?