Stop A Background Process In Flask Without Creating Zombie Processes

I need to start a long-running background process with subprocess when someone visits a particular view. My code: from flask import Flask import subprocess app = Flask(__name__)

Solution 1:

As I've suggested in the comments, one of the cleanest and most robust way of achieving this kind of thing in Python is by using celery.

Celery requires a broker transport for messaging, for which rabbitmq is the default, and at least a process with workers running. However, the thing that increases readbility an dmaintanability is that the worker code can co-exist in the same file or files than your server app. You invoke the remote procedures as though it where a simple function call.

Celery can handle retries, post-task events, and lots of other things for free, everything with mature code hardened by years of use in production.

This is your example after re-writting it for use with Celery:

from flask import Flask
from celery import Celery
import subprocess

app = Flask(__name__)
celery_app = Celery("test")

@celery_app.taskdefrun_process():
    subprocess.Popen(["sleep", "5"])

@app.route("/")defindex():
    run_process.delay()
    return"hi\n"if __name__ == "__main__":
    app.run(debug=True, port=8080)

With this code, in a system with the rabbitmq server running with default options (I installed the package, and started the service - no configurations whatsoever. Of course on production you would have to tune that - but if everything is to be on the same server, it may not even be needed.)

With rabbitmq in place, one starts the worker process with a command line like: celery worker -A bla1.celery_app -D (pip install celery on the same virtualenv you have your Flask). Then just launch the flask server and see it working.

Of course this has even more advantages if you are doing more work in Python itself than just calling an external process. It can have access to your database models, and you can perform assynchronous actions that modify objects in there (and eventually trigger responses for the user, as "flash" messages on the user session, or e-mails)

Solution 2:

I've seen a lot of "poor man's parallel processing" using subprocess.Popen and letting it run freely, but that's often leading to zombie problems as you noted.

You could run your process in a thread (in that case, no need for Popen, just use call or check_call if you want to raise an exception if process failed). call or check_call (or run since Python 3.5) waits for the process to complete so no zombies, and since you're running it in a thread you're not blocked.

import threading

defin_background():
    subprocess.call(["sleep", "10"])

@app.route("/")defindex():
    t = threading.Thread(target=in_background)
    t.start()
    return"hi\n"

Note: To wait for thread completion you'd have to use t.join() and for that you'd have to keep a reference on the t thread object.

BTW, I suppose that your real process isn't sleep, or it's not very useful and time.sleep(10) does the same (always in a thread of course!)