Using Multiprocessing Pool From Celery Task Raises Exception

FOR THOSE READING THIS: I have decided to use RQ instead which doesn't fail when running code that uses the multiprocessing module. I suggest you use that. I am trying to use a mul

Solution 1:

This is a known issue with celery. It stems from an issue introduced in the billiard dependency. A work-around is to manually set the _config attribute for the current process. Thanks to user @martinth for the work-around below.

from celery.signals import worker_process_init
from multiprocessing import current_process

@worker_process_init.connectdeffix_multiprocessing(**kwargs):
    try:
        current_process()._config
    except AttributeError:
        current_process()._config = {'semprefix': '/mp'}

The worker_process_init hook will execute the code upon worker process initialization. We simply check to see if _config exists, and set it if it does not.

Solution 2:

A quick solution is to use the thread-based "dummy"multiprocessing implementation. Change

from multiprocessing import Pool  # or whatever you're using

to

from multiprocessing.dummyimportPool

However since this parallelism is thread-based, the usual caveats (GIL) apply.