Category Archives: Celery

django-celery, eventlet and debugging blocking

I recently wrote a couple Celery tasks that are purely IO bound. So instead of using the default multiprocessing execution pool, I used the Eventlet execution pool. With just a small change in Celery settings, I was off to the races.

Wrong! After some amount of time, it just sits at 100% CPU and no longer processes tasks. Unfortunately, Celery calls monkey_patch a little late when coupled with Django. Django does some magic of its own to various components so monkey_patch needs to be called before Django does any initialization. After a little digging, I found I can just set an environment variable to prevent Celery from doing the monkey patching and at the same time use it to signal to call monkey_patch before the initialization my Django app.

Continue reading

In Python, thread safe does not mean fork safe

We discovered this in a not so easily reproducible way. Here at Mahalo we use this wonderful distributed task queue called Celery. Upon restarting the celeryd server, rarely (until recently) and only some of the workers would throw MemcachedError during their first task. The cause of the issue would be receiving a “STORED” reponse to a memcached_get (wtf?!) or some other variant of invalid response or an EINPROGRESS (which is the socket is already busy doing something). Now another tidbit, we also use Pylibmc which is a wrapper around the libmemcached C library which we’ve written a custom Django cache interface for.
Continue reading