Beware of using Pylibmc 1.1+ and mod_wsgi

Recently, we attempted to bring up a new web server but to our dismay after it was all said and done, the Django application wouldn’t load. Eventually a “gateway timeout” was thrown with no errors in the wsgi or Apache error log. These types of errors are a pain to debug. What made the mystery even better, was the application ran just fine under the Django dev server!

I don’t think I need to go in depth into our debugging practices since that’s not what this post is really about. So I’m just going to skip ahead a little in the story.

Eventually we determined that by downgrading Pylibmc to 1.0 and libmemcached to 0.38 we could get the application to run fine under Apache. It really doesn’t make sense that upgrading should break “legacy” functionality. So I set out to determine the underlying cause.

After some trial and error with trying different version combinations of pylibmc, libmemcached and mod_wsgi, I finally decided to look at changelogs. Lo and behold, they held the answer I was looking for! (Go figure the easy way would have been faster)

Libmemcached 0.40-0.43 had nothing all that drastic.
Pylibmc 1.1 on the other hand had a change that seemed a little suspicious: “Acquire and release the GIL properly”
Sure enough, after digging into the mod_wsgi documentation I found out that it doesn’t play nice with the Python Simplified GIL State API (which is what pylibmc now uses in the “set” method and that line in the changelog references). It’s even referenced in the Python C-API docuemntation that it’s not supported in sub-interpreters!

The fix outlined by the mod_wsgi documentation says to run the application in the global interpreter by adding this to the configuration:
WSGIApplicationGroup %{GLOBAL}

After adding that directive to the VirtualHost block in our Apache configuration; our Django application ran all just fine and dandy (and saves a little bit of memory). Woo!

For most applications running in the global interpreter is not going to cause any issues. However, according to mod_wsgi documentation this may cause issues when running multiple applications that access the same modules. This fix is definitely YMMV. So test THOROUGHLY!

  • Pingback: Tweets that mention Beware of using Pylibmc 1.1+ and mod_wsgi « Dctr Watson -- Topsy.com()

  • http://toxik.myopenid.com/ Ludvig Ericson

    Hi. I was unaware of this; in fact I didn’t really implement GIL handling myself, so I apologize for this behavior from pylibmc– however, I think it GIL practices could be more clear for Python C API developers, seeing as this isn’t exactly my first C extension and yeah…

    Would using Py_{BEGIN,END}_ALLOW_THREADS fix this issue? From my reading the manual, it seems this would be compatible with multiple interpreters within one process.

    • http://www.dctrwatson.com Doctor Watson

      Hi,

      Using Py_{BEGIN,END}_ALLOW_THREADS would fix the issue. The “get” method implements those macros and there are no issues. The thread doesn’t kludge until a “set” is called.

      I agree the GIL practices are a little convoluted for Python C developers. I honestly did not know about the GIL until I started debugging this issue. (I have not done Python C writing though)

  • http://toxik.myopenid.com/ Ludvig Ericson

    I’m not sure the previous comment’s name was updated to my name, but I’m Ludvig Ericson, the main developer of pylibmc.

  • http://www.daybarr.com/ Day Barr

    For others googling their way here after stumbling into this issue, it has been fixed, but not in a version released to pypi. It would be nice to be able to

    pip install pylibmc==1.2

    but until a new 1.2 release is made, doing

    pip install pylibmc -e git+https://github.com/lericson/pylibmc.git@47cb9ef9ff5c3cf25c9c#egg=Package

    will give you a version built from the source in git (latest as of right now). For reference, the github issue is https://github.com/lericson/pylibmc/issues/19

  • Igor

    Thanks a lot !!!! Great help!