Realtime Postfix stats aggregator

With a user base as large as Disqus‘, there is a ton of new comment and reply notification emails to send. Indubitable there is a user that accidentally (maybe even purposefully) subscribes to extremely hot threads. When they start receiving a stream of emails, their email provider doesn’t appreciate the spike in traffic. They usually show their annoyance by temporarily blocking and then rate limiting our Postfix instance from relaying email to that inbox.

Unfortunately, the only decent Postfix stats aggregators I could find were written in Perl (pflogsumm.pl) and consumed log files for some ad-hoc stats generation. Though, I was quite lazy after trying a few variations and finding the same Perl tools over and over so please leave a comment about your favorite Postfix stats aggregator.

I really didn’t need anything too fancy so I decided to take a stab at it myself. After thinking about it for a few minutes, I decided to try out using Python threading to have a small pool of workers run some regex on a queue of lines from syslog. All the stats are then gathered in a dictionary and either spit out to stdout or there is a VERY simple TCP server thread listening for ‘stats’ or ‘prettystats’ to dump the current cumulative stats as a JSON dictionary. Full readme can be found on the Github page. Best part, it requires no 3rd party libraries.

postfix-stats on Github
postfix-stats on PyPi

View more to see example output… Continue reading

django-celery, eventlet and debugging blocking

I recently wrote a couple Celery tasks that are purely IO bound. So instead of using the default multiprocessing execution pool, I used the Eventlet execution pool. With just a small change in Celery settings, I was off to the races.

Wrong! After some amount of time, it just sits at 100% CPU and no longer processes tasks. Unfortunately, Celery calls monkey_patch a little late when coupled with Django. Django does some magic of its own to various components so monkey_patch needs to be called before Django does any initialization. After a little digging, I found I can just set an environment variable to prevent Celery from doing the monkey patching and at the same time use it to signal manage.py to call monkey_patch before the initialization my Django app.

Continue reading

OS X System Equalizer

I don’t have a music collection to listen to using iTunes. I listen to music through online streaming services like Pandora and Slacker. The downside to this is that Google Chrome doesn’t have a built-in audio equalizer like iTunes. I have a decent pair of headphones, Audio-Technica ATH-M50, and the default line-out from an Apple desktop machine has always sounded a little “empty” to me. Like they slightly tweak it for normal/smaller headphones. Today I finally figured out how to get a global system equalizer for OS X (for free) so I can push the bass up a little to compensate for the “emptiness”.
Continue reading

FiSH module for ZNC

So a bunch of us were sitting around talking one night and decided just for the lulz we would also start using FiSH Encryption in one of our back back channels on IRC. Not everyone that is in the channel are neck-beards and use irssi; some even don’t use XChat. To easy the transition for everyone, we setup a ZNC server to be a bouncer that also handles the FiSH encryption (yes this can defeat the purpose of FiSH, that’s why everyone is required to use SSL when connecting.)

As we were setting up, those of us who already were using FiSH couldn’t tell who was talking encrypted and who wasn’t. So @noah256 brought up the idea of prefixing encrypted messages. Later he also realized this can be spoofed; so he suggested to also prefix unencrypted messages from expected encrypted targets. At the same time, we realized when trying to talk to people without FiSH installed, it wasn’t really easy to temporarily send unencrypted messages so we also added a prefix to tell the module to not encrypt the message.

I used the fish.cpp from the fish – ZNC wiki page. For now, the behavior I added is hard coded. I hope when I have free time again, to make the prefixes and disable flag to be configurable.

My modified module can be found in my znc-fish repo on GitHub

Continue reading

FreeBSD mfiutil and the Common RAID Format Spec

Recently I have been setting up a few new boxes that have an LSI MegaRAID controller. As of FreeBSD 8.0, there is a much more friendly tool included called mfiutil. It saves from having to install sysutils/megacli and using its rather complicated arguments (well, at first, after using it for a little while they make sense). Now, I know I setup a RAID10 that spans 3, 2 disk arrays. However, mfiutil is reporting it as a RAID1 volume.

Continue reading

Allow users to use rabbitmqctl

RabbitMQ is our message queue of choice for Celery. One of its main commands is rabbitmqctl; it lets you manage your rabbit nodes. The command itself is just a shell script that uses sudo -u rabbitmq to run an the Erlang rabbitmqctl program. It has to use sudo so that can it run as the rabbitmq user to use the same Erlang cookie that the broker is running with.

If you want to allow more than root or wheel group members to use the command, just use this sudoers privilege specification:

ALL=(rabbitmq) SETENV: NOPASSWD: /usr/local/bin/erl

(rabbitmq) assumes your RabbitMQ broker runs as the ‘rabbitmq’ user. It also assume you have Erlang installed in /usr/local

memcache-top patch to see curr_items

I stumbled upon memcache-top in the Homebrew Formula repository. It’s a neat little Perl application that aggregate stats from a memcached cluster and displays it in a nice little table.

I like to use: memcache-top --servers='server1,server2,server3' --commands --sleep=1

My only qualm is it doesn’t display the curr_items stat. So I wrote a little patch for v0.6 that will add an ITEMS column to the table.

Get the patch from the issue I filed.

MySQL 5.5 Homebrew Formula

I recently switched to a 27″ iMac workstation running Snow Leopard from Ubuntu 10.10 (Maverick Meerkat). The resident Mac fanbois suggested Homebrew over Fink and MacPorts. At first I didn’t heed their warnings and went with MacPorts (I do love the FreeBSD ports system of which MacPorts is based on). However, when it decided to start setting up its own environment I understood the warnings. So I scrapped that and went with Homebrew.

Small issue with Homebrew is it seems to want to stick to ‘Stable’ releases. I really didn’t want to use MySQL 5.1 since it’s a snail compared to MySQL 5.5. Not having an ultra-fast storage array in my iMac this would become an issue with all the work I have to do. So I set out into Ruby land and banged out my first Homebrew formula.

Since it hasn’t been pulled into the official repo, here’s the latest commit: MySQL 5.5 Homebrew Formula

Mainline Homebrew now has 5.5.x as the default MySQL version

Updated: Oracle released MySQL 5.5.8; the first GA release of the 5.5 series.

Resources:
Homebrew Formula cookbook
Building MySQL [5.5] from Source

How to dump memcache key/value pairs fast

Recently we migrated our memcached cluster to a new larger one. This needed to be done mostly for reliability and speed, but it’s also nice being able to have access to new stats like ‘reclaimed’ in 1.4.5.

We decided against migrating data from cluster to cluster because there are long-expire keys we no longer care about. Instead, we just ran some scripts to fill the new cluster with new data. After the scripts finished and the new cluster ran in production for a couple days, it only contained ~70% the data the old cluster possessed. We were curious what that old data was, so we decided to dump it and analyze.
Continue reading

Apache Cassandra FreeBSD rc.d script

We are a FreeBSD shop, so naturally we run our Apache Cassandra cluster on it. Unfortunately, there isn’t an rc.d script for it. Using cassandra/bin/cassandra -p /var/run/cassandra.pid has been working fine without any grief. However, it would be nice to utilize the rc system built into FreeBSD. I took the Tomcat 6.0 rc script that it is in the port system (www/tomcat6) and adapted it for cassandra.

Continue reading