How to Update OpenSSH on Mac OS X

Just for kicks, I wanted to try using an ECDSA key for ssh authentication. Unfortunately, the OpenSSH bundled with Mountain Lion (10.8) does not support ECDSA keys (nor can one even be generated with ssh-keygen.) The man pages for ssh-keygen and ssh-agent say they support ECDSA, but this is due to a naive man page generation assuming that since the OpenSSL library supports it, OpenSSH will too. Also, a PCI compliant OpenSSH isn’t bundled with OS X Lion (10.7) or older so this will also be useful for those users as well. Thankfully, Homebrew already has a recipe for installing an up-to-date OpenSSH so most of the work of upgrading is already done.

Continue reading

Simple PyPi Caching Proxy

Now that PyPi is being accelerated by the Fastly caching network, pip/easy_install already are running faster. However, this can be taken a step further by setting up a simple caching proxy. By caching packages locally (on the machine or in your private network), you don’t have to keep hitting Fastly/PyPi to download them. This is especially useful if you are constantly running builds and/or tests: AKA continuous integration.

Continue reading

Accelerate Phabricator with Nginx

Phabricator is an awesome suite of tools open sourced by Facebook and now maintained by Phacility. At Disqus, it’s the central nexus of our engineering team. Since so much of an engineer’s day revolves around using the web interface, I was tasked with trying to optimize our local instance of it. The quickest win was installing and enabling APC per the Installation Guide. Next up, I opened the network tab of Chrome’s developer tools and found that PHP is handling the serving of static assets. Granted, phabricator does set very sane and liberal headers so that browsers will heavily cache all the assets, each browser still needs to obtain them first. To ease the pain of the first load, I setup Nginx to handle caching them as well. This way PHP only has to serve and/or generate assets once and something that’s far better at serving static content, can handle the heavy lifting from then on out.

Continue reading

Nginx Gzip, High Concurrency and Memory

In the upcoming 0.4 release of the nginx-push-stream-module, it will have support for the Nginx Gzip filter. Being able to gzip messages will free up bandwidth and decrease latency when under high load. However, the default deflate settings Nginx uses are not ideal for the high concurrency and small messages that are typically sent with the push-stream module. By default, Nginx may allocate up to a relatively large (264kb) chunk of memory for zlib upfront for every request that supports gzip. This adds up fast when there are thousands of concurrent connections to Nginx.

Continue reading

Cassandra Metrics Graphite Reporter Agent

With the release of Cassandra 1.2, many new metrics were instrumented with Metrics with CASSANDRA-4009. However, getting those metrics into something like Graphite was still a polling process. Metrics does have Reporters that let Java Agents push metrics stored in the registry to various datastores (Graphite, Ganglia, etc.) Currently, this requires writing the agent code, compiling it and loading it into Cassandra. Soon there will be a way to just configure these reporters using metrics-reporters-config with CASSANDRA-4430. For now though, this simple agent will push metrics into Graphite while filtering out some noise.

Datastax has a blog post with a brief outline of how to enable the GraphiteReporter but it doesn’t go into much detail or release any code. This post augments it with the missing pieces.

Continue reading

Jenkins and Phabricator sitting in a tree

We’ve been using Phabricator for just about a year here at Disqus. It was originally created at Facebook and open sourced in Spring 2011. To sum it up using their own words: “Phabricator is a open source collection of web applications which make it easier to write, review, and share source code.” The small team working on it at Phacility (the SaaS company behind Phabricator) is constantly improving it so it’s on a continuous release cycle.

Jenkins has been used for continuous integration testing here for much longer. I’m not exactly sure for how long since it was setup before I started in September 2011. David Cramer has always been pushing for an ideal continuous integration/deployment system (IE here here) so part of my duties has been to improve what we have to achieve that goal (we’re hiring).

Currently, there isn’t a direct CI hook into Phabricator that is as deep as say Github+Travis. However, with a little script and an simple event listener for Arcanist, we can replicate most of that functionality.

Continue reading

GNU Make, double quotes and lists

Our lead operations engineer, Scott, put together a nice system called fpm-recipes using Git, GNU Make and FPM to keep track of how we build DEB packages of various things at Disqus. Instead of each ops engineer having their own way for building packages that are stored in various places (IE: shell history) we now have a centralized and standardized system. No more do we have to ask each other to update a package they maintain or curse ourselves for not saving the steps somewhere organized/accessible.

In no time I was able to get erlang-nox and zeromq recipes written (since they haven’t been updated in Ubuntu 10.04 LTS (Lucid Lynx) in ages). However, when I went back and tried to add their dependencies, things got a little hairy. GNU Make’s foreach function assumes lists “are whitespace-separated words”, so having something like DEPENDS := "libuuid1 (>= 2.16)" really doesn’t work as intended when passing it to foreach. So I wrote a function, quoted_map, that will map another function of a quoted list of strings. In fpm-recipes, it adds the -d and makes sure it’s quote (-d "libuuid1 (>= 2.16)") and adds to the FPM args list.

See the code: Continue reading

mutt and gmail

Per recommendation from a neckbeard friend, Aaron, I set out to try out Mutt as my email client. Since my email is hosted by Gmail, there’s a little extra configuration needed than just setting up an IMAP inbox. Also, since people actually send multimedia emails, I wrote a small patch for Mutt that detects it’s talking to a Gmail IMAP server and adds a couple custom headers to the message, one of which is the permalink to the email so it can be easily opened in a browser if need be. I’m sure I am one of the few that actually like Google Contacts, so I use Goobook for address completion. And no reason to go through all the trouble of setting up Mutt and not setup GPG for signing/encryption too. I am a fan of Ethan Schoonover’s Solarized color scheme, but I prefer a bit more contrast: I modified the Mutt colors Solarized Dark 16 colors for this preference.

Latest versions of my conf/patch can be found at:
mutt conf GitHub repo
mutt gmail patch GitHub repo

Continue reading

limits.conf and daemons on Ubuntu

I recently was setting up a couple ElasticSearch and RabbitMQ instances when I noticed RabbitMQ was still reporting an abysmally low fd limit in its log file at startup. I double checked my /etc/security/limits.conf and sure enough, limits were properly set to 64000. Yet for some reason it was still only seeing a max of 1024.

It turns out that in Ubuntu 10.04, /etc/pam.d/common-session{,-noninteractive} does not contain:

session required

Adding that, solved my issue:

=INFO REPORT==== 1-Feb-2012::00:05:47 ===
Limiting to approx 63900 file handles (57508 sockets)

UPDATE (Wed Apr 18 15:01:17 PDT 2012)
For RabbitMQ 2.8.x, the init script uses start-stop-daemon. Apply this patch:

--- /etc/init.d/rabbitmq-server.old	2012-04-18 21:54:05.852307662 +0000
+++ /etc/init.d/rabbitmq-server	2012-04-18 21:49:17.594182809 +0000
@@ -35,6 +35,8 @@
 set -e
+[ -r /etc/default/${NAME} ] && . /etc/default/${NAME}
 ensure_pid_dir () {
     PID_DIR=`dirname ${PID_FILE}`
     if [ ! -d ${PID_DIR} ] ; then

And then in /etc/default/rabbitmq-server

ulimit -n 65000