Dctr Watson http://www.dctrwatson.com The odyssey of a philomath of technology Mon, 02 Sep 2013 18:46:30 +0000 en-US hourly 1 http://wordpress.org/?v=4.1.6 How to Update OpenSSH on Mac OS X http://www.dctrwatson.com/2013/07/how-to-update-openssh-on-mac-os-x/ http://www.dctrwatson.com/2013/07/how-to-update-openssh-on-mac-os-x/#comments Wed, 31 Jul 2013 21:19:04 +0000 http://www.dctrwatson.com/?p=227 Just for kicks, I wanted to try using an ECDSA key for ssh authentication. Unfortunately, the OpenSSH bundled with Mountain Lion (10.8) does not support ECDSA keys (nor can one even be generated with ssh-keygen.) The man pages for ssh-keygen and ssh-agent say they support ECDSA, but this is due to a naive man page generation assuming that since the OpenSSL library supports it, OpenSSH will too. Also, a PCI compliant OpenSSH isn’t bundled with OS X Lion (10.7) or older so this will also be useful for those users as well. Thankfully, Homebrew already has a recipe for installing an up-to-date OpenSSH so most of the work of upgrading is already done.

If you don’t already have Homebrew installed, follow its installation instructions first.

1. First we’ll need to add the system duplicates repository to Homebrew.

    $ brew tap homebrew/dupes

2. Compile and install OpenSSH. I want to use a newer OpenSSL and all its optimizations, which Homebrew will happily provide via an option. Also, to make ssh-agent launchd and keychain compatible, there’s a nice undocumented option to apply the necessary patch before compiling too.

    $ brew install openssh --with-brewed-openssl --with-keychain-support

3. Like the caveat notes when brew finishes, you need to update the launchd plist for ssh-agent to use the new Homebrew binary. By replacing /usr/bin/ssh-agent with /usr/local/bin/ssh-agent

    $ launchctl stop org.openbsd.ssh-agent
    $ launchctl unload -w /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist
    $ sudo vi /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist
    $ launchctl load -w -S Aqua /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist

4. The SSH_AUTH_SOCK env var needs to be updated for any open or new terminal sessions. It’s best to logout/login or restart because we cannot modify the variables in the user session scope that all new processes inherit from. However, if that’s not an option, can do this instead.

    $ export SSH_AUTH_SOCK=$(launchctl getenv SSH_AUTH_SOCK)

5. Generate an ECDSA key

    $ ssh-keygen -t ecdsa -b 521

Once the pub key from your new ECDSA key pair is added to .ssh/authorized_keys on your server(s), should be good to go (assuming OpenSSH on your server also supports ECDSA keys.)

Here are openssl speed runs showing considerable improvements in the newer OpenSSL on a Late-2012 rMBP with a 2.9 Ghz i7 (Ivy Bridge):

OpenSSL 0.9.8x 10 May 2012
built on: Apr 27 2013
options:bn(64,64) md2(int) rc4(ptr,char) des(idx,cisc,16,int) aes(partial) blowfish(ptr2)
compiler: -arch x86_64 -fmessage-length=0 -pipe -Wno-trigraphs -fpascal-strings -fasm-blocks -O3 -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DMD32_REG_T=int -DOPENSSL_NO_IDEA -DOPENSSL_PIC -DOPENSSL_THREADS -DZLIB -mmacosx-version-min=10.6
available timing options: TIMEB USE_TOD HZ=100 [sysconf value]
timing function used: getrusage
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              38143.69k   119545.89k   277327.98k   421224.64k   490718.66k
hmac(md5)        44242.06k   134438.19k   293790.30k   422019.19k   494423.98k
aes-128-cbc     170039.77k   180261.34k   180652.73k   183675.79k   184316.33k
aes-256-cbc     134118.17k   139409.38k   137316.74k   141434.10k   141356.07k
sha256           28026.63k    68129.47k   133706.87k   174197.11k   190001.40k
sha512           17881.04k    70711.63k   145642.06k   237958.34k   291830.79k
                  sign    verify    sign/s verify/s
rsa 4096 bits 0.029424s 0.000394s     34.0   2537.2
                              sign    verify    sign/s verify/s
 224 bit ecdsa (nistp224)   0.0002s   0.0008s   5787.7   1251.5
 256 bit ecdsa (nistp256)   0.0002s   0.0010s   4822.8   1004.9
 521 bit ecdsa (nistp521)   0.0005s   0.0026s   1946.9    380.4
OpenSSL 1.0.1e 11 Feb 2013
built on: Tue Jul 30 01:06:27 PDT 2013
options:bn(64,64) rc4(ptr,char) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              73628.17k   215121.49k   448613.97k   613620.39k   693551.10k
hmac(md5)        50714.32k   161230.70k   382580.65k   575982.93k   687741.69k
aes-128-cbc     641533.32k   687496.81k   701280.68k   702194.35k   699692.69k
aes-256-cbc     469374.78k   495378.58k   499366.40k   501357.57k   501394.24k
sha256           54268.49k   118861.46k   208064.34k   255732.74k   275634.77k
sha512           41191.81k   161831.10k   256644.61k   368996.35k   426789.55k
                  sign    verify    sign/s verify/s
rsa 4096 bits 0.008642s 0.000136s    115.7   7374.0
                              sign    verify    sign/s verify/s
 224 bit ecdsa (nistp224)   0.0001s   0.0001s  14421.9   7391.3
 256 bit ecdsa (nistp256)   0.0001s   0.0003s   8765.6   3839.2
 521 bit ecdsa (nistp521)   0.0004s   0.0008s   2636.1   1205.2

http://www.dctrwatson.com/2013/07/how-to-update-openssh-on-mac-os-x/feed/ 13
npm Registry Caching Proxy http://www.dctrwatson.com/2013/06/npm-registry-caching-proxy/ http://www.dctrwatson.com/2013/06/npm-registry-caching-proxy/#comments Fri, 14 Jun 2013 22:45:08 +0000 http://www.dctrwatson.com/?p=222 Prevent your continuous integration/deployment workflow from pummeling the npm registry’s couchdb. Reduce headaches and errors during npm install.

http://www.dctrwatson.com/2013/06/npm-registry-caching-proxy/feed/ 4
Simple PyPi Caching Proxy http://www.dctrwatson.com/2013/06/simple-pypi-caching-proxy/ http://www.dctrwatson.com/2013/06/simple-pypi-caching-proxy/#comments Fri, 14 Jun 2013 22:28:50 +0000 http://www.dctrwatson.com/?p=221 Now that PyPi is being accelerated by the Fastly caching network, pip/easy_install already are running faster. However, this can be taken a step further by setting up a simple caching proxy. By caching packages locally (on the machine or in your private network), you don’t have to keep hitting Fastly/PyPi to download them. This is especially useful if you are constantly running builds and/or tests: AKA continuous integration.

http://www.dctrwatson.com/2013/06/simple-pypi-caching-proxy/feed/ 0
Accelerate Phabricator with Nginx http://www.dctrwatson.com/2013/05/accelerate-phabricator-with-nginx/ http://www.dctrwatson.com/2013/05/accelerate-phabricator-with-nginx/#comments Fri, 31 May 2013 20:05:39 +0000 http://www.dctrwatson.com/?p=219 Phabricator is an awesome suite of tools open sourced by Facebook and now maintained by Phacility. At Disqus, it’s the central nexus of our engineering team. Since so much of an engineer’s day revolves around using the web interface, I was tasked with trying to optimize our local instance of it. The quickest win was installing and enabling APC per the Installation Guide. Next up, I opened the network tab of Chrome’s developer tools and found that PHP is handling the serving of static assets. Granted, phabricator does set very sane and liberal headers so that browsers will heavily cache all the assets, each browser still needs to obtain them first. To ease the pain of the first load, I setup Nginx to handle caching them as well. This way PHP only has to serve and/or generate assets once and something that’s far better at serving static content, can handle the heavy lifting from then on out.

http://www.dctrwatson.com/2013/05/accelerate-phabricator-with-nginx/feed/ 6
Nginx Gzip, High Concurrency and Memory http://www.dctrwatson.com/2013/05/nginx-gzip-high-concurrency-and-memory/ http://www.dctrwatson.com/2013/05/nginx-gzip-high-concurrency-and-memory/#comments Tue, 14 May 2013 22:30:08 +0000 http://www.dctrwatson.com/?p=198 In the upcoming 0.4 release of the nginx-push-stream-module, it will have support for the Nginx Gzip filter. Being able to gzip messages will free up bandwidth and decrease latency when under high load. However, the default deflate settings Nginx uses are not ideal for the high concurrency and small messages that are typically sent with the push-stream module. By default, Nginx may allocate up to a relatively large (264kb) chunk of memory for zlib upfront for every request that supports gzip. This adds up fast when there are thousands of concurrent connections to Nginx.

Normally, Nginx intelligently allocates memory for a gzip-able request using the content length response header. However, the push-stream module, like the name implies, streams messages to the user: the content length is unknown at the beginning of the subscription. Without this hint, Nginx allocates the maximum amount of memory for the connection (default 264kb.)

Thankfully, Nginx has 2 undocumented settings (gzip_window and gzip_hash) that it uses to calculate the maximum memory to preallocate per request using a formula from zconf.h. These parameters are passed to deflateInit2 and are used by zlib for the size of the history buffer and internal compression state. The history buffer affects compression ratio, while the internal state affects both compression ratio and speed. To go in-depth on what these are used for take a look at LZ77 and LZ78 and Huffman Coding.

There is no “one size fits all” value for these parameters (in the context of tuning them for the push-stream-module.) It’s a trade-off of compression ratio to memory required for the number of concurrent users. To come up with settings that will work most of the time (for our system), I followed the algorithm Nginx uses based on the content-length header. I took a 60s sample of messages (3750 messages) and found the average length to be 1120 and a 85th percentile of 1756. So a gzip_window 2k; would work for approximately 85% of the messages. The algorithm starts with gzip_window and halves it until content-length+262 bytes is not larger the the window. So the default gzip_window 32k; has to be halved 4 times: 32768 >> 4 == 2048. Each time gzip_window is halved, so is gzip_hash (to a minimum of 512.) The default of gzip_hash 64k;1, halved 4 times, becomes gzip_hash 4k;. With these settings, the maximum allocation per connection is 32kb (8192 + (gzip_window << 2) + (gzip_hash << 2)).

Shows memory usage and number of users before and after modifying gzip_window and gzip_hash

Shows memory usage and number of users before and after modifying gzip_window and gzip_hash. Machine has 48G of RAM but only receiving ~40% number users as normal for testing.

I'm not too concerned with the compression ratio since the machines that I'm tweaking these settings on have TenG Ethernet cards. Plenty of bandwidth to spare so aiming for memory efficiency (though still seeing about a 40% reduction in bandwidth with these settings.) However, if I were to be trying to optimize the compression ratio, Nginx does a very good job of exposing internal metrics of zlib.

- $gzip_ratio is available for the overall request compression efficiency (good average for all messages in a session)
- Enabling debug and setting error_log to debug level, Nginx will log everything there is to know about the efficiency of deflate (good for per message compression ratio)

  1. Seems Nginx has an off-by-one error in its calculation. Default memLevel of 8 is actually a hash_size of 32k
http://www.dctrwatson.com/2013/05/nginx-gzip-high-concurrency-and-memory/feed/ 9
Cassandra Metrics Graphite Reporter Agent http://www.dctrwatson.com/2013/04/cassandra-metrics-graphite-reporter-agent/ http://www.dctrwatson.com/2013/04/cassandra-metrics-graphite-reporter-agent/#comments Thu, 25 Apr 2013 21:43:33 +0000 http://www.dctrwatson.com/?p=194 With the release of Cassandra 1.2, many new metrics were instrumented with Metrics with CASSANDRA-4009. However, getting those metrics into something like Graphite was still a polling process. Metrics does have Reporters that let Java Agents push metrics stored in the registry to various datastores (Graphite, Ganglia, etc.) Currently, this requires writing the agent code, compiling it and loading it into Cassandra. Soon there will be a way to just configure these reporters using metrics-reporters-config with CASSANDRA-4430. For now though, this simple agent will push metrics into Graphite while filtering out some noise.

Datastax has a blog post with a brief outline of how to enable the GraphiteReporter but it doesn’t go into much detail or release any code. This post augments it with the missing pieces.

As I’m not a Java engineer, I tried to do the bare minimum to get this working. Please submit pull requests to improve it!

1. Clone: git://github.com/disqus/graphite-reporter-agent.git

2. Modify com/disqus/metrics/reporter/GraphiteReporterAgent.java with the host and port of your carbon cache or relay (this should probably be properties or javaagent options but felt lazy due to CASSANDRA-4430).

These were obsoleted by Nikola Petrov by adding Maven support in Pull Request #1

3. Get metrics-graphite 2.0.3: wget http://search.maven.org/remotecontent?filepath=com/yammer/metrics/metrics-graphite/2.0.3/metrics-graphite-2.0.3.jar -O /usr/share/java/metrics-graphite-2.0.3.jar

4. Compile the agent: javac -cp /usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/java/metrics-graphite-2.0.3.jar com/disqus/metrics/reporter/GraphiteReporterAgent.java

5. Make the jar: jar -cfM graphite-reporter.jar .

3. Run: mvn clean package dependency:copy-dependencies

4. Move the jars somewhere Cassandra can load it:
mv target/graphite-reporter-agent-1.0-SNAPSHOT.jar /usr/share/java/
mv target/dependency/metrics-graphite-2.0.3.jar /usr/share/java/

5. Tell Cassandra to start the agent and add metrics-graphite to the classpath.
In Ubuntu, edit /etc/default/cassandra:

JVM_EXTRA_OPTS="$JVM_EXTRA_OPTS -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar"

6. Restart cassandra

7. ???

8. PROFIT!!!!

You should now start seeing stats under cassaandra.$(hostname -s) in your Graphite instance. The current configuration filters out metrics from Connection type, per-client Streaming metrics and the system, system_auth and system_traces column families.

http://www.dctrwatson.com/2013/04/cassandra-metrics-graphite-reporter-agent/feed/ 3
Jenkins and Phabricator sitting in a tree http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/ http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/#comments Wed, 30 Jan 2013 00:16:05 +0000 http://www.dctrwatson.com/?p=185 We’ve been using Phabricator for just about a year here at Disqus. It was originally created at Facebook and open sourced in Spring 2011. To sum it up using their own words: “Phabricator is a open source collection of web applications which make it easier to write, review, and share source code.” The small team working on it at Phacility (the SaaS company behind Phabricator) is constantly improving it so it’s on a continuous release cycle.

Jenkins has been used for continuous integration testing here for much longer. I’m not exactly sure for how long since it was setup before I started in September 2011. David Cramer has always been pushing for an ideal continuous integration/deployment system (IE here here) so part of my duties has been to improve what we have to achieve that goal (we’re hiring).

Currently, there isn’t a direct CI hook into Phabricator that is as deep as say Github+Travis. However, with a little script and an simple event listener for Arcanist, we can replicate most of that functionality.

Jenkins Example Diff

First things, first. Jenkins needs an account in Phabricator to perform actions on the differentials. Make sure the account has the role of System Agent. This keeps it from polluting the user autocomplete in various forms and allows administrative access to its conduit certificate.

Create Jenkins Phab User

You will need to install Arcanist wherever you plan to have Jenkins run these Phabricator builds (master and/or slaves nodes.) However, it doesn’t need the custom PhutilEventListener described below. Instead of running arc install-certificate on each node, it’s easier to just distribute the .arcrc with your Phabricator instance’s conduit uri, Jenkin’s Phabricator username and conduit certificate. Make sure to place .arcrc in Jenkin’s user’s home directory. A good way to test you have arc installed and configured properly is by running arc call-conduit user.whoami from the Jenkins node script console to double check it’s the right user. Make sure to change the conduit-uri to your instance.

Prettyfied response:

To notify Jenkins to do builds on differentials, we need a custom PhutilEventListener. It’s a super simple class with only a couple configuration options. The Arcanist user guide has instructions on how to Create a libphutil library.

Sample .arcconfig

Last but not least, setting up the Jenkins job. As there are numerous ways that jobs can be configured, I’m just going to outline a crude one that can be adapted to your own style. Your Jenkins installation will need at least the PostBuildScript Plugin and Python Plugin.

Since arc diff is going to be kicking off this job, it needs to be a parameterized build that takes a String Paramter DIFF_ID so it knows which patch to run.

Jenkins Parameterized Build

As we use Git for SCM at Disqus, setup the job to follow origin/master and under advanced, enable Prune remote branches before build and Clean after checkout to keep the repo tidy.

Jenkins SCM

There won’t be automatic build triggers since we’re using the event listener to trigger builds on arc diff.

Below is the shell script you can drop into an “Execute shell” build step that will handle applying the diff. This should be the first build step.

Now to post back to Phabricator with the results of the build. This needs to be added to the Post-build Actions. Otherwise, the results of the build won’t be available yet from the Jenkins API. Add “[PostBuildScript] – Execute a set of scripts” action with a build step of “Execute Python Script” of the following script (make sure Python-phabricator is in the PYTHON_PATH):

Huge thanks to Zameer Manji who interned with us Summer 2012 and was instrumental in getting this working!

http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/feed/ 9
GNU Make, double quotes and lists http://www.dctrwatson.com/2012/06/gnu-make-double-quotes-and-lists/ http://www.dctrwatson.com/2012/06/gnu-make-double-quotes-and-lists/#comments Tue, 19 Jun 2012 00:52:23 +0000 http://www.dctrwatson.com/?p=118 Our lead operations engineer, Scott, put together a nice system called fpm-recipes using Git, GNU Make and FPM to keep track of how we build DEB packages of various things at Disqus. Instead of each ops engineer having their own way for building packages that are stored in various places (IE: shell history) we now have a centralized and standardized system. No more do we have to ask each other to update a package they maintain or curse ourselves for not saving the steps somewhere organized/accessible.

In no time I was able to get erlang-nox and zeromq recipes written (since they haven’t been updated in Ubuntu 10.04 LTS (Lucid Lynx) in ages). However, when I went back and tried to add their dependencies, things got a little hairy. GNU Make’s foreach function assumes lists “are whitespace-separated words”, so having something like DEPENDS := "libuuid1 (>= 2.16)" really doesn’t work as intended when passing it to foreach. So I wrote a function, quoted_map, that will map another function of a quoted list of strings. In fpm-recipes, it adds the -d and makes sure it’s quote (-d "libuuid1 (>= 2.16)") and adds to the FPM args list.

See the code:

# Stupid highlighting, let's give it another double-quote "

# ----------------------------------------------------------------------------
# Function:  quoted_map
# Arguments: 1: Name of function to $(call) for each element of list
#            2: List to iterate over calling the function in 1
# Returns:   The list after calling the function on each element
# ----------------------------------------------------------------------------
# Replace spaces with +, explode on double-quote, then call passed function after turning + back into spaces
# This is to support: DEPENDS = "package (>= 1.0)" other_pack "some_other_packge"
# GNU Make implictly thinks a space is a delimiter, so have to change it to read the above line
quoted_map = $(foreach i,$(subst $(DQUOTE),$(SPACE),$(subst $(SPACE),+,$2)),$(call $1,$(subst +,$(SPACE),$i)))

Thanks to GNU Make Standard Library for inspiration/practical examples.

http://www.dctrwatson.com/2012/06/gnu-make-double-quotes-and-lists/feed/ 0
mutt and gmail http://www.dctrwatson.com/2012/03/mutt-and-gmail/ http://www.dctrwatson.com/2012/03/mutt-and-gmail/#comments Sat, 17 Mar 2012 00:17:58 +0000 http://www.dctrwatson.com/?p=100 Per recommendation from a neckbeard friend, Aaron, I set out to try out Mutt as my email client. Since my email is hosted by Gmail, there’s a little extra configuration needed than just setting up an IMAP inbox. Also, since people actually send multimedia emails, I wrote a small patch for Mutt that detects it’s talking to a Gmail IMAP server and adds a couple custom headers to the message, one of which is the permalink to the email so it can be easily opened in a browser if need be. I’m sure I am one of the few that actually like Google Contacts, so I use Goobook for address completion. And no reason to go through all the trouble of setting up Mutt and not setup GPG for signing/encryption too. I am a fan of Ethan Schoonover’s Solarized color scheme, but I prefer a bit more contrast: I modified the Mutt colors Solarized Dark 16 colors for this preference.

Latest versions of my conf/patch can be found at:
mutt conf GitHub repo
mutt gmail patch GitHub repo

Mutt Higher Contrast Solarized Dark

If you’re new to Mutt, take a look at these links:
MuttWiki: MuttGuide
My first mutt

If you’re not a fan of Git or GitHub (why not?), the current bare minimum files as of the writing of this post are below. Everything should live in ~/.mutt


set certificate_file= ~/.mutt/certificates
set header_cache    = ~/.mutt/cache/headers
set message_cachedir= ~/.mutt/cache/bodies
set mailcap_path    = ~/.mutt/mailcap

set editor          = "vim + -c 'set textwidth=72' -c 'set wrap' -c 'set nocp' -c 'set spell' -c '?^$'"
set query_command   = "goobook query '%s'"
bind editor <Tab> complete-query

set folder      = "imaps://imap.gmail.com:993"
set spoolfile   = "+INBOX"
set postponed   = "+[Gmail]/Drafts"

set imap_check_subscribed   = yes
set imap_idle               = yes
set imap_list_subscribed    = yes
set ssl_force_tls           = yes

unset beep
unset collapse_unread
set implicit_autoview   = yes
set mail_check          = 30
unset markers
unset mark_old
set pager_index_lines   = 10
set pager_context       = 5
set pager_stop          = yes
set pipe_decode         = yes
set quit                = ask-yes
unset record
set reverse_alias       = yes
set sort                = threads
set sort_aux            = reverse-last-date-received
unset sort_re
set thorough_search
set thread_received     = yes
set tilde               = yes
set timeout             = 5
set uncollapse_jump     = yes

alternative_order text/plain text/enriched text/html 

set attribution     = "On %d, %n <%a> wrote:"
set status_format   = "-%r-Mutt: %f [Msgs:%?M?%M/?%m%?n? New:%n?%?o? Old:%o?%?d? Del:%d?%?F? Flag:%F?%?t? Tag:%t?%?p? Post:%p?%?b? Inc:%b? %?l? %l?]---(%s/%S)-default-%>-(%P)---"
set index_format    = "%4C %Z  %[%a %b %D %Y %l:%M:%S%p]  %-15.15L (%?l?%4l&%4c?) %s"

unset copy
set edit_headers    = yes
set fast_reply      = yes
set forward_format  = "Fwd: %s"
set forward_quote   = yes
set include         = yes
unset recall
unset reply_self
set reply_to        = yes

ignore *
unignore date: from: to: cc: subject:
unignore x-mailing-list: posted-to:
unignore x-mailer:
unignore x-gm-permalink:
hdr_order x-gm-permalink: date: from: to: cc: subject:

source ~/.mutt/identity
source ~/.mutt/colors
source ~/.mutt/keybindings

source ~/.mutt/gpg-agent.rc
set crypt_autosign              = yes
set crypt_replysign             = yes
set crypt_replysignencrypted    = yes
set pgp_use_gpg_agent           = yes

set sidebar_visible = yes
set sidebar_width    = 30
set sidebar_delim   = '|'
color   sidebar_new brightmagenta   default


set realname    = "MY NAME"
set from        = "example@example.org"
set imap_user   = "example@gmail.com"
set imap_pass   = "PASSWORD"
set smtp_pass   = "PASSWORD"
set smtp_url    = "smtps://example@gmail.com@mtp.gmail.com:465/"
set pgp_sign_as = "0xFFFFFFFF"
# vim: ft=muttrc


# Gmail style keybindings
bind  editor <space> noop
bind  index,pager c  mail       #Compose
macro index,pager e  "<save-message>=[Gmail]/All Mail<enter><enter>" "Archive conversation"
bind  generic     x  tag-entry      #Select Conversation
bind  index       x  tag-thread     #Select Conversation
bind  pager       x  tag-message    #Select Conversation
bind  index,pager s  flag-message   #Star a message
macro index,pager +  <save-message>=[Gmail]/Important<enter><enter> "Mark as important"
macro index,pager !  <save-message>=[Gmail]/Spam<enter><enter> "Report spam"
bind  index,pager a  group-reply    #Reply all
bind  index,pager \# delete-thread  #Delete
bind  index,pager l  copy-message   #Label
bind  index,pager v  save-message   #Move to
macro index,pager I  <set-flag>O    "Mark as read"
macro index,pager U  <clear-flag>O  "Mark as unread"
macro index,pager ga "<change-folder>=[Gmail]/All Mail<enter>"  "Go to all mail"
macro index,pager gs <change-folder>=[Gmail]/Starred<enter> "Go to 'Starred'"
macro index,pager gd <change-folder>=[Gmail]/Drafts<enter>  "Go to 'Drafts'"
macro index,pager gl <change-folder>?               "Go to 'Label'"
macro index,pager gi <change-folder>=INBOX<enter>       "Go to inbox"
macro index,pager gt "<change-folder>=[Gmail]/Sent Mail<enter>" "Go to 'Sent Mail'"
folder-hook +INBOX 'macro index,pager y "<save-message>=[Gmail]/All Mail<enter><enter>" "Archive conversation"'
folder-hook +[Gmail]/Trash macro index,pager y <save-message>=INBOX<enter><enter> "Move to inbox"
folder-hook +[Gmail]/Starred bind  index,pager y flag-message #"Toggle star"

# Other bindings
#macro index \CR "T~N\n;WNT~O;WO\CT~A" "mark all messages read"
macro index \CR "T~N\n;WN;T~O\n;WO;\CT~A\n" "mark all messages read"

# Sidebar bindings
bind index,pager \CJ sidebar-prev
bind index,pager \CK sidebar-next
bind index,pager \CO sidebar-open

folder-hook . "exec collapse-all"


# -*-muttrc-*-
# Command formats for gpg-agent

set pgp_decode_command="gpg --no-verbose --quiet --batch --output - %f"
set pgp_verify_command="gpg --no-verbose --quiet --batch --output - --verify %s %f"
set pgp_decrypt_command="gpg --no-verbose --quiet --batch --output - %f"
set pgp_sign_command="gpg --no-verbose --batch --quiet --output - --armor --detach-sign --textmode %?a?-u %a? %f"
set pgp_clearsign_command="gpg --no-verbose --batch --quiet --output - --armor --textmode --clearsign %?a?-u %a? %f"
set pgp_encrypt_only_command="pgpewrap gpg --batch --quiet --no-verbose --output - --encrypt --textmode --armor --always-trust -- -r %r -- %f"
set pgp_encrypt_sign_command="pgpewrap gpg --batch --quiet --no-verbose --textmode --output - --encrypt --sign %?a?-u %a? --armor --always-trust -- -r %r -- %f"
set pgp_import_command="gpg --no-verbose --import %f"
set pgp_export_command="gpg --no-verbose --export --armor %r"
set pgp_verify_key_command="gpg --verbose --batch --fingerprint --check-sigs %r"
set pgp_list_pubring_command="gpg --no-verbose --batch --quiet --with-colons --list-keys %r"
set pgp_list_secring_command="gpg --no-verbose --batch --quiet --with-colons --list-secret-keys %r"
set pgp_good_sign="`gettext -d gnupg -s 'Good signature from "' | tr -d '"'`"


# Modified solarized dark
# Original: https://github.com/altercation/mutt-colors-solarized/blob/master/mutt-colors-solarized-dark-16.muttrc
# By: Ethan Schoonover <es@ethanschoonover.com>
# make sure that you are using mutt linked against slang, not ncurses, or
# suffer the consequences of weird color issues. use "mutt -v" to check this.

# basic colors ---------------------------------------------------------
color normal        default         defaul
color error         red             defaul
color tilde         black           defaul
color message       cyan            defaul
color markers       red             white
color attachment    white           defaul
color search        brightmagenta   defaul
color status        brightyellow    black
color indicator     brightblack     yellow
color tree          yellow          default                                     # arrow in threads

# basic monocolor screen
mono  bold          bold
mono  underline     underline
mono  indicator     reverse
mono  error         bold

# index ----------------------------------------------------------------

#color index         red             default         "~A"                        # all messages
color index         brightred       default         "~E"                        # expired messages
color index         blue            default         "~N"                        # new messages
color index         blue            default         "~O"                        # old messages
color index         brightmagenta   default         "~Q"                        # messages that have been replied to
#color index         brightgreen     default         "~R"                        # read messages
color index         blue            default         "~U"                        # unread messages
color index         blue            default         "~U~$"                      # unread, unreferenced messages
color index         brightyellow    default         "~v"                        # messages part of a collapsed thread
color index         brightyellow    default         "~P"                        # messages from me
color index         cyan            default         "~p!~F"                     # messages to me
color index         cyan            default         "~N~p!~F"                   # new messages to me
color index         cyan            default         "~U~p!~F"                   # unread messages to me
#color index         brightgreen     default         "~R~p!~F"                   # messages to me
color index         red             default         "~F"                        # flagged messages
color index         red             default         "~F~p"                      # flagged messages to me
color index         red             default         "~N~F"                      # new flagged messages
color index         red             default         "~N~F~p"                    # new flagged messages to me
color index         red             default         "~U~F~p"                    # new flagged messages to me
color index         black           red             "~D"                        # deleted messages
color index         brightcyan      default         "~v~(!~N)"                  # collapsed thread with no unread
color index         yellow          default         "~v~(~N)"                   # collapsed thread with some unread
color index         green           default         "~N~v~(~N)"                 # collapsed thread with unread paren
# statusbg used to indicated flagged when foreground color shows other status
# for collapsed thread
color index         red             black           "~v~(~F)!~N"                # collapsed thread with flagged, no unread
color index         yellow          black           "~v~(~F~N)"                 # collapsed thread with some unread & flagged
color index         green           black           "~N~v~(~F~N)"               # collapsed thread with unread parent & flagged
color index         green           black           "~N~v~(~F)"                 # collapsed thread with unread parent, no unread inside, but some flagged
color index         cyan            black           "~v~(~p)"                   # collapsed thread with unread parent, no unread inside, some to me directly
color index         yellow          red             "~v~(~D)"                   # thread with deleted (doesn't differentiate between all or partial)
color index         yellow          default         "~(~N)"                    # messages in threads with some unread
color index         green           default         "~S"                       # superseded messages
color index         red             default         "~T"                       # tagged messages
color index         brightred       red             "~="                       # duplicated messages

# message headers ------------------------------------------------------

color hdrdefault    default         defaul
color header        green           default         "^(From)"
color header        blue            default         "^(To)"
color header        cyan            default         "^(Subject)"

# body -----------------------------------------------------------------

color quoted        blue            defaul
color quoted1       cyan            defaul
color quoted2       yellow          defaul
color quoted3       red             defaul
color quoted4       brightred       defaul

color signature     green            defaul
color bold          black           defaul
color underline     black           defaul
color body          brightcyan      default         "[;:][-o][)/(|]"    # emoticons
color body          brightcyan      default         "[;:][)(|]"         # emoticons
color body          brightcyan      default         "[*]?((N)?ACK|CU|LOL|SCNR|BRB|BTW|CWYL|
color body          brightcyan      default         "[ ][*][^*]*[*][ ]?" # more emoticon?
color body          brightcyan      default         "[ ]?[*][^*]*[*][ ]" # more emoticon?

## pgp

color body          red             default         "(BAD signature)"
color body          cyan            default         "(Good signature)"
color body          brightblack     default         "^gpg: Good signature .*"
color body          brightyellow    default         "^gpg: "
color body          brightyellow    red             "^gpg: BAD signature from.*"
mono  body          bold                            "^gpg: Good signature"
mono  body          bold                            "^gpg: BAD signature from.*"

# yes, an insance URL regex
color body          red             default         "([a-z][a-z0-9+-]*://(((([a-z0-9_.!~*'();:&=+$,-]|%[0-9a-f][0-9a-f])*@)?((([a-z0-9]([a-z0-9-]*[a-z0-9])?)\\.)*([a-z]([a-z0-9-]*[a-z0-9])?)\\.?|[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)(:[0-9]+)?)|([a-z0-9_.!~*'()$,;:@&=+-]|%[0-9a-f][0-9a-f])+)(/([a-z0-9_.!~*'():@&=+$,-]|%[0-9a-f][0-9a-f])*(;([a-z0-9_.!~*'():@&=+$,-]|%[0-9a-f][0-9a-f])*)*(/([a-z0-9_.!~*'():@&=+$,-]|%[0-9a-f][0-9a-f])*(;([a-z0-9_.!~*'():@&=+$,-]|%[0-9a-f][0-9a-f])*)*)*)?(\\?([a-z0-9_.!~*'();/?:@&=+$,-]|%[0-9a-f][0-9a-f])*)?(#([a-z0-9_.!~*'();/?:@&=+$,-]|%[0-9a-f][0-9a-f])*)?|(www|ftp)\\.(([a-z0-9]([a-z0-9-]*[a-z0-9])?)\\.)*([a-z]([a-z0-9-]*[a-z0-9])?)\\.?(:[0-9]+)?(/([-a-z0-9_.!~*'():@&=+$,]|%[0-9a-f][0-9a-f])*(;([-a-z0-9_.!~*'():@&=+$,]|%[0-9a-f][0-9a-f])*)*(/([-a-z0-9_.!~*'():@&=+$,]|%[0-9a-f][0-9a-f])*(;([-a-z0-9_.!~*'():@&=+$,]|%[0-9a-f][0-9a-f])*)*)*)?(\\?([-a-z0-9_.!~*'();/?:@&=+$,]|%[0-9a-f][0-9a-f])*)?(#([-a-z0-9_.!~*'();/?:@&=+$,]|%[0-9a-f][0-9a-f])*)?)[^].,:;!)? \t\r\n<>\"]"
# and a heavy handed email regex
color body          magenta         default         "((@(([0-9a-z-]+\\.)*[0-9a-z-]+\\.?|#[0-9]+|\\[[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\]),)*@(([0-9a-z-]+\\.)*[0-9a-z-]+\\.?|#[0-9]+|\\[[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\]):)?[0-9a-z_.+%$-]+@(([0-9a-z-]+\\.)*[0-9a-z-]+\\.?|#[0-9]+|\\[[0-2]?[0-9]?[0-9]\\.[0-2]?[0-9]?[0-9]\\.[0-2]?[0-9]?[0-9]\\.[0-2]?[0-9]?[0-9]\\])"

# *bold*
color body          blue            default         "(^|[[:space:][:punct:]])\\*[^*]+\\*([[:space:][:punct:]]|$)"
mono  body          bold                            "(^|[[:space:][:punct:]])\\*[^*]+\\*([[:space:][:punct:]]|$)"
# _underline_
color body          blue            default         "(^|[[:space:][:punct:]])_[^_]+_([[:space:][:punct:]]|$)"
mono  body          underline                       "(^|[[:space:][:punct:]])_[^_]+_([[:space:][:punct:]]|$)"
# /italic/  (Sometimes gets directory names)
color body         blue            default         "(^|[[:space:][:punct:]])/[^/]+/([[:space:][:punct:]]|$)"
mono body          underline                       "(^|[[:space:][:punct:]])/[^/]+/([[:space:][:punct:]]|$)"

# Border lines.
color body          blue            default         "( *[-+=#*~_]){6,}"

# vim: filetype=muttrc

mutt gmail patch

--- imap/command.c.orig	2012-03-07 22:59:11.784404517 -0800
+++ imap/command.c	2012-03-07 22:59:25.517273315 -0800
@@ -66,6 +66,7 @@
+  "X-GM-EXT-1",
--- imap/message.c.orig	2010-08-24 09:34:21.000000000 -0700
+++ imap/message.c	2012-03-07 22:59:25.517273315 -0800
@@ -242,8 +242,10 @@
       char *cmd;
       fetchlast = msgend + 1;
-      safe_asprintf (&cmd, "FETCH %d:%d (UID FLAGS INTERNALDATE RFC822.SIZE %s)",
-                     msgno + 1, fetchlast, hdrreq);
+      safe_asprintf (&cmd, "FETCH %d:%d (%sUID FLAGS INTERNALDATE RFC822.SIZE %s)",
+                     msgno + 1, fetchlast,
+                     mutt_bit_isset (idata->capabilities, X_GM_EXT_1) ? "X-GM-MSGID " : "",
+                     hdrreq);
       imap_cmd_start (idata, cmd);
       FREE (&cmd);
@@ -452,6 +454,12 @@
+  if (mutt_bit_isset (idata->capabilities, X_GM_EXT_1))
+  {
+    fprintf (msg->fp, "X-Gm-Msgid:%llu\n", HEADER_DATA(h)->msgid);
+    fprintf (msg->fp, "X-Gm-Permalink: https://mail.google.com/mail/#all/%llx\n", HEADER_DATA(h)->msgid);
+  }
   /* mark this header as currently inactive so the command handler won't
    * also try to update it. HACK until all this code can be moved into the
    * command handler */
@@ -1142,6 +1152,14 @@
       if1 == NULL)
         return -1;
+    else if (ascii_strncasecmp ("X-GM-MSGID", s, 10) == 0)
+    {
+      s += 10;
+      SKIPWS (s);
+      h->data->msgid = strtoull (s, NULL, 10);
+      s = imap_next_word (s);
+    }
     else if (ascii_strncasecmp ("UID", s, 3) == 0)
       s += 3;
--- imap/message.h.orig	2012-03-07 22:59:00.101513364 -0800
+++ imap/message.h	2012-03-07 22:59:29.586234445 -0800
@@ -37,6 +37,7 @@
   unsigned int parsed : 1;
   unsigned int uid;	/* 32-bit Message UID */
+  unsigned long long msgid;
   LIST *keywords;
--- imap/imap_private.h.orig	2012-03-07 22:58:29.465804586 -0800
+++ imap/imap_private.h	2012-03-07 22:59:29.586234445 -0800
@@ -114,6 +114,7 @@
   IDLE,                         /* RFC 2177: IDLE */
   SASL_IR,                      /* SASL initial response draft */
+  X_GM_EXT_1,       /* Gmail IMAP Extensions */
  1. s = msg_parse_flags (h, s
http://www.dctrwatson.com/2012/03/mutt-and-gmail/feed/ 6
limits.conf and daemons on Ubuntu http://www.dctrwatson.com/2012/03/limits-conf-and-daemons-on-ubuntu/ http://www.dctrwatson.com/2012/03/limits-conf-and-daemons-on-ubuntu/#comments Fri, 16 Mar 2012 22:52:28 +0000 http://www.dctrwatson.com/?p=98 I recently was setting up a couple ElasticSearch and RabbitMQ instances when I noticed RabbitMQ was still reporting an abysmally low fd limit in its log file at startup. I double checked my /etc/security/limits.conf and sure enough, limits were properly set to 64000. Yet for some reason it was still only seeing a max of 1024.

It turns out that in Ubuntu 10.04, /etc/pam.d/common-session{,-noninteractive} does not contain:

session required pam_limits.so

Adding that, solved my issue:

=INFO REPORT==== 1-Feb-2012::00:05:47 ===
Limiting to approx 63900 file handles (57508 sockets)

UPDATE (Wed Apr 18 15:01:17 PDT 2012)
For RabbitMQ 2.8.x, the init script uses start-stop-daemon. Apply this patch:

--- /etc/init.d/rabbitmq-server.old	2012-04-18 21:54:05.852307662 +0000
+++ /etc/init.d/rabbitmq-server	2012-04-18 21:49:17.594182809 +0000
@@ -35,6 +35,8 @@
 set -e
+[ -r /etc/default/${NAME} ] && . /etc/default/${NAME}
 ensure_pid_dir () {
     PID_DIR=`dirname ${PID_FILE}`
     if [ ! -d ${PID_DIR} ] ; then

And then in /etc/default/rabbitmq-server

ulimit -n 65000

http://www.dctrwatson.com/2012/03/limits-conf-and-daemons-on-ubuntu/feed/ 5