<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Dctr Watson</title>
	<atom:link href="http://www.dctrwatson.com/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.dctrwatson.com</link>
	<description>The odyssey of a philomath of technology</description>
	<lastBuildDate>Wed, 18 Apr 2012 22:01:31 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.1</generator>
		<item>
		<title>mutt and gmail</title>
		<link>http://www.dctrwatson.com/2012/03/mutt-and-gmail/</link>
		<comments>http://www.dctrwatson.com/2012/03/mutt-and-gmail/#comments</comments>
		<pubDate>Sat, 17 Mar 2012 00:17:58 +0000</pubDate>
		<dc:creator>dctrwatson</dc:creator>
				<category><![CDATA[Mutt]]></category>
		<category><![CDATA[gmail]]></category>
		<category><![CDATA[mutt]]></category>

		<guid isPermaLink="false">http://www.dctrwatson.com/?p=100</guid>
		<description><![CDATA[Per recommendation from a neckbeard friend, Aaron, I set out to try out Mutt as my email client. Since my email is hosted by Gmail, there&#8217;s a little extra configuration needed than just setting up an IMAP inbox. Also, since people actually send multimedia emails, I wrote a small patch for Mutt that detects it&#8217;s [...]]]></description>
			<content:encoded><![CDATA[<p>Per recommendation from a neckbeard friend, <a href="https://twitter.com/#!/aaronky" target="_blank">Aaron</a>, I set out to try out <a href="http://www.mutt.org" target="_blank">Mutt</a> as my email client. Since my email is hosted by <a href="http://gmail.com" target="_blank">Gmail</a>, there&#8217;s a little extra configuration needed than just <a href="http://wiki.mutt.org/?MuttGuide/UseIMAP" target="_blank">setting up an IMAP inbox</a>. Also, since people actually send multimedia emails, I wrote a small patch for Mutt that detects it&#8217;s talking to a <a href="https://developers.google.com/google-apps/gmail/imap_extensions" target="_blank">Gmail IMAP server</a> and adds a couple custom headers to the message, one of which is the permalink to the email so it can be easily opened in a browser if need be. I&#8217;m sure I am one of the few that actually like <a href="https://www.google.com/contacts/" target="_blank">Google Contacts</a>, so I use <a href="http://code.google.com/p/goobook/" target="_blank">Goobook</a> for address completion. And no reason to go through all the trouble of setting up Mutt and not setup <a href="http://www.gnupg.org/" target="_blank">GPG</a> for signing/encryption too. I am a fan of <a href="http://ethanschoonover.com/solarized" target="_blank">Ethan Schoonover&#8217;s Solarized</a> color scheme, but I prefer a bit more contrast: I modified the <a href="https://github.com/altercation/mutt-colors-solarized/blob/master/mutt-colors-solarized-dark-16.muttrc" target="_blank">Mutt colors Solarized Dark 16 colors</a> for this preference.</p>
<p>Latest versions of my conf/patch can be found at:<br />
<a href="https://github.com/dctrwatson/dotfiles/tree/master/mutt" target="_blank">mutt conf GitHub repo</a><br />
<a href="https://github.com/dctrwatson/mutt-gmail-patch" target="_blank">mutt gmail patch GitHub repo</a></p>
<p><span id="more-100"></span></p>
<p><a href="http://www.dctrwatson.com/wp-content/uploads/2012/03/mutt_custom_dark_solarized.jpg"><img src="http://www.dctrwatson.com/wp-content/uploads/2012/03/mutt_custom_dark_solarized-1024x402.jpg" alt="Mutt Higher Contrast Solarized Dark" title="Mutt Higher Contrast Solarized Dark" width="600" height="235" class="aligncenter size-large wp-image-101" /></a></p>
<p>If you&#8217;re new to Mutt, take a look at these links:<br />
<a href="http://wiki.mutt.org/index.cgi?MuttGuide" title="MuttWiki: MuttGuide" target="_blank">MuttWiki: MuttGuide</a><br />
<a href="http://mutt.blackfish.org.uk/" title="My first mutt" target="_blank">My first mutt</a></p>
<p>If you&#8217;re not a fan of Git or GitHub (why not?), the current bare minimum files as of the writing of this post are below. Everything should live in <strong>~/.mutt</strong></p>
<p><strong>muttrc</strong></p>
<pre class="brush: plain; collapse: true; gutter: false; light: false; title: ; toolbar: true; notranslate">
set certificate_file= ~/.mutt/certificates
set header_cache    = ~/.mutt/cache/headers
set message_cachedir= ~/.mutt/cache/bodies
set mailcap_path    = ~/.mutt/mailcap

set editor          = &quot;vim + -c 'set textwidth=72' -c 'set wrap' -c 'set nocp' -c 'set spell' -c '?^$'&quot;
set query_command   = &quot;goobook query '%s'&quot;
bind editor &lt;Tab&gt; complete-query

set folder      = &quot;imaps://imap.gmail.com:993&quot;
set spoolfile   = &quot;+INBOX&quot;
set postponed   = &quot;+[Gmail]/Drafts&quot;

set imap_check_subscribed   = yes
set imap_idle               = yes
set imap_list_subscribed    = yes
set ssl_force_tls           = yes

unset beep
unset collapse_unread
set implicit_autoview   = yes
set mail_check          = 30
unset markers
unset mark_old
set pager_index_lines   = 10
set pager_context       = 5
set pager_stop          = yes
set pipe_decode         = yes
set quit                = ask-yes
unset record
set reverse_alias       = yes
set sort                = threads
set sort_aux            = reverse-last-date-received
unset sort_re
set thorough_search
set thread_received     = yes
set tilde               = yes
set timeout             = 5
set uncollapse_jump     = yes

alternative_order text/plain text/enriched text/html 

set attribution     = &quot;On %d, %n &lt;%a&gt; wrote:&quot;
set status_format   = &quot;-%r-Mutt: %f [Msgs:%?M?%M/?%m%?n? New:%n?%?o? Old:%o?%?d? Del:%d?%?F? Flag:%F?%?t? Tag:%t?%?p? Post:%p?%?b? Inc:%b? %?l? %l?]---(%s/%S)-default-%&gt;-(%P)---&quot;
set index_format    = &quot;%4C %Z  %[%a %b %D %Y %l:%M:%S%p]  %-15.15L (%?l?%4l&amp;%4c?) %s&quot;

unset copy
set edit_headers    = yes
set fast_reply      = yes
set forward_format  = &quot;Fwd: %s&quot;
set forward_quote   = yes
set include         = yes
unset recall
unset reply_self
set reply_to        = yes

ignore *
unignore date: from: to: cc: subject:
unignore x-mailing-list: posted-to:
unignore x-mailer:
unignore x-gm-permalink:
hdr_order x-gm-permalink: date: from: to: cc: subject:

source ~/.mutt/identity
source ~/.mutt/colors
source ~/.mutt/keybindings

source ~/.mutt/gpg-agent.rc
set crypt_autosign              = yes
set crypt_replysign             = yes
set crypt_replysignencrypted    = yes
set pgp_use_gpg_agent           = yes

set sidebar_visible = yes
set sidebar_width    = 30
set sidebar_delim   = '|'
color   sidebar_new brightmagenta   default
</pre>
<p><strong>identity</strong></p>
<pre class="brush: plain; collapse: true; gutter: false; light: false; title: ; toolbar: true; notranslate">
set realname    = &quot;MY NAME&quot;
set from        = &quot;example@example.org&quot;
set imap_user   = &quot;example@gmail.com&quot;
set imap_pass   = &quot;PASSWORD&quot;
set smtp_pass   = &quot;PASSWORD&quot;
set smtp_url    = &quot;smtps://example@gmail.com@mtp.gmail.com:465/&quot;
set pgp_sign_as = &quot;0xFFFFFFFF&quot;
# vim: ft=muttrc
</pre>
<p><strong>keybindings</strong></p>
<pre class="brush: plain; collapse: true; gutter: false; light: false; title: ; toolbar: true; notranslate">
# Gmail style keybindings
bind  editor &lt;space&gt; noop
bind  index,pager c  mail       #Compose
macro index,pager e  &quot;&lt;save-message&gt;=[Gmail]/All Mail&lt;enter&gt;&lt;enter&gt;&quot; &quot;Archive conversation&quot;
bind  generic     x  tag-entry      #Select Conversation
bind  index       x  tag-thread     #Select Conversation
bind  pager       x  tag-message    #Select Conversation
bind  index,pager s  flag-message   #Star a message
macro index,pager +  &lt;save-message&gt;=[Gmail]/Important&lt;enter&gt;&lt;enter&gt; &quot;Mark as important&quot;
macro index,pager !  &lt;save-message&gt;=[Gmail]/Spam&lt;enter&gt;&lt;enter&gt; &quot;Report spam&quot;
bind  index,pager a  group-reply    #Reply all
bind  index,pager \# delete-thread  #Delete
bind  index,pager l  copy-message   #Label
bind  index,pager v  save-message   #Move to
macro index,pager I  &lt;set-flag&gt;O    &quot;Mark as read&quot;
macro index,pager U  &lt;clear-flag&gt;O  &quot;Mark as unread&quot;
macro index,pager ga &quot;&lt;change-folder&gt;=[Gmail]/All Mail&lt;enter&gt;&quot;  &quot;Go to all mail&quot;
macro index,pager gs &lt;change-folder&gt;=[Gmail]/Starred&lt;enter&gt; &quot;Go to 'Starred'&quot;
macro index,pager gd &lt;change-folder&gt;=[Gmail]/Drafts&lt;enter&gt;  &quot;Go to 'Drafts'&quot;
macro index,pager gl &lt;change-folder&gt;?               &quot;Go to 'Label'&quot;
macro index,pager gi &lt;change-folder&gt;=INBOX&lt;enter&gt;       &quot;Go to inbox&quot;
macro index,pager gt &quot;&lt;change-folder&gt;=[Gmail]/Sent Mail&lt;enter&gt;&quot; &quot;Go to 'Sent Mail'&quot;
folder-hook +INBOX 'macro index,pager y &quot;&lt;save-message&gt;=[Gmail]/All Mail&lt;enter&gt;&lt;enter&gt;&quot; &quot;Archive conversation&quot;'
folder-hook +[Gmail]/Trash macro index,pager y &lt;save-message&gt;=INBOX&lt;enter&gt;&lt;enter&gt; &quot;Move to inbox&quot;
folder-hook +[Gmail]/Starred bind  index,pager y flag-message #&quot;Toggle star&quot;

# Other bindings
#macro index \CR &quot;T~N\n;WNT~O;WO\CT~A&quot; &quot;mark all messages read&quot;
macro index \CR &quot;T~N\n;WN;T~O\n;WO;\CT~A\n&quot; &quot;mark all messages read&quot;

# Sidebar bindings
bind index,pager \CJ sidebar-prev
bind index,pager \CK sidebar-next
bind index,pager \CO sidebar-open

folder-hook . &quot;exec collapse-all&quot;
</pre>
<p><strong>gpg-agent.rc</strong></p>
<pre class="brush: plain; collapse: true; gutter: false; light: false; title: ; toolbar: true; notranslate">
# -*-muttrc-*-
#
# Command formats for gpg-agent
#

set pgp_decode_command=&quot;gpg --no-verbose --quiet --batch --output - %f&quot;
set pgp_verify_command=&quot;gpg --no-verbose --quiet --batch --output - --verify %s %f&quot;
set pgp_decrypt_command=&quot;gpg --no-verbose --quiet --batch --output - %f&quot;
set pgp_sign_command=&quot;gpg --no-verbose --batch --quiet --output - --armor --detach-sign --textmode %?a?-u %a? %f&quot;
set pgp_clearsign_command=&quot;gpg --no-verbose --batch --quiet --output - --armor --textmode --clearsign %?a?-u %a? %f&quot;
set pgp_encrypt_only_command=&quot;pgpewrap gpg --batch --quiet --no-verbose --output - --encrypt --textmode --armor --always-trust -- -r %r -- %f&quot;
set pgp_encrypt_sign_command=&quot;pgpewrap gpg --batch --quiet --no-verbose --textmode --output - --encrypt --sign %?a?-u %a? --armor --always-trust -- -r %r -- %f&quot;
set pgp_import_command=&quot;gpg --no-verbose --import %f&quot;
set pgp_export_command=&quot;gpg --no-verbose --export --armor %r&quot;
set pgp_verify_key_command=&quot;gpg --verbose --batch --fingerprint --check-sigs %r&quot;
set pgp_list_pubring_command=&quot;gpg --no-verbose --batch --quiet --with-colons --list-keys %r&quot;
set pgp_list_secring_command=&quot;gpg --no-verbose --batch --quiet --with-colons --list-secret-keys %r&quot;
set pgp_good_sign=&quot;`gettext -d gnupg -s 'Good signature from &quot;' | tr -d '&quot;'`&quot;
</pre>
<p><strong>colors</strong></p>
<pre class="brush: plain; collapse: true; gutter: false; light: false; title: ; toolbar: true; notranslate">
# Modified solarized dark
#
# Original: https://github.com/altercation/mutt-colors-solarized/blob/master/mutt-colors-solarized-dark-16.muttrc
# By: Ethan Schoonover &lt;es@ethanschoonover.com&gt;
#
#
# make sure that you are using mutt linked against slang, not ncurses, or
# suffer the consequences of weird color issues. use &quot;mutt -v&quot; to check this.

# basic colors ---------------------------------------------------------
color normal        default         defaul
color error         red             defaul
color tilde         black           defaul
color message       cyan            defaul
color markers       red             white
color attachment    white           defaul
color search        brightmagenta   defaul
color status        brightyellow    black
color indicator     brightblack     yellow
color tree          yellow          default                                     # arrow in threads

# basic monocolor screen
mono  bold          bold
mono  underline     underline
mono  indicator     reverse
mono  error         bold

# index ----------------------------------------------------------------

#color index         red             default         &quot;~A&quot;                        # all messages
color index         brightred       default         &quot;~E&quot;                        # expired messages
color index         blue            default         &quot;~N&quot;                        # new messages
color index         blue            default         &quot;~O&quot;                        # old messages
color index         brightmagenta   default         &quot;~Q&quot;                        # messages that have been replied to
#color index         brightgreen     default         &quot;~R&quot;                        # read messages
color index         blue            default         &quot;~U&quot;                        # unread messages
color index         blue            default         &quot;~U~$&quot;                      # unread, unreferenced messages
color index         brightyellow    default         &quot;~v&quot;                        # messages part of a collapsed thread
color index         brightyellow    default         &quot;~P&quot;                        # messages from me
color index         cyan            default         &quot;~p!~F&quot;                     # messages to me
color index         cyan            default         &quot;~N~p!~F&quot;                   # new messages to me
color index         cyan            default         &quot;~U~p!~F&quot;                   # unread messages to me
#color index         brightgreen     default         &quot;~R~p!~F&quot;                   # messages to me
color index         red             default         &quot;~F&quot;                        # flagged messages
color index         red             default         &quot;~F~p&quot;                      # flagged messages to me
color index         red             default         &quot;~N~F&quot;                      # new flagged messages
color index         red             default         &quot;~N~F~p&quot;                    # new flagged messages to me
color index         red             default         &quot;~U~F~p&quot;                    # new flagged messages to me
color index         black           red             &quot;~D&quot;                        # deleted messages
color index         brightcyan      default         &quot;~v~(!~N)&quot;                  # collapsed thread with no unread
color index         yellow          default         &quot;~v~(~N)&quot;                   # collapsed thread with some unread
color index         green           default         &quot;~N~v~(~N)&quot;                 # collapsed thread with unread paren
# statusbg used to indicated flagged when foreground color shows other status
# for collapsed thread
color index         red             black           &quot;~v~(~F)!~N&quot;                # collapsed thread with flagged, no unread
color index         yellow          black           &quot;~v~(~F~N)&quot;                 # collapsed thread with some unread &amp; flagged
color index         green           black           &quot;~N~v~(~F~N)&quot;               # collapsed thread with unread parent &amp; flagged
color index         green           black           &quot;~N~v~(~F)&quot;                 # collapsed thread with unread parent, no unread inside, but some flagged
color index         cyan            black           &quot;~v~(~p)&quot;                   # collapsed thread with unread parent, no unread inside, some to me directly
color index         yellow          red             &quot;~v~(~D)&quot;                   # thread with deleted (doesn't differentiate between all or partial)
color index         yellow          default         &quot;~(~N)&quot;                    # messages in threads with some unread
color index         green           default         &quot;~S&quot;                       # superseded messages
color index         red             default         &quot;~T&quot;                       # tagged messages
color index         brightred       red             &quot;~=&quot;                       # duplicated messages

# message headers ------------------------------------------------------

color hdrdefault    default         defaul
color header        green           default         &quot;^(From)&quot;
color header        blue            default         &quot;^(To)&quot;
color header        cyan            default         &quot;^(Subject)&quot;

# body -----------------------------------------------------------------

color quoted        blue            defaul
color quoted1       cyan            defaul
color quoted2       yellow          defaul
color quoted3       red             defaul
color quoted4       brightred       defaul

color signature     green            defaul
color bold          black           defaul
color underline     black           defaul
#
color body          brightcyan      default         &quot;[;:][-o][)/(|]&quot;    # emoticons
color body          brightcyan      default         &quot;[;:][)(|]&quot;         # emoticons
color body          brightcyan      default         &quot;[*]?((N)?ACK|CU|LOL|SCNR|BRB|BTW|CWYL|
                                                     |FWIW|vbg|GD&amp;R|HTH|HTHBE|IMHO|IMNSHO|
                                                     |IRL|RTFM|ROTFL|ROFL|YMMV)[*]?&quot;
color body          brightcyan      default         &quot;[ ][*][^*]*[*][ ]?&quot; # more emoticon?
color body          brightcyan      default         &quot;[ ]?[*][^*]*[*][ ]&quot; # more emoticon?

## pgp

color body          red             default         &quot;(BAD signature)&quot;
color body          cyan            default         &quot;(Good signature)&quot;
color body          brightblack     default         &quot;^gpg: Good signature .*&quot;
color body          brightyellow    default         &quot;^gpg: &quot;
color body          brightyellow    red             &quot;^gpg: BAD signature from.*&quot;
mono  body          bold                            &quot;^gpg: Good signature&quot;
mono  body          bold                            &quot;^gpg: BAD signature from.*&quot;

# yes, an insance URL regex
color body          red             default         &quot;([a-z][a-z0-9+-]*://(((([a-z0-9_.!~*'();:&amp;=+$,-]|%[0-9a-f][0-9a-f])*@)?((([a-z0-9]([a-z0-9-]*[a-z0-9])?)\\.)*([a-z]([a-z0-9-]*[a-z0-9])?)\\.?|[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)(:[0-9]+)?)|([a-z0-9_.!~*'()$,;:@&amp;=+-]|%[0-9a-f][0-9a-f])+)(/([a-z0-9_.!~*'():@&amp;=+$,-]|%[0-9a-f][0-9a-f])*(;([a-z0-9_.!~*'():@&amp;=+$,-]|%[0-9a-f][0-9a-f])*)*(/([a-z0-9_.!~*'():@&amp;=+$,-]|%[0-9a-f][0-9a-f])*(;([a-z0-9_.!~*'():@&amp;=+$,-]|%[0-9a-f][0-9a-f])*)*)*)?(\\?([a-z0-9_.!~*'();/?:@&amp;=+$,-]|%[0-9a-f][0-9a-f])*)?(#([a-z0-9_.!~*'();/?:@&amp;=+$,-]|%[0-9a-f][0-9a-f])*)?|(www|ftp)\\.(([a-z0-9]([a-z0-9-]*[a-z0-9])?)\\.)*([a-z]([a-z0-9-]*[a-z0-9])?)\\.?(:[0-9]+)?(/([-a-z0-9_.!~*'():@&amp;=+$,]|%[0-9a-f][0-9a-f])*(;([-a-z0-9_.!~*'():@&amp;=+$,]|%[0-9a-f][0-9a-f])*)*(/([-a-z0-9_.!~*'():@&amp;=+$,]|%[0-9a-f][0-9a-f])*(;([-a-z0-9_.!~*'():@&amp;=+$,]|%[0-9a-f][0-9a-f])*)*)*)?(\\?([-a-z0-9_.!~*'();/?:@&amp;=+$,]|%[0-9a-f][0-9a-f])*)?(#([-a-z0-9_.!~*'();/?:@&amp;=+$,]|%[0-9a-f][0-9a-f])*)?)[^].,:;!)? \t\r\n&lt;&gt;\&quot;]&quot;
# and a heavy handed email regex
color body          magenta         default         &quot;((@(([0-9a-z-]+\\.)*[0-9a-z-]+\\.?|#[0-9]+|\\[[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\]),)*@(([0-9a-z-]+\\.)*[0-9a-z-]+\\.?|#[0-9]+|\\[[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]\\]):)?[0-9a-z_.+%$-]+@(([0-9a-z-]+\\.)*[0-9a-z-]+\\.?|#[0-9]+|\\[[0-2]?[0-9]?[0-9]\\.[0-2]?[0-9]?[0-9]\\.[0-2]?[0-9]?[0-9]\\.[0-2]?[0-9]?[0-9]\\])&quot;

# *bold*
color body          blue            default         &quot;(^|[[:space:][:punct:]])\\*[^*]+\\*([[:space:][:punct:]]|$)&quot;
mono  body          bold                            &quot;(^|[[:space:][:punct:]])\\*[^*]+\\*([[:space:][:punct:]]|$)&quot;
# _underline_
color body          blue            default         &quot;(^|[[:space:][:punct:]])_[^_]+_([[:space:][:punct:]]|$)&quot;
mono  body          underline                       &quot;(^|[[:space:][:punct:]])_[^_]+_([[:space:][:punct:]]|$)&quot;
# /italic/  (Sometimes gets directory names)
color body         blue            default         &quot;(^|[[:space:][:punct:]])/[^/]+/([[:space:][:punct:]]|$)&quot;
mono body          underline                       &quot;(^|[[:space:][:punct:]])/[^/]+/([[:space:][:punct:]]|$)&quot;

# Border lines.
color body          blue            default         &quot;( *[-+=#*~_]){6,}&quot;

# vim: filetype=muttrc
</pre>
<p><strong>mutt gmail patch</strong></p>
<pre class="brush: diff; collapse: true; gutter: false; light: false; title: ; toolbar: true; notranslate">
--- imap/command.c.orig	2012-03-07 22:59:11.784404517 -0800
+++ imap/command.c	2012-03-07 22:59:25.517273315 -0800
@@ -66,6 +66,7 @@
   &quot;LOGINDISABLED&quot;,
   &quot;IDLE&quot;,
   &quot;SASL-IR&quot;,
+  &quot;X-GM-EXT-1&quot;,

   NULL
 };
--- imap/message.c.orig	2010-08-24 09:34:21.000000000 -0700
+++ imap/message.c	2012-03-07 22:59:25.517273315 -0800
@@ -242,8 +242,10 @@
       char *cmd;

       fetchlast = msgend + 1;
-      safe_asprintf (&amp;cmd, &quot;FETCH %d:%d (UID FLAGS INTERNALDATE RFC822.SIZE %s)&quot;,
-                     msgno + 1, fetchlast, hdrreq);
+      safe_asprintf (&amp;cmd, &quot;FETCH %d:%d (%sUID FLAGS INTERNALDATE RFC822.SIZE %s)&quot;,
+                     msgno + 1, fetchlast,
+                     mutt_bit_isset (idata-&gt;capabilities, X_GM_EXT_1) ? &quot;X-GM-MSGID &quot; : &quot;&quot;,
+                     hdrreq);
       imap_cmd_start (idata, cmd);
       FREE (&amp;cmd);
     }
@@ -452,6 +454,12 @@
     }
   }

+  if (mutt_bit_isset (idata-&gt;capabilities, X_GM_EXT_1))
+  {
+    fprintf (msg-&gt;fp, &quot;X-Gm-Msgid:%llu\n&quot;, HEADER_DATA(h)-&gt;msgid);
+    fprintf (msg-&gt;fp, &quot;X-Gm-Permalink: https://mail.google.com/mail/#all/%llx\n&quot;, HEADER_DATA(h)-&gt;msgid);
+  }
+
   /* mark this header as currently inactive so the command handler won't
    * also try to update it. HACK until all this code can be moved into the
    * command handler */
@@ -1142,6 +1152,14 @@
       if ((s = msg_parse_flags (h, s)) == NULL)
         return -1;
     }
+    else if (ascii_strncasecmp (&quot;X-GM-MSGID&quot;, s, 10) == 0)
+    {
+      s += 10;
+      SKIPWS (s);
+      h-&gt;data-&gt;msgid = strtoull (s, NULL, 10);
+
+      s = imap_next_word (s);
+    }
     else if (ascii_strncasecmp (&quot;UID&quot;, s, 3) == 0)
     {
       s += 3;
--- imap/message.h.orig	2012-03-07 22:59:00.101513364 -0800
+++ imap/message.h	2012-03-07 22:59:29.586234445 -0800
@@ -37,6 +37,7 @@
   unsigned int parsed : 1;

   unsigned int uid;	/* 32-bit Message UID */
+  unsigned long long msgid;
   LIST *keywords;
 } IMAP_HEADER_DATA;

--- imap/imap_private.h.orig	2012-03-07 22:58:29.465804586 -0800
+++ imap/imap_private.h	2012-03-07 22:59:29.586234445 -0800
@@ -114,6 +114,7 @@
   LOGINDISABLED,		/*           LOGINDISABLED */
   IDLE,                         /* RFC 2177: IDLE */
   SASL_IR,                      /* SASL initial response draft */
+  X_GM_EXT_1,       /* Gmail IMAP Extensions */

   CAPMAX
 };
</pre>
]]></content:encoded>
			<wfw:commentRss>http://www.dctrwatson.com/2012/03/mutt-and-gmail/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>limits.conf and daemons on Ubuntu</title>
		<link>http://www.dctrwatson.com/2012/03/limits-conf-and-daemons-on-ubuntu/</link>
		<comments>http://www.dctrwatson.com/2012/03/limits-conf-and-daemons-on-ubuntu/#comments</comments>
		<pubDate>Fri, 16 Mar 2012 22:52:28 +0000</pubDate>
		<dc:creator>dctrwatson</dc:creator>
				<category><![CDATA[RabbitMQ]]></category>
		<category><![CDATA[Ubuntu]]></category>
		<category><![CDATA[elasticsearch]]></category>
		<category><![CDATA[rabbitmq]]></category>
		<category><![CDATA[ubuntu]]></category>

		<guid isPermaLink="false">http://www.dctrwatson.com/?p=98</guid>
		<description><![CDATA[I recently was setting up a couple ElasticSearch and RabbitMQ instances when I noticed RabbitMQ was still reporting an abysmally low fd limit in its log file at startup. I double checked my /etc/security/limits.conf and sure enough, limits were properly set to 64000. Yet for some reason it was still only seeing a max of [...]]]></description>
			<content:encoded><![CDATA[<p>I recently was setting up a couple <a href="http://www.elasticsearch.org" target="_blank">ElasticSearch</a> and <a href="http://www.rabbitmq.com" target="_blank">RabbitMQ</a> instances when I noticed RabbitMQ was still reporting an abysmally low fd limit in its log file at startup. I double checked my <strong>/etc/security/limits.conf</strong> and sure enough, limits were properly set to 64000. Yet for some reason it was still only seeing a max of 1024.</p>
<p>It turns out that in <a href="http://releases.ubuntu.com/lucid/" target="_blank">Ubuntu 10.04</a>, <strong>/etc/pam.d/common-session{,-noninteractive}</strong> does not contain:<br />
<code><br />
session required pam_limits.so<br />
</code></p>
<p>Adding that, solved my issue:<br />
<code><br />
=INFO REPORT==== 1-Feb-2012::00:05:47 ===<br />
Limiting to approx 63900 file handles (57508 sockets)<br />
</code></p>
<p><strong>UPDATE (Wed Apr 18 15:01:17 PDT 2012)</strong><br />
For RabbitMQ 2.8.x, the init script uses start-stop-daemon. Apply this patch:</p>
<pre class="brush: plain; light: true; title: ; notranslate">
--- /etc/init.d/rabbitmq-server.old	2012-04-18 21:54:05.852307662 +0000
+++ /etc/init.d/rabbitmq-server	2012-04-18 21:49:17.594182809 +0000
@@ -35,6 +35,8 @@
 RETVAL=0
 set -e

+[ -r /etc/default/${NAME} ] &amp;&amp; . /etc/default/${NAME}
+
 ensure_pid_dir () {
     PID_DIR=`dirname ${PID_FILE}`
     if [ ! -d ${PID_DIR} ] ; then
</pre>
<p>And then in <strong>/etc/default/rabbitmq-server</strong><br />
<code><br />
ulimit -n 65000<br />
</code></p>
]]></content:encoded>
			<wfw:commentRss>http://www.dctrwatson.com/2012/03/limits-conf-and-daemons-on-ubuntu/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Realtime Postfix stats aggregator</title>
		<link>http://www.dctrwatson.com/2011/12/realtime-postfix-stats-aggregator/</link>
		<comments>http://www.dctrwatson.com/2011/12/realtime-postfix-stats-aggregator/#comments</comments>
		<pubDate>Fri, 30 Dec 2011 02:58:23 +0000</pubDate>
		<dc:creator>dctrwatson</dc:creator>
				<category><![CDATA[Postfix]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[monitoring]]></category>
		<category><![CDATA[postfix]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.dctrwatson.com/?p=96</guid>
		<description><![CDATA[With a user base as large as Disqus&#8216;, there is a ton of new comment and reply notification emails to send. Indubitable there is a user that accidentally (maybe even purposefully) subscribes to extremely hot threads. When they start receiving a stream of emails, their email provider doesn&#8217;t appreciate the spike in traffic. They usually [...]]]></description>
			<content:encoded><![CDATA[<p>With a user base as large as <a href="http://disqus.com/about/">Disqus</a>&#8216;, there is a ton of new comment and reply notification emails to send. Indubitable there is a user that accidentally (maybe even purposefully) subscribes to extremely hot threads. When they start receiving a stream of emails, their email provider doesn&#8217;t appreciate the spike in traffic. They usually show their annoyance by temporarily blocking and then rate limiting our <a href="http://www.postfix.org/">Postfix</a> instance from relaying email to that inbox.</p>
<p>Unfortunately, the only decent Postfix stats aggregators I could find were written in <a href="http://jimsun.linxnet.com/postfix_contrib.html">Perl (pflogsumm.pl)</a> and consumed log files for some ad-hoc stats generation. Though, I was quite lazy after trying a few variations and finding the same Perl tools over and over so please leave a comment about your favorite Postfix stats aggregator.</p>
<p>I really didn&#8217;t need anything too fancy so I decided to take a stab at it myself. After thinking about it for a few minutes, I decided to try out using Python threading to have a small pool of workers run some regex on a queue of lines from syslog. All the stats are then gathered in a dictionary and either spit out to stdout or there is a <strong>VERY</strong> simple TCP server thread listening for &#8216;stats&#8217; or &#8216;prettystats&#8217; to dump the current cumulative stats as a JSON dictionary. Full readme can be found on the Github page. Best part, it requires no 3rd party libraries.</p>
<p><a href="https://github.com/disqus/postfix-stats">postfix-stats on Github</a><br />
<a href="http://pypi.python.org/pypi/postfix-stats">postfix-stats on PyPi</a></p>
<p>View more to see example output&#8230; <span id="more-96"></span></p>
<pre class="brush: jscript; title: ; notranslate">
{
  &quot;recv&quot;: {
    &quot;status&quot;: {
      &quot;sent&quot;: 279518
    },
    &quot;resp_codes&quot;: {
      &quot;2.0.0&quot;: 279518
    }
  },
  &quot;clients&quot;: {
    &quot;209.85.210.170&quot;: 150,
    &quot;209.85.216.170&quot;: 67,
    &quot;74.125.82.170&quot;: 55,
    &quot;204.231.192.41&quot;: 198,
  },
  &quot;local&quot;: {
    &quot;notifications&quot;: 28129
  },
  &quot;send&quot;: {
    &quot;status&quot;: {
      &quot;deferred&quot;: 26739,
      &quot;bounced&quot;: 25406,
      &quot;sent&quot;: 259183
    },
    &quot;resp_codes&quot;: {
      &quot;4.7.1&quot;: 2295,
      &quot;2.0.0&quot;: 255801,
      &quot;5.1.1&quot;: 4817,
      &quot;4.2.0&quot;: 16,
      &quot;5.4.4&quot;: 7,
      &quot;4.0.0&quot;: 9661,
      &quot;5.4.6&quot;: 3,
      &quot;4.1.1&quot;: 75,
      &quot;4.1.8&quot;: 7,
      &quot;4.2.1&quot;: 261,
      &quot;5.7.0&quot;: 2,
      &quot;2.1.5&quot;: 22,
      &quot;5.5.0&quot;: 1,
      &quot;5.2.2&quot;: 15416,
      &quot;4.7.0&quot;: 1,
      &quot;5.2.1&quot;: 24,
      &quot;2.5.0&quot;: 1218,
      &quot;5.3.0&quot;: 2,
      &quot;5.0.0&quot;: 4898,
      &quot;2.6.0&quot;: 2142,
      &quot;4.4.1&quot;: 11122,
      &quot;4.4.3&quot;: 2706,
      &quot;4.4.2&quot;: 570,
      &quot;4.2.2&quot;: 5,
      &quot;5.7.1&quot;: 236,
      &quot;4.3.5&quot;: 2,
      &quot;4.3.2&quot;: 9,
      &quot;4.3.0&quot;: 9
    }
  },
  &quot;in&quot;: {
    &quot;status&quot;: {
      &quot;sent&quot;: 28129
    },
    &quot;resp_codes&quot;: {
      &quot;2.0.0&quot;: 28129
    }
  }
}
</pre>
]]></content:encoded>
			<wfw:commentRss>http://www.dctrwatson.com/2011/12/realtime-postfix-stats-aggregator/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>django-celery, eventlet and debugging blocking</title>
		<link>http://www.dctrwatson.com/2011/12/django-celery-eventlet-and-debugging-blocking/</link>
		<comments>http://www.dctrwatson.com/2011/12/django-celery-eventlet-and-debugging-blocking/#comments</comments>
		<pubDate>Fri, 30 Dec 2011 02:11:16 +0000</pubDate>
		<dc:creator>dctrwatson</dc:creator>
				<category><![CDATA[Celery]]></category>
		<category><![CDATA[Django]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[celery]]></category>
		<category><![CDATA[django]]></category>
		<category><![CDATA[eventlet]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.dctrwatson.com/?p=94</guid>
		<description><![CDATA[I recently wrote a couple Celery tasks that are purely IO bound. So instead of using the default multiprocessing execution pool, I used the Eventlet execution pool. With just a small change in Celery settings, I was off to the races. Wrong! After some amount of time, it just sits at 100% CPU and no [...]]]></description>
			<content:encoded><![CDATA[<p>I recently wrote a couple <a href="http://celeryproject.org/">Celery</a> tasks that are purely IO bound. So instead of using the default multiprocessing execution pool, I used the <a href="http://eventlet.net/">Eventlet</a> execution pool. With just a small change in <a href="http://celery.readthedocs.org/en/latest/userguide/concurrency/eventlet.html">Celery settings</a>, I was off to the races.</p>
<p>Wrong! After some amount of time, it just sits at 100% CPU and no longer processes tasks. Unfortunately, Celery calls monkey_patch a little late when coupled with <a href="https://www.djangoproject.com/">Django</a>. Django does some magic of its own to various components so monkey_patch needs to be called before Django does any initialization. After a little digging, I found I can just set an environment variable to prevent Celery from doing the monkey patching and at the same time use it to signal manage.py to call monkey_patch before the initialization my Django app.</p>
<p><span id="more-94"></span></p>
<p>Just add this to the top of your <strong>manage.py</strong>:</p>
<pre class="brush: python; title: ; notranslate">
if os.environ.get('EVENTLET_NOPATCH'):
    import eventlet
    import eventlet.debug

    eventlet.monkey_patch()
    eventlet.debug.hub_prevent_multiple_readers(False)

    eventlet_timeout = os.environ.get('EVENTLET_TIMEOUT')
    if eventlet_timeout:
        eventlet.debug.hub_blocking_detection(True, float(eventlet_timeout))
</pre>
<p>Now when starting celeryd just add the environment variable <code>EVENTLET_NOPATCH='yes'</code> to your manage.py command<br />
<code><br />
EVENTLET_NOPATCH='yes' python ./manage.py celeryd -c 1000 -Q eventlet_tasks -P eventlet<br />
</code></p>
<p>Also, if you&#8217;re worker seems to be running a little slow, you can now add <code>EVENTLET_TIMEOUT='1.0'</code> to cause Eventlet to print a stacktrace of the blocking thread to stderr. <a href="http://eventlet.net/doc/modules/debug.html#eventlet.debug.hub_blocking_detection">hub_blocking_detection</a> takes a float in number of seconds to set the alarm for.<br />
<code><br />
EVENTLET_TIMEOUT='0.1' EVENTLET_NOPATCH='yes' python ./manage.py celeryd -c 10 -Q eventlet_tasks -P eventlet<br />
</code></p>
]]></content:encoded>
			<wfw:commentRss>http://www.dctrwatson.com/2011/12/django-celery-eventlet-and-debugging-blocking/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>OS X System Equalizer</title>
		<link>http://www.dctrwatson.com/2011/06/os-x-system-equalizer/</link>
		<comments>http://www.dctrwatson.com/2011/06/os-x-system-equalizer/#comments</comments>
		<pubDate>Sat, 11 Jun 2011 21:30:02 +0000</pubDate>
		<dc:creator>dctrwatson</dc:creator>
				<category><![CDATA[OS X]]></category>
		<category><![CDATA[os x]]></category>

		<guid isPermaLink="false">http://www.dctrwatson.com/?p=86</guid>
		<description><![CDATA[I don&#8217;t have a music collection to listen to using iTunes. I listen to music through online streaming services like Pandora and Slacker. The downside to this is that Google Chrome doesn&#8217;t have a built-in audio equalizer like iTunes. I have a decent pair of headphones, Audio-Technica ATH-M50, and the default line-out from an Apple [...]]]></description>
			<content:encoded><![CDATA[<p>I don&#8217;t have a music collection to listen to using <a href="http://www.apple.com/itunes/">iTunes</a>. I listen to music through online streaming services like <a href="http://www.pandora.com">Pandora</a> and <a href="http://www.slacker.com">Slacker</a>. The downside to this is that <a href="http://www.google.com/chrome">Google Chrome</a> doesn&#8217;t have a built-in audio <a href="http://www.apple.com/itunes/features/#musicplayback">equalizer like iTunes</a>. I have a decent pair of headphones, <a href="http://www.audio-technica.com/cms/headphones/0edf909675b1be4d/index.html">Audio-Technica ATH-M50</a>, and the default line-out from an Apple desktop machine has always sounded a little &#8220;empty&#8221; to me. Like they slightly tweak it for normal/smaller headphones. Today I finally figured out how to get a global system equalizer for OS X (for free) so I can push the bass up a little to compensate for the &#8220;emptiness&#8221;.<br />
<span id="more-86"></span></p>
<p>These steps are done on a Snow Leopard iMac (10.6.7 specifically) but I&#8217;m pretty sure they would work on a Tiger/Leopard (10.4/10.5) machine as well.</p>
<p>First thing is to install <a href="http://cycling74.com/products/soundflower/">Soundflower</a> which requires a restart. It&#8217;s similar to <a href="http://www.rogueamoeba.com/audiohijackpro/">Audio Hijack Pro</a> in the fact that it adds virtual sound devices but without any other capabilities like recording/sfx/etc (and it&#8217;s freeware).</p>
<p>Next you will need is AU Lab from Apple. It&#8217;s a bundled application with Xcode so either <a href="http://developer.apple.com/xcode/">download it from Apple</a> or<a href="http://youtu.be/YhiRzCCK2t0"> install it from your OS X Install DVD</a> if you don&#8217;t want to deal with the multi-GB download.</p>
<p>And that&#8217;s it for software requirements!</p>
<h4>Step 1) Change Audio Output Device</h4>
<p>Change the output device in System Preferences->Sound, choose the Output tab. You should see Soundflower (2ch) in the list of devices. Select that one.<br />
<a href="http://www.dctrwatson.com/wp-content/uploads/2011/06/Sound_Preferences.png"><img src="http://www.dctrwatson.com/wp-content/uploads/2011/06/Sound_Preferences.png" alt="OS X Sound Preferences" title="OSX Sound Preferences" width="500" height="375" class="aligncenter size-full wp-image-87" /></a></p>
<h4>Step 2) Open AU Lab</h4>
<p>Easiest to search for it in Spotlight or it&#8217;s default location is at /Developer/Applications/Audio/AU Lab.app<br />
<a href="http://www.dctrwatson.com/wp-content/uploads/2011/06/AU_Lab_Location.png"><img src="http://www.dctrwatson.com/wp-content/uploads/2011/06/AU_Lab_Location-1024x433.png" alt="AU Lab Location" title="AU Lab Location" width="500" height="375" class="aligncenter size-large wp-image-88" /></a></p>
<h4>Step 3) Setup Input and Output devices in AU Lab</h4>
<p>You&#8217;ll be prompted with &#8220;Document Configuration&#8221;, use the Factory Configuration &#8220;Stereo In/Stereo Out&#8221;. Change the Audio Input Device to &#8220;Soundflower (2ch)&#8221; and the Audio Output Device to &#8220;Built-in Output&#8221;. Finally, click Create Document.<br />
<a href="http://www.dctrwatson.com/wp-content/uploads/2011/06/AU_Lab_Doc_Settings.png"><img src="http://www.dctrwatson.com/wp-content/uploads/2011/06/AU_Lab_Doc_Settings.png" alt="AU Lab Document Settings" title="AU Lab Document Settings" width="500" height="375" class="aligncenter size-full wp-image-89" /></a></p>
<h4>Step 4) Add EQ Effect</h4>
<p>Under &#8220;Ouput 1&#8243; apply the effect Apple->AUGraphicEQ<br />
<a href="http://www.dctrwatson.com/wp-content/uploads/2011/06/AU_Lab_EQ_Effect.png"><img src="http://www.dctrwatson.com/wp-content/uploads/2011/06/AU_Lab_EQ_Effect.png" alt="AU Lab EQ Effect" title="AU Lab EQ Effect" width="500" height="375" class="aligncenter size-full wp-image-90" /></a></p>
<h4>Step 5) Edit the EQ</h4>
<p>Tweak like mad with 31 bands or change it to 10 bands to get an EQ that looks more like the iTunes one.<br />
<a href="http://www.dctrwatson.com/wp-content/uploads/2011/06/AU_Lab_31_band_EQ.png"><img src="http://www.dctrwatson.com/wp-content/uploads/2011/06/AU_Lab_31_band_EQ.png" alt="AU Lab 31 band EQ" title="AU Lab 31 band EQ" width="500" height="375" class="aligncenter size-full wp-image-91" /></a><br />
<a href="http://www.dctrwatson.com/wp-content/uploads/2011/06/AU_Lab_10_band_EQ.png"><img src="http://www.dctrwatson.com/wp-content/uploads/2011/06/AU_Lab_10_band_EQ.png" alt="AU Lab 10 band EQ" title="AU Lab 10 band EQ" width="500" height="375" class="aligncenter size-full wp-image-92" /></a></p>
<h4>Step 6) Save your preset</h4>
<p>The 3rd drop down from the left of the EQ window lets you save/change presets.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.dctrwatson.com/2011/06/os-x-system-equalizer/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>FiSH module for ZNC</title>
		<link>http://www.dctrwatson.com/2011/03/fish-module-for-znc/</link>
		<comments>http://www.dctrwatson.com/2011/03/fish-module-for-znc/#comments</comments>
		<pubDate>Wed, 09 Mar 2011 01:01:41 +0000</pubDate>
		<dc:creator>dctrwatson</dc:creator>
				<category><![CDATA[ZNC]]></category>
		<category><![CDATA[fish]]></category>
		<category><![CDATA[znc]]></category>
		<category><![CDATA[znc module]]></category>

		<guid isPermaLink="false">http://www.dctrwatson.com/?p=84</guid>
		<description><![CDATA[So a bunch of us were sitting around talking one night and decided just for the lulz we would also start using FiSH Encryption in one of our back back channels on IRC. Not everyone that is in the channel are neck-beards and use irssi; some even don&#8217;t use XChat. To easy the transition for [...]]]></description>
			<content:encoded><![CDATA[<p>So a bunch of us were sitting around talking one night and decided just for the lulz we would also start using <a href="http://fish.secure.la/">FiSH Encryption</a> in one of our back back channels on IRC. Not everyone that is in the channel are neck-beards and use <a href="http://irssi.org/">irssi</a>; some even don&#8217;t use <a href="http://xchat.org/">XChat</a>. To easy the transition for everyone, we setup a <a href="http://en.znc.in/wiki/ZNC">ZNC</a> server to be a bouncer that also handles the FiSH encryption (yes this can defeat the purpose of FiSH, that&#8217;s why everyone is required to use SSL when connecting.)</p>
<p>As we were setting up, those of us who already were using FiSH couldn&#8217;t tell who was talking encrypted and who wasn&#8217;t. So <a href="http://twitter.com/noah256">@noah256</a> brought up the idea of prefixing encrypted messages. Later he also realized this can be spoofed; so he suggested to also prefix unencrypted messages from expected encrypted targets. At the same time, we realized when trying to talk to people without FiSH installed, it wasn&#8217;t really easy to temporarily send unencrypted messages so we also added a prefix to tell the module to not encrypt the message.</p>
<p>I used the <a href="http://znc-msvc.googlecode.com/svn/trunk/flakes/fish.cpp">fish.cpp</a> from the <a href="http://en.znc.in/wiki/Fish">fish &#8211; ZNC wiki page</a>. For now, the behavior I added is hard coded. I hope when I have free time again, to make the prefixes and disable flag to be configurable.</p>
<p>My modified module can be found in my <a href="https://github.com/dctrwatson/znc-fish">znc-fish repo on GitHub</a></p>
<p><span id="more-84"></span></p>
<pre class="brush: plain; gutter: false; title: ; notranslate">
&quot;(e) &quot; denotes the message was received encrypted
&quot;(d) &quot; denotes the message was received unencrypted

If you send a message prefixed with &quot;-e&quot;,  it won't encrypt the message.
</pre>
]]></content:encoded>
			<wfw:commentRss>http://www.dctrwatson.com/2011/03/fish-module-for-znc/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>FreeBSD mfiutil and the Common RAID Format Spec</title>
		<link>http://www.dctrwatson.com/2011/02/freebsd-mfiutil-and-the-common-raid-format-spec/</link>
		<comments>http://www.dctrwatson.com/2011/02/freebsd-mfiutil-and-the-common-raid-format-spec/#comments</comments>
		<pubDate>Fri, 18 Feb 2011 22:35:45 +0000</pubDate>
		<dc:creator>dctrwatson</dc:creator>
				<category><![CDATA[FreeBSD]]></category>
		<category><![CDATA[freebsd]]></category>
		<category><![CDATA[lsi]]></category>
		<category><![CDATA[megaraid]]></category>
		<category><![CDATA[mfiutil]]></category>
		<category><![CDATA[RAID]]></category>

		<guid isPermaLink="false">http://www.dctrwatson.com/?p=81</guid>
		<description><![CDATA[Recently I have been setting up a few new boxes that have an LSI MegaRAID controller. As of FreeBSD 8.0, there is a much more friendly tool included called mfiutil. It saves from having to install sysutils/megacli and using its rather complicated arguments (well, at first, after using it for a little while they make [...]]]></description>
			<content:encoded><![CDATA[<p>Recently I have been setting up a few new boxes that have an <a href="http://www.lsi.com/channel/products/raid_controllers/index.html">LSI MegaRAID</a> controller. As of FreeBSD 8.0, there is a much more friendly tool included called <a href="http://www.freebsd.org/cgi/man.cgi?query=mfiutil&#038;sektion=8&#038;manpath=FreeBSD+8.0-RELEASE">mfiutil</a>. It saves from having to install <strong>sysutils/megacli</strong> and using its rather complicated arguments (well, at first, after using it for a little while they make sense). Now, I know I setup a RAID10 that spans 3, 2 disk arrays. However, mfiutil is reporting it as a RAID1 volume.</p>
<p><span id="more-81"></span></p>
<p>After setting up a RAID10 array using the WebBIOS tool and installing FreeBSD I ran <code>mfiutil show volumes</code> to make sure I was referencing the right virtual drive and to my astonishment I see:</p>
<pre class="brush: plain; light: true; title: ; notranslate">
mfi0 Volumes:
  Id     Size    Level   Stripe  State   Cache   Name
 mfid0 (  408G) RAID-1      64K OPTIMAL Enabled
</pre>
<p>Say WHAT?! That volume is <strong>NOT</strong> supposed to be RAID1. So I ran <code>mfiutil show config</code> to get some more details about my configuration.</p>
<pre class="brush: plain; light: true; title: ; notranslate">
mfi0 Configuration: 3 arrays, 1 volumes, 0 spares
    array 0 of 2 drives:
        drive 8 (  137G) ONLINE &lt;SEAGATE ST3146356SS 0005 serial=SERIAL&gt; SAS enclosure 1, slot 0
        drive 9 (  137G) ONLINE &lt;SEAGATE ST3146356SS 0005 serial=SERIAL&gt; SAS enclosure 1, slot 1
    array 1 of 2 drives:
        drive 10 (  137G) ONLINE &lt;SEAGATE ST3146356SS 0005 serial=SERIAL&gt; SAS enclosure 1, slot 4
        drive 11 (  137G) ONLINE &lt;SEAGATE ST3146356SS 0004 serial=SERIAL&gt; SAS enclosure 1, slot 5
    array 2 of 2 drives:
        drive 12 (  137G) ONLINE &lt;SEAGATE ST3146356SS 0004 serial=SERIAL&gt; SAS enclosure 1, slot 6
        drive 13 (  137G) ONLINE &lt;SEAGATE ST3146356SS 0005 serial=SERIAL&gt; SAS enclosure 1, slot 7
    volume mfid0 (408G) RAID-1 64K OPTIMAL spans:
        array 0
        array 1
        array 2
</pre>
<p>Alright, that looks right but it <strong>still</strong> says RAID1 as well. Now this kind of bugs me, since I even followed the <a href="http://www.lsi.com/DistributionSystem/User/AssetMgr.aspx?asset=52770">MegaRAID SAS User Guide</a> to setup the RAID10 array. Alright, let&#8217;s get mfiutil to spit out some debug output; which requires <a href="http://unix.derkeiler.com/Mailing-Lists/FreeBSD/hackers/2010-10/msg00009.html">applying a patch and recompiling</a>.</p>
<pre class="brush: bash; title: ; notranslate">
cd /usr/src/usr.sbin/mfiutil
make clean
patch fix-mfiutil-debug.diff
make -DDEBUG
/usr/obj/usr/src/usr.sbin/mfiutil/mfiutil debug
</pre>
<p>Ok, now this looks right to me. Primary RAID level is 1 and secondary RAID level is 0.</p>
<pre class="brush: plain; light: true; title: ; notranslate">
mfi0 Configuration (Debug): 3 arrays, 1 volumes, 0 spares
  array size: 288
  volume size: 256
  spare size: 40
    array 0 of 2 drives:
      size = 285155328
        drive 8 ONLINE
          raw size: 286749488
          non-coerced size: 285700912
          coerced size: 285155328
        drive 9 ONLINE
          raw size: 286749488
          non-coerced size: 285700912
          coerced size: 285155328
    array 1 of 2 drives:
      size = 285155328
        drive 10 ONLINE
          raw size: 286749488
          non-coerced size: 285700912
          coerced size: 285155328
        drive 11 ONLINE
          raw size: 286749488
          non-coerced size: 285700912
          coerced size: 285155328
    array 2 of 2 drives:
      size = 285155328
        drive 12 ONLINE
          raw size: 286749488
          non-coerced size: 285700912
          coerced size: 285155328
        drive 13 ONLINE
          raw size: 286749488
          non-coerced size: 285700912
          coerced size: 285155328
    volume mfid0 RAID-1 OPTIMAL
      primary raid level: 1
      raid level qualifier: 0
      secondary raid level: 0
      stripe size: 7
      num drives: 2
      init state: 0
      consistent: 1
      no bgi: 0
      spans:
        array 0 @ 0 : 285155328
        array 1 @ 0 : 285155328
        array 2 @ 0 : 285155328
</pre>
<p>So why is it still reporting RAID1? Well let&#8217;s dive into the source of mfiutil. Ah-ha! The issue stems from <strong>mfi_cmd.c</strong></p>
<pre class="brush: cpp; first-line: 124; highlight: [133,134]; title: ; notranslate">
const char *
mfi_raid_level(uint8_t primary_level, uint8_t secondary_level)
{
        static char buf[16];

        switch (primary_level) {
        case DDF_RAID0:
                return (&quot;RAID-0&quot;);
        case DDF_RAID1:
                if (secondary_level != 0)
                        return (&quot;RAID-10&quot;);
                else
                        return (&quot;RAID-1&quot;);
</pre>
<p>Well, okay then. That is only partially correct. According to <a href="http://www.snia.org/tech_activities/standards/curr_standards/ddf/SNIA_DDF_Technical_Position_v2.0.pdf">Common RAID Disk Data Format Specification</a> that is an incorrect assumption. I dug a little deeper into the issue and found <a href="http://old.nabble.com/mfiutil-and-raid-level-td30586641.html">this thread</a> on the <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-current">freebsd-current mailing list</a>. The post that made it clear why mfiutil may make this assumption was by <a href="http://www.baldwin.cx/~john/">John Baldwin</a></p>
<pre class="brush: plain; light: true; title: ; notranslate">
Previous RAID-10 volumes that I've seen MFI BIOSes create used a non-zero
secondary raid level (they all used '3', which is what mfiutil uses to
create RAID-10 volumes itself). 

--
John Baldwin
</pre>
<p>I found a <a href="http://comments.gmane.org/gmane.linux.hardware.dell.poweredge/39365">thread</a> on the <a href="https://lists.us.dell.com/mailman/listinfo/linux-poweredge">Dell Linux-PowerEdge mailing list</a> and according to I believe a Dell rep, saying their PERC controllers (which is based on MegaRAID) reports RAID levels based on the SNIA DDF standard. So I followed his suggestion and started reading the SNIA DDF specification I linked above.</p>
<p>On Page 84:</p>
<hr />
<strong>4.3 Secondary RAID Level</strong></p>
<p>Table 15 lists values used in the Secondary_RAID_Level field of the Virtual Disk Configuration Record<br />
(Section 5.9.1) and their definitions. The table defines secondary RAID levels such as Striped, Volume<br />
Concatenation, Spanned, and Mirrored for hybrid or multilevel virtual disks.  The Secondary_RAID_Level<br />
field in the Virtual Disk Configuration Record MUST use the values defined in Table 15. </p>
<p><strong>Table 15: Secondary RAID Levels</strong></p>
<table>
<thead>
<tr>
<th>Name</th>
<th>SRL Byte</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>Striped</td>
<td>0&#215;00</td>
<td>Data is striped across Basic VDs. First strip stored on first BVD and<br />
next on next BVD.<br />
<strong>NOTE</strong>: BVD sequence is determined by the Secondary_Element_Seq<br />
field in the Virtual Disk Configuration Record (Section 5.9.1).</td>
</tr>
<tr>
<td>Mirrored</td>
<td>0&#215;01</td>
<td>Data is mirrored across Basic VDs.</td>
</tr>
<tr>
<td>Concatendated</td>
<td>0&#215;02</td>
<td>Basic VDs combined head to tail.</td>
</tr>
<tr>
<td>Spanned</td>
<td>0&#215;03</td>
<td>A combination of stripping and concatenations involving Basic VDs of<br />
different sizes.<br />
<strong>NOTE</strong>: BVD sequence is determined by the Secondary_Element_Seq<br />
field in the Virtual Disk Configuration Record (Section 5.9.1). </td>
</tr>
</tbody>
</table>
<hr />
<p>So now it all makes sense. The WebBIOS tool is using a secondary level of 0 because my VDs are all the same size so it has no reason to instruct the controller to use a secondary level 3. Now I don&#8217;t have to worry that I may of configured my array incorrectly.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.dctrwatson.com/2011/02/freebsd-mfiutil-and-the-common-raid-format-spec/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Allow users to use rabbitmqctl</title>
		<link>http://www.dctrwatson.com/2010/12/allow-users-to-use-rabbitmqctl/</link>
		<comments>http://www.dctrwatson.com/2010/12/allow-users-to-use-rabbitmqctl/#comments</comments>
		<pubDate>Mon, 27 Dec 2010 22:34:00 +0000</pubDate>
		<dc:creator>dctrwatson</dc:creator>
				<category><![CDATA[FreeBSD]]></category>
		<category><![CDATA[RabbitMQ]]></category>
		<category><![CDATA[rabbitmq]]></category>
		<category><![CDATA[sudoers]]></category>

		<guid isPermaLink="false">http://www.dctrwatson.com/?p=78</guid>
		<description><![CDATA[RabbitMQ is our message queue of choice for Celery. One of its main commands is rabbitmqctl; it lets you manage your rabbit nodes. The command itself is just a shell script that uses sudo -u rabbitmq to run an the Erlang rabbitmqctl program. It has to use sudo so that can it run as the [...]]]></description>
			<content:encoded><![CDATA[<p><a href="http://www.rabbitmq.com/">RabbitMQ</a> is our message queue of choice for <a href="http://celeryproject.org/">Celery</a>. One of its main commands is <code>rabbitmqctl</code>; it lets you manage your rabbit nodes. The command itself is just a shell script that uses <code>sudo -u rabbitmq</code> to run an the <a href="http://www.erlang.org/">Erlang</a> <a href="http://www.rabbitmq.com/man/rabbitmqctl.1.man.html">rabbitmqctl</a> program. It has to use sudo so that can it run as the rabbitmq user to use the same Erlang cookie that the broker is running with. </p>
<p>If you want to allow more than root or wheel group members to use the command, just use this sudoers privilege specification:<br />
<code><br />
ALL=(rabbitmq) SETENV: NOPASSWD: /usr/local/bin/erl<br />
</code></p>
<p>(rabbitmq) assumes your RabbitMQ broker runs as the &#8216;rabbitmq&#8217; user. It also assume you have Erlang installed in /usr/local</p>
]]></content:encoded>
			<wfw:commentRss>http://www.dctrwatson.com/2010/12/allow-users-to-use-rabbitmqctl/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>memcache-top patch to see curr_items</title>
		<link>http://www.dctrwatson.com/2010/12/memcache-top-patch-to-see-curr_items/</link>
		<comments>http://www.dctrwatson.com/2010/12/memcache-top-patch-to-see-curr_items/#comments</comments>
		<pubDate>Fri, 17 Dec 2010 04:58:59 +0000</pubDate>
		<dc:creator>dctrwatson</dc:creator>
				<category><![CDATA[Memcached]]></category>
		<category><![CDATA[Perl]]></category>
		<category><![CDATA[memcache-top]]></category>
		<category><![CDATA[memcached]]></category>
		<category><![CDATA[perl]]></category>

		<guid isPermaLink="false">http://www.dctrwatson.com/?p=75</guid>
		<description><![CDATA[I stumbled upon memcache-top in the Homebrew Formula repository. It&#8217;s a neat little Perl application that aggregate stats from a memcached cluster and displays it in a nice little table. I like to use: memcache-top --servers='server1,server2,server3' --commands --sleep=1 My only qualm is it doesn&#8217;t display the curr_items stat. So I wrote a little patch for [...]]]></description>
			<content:encoded><![CDATA[<p>I stumbled upon <a href="http://code.google.com/p/memcache-top/">memcache-top</a> in the <a href="https://github.com/mxcl/homebrew/tree/master/Library/Formula">Homebrew Formula repository</a>. It&#8217;s a neat little Perl application that aggregate stats from a <a href="http://memcached.org/">memcached</a> cluster and displays it in a nice little table.</p>
<p>I like to use: <code>memcache-top --servers='server1,server2,server3' --commands --sleep=1</code></p>
<p>My only qualm is it doesn&#8217;t display the <strong>curr_items</strong> stat. So I wrote a little patch for v0.6 that will add an <strong>ITEMS</strong> column to the table. </p>
<p>Get the patch from the <a href="http://code.google.com/p/memcache-top/issues/detail?id=8">issue I filed</a>.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.dctrwatson.com/2010/12/memcache-top-patch-to-see-curr_items/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>MySQL 5.5 Homebrew Formula</title>
		<link>http://www.dctrwatson.com/2010/12/mysql-5-5-homebrew-formula/</link>
		<comments>http://www.dctrwatson.com/2010/12/mysql-5-5-homebrew-formula/#comments</comments>
		<pubDate>Wed, 15 Dec 2010 02:19:54 +0000</pubDate>
		<dc:creator>dctrwatson</dc:creator>
				<category><![CDATA[Homebrew]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[OS X]]></category>
		<category><![CDATA[Ruby]]></category>
		<category><![CDATA[homebrew]]></category>
		<category><![CDATA[os x]]></category>
		<category><![CDATA[ruby]]></category>

		<guid isPermaLink="false">http://www.dctrwatson.com/?p=73</guid>
		<description><![CDATA[I recently switched to a 27&#8243; iMac workstation running Snow Leopard from Ubuntu 10.10 (Maverick Meerkat). The resident Mac fanbois suggested Homebrew over Fink and MacPorts. At first I didn&#8217;t heed their warnings and went with MacPorts (I do love the FreeBSD ports system of which MacPorts is based on). However, when it decided to [...]]]></description>
			<content:encoded><![CDATA[<p>I recently switched to a 27&#8243; iMac workstation running Snow Leopard from Ubuntu 10.10 (Maverick Meerkat). The resident Mac fanbois suggested <a href="http://mxcl.github.com/homebrew/">Homebrew</a> over <a href="http://www.finkproject.org/">Fink</a> and <a href="http://www.macports.org/">MacPorts</a>. At first I didn&#8217;t heed their warnings and went with MacPorts (I do love the FreeBSD ports system of which MacPorts is based on). However, when it decided to start setting up its own environment I understood the warnings. So I scrapped that and went with Homebrew.</p>
<p>Small issue with Homebrew is it seems to want to stick to &#8216;Stable&#8217; releases. I really didn&#8217;t want to use MySQL 5.1 since it&#8217;s a snail compared to MySQL 5.5. Not having an ultra-fast storage array in my iMac this would become an issue with all the work I have to do. So I set out into <a href="http://www.ruby-lang.org/en/">Ruby</a> land and banged out my first Homebrew formula.</p>
<p><del datetime="2011-09-29T17:23:02+00:00">Since it hasn&#8217;t been pulled into the official repo, here&#8217;s the latest commit: <a href="https://github.com/dctrwatson/homebrew/blob/5fe26cbc27eceb0955836f9b5434c3ed6ef0de76/Library/Formula/mysql55.rb">MySQL 5.5 Homebrew Formula</a></del></p>
<p><strong>Mainline Homebrew now has 5.5.x as the default MySQL version</strong></p>
<p>Updated: Oracle released MySQL 5.5.8; the first GA release of the 5.5 series.</p>
<p>Resources:<br />
<a href="https://github.com/mxcl/homebrew/wiki/Formula-Cookbook">Homebrew Formula cookbook</a><br />
<a href="http://dev.mysql.com/doc/mysql-sourcebuild-excerpt/5.5/en/index.html">Building MySQL [5.5] from Source</a></p>
]]></content:encoded>
			<wfw:commentRss>http://www.dctrwatson.com/2010/12/mysql-5-5-homebrew-formula/feed/</wfw:commentRss>
		<slash:comments>6</slash:comments>
		</item>
	</channel>
</rss>

