Jenkins and Phabricator sitting in a tree

We’ve been using Phabricator for just about a year here at Disqus. It was originally created at Facebook and open sourced in Spring 2011. To sum it up using their own words: “Phabricator is a open source collection of web applications which make it easier to write, review, and share source code.” The small team working on it at Phacility (the SaaS company behind Phabricator) is constantly improving it so it’s on a continuous release cycle.

Jenkins has been used for continuous integration testing here for much longer. I’m not exactly sure for how long since it was setup before I started in September 2011. David Cramer has always been pushing for an ideal continuous integration/deployment system (IE here here) so part of my duties has been to improve what we have to achieve that goal (we’re hiring).

Currently, there isn’t a direct CI hook into Phabricator that is as deep as say Github+Travis. However, with a little script and an simple event listener for Arcanist, we can replicate most of that functionality.

Jenkins Example Diff

First things, first. Jenkins needs an account in Phabricator to perform actions on the differentials. Make sure the account has the role of System Agent. This keeps it from polluting the user autocomplete in various forms and allows administrative access to its conduit certificate.

Create Jenkins Phab User

You will need to install Arcanist wherever you plan to have Jenkins run these Phabricator builds (master and/or slaves nodes.) However, it doesn’t need the custom PhutilEventListener described below. Instead of running arc install-certificate on each node, it’s easier to just distribute the .arcrc with your Phabricator instance’s conduit uri, Jenkin’s Phabricator username and conduit certificate. Make sure to place .arcrc in Jenkin’s user’s home directory. A good way to test you have arc installed and configured properly is by running arc call-conduit user.whoami from the Jenkins node script console to double check it’s the right user. Make sure to change the conduit-uri to your instance.

Prettyfied response:

To notify Jenkins to do builds on differentials, we need a custom PhutilEventListener. It’s a super simple class with only a couple configuration options. The Arcanist user guide has instructions on how to Create a libphutil library.

Sample .arcconfig

Last but not least, setting up the Jenkins job. As there are numerous ways that jobs can be configured, I’m just going to outline a crude one that can be adapted to your own style. Your Jenkins installation will need at least the PostBuildScript Plugin and Python Plugin.

Since arc diff is going to be kicking off this job, it needs to be a parameterized build that takes a String Paramter DIFF_ID so it knows which patch to run.

Jenkins Parameterized Build

As we use Git for SCM at Disqus, setup the job to follow origin/master and under advanced, enable Prune remote branches before build and Clean after checkout to keep the repo tidy.

Jenkins SCM

There won’t be automatic build triggers since we’re using the event listener to trigger builds on arc diff.

Below is the shell script you can drop into an “Execute shell” build step that will handle applying the diff. This should be the first build step.

Now to post back to Phabricator with the results of the build. This needs to be added to the Post-build Actions. Otherwise, the results of the build won’t be available yet from the Jenkins API. Add “[PostBuildScript] – Execute a set of scripts” action with a build step of “Execute Python Script” of the following script (make sure Python-phabricator is in the PYTHON_PATH):

Huge thanks to Zameer Manji who interned with us Summer 2012 and was instrumental in getting this working!

  • Edson Chavez

    hi i following your tutorial i sucess in execute

    def proc = [“sh”, “-c”, “echo {} | arc call-conduit –conduit-uri=https://phabricator.example.com user.whoami”].execute()

    proc.waitFor()

    println proc.in.text

    here my reply:

    {“error”:null,

    “errorMessage”:null,

    “response”:{“phid”:”PHID-USER-mqq6stjwsm6jsyxqgiru”,

    “userName”:”jenkins”,

    “realName”:”jenkins”,

    “image”:”/res/1c5f2550/rsrc/image/avatar.png”,

    “uri”:”http://phabricator.mycentos.org/p/jenkins/”,

    “roles”:[“agent”,”verified”]}

    }

    i think this step is ok

    but i´m stuck in the execute shell step
    first my csv is svn then i made some changes in the script here my version:

    echo “## Getting base rev for diff”

    BASE_GIT_COMMIT=$(echo {“diff_id”: ${DIFF_ID}} | arc call-conduit differential.getdiff | awk -v RS=’,’ -v FS=':’ ‘$1~/”sourceControlBaseRevision”/ {print $2}’ | tr -d “)

    echo “## Reset to base commit”

    svn checkout -r ${BASE_GIT_COMMIT} http://10.10.2.112:1024/svn/evaluacion_phabricator/trunk/evaluacionPhabricator –username=jenkins –password=password

    echo “## Apply diff”

    arc patch –nobranch –no-ansi –diff $DIFF_ID –nocommit

    I stuck in the first step then i have a question: ${DIFF_ID} is the id for the revision? ‘d3′ (see picture) or the diff in the revision? ‘3’ (see picture 2)?

    i tried both and always this sentence return blank here the log from jenkins:

    + echo ‘## Getting base rev for diff’
    ## Getting base rev for diff
    ++ echo ‘{“diff_id”:’ ‘3}’
    ++ awk -v RS=, -v FS=: ‘$1~/”sourceControlBaseRevision”/ {print $2}’
    ++ tr -d ‘”‘
    ++ arc call-conduit differential.getdiff
    + BASE_GIT_COMMIT=
    + echo ‘## Reset to base commit’
    ## Reset to base commit
    + svn checkout -r http://10.10.2.112:1024/svn/evaluacion_phabricator/trunk/evaluacionPhabricator –username=jenkins –password=password
    svn: E205000: Syntax error in revision argument ‘http://10.10.2.112:1024/svn/evaluacion_phabricator/trunk/evaluacionPhabricator’
    Build step ‘Execute shell’ marked build as failure

    as you can see BASE_GIT_COMMIT is blank i miss something?

    regards
    Edson

    • http://www.dctrwatson.com John Watson

      DIFF_ID would be the 3 in picture 2. (The 3 in D3 is the revision_id)

      It looks right. I would try debugging the shell expansion for BASE_GIT_COMMIT.
      Try running from the node script console and see if it contains sourceControlBaseRevision or maybe even an error:
      def proc = [“sh”, “-c”, “echo {“diff_id”: 3} | arc call-conduit –conduit-uri=https://phabricator.example.com differential.getdiff”].execute()
      proc.waitFor()
      println proc.in.text

  • Craig Silverstein

    The phabricator folks mentioned to me that they’ve made recent improvements that can streamline this process a bit. In particular, you can just put JenkinsDiffEventListener into src/extensions, instead of building a libhutil lib.

  • http://www.deepdreamer.net Matthew Cunningham

    Hi John,

    Great post, although I do find I am having a few problems. My main one of which is with the Jenkins account on Phabricator part. I’ve followed you steps, even got my IT guys to set up and email specifically for the bot so I can get the registration email from Phabricator, but it’s not coming. Do the bots not get the welcome email?

    Thanks
    – Matthew

    • http://www.dctrwatson.com John Watson

      I do not believe bots get welcome emails. I want to say at some point, Evan made it so bots didn’t need them or added a way an admin could manually verify them.

  • http://jacob-walker.com jacobwalker0814

    Hi John. This article has helped me a bunch; thanks for writing it! I want to mention that “differential.getdiff” has been deprecated in favor of “differential.querydiffs”. I don’t know if this is a bug or what, but in my experience “differential.getdiff” does not always contain the property “sourceControlBaseRevision”.

    • http://jacob-walker.com jacobwalker0814

      Oh, uh…. I spoke too soon about getdiff not containing the “sourceControlBaseRevision”. That was a problem on my end. Either way I guess somebody might care that it’s deprecated :)

  • MIchael

    If you’re using git submodules, you might want to go with “git clean -ffdx” (note double f). This will clear out old git submodules if you delete them from the repo.

  • Aiden Scandella

    If anybody is looking for an easier way to accomplish this, we just open sourced a Jenkins plugin to perform this integration: https://github.com/uber/phabricator-jenkins-plugin