Setting up Test Swarm

Near the end of 2011 ( before I disappeared for 2 weeks ), the Popcorn.js team and I went out for an end of year dinner.  Throughout dinner a discussion came up about how our testing infrastructure could be improved and some of the things that would be nice to have.  Something that stuck with me that David Humphrey mentioned was how we could benefit from continuous integration within our tests.  Continuous integration is essentially creating a way to test our code as we are developing it.  This means running manual tests less frequently, being able to identify bugs sooner, and prevent regressions from occurring.  After coming off my 2 - 3 months working primarily on improving Popcorn's documentation, I figured I could use a serious change of pace.  So at the beginning of 2012 I told myself that this is what I wanted to do.

One of the difficult issues that arose right out of the gate was which CI ( continuous integration ) framework did I want to use, and more importantly, which one was right for us.  After doing a bunch of googling a came up with numerous different options, but none of them seemed right for what we wanted.  Most of the one's that came close failed in one major aspect, that they didn't test on actual browsers.  What I mean by this is a lot of the options that came close offered there ghetto webkit rendition that the tests would run on.  The problem lies in the fact that some tests pass on lets say Firefox, but fail on Safari.  We can't rely on a single point of testing and needed a way to continue to test on all of the modern browsers.  After a bit more searching I came across an option that seemed to fit our needs and was even written by John Resig ( previous Mozilla employee and creator of JQuery ), so I had high hopes.

The project was called Test Swarm and described as the following:

The primary goal of TestSwarm is to take the complicated, and time-consuming, process of running JavaScript test suites in multiple browsers and to grossly simplify it. It achieves this goal by providing all the tools necessary for creating a continuous integration workflow for your JavaScript project.

After talking it over with a Jon Buckley and Rick Waldron a bit ( fellow Popcorn.js devs ) we decided to give it a shot.  Rick was awesome enough to get us a few minutes to talk with Corey Frang ( member of the JQuery Core and UI teams ) to lend us a bit of his expertise in the area of setting up Test Swarm.  Corey and Rick both cautioned that setting it up was not a simple task and they ended up being right as it took me about a week of work to get to  where I am now.  Initially I searched for other's who had set up test swarm and the problems that they encountered because I figured if they hit these issues I was sure too.  I was surprised that there really wasn't a detailed guide to setting up Test Swarm and to learn that it isn't that widely used.  So I started working with what I had, which was the README from the Test Swarm repo and a small set of instructions on the wiki page.

Setting up the server, for the most part, wasn't that bad.  I ran into a few problems trying to set up the server on my localhost and ended up throwing it up on a web server for simplicities sake ( I'd recommend doing so for any else who is trying this as well ). The main problem that I ran into in this part of the installation was configuring apache correctly.  Up to this point in my exposure to actual programming ( what I refer to as about the last year of my life ) I have just ignored setting up apache for myself and most of the time got help from others when I needed something.  When I started working on this project it turned out that I no longer had to luxury of remaining ignorant.  I ran into a slew of problems with permissions, how .htaccess files worked, and how to enable a rewrite module.  If anyone is wondering, all of this work was done on Ubuntu.

The first thing problem I listed was setting up permissions properly with apache ( and Ubuntu for that matter ).  This was a problem because I needed certain scripts to be able to call other scripts, perform git actions, write to a databases, and so on.  This involved A LOT of googling in order to figure out just how all of this is done.  For the most part, the searching was pretty straightforward in the fact that I could essentially google what I was thinking and get helpful results.  The next issue I ran into was working with the .htaccess files.  From what I learned, a .htaccess file is essentially a configuration file on a per directory basis.  This means I can create a file to specify options that are only applied in the context of that directory.  By default, .htaccess files are not used and have to be turned on in apache settings ( from my reading online, it seems .htaccess files are frowned upon for some reason ).  According to the README, all I needed to do was change the AllowOverride attribute from None to All.  Alright great, but HOW do I do that and where exactly should I look?  I started of by searching for "AllowOverride" in the main apache config file and found a few instances of it kicking around and figured I would just set all of those to All instead of None and see what happened.  I was hoping to figure out which one I needed to actually be set to All by process of elimination, but as it turns out, none of them were the right one to set to All.  Well, as of this point, I had never touched another apache config file so it was back to Google for me.  After about a day of searching I finally found out what I needed to do via a helpful blog post I found.  The post outlined exactly what I needed to do and after actually doing it I had a fair bit better understanding of what I needed to do. At this point I had our Test Swarm server up and running.  The next step was to set up a git post commit hook and create jobs based on each commit.

The way we envisioned this all working was that when a commit is pushed to our develop branch that it would automatically checkout that commit and run all of our tests.  After the tests have finished we would get a report of some sort back with the results of the tests that were run.  You can imagine how this is benefitial to any project, especially the bigger they get.  Popcorn.js currently has ~1500 tests in total, so running those tests can become quite time consuming ( especially when you factor in a complete and minified version ).  The next step that I needed to take was to create a git hook the would send our server information each time a commit was made to our develop branch.  I did this by doing the following:

  1. Set up the post-commit hook on github to send a POST message to our server when something was committed
  2. Create a php file that would receive the POST message.  All this file did essentially was check to see if the branch that the commit came from was our develop branch ( only want to run tests based of develop ( for now at least ) ).  Once we verify that the commit came from develop, I call a bash script that I created passing along the URL to the git repo and the commit id as arguments.
  3. The bash file essentially created a git repo ( with the passed in URL ) with a directory named after the commit.  Naming the directory after a commit is essential here because if numerous commits come in, we need to separate them and not have them overwrite one another.  I then checkout the commit and call a perl script.
  4. Inside our Popcorn test suite I had to add an injection script in order to report passes/fails back to the swarm when I test was done.  This was added inside our project repo and not on the server.  The script itself is pretty small ( ~150 lines ) and is included in the Test Swarm repo in the  js/ directory.
  5. The perl script does most of the heavy lifting for us.  The Test Swarm repo actually has a few sample scripts setup ( they fortunately already had one setup for QUnit! ) so I used it as a starting point.  For the most part it was as simple as replacing there directory paths and options with what we needed.  I had to modify a bit of stuff as it just didn't work otherwise ( for some reason there was code in there to shorten the commit id, which didn't work when trying to locate our repo ).
  6. That was essentially it in a nutshell.  There was a ton of debugging that went along with the process but I actually didn't have to write much code at all.

As of right now I am able to commit to my Popcorn repo, have that commit automatically pulled in, a job created based off that commit, and that job be pushed out to all of the connected clients.  The tests are then run and the results are reported.  The next things I need to get done are:

  • removing a git repo when that commit's tests have finished. There is no need to keep old repos around, it is only taking up disk space.
  • set up a few dedicated boxes here at work each running VM's for all of the browsers that we tests off of.  The reason for this is that we have many tests that will fail if the browser is not in focus ( which is why we need to run a VM for each browser on each OS ).
  • The tests are running, and I am getting results back, but in our report chart, even if a test fully passes the square still shows up as black ( should be green ).  I image this is probably a problem with the injection script and how we are reporting the amount of passes/fails back.
  • It would be nice to not have to run all of the plugin tests if a commit only pertains to core tests and vice versa.  This would mean that we would get results back faster and new commits could be tested sooner.
  • Im sure there are a ton of things that I am missing

All in all this has been a great experience and an awesome change of pace from writing documentation. Once this is fully up and running I think everyone will benefit from it, I mean lets be honest, no one likes sitting there manually running tests!  Having a system such as this should make releases let strenuous and ( hopefully ) we shouldn't be presented with any surprise bugs come release day.