Cross-browser testing with Karma

This project is based on James Shore: Let's Code: Test Driven Javascript lessons (Chapter 7, Cross-Browser Testing). We will set up a client side testing framework using Grunt as task-runner, Karma as test-runner, Mocha as testing framework and Expect as an assertion library.

Setting up Karma

Karma is a test runner. You can test your javascript code on many browsers at the same time. The test process can be automated, and it can be run automatically as your code changing. As its home page says

Karma is essentially a tool which spawns a web server that executes source code against test code for each of the browsers connected. The results for each test against each browser are examined and displayed via the command line to the developer such that they can see which browsers and tests passed or failed.

Karma runs on Node.js. If you are working on a Windows machine, or you have not used Node.js yet, this post may help you try out these tools. Ok, so, let's install Karma.

(we suppose that you installed Node.js and you have a package.json file in your working directory)

$ npm install karma --save-dev

After installation make a script which can launch Karma:

#!/bin/sh

# karma.sh
# This script launches Karma from your working directory

node_modules/karma/bin/karma $*  

Running Karma without any parameter we get this message:

$ ./karma.sh  
Command not specified.  
Karma - Spectacular Test Runner for JavaScript.

Usage:  
  node ./node_modules/karma/bin/karma 

Commands:  
  start [<configFile>] [<options>] Start the server / do single run.
  init [<configFile>] Initialize a config file.
  run [<options>] [ -- <clientArgs>] Trigger a test run.
  completion Shell completion for karma.

Run --help with particular command to see its description and available options.

Options:  
  --help     Print usage and options.
  --version  Print current version.

Now we can initialize Karma:

$ ./karma.sh init

Which testing framework do you want to use ?  
Press tab to list possible options. Enter to move to the next question.  
> mocha

Do you want to use Require.js ?  
This will add Require.js plugin.  
Press tab to list possible options. Enter to move to the next question.  
> no
WARN [init]: Failed to install "karma-mocha"  
  Please install it manually.

Do you want to capture any browsers automatically ?  
Press tab to list possible options. Enter empty string to move to the next question.  
>

What is the location of your source and test files ?  
You can use glob patterns, eg. "js/*.js" or "test/**/*Spec.js".  
Enter empty string to move to the next question.  
> test/**/_*_test.js
WARN [init]: There is no file matching this pattern.

>

Should any of the files included by the previous patterns be excluded ?  
You can use glob patterns, eg. "**/*.swp".  
Enter empty string to move to the next question.  
>

Do you want Karma to watch all the files and run the tests on change ?  
Press tab to list possible options.  
> no

Config file generated at "d:\dev\cross-browser-testing\karma.conf.js".

Install Mocha

We need to install Mocha (or other test framework) to use Karma. Besides Mocha, we install karma-mocha, an adapter for Mocha:

$ npm install mocha --save-dev
$ npm install karma-mocha --save-dev

Now we can start Karma server:

$ ./karma.sh start  
INFO [karma]: Karma v0.12.16 server started at http://localhost:9876/  
WARN [watcher]: Pattern "d:/dev/cross-browser-testing/test/**/_*_test.js" does not match any file.

As you can see Karma server started at http://localhost:9876/. I you run a browser, and type this url, the Karma connects to the browser:

$ ./karma.sh start  
INFO [karma]: Karma v0.12.16 server started at http://localhost:9876/  
WARN [watcher]: Pattern "d:/dev/cross-browser-testing/test/**/_*_test.js" does not match any file.  
INFO [Chrome 35.0.1916 (Windows 7)]: Connected on socket Kfl1H4gyhDzlFNQZ3eyJ with id manual-6936


You can connect as many browsers as you want (check out the available browsers here). If you want to launch a browser from Virtualbox (e.g. IE8 on Windows), here are some good tips about how to address localhost.

Run test

We have Karma server, but nothing to test yet. Let's install karma-expect, the Expect.js adapter first:

$ npm install karma-expect --save-dev

Then we have to add expect to frameworks in the Karma config file (karma.conf.js):

frameworks: ['mocha','expect'],

Let's write a simple test that compares two strings:

describe("A simple test", function(){  
  it("should run", function(){
    expect("foo").to.equal("foo");
  });
});

Now start again the Karma server (because we have modified its config file just now), and run the test in an other terminal (and make sure you connected at least one browser to Karma):

$ ./karma.sh run  
[2014-07-11 13:30:47.438] [DEBUG] config - Loading config d:\dev\cross-browser-testing\karma.conf.js

IE 9.0.0 (Windows 7): Executed 1 of 1 SUCCESS (0.034 secs / 0 secs)  
IE 8.0.0 (Windows 7): Executed 1 of 1 SUCCESS (0.028 secs / 0.001 secs)  
Chrome 35.0.1916 (Windows 7): Executed 1 of 1 SUCCESS (1.166 secs / 0.004 secs)  
TOTAL: 3 SUCCESS

Automation

We are going to use Grunt for task automation. The work flow is the following:

  • start Karma server
  • launch the browsers manually (if you use virtual machine for some exotic browsers, start the machine and launch those browsers there)
  • connect the browsers to the Karma server (see above)
  • start grunt
  • grunt runs Karma to test your code, and checks if the tests pass or fail

First you need Grunt. After installing Grunt let's run it:

$ grunt  
A valid Gruntfile could not be found. Please see the getting started guide for  
more information on how to configure grunt: http://gruntjs.com/getting-started  
Fatal error: Unable to find Gruntfile.

We have to create a simple Gruntfile and configure Karma (or if Gruntfile exists, add Karma to this). We'll accomplish this with Node.js child_process.exec():

// Gruntfle.js

module.exports = function(grunt) {

  grunt.registerTask('karmaRun', 'Running tests on Karma server', function(){

    var done = this.async(); 
    var childProcess = require("child_process");   
    var isWin = /^win/.test(process.platform);
        // http://stackoverflow.com/questions/8683895/variable-to-detect-operating-system-in-node-scripts
      var processName = isWin ? "karma" : "./karma";
      var execKarmaRun = childProcess.exec(processName + " run", function(error, stdout, stderr) { 
        console.log(stdout, stderr);
      // on success, error equals null
      if (error === null) done();  
      // passing false to the done() tells Grunt that the task has failed.
      else done(false); 
    });
  });

  // Default task(s).  
  grunt.registerTask('default', ['karmaRun']);
};

Grunt works synchronously, but it can be switched to asynchronous by calling this.async() within the task body (for more information see Grunt documentation).

You may want to get a feedback if the tests passed. According to the Node.js documentation the error argument - getting by the callback - will be null on success. All we need to do is passing false to the done() method to signal at Grunt if the tests failed.

If we start Karma server, launch our browsers and connect them, and run Grunt, then we can see the following:

$ grunt

Running "karmaRun" task  
[2014-07-19 23:20:21.778] [DEBUG] config - Loading config /Users/tompascall/dev/playground/cross-browser-testing/karma.conf.js
Mobile Safari 7.0.0 (iOS 7.1): Executed 1 of 1 SUCCESS (0.014 secs / 0.002 secs)  
IE 9.0.0 (Windows 7): Executed 1 of 1 SUCCESS (0.142 secs / 0.008 secs)  
IE 8.0.0 (Windows 7): Executed 1 of 1 SUCCESS (0.044 secs / 0.003 secs)  
Chrome 35.0.1916 (Mac OS X 10.9.4): Executed 1 of 1 SUCCESS (1.915 secs / 0.002 secs)  
Safari 7.0.5 (Mac OS X 10.9.4): Executed 1 of 1 SUCCESS (0.014 secs / 0.002 secs)  
Firefox 30.0.0 (Mac OS X 10.9): Executed 1 of 1 SUCCESS (0.091 secs / 0.028 secs)  
TOTAL: 6 SUCCESS


Done, without errors.

And now only with Chrome, let's see what happens if the test fails:

$ grunt

Running "karmaRun" task
[2014-07-20 15:42:26.374] [DEBUG] config - Loading config /Users/tompascall/dev/playground/cross-browser-testing/karma.conf.js Chrome 35.0.1916 (Mac OS X 10.9.4) Something should run FAILED
Error: expected 'foo' to equal 'fo' at Assertion.assert (/Users/tompascall/dev/playground/cross-browser-testing/nodemodules/karma-expect/nodemodules/expect.js/index.js:96:13) at Assertion.be.Assertion.equal (/Users/tompascall/dev/playground/cross-browser-testing/nodemodules/karma-expect/nodemodules/expect.js/index.js:216:10) at Context. (/Users/tompascall/dev/playground/cross-browser-testing/test/equaltest.js:8:22) at callFn (/Users/tompascall/dev/playground/cross-browser-testing/nodemodules/mocha/mocha.js:4338:21) at Test.Runnable.run (/Users/tompascall/dev/playground/cross-browser-testing/nodemodules/mocha/mocha.js:4331:7) at Runner.runTest (/Users/tompascall/dev/playground/cross-browser-testing/nodemodules/mocha/mocha.js:4728:10) at /Users/tompascall/dev/playground/cross-browser-testing/nodemodules/mocha/mocha.js:4806:12 at next (/Users/tompascall/dev/playground/cross-browser-testing/nodemodules/mocha/mocha.js:4653:14) at /Users/tompascall/dev/playground/cross-browser-testing/nodemodules/mocha/mocha.js:4663:7 at next (/Users/tompascall/dev/playground/cross-browser-testing/node_modules/mocha/mocha.js:4601:23) Chrome 35.0.1916 (Mac OS X 10.9.4): Executed 1 of 1 (1 FAILED) ERROR (1.871 secs / 0.003 secs)

Warning: Task "karmaRun" failed. Use --force to continue.

Aborted due to warnings.

Summary

Our aim was to set up a simple cross-browser testing framework, using Karma. We needed a test framework - it was Mocha - and some assertion library (I chose Expect.js). First we installed Karma and configured it, then installed Mocha and Expect.js adapter for Karma. After installations we were able to start Karma server and connect browsers to it. We wrote a simple test, ad ran it with Karma. We wanted to automate the test-running, using Grunt, the task runner. We configured a task that can launch Karma as a child process, and check if the tests pass or fail.

As I was working on setting up the framework, I tried grunt-karma, a Karma runner plugin for Grunt. It is really useful, you can simply configure Karma with this. After installing additional browser-launcher plugins, you don't need to start Karma and connect the browser manually, Grunt does the work, it starts the server, launches the given browsers, run the tests, closes the browsers and the Karma server (Single Run mode). Beyond that you can set - whenever your test files are being modified - to run Karma automatically. But I had difficulties when I tried to connect browsers launched in virtual machine (I need vm to run different kinds of Internet Explorers). The Karma runner couldn't see that browsers. I tried 'node-virtualbox', and a lot of plugins that works with virtual-machines, but I couldn't tune up them to work with grunt-karma plugin. After experimenting with grunt-karma, I left it and figured out a work flow, that needs to start the Karma server and the given browsers manually, but the testing can be automated by Grunt.

You can download the project from here.



comments powered by Disqus