UI/API Test Framework Github Repos

  • March 30, 2016
  • BDD, behat, ...


Build Status



  • Install NodeJS
  • npm install -g phantomjs
  • npm install -g gulp
  • cd /repo root folder
  • node_modules/.bin/webdriver-manager update
  • node_modules/.bin/webdriver-manager start


  • Install NodeJS
  • Intsall GitSCM
  • Install Visual Studio Express
  • (probably good point for reboot)
  • From gitbash prompt:
    • npm install -g node-gyp
    • npm install -g gulp
  • node_modules/.bin/webdriver-manager update
  • node_modules/.bin/webdriver-manager start

    To run headless phantomjs

    • Download window phantomjs binary
    • Extract somewhere and rename phantomjs.exe to phantomjs
    • Copy to \Users[user id]\AppData\Roaming\npm
    • Change browserName: 'phantomjs' in conf.js file

How do I run tests?

from repo root folder run:

  • npm install (just once)
  • Symlink to api module: ln -s node_modules/apickli/apickli-gherkin.js e2e/step-definitions/api/apickli-gherkin.js
  • Always run export NODE_ENV={environment} prior to running any tests (e.g. export NODE_ENV=development) _By default, if this value of NODEENV is not set before running tests, the tests will run against development url. Currently the two settings available are 'development' and 'localhost'


  • Run gulp api to run API tests

UI (Using webdriver)

  • gulp webdriver-update
  • gulp webdriver-standalone
  • Run gulp ui to run UI tests
  • Run gulp api to run headless-browser UI tests

Run UI tests in parallel

  • Run gulp parallel-ui


  • Run gulp bs for regression tests on browserstack service

To view the the report after running the above gulp commands, run the following gulp tasks to generate the html report.

  • gulp clean-json
  • gulp protractor-report

** Or use shell scripts provided

  • Run ./ to run api tests
  • Run ./ to run ui tests
  • Run ./ to run ui tests in parallel mode
  • Run ./ to run api tests in parallel mode

Run selected test with optional tag

  • An optional tag can be passed during when the gulp task is invoked e.g gulp ci --tag @tagNameHere will only run tests tagged with @tagNameHere.
  • If the optional tag is missing the gulp task will run all tests set in the {{gulTaskConfig}}.config.js file as a fallback


Build Status

Key components:

Behat 3 Mink Extension PageObjects WebAPIContext


From Behat repo root folder run following commands:-


Before running behat to test the feature files in features directory, ensure the following commands are executed :-

  • java -jar selenium-server-standalone-[version].jar

To run tests (open another terminal window):-

  • bin/behat features

Second test runs using Guzzle (for API), the rest using Firefox


  • phantomjs --webdriver=4444
  • bin/behat -p phantomjs features


  • apt-get install parallel
  • java -jar selenium-server-standalone-2.43.1.jar --role hub
  • find features -iname '*.feature'| parallel --gnu -j5 --group bin/behat --ansi {}


Using saucelabs service, you can run tests against most OS/browser combinations and mobile platforms too.

I added an example profile for IE8, as example. To run it, first run sauceconnect config:-

  • bin/sauce_config saucelabs_user_id saucelabs_api_key

Now try running the tests ....

  • bin/behat -p saucelabs_ie8 features/

If you want to use saucelabs service against a localhost url, or any url behind a firewall, then follow these steps (assuming linux):

  1. sudo wget
  2. sudo tar -xvf sc-4.3.8-linux.tar.gz
  3. cd sc-4.3.8-linux
  4. bin/sc -u saucelabs_user_id -k saucelabs_api_key


As well as a html style report, there is a graphical report-based version using Twig. These are generated in the "reports" folder. Below is example of the Twig report output.


Build Status

Demo test application and protractor tests.


git clone
cd protractor-demo
npm install

To run

Get ChromeDriver set up: Run ./node_modules/.bin/webdriver-manager update.

Start the test application server with node app/expressserver.js Or if you're feeling lazy, just npm start.

Run the tests with node_modules/.bin/protractor test/conf.js Or npm test.


Behat with Docker


This repository is the source of docker-behat which brings :

  • a basic shell with oh-my-zsh
  • php5-cli with PHP 5.5
  • behat 3.0 / mink 1.5
  • all needed dependencies


docker pull jaffamonkey/docker-behat


docker run -t -i -h docker-behat-1 -v $(pwd):/home/behat:rw jaffamonkey/docker-behat /bin/zsh

You should see a prompt containing [ docker-behat ] and have behat command available.


If you need adapt the project to your needs, clone, modify the Dockerfile and from the source directory, run:

docker build -t docker-behat .


I am running simple test from the behat 3 kickstart (as you can see in last line in Dockerfile) - please read following for more guidance on running behat tests:

cucumberjs/protractor – dealing with non-angular popups

When creating cucumberjs/protractor test that involves dealing with a non-angular pop-up window, you can use combination of the getAllWindowHandles, and the ignoreSynchronization flag. The example below is a facebook SSO login (previous step was simply clicking link to login with facebook, to bring up the popup)

 this.Given(/^I login with facebook$/, function () {
        browser.getAllWindowHandles().then(function (handles) {
            var buttonName = 'fb login';
            // this switches focus of protractor the facebook login popup
            browser.ignoreSynchronization = true;
            // this tells protractor it's now a non-angularjs page
            fillField('facebook id', '{facebook login email');
            fillField('facebook password', '{facebook password}');
            return getVariable[buttonName.replace(/\s+/g, '')].click().then(function () {
                // this switches focus of protractor back to main angularjs window
                browser.ignoreSynchronization = false;
               // this tells protractor it's now a angularjs page

Waiting for elements to appear/disappear with cucumberjs/protractor

The cucumberjs/protractor combination can be awkward, testing angularjs, and using chai-as-promised helped, but reliability still can become an issue. So I started to lean towards more purist protractor code in these circumstances, to get round issue that waits are not always intelligent in cucumberjs and chai-as-promised didnt always provide solution (but is still great extension!). This following code has proved a reliable wait function.

this.Given(/^I wait for something to finish$/, function () {
function waitForElementToNotBePresent(elementToWaitFor) {
		var EC = protractor.ExpectedConditions;
		var el = element(;
		return browser.wait(EC.visibilityOf(el));
// Or to check if element is not there, return browser.wait(EC.invisibilityOf(el));

What is happening here is that protractor is waiting for the element to be be present or not, on page, before completing current step action.

What is the point of test automation?

  • February 23, 2016
  • Agile

Easily very little.  Whether it the most heinous and popular of crimes, the throwaway UI tests or (closely followed by) tests with test data and/or inter-test dependencies.  But even if all this is perfect, if it’s running on one machine, for one person, it’s just a good college project.  Or might as well be, for all the real use you get out of it.

The first step, after avoiding the perils of what’s outlined above, is to make it bigger. Drive tests using a specification language (i.e. natural language), but don’t forget to approach that as code, or too easily becomes a tangled library of accidental repetition.  The advantage here is quickly identifying common steps.  Everyone thinks their website UX is unique, but there is a lot that is common. Generating a good html report (with screenshots of failures), and a junit one for easy processing on systems such as Travis or Jenkins. Which brings me to important point – running tests on CI.

Integrating acceptance tests (which I what I generally call tester-generated automated tests).  It’s imperative to get maximum benefit from test automation.  The advantage of this approach, is it keeps tests lean and relevant and unique.  With no constrictions, you will no doubt have experienced testers or consultancies, who can generate hundreds of automated tests that provide “wow” factor, simply by quantity.  But run those same tests again and again, and the cracks will soon start to show.  Data dependencies, too many tests need updating with a code change, dependencies on other tests that start a cascade fail.  This is programming, remember, and same risks apply 🙂

While UI and API acceptance testing doesn’t replace unit testing, developers find the test framework useful in developing reusable tests (especially flexible API ones). Keeping in mind what is useful, and to who, help focus the mind.  My mind is very prone to veering off rapidly on tangents, so this was simple part of the way I learnt to avoid scatterbrained development.

Above all don’t play at it, do it. I have little patience for wasting time developing acceptance test frameworks that are unused.  Too many just “want it”, but don’t actually understand why, they just know it’s a “good thing”.  Which is why so many go through some consultancies pain before realising it can be a one-person job, well at least to kick-start it.

Manual tester are fast becoming a role that has become redundant in many ways, but others do not want to take it on. Hello to some PO’s and Stakeholders out there 😉 However, test frameworks and automation requires programming skills.  The best way to think about it is that’s it’s another developer on the team. (or more relevant test frameworks) can help bridge the old requirements to code gap.

Running Cucumberjs/Protractor tests using Phantomjs on Windows

Using gitbash ( from gitSCM) or similar Windows BASH emulation software:

  • npm install webdriver-manager
  • download window binary and extract somewhere
  • rename phantomjs.exe to phantomjs and copy to \\Users\[user id]\AppData\Roaming\npm

Change capabilities in conf file:

capabilities: {
browserName: 'phantomjs',
debug: true

webdriver-manager start

DISCLAIMER: I am not syaing this is the right way – this is the way I got the damn set-up to work on Windows 😉