Blog


Be your own SaaS

You take a few products from open source, integrate, customise. It’s still called development, and it still requires skill – that is is you want your test framework to be stable and grow. It needs skill, as you need to minimise product-dependencies ongoing, or you can quickly get lost in spiral of dependency failures, as you update core framework.

You can chase rainbows for the ultimate test automation tool – there are those using open source, with same dire “all your problems are over” style taglines for their SaaS.

While you are chasing your rainbow, I will have built a framework and maintained it for resusability and open to all to use – tech and non-tech people alike. By trusting a third-party vendor or SaaS, you constrict your team immediately to work a certain way. It limits creativity and relevance.




Testing REST API with Cukes




Test engineering

  • May 25, 2016
  • Agile

OK, let’s cut to the chase. Testers are part of your team, not an anonymised bolt-on. This was an attitude prevalent 20 years ago, but at least then, testers were taken seriously. What we have now is too many making evaluations of what testing is and where it should fit in. Testing is a free-for-all area that many self-appointed experts reside in. And the more testing becomes test engineering, the more foolish those people look.

Who (as a tester) has not heard the response to you job “what, you just sit around surfing website all day?”. Well, these people are not worth correcting, but it illustrates a point – even with 20 years QA experience, it amazes me how many feel need to tell me what I do, and why I do it. My twitter feed gauge my annoyance levels, full as it is, of strongly-worded diatribes on this subject. It is bad enough that in contracting, I am generally following in wake of amateur balls-ups. And balls-ups that went on for too long, as no-one took interest or monitoring the ongoing development of tests.

Part of problem is that testing world is full of enthusiasm, but few with real test engineering skills. Want to impress your average company? Create hundreds of brittle UI automated tests, built on latest “funky framework”, then bask in the multiple “oooooooh!”, then time exit before anyone notices that:

  • No-one is using the test framework repo, apart from the tester.
  • No-one else is checking out the repo.
  • The tester becomes “Manuel Jenkins”, as the test framework only ever running on their pc.
  • The hundreds of tests are pointless, useless and throwaway

Then I come in to mop up the mess, and deal with bad perceptions/understanding about test engineering. There is a problem around the decision-making process, usually made by project members who view a project like this:

Airplane

Generally, process and products are selected based on personal taste or even fashion (the worst). These are always sold with over-optimistic promises of improving things, simply by their existence on project. These helpful individuals may not even be connected with tech – maybe they just enjoyed a pretentious demonstration at a conference. And react like they have never heard the time-honoured sales line “All you problems are solved with this product!”. The people that decide test framework are the team, because they are the saps that have to use and maintain it.

Though a broad statement, test engineering is very effective bridge between development and devops. Yes, developers can factor in test framework maintenance and coding tests, into their workload. But why? It’s far better to have a well-maintained test framework they can use as necessary, rather than being concerned with it’s mechanics. Calling test engineering “developing in test” provided more confusion, as many role descriptions were essentially for a developer who was also prepared to write acceptance tests. By that, I mean tests that directly map to requirements, not unit tests.

If the test engineer is charged with creating and maintaining UI and API tests, this can save the developers a lot of unit test work, and if test framework set running on the build server, an excellent sanity check for every build. And based of real-life scenarios. The common whine when trying to get is that it gets in the way. Really? A failing test is an obstruction, and should be (whatever the reason it fails). Acceptance tests should be part of the build pipeline. Tests don’t obstruct, they inform.

I am generally following in wake of amateur balls-ups. And balls-ups that went on for too long, as noone took interest or monitoring the ongoing development of tests. Want to impress your average company? Create hundreds of brittle UI automated tests, bask on the multiple “oooooooh!”, then time exit before anyone notices that:

  • No-one is using the test framework repo, apart from the tester.
  • No-one else is checking out the repo.
  • The tester becomes “Manuel Jenkins”, as the test framework only ever running on their pc.
  • The hundreds of tests are pointless, useless and throwaway

Then I come in to mop up the mess, and deal with bad perceptions/understanding about test engineering. There is a problem around the decision-making process, usually made by project members who view a project like this:

Airplane

So process and products are selected based on personal taste, fashion (the worst). These are always sold with over-optimistic promises of improving things, simply by their existence on project.

Though a broad statement, test engineering is very effective bridge between development and devops. Yes, developers can factor in test framework maintenance and coding tests, into their workload. But why? It’s far wetter to have a well-maintained test framework they can use as necessary, rather than being concerned with it’s mechanics. Calling test engineering “developing in test” provided more confusion, as many role descriptions were essentially for a developer who was also prepared to write acceptance tests. By that, I mean tests that directly map to requirements, not unit tests.

If the test engineer is charged with creating and maintaining UI and API tests, this can save the developers a lot of unit test work, and if test framework set running on the build server, an excellent sanity check for every build. And based of real-life scenarios. The common whine when trying to get is that it gets in the way. Really? A failing test is an obstruction, and should be (whatever the reason it fails). Acceptance tests should be part of the build pipeline. Tests don’t obstruct, they inform.




Elementor: An improved element finder for protractor

elementor – An improved element finder for protractor

Source: andresdominguez/elementor: An improved element finder for protractor




UI/API Test Framework Github Repos

  • March 30, 2016
  • BDD, behat, ...

CucumberJS/Protractor

Build Status

README

PREP MAC/LINUX

  • Install NodeJS
  • npm install -g phantomjs
  • npm install -g gulp
  • cd /repo root folder
  • node_modules/.bin/webdriver-manager update
  • node_modules/.bin/webdriver-manager start

PREP WINDOWS

  • Install NodeJS
  • Intsall GitSCM
  • Install Visual Studio Express
  • (probably good point for reboot)
  • From gitbash prompt:
    • npm install -g node-gyp
    • npm install -g gulp
  • node_modules/.bin/webdriver-manager update
  • node_modules/.bin/webdriver-manager start

    To run headless phantomjs

    • Download window phantomjs binary
    • Extract somewhere and rename phantomjs.exe to phantomjs
    • Copy to \Users[user id]\AppData\Roaming\npm
    • Change browserName: 'phantomjs' in conf.js file

How do I run tests?

from repo root folder run:

  • npm install (just once)
  • Symlink to api module: ln -s node_modules/apickli/apickli-gherkin.js e2e/step-definitions/api/apickli-gherkin.js
  • Always run export NODE_ENV={environment} prior to running any tests (e.g. export NODE_ENV=development) _By default, if this value of NODEENV is not set before running tests, the tests will run against development url. Currently the two settings available are 'development' and 'localhost'

API

  • Run gulp api to run API tests

UI (Using webdriver)

  • gulp webdriver-update
  • gulp webdriver-standalone
  • Run gulp ui to run UI tests
  • Run gulp api to run headless-browser UI tests

Run UI tests in parallel

  • Run gulp parallel-ui

Browserstack

  • Run gulp bs for regression tests on browserstack service

To view the the report after running the above gulp commands, run the following gulp tasks to generate the html report.

  • gulp clean-json
  • gulp protractor-report

** Or use shell scripts provided

  • Run ./run_api_tests.sh to run api tests
  • Run ./run_ui_tests.sh to run ui tests
  • Run ./run_ui_tests_parallel.sh to run ui tests in parallel mode
  • Run ./run_api_tests_parallel.sh to run api tests in parallel mode

Run selected test with optional tag

  • An optional tag can be passed during when the gulp task is invoked e.g gulp ci --tag @tagNameHere will only run tests tagged with @tagNameHere.
  • If the optional tag is missing the gulp task will run all tests set in the {{gulTaskConfig}}.config.js file as a fallback

Behat/Mink

Build Status

Key components:

Behat 3 Mink Extension PageObjects WebAPIContext

SETUP

From Behat repo root folder run following commands:-

RUNNING SELENIUM BROWSER TESTS

Before running behat to test the feature files in features directory, ensure the following commands are executed :-

  • java -jar selenium-server-standalone-[version].jar

To run tests (open another terminal window):-

  • bin/behat features

Second test runs using Guzzle (for API), the rest using Firefox

RUNNING PHANTOMJS TESTS

  • phantomjs --webdriver=4444
  • bin/behat -p phantomjs features

PERFORMANCE/PARALLEL TESTING

  • apt-get install parallel
  • java -jar selenium-server-standalone-2.43.1.jar --role hub
  • find features -iname '*.feature'| parallel --gnu -j5 --group bin/behat --ansi {}

CROSS BROWSER

Using saucelabs service, you can run tests against most OS/browser combinations and mobile platforms too.

I added an example profile for IE8, as example. To run it, first run sauceconnect config:-

  • bin/sauce_config saucelabs_user_id saucelabs_api_key

Now try running the tests ....

  • bin/behat -p saucelabs_ie8 features/

If you want to use saucelabs service against a localhost url, or any url behind a firewall, then follow these steps (assuming linux):

  1. sudo wget https://saucelabs.com/downloads/sc-4.3.8-linux.tar.gz
  2. sudo tar -xvf sc-4.3.8-linux.tar.gz
  3. cd sc-4.3.8-linux
  4. bin/sc -u saucelabs_user_id -k saucelabs_api_key

REPORTING

As well as a html style report, there is a graphical report-based version using Twig. These are generated in the "reports" folder. Below is example of the Twig report output.

Jasmine/Protractor

Build Status

Demo test application and protractor tests.

Setup

git clone https://github.com/jaffamonkey/protractor-jasmine.git
cd protractor-demo
npm install

To run

Get ChromeDriver set up: Run ./node_modules/.bin/webdriver-manager update.

Start the test application server with node app/expressserver.js Or if you're feeling lazy, just npm start.

Run the tests with node_modules/.bin/protractor test/conf.js Or npm test.

CucumberJS/Protractor(Basic)

Build Status

cucumberjs with protractor example

  • npm install
  • npm run Protractor

Generate html report

  • grunt cucumber_html_report

Run tests through saucelabs cross-browser SaaS

  • npm run saucelabs

Behat with Docker

docker-behat

This repository is the source of docker-behat which brings :

  • a basic shell with oh-my-zsh
  • php5-cli with PHP 5.5
  • behat 3.0 / mink 1.5
  • all needed dependencies

Install

docker pull jaffamonkey/docker-behat

Usage

docker run -t -i -h docker-behat-1 -v $(pwd):/home/behat:rw jaffamonkey/docker-behat /bin/zsh

You should see a prompt containing [ docker-behat ] and have behat command available.

Build

If you need adapt the project to your needs, clone, modify the Dockerfile and from the source directory, run:

docker build -t docker-behat .

behat

I am running simple test from the behat 3 kickstart (as you can see in last line in Dockerfile) - please read following for more guidance on running behat tests:

https://github.com/jaffamonkey/behat-3-kickstart/blob/master/README.md