what is the difference between QA and Testing?

First of all what is the difference between QA and Testing? Is there one?

Standard IEEE definitions are:-
TESTING means “quality control”
QUALITY ASSURANCE measures the quality of processes used to create a quality product.

This is fair assessment, and it is Quality Assurance that this book is focussed on, as testing is simply a derivative a QA process. What has inspired/driven me to write this book, is to sum up my experiences into a guide for other QA Managers to deal with the unusual and bizarre – the world of RAD, of which Agile is the most popularist exponent.  Before moving onto this topic, lets see if we can define Agile. This is a challenge as the perception of what agile is differs from company to company, and individual to individual.

This is a good overview of the aims of agile, but the biggest error is to take this methodology and attempt to build a complete project around it. The main problem with methodologies that claim to encompass a complete project lifecycle, is the dangerous assumption everyone is working off the same page; that everyone understands the implications and possible caveats.  Agile has been dogged by theory and tools, rather than focussing on real issue with modern software development – communication.  I have been in several bizarre meeting where the topic has been communication issues.  Unfortunately the upshot of these meetings is usually “lets email even more” or worse “we are right, they are wrong”. The customer is the driving force of a software project, be it an internal or external one.

The raison d’etre of agile is to develop fast, with minimal interference from business.  The focus is on the customer, but the customer is rarely involved in the day-to-day Agile process.  We can see the assumptions mounting up, so rather than following a more critical view of Agile, I am suggesting that development are doing what is required, it is other project members that are not adapting their important roles.

The traditional role of QA management, has been adapted over time by necessity, rather than any guidelines.  And the additional responsibilities the QA Manager has, extends into release and project management (these roles have been diluted over time, especially on web/media project.  The question I still have for myself, during consultancy work, is should I be following company policy rather than defining my own path? They hired a manager, and the assumption is that you can think for yourself, and go against the grain if required. QA requires a thick skin, and complete faith in your beliefs!

Agile – Customer Focussed?

I believe the negative flipside of Agile, is that though the positives of rapid development, creativity are core to the methodology, the other core principle that that the customer/client (be it internal or external) can be lost in that environment. In such situations more traditional process, and communication can salvage the project, without damaging the overall Agile process.

As with most new methodologies, there is mistaken assumption that by simply adopting the methdology in development will somehow work. I reality, all project members have to be working not only to same guidelines, but also not to forget that each project member represents his/her own project area. Project Management and QA are usually the common areas of failure in Agile environments, which of course leads to damage to relationships external to a company.

There are tools specific to Agile, and MSProject is not my first choice as a project management tool in any case. I always found dotproject was a better integrated project/QA management tool, which is more suited to rapidly evolving Agile projects. However the core of Agile principles is more to do with a way of doing things, rather than software enabling this to happen.

Nothing new has really been introduced, as the idea of development and testing working closely together is just sound software development practice, and shorter release cycles is more appropriate the demands of modern software projects. What is new is better accountability of development, and more interactivity between a company and their client. i.e. more transparency generally in the project lifecycle.

This information was based on my own experience, and in media/gaming way, QA has always been more of a challenge, due to the creative and emotional level of people that work in these industries. Automation is heavily featured in Agile methodology, but in my experience the biggest challenge with Agile is people, not technology.

Open Source Automated Test Software

One of the most impressive uses of test tools I saw in action was the usage of open source test tool STAF (Software Test Automated Framework) in conjunction with Junit. The main advantage was that the company was able to create a very relevant load test environment, on a lower budget. It also does not require expensive hardware to run on.

STAF is a load test tool primarily, and allows you to build a load test framework, and uses XML configuration files to create different test scenarios. It is very scalable, and we were able to simulate 100k transactions spread over a 2/3 day period. This is by no means to limit of the system’s capability. It requires good technical skills and buy-in from project development team, as their input and customisation work is key to successful implementation of STAF.

Agile Complacency

It appears that Agile methodology works well for development by default, but can create issues in other areas, notably support and data storage. As the benchmarks can be changed rapidly on an agile project, this information needs to be filtered down rapidly to prevent problems at product launch stage.

One solution to alleviate this potential pitfall is to keep trainers, support and DBA’s in the agile communication loop. In the end, its the people who have to work with the software ongoing that will experience the knock-on negative effects of Agile. Looking at Agile from higher level, it is easier to see where communication can break down, and where dangerous assumptions are made.

This is not a negative view of Agile, but more of a note that evolvement of the methodology must continue, as well as education to broader scope of people.

Agile vs QA

I am interested in testing opinions on Agile Methodology, as I have been working in test management in these “challenging” environments for a few years now. My main issue with Agile is that the projects tend to isolate themselves from business, in mistaken belief that development “knows best”. Though the software development methodologies (based largely on open source movement) are currently exciting and innovative, it’s application is subject to numerous interpretations. The main error is assumption that agile is new, and it’s not, and most/all of the principles are rewriting core programming principles, for modern software development demands. Would be interested to hear experiences from QA/Test people – positive and negative. I regularly find myself defining QA processes in these environments, as there are few standards from QA perspective.

Chris Garrett (Blogger, Online Marketing and web development consultant)
I’m not a QA/test person but I find any methodology will only work with the full buy in and communication of all the people who are (and should be) involved. Agile is fashionable, particularly in the current “everything beta, all the time” environment but it is as good to follow as any other and success will have more to do with personalities and expertise as what approach is used.

Cynthia J Perkins PMP (Program Manager – Information Technology)
I am not a testing expert, but I find your insights on Agile very interesting. I just started working at Klipsch (maker of speakers) and we have been asked to review the tool to perhaps use it in place of MSProject. The engineers currently use it for product roll-out. What version of Agile are you using? I have not started my research yet. It’s on my “to-do” list.


Cathryn Symons (Independent IT Project Manager)

I am a project manager, and we’re using agile in my current project (Scrum). We have business staff embedded in the team, which means that it hasn’t become isolated from the business. However, I do see some concerns, particularly around keeping a longer term view of the entire project (as funded) rather than just the current sprint, and making sure that dependencies are clear. I’m not convinced that Agile alone is enough to manage an entire project, though can see definite advantages for the software development side. Not having to have the requirements defined in detail up front is good, and it seems to lead to far better team working between the developers and business in our case.
Olivier Azeau (Engineering Project Manager at Varian Medical Systems)
In his “Collaborative Software Testing” blog (link 1). Jonathan Kohl regularly writes about QA in agile environments. The “Who do we serve” couple of articles (link 2) is a must read to understand the not-so-easy position of a tester in an environment which claims to focus on testing at developer’s level.

As far as I am concerned, I am not a QA/Test person but I think agile methodologies are good for product quality. The tricky part for the QA/Test people is to be able to become, in some way, ubiquitous : standing beside the developer to avoid the tunneling effect of waterfall while staying in touch with the real users world. I wrote about this issue in “Every process is an iterative waterfall” (link 3).



Justin Brister (Director of Technology, Virtusa)
Interesting question; In general, Agile does not call for a separate QA function. Continuous integration, test driven development and prototyping all form an integrated part of the process. In highly complex deployment environments, you may still require a more formal integration testing. Projects which isolate themselves form the end users are not by definition Agile, as Customer involvement is a key aspect of the methodology. It sounds like the issues that you are encountering are down to the fact that traditional waterfall practices are still being used. It doesn’t matter how Agile the development teams try to be, unless you have strong project management and total commitment from the business, you will fail.

Things get even more interesting when you throw offshoring into the mix.


Ben Levin (Manager, User Experience Architecture – FSG)
Having been through a couple of Agile projects ($1MM+ development budgets), and coming from a User-Centered Design background (which shares similar goals of continuous testing, continuous customer involvement), I can say that the methodology is quite exciting, and also about as prone to failure as any other. You certainly can’t do it half-assed, but to get “total committment” you need everyone (from the people doing the development to the team that supports them to the business stakeholders who are paying for it) to buy in to several assumptions:

1. That it’s better to launch something that’s useful than to spend time figuring out exactly what you should build and why.
2. That your development team cost is more or less “fixed”. By this I mean Agile tries to optimize the work of the development team towards maximizing the amount of working code they can produce for the amount that they’re paid. This is more or less a given when you’re in a corporation with programmers who are on staff, who might not be doing anything productive for the organization if they’re not focused on developing the next itteration of your product. It isn’t quite as much of a given when you’re outsourcing your development, and even less so when you are the outsourced developer.
3. The business stakeholders have a good reason for wanting to develop something, and have thought through the cost/benefit analysis.

There are a lot of other very valid opinions about whether or not Agile is “good” or “bad”. My overall judgement is that it’s as good as the planning and forethought that goes into using it. In the end, it’s just a methodology, and no methodology can guarantee success (no matter how strictly you follow it.)
Kerry Buckley (Software developer at BT)
Given that one of the principles of agile development is that “Business people and developers must work together daily throughout the project.”, I’d suggest that the projects you’ve experienced weren’t really agile at all.


Heath Newburn (Product Manager, Motivational Speaker, Change Agent, QA Geek)
There is a Yahoo Group on agile methodologies, it’s pretty much hit or miss. The real issue is treating QA as a separte part of Agile methodologies. It has to go hand-in-hand with each cycle. Unformtunately, I haven’t seen it work in any sort of scale.


Elisabeth Hendrickson (Owner, Quality Tree Software, Inc. and Computer Software Consultant)
I’m a testing consultant who works with Agile teams. I’ve spoken and written a fair bit about my experiences and insights. The links I’ve included below go to my Google Tech Talk, my PNSQC conference paper “Agility for Testers”, and a collection of writings on my blog. From my blog, you’ll find links to other experts in Agile QA/Testing including Lisa Crispin, Brian Marick, and (as mentioned by Olivier earlier) Jonathan Kohl.


Markus Mikkolainen (CEO at kabuto,Software professional, a jack of all trades and master of some)
The main idea of most agile practises is constant user feedback enabled with constant workable versions which are the result of constant integration.

So if your developers arent communicating with the customer, then you are losing the whole point and the most important gain from using agile methodologies. The customer must be in the loop and giving you feedback after every iteration/sprint , this way the customer will be able to refine his view on where the software should go and he will get a warm and fuzzy feeling that software is getting completed. This will also give the client and immediate control on the timetable, do you want this now or that later? do we do the feature or an immediate release?

Testingwise the challenge in agile methods is knowing what to test since usually there isnt an up-to-date req.spec/test plan. You will have to work from a list of cards (XP) written by the customer and you have to update the test plan on the fly. On the other hand you will be able to do testing most of the time and you can (and should) be in the loop with the developers so they can fix small bugs asap and big ones can be listed on the backlog of things to do.

Agile processes are the best way to combat an environment where the requirements keep coming in and the environment changes very fast.. which is unfortunately all too common nowadays. Also in a case where the customer doesnt really know what they want and what are the possibilities of the solution, it is very useful to be able to get the customer in-loop with the developers and keep getting input on up-to-date versions of the software.

btw. usually if development isolates itself that is a sign of a broken process in the people who are trying to feed the development with too much stuff. Usually this is sales or product development whole have gotten into a feature creep loop or are selling non-existant features. ie. have no idea of how to manage software development, yet they are trying to manage it ..either directly or via unofficial channels.

Phil Goodwin (Software author, engineer, and mentor)
“My main issue with Agile is that the projects tend to isolate themselves from business, in mistaken belief that development “knows best”.”

This is a misconception that I hadn’t encountered before. One of the common threads that run through Agile methodologies is the clear definition and separation of business decisions and engineering decisions. Agile projects are supposed to be run as a cooperative effort with business people providing information about the content and priority of requirements and developers providing information about the available strategies and costs of implementing the requirements.

“I would be interested to hear experiences from QA/Test people – positive and negative. I regularly find myself defining QA processes in these environments, as there are few standards from QA perspective.”

Agile methodologies, and Extreme Programming in particular, define a testing strategy that does not specifically include a traditional QA role, but which specify activities that seem to me to be ideally suited to being carried out by individuals with a traditional QA skill set. Automated unit testing and Test Driven Design are core Agile practices that seem to obviate the need for external testing. In some situations there is a close enough relationship between developers and the project’s ultimate consumers that the code quality assurances provided by a unit test suite are adequate support for a successful launch. However, on larger projects, particularly when there is a strong need for a positive initial customer reaction to new releases there is a substantial benefit to having dedicated QA personnel on the project.

You hinted strongly at their role in your initial comments. There is a need for the business side of the house to be able to confirm that the software produced meets the actual requirements of the system in which it is to fit. A QA organization brings the expertise that allows the software’s conformance to specification to be measured and quantified. It also continues to play the traditional role of causing the articulated requirements to converge on the actual requirements through a process of consecutive approximation. There’s no replacement for a bloody-minded tester torturing a system with inputs that only a madman, genius or end-user could possibly come up with to flush out hidden requirements (having a good writer document the system will do the same thing along an entirely different axis.) A QA team will also bring the technical skills and discipline to ensure that acceptance testing is automated and synchronized with build process in a way that produces repeatable results and useful diagnostic data. In an Agile environment a QA team can play a very strong supporting role to the business responsibilities of defining and, of course, testing acceptance criteria. In addition they also serve as a proxy for the On-site Customer as defined by Extreme Programming.

I am including a link to the site for the Fitnesse testing tool. The material on this site should give a good gut feel for the way that acceptance testing, which is close to traditional QA testing, fits into some existing Agile projects, and the Fitnesse tool itself has a good reputation in some circles for its ability to provide non-technical business people to express requirements in a testable format.


James Eager (Database Administrator / Data Analyst at Federal Express National LTL)
From a development standpoint, Agile is a rehash of something we used before. We used to call it “prototyping” back in the mid 80s.

From the tech support (DBA) side, Agile is a disaster. We have a couple of projects runnng now. These projects consume 3 times the tech support resources that the regular projects do. They are constantly re-changing the same tables, requiring re-work on our part to fix the things they didn’t start out with.
Matt Davidson (R&D Engineer – Math Geek)
Agreed, from a database perspective it is a nightmare. From previous experience I would say it is better to apply agile or “refactoring” techniques to the software rather than it’s data source.

For applications development with teams of a dozen or less on a single project, I’ve found success with a combination of automated unit tests, and a principle of refactoring using higher level languages like .net to prototype UI’s etc… while the real work is done with C++ core libraries.

Mostly, I’d say it is common sense, don’t over design, don’t under design. Everything is balance.


Nik Clayton (Vice President / Senior Engineer at Citigroup)
You wrote “My main issue with Agile is that the projects tend to isolate themselves from business, in mistaken belief that development “knows best”.”

That seems to me go against one of the core tenents of any methodology that claims to be agile, in that the customer must be kept in the loop, with regular deliveries of functionality to the customer so that they can confirm that what’s being developed meets their needs.

And that’s without going in to the initial “customer story” design phase in which the customer’s initial requirements are supposed to be captured (on 5×3 index cards if you’re a purist…)


Thierry Jeremie (Aegeva – President/Founder)
I have being in the SQA world for quite sometimes now, and found that like everything else there is room for waterfall or iterative methodology. I have worked in a mixed environment, and for a formal implementation (ERP, CRM,…) the waterfall method fits well. However, for fast development and new pre-ipo projects, Agile is a great fit, cutting on the time to market and also having the ability to quickly inject customer enhancements in the product at a low cost enabling the new venture to take off quickly and gain market shares.

For the XP flavor of Agile, the all idea of getting the testing done first calls for Test Engineers that are bread between Developers and Test Engineers. Using as an example the Java framework they should be at ease with Java, Junit, Subversion, Cruisecontrol, ANT,… However they need to also be fluent in formal QA practice once a build(s) reach the integration phase for formal certification, such as: performance, load and stress testing. In some instances, Agile can be so intense, demanding, and fascinating that compliance and governance can be easily forgotten… And if misused Agile could dilute milestones in a large number of iterations with a end result of having an application delivery that is incomplete or late.

Finally, just as any other methodology upper management needs to be educated and understand the benefits as well as cost/time/resource needed for implementing continuous integration processes and tools to support Agile.


Paul Davis (Sr Java Developer at Idea Integration)
“the projects tend to isolate themselves from business, in mistaken belief that development “knows best”.”
– This looks like a mis-understanding.
Most agile process stress iterations where MORE time is spent with the customer. Customers decide priority of features, etc.. So, if you turn that statement around you’ll be closer to the truth.

On to agile in general. Its more about facing reality.
The reality is, nobody really has an effective process for predictably manging software development. How can you when, most development involves a large about of innovation and when the productivity difference your best and 2nd best developer can be 10x.
Agile methodologies basic just accept the reality and cope with it through iterations. Each iteration increases the amount of predictability of the project.
Its just a stepping stone. As time goes by, and the knowledge base increases, we’ll learn more about how to ‘manage’ development and develop better processes.

On the QA side, the problem of so few standards comes from the extreme variation of the quality of QA in organizations. Some have really high quality with people who really understand what quality assurance is while others get starbucks rejects who just click links. Many business owners aren’t willing to commit the amount of resources needed for good QA. The result of this is that QA can be a tough career choice so, few choose it.