Agile vs QA

I am interested in testing opinions on Agile Methodology, as I have been working in test management in these “challenging” environments for a few years now. My main issue with Agile is that the projects tend to isolate themselves from business, in mistaken belief that development “knows best”. Though the software development methodologies (based largely on open source movement) are currently exciting and innovative, it’s application is subject to numerous interpretations. The main error is assumption that agile is new, and it’s not, and most/all of the principles are rewriting core programming principles, for modern software development demands. Would be interested to hear experiences from QA/Test people – positive and negative. I regularly find myself defining QA processes in these environments, as there are few standards from QA perspective.

Chris Garrett (Blogger, Online Marketing and web development consultant)
I’m not a QA/test person but I find any methodology will only work with the full buy in and communication of all the people who are (and should be) involved. Agile is fashionable, particularly in the current “everything beta, all the time” environment but it is as good to follow as any other and success will have more to do with personalities and expertise as what approach is used.

Cynthia J Perkins PMP (Program Manager – Information Technology)
I am not a testing expert, but I find your insights on Agile very interesting. I just started working at Klipsch (maker of speakers) and we have been asked to review the tool to perhaps use it in place of MSProject. The engineers currently use it for product roll-out. What version of Agile are you using? I have not started my research yet. It’s on my “to-do” list.


Cathryn Symons (Independent IT Project Manager)

I am a project manager, and we’re using agile in my current project (Scrum). We have business staff embedded in the team, which means that it hasn’t become isolated from the business. However, I do see some concerns, particularly around keeping a longer term view of the entire project (as funded) rather than just the current sprint, and making sure that dependencies are clear. I’m not convinced that Agile alone is enough to manage an entire project, though can see definite advantages for the software development side. Not having to have the requirements defined in detail up front is good, and it seems to lead to far better team working between the developers and business in our case.
Olivier Azeau (Engineering Project Manager at Varian Medical Systems)
In his “Collaborative Software Testing” blog (link 1). Jonathan Kohl regularly writes about QA in agile environments. The “Who do we serve” couple of articles (link 2) is a must read to understand the not-so-easy position of a tester in an environment which claims to focus on testing at developer’s level.

As far as I am concerned, I am not a QA/Test person but I think agile methodologies are good for product quality. The tricky part for the QA/Test people is to be able to become, in some way, ubiquitous : standing beside the developer to avoid the tunneling effect of waterfall while staying in touch with the real users world. I wrote about this issue in “Every process is an iterative waterfall” (link 3).



Justin Brister (Director of Technology, Virtusa)
Interesting question; In general, Agile does not call for a separate QA function. Continuous integration, test driven development and prototyping all form an integrated part of the process. In highly complex deployment environments, you may still require a more formal integration testing. Projects which isolate themselves form the end users are not by definition Agile, as Customer involvement is a key aspect of the methodology. It sounds like the issues that you are encountering are down to the fact that traditional waterfall practices are still being used. It doesn’t matter how Agile the development teams try to be, unless you have strong project management and total commitment from the business, you will fail.

Things get even more interesting when you throw offshoring into the mix.


Ben Levin (Manager, User Experience Architecture – FSG)
Having been through a couple of Agile projects ($1MM+ development budgets), and coming from a User-Centered Design background (which shares similar goals of continuous testing, continuous customer involvement), I can say that the methodology is quite exciting, and also about as prone to failure as any other. You certainly can’t do it half-assed, but to get “total committment” you need everyone (from the people doing the development to the team that supports them to the business stakeholders who are paying for it) to buy in to several assumptions:

1. That it’s better to launch something that’s useful than to spend time figuring out exactly what you should build and why.
2. That your development team cost is more or less “fixed”. By this I mean Agile tries to optimize the work of the development team towards maximizing the amount of working code they can produce for the amount that they’re paid. This is more or less a given when you’re in a corporation with programmers who are on staff, who might not be doing anything productive for the organization if they’re not focused on developing the next itteration of your product. It isn’t quite as much of a given when you’re outsourcing your development, and even less so when you are the outsourced developer.
3. The business stakeholders have a good reason for wanting to develop something, and have thought through the cost/benefit analysis.

There are a lot of other very valid opinions about whether or not Agile is “good” or “bad”. My overall judgement is that it’s as good as the planning and forethought that goes into using it. In the end, it’s just a methodology, and no methodology can guarantee success (no matter how strictly you follow it.)
Kerry Buckley (Software developer at BT)
Given that one of the principles of agile development is that “Business people and developers must work together daily throughout the project.”, I’d suggest that the projects you’ve experienced weren’t really agile at all.


Heath Newburn (Product Manager, Motivational Speaker, Change Agent, QA Geek)
There is a Yahoo Group on agile methodologies, it’s pretty much hit or miss. The real issue is treating QA as a separte part of Agile methodologies. It has to go hand-in-hand with each cycle. Unformtunately, I haven’t seen it work in any sort of scale.


Elisabeth Hendrickson (Owner, Quality Tree Software, Inc. and Computer Software Consultant)
I’m a testing consultant who works with Agile teams. I’ve spoken and written a fair bit about my experiences and insights. The links I’ve included below go to my Google Tech Talk, my PNSQC conference paper “Agility for Testers”, and a collection of writings on my blog. From my blog, you’ll find links to other experts in Agile QA/Testing including Lisa Crispin, Brian Marick, and (as mentioned by Olivier earlier) Jonathan Kohl.


Markus Mikkolainen (CEO at kabuto,Software professional, a jack of all trades and master of some)
The main idea of most agile practises is constant user feedback enabled with constant workable versions which are the result of constant integration.

So if your developers arent communicating with the customer, then you are losing the whole point and the most important gain from using agile methodologies. The customer must be in the loop and giving you feedback after every iteration/sprint , this way the customer will be able to refine his view on where the software should go and he will get a warm and fuzzy feeling that software is getting completed. This will also give the client and immediate control on the timetable, do you want this now or that later? do we do the feature or an immediate release?

Testingwise the challenge in agile methods is knowing what to test since usually there isnt an up-to-date req.spec/test plan. You will have to work from a list of cards (XP) written by the customer and you have to update the test plan on the fly. On the other hand you will be able to do testing most of the time and you can (and should) be in the loop with the developers so they can fix small bugs asap and big ones can be listed on the backlog of things to do.

Agile processes are the best way to combat an environment where the requirements keep coming in and the environment changes very fast.. which is unfortunately all too common nowadays. Also in a case where the customer doesnt really know what they want and what are the possibilities of the solution, it is very useful to be able to get the customer in-loop with the developers and keep getting input on up-to-date versions of the software.

btw. usually if development isolates itself that is a sign of a broken process in the people who are trying to feed the development with too much stuff. Usually this is sales or product development whole have gotten into a feature creep loop or are selling non-existant features. ie. have no idea of how to manage software development, yet they are trying to manage it ..either directly or via unofficial channels.

Phil Goodwin (Software author, engineer, and mentor)
“My main issue with Agile is that the projects tend to isolate themselves from business, in mistaken belief that development “knows best”.”

This is a misconception that I hadn’t encountered before. One of the common threads that run through Agile methodologies is the clear definition and separation of business decisions and engineering decisions. Agile projects are supposed to be run as a cooperative effort with business people providing information about the content and priority of requirements and developers providing information about the available strategies and costs of implementing the requirements.

“I would be interested to hear experiences from QA/Test people – positive and negative. I regularly find myself defining QA processes in these environments, as there are few standards from QA perspective.”

Agile methodologies, and Extreme Programming in particular, define a testing strategy that does not specifically include a traditional QA role, but which specify activities that seem to me to be ideally suited to being carried out by individuals with a traditional QA skill set. Automated unit testing and Test Driven Design are core Agile practices that seem to obviate the need for external testing. In some situations there is a close enough relationship between developers and the project’s ultimate consumers that the code quality assurances provided by a unit test suite are adequate support for a successful launch. However, on larger projects, particularly when there is a strong need for a positive initial customer reaction to new releases there is a substantial benefit to having dedicated QA personnel on the project.

You hinted strongly at their role in your initial comments. There is a need for the business side of the house to be able to confirm that the software produced meets the actual requirements of the system in which it is to fit. A QA organization brings the expertise that allows the software’s conformance to specification to be measured and quantified. It also continues to play the traditional role of causing the articulated requirements to converge on the actual requirements through a process of consecutive approximation. There’s no replacement for a bloody-minded tester torturing a system with inputs that only a madman, genius or end-user could possibly come up with to flush out hidden requirements (having a good writer document the system will do the same thing along an entirely different axis.) A QA team will also bring the technical skills and discipline to ensure that acceptance testing is automated and synchronized with build process in a way that produces repeatable results and useful diagnostic data. In an Agile environment a QA team can play a very strong supporting role to the business responsibilities of defining and, of course, testing acceptance criteria. In addition they also serve as a proxy for the On-site Customer as defined by Extreme Programming.

I am including a link to the site for the Fitnesse testing tool. The material on this site should give a good gut feel for the way that acceptance testing, which is close to traditional QA testing, fits into some existing Agile projects, and the Fitnesse tool itself has a good reputation in some circles for its ability to provide non-technical business people to express requirements in a testable format.


James Eager (Database Administrator / Data Analyst at Federal Express National LTL)
From a development standpoint, Agile is a rehash of something we used before. We used to call it “prototyping” back in the mid 80s.

From the tech support (DBA) side, Agile is a disaster. We have a couple of projects runnng now. These projects consume 3 times the tech support resources that the regular projects do. They are constantly re-changing the same tables, requiring re-work on our part to fix the things they didn’t start out with.
Matt Davidson (R&D Engineer – Math Geek)
Agreed, from a database perspective it is a nightmare. From previous experience I would say it is better to apply agile or “refactoring” techniques to the software rather than it’s data source.

For applications development with teams of a dozen or less on a single project, I’ve found success with a combination of automated unit tests, and a principle of refactoring using higher level languages like .net to prototype UI’s etc… while the real work is done with C++ core libraries.

Mostly, I’d say it is common sense, don’t over design, don’t under design. Everything is balance.


Nik Clayton (Vice President / Senior Engineer at Citigroup)
You wrote “My main issue with Agile is that the projects tend to isolate themselves from business, in mistaken belief that development “knows best”.”

That seems to me go against one of the core tenents of any methodology that claims to be agile, in that the customer must be kept in the loop, with regular deliveries of functionality to the customer so that they can confirm that what’s being developed meets their needs.

And that’s without going in to the initial “customer story” design phase in which the customer’s initial requirements are supposed to be captured (on 5×3 index cards if you’re a purist…)


Thierry Jeremie (Aegeva – President/Founder)
I have being in the SQA world for quite sometimes now, and found that like everything else there is room for waterfall or iterative methodology. I have worked in a mixed environment, and for a formal implementation (ERP, CRM,…) the waterfall method fits well. However, for fast development and new pre-ipo projects, Agile is a great fit, cutting on the time to market and also having the ability to quickly inject customer enhancements in the product at a low cost enabling the new venture to take off quickly and gain market shares.

For the XP flavor of Agile, the all idea of getting the testing done first calls for Test Engineers that are bread between Developers and Test Engineers. Using as an example the Java framework they should be at ease with Java, Junit, Subversion, Cruisecontrol, ANT,… However they need to also be fluent in formal QA practice once a build(s) reach the integration phase for formal certification, such as: performance, load and stress testing. In some instances, Agile can be so intense, demanding, and fascinating that compliance and governance can be easily forgotten… And if misused Agile could dilute milestones in a large number of iterations with a end result of having an application delivery that is incomplete or late.

Finally, just as any other methodology upper management needs to be educated and understand the benefits as well as cost/time/resource needed for implementing continuous integration processes and tools to support Agile.


Paul Davis (Sr Java Developer at Idea Integration)
“the projects tend to isolate themselves from business, in mistaken belief that development “knows best”.”
– This looks like a mis-understanding.
Most agile process stress iterations where MORE time is spent with the customer. Customers decide priority of features, etc.. So, if you turn that statement around you’ll be closer to the truth.

On to agile in general. Its more about facing reality.
The reality is, nobody really has an effective process for predictably manging software development. How can you when, most development involves a large about of innovation and when the productivity difference your best and 2nd best developer can be 10x.
Agile methodologies basic just accept the reality and cope with it through iterations. Each iteration increases the amount of predictability of the project.
Its just a stepping stone. As time goes by, and the knowledge base increases, we’ll learn more about how to ‘manage’ development and develop better processes.

On the QA side, the problem of so few standards comes from the extreme variation of the quality of QA in organizations. Some have really high quality with people who really understand what quality assurance is while others get starbucks rejects who just click links. Many business owners aren’t willing to commit the amount of resources needed for good QA. The result of this is that QA can be a tough career choice so, few choose it.

No Comments

You can leave the first : )

Leave a Reply