Blog


Agile vs QA

I am interested in testing opinions on Agile Methodology, as I have been working in test management in these “challenging” environments for a few years now. My main issue with Agile is that the projects tend to isolate themselves from business, in mistaken belief that development “knows best”. Though the software development methodologies (based largely on open source movement) are currently exciting and innovative, it’s application is subject to numerous interpretations. The main error is assumption that agile is new, and it’s not, and most/all of the principles are rewriting core programming principles, for modern software development demands. Would be interested to hear experiences from QA/Test people – positive and negative. I regularly find myself defining QA processes in these environments, as there are few standards from QA perspective.

Â
Chris Garrett (Blogger, Online Marketing and web development consultant)
I’m not a QA/test person but I find any methodology will only work with the full buy in and communication of all the people who are (and should be) involved. Agile is fashionable, particularly in the current “everything beta, all the time” environment but it is as good to follow as any other and success will have more to do with personalities and expertise as what approach is used.
Â

Cynthia J Perkins PMP (Program Manager – Information Technology)
I am not a testing expert, but I find your insights on Agile very interesting. I just started working at Klipsch (maker of speakers) and we have been asked to review the tool to perhaps use it in place of MSProject. The engineers currently use it for product roll-out. What version of Agile are you using? I have not started my research yet. It’s on my “to-do” list.

Â

Cathryn Symons (Independent IT Project Manager)

I am a project manager, and we’re using agile in my current project (Scrum). We have business staff embedded in the team, which means that it hasn’t become isolated from the business. However, I do see some concerns, particularly around keeping a longer term view of the entire project (as funded) rather than just the current sprint, and making sure that dependencies are clear. I’m not convinced that Agile alone is enough to manage an entire project, though can see definite advantages for the software development side. Not having to have the requirements defined in detail up front is good, and it seems to lead to far better team working between the developers and business in our case.
Olivier Azeau (Engineering Project Manager at Varian Medical Systems)
In his “Collaborative Software Testing” blog (link 1). Jonathan Kohl regularly writes about QA in agile environments. The “Who do we serve” couple of articles (link 2) is a must read to understand the not-so-easy position of a tester in an environment which claims to focus on testing at developer’s level.

As far as I am concerned, I am not a QA/Test person but I think agile methodologies are good for product quality. The tricky part for the QA/Test people is to be able to become, in some way, ubiquitous : standing beside the developer to avoid the tunneling effect of waterfall while staying in touch with the real users world. I wrote about this issue in “Every process is an iterative waterfall” (link 3).

Links:
• http://www.kohl.ca/blog/
• http://www.kohl.ca/blog/archives/000174.html
• http://agilitateur.azeau.com/index.php?itemid=33

Â

Justin Brister (Director of Technology, Virtusa)
Interesting question; In general, Agile does not call for a separate QA function. Continuous integration, test driven development and prototyping all form an integrated part of the process. In highly complex deployment environments, you may still require a more formal integration testing. Projects which isolate themselves form the end users are not by definition Agile, as Customer involvement is a key aspect of the methodology. It sounds like the issues that you are encountering are down to the fact that traditional waterfall practices are still being used. It doesn’t matter how Agile the development teams try to be, unless you have strong project management and total commitment from the business, you will fail.

Things get even more interesting when you throw offshoring into the mix.
Links:
• http://jbrister.blogspot.com/

Â

Ben Levin (Manager, User Experience Architecture – FSG)
Having been through a couple of Agile projects ($1MM+ development budgets), and coming from a User-Centered Design background (which shares similar goals of continuous testing, continuous customer involvement), I can say that the methodology is quite exciting, and also about as prone to failure as any other. You certainly can’t do it half-assed, but to get “total committment” you need everyone (from the people doing the development to the team that supports them to the business stakeholders who are paying for it) to buy in to several assumptions:

1. That it’s better to launch something that’s useful than to spend time figuring out exactly what you should build and why.
2. That your development team cost is more or less “fixed”. By this I mean Agile tries to optimize the work of the development team towards maximizing the amount of working code they can produce for the amount that they’re paid. This is more or less a given when you’re in a corporation with programmers who are on staff, who might not be doing anything productive for the organization if they’re not focused on developing the next itteration of your product. It isn’t quite as much of a given when you’re outsourcing your development, and even less so when you are the outsourced developer.
3. The business stakeholders have a good reason for wanting to develop something, and have thought through the cost/benefit analysis.

There are a lot of other very valid opinions about whether or not Agile is “good” or “bad”. My overall judgement is that it’s as good as the planning and forethought that goes into using it. In the end, it’s just a methodology, and no methodology can guarantee success (no matter how strictly you follow it.)
Â
Kerry Buckley (Software developer at BT)
Given that one of the principles of agile development is that “Business people and developers must work together daily throughout the project.”, I’d suggest that the projects you’ve experienced weren’t really agile at all.
Links:
• http://agilemanifesto.org/principles.html

Â

Heath Newburn (Product Manager, Motivational Speaker, Change Agent, QA Geek)
There is a Yahoo Group on agile methodologies, it’s pretty much hit or miss. The real issue is treating QA as a separte part of Agile methodologies. It has to go hand-in-hand with each cycle. Unformtunately, I haven’t seen it work in any sort of scale.

Â

Elisabeth Hendrickson (Owner, Quality Tree Software, Inc. and Computer Software Consultant)
I’m a testing consultant who works with Agile teams. I’ve spoken and written a fair bit about my experiences and insights. The links I’ve included below go to my Google Tech Talk, my PNSQC conference paper “Agility for Testers”, and a collection of writings on my blog. From my blog, you’ll find links to other experts in Agile QA/Testing including Lisa Crispin, Brian Marick, and (as mentioned by Olivier earlier) Jonathan Kohl.
Links:
• http://testobsessed.com/wordpress/wp-content/uploads/2007/01/aftpnsqc2004.p
• http://www.testobsessed.com/index.php?s=agile
• http://video.google.com/videoplay?docid=-3054974855576235846

Â

Markus Mikkolainen (CEO at kabuto,Software professional, a jack of all trades and master of some)
The main idea of most agile practises is constant user feedback enabled with constant workable versions which are the result of constant integration.

So if your developers arent communicating with the customer, then you are losing the whole point and the most important gain from using agile methodologies. The customer must be in the loop and giving you feedback after every iteration/sprint , this way the customer will be able to refine his view on where the software should go and he will get a warm and fuzzy feeling that software is getting completed. This will also give the client and immediate control on the timetable, do you want this now or that later? do we do the feature or an immediate release?

Testingwise the challenge in agile methods is knowing what to test since usually there isnt an up-to-date req.spec/test plan. You will have to work from a list of cards (XP) written by the customer and you have to update the test plan on the fly. On the other hand you will be able to do testing most of the time and you can (and should) be in the loop with the developers so they can fix small bugs asap and big ones can be listed on the backlog of things to do.

Agile processes are the best way to combat an environment where the requirements keep coming in and the environment changes very fast.. which is unfortunately all too common nowadays. Also in a case where the customer doesnt really know what they want and what are the possibilities of the solution, it is very useful to be able to get the customer in-loop with the developers and keep getting input on up-to-date versions of the software.

btw. usually if development isolates itself that is a sign of a broken process in the people who are trying to feed the development with too much stuff. Usually this is sales or product development whole have gotten into a feature creep loop or are selling non-existant features. ie. have no idea of how to manage software development, yet they are trying to manage it ..either directly or via unofficial channels.

Â
Phil Goodwin (Software author, engineer, and mentor)
“My main issue with Agile is that the projects tend to isolate themselves from business, in mistaken belief that development “knows best”.”

This is a misconception that I hadn’t encountered before. One of the common threads that run through Agile methodologies is the clear definition and separation of business decisions and engineering decisions. Agile projects are supposed to be run as a cooperative effort with business people providing information about the content and priority of requirements and developers providing information about the available strategies and costs of implementing the requirements.

“I would be interested to hear experiences from QA/Test people – positive and negative. I regularly find myself defining QA processes in these environments, as there are few standards from QA perspective.”

Agile methodologies, and Extreme Programming in particular, define a testing strategy that does not specifically include a traditional QA role, but which specify activities that seem to me to be ideally suited to being carried out by individuals with a traditional QA skill set. Automated unit testing and Test Driven Design are core Agile practices that seem to obviate the need for external testing. In some situations there is a close enough relationship between developers and the project’s ultimate consumers that the code quality assurances provided by a unit test suite are adequate support for a successful launch. However, on larger projects, particularly when there is a strong need for a positive initial customer reaction to new releases there is a substantial benefit to having dedicated QA personnel on the project.

You hinted strongly at their role in your initial comments. There is a need for the business side of the house to be able to confirm that the software produced meets the actual requirements of the system in which it is to fit. A QA organization brings the expertise that allows the software’s conformance to specification to be measured and quantified. It also continues to play the traditional role of causing the articulated requirements to converge on the actual requirements through a process of consecutive approximation. There’s no replacement for a bloody-minded tester torturing a system with inputs that only a madman, genius or end-user could possibly come up with to flush out hidden requirements (having a good writer document the system will do the same thing along an entirely different axis.) A QA team will also bring the technical skills and discipline to ensure that acceptance testing is automated and synchronized with build process in a way that produces repeatable results and useful diagnostic data. In an Agile environment a QA team can play a very strong supporting role to the business responsibilities of defining and, of course, testing acceptance criteria. In addition they also serve as a proxy for the On-site Customer as defined by Extreme Programming.

I am including a link to the site for the Fitnesse testing tool. The material on this site should give a good gut feel for the way that acceptance testing, which is close to traditional QA testing, fits into some existing Agile projects, and the Fitnesse tool itself has a good reputation in some circles for its ability to provide non-technical business people to express requirements in a testable format.
Links:
• http://fitnesse.org/

Â

James Eager (Database Administrator / Data Analyst at Federal Express National LTL)
From a development standpoint, Agile is a rehash of something we used before. We used to call it “prototyping” back in the mid 80s.

From the tech support (DBA) side, Agile is a disaster. We have a couple of projects runnng now. These projects consume 3 times the tech support resources that the regular projects do. They are constantly re-changing the same tables, requiring re-work on our part to fix the things they didn’t start out with.
Matt Davidson (R&D Engineer – Math Geek)
Agreed, from a database perspective it is a nightmare. From previous experience I would say it is better to apply agile or “refactoring” techniques to the software rather than it’s data source.

For applications development with teams of a dozen or less on a single project, I’ve found success with a combination of automated unit tests, and a principle of refactoring using higher level languages like .net to prototype UI’s etc… while the real work is done with C++ core libraries.

Mostly, I’d say it is common sense, don’t over design, don’t under design. Everything is balance.

Â

Nik Clayton (Vice President / Senior Engineer at Citigroup)
You wrote “My main issue with Agile is that the projects tend to isolate themselves from business, in mistaken belief that development “knows best”.”

That seems to me go against one of the core tenents of any methodology that claims to be agile, in that the customer must be kept in the loop, with regular deliveries of functionality to the customer so that they can confirm that what’s being developed meets their needs.

And that’s without going in to the initial “customer story” design phase in which the customer’s initial requirements are supposed to be captured (on 5×3 index cards if you’re a purist…)

Â

Thierry Jeremie (Aegeva – President/Founder)
I have being in the SQA world for quite sometimes now, and found that like everything else there is room for waterfall or iterative methodology. I have worked in a mixed environment, and for a formal implementation (ERP, CRM,…) the waterfall method fits well. However, for fast development and new pre-ipo projects, Agile is a great fit, cutting on the time to market and also having the ability to quickly inject customer enhancements in the product at a low cost enabling the new venture to take off quickly and gain market shares.

For the XP flavor of Agile, the all idea of getting the testing done first calls for Test Engineers that are bread between Developers and Test Engineers. Using as an example the Java framework they should be at ease with Java, Junit, Subversion, Cruisecontrol, ANT,… However they need to also be fluent in formal QA practice once a build(s) reach the integration phase for formal certification, such as: performance, load and stress testing. In some instances, Agile can be so intense, demanding, and fascinating that compliance and governance can be easily forgotten… And if misused Agile could dilute milestones in a large number of iterations with a end result of having an application delivery that is incomplete or late.

Finally, just as any other methodology upper management needs to be educated and understand the benefits as well as cost/time/resource needed for implementing continuous integration processes and tools to support Agile.

Â

Paul Davis (Sr Java Developer at Idea Integration)
“the projects tend to isolate themselves from business, in mistaken belief that development “knows best”.”
– This looks like a mis-understanding.
Most agile process stress iterations where MORE time is spent with the customer. Customers decide priority of features, etc.. So, if you turn that statement around you’ll be closer to the truth.

On to agile in general. Its more about facing reality.
The reality is, nobody really has an effective process for predictably manging software development. How can you when, most development involves a large about of innovation and when the productivity difference your best and 2nd best developer can be 10x.
Agile methodologies basic just accept the reality and cope with it through iterations. Each iteration increases the amount of predictability of the project.
Its just a stepping stone. As time goes by, and the knowledge base increases, we’ll learn more about how to ‘manage’ development and develop better processes.

On the QA side, the problem of so few standards comes from the extreme variation of the quality of QA in organizations. Some have really high quality with people who really understand what quality assurance is while others get starbucks rejects who just click links. Many business owners aren’t willing to commit the amount of resources needed for good QA. The result of this is that QA can be a tough career choice so, few choose it.




RUP

Another well-known process to have come out of the object-oriented community is the Rational Unified Process (sometimes just referred to as the Unified Process). The original idea was that like the UML unified modeling languages the UP could unify software processes. Since RUP appeared about the same time as the agile methods, there’s a lot of discussion about whether the two are compatible.

RUP is a very large collection of practices and is really a process framework rather than a process. Rather than give a single process for software development it seeks to provide a common set of practices for teams to choose from for an individual project. As a result a team’s first step using RUP should be to define their individual process, or as RUP calls it, a development case.

The key common aspects of RUP is that it is Use Case Driven (development is driven through user-visible features), iterative, and architecture centric (there’s a priority to building a architecture early on that will last the project through).

My experience with RUP is that it’s problem is its infinite variability. I’ve seen descriptions of RUP usage that range from rigid waterfall with ‘analysis iterations’ to picture perfect agile. It’s struck me that the desire of people to market the RUP as the single process led to a result where people can do just about anything and call it RUP – resulting in RUP being a meaningless phrase.




SCRUM

Scrum also developed in the 80’s and 90’s primarily with OO development circles as a highly iterative development methodology.

Scrum concentrates on the management aspects of software development, dividing development into thirty day iterations (called ‘sprints’) and applying closer monitoring and control with daily scrum meetings. It places much less emphasis on engineering practices and many people combine its project management approach with extreme programming’s engineering practices. (XP’s management practices aren’t really very different.)




XP (Extreme Programming)

During the early popularity of agile methods in the late 1990’s, Extreme Programming was the one that got the lion’s share of attention. In many ways it still does.

The roots of XP lie in the Smalltalk community, and in particular the close collaboration of Kent Beck and Ward Cunningham in the late 1980’s. Both of them refined their practices on numerous projects during the early 90’s, extending their ideas of a software development approach that was both adaptive and people-oriented.

Kent continued to develop his ideas during consulting engagements, in particular the Chrysler C3 project, which has since become known as the creation project of extreme programming. He started using the term ‘extreme programming’ around 1997. (C3 also marked my initial contact with Extreme Programming and the beginning of my friendship with Kent.)

During the late 1990’s word of Extreme Programming spread, initially through descriptions on newsgroups and Ward Cunningham’s wiki, where Kent and Ron Jeffries (a colleague at C3) spent a lot of time explaining and debating the various ideas. Finally a number of books were published towards the end of the 90’s and start of 00’s that went into some detail explaining the various aspects of the approach. Most of these books took Kent Beck’s white book as their foundation. Kent produced a second edition of the white book in 2004 which was a significant re-articulation of the approach.

XP begins with five values (Communication, Feedback, Simplicity, Courage, and Respect). It then elaborates these into fourteen principles and again into twenty-four practices. The idea is that practices are concrete things that a team can do day-to-day, while values are the fundamental knowledge and understanding that underpins the approach. Values without practices are hard to apply and can by applied in so many ways that it’s hard to know where to start. Practices without values are rote activities without a purpose. Both values and practices are needed, but there’s a big gap between them – the principles help bridge that gap. Many of XP’s practices are old, tried and tested techniques, yet often forgotten by many, including most planned processes. As well as resurrecting these techniques, XP weaves them into a synergistic whole where each one is reinforced by the others and given purpose by the values.

One of the most striking, as well as initially appealing to me, is its strong emphasis on testing. While all processes mention testing, most do so with a pretty low emphasis. However XP puts testing at the foundation of development, with every programmer writing tests as they write their production code. The tests are integrated into a continuous integration and build process which yields a highly stable platform for future development. XP’s approach here, often described under the heading of Test Driven Development (TDD) has been influential even in places that haven’t adopted much else of XP.




Website Usability Example: a Screencast

My first experiment with screencasting – using a website usability problem as an example.