New Methodology

After seeing post I raised related to Agile and other related methodologies, and how QA can adapt and improve (see post), a friend of mine raised a very valid suggestion – write a new methodology. In the course of consulting QA management work, adaptation of traditional QA processes has been essential, as well as being the unpopular guy who points out faults in a company’s development methodology. Although I have the experience, creating a methodology is a challenge I am happy to take on.Â

I am not looking to reinvent the wheel, just for the sake of it.  But my approach will attempt to include positive elements from existing methodologies combined with with my experience in the field. Media/Gaming/Mobile projects are ideal to base a new methodology around.

Risk Based Test Strategy

Risk based testing is a common approach in the fast demanding modern methodologies in development. In this strategy we assume that it’s undoable to test everything. From a economic point of view it doesn’t even make sense; spending lots of time to parts of a system where the changes of having a bug are low, or even if a bug has been found, where the impact of it will be low. Risk and requirements based testing helps you to determine what to test first, in which sequence, so you spend the time you have to the parts that really matter.

The strategy starts with a risk analysis to determine the functions (requirements) with the highest risk, and plan your test activities guided by this analysis.

To help you identify the risks involved in all your requirements, consider the following aspects:

  • Functions often used by the users
  • Complex functions
  • Functions that have a lot of updates or bugfixes
  • Functions that require high availibility
  • Functions that require a consistent level of performance
  • Functions that are developed with new tools
  • Functions that require interfacing with external systems
  • Functions with requirements with a low level of quality
  • Functions developed by more programmers at the same time
  • New functions
  • Functions developed under extreme time pressure
  • Functions that are most important to the stakeholders
  • Functions that reflect a complex business process

Reverse engineering requirements

One of the great challenges of modern development projects is the phenomena that would have been unthinkable 10 years ago, which is very real possibility of little/no business requirements, or very out-of-date requirements.  Generally, once a project reaches development phase, most methodologies will assume requirements are in place. If they are not in place, then there will be some kind of development requirements. While these are sufficient to develop software, they are not sufficient to deliver software to a customer. In these circumstances, reverse enginnering requirements can help solve the issue.

No-one wants to go backwards, but sometimes it is a necessary evil. In my QA manager capacity, I have seen this situation many times. The way to reverse engineer requirements, by stealth, is a combination of:-

1. Get the customer more involved in the test cycle (so they can realise what their requirements are)
2. Generate some standard business requirements based on generic software standards.
3. Generate some standard business requirements based on conversations.
4. Use higher level development requirements.

Once you have a baseline set of business requirements, you can then put them forward for review.

Localization on a zero budget

This was an interesting exercise is battling the odds. I was providing QA management on a mature successful MMORPG that was approaching localization phase. The game was based on open source technology, and had been developed in Agile/SCRUM environment in the US office. The localization was managed from the UK. The first big hurdle was that the game had been totally hardcoded, so initial work was done to externalise all game data. Translations were done by external translations company, but up to the point when I arrived, the translation had been very ad-hoc, leading to basic errors. The localized versions were hosted by various ISP’s in different countries, and their involvement was limited to verifying the translations.

There was no real budget allowance for this phase of project, so the first move I made was to streamline the translation process, from the translation company, through development and through to the testing at the end of the cycle. As the data had been externalised into one comprehensive localizer file, I took this as the first point in the cycle. This file was provided (intact) to the translations company. To speed up process, and to ensure file integrity, I specified acceptable methods of viewing and editing this file (I chose Vi, as this had support for multiple language character sets, and there was free version available). In order to keep this file integrity, I used QA as the conduit between various people.

Once the translations company had translated the complete localizer file, it was uploaded to shared ftp drive, where QA validated the file integrity, before sending to US dev team for implementation on QA server. The ISP was then asked to validate the localizer file, and then perform some in-game testing to verify correct use of grammar, and identify issues, such as graphic overspill, caused by elongated text. I also used this process as a formal signoff stage, as the informality of the environment meant there was little or no formality. For the business this could prove disastrous as in the event of issues, it was difficult to trace where problem was first initiated.

QA & Modern Development

First of all what is the difference between QA and Testing? Is there one? Standard IEEE definitions are:-
TESTING means “quality control”
QUALITY ASSURANCE measures the quality of processes used to create a quality product.

This is fair assessment, and it is Quality Assurance that this book is focussed on, as testing is simply a derivative a QA process. What has inspired/driven me to write this book, is to sum up my experiences into a guide for other QA Managers to deal with the unusual and bizarre – the world of RAD, of which Agile is the most popularist exponent. Before moving onto this topic, lets see if we can define Agile. This is a challenge, as the perception of what agile is differs from company to company, and individual to individual.

This is a good overview of the aims of agile, but the biggest error is to take this methodology and attempt to build a complete project around it. The main problem with methodologies that claim to encompass a complete project lifecycle, is the dangerous assumption everyone is working off the same page; that everyone understands the implications and possible caveats. Agile has been dogged by theory and tools, rather than focussing on real issue with modern software development – communication. I have been in several bizarre meeting where the topic has been communication issues. Unfortunately the upshot of these meetings is usually “lets email even more” or worse “we are right, they are wrong”. The customer is the driving force of a software project, be it an internal or external one.

The aim of agile is to develop fast, with minimal interference from traditional business processes. The focus is on the customer, but the customer is rarely involved in the day-to-day Agile process. We can see the assumptions mounting up, so rather than following a more critical view of Agile, I am suggesting that development are doing what is required, it is other project members that are not adapting their important roles.Â

Firstly to challenge the traditional role of QA management, and discusses what role attributes should remain, what should be adapted, and additional responsibilities the QA Manager has – whether or not this is defined by the company. Should I be following company policy, rather than defining my own own? They hired a manager, and the assumption is that you can think for yourself, and go against the grain if required. QA requires a thick skin, social aptitude, and complete faith in your beliefs. That is why so many testers did not make it from 20th to the 21st century.