Posted by: jsonmez | June 16, 2010

Agile Testing is Different

I come from a QA background.

I started off my career doing testing.  I learned about testing the same way I learned about development… reading lots of books and applying what I learned.

This is how you are supposed to do QA:

  1. Write a test plan for the project.
  2. Gather requirements.  Put them into a repository and give them cold hard labels like REQ1.2.5.
  3. Create test cases from the requirements.  Store those test cases into a tracking system, which can map them back to individual requirements.
  4. Execute test cases, each test step should either pass or fail.  Record the results.  File any bugs found into the bug tracking system, which maps the bug to the test that failed and the associated requirement.

There is more to it, but that is the basic outline.

But it doesn’t work in Agile!

Heck, it doesn’t work anywhere.  At least not the requirements part.  That is the part that is always missing, or woefully inadequate.

Don’t panic though.  It makes sense.  Let’s think this through.  The reason we all started jumping off the waterfall ship and into Agile is because we realized that it is almost impossible and certainly worthless to try and define requirements up front.

With that idea in mind, how do we test?  How can we test in an Agile environment?

First goal is automation

Let me explain to you why this is the most important part.  It may seem a little out of order, but bear with me.

When you are doing iterative development, you are building things little by little.  You’re not waiting until the end to test things, so there is a really good chance something you do in any iteration will break something you did earlier.

How do you deal with this?  It’s called “regression.”  Yes, you probably need it in waterfall, but in Agile it is an absolute must, otherwise you’ll find that for every step forward you take, you take a step backward as you fix something that you broke.


Running an Agile project without automation is like walking the tight-rope without a safety net, except the tight-rope is on fire, someone is strumming it like a banjo, and you’re wearing two left shoes.

People groan when I say this, but I firmly believe it and I have done it several times.

Part of every backlog’s done criteria should be that it has automated tests!

Unless you have automated tests for each piece of functionality you build (or at least the ones you care if they stay working), or an army of manual testers running your regression suite every iteration, your software will break in your customer’s hands.

How to build an Internal DSL for Automated Functional Testing

Automated UI Testing Framework… A Real Example

Next goal is change the focus

Testing is traditionally an activity that happens at the end of development and is the gate-keeping for software.  It has to go through QA before it gets released.

That is not a good model for Agile, because what happens when there is a bug?  If the testing happens at the end of the iteration, and there is a bug found, there is no time to fix it.

What if we change the order of things, change the focus?

When you write your code, you write unit tests first, right?  (Or if you like BDD, you write the specifications, or behaviors.)  And you run them continuously as you’re working on your code to make sure you don’t break them, right?

We can apply the same thing at the higher level.

  • We can focus on creating a failing automated test before implementing the software.
  • We can iteratively build more failing automated tests and make each one pass by building the software.
  • We can run those tests as we build the software.

How many times have you built a feature only to have it tested and find out the test cases were not at all testing what you built? (I’ll raise my hand.)

That happens when we try to independently create the test cases from the development of the software.

What we want to do is blur the lines between testing and development.  We want to get close to continuous testing.  Just like we do when we are running unit tests as we are writing code.  Short feedback cycles are the key.

Sounds good, but how do we do it?

Here is a walk through of what your workflow might be when doing Agile testing.

  1. Backlog item is pulled into iteration.
  2. Team members talk to backlog owner about the backlog.  The goal of this conversation is not to get all the details of the backlog, but to get enough information to understand the bigger picture of the backlog and get enough information to get started.
  3. Team members talk about what is the basic, first, automated test to write for this backlog.
  4. Team starts writing the automated test.
  5. Team starts writing the code to make the automated test pass.
  6. Team goes back to backlog owner to demonstrate completed parts, get feedback, and ask for more clarification.
  7. Team continues to write automated tests, make them pass, and have conversations and demonstrations with backlog owner.
  8. When backlog owner is happy and backlog is complete, and everyone considers it well-tested, the backlog is done.

Observations about that process:

  • Steps can happen parallel or out of order, steps can overlap.
  • Steps can be split among team members.
  • I erased the words “QA person” and “Developer” and replaced them with “Team,” we don’t care so much about roles, we want to blur those lines as much as possible.
  • There aren’t really phases, for development, QA, etc.  Development involves testing as you go, when the backlog is done being “developed,” it is done.
  • Focus is on making customer happy, not passing an arbitrary set of requirements the customer gave you upfront, or you tried to interpret.  Take a look at my post on comparing user stories requirements gathering to hanging a picture for more on this.
  • There is no useless documentation.  All documentation is executable.
  • There is no “gate-keeper,” quality is being built as we go, we are not trying to “test it in” at the end of the iteration.

What is the role of the QA person?

It might seem that we have eliminated that role by this process, or diminished the value, but nothing could be further from the truth.

What we have actually done here is elevated the role from writing documentation and manually executing tests to being the expert on quality and the representative of the customer on the team.

You can think of the QA person as the “quality coach” for the team.  They might not always be the one creating the automated tests or running them, but they are one of the main forces guiding the direction of the creation of those tests, and making sure the customer’s interests are represented.

The QA team members will also occasionally manually test things that cannot be automated, and exploratory test as needed, but the focus should shift towards being the “quality coach” for the team instead of doing all of the quality work.

I found this excellent presentation on Agile Testing by Elisabeth Hendrickson from Quality Tree Software if you are interested in further reading on the topic.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.



  1. […] Agile Testing is Different (John Sonmez) […]

  2. QA is ‘Quality Assurance’. The classical software test approach is actually Quality Control:

    Testing after the fact is QC. Making sure you don’t spend time doing the wrong thing in the first place is QA. Development processes are finally heading towards QA long after the term ‘QA’ was bent around to be applied instead to QC!

    QC is testing. QA never was (although that tool might be used for some purposes). QA is about process.

    It is fun to watch. The purpose of actual QA is to change the workflow to avoid wasting time and effort. And now, you mention ‘You can think of the QA person as the “quality coach” for the team’.

    That was actually always the idea.

    Please don’t read this in any way as a criticism of you or what you are saying – I agree with you, and I’m pleased to read the perspective is changing “in the wild”. I would say from my own experience that poor and ignorant management has always been what has held software development back, and continues to now. But it is heartening to see a widening recognition of this point.


    • Thanks Bret,

      I agree with you completely. Appreciate the comments and your comparison of QC to QA.

  3. […] fully subscribe to the concept of an agile testing i.e. no clear distinction between a developer and tester, just a team who are continually writing […]

  4. Nice writing !

  5. It is a nice article but things mentioned are not just relavant to Agile but also a better approach of doing in waterfall model too.

    Agile is nothing more than an excuse for lazy spec writers. The things that could have been identified and decided earlier in waterfall model now takes almost project end to get decided.

    Agile is becoming fast industry standard because it is lead by lazy developers who don’t want to take blame their in-efficiencies to forsee the things in advance and lazy spec writers who gets an excuse in the form of development methodology.

    People tend to think Agile is faster development methodology but I counter the argument. The thing that could have been finalized by specs writer in cordination with requirement initiator or product manager now involves the whole team. If we do a cost analysis between Agile and Waterfall my firm believe is that we can be much more efficient using waterfall model.

    Consider an example of building a Building. Initially you don’t know what color tiles you want in your bathroom, you paste one tile, but since this is not suiting the floor you scrapped it and start tiling your bathroom again with different set of tiles.

    Software products are no different. But I doubt this industry will ever align with my thoughts because at last the industry itself is driven by developers and analysts who want to play it safe.

    But, not everything about Agile is wrong. Definite and well defined work and amount of work is one of the best aspect of Agile technology.

    Sorry, if the forum was not right for this message.

  6. Hi

    I’m late to this post but let me completely agree with you that automated tests are utterly vital in a fully agile / scrum environment. Having been a Software Development Manager for many years, I was convinced that automated tests were the way to go. Now, as a Scrum evangelist, I consider automated tests as vital.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: