Posted by: jsonmez | November 25, 2009

Simplified Testing


I have been recently struggling with the collision of worlds from the waterfall approach to testing and the agile approach to testing.  There really isn’t a good solid statement being made about what does “end to end” and “system testing” look like in agile.

In concordance with keeping things simple.  Lets break it down starting with the end goal.

To produce shippable code at the end of an iteration.

This comes from the agile principle of

Deliver working software frequently, from a

couple of weeks to a couple of months, with a

preference to the shorter timescale.

In order to deliver working software, that software must be usable or done at that point.  Almost every agile process involves completing a user story, before moving on to the next.  (At least all the user stories in an iteration.)

Shippable software, is software that has finished being tested.   Getting to that point in one iteration is not something that is attainable if you hold on to the fallacy that is “system testing.”  What is “system” or “end to end testing” anyway?  I have never seen a true system level defect that did not exist also at a functional level.  What value are you getting by testing the whole system when you only changed a small part of the functionality?  And when you find defects at this high level of testing you end up having to triage them down to a root cause.

So what if we cut that out.  What if instead we said all functionality in the system that is desired is represented by automated tests which are produced in conjunction with the code.

What if we did 2 very simple things:

  1. For each user story we write automated tests which define what we expect from the system, when those pass the story is done.
  2. Each time we finish a story we run all the previous automated tests to insure nothing is broken.

If we do those 2 things, and we make sure the user of the system (or person representing the user) has agreed that the automated tests adequately cover what they want the story to accomplish is there a need to do anything else?  Or is it perhaps that this very complex methodology of testing really is just that simple?



  1. Very nice blog. Just a question for you. You’ve never checked the money issued the ATM? Do you really trust in the whole process that it’s used to give your money just because it’s automated? I’m thinking in the simplest thing and considering that you’re right just in parts. Automation it’s needed but you need some end-to-end testing, even manually, except you also use a little machine to check the check the money the ATM gave you.

    • You are right. It depends on the kind of thing you are building. If you are purely building software, automation is in software.

      If you are building software and hardware, automated tests are a combination of software and physical automation.

      The physical automation might not be possible, or cost effective, so it would have to be manual for those parts of the system.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: