Posted by: jsonmez | February 5, 2011

Back to Basics: Becoming BAT Man

In my Back to Basics post on my conclusions about blackbox automated tests (BATs) and unit testing, I said that we should:

Spend a majority of your effort, all the time you would have spent writing unit tests, instead writing what I will call blackbox automated tests or BATs.

In this post, I am going to outline what I think it takes for a team to go from BAT-less to BAT-man (or BAT-woman) without going batty.

I want to take a practical approach to getting from 0 blackbox automated tests to building a sustainable process of developing BATs as a integral component of the acceptance criteria for calling a backlog item “done.”

Let me paint a picture for you


I want to paint the picture for you of the end goal that we are seeking to achieve, from start to finish, of a single backlog.

  1. A backlog is selected for work.
  2. Developers, QA and product owner work together to clearly define the acceptance criteria which will enable that backlog to be signed off or called “done.”
  3. QA and developers work together to transform the acceptance criteria into a high level description of BATs which will be created to verify each aspect of the acceptance criteria.
  4. Developer begins coding with this high level set of BATs in mind (BATs drive the development), QA begins developing BATs.
  5. As developer completes enough functionality to pass a BAT and not break existing BATs that functionality is checked into trunk.
  6. Each check-in causes the entire battery of BATs to be kicked off split across multiple VMs which run 100s of hours worth of automated tests in minutes.
  7. Once all BATs for a backlog are passing, that backlog is done and those BATs become part of the battery of tests which are run with each continuous integration build.

We should keep this simple process in mind and strive to reach it, knowing that when we are able to reach this process we will be able to have a HUGE amount of confidence in the functionality and non-regression of our system.

Put aside for a second your notion that unit testing is the correct thing to do.  Put aside the idea that blackbox automated testing is too hard and too fragile, and imagine the world of software development that flows like the picture I painted above.

Now listen up, because this part is important…

Before you put forth the effort to write one more unit test, before you dip your test double pen in the ink well of “mock,” make sure you are first taking efforts to develop a process to create BATs.

The value proposition

We really need to take the effort to understand the value proposition being presented here.

I’m going to use some real fake numbers to try and really convey this important point that I think is likely to be overlooked.

Consider first the amount of time that you spend doing two things:

  1. Creating infrastructure to allow your code to be unit tested in isolation.  (Dependency injection, interfaces, etc.)
  2. Time spent writing unit tests.

Now, before we go any further, I just want to reiterate.  I am not advocating completely abolishing the writing of unit tests.  See my original conclusions post, or my post on unit testing without mocks for a better understanding of what I am advocating in regards to unit testing.

So back to my point.  Think about all the time it takes to do this.  To properly isolate and unit test code most developers would probably estimate that for every hour worth of work writing the code there is about another hour to two hours worth of unit testing and unit test prep time.

There is a reason why unit tests take longer to write than the code they test; it takes MORE lines of code to test a line of code in isolation.

BATs developed using a good automation framework are just the opposite.  While unit test code might have a ratio of 4 lines of unit test code to 1 line of production code or a 4:1 ratio, BAT code has a completely opposite ratio.  It is very likely that 1 line of BAT code can test 20 lines or more of production code, a 1:20 ratio.  (Where do I get these numbers?  Looking at some of my real production code being tested with unit tests and BATs.)

Even if unit tests and BATs were equally effective in preventing the regression of your software system, and equally effective at providing an absolute assurance of meeting the acceptance criteria (which I would argue that BATs are much more effective at both), you can see easily that you are going to get a much higher return on your investment by investing your time in BATs vs. investing your time in unit tests.

I don’t make these statements lightly or in a theoretical tone.  I have real world experience with successfully implementing automation frameworks for writing BATs that reinforce these conclusions.

How to get there

If I have done my job, you are now at least convinced that getting to the point of having BATs for all backlogs is a good goal, but if you are like most people I talk to, you are skeptical of the costs and feasibility of doing it.

What you need is a good guide.  A step by step guide on how to do it.

I am going to conclude the Back to Basics posts and segue into a new series with the goal of providing a step by step guide to getting to fully automated blackbox testing.

Let’s take a look at the steps that I will cover in the next series of posts.

  1. Hiring an automation lead – it’s important to either find someone who’s done this before, or a developer resource to this role.
  2. Deciding on the underlying browser driver or browser automation tool – assuming web apps here, there are several to choose from, WatiX, Selenium and more.
  3. Designing an automation framework – building a framework tailored specifically to your application under test will create an effective domain specific language for testing your application.
  4. Creating your first smoke tests – building some smoke tests will help you prove and build out your framework and provide you the highest value tests first.
  5. Adding smoke tests to your build – with smoke tests being run as part of the build, you can immediately start seeing value.
  6. Adding BATs to your acceptance criteria – we need to start out slowly here, but eventually make all backlogs require BATs for each acceptance criterion.
  7. Scaling out – as you start to get a large amount of BATs you’ll need to figure out a way to virtualize more servers and parallelize the test runs to be able to run all the BATs in a short amount of time.
  8. Building a true DSL – once you get this far, it may be time to start thinking about creating a domain specific language that even business analysts can write tests with.
As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.


  1. […] all the time you would have spent writing unit tests, instead writing what I will call blackbox… [full post] jsonmez Making the Complex Simple automationbest practicesfunctionalprocess improvement […]

  2. Hey John, would you say that BATs have a diminishing return on investment as the complexity of the software / business domain goes down? In other words, if I have to create a fairly simple site, would I be better off writing a few unit tests, or would full on automated tests still make sense?

    Really looking forward to the new series!

    • Hmm, good question. I am not completely sure one that one.

      I think to some degree that would be correct, but I think it depends on how good you are with automation tools and building a framework.
      What I mean by this is that if your company already has the competency in BATs the cost to get BATs up for a new product might be very low. So even for non-complex software the ROI might be there, but on the other hand if your company is not already invested in BATs then the initial cost might be very heavy in relation to the benefit.

  3. […] my last post in the Back to Basics series, I mentioned that I would begin this series with the intent of showing you step by step how to go […]

  4. […] you have been following my posts so far on Becoming Bat Man, you should already have hired an automation lead, figured out what browser driver you are going to […]

  5. I know this is very very old. Just stumbled upon it.
    I’m a automation-test programmer at a company with a very old legacy codebase, growing since 2005, working since 2010 on wholistic blackbox-integration tests (the ones you call BATs). I have written such BATs for over a year, now switching to SpecFlow-based component tests, which are still in the business domain, but expressed very verbose, are in-process, and such stable and as fast as unit tests. Also, they are direct usecase descriptions and thus living specifiations, used by the business, project leaders, used in communication with customers…
    The BATs from before.. we had to throw them away, they were a maintenance burden.
    It’s simple in testing: if you can’t maintain it, meaning that the cause for an error (much more likely in BATs than in Unit tests) has to be able to be pinpointed in minutes. But this time gorws exponentially with the odeamount covered by a single test. Thus, BATs are.. very very bad.

    I actually can now get much better productivity in the longterm with SpecFlow based component tests as living documentation + very slim integration tests (mostly smoketests via automated LinqPad html output on points like Exchange server invocation, Sql Server invocation, etc.) on the “outside”. Beautyfull concept, will never look back at BATs in my life, I’m sure there..

  6. “It’s simple in testing: if you can’t maintain it, meaning that you can locate the cause for an error (much more likely in BATs than in Unit tests) in few minutes, yu shouldn’t do it. But this time grows exponentially with the amount of code covered by a single test.”
    Sorry, very late here in germany..

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: