Of Planets and Test Automation

This coming July (year 2015), the New Horizons spacecraft will fly by Pluto. Along with that fantastic human achievement, the controversy-that-refuses-to-die will be rejuvenated and kicked around ad nauseam: Is Pluto a planet or not?

The irony here is that since the objective is pure science, as long as communication is served, the labels that people use for the object don’t matter at all. It’s no more than semantics and politics. We can call Pluto a planet, a minor planet, a dwarf planet, a Kuiper Belt object, or a great big sleeping comet, but it makes no difference. Planet or Kuiper Belt object or whatever, Pluto is Pluto.

In the world of software quality, there’s a similar controversy around automated tests: aren’t these just the same thing as manual tests? “Manual test” and “automated test” are the same thing, right, so would it not be more efficient and correct to call them “tests?”

No, automated tests and manual tests are not the same. The objective of a test is pure information, or more precisely, technical value in quality measurement. The difference between “manual” and “automated” is real and valuable, because the value to the software project between manual and automated tests is generally very different.

Generally, manual test have these values:

·         Whether scripted or exploratory, they benefit from human intelligence and powers of observation

·         People can be flexible when running them, to be more efficient

·         The test results benefit from tester smarts

And, these downsides:

·         They can get repetitive or boring

·         Humans make mistakes

·         Humans get tired, have to sleep, must do something else sometimes

·         Test repeatability isn’t always good, due to above factors

Automated tests have these values:

·         Can be fast or extremely fast

·         If well-written, they’re very repeatable

·         If well-designed, they’re very scalable

·         Geography and time-of-day doesn’t matter anymore

·         Automatons don’t tire or get bored

·         Automatons are great for processing large amounts of information efficiently and accurately

And, these downsides:

·         Automatons are idiots, and depending on how the automation is written, they can make huge, immensely stupid mistakes repeatedly

·         An automated test will miss major issues that are both important and obvious to a tester

·         Human tester/programmers have to tell the automaton exactly what to do, but if given a chance, the automaton will mess up anyway

·         Poorly written automation can fail and drop important root cause information bits on the floor, which requires a tester/developer person to follow up and figure out what happened

To create a scripted test means writing out the steps. Almost all of them are run manually at some point, but note that by “manual” I mean that it’s a human who determines the initial result of the test as pass, or blocked or fail with additional information about the blockage or failure. It’s a manual test even if it’s a tester that uses a web browser to call a REST API with XML over HTTP, because the tester completes the test with a result.

At some point, the steps of the script may be automated, so it becomes an automated test; it’s an automated process, not a tester, which determines the initial test result at that point.

Given the above differences between automation and manual tests, is the test the same now that it’s automated? Faster and cheaper, maybe? Faster and cheaper, maybe, but the same value, not at all.

This is why I use the word “check” to describe the automated test: the difference is important, and not understanding the difference could introduce significant business risk. (More on the “risk” below.) Calling it a “check” rather than a “test” is much less of an invitation to the business owners to introduce poorly understood risk. A check is just a kind of test, though, but with well-defined and well-understood verification(s). It’s just like driving around at night when the passenger says to the driver, “Hey, check that the lights are on.”

The risk of mistaking an automated procedure with defined steps for the same steps done by a manual tester is that the tester can see so much more. Testers are smart and observant, automatons not. To automate the test means making the team blind to things that it used to be able to see.

For example, here’s a manual test for a credit union web site

1.       Go to credit union site

2.       Log in

3.       Go to checking page

4.       Check balance

5.       To go “Account Transfer” page

6.       Select $100

7.       Select “From” as “Savings”

8.       Select “To” as “Checking”

9.       Click “Transfer Funds” button

10.   Confirm

11.   Go to checking page

12.   Verify that the transfer happened

13.   Verify correct balances reflected

14.   Logout

 

A tester running this test notices “hey, the nav bar looks out of whack…” takes and annotates a screenshot and enters a bug.

The bug is fixed and regressed by the tester with a note “looks good now.” The above test is also automated, but the navigation bar is not the top priority and one can’t automate a check for “looks good” so that never gets automated.

Assuming that the automation is written and handled well, it regresses the functionality of the steps and makes the verifications often. But, the presentation of the navigation bar isn’t verified any more. The business owners think that the web site and transfer functionality is regressed entirely, but that’s not the case; the automation runs, and the verifications in step 12 and 13 happen, but whether the page is readable to a human or not is simply not tested when the automation is run.

That discrepancy creates the risk that comes with misunderstanding what automation does for you, which in turn is related to the common mistake that automated tests and manual tests are the same thing.

Clarity is important here: automated tests and manual tests are so different, they need to be considered differently to avoid the business risk described above.

For Pluto, whether it’s a planet or not, isn’t important. But, I sure am looking forward to what New Horizons can tell us about it.