Manual Testing Does Exist and It Is Bad

This content is syndicated from by Ron Jeffries. To view the original post in full, click here.

My esteemed colleague and sometime nemesis Michael Bolton has written a screed against the terms “manual testing” and “automated testing”. Check it out, sapiently.

In his article, Michael, for reasons which have not become clear to me, attempts to show that these terms are not useful, by arguing that there are other human activities which do not parse nicely into the notions of manual and automated. Unfortunately, there is a useful distinction in software testing that is made by these terms. There might be better terms. Michael himself has offered others from time to time. He has offered the notion of “test” vs “check”. This is a useful notion but it turns out that both manual tests and automated tests in our parlance are checks in his. We would have to say “manual checks” and “automated checks” — in fact sometimes we do — but then we’d have to have this discussion all over again.

Michael has also offered or borrowed — I hope the latter — the term “sapient”. I don’t claim to grok this in fullness but I take it to mean, roughly, “thoughtful”. We humans can do things more or less thoughtfully, and to the extent that sapient means “wise”, not just “thoughtful”, we are certainly capable of doing things more or less wisely. I am living proof of that last bit.  Sapience is a useful notion as well, but it turns out that while both manual tests and automated tests can be prepared “sapiently”, and manual tests can even be executed sapiently — but often are not — automated tests of today cannot be executed in a sapient fashion.

So neither the notion of check versus test, nor the notion of sapient versus … what … non-sapient? quite captures the distinction intended by manual versus automated tests. What do we mean by manual versus automated? I’m glad you asked.

There is a kind of “testing” where there is a book that tells the “tester” what to enter into the computer, and what the output should be. They enter the input the book calls for, and check to see if the computer says what it was supposed to. If it does not, they — I don’t know — blow a whistle or write it down or something. This alerts someone to look at the situation, think about it, determine if there is a problem, and fix it. The person who does all this is often not the person doing the typing and checking. Insultingly, they are often thought of as “smarter” if not more thoughtful or sapient.

That kind of “testing” is what we call “manual testing”. We do not like it. It is inhumane. It is intentionally designed to require little thought. It could probably be taught to chickens were they not so tasty.

Yes, we know there is a computer involved. Not everything with a computer involved is “automated”. For example, I am thinking about this reply, composing it, trying to be just edgy enough but not too edgy. I am typing this reply. Through some means I do not fully understand, it is going into my computer and ultimately into this little rectangle I’m typing into, and, I hope, onto my web site. It’s not as manual as when we used to cut little wedge marks into clay tablets and then bake them, and it’s not as automated as it would be if I could just think “Sort Michael out” and carefully crafted ideas would appear on my site — or directly in your head. If you feel yourself getting smarter, don’t worry, it’s not me putting ideas directly into your head. I can’t do that. As far as you know.

I would say that I wrote this reply thoughtfully. As for sapiently, that remains to be seen.

I write code thoughtfully as well. I don’t know any way to write code, or blog entries, other than thoughtfully. I cannot write them manually, and in both situations, my writing cannot be automated, as far as I know. The whole notion of concluding that there is no manual testing because we don’t call compiling automated programming is empty. (By the way, in the olden days, after the clay tablets but before today, compiling and linking were in fact called automated, or automatic, programming. The term did not gain currency because it was so obviously just another kind of programming, of creating a work of the mind and getting it into a computer.)

Testing, however, can be set up to be done thoughtlessly, and in my experience it often is. Non-sapiently, if you must. That process is what we call manual testing. The creation of the book full of things to check was perhaps thoughtful. The execution of what’s in the book is not very thoughtful. We think that the initial thinking should be put into a program, not into a book, and that the execution should be done by a computer, leaving people to do work that is more suited to humans. “Exploratory” testing, for example. We’ll leave it to another day to talk about that term, which is not automated, presumably sapient, and — unfortunately — sometimes looks a lot like manual testing. It is distinguished from manual testing in that even if it uses a book, it is not rote in nature. We reserve the term “manual testing” for the kind of testing that could be taught to a chicken — or even to a computer.

Much of the process of building toward certainty that software works, especially when we work in an incremental and iterative fashion, involves checking to be sure that the answers are right — as we understand right — and re-rhecking to be sure that the software is still producing that answer. We do that with tools that encode the ideas of “right”, execute the software, and check to see if the computer says what was expected. If it does not, the tools often blow a whistle or write it down or something. This alerts someone to look at the situation, think about it, determine if there is a problem, and fix it. This is always a person today. Perhaps one day the computer will be able to do more to help. Today, all the computer can really do is alert us when something changes.

Because we need to know quite often — weekly, daily, even moment-to-moment — whether something has changed, we prefer to have “automated tests”, which we can execute at the press of a button, rather than “manual tests”, where we would have to call over a person — or a chicken — to perform the tests and get back to us. Should those tests fail, we then look into the situation further, or, if we are fortunate to have one on hand, ask a sapient tester to look into it with us or on our behalf.

Now these manual tests which we abhor might better be named manual checks, and in fact I try to call them that in day-to-day use. I call them tests here to maintain continuity with Michael’s article. In execution, they are not thoughtful, and not “sapient”, though their original definition was both.

There are many words that describe ways of improving our certainty that our software works. There is a particular value to defining checks that can be done to help us gain confidence. Those checks can be made by the computer running a checking program, or by a person running through a list of things to type. We call the former “automated”, and the latter “manual”. We don’t want humans doing manual testing. We don’t even want chickens doing it. We want computers doing it. The terms manual and automated are useful, and that’s why we use them.

Leave a Reply

What is 9 + 5 ?
Please leave these two fields as-is:
Please do this simple sum so I know you are human:)

There are 101 ways to approach anything.
To find the best way, sometimes you need expert help

What People Say

“Kelly revolutionised the way our digital department operated. A true advocate of agile principles, he quickly improved internal communication within our teams and our internal clients by aligning our business and creating a much enhanced sense of transparency in the decisions the business was making. Kelly also introduced a higher sense of empowerment to the development teams...”


“Kelly’s a leading program director with the ability to take charge from day one and keep strong momentum at both a program and project level driving prioritisation, resourcing and budgeting agendas. Kelly operates with an easy-going style and possesses a strong facilitation skill set. From my 5 months experience working with Kelly, I would recommend Kelly to program manage large scale, complex, cross company change programs both from a business and IT perspective.”


“Kelly is an extremely talented and visionary leader. As such he manages to inspire all around him to achieve their best. He is passionate about agile and has a wealth of experience to bring to bear in this area. If you're 'lucky' he might even tell you all about his agile blog. Above all this, Kelly is great fun to work with. He is always relaxed and never gets stressed - and trust me, he had plenty of opportunity here! If you get the chance to work with Kelly, don't pass it up.”


“Kelly is an Agile heavy-weight. He came in to assess my multi-million $ Agile development program which wasn’t delivering the right throughput. He interviewed most of the team and made some key recommendations that, when implemented, showed immediate results. I couldn’t ask for more than that except he’s a really nice guy as well.”


“Kelly and I worked together on a very large project trying to secure a new Insurer client. Kelly had fantastic commercial awareness as well as his technical expertise. Without him I would never had secured this client so I owe a lot to him. He is also a really great guy!”


“Kelly came to the department and has really made a huge impact on how the department communicates, collaborates and generally gets things done. We were already developing in an agile way, but Kelly has brought us even more into alignment with agile and scrum best practices, being eager to share information and willing to work with us to change our processes rather than dictate how things must be done. He is highly knowledgable about agile development (as his active blog proves) but his blog won't show what a friendly and knowledgeable guy he is. I highly recommend Kelly to anyone looking for a CTO or a seminar on agile/scrum practices - you won't be disappointed!”


“Kelly was a great colleague to work with - highly competent, trustworthy and generally a nice bloke.”


“Kelly was engaged as a Program Director on a complex business and technology transformation program for Suncorp Commercial Insurance. Kelly drew on his key capabilities and depth of experience to bring together disparate parties in a harmonised way, ensuring the initiate and concept phases of the program were understood and well formulated. Excellent outcome in a very short time frame. ”


“I worked with Kelly on many projects at IPC and I was always impressed with his approach to all of them, always ensuring the most commercially viable route was taken. He is great at managing relationships and it was always a pleasure working with him.”


“I worked with Kelly whilst at Thoughtworks and found him to be a most inspiring individual, his common-sense approach coupled with a deep understanding of Agile and business makes him an invaluable asset to any organisation. I can't recommend Kelly enough.”


“Kelly was a brilliant CTO and a great support to me in the time we worked together. I owe Kelly a great deal in terms of direction and how to get things done under sometimes difficult circumstances. Thanks Kelly.”