Tales from the trenches: Keyword based testing

Setting the scene

A few weeks back I got presented with a very interesting challenge from a colleague. It differed from the norm in that I was presented with nothing more than a scenario & task, with no functional pieces to grab a hold of and test.  Fun!  A thinking man’s challenge, I like those

What will follow is my approach to that challenge, I hope you’ll find this useful & as a test technique this can prove to be very powerful.

I’ve called it “Keyword based testing” just as a way to describe it, I’ve no doubt many people use similar techniques.  I’m not laying claim to anything you can call it whatever you like.  Just like all things in testing, there are no radical new idea’s, just common sense approaches to solving problems. On a side note the name comes from my colleague Andrew who I’d asked to help me name it, as for me that proved the most difficult part, picking a name to call this.

The challenge

So let’s quickly recap that challenge, Stuart talked about it here previously, if you haven’t read that, have a look, it’s very interesting.

Scenario

Your good friend Dr MacDonald has been developing a time-traveling car which he aims to use to allow a person to travel backwards or forwards in time.  He states that he has tested each of the components separately and has tested the system as a whole by using two synchronized clocks (one of which is sent 1 minute into the future with the car.  When the car reappeared, there was exactly 1 minute of difference between the clocks).

He states that he is now happy to use the system and wishes to be the first person to travel in time.

Task

Dr MacDonald has hired you at this late stage in development to consult about the level of testing he has carried out so far and if he needs to do any more.

So, the task is simply this.  What questions would you ask Dr MacDonald about the system and its testing?

Rules

Please take no longer than 30 minutes to come up with as many questions as you can.

Initial reactions

So from a quick glance you can see that we don’t have a lot of time to complete the challenge & we have to work out what we would test in this system.  We don’t have Dr MacDonald at hand to answer our questions as we ask them so we’ll have to quickly jot them all down and present them to him in one go.  In hindsight though wouldn’t that have been an excellent first question?  “Dear Dr MacDonald, with these short time constraints you’ve placed me under, I’d like to ask if you could be present to answer my questions as I ask them?”

Now I don’t have an application to test, but I do have a description, assumptions, a deadline & most importantly a what & how!

In this situation a mind map would make perfect sense as I could quickly generate questions rapidly and structure this scenario into it.  Having previously talked on my blog about using mind maps to design lean test cases I know how powerful a tool they can be & an essential tool at that for any tester’s toolkit.

So you think you know where this is going right?  Wrong!  Keep reading

Breaking it down

So with my favorite mind map tool loaded (XMind) I began to break down the scenario into sections.  Flipping aspects of this would lead to my initial questions.

So let’s do a demonstration of how I did this:

We have the scenario “Your good friend Dr MacDonald has been developing a time-traveling car which he aims to use to allow a person to travel backwards or forwards in time.  He states that he has tested each of the components separately and has tested the system as a whole by using two synchronized clocks (one of which is sent 1 minute into the future with the car.  When the car reappeared, there was exactly 1 minute of difference between the clocks).” Chop, chop, chop!  Yeah let’s break it up & start flipping words around

“Your good friend Dr MacDonald”  Good?  Hmm I’m not to sure about that one, lets flip that and call him bad!

Now all we’ve done so far is break a small section off the scenario & took a keyword from this and flipped it to be reversed.  Very simple stuff.  This is also a very powerful test technique, read on and you’ll see why.

Mapping it out

So we flipped good for bad and got the following “Your bad friend Dr. MacDonald”, now I’m already starting to fill my head with distrust for this guy.  So what can we question about this now?  For starters how about the legality of the whole experiment?  Security?  Trust?  Intentions?  Those are probably good initial nodes for our mind map, which we can later re-look at and expand into more questions.

Great minds!

Lets break up the scenario more “has been developing a time-traveling car”, another keyword “car” we could turn that into “cars” right?  Hmmm more questions are jumping into my head now!  Why not cars?  Where is the redundancy?  How easy is this car to replicate?  In hindsight I could probably have identified “has” & “developing” as keywords.  Think about those two for a minute.  See you’ll already be coming up with your own questions on these now

The fun thing is, everyone will identify their own keywords & come up with their own questions.  This can be easily be used as a powerful collaborative brainstorming technique, try it out some time with some requirements.

Assumptions can bite you in the …

“he aims to use to allow a person”  That little section is very interesting in itself!  We can take two things from this: “Aims” means we can assume (even though we find out later this is true) that he hasn’t proved yet that a person can travel in this car.  So material springs to mind!  Can it transport living matter?  We later find he’s only testing non-living matter, but we now have a question which might have been lost by an assumption.  There is a very good lesson to be learned here, always question your assumptions!

The other thing we can take from this is the keyword “person”.  Most cars that I’ve seen carry two people, but we don’t know anything about this car.  So we could easily come up with a whole bunch of questions around the design of the car, which in themselves may lead to more sub questions.  I didn’t notice this one first time around, only as I’m talking about this here, but we could ask for wire frames before we meet him “if” we are allowed to meet him and his car, which in itself is another assumption we had that we have now questioned. 

Ok so now you’re starting to see the benefits of this technique.  Your lovely mind map will be quickly filling out, on what at the beginning might have looked like a challenging challenge.  Oh but wait how far along are we now?  “Your good friend Dr MacDonald has been developing a time-traveling car which he aims to use to allow a person”  Ouch!  This may well turn out to be a very long post!  Don’t worry though I’ll speed things up now

Gear change!

“to travel backwards or forwards in time”  So we have our directions, lets map those as initial nodes on our mind map.  We can remove them later if needed.  So if I can go in two directions to (assumption) any place?  Ah ha!  Why can’t I just teleport to another location?  Let’s also take a note of that assumption so we can question it later.

“He states that he has tested each of the components separately”  *cough* assumption!  *ahem* question!  Again do we believe this?  Also let’s flip separately to say together!  Uh oh!  No systems integration testing!  Holy crap!  I should probably stick SIT & CIT & CT on my map just now as initial nodes right?  I can modify it later if I want.

“and has tested the system as a whole by using two synchronized clocks”  Lets change two with three & lets change system with car.  So why two clocks?  Why not more?  What makes the clock valid, is it secure (type of testing, map it!) Ah!  Secure!  Is the system secure?  Where is it located?  Does this machine put that country at risk?  And so on and so on…  System!  What is a system to Dr Macdonald?  Why do we use clocks can we try some other means to verify it?  What about the other end (where it travels to) is that secure?

Old school testing

“one of which is sent 1 minute into the future with the car” Lets flip the keyword one minute with two & future with past.  Oh!  He’s only verified one directional travel?  He’s done only 1 minute?  Why not more?  Does it take negative values?  We could try some really large numbers, really small, the current time?  There are so many questions to be asked here alone!

“When the car reappeared” Lets change when to did, we can also change reappeared to disappeared.  Again lots of redundancy, security, validity questions to be asked here.  We also have an assumption in this line & by switching the keyword when with did we have questioned that assumption.

“there was exactly 1 minute of difference between the clocks”  Another assumption!  Question it!

A powerful technique

You can see we’ve generated a whole pile of questions already & we’ve only got through one paragraph of the challenge; there is lots more to go!  I’ll not bore you with that though I think I’ve demonstrated my approach enough now

So by using this technique of breaking up the scenario, we can easily identify keywords which if manipulated can lead to further questions.  Often you’ll be able to do it all in your head & just map out your initial nodes on you mind map.  Most importantly we need not think of this as a technique to be used only for challenges.  Think about those vague requirements you often get, this can be excellent for them!  Lets break down the vagueness and question it!  Those questions you map out can be further broken down, which will lead to even more questions!  Before you know it you’ve discussed all these questions with your stakeholder (or someone!) & you now have a whole raft of information to play with

Oh!  I nearly forgot one thing!  Did we have any rules?  We had a limit of thirty minutes.  Do you not think we should question that also? 

I never did question that myself, I stuck to my thirty minute limit even though the questions were still flowing! However rules are there to be broken & most importantly since no one made any (minus time) we can do what ever the hell we like

So what did you do?

If you’re interested in what my response to this challenge was, have a look at my mind map below.  A larger picture of it can be viewed here.

Lets learn from each other!

Just like the previous challenge I’d given my team, my main interest in this one was the approach people used for it.  As much as I enjoy looking at what people have found that I hadn’t & believe me, I really do enjoy it!  It’s from peoples approaches that I’ll learn the most valuable lessons.  I’m always sitting on the edge of my seat waiting for one to come along that will make me go wow!  I could use that.  That’s very useful!

I’d love to see more people discussing how they approach testing certain things & challenges are a good way to learn and talk about them.  So I challenge everyone reading this, to go away and take a simple challenge & talk about how you approached it.  I’d love to read it!  Thanks for reading.