The Friday challenge

Ah! A Friday Challenge!

The 19th of November marked the day of the first Friday challenge in our office.  The idea was that we’d have one each week & these would be fun, short, simple & provide people with the chance to learn & discuss what they’d tried to solve.  I must say when I heard about the idea of doing one each week I thought it might be a little bit too much!  None the less one sprung to mind that I thought I’d give a try.

So with fun in mind, this challenge was all about those nasty little things that we call assumptions, which at times can be useful but more often than not dangerous & wasteful.

The challenge ran in parallel with our R&D team & also on twitter with the testing community.  I’ll start of with our team before taking it global.

The challenge

So the challenge was very simple I’ll recap it below:

The world could end in five minutes! You’re currently in the office using a program that a wannabe programmer “Tiffany Swift” has developed for your team. It requires you to be present to press a key every 3 minutes. Your boss is expecting the results from this program when it completes! He’s stressed it’s very important.

What do you do?”

I already knew what I was looking for obviously.  Each of these would be found simply from people questioning any assumption they’d made.  In fact, between doing this with our R&D team & on Twitter each aspect of this challenge was cracked!  I was impressed with that, because there is a lot that we can derive from this in the form of assumptions.

What the locals did!

So I’ll start with a summary of the responses from our R&D team.  Be warned some of them are a bit silly; it appears to be mandatory to crack at least one joke with every email they’d sent :-)

Recruitment

  • Our product manager suggested using “name bias” as a recruitment policy.  Apparently he wouldn’t hire someone with the name Tiffany Swift *tut tut*

Automation

  • Three people suggested using automation to prevent you from having to press the key manually. I’ve included these below.
    • Does the requirement to constantly press a button match the initial requirements and could this not be carried out via automation?
    • Do what Homer did in the Simpson’s – get one of those little nodding bird toys to press the key for you!
    • I’d spend the 2 minutes, 59.9 seconds between key presses training a monkey to do the job.

Compliance

  • Two people suggested just continuing to use the program until it produced your results.

Documentation

  • Someone wrote some test cases & highlighted what to do if the world ended or not.

Selfishness

  • Someone suggesting nuking the site from orbit!  I’m calling them selfish as I’d assume they’d not be on earth when they did that.  Either that or they’re just stupid :-)
  • Someone also suggested building a time machine.  Now, if you’ve ever watched Star Trek you’ll know this is a really bad idea!

Requirements

  • Someone questioned the purpose of the program.
  • The same person also questioned who it was intended for.

Approach

  • Someone suggested sitting Tiffany down and using the program with her, taking notes & questioning aspects of the program while she uses it.

Assumptions

  • Most people assumed the program was related to the world ending
  • Most people assumed the world would end.
  • Most people assumed pressing a key on a program every three minutes was acceptable

Risk

  • Two people questioned the probability of the world ending in five minutes
  • One person questioned the relationship between the world ending and the actual program
  • Someone wanted to know how the program would make the world end.
  • Someone stated that the world could end at any time!

Design

  • Two people questioned whether the time you pressed the key at mattered
  • Someone asked if you could just hold down the key
  • Two people wanted to know how long the program took to complete
  • Three people wanted to know what results it produced
  • Someone questioned why it produces these results

Other

  • Someone highlighted that if Tiffany was hot, they’d work harder for her!

So some pretty good stuff came from our R&D team, for me they had cracked all but one aspect of the challenge.  However a technique was suggested which would have probably cracked that last aspect as well.

Thoughts from afar

I’ll move on then to the testing community who had also taken me up on my challenge on twitter.

Thank you to: Michel KraaijPeter Haworth-LangfordSimon MorleyStefan KlänerStephen HillSteven HedleyThomas Ponnet who all joined in & despite my best attempts to put them off they still kept coming back with more questions like all good testers do.

I kept a feed of everyone’s responses to the challenge here.  There were some very interesting questions & funny remarks in this; I’d recommend giving it a read if you have time.

So here’s a breakdown of the responses from the testing community that differed from our R&D team:

Requirements

  • Someone asked what type of testing the program was doing
  • Someone asked for the non functional requirements of the program
  • Someone asked for more information & got it!
  • Someone asked if they were using or testing the program

Stakeholders

  • Two people asked who it was important to
  • Someone asked the boss to provide quality related questions
  • Someone asked why the boss cared about the results

Compatibility

  • Someone wondered if the modifier keys on Windows / Apple would be any different.

Program

  • Someone asked what would happen if we didn’t press a key
  • Someone asked if you could press the same key more than once
  • Someone quizzed why we had to press the key every three minutes
  • Someone asked what benefit the program provides
  • Someone thought outside the box & asked if pressing his car key would work

Usability

  • Someone suggested raising a usability issue against the program!

Automation

  • Two people expressed that it should be automated, one declaring “no testing occurs here”

Selfishness

  • Someone decided you should tell your boss to do it, and then do a runner!

Doomsday

  • After what seemed like forever someone finally questioned how the world would end :-)
  • Someone questioned how the program was related to the world ending

When to say no

  • Someone related the test with a monkey!  Then stated they could waste their whole weekend pressing this key
  • Someone expressed that they would be reluctant to accept the application if they had to press it every three minutes

Time frame

  • Someone questioned the time frame of five minutes

Approach

  • Someone stated I’m testing without a mission, as after quizzing me they’d realised I’d no idea what the program did!
  • Someone suggested using Michael Bolton’s “I feel stuck” heuristic

Techniques

  • Lots of people expressed boundary tests

So between the two groups of participants there were a lot of responses, with some pretty good questions asked.  A lot of people made assumptions and never bothered to question them, leading them astray!  Remember assumptions can be dangerous!

Breaking it down

So let’s quickly look at how you could have solved the challenge.

To solve the first part we could have looked at the challenge as a whole, and broke it into sections.

1. The world could end in five minutes!
2. You’re currently in the office using a program that a wannabe programmer “Tiffany Swift” has developed for your team. It requires you to be present to press a key every 3 minutes. Your boss is expecting the results from this program when it completes! He’s stressed it’s very important.

We can see there are two main things; the fact that the world could end & a program.  From breaking this up we can begin to raise questions on this & some people did do this.  A few people directly questioned the relationship between the fact that the world could end & the program.  That’s one of the things I was looking for; they did in fact have no relationship.

We could further break this down now that we know the relationship is void.

1. You’re currently in the office using a program that a wannabe programmer “Tiffany Swift” has developed for your team.
2. It requires you to be present to press a key every 3 minutes.
3. Your boss is expecting the results from this program when it completes! He’s stressed it’s very important.

But why press the button?

Some people considered the technical ability of Tiffany which was good.  Others focused on the fact that you had to press this key every three minutes.  A bunch of people suggested using automation to solve this, which was also very good!  Although not quite what I was looking for.   Michael Johnston suggested observing Tiffany using the application and taking notes & questioning her about the program, which in itself could be looked on as a type of usability evaluation.  This was really good, by taking this approach we could have found one of the other aspects I was looking for.  Stephen Hill said he would just raise a usability issue against the program!  You’re correct, why do we have to press this stupid button in the first place anyway?  Sure, automation would provide a workaround to the problem, but the issue here is having such a bad usability flaw in the first place.  That was Tiffany’s flaw, she wasn’t a programmer & while she had created a useful program she never could figure out how to stop it from forcing you to press a key to continue every three minutes.

Reminds me of a time…

We don’t even have to raise a bug always; we may find that just by communicating our problem with others, someone with the skills to help may be able to fix it easily.  Or better still, they might provide a more effective solution or alternative.

I remember once my product manager at that time had asked me to write up a script to extract some stats from our session logs.  The aim of this script was to find the amount of database calls being made at any given time when using our product.  It was fun, I enjoyed writing up this little script to do this, and the next day I sent out an email to the rest of our R&D team showing them this script that I’d made, hoping it would be useful for them also.  I got a reply pretty quickly from a developer “Oh that’s a lot like an awk script I wrote a year or so back!”, he then went on to share his script with the rest of the team as well.  His script was excellent; he’d spent a lot more time on it than me & had made it do lots of cool stuff.  To me that had highlighted the power of communication.  Just from discussing what we do, have done, or are doing, we most often come away with more useful knowledge, which could have been lost had we not communicated it in the first place.  Now this developer on the other hand, could have shared his useful script a long time ago, which he probably viewed as a personal tool to aid his job.  That script is now heavily used by the rest of the team :-)

Times running out!

Anyway back to the challenge & sorry for jumping off topic slightly.   There was only one other assumption that I was hoping someone would question & someone did eventually.  One of our developers Stuart Chalmers pointed out that the world could end at any time & this had made no sense to him in the context of this challenge.  Ah!  Most people assumed that the fact the world could end was fact!  Only one person had thought to question that assumption.  Could, would, might, possible, won’t?  In hindsight keyword based testing would probably have been useful here.

So that’s all I was looking for.  Someone to determine the program and the fact that the world could end had no relationship.  Likewise the fact that the world could end didn’t really matter, it could end this second for all we know :-)   Finally having to press a button on a program every three minutes is a waste of people’s time!  Let’s fix that so we don’t have to press the button :-)

Lessons learnt

Are we done yet?  Well you could be if you like; I mean the challenge is solved now after all.  However you might like to read on a bit more to see what we can learn from this, after all that’s the best part of any challenge :-)

The challenge highlighted to me that people skip usability quite easily if they hadn’t considered it as an aspect in their model.   Some people provided workarounds to the usability issue, but no one had a solution.  My company is excellent at considering usability at the requirements stage, they also consider it at design & after this some people will highlight a few things here and there.  So for me I think my company would probably benefit by inputting usability feedback at the later stages.  One of the things I’d been looking at doing myself was usability evaluations for our new features with end users.  This would help for sure; we could also look at reviewing standards such as Elmer and possibly training up some people on community recognised courses.  Even though we are good at considering usability as a business, I think you can & always should improve where possible.  To me these seem like cheap suggestions to try, so hopefully we will.

Questions, questions, questions!

Twitter’s testing community or at least those that joined in on the challenge, wow!  What can I say?  You don’t give up do you?  At times it must have felt like you were asking questions to a brick wall!  Such were my responses.  Probably the most useful question anyone could ask came from Michel Kraaij who had simply asked if there was any more information.  He got his extra information & asked for more, from which I told him I didn’t have anymore but dropped him a subtle hint!  He missed the hint which was easy to miss anyway, but as a tester that’s an excellent question to ask!  Just even asking people questions in the first place is a powerful technique in itself.  Have you any more information?  What testing would you recommend here?  What tests have you done?  What does the customer think?  You could go on forever!  I think though some people have a fear of looking silly.  Remember, you’ll only look silly the first time you ask something.  Even then it might be just yourself who thinks they look silly.

Michael Bolton told us a story at his Rapid Software Testing course & the details that I can recall are sketchy, but from what I remember of the story he was a product manager who asked questions all the time.   I mean he’d spend his day asking a whole pile of questions to different people so he could gather up the information he needed to do his job.  After some time, people began asking him the questions.  He became the go to guy!  Why?  Well people thought he knew most things, even though he was the one doing all the asking!  Funny thing that, but hey, in the end through communication from asking all those questions he probably did know most things, as by asking those questions he was learning.

What problem are we solving here?

So what other cool questions got asked?  I quite enjoyed Stefan Kläner’s observation that I was testing without a mission, as after all I was just pressing a button waiting for the program to finish.  Was I testing though?  Was I even checking?  I don’t think I was doing either, in fact I think my valuable time was being wasted by this programs flaw.  A few people asked who the program was important to & who actually cared about its results.  Both of these are excellent questions, both could help you gather more information on your users & stakeholders.

Another good one from Stefan Kläner was asking about what benefit the program would provided?  After all we build programs to solve people’s problems right?  If there isn’t a problem to solve, then why are we wasting valuable time & money on it?  That’s a question I use often “What problem is this trying to solve?”

Help!

A few people questioned the timeframe which was good; it’s always good to know what our influences on time are e.g. budget, resource, key events and so on.  Michel Kraaij also suggested he might use the “I Feel Stuck!” (Michael Bolton) or “Mission Rejected” (Cem Kaner) heuristic.  I liked his response a lot because often testers don’t know when to raise the white flag & admit defeat.  Often just a break or some help is all that’s needed.  Sometimes even, it might just be impossible to continue.  To me asking for help is probably the most important one here, I’d consider that a key skill in itself, knowing your limits and being able to ask for assistance.  Its problem solving after all, just asking someone for help, you’ll solve it more efficiently.  Always asking though, that’s another thing!  Don’t do that :-)

So that’s it, I’m sure I could pick out many other things to discuss from this, but I think I’ve went on long enough.  Plus my fingers are getting sore now from pressing all these buttons!

Thank you to everyone who joined in it was fun & thanks to all those who took the time to read this :-)

Related posts:

  1. Challenge: Testing the Future
  2. Challenge: Escape! Games+testing = fun
  3. The Impossible Challenge?
  4. The usability challenge!