Challenge: Escape! Games+testing = fun

A little background

I recently watched Markus Gärtner’s excellent webinar “Alternative Paths For Self-Education In Software Testing”.  It was very enjoyable & I picked up a couple of things: Matthew Heusser has a testing school called the “Miagi-Do School of Software Testing”.  I never knew this, but I’m putting it on my to-do list (beware Matthew I’ll be pestering you in the near future about this).  Another which also had a funky name was “Testing Dojo’s” I’d never heard of these either & the way Markus had described them sounded much like something else which was on my forever expanding to-do list: Doing a screen cam of myself testing.  I’d planned to use this to share with my team to see if they could pick up anything from the way I test.  Dojo’s sounded much better though and is more common sense than just showing someone how I test.

The way Markus had described Dojo’s was that these are a challenge on a group of testers, from which they could learn from one another.  These could be group sessions, singleton or paired but at the end you discuss and compare results.  Much like the usability challenge I’d done previously.

If you don’t have time to catch his webinar in the EuroStar Achieves check out Rob Lamberts summary of the talk Markus did at AgileTD.  Rob is probably very angry right now at how much I’ve talked about nothing, but I think the background is important, so I’ll continue .

Anyway I later had a chat with Markus over twitter about this, it was at that point Thomas Ponnet talked about how he’d done these in the past with his team & offered to share his results with me.  He did & it was an excellent insight which allowed us to see the pro’s and cons of various approaches to these Dojo’s, thanks again Thomas for taking the time out to do that.

So all that was left to do was to find something to test & I did thanks to Albert Gareev who’d previously done a challenge on a game called Escape, that he’d titled “Real time input game challenge”.  Thankfully this post never gave nothing away & I held myself back from reading his next entry which was the results of this challenge, that would have been cheating!

The Challenge

So the fun commenced.  I’d previously talked about my intentions to do a challenge so my team were expecting the email that followed:

“Here’s the challenge Escape is a very simple flash game, it has a few bugs.  Your job as a tester is not essentially to find the bugs but to test the application and report your results.

It’s neither a competition or a comparative evaluation on anyone, it’s simply a learning exercise from which I hope I’ll learn a lot & you’ll learn a lot, making us better testers all round.

Rules:
-No more than 2hrs are allowed
-No cheating

I’m not going to tell you how or what to do just email me whatever you think is appropriate.

Kiss Kiss Darren.”

The last line I made up, but it got your attention right?

The results would be sent to myself & my boss Michael, we both promised not to look at anyone’s results until after we’d took the challenge ourselves.

The Results

If you have any intention of trying this challenge with your team then stop reading now!  You’ll only spoil it for yourself.  Go away and take the challenge then come back and read the rest later.  You could even share your experiences with me, I’d love to hear them.

So here follows a summary of everyone’s results:

Participants: 12 people

Recommendations:

• 3 people recommended further testing
• 3 people declared if it was fit for release or not (1 no, 2 yes )
  • 1 person recommended a couple of major defects be fixed to make it shippable
• 1 person recommended a code review
• 1 person recommended the game be centred
• 1 person recommended sounds
• 1 person recommended a scoring system
• 1 person recommended security testing

Exploits:

• 3 people found cheats
  • 2 people found the right click infinite score cheat
  • 2 people found the escape key never ending game cheat
    • 1 person exploited this for a soak test
  • 1 person found both
• 1 person cheated and hacked the source code *read the rules*

Questions:

• 1 person questioned the name of the game “Escape” when you can’t actually escape
• 1 person asked what the intended audience was & recommended removing the decimal places from score times if it was for very young children to aid learning

Types of testing:

• Most people attempted cross browser testing, although not all managed more than three browsers
• 3 people inspected the source code
  • 1 person traced root causes of defects from the source code
• 1 person tested what happens if JavaScript is disabled
• 1 person did manual stop watch tests
• 1 person analyzed patterns in the games behaviour
• 1 person did accessibility testing for colour blind people
• 1 person did a soak test to check for memory leaks & CPU increases, none being found
• Everyone did cosmetic testing
• 1 person noticed a single pixel abnormality
• A couple of people found the instructions sufficient
  • Mostly everyone else didn’t
• For some people obvious defects got missed
• Some were just interested in reporting bugs
• Some documented more and as a result found less, but over time could possibly have found as many with a documented approach?
• A few considered usability aspects
• Two people recommended backwards compatibility testing

Documentation:

• 1 person wrote test cases
• 3 people planned their testing, documenting the types of testing before hand.
• 3 people documented their approach
• 3 people documented browsers used
• 1 person documented the name of the application they were testing
• 1 person prioritized defects
• 3 or 4 people used media (video & screenshots) to report defects, although only when words would not provide a good enough description

Phew, quite a read huh?  I’m sure you’ll agree the results proved very interesting.  I later had a look at Albert Gareev’s results page from when he did this challenge, thankfully we didn’t miss much.  I think the only thing that wasn’t noted in anyone’s results was Albert’s modification of local machine time, to get higher scores, nice catch Albert.

The Imposter!

Unknown to my team, I’d talked my partner Karen into trying this out.  She spent 15-20 minutes on it before getting bored.   She did pretty good I thought, one of her questions was excellent “Who is this game intended for?  I’d ask that first because if it’s intended for small children to aid learning then the decimal point in the score will confuse them.  If so I’d recommend removing that”.  Wow, that’s pretty cool!  It also highlights what Kaner, Bach, Bolton & many others say “We can learn a lot in testing from looking at other professions”.

Time for a debrief

A few days after the challenge we had a debrief, at this point in time no one knew each others results, all they had was the summary above.  This also proved interesting, I’ll summarise this debrief:

Breakdown:

• Everyone really enjoyed the challenge
• Everyone felt they learned a lot from it
• 2 felt they’d missed a lot of issues
• 1 suggested this would be a good idea when a feature was added but had no requirements
• 1 said he’d learned he needs to plan his time better with time constrained tasks like these
• 1 person saw our weakness as a team is not reviewing code & suggested someone do a tutorial to the rest of the team to help us improve in this area.
• I questioned what’s more important approach or defects?
  • Everyone agreed approach is more important
  • We noticed people with an approach covered more & found more issues
  • This challenge specifically stated that the approach was more important but only three people did this.  I’m not sure what this means just now but perhaps this is something to follow up on.
• I asked if people would like to share their results with each other so we could learn more
  • 5 people said no
  • 4 said yes (two not present at the time), these 4 went on to share theirs
• We got a volunteer to arrange the next challenge
• Two people recommended involving developers in the next challenge

For me learning was the most important part.  I think we all learned as a team how to approach these challenges better & the different types of testing we can use.  We shouldn’t be concerned only about bugs, although these are important, finding bugs only accounts for a small proportion of a testers role.  We can question, recommend, and investigate further.  Most importantly, we should document our findings in such a manner that when we present them to someone who can make informed business decisions they make sense & allow them to easily make those decisions.

I think starting out with a game put people at ease, I know myself I was pretty hyped up about the fact I was coming into work to play a game for a couple of hours .  This just shows how much our company promotes learning, we’re all very lucky to have that opportunity, believe me I’ve personally been told many a horror tale from testers working elsewhere.

Further reading

So if you’re interested in trying a challenge with your team mates remember these can be on anything, they could even be a challenge on an individual.   I’d recommend reading James Bach’s excellent post on “How challenging each other helps the craft”, Lanette Creamer also did an excellent challenge to her readers with a sci-fi twist “Beam me up challenge”.  The beam me up challenge is what probably first spiked my interest in these.  You can also have a look at the previous usability challenge I set my team.  These can be an excellent source for learning I’d highly recommend you give them a try.

Remember this post although very long is only a summary of the challenge we did, I don’t think it highlights how beneficial this challenge was to our team & how much we are still learning, in fact even as I write we have a sub challenge from the results of this on going.  Further to that I just got dropped an email today with another challenge from one of the team leads in our company   Gosh, word is spreading fast!

So that’s it, thanks for reading.  If you’d like to post a challenge to my team please do, I’m sure we’d be up for it & please let me know how you get on if you do something similar.

Related posts:

1. The usability challenge!