Overcoming the limitations of traditional test design

Test design is often a very time consuming process when it need not be.  People are often weighed further down by tools such as jack of all trades, master of none test management systems which dictate that tests are written in a less than optimal format, or introduce additional frustrations to those that try to use them.   External pressures outwith the team may see them required to produce sufficient quantities of tests in relation to features or requirements being developed.  Problematic, this certainly is.   When it seems everything is conspiring to make what should be a very creative experience, regimented and structured, then it is time to step back and think of alternative options.

Mind mapping helps reduce the cost of test design significantly by visualizing the item being designed and providing a holistic view point to develop ideas rapidly.   Essentially what this really means is that the creative side of our brains is much more engaged whilst developing ideas in a visual format.   Additionally by providing a birds eye view of the elements under test, ideas will spring to mind which may have not been consider in a linear thought process associated with writing tests in a document, or test management system.

Obviously this is not a straight forward technique that will work for everyone, so in the following series of posts I would like to introduce approaches to test case design using mind maps, and introduce considerations to make your experience with the concept more beneficial and powerful.

Test Conditions Vs Test Ideas

We will start off simple and look at the idea of test conditions Vs test ideas.  Let’s explain each briefly to remove any uncertainty behind them.   A test condition is a check that states based upon our knowledge of the item under test, we expect something to occur.  An example of a test condition might be:

“The login functionality should be easily extensible, so that in the future, should we want to add in an additional field such as a user pin, we could do this easily by extending the capabilities of the login API.”

Now if we were to take such an extensive approach for everything we thought of, the process of designing tests would become very long.

Test ideas on the other hand essentially allow you to note your thoughts down in a rapid but less extensive format.  Let’s look at the previous example for a test condition and switch it to a test idea:

“Extensibility > Login > Additional fields > e.g. user pin”

Now this is certainly less readable, and may prove more problematic for others to understand than a test condition, but it can greatly reduce the time it takes to design your initial tests.   You can also see how the format of test ideas blend well to mind mapping, which benefits from quick sharp notes, as opposed to longer sentences.

Additionally, consider a persons level of understanding of a type of testing.   It is much more difficult to write test conditions for something which we don’t have a full understanding of yet, but in a test idea format, we could cover this more quickly without going into too much detail.  Take for example, encryption used to secure credentials for a login process.   There are many elements to consider here, such as the type and strength of encryption used.  Has it been implemented correctly?  The distribution of keys and countless others concerns which go beyond my experience.

Summary

In the following post I go over a lean form of requirements gathering and planning that I did with my new team at Maidsafe.  I provide examples, and tips for those who might find themselves in similar situations.  I finish up by identifying the key lessons we can learn from this as testers.

The problem with visions

As most readers know I recently switched jobs to go work for some crazy company on the west coast of Scotland, who’d planned to change the world by bringing back a little thing called freedom, or at least in its online sense.  Most of my time during the first couple of months at MaidSafe had been consumed with other things.  The biggest other thing I got the pleasure of doing was a scoping exercise, for two of our new products being developed for our sister company Sigmoid Solutions.  Now having spent so many years at the other end of scoping exercises looking for gaps, flaws, risks, and even identifying new requirements, this for me was a very fun experience, and it gave me a real chance to make use of the testing skills I’d developed over the years.

I guess it was my own fault that I got side tracked from testing to write up requirements for two of our current projects.  You see, I came in with a fresh pair of eyes and realised that the team had been developing from a vision; a great one at that, but still a vision.  For me a vision is essentially someones dream, or an idea which they know roughly how it should end up, but not the exact steps to get there.  As work begins on a vision, the steps become more apparent, and evolve over time.

The problem for me when working with visions is that they’re large fuzzy targets somewhere far in the distance that people are aiming for.  The process of actually reaching that target can be pro-longed by not having clearly scoped out what is required to achieve these goals.  More often than not, teams will seek perfection whilst trying to meet these them, adding in nice to haves that aren’t really essential for the business, or at least for the current context.  Feedback loops suffer, and god forbid anyone asks how much longer it’ll be before something is in a releasable state!  After all we’ll only be learning what’s required as we go, right?

Well that’s my problem with visions, and don’t get me wrong, they actually can work initially for small close knit teams, who can easily communicate and develop initial visions rapidly.  That’s how MaidSafe begun, a team of three people developing a radical idea for the first couple of years, and it worked!  They got their initial concepts coded, and certainly for them in the first couple of years, working from a vision in such a small team had paid off.

Teams grow, and as more people become involved things inevitable change.  That small sharp communication ring between three people suddenly becomes unmanageable.  Pockets of unique knowledge begin to form in lots of small clusters throughout the team.  The business side begins to ask difficult questions on current progress, pilot dates, and how much more needs done.  No one person can tell the whole story anymore.  Teams grow, and processes need to adapt to manage that growth.

Creating a manageable process

So I highlighted the issue of working from a vision to the team and drew up on a whiteboard a three tier hierarchy of what I thought we should be doing.

1. Vision – (Customer and stakeholders needs)
2. High level requirements – (Business requirements document / scope mind map)
3. Technical breakdown – (User stories)

A simple model that gets the objective across, but still allows room for stakeholders to have their visions.   We simply break these visions down now, into something which is much more manageable.

Now the problem a lot of companies have with documenting requirements and any form of planning, is that it takes time.  Those that would rather just jump right in and develop features, realise that after time it pays off to do some form of requirements scoping exercise, prior to writing any code.

I’m not convinced that a team has to spend weeks, or even months planning and writing up requirements before they start development.  With this in mind I proposed a rapid scoping exercise that would break down the existing visions for each product into high level requirements.  I would then be able to write up these requirements into a Business Requirements Document (BRD), and then have the key stakeholders decide from this document, what was required for each of our products upcoming Alpha releases.

The requirements we agreed to do from the BRD would then be taken to each team and broken down during technical requirements meetings.   This would result in a bunch of user stories, which could then go through a form of planning poker, giving us an estimation of workload and confidence against each broken down user story.  Lastly, lead developers and stakeholders would agree on a monthly allocation of work to meet a key deliverable.

So let’s quickly look over what we essentially did.  I find this to be a much more lean form of requirements gathering, which with the methods used gives a good middle ground for teams not wanting to spend too much time on requirements gathering and planning, but still want a decent level of task breakdown.

Step 1: Breaking down visions

We had two products Alpha releases to break down.  We knew what we had done so far, but weren’t sure about what we still had to do.  Being a mind map fanatic, I thought a session of collaborative mind mapping would work well here.  We also used a series of Lightweight Personas to help expand out scope and generate fresh ideas.

To allow us to map out what was required for the first product (which had much more development work required than the other) we drew up a checklist on a white board of the key product areas to be worked on.  From each we then mind mapped out what was required in terms of business requirements.  We’d also drew up on another white board a list of lightweight personas that might interact with the product in some way or another:

• User
• Administrator
• Seller
• Buyer
• Hacker
• External Developer
• Tester

I used a different colour pen for each persona.  Strange?  Not really, the colour separation engages the right side of your brain, which deals with your creative side, as a result inspiring fresh ideas.

The collaborative mind mapping went well, and within the space of a few hours, we’d scoped out our Pro products requirements.

A collaborative scoping mind map for our Pro products management interface.  We clearly separate dependencies from the other version of the product, which will feed into the scoping exercise for that product if needed.

Another collaborative scoping mind map, this time for our vault and product installers.

This map was done on a day I was on holiday.  A collaborative brainstorm between two of our lead developers, to work out what was required for out distributed vault network.

The other product Core, only took a couple of hours as well in a later brainstorming session.  We opted to map the entire scope onto a single brainstorming map, thinking there wasn’t as much to it.   We used dashed lines to highlight dependencies between elements on the map .

This is our Core Beta scope mind map. Its broken down tasks later went on to be collectively know as Alpha scope since we’d realised that we had more to do than we’d originally thought.  Notice the use of an actor on this map, in this case customers that we’d like certain tasks to occur within their environment.

So you can easily see that the task of getting the initial business requirement scoped out, can be done very quickly.  That’s the real power of mind mapping, and using additional tools in conjunction such as lightweight personas and checklists of product feature areas.

If you plan on facilitating a collaborative mind mapping session yourself, keep it simple, only have the people necessary in the room.  Keep people on track, and try to steer the mind mapping exercise yourself, using peoples input to develop the map.  Prompt often upon lightweight persona’s to inspire ideas.  Probably the most important point if you’re a facilitator, is to make sure that you’re comfortable with mind mapping, since you’ll be doing most of the drawing.  The facilitation, drawing, and thinking of ideas can be quite a handful for some, so having experience mind mapping certainly helps.

Step 2: Deciding what’s important

Now with the business requirements mind mapped, I had to take these and present them in a format suitable for our key stakeholders to decide upon  what was important or not.  I decided to write a Business Requirements Document (BRD) for this, similar in style to the spoof requirements document I wrote for a Week Night Testing challenge a while back.

I had decided upon this because by its nature, it allows for further expansion of requirements and identification of new ones (gaps).  In hindsight though, I’d probably not go to this amount of effort again and simply create a mind map of all the scope combined instead.

So the idea here is that I got the key stakeholders into a room with the lead developer for the project and discussed each requirement with them.  The stakeholders, upon hearing the details about each requirement,  would then assign a MoSCoW priority to it (with the woulds being out of scope for this release).  The whole process took about two hours, and at the end we’d clearly identified what was important for the business, for this months release, and what wasn’t.  Some of the what wasn’t and lesser important tasks, would easily have been done previously if we’d been working from visions.

The stakeholders loved the BRD document.  I honestly think though, that they’d have been just as comfortable looking at a digital mind map.  It would have saved time, as it took me a couple of days to write up the document, which would likely have been a couple of hours had it been a mind map.

On the plus side the document helped identify additional gaps, whilst I was writing it.  I’d also used a who, what, why format for writing business requirements.  This resulted in me struggling on a why for some, therefore making me directly question the purpose of such requirements.  After all, if you can’t justify why your doing something, then why do it at all?  I found that to be really useful, and got discussions going upon the purpose of some requirements, sometimes leading to new scope, or the removal of items.  I doubt though that I’d go to the effort of creating a BRD document again, as I think a mind map would have caught as many gaps potentially.

Step 3: Breaking down the business requirements

Now I knew what was important in terms of business requirements, I could arrange a meeting with the team who’d be working on each project, to break these requirements down further.

I decided to buy a stack of index cards, and began to write out a business requirement from the BRD on each card.  After I’d finished I got the feature team into a room and explained my plans for the meeting.  The plan was simple, a two stage meeting.   The first part would involve me discussing each requirement from the story cards, then us all discussing what was involved technically, and as a result giving us broken down stories to work from for each requirement.  This is the technical breakdown from the three tier plan that I’d originally proposed to the team.   The second part involved the team putting an estimate against each story, along with a level of confidence (high, medium low) on getting that work done in the estimated points.

I used a form of planning poker for the estimating, with a slight twist.  Since the team worked in points, we didn’t care about time, we only had to estimate in points (1-3) and let our velocity determine how many points we could complete in the month.  I’d also added an additional card into the deck (a split card).  Essentially if the task was too large to possibly estimate, and if the majority had voted to split it, we’d attempt to break it down further, and then estimate the broken down cards.

For both teams this probably took the best part of a day to complete.

Step 4: Agreeing tasks for the next sprint.

I’ve talked in the past about how mind maps can be useful for visual feedback of information.  To help me present the scope of work to be done for each products Alpha releases, I gathered it all in the form of a  large scope proposal mind map, developed using my favourite mind mapping tool Xmind.

Me standing beside a scope mind map for our Pro product that we’d printed using an industrial sized spool printer.  An equally large sized map later joined this on the wall for our Core product.

This might look a bit crazy, but in reality it actually worked out quite well and allowed us to easily pick off what was important for that sprint.  As work progressed we marked out areas on the map which had been completed, and had the other tasks for the month circled out too, allowing us to clearly see our progress.

I’ll provide hard links to these massive mind maps as you really need to zoom into them to see them.  You can view the Core one here, and the Pro one here.

We took a rough guesstimate of how many points we’d get through in the month and decided on a different main sprint objective for each product.  Our guesstimate on how much work we could get through turned out to be way off, but we knew that would be the case at the start.  We’ll learn over time and get better at this.

Step 5: Staying focused.

With everything planned, and the scope agreed and signed off, we had a clear target for the month.  To stop us going off track I’d previously introduced two things, with the later not solely being for this purpose.

Change requests, change control, or whatever you want to call them often gets lots of bad press.  People generally don’t like being told they can’t just do innovative things, or code those little nice to have improvements here and there,  that generally are never critical to shipping the product.  When I first heard of this I hated it too!  I quickly saw why it was necessary though.  After all, scope creep, and the pursuit of perfection can potentially cripple teams, and in the long run demoralise people.  So you need to get the balance right somewhere in between.  That’s why if your change request / control policy is implemented well, and continually monitored, it can work wonders.

The other was just a basic daily standup, which remember in the past for a small team wouldn’t have been necessary.  All I’ve did here is come into a growing team with a fresh pair of eyes and said “Hey, this could work well for us, let’s try it!”  Daily standups are good for watching if people are going off track as you can listen out for telling signs such as:

• It would be good if we had …
• At some point we should …
• There is a few good ideas for …
• I rejected this because there is a better way to …
• There’s some more to be done in blah blah blah, will I add a story in for that?

You get the idea.  Divert these to your change request process, and if you manage it well people will begin to appreciate it in the long run.   If it becomes that place people know as the graveyard for ideas, then you’re clearly doing it wrong, and need to work at improving this process, or find something better that fits your context.

From our end of sprint retrospective, we got a lot of good feedback on the new approach to planning.  People generally enjoyed the greater visibility from being involved in scoping, and technical breakdown / estimation meetings.  They also enjoyed having a clear achievable target for that month, as opposed to a long distant goal.  There is lots we can improve on, and no doubt this entire process will change significantly over time.

Having short snappy sprints, with retrospectives at the end certainly keeps a real sense of momentum going, and creates an aura of excitement at times (or perhaps that’s just me?)  As long as we continue to self improve ourselves and learn from our mistakes, we can do wonderful things.

Lessons learnt

Personally I’ve enjoyed the whole experience of sitting on the other side of the fence, scoping out and planning releases, and even getting down to the nitty gritty of writing up requirements.  Although it wasn’t the experience I’d have expected to gain when I took the job, it has certainly proved very worthwhile and gives me skills which I can apply directly to my job as a tester.

It certainly proves to me how invaluable collaborative mind mapping can be.  I’ll continue to use this as a means of generating and expanding test ideas with others, and to help solve difficult testing problems we encounter, such as tackling scalability, or proving difficult product claims.

Lightweight personas helped spark fresh ideas when mapping out requirements scope.  As a tester these will always prove essential for me in test case design, and big picture ideas.  They’ll help me think more in terms of problems being solved by the product, that in turn identifying key issues,  such as extensibility problems with design which could affect third party developers at a later date.  Perhaps even, accessibility concerns for a persona we’d not considered before.

Mnemonics, continue to prove useful as long as you can directly apply your own, when needed.  The who, what, why mnemonic I’d used to develop user stories around the business requirements, came into my head from me trying to solve a problem.  That problem being, the presentation of requirements in a readable, and justifiable format that could be read, and understood by key stakeholders.  While other peoples mnemonics may prove useful in the short term, being able to build your own set will make them much more beneficial for you to use, in your context, in the long term.

There is a lot more I’ve learnt from this experience, but hopefully you’ll have picked this up yourself from what you’ve read already.  Rather than turn this already long post into what would feel like a small e-book, I’ll finish up here, and extend a warm thank you for taking the time out to read my thoughts.

So test case design can be a very powerful technique if used correctly.  It can play part in all aspects and stages of the software development process.  Often it is misunderstood, and false beliefs are imposed upon its limitations.  We can see easily that such beliefs are unfounded, and the benefits are there to be taken, and harnessed by your teams.

I hope that from sharing my thoughts, others will in turn share techniques that they have found useful for test case design.  Thanks for reading.