Tales from the trenches: Lean test phase planning

Out with the old, in with the new

Today I’d like to talk about a lean approach I use for planning out test phases.

I’m a strong believer in reducing waste.  In fact, if I don’t see the value in something I’ll actively question its purpose.  If I still don’t see value based upon the response I receive I’ll look at ways to change it.  Not always of course, as the process of change might not reap worthwhile rewards; perhaps it’s just cheaper to live with it?

Now our traditional approach to test phase planning has probably been like many other companies.  It’s a task near the end which we’ve guessed upon how long it’ll take from past experience and early predictions based upon risk when we’ve first seen or indeed didn’t see the initial requirements.

Now that’s not a bad thing, and of course at that early stage we’ve not much of an idea other that our traditional regression test tasks of what will be in our test phase.  We might have an idea from some tasks, however like all plans things will likely change.

So in order to devise some sort of plan for our test phase we’d get together and discuss as a team what we should be doing and discuss risks along with some possible others tasks that we could do.

So where does lean fit in to this?  We’ll it doesn’t right?  Aspects of it might apply to an extent, but we still have waste!  It’s not very flexible!  Oh and what about the customer?   Don’t they get a say in this?

So what follows is an approach I’ve used successfully which encompasses all aspects of lean when planning your test phase.   Remember though lean is only a concept, and by the definition of lean itself by taking only the aspects we find useful, we are in essence being lean.  Likewise, hopefully you’ll take from this what you find useful and adapt it to suit your needs

Step 1: Realise that plans change

It’s true, and we all know it very well.  Even the best planned plans change!  As such why would you even consider early test phase planning?

Sure, you can get an idea of the tasks you may or may not do early.  One might even have as part of his verification checklist “Possible test phase task?”.  By that I mean identifying as we test high risk items that might be worthwhile taking note of early for inclusion into our test phase.

I personally think the whole idea of an initial plan is pointless at this stage.  Perhaps there may be value in it should you generate that collaboratively with the correct people involved.  Your true source of knowledge will be your tester who has been testing that feature right?

Step 2: Empower your testers

Since the testers have the experience and knowledge allow them to generate the initial scope.

Something I did which quickly got adopted by the rest of our team was to mind map the initial scope of possible tasks we might choose to do in our test phase.

You can see from the map below that they’ve been sectioned into types and prioritised.  The priority comes from risk.  Risk in this case being the level of test coverage this area has had already and the possible risk to the business should this task not be executed.

Please be aware that all images used in this post can be expanded larger in a pop up by clicking on them. There is quite a lot of detail in them so you’ll likely have to do this to see their contents clearly.

This was excellent for us as we didn’t have to care about our time constraints, we’d just note down what is important to be tested.  If you had a test phase that lasted two weeks or twenty it didn’t matter.  You’d note down as much as you could think of that would add value to the testing that’s already been done throughout the release and you’d also note the high risk items that are essential to be retested.

Anything that doesn’t fit into the time constraints can go.  We’ve addressed risk so we already know what is most important to us.  We’re not running the risk of not thinking of possible tasks because we are so obsessed by what will fit or not.

I’d like to think of this as a scope proposal at this stage.

Step 3: Collaborate

Now that you’ve devised a set of tasks you think should be done, involve other parties.  Get feedback initially by emailing or discussing with your developers who’ve built the feature/s.  See if anything else needs added or if suggested tasks won’t add much value.  Discuss your priorities.

In general people like reading mind maps if they’re organised nicely.  I’ve found that when used as a form of reporting they receive positive feedback as a mechanism for displaying information.  Certainly for management, they’d much rather look at the mind map and have me discuss aspects of it with them, than read a lengthy chunk of text, or even attempt to discuss something with them without a visual cue.  Generally people will find it easier to feedback on a mind map than a complex conversation or lengthy email.

Get your customer or customer representative involved.  In my case I would discuss this with a customer representative and talk them through each of the suggested tasks, seeing if they were acceptable and if anything else needed to be considered or dropped.

By talking initially with your developers and then to the customer you’ll firm up your scope of possible tasks for your test phase.  The likelihood, or at least in my experience is that the developers you’ve been working with will actually be running these tasks throughout the test phase along side your test team colleagues.  So by having those initial discussions you’re already getting their buy in to the process, which is very important if you want them to see benefit in the tasks at hand.

Myself, I’d run the tasks I’ve proposed by the developers I’d been working with and my customer representative.  I’d then involve my other test team colleagues and then finally the feature team lead.  After all that even if my scope hadn’t changed much I’d at least have more realistic task priorities from knowledge & business risks which I wouldn’t have know about.  Again, everyone would be bought in to my scope proposal as they’d had their say.

You should now have test scope for a feature.  If you’re a test manager you’ll have had each of your testers devising similar maps of possible tasks to test the features you’ve been working on.  You’ve now got an idea of what will likely be in the test phase in addition to traditional test tasks.  From simply empowering your testers you’ve not only gained their respect, you’ve also got a much more solid set of tasks that the whole development team are hopefully bought into.  Even if some are not you’re in a much better position for buy in than in the past.

Step 4: Formulate a plan

So you now have your scope.  You’ve prioritised it and know what is essential or not, so you can begin to devise a plan.

I said previously “empower your testers” I still think that’s important.  At this stage if you’re a test manager I’d be reluctant to try and take over.  Advise on time you’ll need from people and allow you’re testers to plan their feature teams tasks.

Collaborate on estimations for these tasks.  Find out from your team how long they think these tasks will take.  This could be as simple as a quick thirty minute meeting.  You could even take a guess at estimating them yourself and then asking for feedback.

Begin planning tasks in.  My advice here would be to get an initial quick sweep of the functionality and address high risk items.  This will be your initial test cycle from which you can say by X date we will have covered most of the functionality; this is essentially our critical plan which we don’t want to drop tasks from.  These critical tasks can overrun, or be impacted by sudden holidays or sickness, but we know that at most other less riskier tasks will have to drop.  Hopefully you’ll have planned in some contingency to negate this risk anyway.

Once you’ve done you’re initial sweep it’s probably best to plan in a second cycle of testing which might just involve a one day sanity of the features by everyone in your team or just some of your team, depending on the risk & complexity of the feature.

Less risky tasks can now begin to be added into the later stages of the plan.

I’ve provided an example of the initial plan our team worked from when testing our text chat feature during our currently on going test phase in the picture below.  This plan did go on to change slightly as some tasks had overrun, and when sickness or holidays occurred.  However it didn’t drastically change, and at most some automation and low risk tasks were dropped.

We still have a little time left until our test phase is actually complete, but at this late stage things will likely not change much.

You can see from the plan that our initial test blast provided us enough coverage and feedback by the eleventh of January.  This, if you take into account the Christmas holidays and reduced workforce over that period was about seven days into a planned four week test phase.  By that point we’d known what our real actual high risks were.

A few days later followed a second cycle of testing followed by lower risk test tasks, such as extensibility proving tasks.  Of course I’m not saying that extensibility isn’t risky, for us it was less a risk as we’d raised awareness of this and planned in development extensibility proving tasks throughout the release.

It’s probably worth mentioning the importance of planning in time for continual bug fixing throughout your test phase.  Defect predictions are often highly inaccurate, but at least allow for allocation of time for bug fixing.  That’s why doing an initial scope is good as you don’t have to worry about cycles of testing, bug fixing, automation request from management and so on.  You can just devise what you think is important and plan in the important stuff that can fit into your time constraints.

Step 5: Communicate some more

It’s important you keep track of your plan and adjust it accordingly, that’s obvious.  I think it’s also important that you continue to communicate with those involved throughout to see if things are still adding value.

An example I did was simply sitting the participants down after four days had passed at the start of our test phase, and discussed aspects of the plan with them.  From this the group quickly communicated their thoughts and problems allowing me to make some changes to estimates and help resolve the intentions of any tasks that were unclear to them.

Remember it’s easy for things to go astray, often people will just continue as-is regardless, should it be a task that will overrun or simply something someone doesn’t see the benefit in doing.  That’s why it’s so important to keep constant communication going so you know then that there is a problem, rather than when it’s too late.

Summary

To wrap this up I’ll quickly summarise lean test planning:

• Realise that plans change:
  • Plan as late as possible.
  • Reduce waste.
• Empower your testers:
  • Having worked with your feature teams they’re silo’s of knowledge.
    • Let them propose tasks.
    • Let them plan their feature teams tasks.
    • Let them involve the customer.
• Mind mapping scope:
  • Increases creativity.
  • Reduces time to create scope.
  • Increases visibility of the bigger picture.
  • Identifies risk easily.
  • Doubles as an easy reporting method for feedback.
• Initial scope proposals ignore time constraints:
  • We lessen the risk of forgetting tasks due to thinking about time constraints.
• Collaborate:
  • Involve the customer.
  • Involve all other essential parties.
  • From direct collaboration, we iron out actual risk.
• Lean planning:
  • We’ve identified scope.
  • We’ve identified risk.
  • We’ve got buy in.
  • We’ve collaborated with essential parties.
  • Now actually formulating a plan is simple.
    • It will provide high coverage and feedback quickly.
    • Being flexible it will allow for change.
      • Lower risk tasks can drop if needed.
• Keep communicating:
  • Don’t ignore the value of communication.
  • Empower the customer to decide what should be de-scoped from an overrun plan.

On that note we’ve come to the end of this discussion; admittedly one way so far being just my thoughts & experiences.  Hopefully though I’ll have some feedback from the community be that positive or negative, any is always welcome

Thanks once again for taking the time out to read my thoughts and experience reports.

I’m planning a couple of follow up posts in relation to this.  One on the usability evaluation we did.  The other around the actual tasks I’d planned in as I think potentially there might be a few valuable insights from discussing them that others could benefit from.

Related posts:

1. Tales from the trenches: Lean test case design
2. Tales from the trenches: Keyword based testing
3. Tales from the trenches: Proactive testing
4. Proactive testing: Tips from Michael Bolton