Summary

In the following post we’ll look at attempting to identify the role a tester plays in a prototyping team.  We’ll discuss feedback provided by the testing community and devise an initial approach that a tester could take.

Background

My workplace has decided to put a big effort on prototyping for the next couple of months.  It’s something new for us as a company as we’ve always just went away and built fully functioning software that’s been asked for.  We’ve never had to prototype anything before.

So certainly it’s a step out of the comfort zone for everyone involved.  One which will require a lot of thought and collaboration from all parties involved.

For myself as a tester it makes me question where I belong in this team, since a traditional functional testing effort will have less importance here; something which would have took up a large amount of my time previously.  I can certainly think of places where I can provide value; however there is still a lot of uncertainty in my mind as to the role we play.  It’s not a traditional test effort for sure, and requires a new approach.

So what role does a test play in a team developing prototypes?

It’s something both myself and my colleague Stuart MacDonald wanted to find out.  So we sat down and had a brief fifteen minute discussion on what roles we could play to provide value to the team.  Jotting down a mind map of our ideas on some paper, we quickly identified some crucial areas we could provide assistance in.

Insights from the community

Now I’m sure other people out there will have a lot more experience than us when it comes to the role a tester plays in a prototype team.  As such I turned to twitter for assistance asking “How does a tester fit within a prototype team?”  A bunch of people responded, and the insights they provided were very helpful for me, helping me clarify thoughts in my head and introduce new ones.

Proactive Testing

Albert Gareev highlighted the need for prospective testing skills.  Prospective being something I refer to as proactive testing.  Certainly, proactive/prospective testing is essential here!  But to what extent?

Feeding back on requirements

As testers a key role we play is gathering information, assessing it and providing feedback on it.  For a prototype team we will have some form of requirement, or even a vision if you like?  So thinking proactively around the testing we could do for a requirement, we could review whatever information we currently have and gather any addition information that we need, review this and provide feedback on it.  Or for a vision we could do exactly the same but obviously put more emphasis on information gathering and collaboration (questioning stakeholders) in order to provide our feedback.

So now as a result the feedback, or feedback cycles we have provided will have helped stabilise scope, reduce risk, and most fundamentally assisted in determining exactly what the end goal of the prototype will be.

Collaboration and determining business expectations

Is that as far as we can go proactively?  Of course not!  Prototyping will evolve over time.  A good approach to prototyping will involve lots of interaction with it’s stakeholders to make sure essentially that the prototype meets their expectations, or at least indicates early when expectations are not achievable.

An approach we can take as testers here is to begin thinking of the difficult questions a stakeholder might ask of a prototype, and asking them early!  From doing so we reduce the risk of developers being embarrassed and the stakeholder being upset.

Our teams are being asked to produce something demonstrable, which can be picked up and demoed to customers by our demo team.  As such we already have an expected outcome placed upon us from the start.  So we could begin asking difficult questions around the business expectations of the prototypes such as:

• “Are you happy to invest X amount of time in throwaway but demonstrable code?”
• Do you expect anything to be reusable from this prototype?
• Are you happy to invest knowing that what is being asked might not be achievable?
• If you do expect some re-use what are the non functional requirements of this prototype.
• Are you willing to fully commit to making these a success via frequent collaboration?

Feeding back on visuals

Rob Lambert and Peter Haworth-Langford stated that testers could play key roles in providing input to designs, process flows and UI.

I agree, and although some might question if a testers role is to input on design decisions at all, I think it can be essential at times.  We might make suggestions for improvement here around usability issues we’ve identified, or realise that a piece of design or in fact the overall design itself doesn’t actually provide any business benefit.

Process flows can be mapped out early from requirements if we have wire frames.  If we don’t we can communicate with the stakeholders to determine the process flow of the proposed prototype.  From assessing these flows we can identify usability concerns early, for example could the flow to point X be quicker if we combined points C and D?

UI, of course there is a lot we could feed back on early here.  We could even check if our competition is doing something similar?  What makes their paradigms special?  How do we make ours better?

Preventing rejected systems

The following two I liked a lot:

• Anders Dinsen stated “The tester understands the objectives, but questions the means to get there because he sees the problems.”
• Jamie Cuthbertson stated that a tester’s role should be to challenge assumptions, viewing things from a different perspective and making suggestions for improvement to the product.

Anders highlights the need for a tester to think of the bigger picture.  It’s a difficult job that in itself, and for a prototype we’d need to fully understand what the business expectations of the prototype are first before we can begin to see if we are on the correct path.

Jamie correctly highlights the need to challenge assumptions.  I’d like to add to this “assumptions that matter”.  Certainly for a prototype we will be looking at things from a different perspective.  I’m still unsure myself what perspective we’ll take, but for now I see myself and my colleague Stuart being focused on making sure that the end delivery satisfies business needs as opposed to individuals visions, or worse still conflicting visions.  After all it’s easy for a prototype to free-fall into something which sounds and looks pretty but delivers no business value as it doesn’t actually solve a problem, or one that’s worth solving.  Peter Haworth-Langford highlighted on this when he said that we should be making sure this is actually something someone wants.

Peter Haworth-Langford also added that a tester’s role might be to help a customer or stakeholder formulate their concept.  I think that’s a very good point.  We could collaborate with them to generate scenarios, persona’s and solid use cases around their proposals.  This in turn would help identify if their was a solid business case for the prototype after all.

Functional test involvement?

Basim Baassiri highlighted that testers could provide feedback by doing exploratory tests.  Stephen Hill said something similar when he stated that they could look for areas where the prototype might break.  I agree to an extent here, and if the prototype is being demonstrated as its being developed we could provide assistance here to avoid embarrassment during the demonstrations.  Of course exploratory testing doesn’t just stop at functional testing, from providing feedback to the requirements we will be exploratory testing as well.

Alternative prototyping

Markus Gärtner asked a series of questions to help him understand the definition of prototyping in my company.  In one question I responded that traditional functional testing would be of less importance here, to which he asked “So, how would you test a teleporter like the one from NCC-1701 D?”  To which I replied that it depends on what this prototype teleporter was required to do.  As a prototype it might not even need to teleport at all.

A famous modern example of prototyping without actually having a working model comes from none other than Apple, who had created several initial prototypes of a proposed iPod design out of foam.  Drawing the controls on the front they used these foam models as if they were actual real working mp3 players for some time.  Walking around with them and pretending to interact with the controls.  This allowed them to understand if the actual look and feel could be appealing, prior to investing in a working model.

So what is the role of a tester in a prototyping team?

To get more of an understanding of the role I’ll be playing I’ve generated a mind map below.

This image has been cropped, please select it to see it in an expanded size.

Suggested approach

So the Twitter testing community provided me with a lot of help in identifying what a testers role should be in a prototyping team.  As the above is just a collection of my thoughts on peoples responses I’d like to see if I can model an approach of some kind, and hopefully you can help me some more by providing feedback on that.

Step 1:  Clarify business expectations

We need to quickly identify what the expectations on a prototype are from the business.  Hopefully this will have been done already, but if not we’ll have to input these into the chain of command.  Some example questions here might be:

• What deliverables are you expecting during or at the end of the prototyping phase?
• Are you expecting re-useable code?
• If something is not realistically achievable in the given constrains how will you react?

Step 2:  Feedback on requirements or gather an understanding of visions.

Obviously as information gatherers we will identify and feedback on potential issues, gaps and risks with the information we have.  Most importantly we’ll be in constant communication with the key parties involved in this prototype, keeping an eye out for conflicting information.

Step 3:  Validate designs

Often a prototype can have little or no requirements.  We can assist both the developers and the stakeholders here by helping (in collaboration with the stakeholder) produce real life scenarios of how a customer would use the proposed prototype.  We might produce persona’s as well, to highlight the different kinds of user of a system, and in doing so validate our paradigms against these, and uncover potential usability issues.

Step 4:  Determine functional test involvement if any

We might be able to assist with functional testing of a prototype prior to stakeholder demonstrations.  There is a high possibility that our input here might not be required at all.

Step 5:  Ask the difficult questions early

This can be two fold.  We can assist developers by putting ourselves into a stakeholders / business mindset and thinking of possible questions they might ask of this prototype.  If the prototype has frequent progress demonstrations (and it should have) to stakeholders, we can ask the difficult questions of them also.  For example the prototype might be at risk because a clear decision has not been on a paradigm to use, we can ask why that decision has not been made and when it will be made highlighting that the project is at risk by the decision delays, and clarifying that their expectations will not be met if this progresses.

Step 6:  Track expectations

As a prototype might have one or more stakeholders, combined with other people wanting to input their ideas, we’ll need to do our best to keep track of the initial expectations, and make sure if these change that everyone is aware of it and in agreement.

Step 7: Be aware that business expectations may change

Even the best laid out plans for a prototype may become obsolete overnight if the expectations of the business change.  They might like the concept so much that they decide it should be part of the product, and ask that you begin building it in production standard code, which comes with a large test effort in itself.  Likewise the concept might be sold to a customer and require all current prototyping efforts to be dropped to work on developing this sold concept for the customer.

Thanks for reading.

Out with the old, in with the new

Today I’d like to talk about a lean approach I use for planning out test phases.

I’m a strong believer in reducing waste.  In fact, if I don’t see the value in something I’ll actively question its purpose.  If I still don’t see value based upon the response I receive I’ll look at ways to change it.  Not always of course, as the process of change might not reap worthwhile rewards; perhaps it’s just cheaper to live with it?

Now our traditional approach to test phase planning has probably been like many other companies.  It’s a task near the end which we’ve guessed upon how long it’ll take from past experience and early predictions based upon risk when we’ve first seen or indeed didn’t see the initial requirements.

Now that’s not a bad thing, and of course at that early stage we’ve not much of an idea other that our traditional regression test tasks of what will be in our test phase.  We might have an idea from some tasks, however like all plans things will likely change.

So in order to devise some sort of plan for our test phase we’d get together and discuss as a team what we should be doing and discuss risks along with some possible others tasks that we could do.

So where does lean fit in to this?  We’ll it doesn’t right?  Aspects of it might apply to an extent, but we still have waste!  It’s not very flexible!  Oh and what about the customer?   Don’t they get a say in this?

So what follows is an approach I’ve used successfully which encompasses all aspects of lean when planning your test phase.   Remember though lean is only a concept, and by the definition of lean itself by taking only the aspects we find useful, we are in essence being lean.  Likewise, hopefully you’ll take from this what you find useful and adapt it to suit your needs

Step 1: Realise that plans change

It’s true, and we all know it very well.  Even the best planned plans change!  As such why would you even consider early test phase planning?

Sure, you can get an idea of the tasks you may or may not do early.  One might even have as part of his verification checklist “Possible test phase task?”.  By that I mean identifying as we test high risk items that might be worthwhile taking note of early for inclusion into our test phase.

I personally think the whole idea of an initial plan is pointless at this stage.  Perhaps there may be value in it should you generate that collaboratively with the correct people involved.  Your true source of knowledge will be your tester who has been testing that feature right?

Step 2: Empower your testers

Since the testers have the experience and knowledge allow them to generate the initial scope.

Something I did which quickly got adopted by the rest of our team was to mind map the initial scope of possible tasks we might choose to do in our test phase.

You can see from the map below that they’ve been sectioned into types and prioritised.  The priority comes from risk.  Risk in this case being the level of test coverage this area has had already and the possible risk to the business should this task not be executed.

Please be aware that all images used in this post can be expanded larger in a pop up by clicking on them. There is quite a lot of detail in them so you’ll likely have to do this to see their contents clearly.

This was excellent for us as we didn’t have to care about our time constraints, we’d just note down what is important to be tested.  If you had a test phase that lasted two weeks or twenty it didn’t matter.  You’d note down as much as you could think of that would add value to the testing that’s already been done throughout the release and you’d also note the high risk items that are essential to be retested.

Anything that doesn’t fit into the time constraints can go.  We’ve addressed risk so we already know what is most important to us.  We’re not running the risk of not thinking of possible tasks because we are so obsessed by what will fit or not.

I’d like to think of this as a scope proposal at this stage.

Step 3: Collaborate

Now that you’ve devised a set of tasks you think should be done, involve other parties.  Get feedback initially by emailing or discussing with your developers who’ve built the feature/s.  See if anything else needs added or if suggested tasks won’t add much value.  Discuss your priorities.

In general people like reading mind maps if they’re organised nicely.  I’ve found that when used as a form of reporting they receive positive feedback as a mechanism for displaying information.  Certainly for management, they’d much rather look at the mind map and have me discuss aspects of it with them, than read a lengthy chunk of text, or even attempt to discuss something with them without a visual cue.  Generally people will find it easier to feedback on a mind map than a complex conversation or lengthy email.

Get your customer or customer representative involved.  In my case I would discuss this with a customer representative and talk them through each of the suggested tasks, seeing if they were acceptable and if anything else needed to be considered or dropped.

By talking initially with your developers and then to the customer you’ll firm up your scope of possible tasks for your test phase.  The likelihood, or at least in my experience is that the developers you’ve been working with will actually be running these tasks throughout the test phase along side your test team colleagues.  So by having those initial discussions you’re already getting their buy in to the process, which is very important if you want them to see benefit in the tasks at hand.

Myself, I’d run the tasks I’ve proposed by the developers I’d been working with and my customer representative.  I’d then involve my other test team colleagues and then finally the feature team lead.  After all that even if my scope hadn’t changed much I’d at least have more realistic task priorities from knowledge & business risks which I wouldn’t have know about.  Again, everyone would be bought in to my scope proposal as they’d had their say.

You should now have test scope for a feature.  If you’re a test manager you’ll have had each of your testers devising similar maps of possible tasks to test the features you’ve been working on.  You’ve now got an idea of what will likely be in the test phase in addition to traditional test tasks.  From simply empowering your testers you’ve not only gained their respect, you’ve also got a much more solid set of tasks that the whole development team are hopefully bought into.  Even if some are not you’re in a much better position for buy in than in the past.

Step 4: Formulate a plan

So you now have your scope.  You’ve prioritised it and know what is essential or not, so you can begin to devise a plan.

I said previously “empower your testers” I still think that’s important.  At this stage if you’re a test manager I’d be reluctant to try and take over.  Advise on time you’ll need from people and allow you’re testers to plan their feature teams tasks.

Collaborate on estimations for these tasks.  Find out from your team how long they think these tasks will take.  This could be as simple as a quick thirty minute meeting.  You could even take a guess at estimating them yourself and then asking for feedback.

Begin planning tasks in.  My advice here would be to get an initial quick sweep of the functionality and address high risk items.  This will be your initial test cycle from which you can say by X date we will have covered most of the functionality; this is essentially our critical plan which we don’t want to drop tasks from.  These critical tasks can overrun, or be impacted by sudden holidays or sickness, but we know that at most other less riskier tasks will have to drop.  Hopefully you’ll have planned in some contingency to negate this risk anyway.

Once you’ve done you’re initial sweep it’s probably best to plan in a second cycle of testing which might just involve a one day sanity of the features by everyone in your team or just some of your team, depending on the risk & complexity of the feature.

Less risky tasks can now begin to be added into the later stages of the plan.

I’ve provided an example of the initial plan our team worked from when testing our text chat feature during our currently on going test phase in the picture below.  This plan did go on to change slightly as some tasks had overrun, and when sickness or holidays occurred.  However it didn’t drastically change, and at most some automation and low risk tasks were dropped.

We still have a little time left until our test phase is actually complete, but at this late stage things will likely not change much.

You can see from the plan that our initial test blast provided us enough coverage and feedback by the eleventh of January.  This, if you take into account the Christmas holidays and reduced workforce over that period was about seven days into a planned four week test phase.  By that point we’d known what our real actual high risks were.

A few days later followed a second cycle of testing followed by lower risk test tasks, such as extensibility proving tasks.  Of course I’m not saying that extensibility isn’t risky, for us it was less a risk as we’d raised awareness of this and planned in development extensibility proving tasks throughout the release.

It’s probably worth mentioning the importance of planning in time for continual bug fixing throughout your test phase.  Defect predictions are often highly inaccurate, but at least allow for allocation of time for bug fixing.  That’s why doing an initial scope is good as you don’t have to worry about cycles of testing, bug fixing, automation request from management and so on.  You can just devise what you think is important and plan in the important stuff that can fit into your time constraints.

Step 5: Communicate some more

It’s important you keep track of your plan and adjust it accordingly, that’s obvious.  I think it’s also important that you continue to communicate with those involved throughout to see if things are still adding value.

An example I did was simply sitting the participants down after four days had passed at the start of our test phase, and discussed aspects of the plan with them.  From this the group quickly communicated their thoughts and problems allowing me to make some changes to estimates and help resolve the intentions of any tasks that were unclear to them.

Remember it’s easy for things to go astray, often people will just continue as-is regardless, should it be a task that will overrun or simply something someone doesn’t see the benefit in doing.  That’s why it’s so important to keep constant communication going so you know then that there is a problem, rather than when it’s too late.

Summary

To wrap this up I’ll quickly summarise lean test planning:

• Realise that plans change:
  • Plan as late as possible.
  • Reduce waste.
• Empower your testers:
  • Having worked with your feature teams they’re silo’s of knowledge.
    • Let them propose tasks.
    • Let them plan their feature teams tasks.
    • Let them involve the customer.
• Mind mapping scope:
  • Increases creativity.
  • Reduces time to create scope.
  • Increases visibility of the bigger picture.
  • Identifies risk easily.
  • Doubles as an easy reporting method for feedback.
• Initial scope proposals ignore time constraints:
  • We lessen the risk of forgetting tasks due to thinking about time constraints.
• Collaborate:
  • Involve the customer.
  • Involve all other essential parties.
  • From direct collaboration, we iron out actual risk.
• Lean planning:
  • We’ve identified scope.
  • We’ve identified risk.
  • We’ve got buy in.
  • We’ve collaborated with essential parties.
  • Now actually formulating a plan is simple.
    • It will provide high coverage and feedback quickly.
    • Being flexible it will allow for change.
      • Lower risk tasks can drop if needed.
• Keep communicating:
  • Don’t ignore the value of communication.
  • Empower the customer to decide what should be de-scoped from an overrun plan.

On that note we’ve come to the end of this discussion; admittedly one way so far being just my thoughts & experiences.  Hopefully though I’ll have some feedback from the community be that positive or negative, any is always welcome

Thanks once again for taking the time out to read my thoughts and experience reports.

I’m planning a couple of follow up posts in relation to this.  One on the usability evaluation we did.  The other around the actual tasks I’d planned in as I think potentially there might be a few valuable insights from discussing them that others could benefit from.