Tales from the trenches: Essential test case design

Summary

Test Case Design is a rather large topic isn’t it?  Well I’ve no intentions of covering what others have already discussed in great detail in articles, courses and entire books on the subject.  Instead I’d like to talk about some cheap, effective techniques and approaches that you can use today, in your workplace, to improve your testing.

An often confused practice

Everyone has been singing the praises of exploratory testing for some time now.  I often read about people arguing over scripted vs exploratory testing, as if they are to distinct approaches that could never overlap.  Sometimes I see educated people on Twitter and various testing discussion forums having the same debate, scoffing at the thought of ever writing a scripted test case, as if it was some monkey job that was below them.

Now, for those informed, or perhaps, those who have put some thought into it, might actually realise that when they’re in the process of writing heavily detailed, scripted test cases, they are in-fact doing a form of exploratory testing.  Some time back I wrote about my method for writing lean test cases, and talked about how I’d used a model of testing types, combined with the power of mind mapping to write heavily detailed, conditional, scripted test conditions.  As I was using my brain and thinking of conditions in which I could find flaws with the system under test, I was in fact doing a form of exploratory testing.

Now test case design, if done in the correct way, can be a very powerful technique for deciding upon the, what, where, when, how and who of testing.  You might not think that you’d get all of this from a traditional scripted test case, but you can, and should.  Over the next few post’s I’ll delve into techniques with mind mapping, and indeed entire project planning with mind maps which will further explore this concept.  For now though I’ll jump onto the cheap techniques and approaches that I’d promised.

When to write test cases

Most often than not, I’ve found test case design to be most useful when I don’t actually have a system to test.  Why?  Well if you think about it, when we do have a system to test we are often far to busy testing it to stop and consider writing test cases around it.  It’s often easier to jump right in and get your hands dirty testing the application; after all it will provide you with invaluable quick feedback.  That being said, sometimes you can reap huge rewards to stop, and think about the types of testing and conditions to apply.

So when does it make sense to write test cases, as opposed to testing?  Well, sometimes you think you know all the aspects to test, and quickly jump in and begin testing.  Sometimes you are so time constrained that you really have no other option.  However it does pay to think over what you actually want to test, and how.

Most often people write a test plan document which, amongst other things, would allow them to consider aspects of testing that they might not have previously thought about.  The problem I see with such documents is that, whilst some of them provide some visibility to other interested parties such as project managers, and perhaps customers, most often than not they become quite lengthy, and once written never, or very seldomly get refereed to again by the author.  Often plans change and documents remain as-is, quickly becoming outdated and set aside on the document scrap heap that no one wants to acknowledge.

Taking that information out of a document and into an environment where people will take notice of it is one possible solution; where is such a place though?  Well, lets look at some possible high visibility, high impacts test techniques that might solve this problem.

Story bound acceptance tests

Acceptance tests are an easy candidates for such a move, and can exist in many meaningful places.  If you work in an agile environment you’ll likely use story cards; these are placed upon your story board or task management application such as Jira.  A really cheap technique that will allow you to place tests into the field of view of developers and those that matter, is to write acceptance tests for each story, either on the back of the story card if you are using a whiteboard, or along with the content of the story on your task management application.  If you want to improve the process further, have the customer write these acceptance tests in conjunction with you, the tester.  It seems so obvious, but this little gem is often forgotten about, or simply ignored.  This can be so beneficial to team, and often uncovers key gaps, and risks in deliverables.

Now you have a pros and cons with this technique.  The benefits being obviously the direct involvement with the customer, and placing test directly into the acceptance criteria for delivery of a story.  The downside to this, is that you have this linear view of the world.  Often you never see the big picture, or if you like, the holistic view of the feature when combined with the product.  As such your tests often reflect this, and are constrained to that story, or sometimes with some minor interactions with different stories, within the same feature.  As long as you are aware of the limitations of this technique and counter for it, this can work wonders.

So obviously writing, or linking acceptance cards to user stories provides benefits, but we know it has its limitations; specifically in terms of providing a holistic view of the system.  We know however that it does make sense to use this technique, and you could argue that if the tester, in conjunction with the customer can’t write these acceptance tests, then the customer really doesn’t know what they’re asking for.  Therefore it’s probably a good thing that we had got that initial communication going.   This technique is also very much so, a form of proactive testing, which not only reduces potential defects; it also promotes higher quality software throughout the team.  So certainly in this case it makes perfect sense to write test cases, as opposed to testing the system.

Mind mapping acceptance tests

Getting back to the the subject of writing test plans to provide you with that early big picture overview, writing acceptance tests in the form of a mind map can certainly replace most aspects of this traditional, write once, never read again document.  “How so?”  I hear you ask.  Well, if done correctly, mind maps of proposed features, or indeed entire products can provide you with a holistic view of the system.  With that big picture overview, you can more easily see the what, where, when, how and who of testing.  I’ll provide some real life examples in later post’s of using mind maps like this.

I normally begin by building my map up with an overview of the proposed feature.  If the feature has not been implemented yet, then I’ll use existing documentation such as requirements, specifications and designs to help me determine what it consists of.  I’ll also lean heavily on customers, and key stakeholders to make sure expectations are aligned, taking notes of any initial risks, and highlighting gaps in the proposed feature.  Since I already have a feature overview created, or in development in the form of a mind map, it makes the process of identifying gaps when communicating with stakeholders, that much easier.

If the feature has already been developed, I can use the previously discussed documentation to work out if what has been delivered met initial expectations, and if not I can begin looking into the why.  Either way, at the end I’ll have mapped out what this proposed feature consists of, and preferably how the components interact with each other and the system as a whole.  I can then use this as a means of developing acceptance tests, since now I have that big picture overview of the scope, interaction points, and external dependencies of that feature.

If you read my post on lean test case design, you’ll see in much more detail how powerful this technique can be.  An important aspect to take away from this is the use of a testing model to use as a reference for developing test cases against.  Your model will obviously be built to fit your products needs, but obvious candidates for inclusion would be types of testing you might carry out, such as performance, extensibility, usability and so on.  Likewise, lightweight persona’s can expose aspects of testing you might not have previously considered.  For example, where in your system would a hacker likely target?  How would an programmer external to your company use your public API’s?  Do they fulfill his needs?  How would a user interact with this feature?  Are they jumping through hoops trying to use your system?  Another useful element to include in your model is rules.  Rules simply being, expectations of the system.  These a direct expectations from the stakeholder, and sometimes can be lifted directly from requirements documentation.

Now I’ve obviously not went into much detail here on how to build your map, but you can find more about that in my lean test case design article.  I will however, at some point in the near future expand upon on mind map design techniques in much more detail.

So we now have two cheap, effective techniques to produce acceptance tests.  Both hopefully could be used in a proactive sense, to further improve their benefits.

Pair test case design

Now I’m a big fan of collaboration when it comes to test case design.  One obvious candidate which would blend well to pairing with another, or indeed a group, is mind mapping.  I’ve talked in the past about the benefits mind maps provide for test case design; what I haven’t touched on much is pair mind mapping.  It sounds very simple, but there are techniques that can make this experience much more powerful.

In my experience pair mind mapping can provide fantastic results for test case design.  If you do intend to pair I’d recommend using a tool such as Mindmeister for this, as opposed to paper and pens, or a whiteboard.  The reason I suggest using a tool as opposed to directly pairing, is that most often than not someone takes charge, and begins to dictate the mapping experience.  It is then often difficult for the other pair to slip out of a linear thinking mindset, to provide worthwhile contributions.  Separating the pairs, and forcing them to use a tool as a means of communication results in both participants opening up their creative side, and as a side effect providing much better results.

Both testers could let themselves loose and begin mapping test ideas onto the map, however what I’ve found works well is to assign the tester with the most experience in mind mapping a lead role.  The lead works his way around the map, jotting down initial test ideas, while the other tester shadows his movements filling in gaps and contributing new ideas to the mapped areas.  The lead tester should frequently revisit previous nodes, taking note of changes added by the other tester, and by doing so this will spark new fresh ideas, producing much better results.  Mindmeister is fantastic for this approach, as it’s very evident when changes are made by another party on the map, in real time.

Group test case design

A tool like Mindmeister also works well for group contributions.  For example, you may design tests for a new feature, and then share it with other members of your team to review and expand upon.  Each member can then review and expand your test ideas at their own leisure.  When changes are made to the original map you are notified by email, and can then log in and review, step by step, all changes made.

Now obviously if you have a group to work with, using a tool to design test cases isn’t going to work if you’re all trying to interact with the map at the same time.  Re-locating the group to a quite room, with a whiteboard and some pens gives you all a much better chance of producing good results.

Much like a pair mind mapping session, it works well to assign someone with good experience in mind mapping a lead role.  Rather though than them mapping out nodes themselves initially on the whiteboard, you would have them steer discussions and encourage participation from other people in the group.  They will also be responsible for mapping out discussions on the whiteboard, but other participants should also be encouraged to contribute to the map when they feel their ideas are not expressed well verbally.

Lean heavily upon lightweight persona’s.  If you have a second whiteboard available, or some paper which can be shared with others, start the exercise by asking the group to think about which type of people would interact with the feature in some form, at some point.  Take note of these users and have the group refer to them throughout the mapping exercise to generate new ideas.  Question, how would this persona interact with this feature area, often.

If possible, get key stakeholders involved.  Customer and stakeholder input can be invaluable, and often exposes gaps, or risks with design.

Group test case design with mind maps can provide really detailed test cases, more so, they often expose much larger tasks which will require adjustment to initial plans.  Often you’ll come away with a long list of follow up items which need clarification, that’s why having customers and key stakeholders involved in these activities to clarify issues is worthwhile.

Input / Output mapping

A golden gem, which I’ve used to unearth many a major defect, and countless lower severity issues, is the mapping of inputs and outputs.

So what is it?  Well it is exactly as it sounds.  For every input, more often than not we’d expect and output.  Likewise, for every output we’d expect there to be an input in most cases.  Developers often see just that, an input for an output.  Their tests often reflect this, and error handling code, likewise is the same.  An input for an output, and an output for an input.

Of course we can fully expect that all inputs or outputs can have one or many of the opposite which directly affect them.  As a developer when working on a block of code for a specific area, the holistic view of the product, or indeed the feature is often lost.  It’s hard to see the whole picture when you are so focused on a single task at hand.  Therefore their code and tests often miss out other possible areas, which might affect that piece of code they’ve just written.

Previously when I discussed mind mapping acceptance tests, I talked about how this could provide you with a holistic view of the system and feature under test.   We can use this big picture overview to our advantage, to begin listing possible inputs into the system.  We can then make a second list of possible outputs from the system.  If we then have time, we can begin linking multiple inputs to our outputs and vice-versa.  If we don’t have the time, we can simply pick out what we feel are they key high risk area’s and work on those.

By noting multiple alternative inputs or outputs for a feature area, we can identify areas which most certainly would have low or non existent test coverage by the developer.

Don’t consider only what you can see; configuration files, administrative settings, database configuration tables, operating systems, browsers and so on can all provide a means of affecting possible outputs.  Some of these such as that hidden away configuration file, or administrative screen to adjust settings can all be a rich source of potential issues.

More often than not, consistency can be an issue with outputs.  The same information presented in several area’s when modified, can often result in inconsistent, or non updated outputs.

I’d highly recommend trying this out.  Sure, we all do it to some extent when testing, but set aside some time to sit down and map these test conditions out.  You’ll be surprised with the results.

API Test Cases

I’ll finish off this rather long post by talking about API test case design.  Rather than jumping in and coding API tests, I’ll suggest an alternative approach that can be done either by yourself, or in a pairing task with a developer.

API tests blend very well to discussions on test case design, or indeed a thought process to design test conditions.  Indeed, we have many automated documentation tools which can generate documentation directly from the source code.  Doxygen is an example of one such tool used in my work place, which with a little effort from the developer can produce pretty decent documentation of code.  Even without an automated documentation tool, the explanation of an API is a rather simple task, and one that can be done quickly.  Drafting up some quick documentation does not take much time either.   Likewise if you’d like to explain tests around code, commenting around them in Doxygen format, or a similar documentation tool would result in a list of tests that provide code coverage, which make sense, to almost anyone.

Sometimes if you’re lucky a team may attempt to define API’s up front as part of a technical requirements scoping exercise.  In such cases it would be foolish to not assist with test case design upon the API.  You have a real chance here to ensure that good tests are being written.  In the past I’ve reviewed API’s and suggested test cases to improve coverage, only to find that whilst they would be beneficial, the resources to do the task simply didn’t exist at that point in time

What I find works best is to produce a series of one line descriptive tests.  For example, if I provide a get function a specific parameter, I can expect a specific return type.  If you run over the API initially and map out basic acceptance tests, covering all parameters and returns types we can assume that we have decent coverage of that API.  However, there may be dependencies, or specific functions which need initialised first.  Obviously we can come up with lots of useful scenarios to test here, and negative tests to give improved coverage, and hopefully expose some issues.

If you’re pairing with a developer to do these test conditions, great!  If not, hopefully you’ll have a nice list of one liner tests to provide them with to ensure decent coverage.  This is particularly important I feel with public facing API’s, which might be exposed to other applications or 3rd party developers.  Indeed in such a case, such API’s not only need to be robust, they also need to be usable and fit for purpose.  In such a case strong readable documentation is essential, along with validation that these public facing API’s fulfil users needs.  One method to ensure the later works, is to think of application scenarios that might make use of these API’s, and quickly mock up these applications to expose limitations and fundamental issues.

Sometimes it’s also worthwhile writing a test harness against the API, which can then be used by anyone with little, or no training.  These can often be written very quickly, to allow quick testing.  Often, resulting in scenarios executed by the tester that you might not have considered yourself.

Conclusions