Why No Test Cases

Why No Test Cases

The test team I lead do not use test cases.  We believe that testing is an active,  changing exercise and that rigid test cases does not support this rate of change.  I will give my arguments for not using test cases before explaining how we do plan and track our testing

What is a test case?

A test case defines a process that will show if the software you are testing displays a particular desirable attribute. It consists of 3 main parts:

1. Starting condition
2. Method
3. End condition

And may have other parameters

1. Priority
2. Owner
3. Status (open, in progress, blocked, closed)
4. Weight

A low level test plan is a collection of these test cases for a particular module or capability of the software being tested.  The test plan is reviewed and approved by stakeholders rather than an individual review of each test case.

Reasons for change

Not all test cases are created equal

To paraphrase George Orwell "all test cases are created equally, but some test cases are more equal than others".   Each test case differs from others in terms of it’s:

• Risk
• Complexity
• Duration

Tests when executed reduce the risk of faulty software being shipped to the customer.  From the customers perspective there are different levels of risk, some acceptable and some not.  The risk of a small input validation error within a utility program might be just annoying but other errors could cause a loss of money within a business or even loss of life for aviation or medical software.  Different tests mitigate different amounts of risk.

The complexity of a test case can also vary.  Modern software is rarely executed on it’s own and will likely have many integration points.  Configuring, executing and monitoring the software during the test can be a complex task.  Some simpler test cases though are a lot less complex.

Tests not only take time to execute but also can take considerable time to prepare the required test configurations and test harness to execute the test.  As well as time required by the tester to complete these tasks, longevity or workload tests can then take an extended period of elapsed time to execute the test. Validation of the test execution data once the test has executed will also add to the duration of the test case.

Some tests will have a natural ordering between them.  Quite often a test will build on top of previously run tests.  These tests will require the tests run earlier to pass in order for this test to be run.  

The above points show how varied and different a set of tests can be.  Defining these varied and many tests into a single unified ‘test case’ hides the differences between them.  Once in this form they are counted as if they were all identical in duration, risk and complexity. Boiling this complexity into a single test case template just hurts the testing.

Preparatory work

Any test case will require some degree of preparatory work before the 'actual test' can get underway.  This might be configuring automation, defining workloads or developing test harness to run the test.  Is this work part of the testing process or just necessary delay that has to be done before the real work starts.  Fellow testers seem to fall into one of two main schools of thought.  One group see prep work as a required evil, work that must be done but isn't part of the actual testing.  The other group are of the thinking that as soon as you start to interact with the new parts of the software, the process of testing has begun.  Either way how do you classify this work?  It doesn't fit into the test case template at all and yet the test cases are dependant on it being done.  When a team has 10 test cases remaining you don't know what work needs to be done to execute those tests?  They might need 3 days of prep work to be done first.

Once the prep work is completed you are left with a set of test cases and a period of time to complete them in.  This turns the test case into a metric and ...

Someone always wants to track them

There are x number of test cases left to do and y amount of time left to do it.  Simplistically this is modelled by a straight line graph.  The model assumes that each test can be executed in the same amount of time.

However as we have said before not all test cases are equal, so to assume that we can execute through them in a linear fashion is short sighted.  Also as test work progresses defects are raised which will require recreation and verification and this takes time, time that cannot then be spent on executing tests.  Responding to this the chart is often re-drawn as an s-curve like this one:

Progress is charted against the graph and teams either are congratulated or receive 'management focus' depending on the state of the chart.  This misses the point and it's largely the fault of the test cases.  Progress through a list of predefined tests means nothing.  For the progress to be meaningful you need to assume that the quality of the tests are high and that the tests mitigate the majority of the risk for the customer.  Even if you can make that assumption, progress through the test cases isn't enough to judge progress you also need to consider:

• How many defects testers are finding.
• The spread of defects across the product
• What tests have actually been done yet (have we just done the easy ones)
• what if we have completed 90% of the test cases and found no defects?

Without this additional information the complexity of testing is abandoned in favour of an overly simplified metric which is too narrow to be of any real use.

They don't react well to change

Testing is an exploratory sport, an experienced tester working through any set of test cases will question their approach based upon; progress through the test case, prior defects and often just a gut feeling.  Regardless of how much thought and effort is applied to the formulation of the test plan, the planned testing and the actual testing will be different.  This is because the tester is learning about the software and how it was implemented as they execute the tests.  A single test case might expose a bug which is part of a bug cluster within a piece of code.  To fully examine the area new test cases will be created and executed on the fly. This ad-hoc, off piste or exploratory testing should result in further test cases being written and added to the plan.  Adding extra tests often infuriates managers trying to track progress.  The new tests are driving further quality into the product and their value isn't in the testcase being written but in it being executed and the defects that they find.  

Test approaches such as scenario based, use-case or exploratory do not lend themselves to a formal test case report.  They often do not have a particular method or starting condition but instead tend to rely on experienced confident testers examining the product in the same way the intended customer would.  These approaches are better tracked using a time box or complexity rating like duration or story points.  

Test cases seem like a good idea.  However in reality they are too simple a model to use in practice.  A single test case can define the intent of a test but the model breaks when used to suggest progress or the quality of the product.

I explained this to a junior member of my team and drew parallels between some exploratory testing he wanted to achieve and a prototyping story that development were planning.  Both didn't have a clear plan of implementation but they did have allotted time and a focus on learning how to achieve their aim by 'working with the software'.

This was when an alternative became clear.  This is just software engineering.  Can't we measure software development and texecute.work in the same way?  Would that makes things clearer

Whats the alternative

Test work is just engineering work

If you stop thinking about testing as being the execution of test cases and instead as another type of software engineering work then an alternative approach becomes obvious.  Testers still write code, examine specifications and solve problems in the same way as a developer.  We don't track developers by the number of changesets they produce so why not treat testers in the same way and use the same tracking artefacts that are used in development.

We now use test stories for our system. Test work.  No test cases in sight.

A Test Story is similar to a development story.  It outlines the objective of the work, priority, owner and complexity.  Except in a test story the objective could be more deconstructive than that of development story.  As a dev story is constructing something new, test are de-constructing it to reveal defects.

Test Stories can be planned for an iteration and split into multiple child tasks that describe each individual unit of work.  The test story is given an estimate of the time it might take, a priority of how important it is that this test is executed and a story point rating to describe how complex this test is to implement and execute

Handling defects

Defects found during the story executuon are linked to the work item to show the 'fruit' of that story.  Development and test stories can be associated to show the testing that will be applied to new capability

Defects that are found will often require recreating to provide further diagnostics and require verification once a fix is supplied. This work will need to be planned and committed to an iteration just like any other piece of work.  We have created queries to total the amount of time spent 'working' these defects.  This can help to explain delays in test progress as testers are working on defects.

Showing progress

We can now show our progress by explaining the work we have committed in an iteration and our burn down throughout the iteration.  Not using the simple metric of testcases focuses people to ask more interesting questions to determine the quality of the project:

• What types of testing have we done?
• What defects have we found?
• How much time do we need to complete the high priority work

All more valid and useful than a single percentage of test case completion.

We have been using test stories as a replacement for test cases in the system test team for a few releases.  Using them has given us the following benefits:

• The total amount of  test work that needs to be done including  prep, defect recreation and verification as well as actual testing.  We can plan our iterations a lot better.
• Tracking is a lot easier as we can understand the duration as well as the complexity of the work rather than simple counting measures
• We are starting to ask better questions about the state of the test project.

We still are not perfect.  We are still refining our process to ensure we estimate tasks better and ensure we don't fall into the trap of just counting test stories.  Overall we have seen an improvement in out efficiency.  Who needs test cases?  Not us!

Taking back the web

Back in the good old days when your modem only supported 28.8kbps and accessing the local BBS didn't allow you to 'like' posts, there weren't the rich services that we no benefit from such as free mail, cloud storage, hosting etc.  If you wanted any of the above you had to build a server yourself.  These servers were big grey boxes with large monitors.

Now in 2014 I have decided to build a server myself and use it to host 'something'.  This is itself isn't a real technical challenge.  Most modern linux distributions allow the user to install a preconfigured LAMP stack with a single command.  Neither is building a web site, thats just a bunch of HTML, CSS and other scripting languages.  I however want to do this for NO MONEY!  I will only use hardware and services that I have either had donated or have free options.  The point of this is that to prove that anyone with a bit of knowledge can 'take back the web'  by building the infrastructure and hooking it up to the rest of the internet. This post consists of 2 parts:

1. The web server
2. Domains / DNS and port forwarding.

At the end I will have a basic HTTP server running on some hardware and available on the internet.  Future posts will discuss the services I will build on this stack.

The web server

I like my smartphones, for the past 2 years I have had a Samsung Galaxy Nexus.  I love this device.  It was my email, calendar, music, communication device and it went everywhere with me.  Over Christmas I upgraded to a Nexus 5 and for the first time felt a pang of pain as I removed the SIM from my old device and turned it off for the last time.  I have never felt like that before with some electronics.  Deciding to turn it into a web server somehow eased the pain and is a nice example of upcycling a device.  In some ways it is a great choice:

• Silent running
• low power consumption
• Small physical footprint

It's not going to be able to serve the next facebook or wikipedia. but to host some personal stuff it works absolutely fine.  If I ever require something with a bit more capacity then I can always find some other devices and use them as database servers or content servers to lighten the load.

I am using a web server called kWS which is free in the play store.  it is just a basic HTTP server with pro features in the paid version.  Right now it fits my requirements as in it serves basic HTML and is free.  Later on I might need to change to a different server.

Because I have not rooted my android device I am not allowed to bind to any port running <= 1024 so therefore I am using port 8080.

Domains / DNS and port forwarding

I don't have my own domain and buying one would be outside of the rules so I have used the free service of:

www.noip.com which provide a free dynamic DNS service.  They host the domain and I am allowed to update the associated IP address using a small client running either on my ADSL modem router or on my web server.  

My router (provided by sky) only supports dynamic dns updating if you are using the dyndns product.  However this is no longer free.  I am using a small client that sits on my android web server and uses the no-ip.com api to update the DNS record.  I am not expecting the external DNS to change much, but if it does I am covered.

Finally I just needed to reserve an IP address for the MAC address of my phone ensuring that my new web server always gets allocated the same IP address.  One small hole in the firewall allowing inbound traffic on port 8080 to go to the IP address on my phone and I was all ready.

The site http://hobbit1983.no-ip.biz:8080/ is now live and running from my smartphone!

Connecting a kindle to a linux machine

I have a 50mb pdf that I want to read on my kindle.  I cannot use the email to whispersync bridge that amazon provide as the 50mb attachment is too large.

I tried to connect the kindle to my linux laptop and apart from the kindle charging there was no recognition in KDE that I had plugged a device in.  A quick google later discovered that the kindle uses the MTP protocol rather than Mass Storage Device Class protocol like most other USB devices.  Installing MTP support is easy:

sudo apt-get install mtpfs

Plugging in the kindle now gets KDE all excited and allows me to view the content.

Except after unmounting the kindle I couldn't see the PDF on the menu screen.  I re-attached my kindle to the computer and could see the file still there, so why didn't the kindle like it.  The file I copied was called:

10_PRINT_121114.pdf

After renaming it to:

Print Code.pdf

It started to show on the kindle screen.  All I can think of is that the kindle doesn't like files that either start with a numeric or have a '_' character in their names.  After some more renaming I can conclude:
The kindle doesn't like underscore characters in file names.  I ended up just changing the "_" to a " " and everything works fine now.

For those interested the book I am copying to the kindle is called:

10 PRINT CHR$(205.5+RND(1)); : GOTO 10

From 10print.org.  This book tells the story of computer code in society and culture through the lens of a single line of C64 basic code.

Retro Geek

I am the luckiest man alive.  Not only does my wife acknowledge that I am a geek and have an unhealthy obsession with computers but she feeds this geek behaviour with some outstanding presents. However she has outdone herself this time as she has bought me a commodore 64.

It was meant to be a Christmas present for me.  However the guy that sent it didn't wrap it and as I answered the door when it arrived the cat prematurely escaped from the bag.  The commodore 64 was the first computer that I owned.  We did used to have a dragon as well, however that was more my dad's computer.  The commodore was just mine.  I learnt a lot about computers from that machine, from how a light gun worked through to how data was stored and some very basic programming.  That machine will always be special to me.  To have one now at the age of 30 is incredible.

I took it to the local maplins store, where a huddle of geeks clustered around me and the commodore, admiring it's good condition and asking to have their photo taken next to it.  The store manager even walked out of the store as I entered asking if the box truly contained that which was advertised on it.  We needed to go to maplins to purchase a RF cable to connect the Commodore to my LCD TV.  Naturally this being maplins they knew exactly what I needed and passed it to me.  We took it home and started the unboxing.

Like I said the box was not wrapped when it was sent so it was COVERED in parcel tape.  This is annoying.  The original box is part of the vintage retro feel.  5 minutes of painstaking picking and pulling and the unboxing could commence.

Once unboxed and all connected up the fun could start.  I found that my TV needed to be tuned to 590Mhz to pick up the output from the computer.

Amy graciously allowed me to then spend the whole evening playing vintage games.  Working out that a light gun will not work on a LCD screen (It uses cathode ray gun timings, which of course, are not available on a LCD screen), getting beaten by her on Mike Reids pop quiz etc etc and writing my first program (again)

Thank you to Amy for allowing me to geek out for the evening and re-live my childhood again.  It is a wonderful gift that I will never forget.

Quis custodiet ipsos custodes?

"Quis custodiet ipsos custodes?" or "Who will guard the guardians?" or in my case as a software tester - who tests the testers?

As a system tester I am the final link between software being 'in development' and it being live for real users to use. We are often seen as the guardians of the users, ensuring that before it is made generally available the software is as bug free as we can possibly make it.

But who's job is it to test the system testers? who are we culpable to? or to paraphrase who "invites me for tea, biccies and a little chat" when things go wrong?

During my day job at IBM I provide the development organisation with High level plans of what testing we plan to inflict upon their software. This plan is reviewed by both them and my test colleagues. During test execution we report how well we are doing against the plan.

So our intentions are reviewed and approved by the development stakeholders. However testing is not a exhaustive activity. We can never reach 100% completion and state that the product is bug free!

The people that get hit are the customers of the product. Using some configuration, use case, or pattern that we didn't think of first, the customer found a bug. Personally I hate it when a bug is found in a product that I was testing. I will ask myself; why didn't I find that, why didn't I try that configuration etc.

Apart from the moral pain it may cause the tester, for the customer it may mean delays to their work schedule as they workaround or wait for a fix to the problem. But there is another organisation that will also get impacted.

Most (if not all) software houses will have a support organisation whose job it is to fix bugs that customers have found in the product. They bear the brunt of a customer complaint when the software does not behave as expected. This group of people more than test or development understand what customers are doing when the software fails. They understand the usage patterns that a customer is using when they notice a bug in the software.

It is the service team that 'pay the price' along with the customer when things go wrong. Perhaps it is this position that means that they should be allowed to guard the guardians and have close scrutiny of test plans to ensure that the tester has thought of common usage patterns that a customer may use.

I'm not suggesting that this approach will mean we hit 100% bug free software but perhaps it will mean that customers will not hit bugs when they are doing something that 'is normal for them' and that has to be a good thing.

When software doesn't just work but delight

As users the software we use is often just 'OK', I mean it does what we ask of it, doesn't break too often and helps us to complete a task. This unfortunately is the norm for most software.

So what can change what can we expect software to do that is better than the above? I think software should delight users not just behave as expected. I know of no piece of software that delights in all of its functions. However occasionally a piece of software does delight me.

What do I mean by being delighted by software. I mean the software almost second guessing my intentions and then making intelligent choices about what options are available to me and changing settings to support what I am doing. Or it can mean software that streamlines a set of tasks required to complete a goal without much intervention by me. Software that has delighted me makes me stop what I am doing and smile back at the computer.

When I say that software should second guess my intentions I add a caveat that the this 'guessing' should be intelligent and treat me with respect. Making non-intelligent guesses about my intentions only makes using the software VERY difficult and annoying. A perfect example is the paperclip from MS office. It tried to guess what I was doing and gave me a list of tasks that I might want to use. The problem was that it often incorrectly guessed my intentions and thus annoyed me with a list of tasks that I DIDN'T want to perform.

However today I was impressed by software so will sing it's praises here. Today my eclipse environment decided to remove all of my packages from the package explorer view. The files still existed in the workspace but I couldn't see them through eclipse, thus I couldn't work. eclipse fail!

I did manage to re-import the projects back into the workspace which added them back to the package explorer view. However none of the projects had been connected back into Rational Team Concert. This meant that I wouldn't receive any changes from my colleagues and any changes I made would not be reflected back in the repository.

I tried to 're-share' a project back into the repository and this was when RTC delighted me. It prompted me that the project was already shared in the repository and that the best thing to do would be to re-link it to the repository. Excellent that was what I wanted to do. RTC then prompted to say that it had found 50 other projects that were in a similar disconnected state and that it could fix these projects at the same time. How amazing is that. I felt that the software had gone out of it's way to actually work out what I was trying to do and then search for other similar options that I might need to do as well.

I am a massive fan of Rational Team Concert but today's incident has made me love it even more as it has now made that rare step and started to delight me rather than just meet my expectations. Well done IBM rational jazz team.

Wishing I had ...

My wife and I have been doing a lot of exercise to try and lose weight recently. As part of this we record how much weight we have lost everyweek and how much exercise we have done.

Now the key to losing weight is to simply use more energy than you eat, simple surely?

Working out the energy in value is easy, just use food wrappers and the information is there. However calculating the energy used is a lot harder. Gym equipment can help but isn't completly accurate, also what about energy burned when not connected to a treadmill? Just writing a blog - that must use energy!

So I wish that the human body had a data access port (USB would be my preference) to allow me to download how much energy I have used and such like. This would also make a docotrs job a lot easier as they would never need to run any diagnostic tests, just connect you to their laptop and get a complete view of your health.

We could even write software to interpret the data, similar to the jvm health checker. I think this could be a really good idea.

Alas since we don't have such a wonderful port I guess that I will just have to go and get weighed again tomorrow and hope that I am a little thinner and lighter than I was a week ago
Follow me: http://twitter.com/hobbit1983
Blog: http://outspokenhobbit.blogspot.com/

Function Vs System Tests

I'm often asked "what is the difference between a functional and a system test - is a system test just a larger more complex functional test?"

A functional test (FVT) test proves that a particular isolated function of a system works as designed. The function is given a particular input and the response from the system is checked against the 'known' good response. These tests will test what happens when incorrect data is used as input and input data that is on the edge of 'good' and 'bad'. I suppose the key phrase is that an FV test is quantitative. The input and ouput data can be described mainly because the function under test has been isolated from the rest of the system and is simple enough to be tested in this way.

System Testing (SVT) is often seen as being 'bigger better FV testing'. People often see SVT like this because a lot of effort has gone into developing the FV tests and running them again but harder and faster seems like an efficient use of an existing resource. However such a concept misses the real point. Customers do not use the functions of a system in isolation. You can see this in domestic application as well as large scale enterprise software. Take a word processor for example. A user does not use the save function in isolation. They use it alongside a document that; contains pictures, is in a different format, is actually stored on a network drive etc. thus just testing the function in isolation is not what your customer will do.

The real aim of SVT is to test the System as a whole in the way a customer will use it. This means using all useful combinations of the functions of a system to perform a particular task. When doing this we are looking to see if the system behaves in a 'reasonable' way. We can't use a specific input and expect a specific output as in a good system test there are just too many different options and circumstances for this to be a efficient method.

An Explosion of bytes

I haven't updated this for a while but since my wife has started blogging over on http://weightlosslou.blogspot.com/ I thought I would update here if I ever felt like I had something worthwhile to say.

I'm lunching at my desk at work and started the weekly download of podcasts I listen to:

Chris Moyles podcast (http://www.bbc.co.uk/podcasts/series/moyles)- I'm a big fan of his, not always the music that he plays but the banter he has with the studio team and with the nation. Thus the podcast is always a good listen (all chat no music)

The God Journey (http://thegodjourney.com/) Although I am a Christian I have reserved feelings for the way 'church' is managed and run. This is another blog topic for the future I feel. However Wayne and Brad often express the same feelings that I have had in the past

The Friday night comedy podcast. Currently this is the now show which is a VERY funny show.

Anyway - why an explosion of bytes? well I have downloaded the above 3 podcasts onto my Blackberry device to play on my drive home from work during the week. In total 60meg of data.!!!

Now I am not a ludite by any means but I do remeber when 60meg was a significant amount of data to obtain (esp through a 36kbps modem) and to transfer about (just over 40 1.4mb disks) However now I have downloaded and transferred the lot in a couple of minutes.

So in todays terms what is a considerable amount of data are we talking terrabytes, petabytes ???