Software Testing - Acceptance Testing
Acceptance Testing
Large scale software projects often have a final phase of testing called “Acceptance Testing”.
Acceptance testing forms an important and distinctly separate phase from previous testing efforts and its purpose is to ensure that the product meets minimum defined standards of quality prior to it being accept by the client or customer.
This is where someone has to sign the cheque.
Often the client will have his end-users to conduct the testing to verify the software has been implemented to their satisfaction (this is called “User Acceptance Testing” or “UAT”). Often UAT tests processes outside of the software itself to make sure the whole solution works as advertised.
While other forms of testing can be more ‘free form’, the acceptance test phase should represent a planned series of tests and release procedures to ensure the output from the production phase reaches the end-user in an optimal state, as free of defects as is humanly possible.
In theory Acceptance Testing should also be fast and relatively painless. Previous phases of testing will have eliminated any issues and this should be a formality. In immature software development, the Acceptance Test becomes a last trap for issues, back-loading the project with risk.
Acceptance testing also typically focusses on artefacts outside the software itself. A solution often has many elements outside of the software itself. These might include : manuals and documentation; process changes; training material; operational procedures; operational performance measures (SLA's).
These are typically not tested in previous phases of testing which focus on functional aspects of the software itself. But the correct delivery of these other elements is important for the success of the solution as a whole. They are typically not evaluated until the software is complete because they require a fully functional piece of software, with its new workflows and new data requirements, to evaluate.
14
Organisations often seek to reduce the cost of testing. Most organisations aren't comfortable with reducing the amount of testing so instead they look at improving the efficiency of testing. Luckily, there are a number of software vendors who claim to be able to do just this! They offer automated tools which take a test case, automate it and run it against a software target repeatedly. Music to management ears!
However, there are some myths about automated test toolsthat need to be dispelled :
Automated testing does not find more bugs than manual testing – an experienced manual tester who is familiar with the system will find more new defects than a suite of automated tests.
Automation does not fix the development process – as harsh as it sounds, testers don’t create defects, developers do. Automated testing does not improve the development process although it might highlight some of the issues.
Automated testing is not necessarily faster – the upfront effort of automating a test is much
higher than conducting a manual test, so it will take longer and cost more to test the first time around. Automation only pays off over time. It will also cost more to maintain.
Everything does not need to be automated – some things don’t lend themselves to automation, some systems change too fast for automation, some tests benefit from partial automation – you need to be selective about what you automate to reap the benefits.
But, in their place, automated test tools can be extremely successful.
The funniest business case I have ever seen for automated test tools ran like this :
– Using manual testing, we find X number of defects in our software
– It costs $Y to find and fix these defects each year (developer + tester time)
– We can buy an automated test tool for $Z/year
– Since $Z < $Y we should buy the tool and, as a bonus, it will find < X defects
So, not only are you comparing the one off cost for buying tools (without set-up or maintenance) with the cost of manual testing, but the tool will do away with the manual testers as well – the ones who find all those pesky defects!
The Hidden Cost
The hidden costs of test automation are in its maintenance.
An automated test asset which can be written once and run many times pays for itself much quicker than one which has to be continually rewritten to keep pace with the software.
And there's the rub.
Automation tools, like any other piece of software, talk to the software-under-test through an interface. If the interface is changing all the time then, no matter what the vendors say, your tests will have to change as well. On the other hand, if the interfaces remain constant but the underlying functional code changes then your tests will still run and (hopefully) still find bugs.
Many software projects don't have stable interfaces. The user-facing interface (or GUI) is in fact the area which has most change because its the bit the users see most. Trying to automate the testing for a piece of software which has a rapidly changing interface is a bit like trying to pin a jellyfish to the wall.
