User Acceptance Testing and the Application Lifecycle

User Acceptance Testing (UAT) is an important part of the development process. If carried out as early as possible and as regularly as possible, it not only alerts the development team to aspects that don't yet meet the requirements of the users, but also gives governance a better idea of progress. If UAT is delayed, defects become expensive and troublesome to fix. Sophia Segal gives an experienced summary view of UAT

A software application can be robust, scalable, easily maintained, and easy to use, but if it doesn’t do what the customer needs then it is useless. The success of a new product or software application is at the mercy of how precisely it fulfils the requirements of its users. Even if a company has successfully gone live with an innovative product, but has failed to ensure that it meets the requirements of the end users because its suitability was never properly checked, the consequences will be likely to be failure, poor consumer experience, brand deterioration and major financial loss because of having to identify defects and fix them. Despite all this, there is little enthusiasm in the industry for User Acceptance testing (UAT), and project governance continues to make the repeated mistake of undervaluing the process, whether using Agile or Waterfall methodologies.

Why UAT is critical to Software Application Projects.

Without effective UAT (User Acceptance Testing), the chances of the success of a development project are severely diminished. That it is why it is such an important part of the delivery process. UAT can be carried out using an agile approach. In Agile, it is not an independent phase, but instead an iterative collaboration between all the project stakeholders, ensuring continuous test and feedback cycles.

For User Acceptance testing, a Requirements Subject Matter Expert (Requirements SME), or Business Domain expert, ensures that the requirements continues, throughout the delivery process, to be correctly understood and documented within the delivery team. These can then be verified by means of UAT. It is a small investment to make to maximize their revenue and increase brand reputation in the market. It removes unpleasant surprises after release and deployment, and gives Governance a far better idea of development progress. It takes care to ensure that business processes that are essential for the business or which carry a risk of financial or reputational loss are rigorously checked.

User Accepted Testing (UAT) is a structured testing process that makes sure that all user requirements are performing as the user wants and expects. Does the product follow the company’s business regulations for new registration? Does the vendor product screen display under ten seconds when the user selects it? Do the online forms support a department’s legal obligations? Does the business flow handle exceptions and alternative flows? These are all valid UAT requirements which need to be fleshed out meticulously with their corresponding acceptance criteria. The software or product must satisfy these criteria, before the customer or user will accept it.

Requirements SME, who possess direct knowledge of end-to-end business processes, are a key part of the governance of an IT project. They can manage stakeholders to simulate accurate business scenarios, understand the key system functionalities, are able to lead training efforts and can accurately assess the severity of defects and risks on the business.

They will also have a critical role to prepare UAT and to design the overall structure of the tests, by means of a series of concise unambiguous requirements, which form the base for test cases. By shepherding real business users through the process of running these tests, UAT reveals all the shortcomings of the application under development. This is only effective if there are concise user requirements, which will evolve over time as they are reviewed and refined in software projects.

Any application development has to meet its requirements. As well as ensuring that these are all identified and understood, a Requirements SME needs to specify the acceptance criteria in terms that can be understood by the entire team. As soon as an issue or defect is identified, it is fixed or resolved before the product is deployed, thereby saving time and cost.

The Warning Signs

Concentrating on other types of testing such as unit testing, integration testing or performance tests at the expense of UAT because of project time constraints

If testing focuses on the technical functionality of the software instead of looking at the complete end-to-end testing process, defects will only be discovered after the product has shifted to production, and will be likely to cost at least ten times to fix than had it been captured earlier in the process

Domain experts and business staff who lack the know-how and cannot accurately identify scenarios relating to the operation of the whole business process which needed to be tested.

With no expert input to extensively prepare and manage a user acceptance testing process and risk assessment errors will not be captured before the system goes live, the software application will not behave as the end users anticipate and the developers will be swamped by change requests from the business at the worst possible time to find and resolve major defects.

Requirements Subject Matter Experts must understand business and application software requirements and critical business flows, and be able to competently prepare and validate data which are realistic and pertinent to the business.

UAT is seen as part of the deployment process rather than development

UAT should be an integral part of the development process so that development teams can get immediate warning of any misunderstanding of the acceptance criteria. This allows teams to make changes or a modification to the criteria at a point in the application lifecycle where it is easy to do so.

There is no clear UAT plan in place

An effective UAT plan needs to detail out the key focus areas, do a risk assessment and prepare the user acceptance test cases to test the system. These should use real world test scenarios for the applications and be in sufficient detail to implement scenario-based testing to conduct the UAT process.

There is no Change Control process

Change control is as important as issue tracking in development. It makes all changes visible to the team and makes sure that team members can bring their expertise to bear on decisions on any proposed changes. This provides a better understanding of their impact and risks are understood. If Change control interferes with progress, it means that there are too many changes. Complaining about change control is like complaining about the loud noise of a fire alarm when the building is in flames.

UAT defect management is ad-hoc

There must be an effective defect tracking and management processs that allows the delivery team to review test progress and metrics with the testing team, determine priorities and to resolve all defects found during UAT testing. All high-priority defects that would prevent a deployment can be quickly.

Testing is done in isolation by specialized testers

UAT involves testers, developers, business representatives, Governance, administrators and third parties. It is an aspect of development. If, for example, compliance experts aren’t involved, then it is easy to miss a statutory requirement that affects the basic test scenarios. If business representatives aren’t involved, then obvious defects can easily be missed. All team members that are involved in the UAT process need a clear understanding of their responsibility, roles and tasks. All test needs and models must be discussed with the developers, QA team representatives, database administrators, Operations staff and the testing team.

Ambiguity in the definitions of requirements

Ambiguous requirements often leads to testers and developers to unknowingly misinterpret them, which causes requirement errors to slip through UAT and morph rapidly into defects. These defects surface after implementation, at a time when it costs tenfold to pursue and resolve the same defect. It is therefore important to capture and define these shifting, ambiguous requirements and flesh out meticulous acceptance criteria so that the intended results are visible to all. Ambiguity can be one of a range of critters. These include

  • Ambiguous terms : subjective or vague terms that cannot be measured
  • Conflicting requirements : Two or requirements that conflict each other
  • Incomplete requirements : Missing values, business rules, etc.
  • Missing requirements : Possible missing requirements that have not been defined
  • Unclear requirements : Requirements that can be interpreted in multiple ways
  • Glossary : Term is not found in the glossary reference document
  • Grammar, spelling and wording : Spelling mistake, grammar, rewording suggestion

Key Steps When Implementing an End-to-End UAT Process

User Acceptance Testing is a challenging part of software delivery. If the process is fudged, it is too easy for buried issues to materialize quickly and escalate to the point that they risk the successful delivery of the development project. Because software delivery relies on being able to change rapidly in response to changes in the business requirements and a better understanding of the business domain, UAT must rapidly change its criteria and scenarios to remain in step. This means that it is a process with its own lifecycle. It can be agile, but it must never lose precision. We will discuss the process in a series of sections.

Planning

The UAT plan should be an accurate description of how to conduct the User Acceptance Test, couched in plain language that assumes the minimum of technical language. The test plan describes the effort and resources, entry and exit criteria for UAT, scheduling, test scenarios, test cases strategy, risk management and assessment and timelines of UAT testing. Because priorities will be assigned to the business requirements, the planning document is used to confirm these priorities with the business and gain both feedback and agreement on critical business flows, the scope and complexity of the testing and defining acceptance criteria.

The Test Plan is referenced by team members on how the testing activity is going to be approached and should be current at all times, easily accessed by the team. Here is a suggestion for the framework of the plan

Intended Audience

Write a description of the target audience to identify the consumers which also prevents this document from being exercised inappropriately.

In Scope

Describe the areas, scenarios or functionality that have to be tested and are within the scope of this UAT.

Out of Scope

Outline the high level functionality, scenarios, areas that will be excluded from scope, especially where the intended audience might assume the inclusion of those areas. E.g. What aspects, including business Requirements, flow, and functionality are not going to be tested in UAT for this project?

Assumptions

List all the conditions that are expected to be true so you can carry out the UAT.

Test Documentation

Test Cases, Test data, Setting up the environment, Test execution, how many Test cycles are required, timelines for cycles, test scenarios

Roles and Responsibilities

All team members are listed, their roles and responsibilities, and their contact details. As an example:
Sophia Segal.
Role: UAT Lead
Responsibilities:

  • Provides management, supervision and logistics
  • Agrees on goals and objectives.
  • Obtains resources
  • Reporting
  • Champion of UAT interests
  • Leads and communicates with team
  • Identifies target user group
  • Evaluates UAT performance

Deliverables

What documents are necessary and the estimated time frames? What is expected from each document? E.g. UAT Scenarios, Test Cases, Test Results, Defect Log, Defect Management Plan, Metrics, Risk identification.

Environment

What are the environment requirements? E.g. hardware, software, test data, staffing, skills training.
What actions to follow if an error occurs. Describe the environment in which the UAT will be executed.

Tools

What tools including software are going to be used for Productivity and Support?
Tools for incident management, test scripts automation tool, e.g. Mantis, Jira.
How to login for the UAT process.

Defect Management

When and who do we report the defects to? Is there a defect management procedure to follow? Do we have a baseline? How are we going to report these defects? Do we provide screenshots? Do we provide metrics?

Risk Management

Identify risks that may affect the execution of the UAT, assess likelihood and quantify project impact, detail a mitigation strategy and contingency plan for each risk. Any critical risks known or vulnerable system areas.

Communication

How are you going to communicate the schedules to the users? How are you going to communicate the defect management process? Will there be regular status meetings or email updates?

Documentation Control

Outline what system repository, documents will be saved to. Everyone on the team should be made aware of and adhere to document version control, to avoid issues such as test cases referencing old versions of requirements which will make test results inaccurate and traceability confusing.

Enter criteria

What is the criteria that must be met before UAT can begin. E.g. Business requirements must be available or User stories or use cases must be available.

UAT Environment

it is usually important to execute the tests in a test environment that simulates the real world production environment as closely as possible. This means that, resources will need to be installed, setup and configuration of the test environment needed to be in place, including hardware that was required for the execution of the testing and suitable test data in place. The testing environment should be ready before execution can begin, as sometimes there is more than one environment running, which can be confusing if a defect is discovered, but cannot be targeted to its environment, if multiple environments exist. Also it is important to agree on the data required for testing and if scripts will needed to be run, or will UAT be using data sets.

Understanding your target User Group

Incentive Level

IT skills level

Test Complexity

UAT Training Effort

Low

Low

Low

High

Medium

Low

Low – Moderate

Medium – High

Medium

Medium

Low – High

Medium

High

Low

Low – Moderate

Medium – Low

High

High

Low – High

Low

The target users must be engaged if the testing and project is to be successful. There are plenty of influences that can prevent this happening. Users could be already fully-occupied in the workplace; their own jobs might be jeopardized by successful introduction of the application. Or they might not be completely convinced by the importance of the end-product and so lack the desire or incentive to collaborate. Alternatively, they may not have grasped enough IT skills to accomplish the test scenarios and will need some training. In the past, I have characterized users and grouped them in terms of their computer skills, incentive level and training effort and marketed those to the users. It is important to gauge on a user’s strength, weaknesses and desire to understand direction, to be aware of key user roles and form the right balance of IT skills and training effort.

User engagement and transparency can be increased by collaborating with users to flesh out their critical business concerns and devoting UAT efforts in these key areas, or by developing additional step-by-step scenarios or collaboratively walking through process pain points in detail.

I cannot stress how important it is to engage with users earlier on. It also helps to identify change agents from the user group, who can verify business users are all well represented in UAT and help in the running smoothly of UAT and champions its value.

Designing Test Cases

Test cases must be real-life scenarios that the user needs to perform using the application. For composing Test Cases, I always trace back to the business requirements document. The Business Requirements should be available from a signed-off Business Requirements Document or Use Case and reference an actual version number of the document as stored in the documents repository, to make sure that the most current document is being accessed. I have also facilitated sessions with users adopting Requirements Traceability Matrix, Functional Specifications, Use Cases and Business Process workflow diagrams as input for creating test cases. This is also a great technique for requirements validation, scope definition, identifying task inputs, outputs and all triggers to illustrate dependencies and impact analysis for test cases and test scenarios.

Each test case covers a specific scenario of the software and captures all the steps accurately which the user has to carry out, with respect to high level business flows and also be able to verify if the software application or product is working as predetermined.

These test cases can be very detailed with refined step-by-step instructions or captured at a higher level. The challenge with high-level test cases is that there are usually multiple business flows on functionality, including exceptions and alternate flows which could easily be missed and so not properly tested. It is important to get the right level of granularity especially if there is a new critical path through the application that needs testing or many flows which need to be captured.

It is also important to capture, with the users, not just the ‘happy path’ but also the negative paths that capture exceptions and alternative flows of accomplishing a user task. This ensures the whole scope of how the user behaves with the system and what they need to achieved is considered and needs to be logically well thought out.

Test Case ID

Test Case Name

Test Description

Pre-condition

Step

Execution Steps

Expected Result

Test Data

001

Login Page

A Registered user should successfully login to Vendor Launch site.

The user should be registered, have a use rid and password

01

Open Vendor Launch Login Screen

The Vendor login Launch screen is displayed

001

02

Enter Userid

Userid is correct

Correct Userid (6-12 characters)

001

03

Enter password

Password is correct

Correct password (4-16 characters)

001

04

Select Enter button

All mandatory fields validated and user is successful in logging in to Vendor Launch site.

All Test Scenarios should be documented if they are a Pass or Fail. The above Test Case could have multiple scenarios for exceptions and alternative flows and additional steps would need to be captured for the following scenarios.

  • user cannot login
  • userid or password is incorrect
  • field is left blank
  • the user is not registered
  • uppercase is not permitted for password
  • password is being displayed
  • steps on displaying error messages
  • displaying information messages
  • exiting the screen
  • navigating to a different application
  • printing and reporting capabilities

Execution

In my last project, Beta Testing took place in the customer’s environment and involved extensive testing by users. They executed all steps written in the scenarios, and, depending on the results returned, checked either Pass or Fail. This process was repeated for all Test Cases. For any failed Test Cases, defects were raised. The users provided feedback, which led to refinements of the software application being tested. During execution, screenshots were taken to capture how abnormalities were triggered. This helps when testers, developers or business teams are dispersed. I also took note of any timestamps when defects occurred, to cross-reference this with any dependencies in the backend. Executing and documenting the results should happen simultaneously to avoid losing any data and consolidate an overview of critical issues and metrics.

Evaluating UAT Results

How many open defects are there that need to be fixed? How many defects need to be re-tested? How many defects are closed? How many defects were captured in the first release? This are some of the analysis that is required during this phase. The evaluation phase is an extensive process, since each test case needs to be scrupulously analyzed, placed into quantifiable context, and referencing pre-defined acceptance criteria, evaluated to see if it has been tested adequately.

The quantitative and qualitative data collected was documented and translated into meaningful insights. For example how many defects are ‘In Test’? If this is higher than expected, should we acquire more resources? Do we have high-priority defects in an open state all assigned to one person? If yes, then we need to find more development resources. How many defects have been raised and how many have been resolved? I also documented status reports to illustrate high-priority insights including metrics on test coverage, top priority defects, defect statistics and added filter parameters to reports to allow stakeholders to find data that was important to them.

How do we calculate Metrics?

Defect Detention Percentage (DDP)

DDP = Number of defects found during UAT / Number of defects found in total.

DDP measures the value of testing. For example, if the number of defects found during UAT is 70 and the number of defects found going live is 30. That’s a DDP of 70%, which is satisfactory. Additionally, DDP can be used to measure the quality of testing when making changes in the UAT cycle. Example of these changes are switching user groups, testing environments or between different releases. Always measure the defects before any change, and then after, to measure the quality of testing.

Completion of Acceptance Criteria

Defining concise acceptance criteria is key to a complete Test Case. Not only does it clearly illustrate what the user expects from a scenario, how the requirement should be met, but also verifies the quality and scope of a test case scenario and exit criteria. This is calculated by counting all the acceptance criteria, including scenarios, and dividing the number by how many acceptance criteria have been completed with the expected results.

Measuring the cost of a defect

Usually, as an initial step I list all defects in a UAT release. As I always have a defect management process in place, for one defect found, I list the time taken from defect identification to close, in hours and multiply this by the hourly rate. Below is an example for one defect.

Defect 001

Description of Task

Number of hours
dedicated to defect

Defect Identified and being analyzed

1

Defect in progress

3

Defect resolved and being tested

2

Documenting of defect

2

Update Testing Plan documentation

1

Update software with fixed defect

2

Communication of defect to team

2

Run software with revised, fixed defect

2

Log results and impact analysis

3

Total Number of hours/effort

18

From above, in total 18 hours were resourced to this one defect. This does not take into consideration, the hours needed if the defect was not resolved in the first cycle. In this example, the cost of the defect is
      Cost of defect = Total Number of hours/effort * average hourly rate
From this, you can calculate the total costs of all defects in a release.

Managing Defects.

Defects caught in UAT, are because the product or software is tested and analyzed from a user viewpoint and is based solely on requirements. Because of this, previous testing methods tend to miss defects which can be critical.

It is important to have a comprehensive process for tracking defects. This is important, because the team have to make sure all fixes are complete before the product is delivered to the customer. Defects should be re-tested, once fixed, as this not only generates user confidence, due to managing a transparent defect process but also confirms that you are fixing what has been agreed by all.

For one of my UAT projects I managed, all data feeds successfully loaded into the data warehouse. There was one defect. It was a data issue, where a unique product number format needed some manipulation to conform to the company-wide format. This was resolved quickly by using a systematic defect management process for defect status, tracking and communication, including tracking the actual release the defects were fixed in. Any critical defects which remained open were considered risks and pushed on to Risk Management.

Open

Identified bugs are assigned. If there are any familiarities with previous bugs, I make a reference.

In Progress

Bug has a description and been assigned.

Resolved

The bug has been resolved and the updated version sent to retest.

Re-Test

The bug is being tested again, after repair.

Re-opened

The repair has not worked. The bug goes back to status ‘Open’

Closed

The bug is resolved. Closed and released.

Requirements Defect Severity classification

An additional technique, illustrated below, measures severity of requirement defects and is very important to assess critical issues and provides a useful lens for prioritizing repairs in a software project. These comments are usually noted in requirement documents or technical documents such as Test Cases which originate from Requirement documents.

Comment Type
 Definitions.

Type 1

Comments on a possible gap or miss in defining requirements with no work around. Type 1 issues can become a show stopper.

Type 2

Comments on adding more information or clarity to requirements. Type 2 issues are not show stoppers but can cause ambiguity that might lead to confusion for developers and testers.

Type 3

Comments on semantics and grammar and other defects of lesser importance. Type 3 issues are usually about the quality of the document.

Risk Management

Not every project requires comprehensive testing, however, projects that I have managed, involved extensive validation because of business functionality that carries with it a risk of financial loss if it fails, or has errors, and so risk assessment was required. I reference the master project plan to begin a list of risks. As a further technique to cover all the potential risks associated with UAT, I perform a risk assessment and documented the probability of risk occurrence, risk impact and risk mitigation. It is important to manage and control any unresolved defects and low risk from UAT, as they can easily metamorphosis into dangerous risks.

Risk Assessment

Risk assessment involves identifying the critical systems and business areas most vulnerable to risk, so that resources can be assigned where the risk impact is highest. In this scenario, the highest risk impact has a direct correlation to defects that potentially have serious consequences. For all the risks, assign a score to each one, probability of occurrence and its impact on the user and contingency plans if the risks occur. Compile a risk report, assigning each risk a unique id and a description and review it once a week with the team. Any new risks, derived from open defects, should be raised as they are identified. Other risks can be, and usually are, adjusted in terms of significance as more information comes to the surface about them.

Risk

Probability

Impact

Mitigation

Defects are found late in the UAT cycle

High

High

Defect management plan is at hand to ensure clear communication and resolving.

Types of Risk

  • Ambiguous requirements
  • Natural disaster
  • Third party products and services
  • New versions of interfacing
  • The test team’s ability to use technology required for the test effort.
  • Working across multiple sites
  • UAT environment not available
  • Adoption of any new technologies
  • Schedule slippage and its impact on the test schedule
  • Changing requirements

Sign off

The users of the application should be presented with a formal test completion report where critical metrics and results from UAT are presented. They will then need to make a decision whether these results are acceptable and in accordance with their criteria and expectations and, if so, approve the UAT. This is also called the Go/No-Go decision. Users should be sent an email for sign-off and post sign-off, stating whether the product or software application is ready for deployment.

Sign-off from the users is a key phase in the UAT life cycle, as it not only illustrates that business and IT are on agreement and aligned, the results delivered are ultimately what the business requires, but moving forward, if the results from the UAT are not accepted or what the business really asked for, in reality, it will deliver zero business value.

Agile or Waterfall.

Agile UAT is usually conducted concurrently with development, and reviewed with users at each Sprint and any defects are resolved before the next iteration. In waterfall, the UAT is performed after development, before product is deployed.

Just as a UAT test plan is key, when using a waterfall approach; with agile, a comprehensive product backlog provides a dashboard for all the product information you need, to share with your team.

Each backlog item, when drilled down, should detail the user story requirements , and capture what functionality the user needs. It should also include, defined acceptance criteria, signalling when this functionality is complete.

The important take away, is regardless of the project approach, collaboration with your user group to validate un-ambiguous, testable requirements is critical, in order to to add value to a product or software requirements project

Conclusion.

There must be no surprises during deployment, unless they are pleasant ones. Software deployment is generally fraught because User Acceptance Tests are usually mismanaged, and left until the end of a software development phase, or skimmed through chaotically by untrained staff. The consequences are horrible, because defects turn up in the production system, and can cause direct financial loss to the business. Additionally, this is the worst time to put them right, so the cost of the project will increase rapidly.

Whether in waterfall or incremental Agile software applications, it is clear that the success of a new product or software application will depend on how well it meets the requirements and demands of its users. Expert-led UAT practice is needed to check whether a system meets the business requirements of the users. This practice brings out all-important functionality and business problems while mitigating risks and reducing high costs of redevelopment.

  • 8218 views

  • Rate
    [Total: 56    Average: 4.8/5]
  • Ich Apps

    Its very important role in software sector when the application is completed then finally it is checked out inclient place.

  • Alicia.B

    Great article, very helpful and informative. I’m going to use this for a project I’m working on currently, very valuable to use as a guide!