Ensuring a software application is thoroughly tested is crucial for most software projects. By conducting comprehensive testing, any potential bugs or defects can be identified and addressed before they have a chance to cause any negative experiences for users.
Today, a typical QA process may consist of the following tasks:
- Analyzing requirements
- Planning and designing tests
- Executing tests and reporting bugs
- Running regression tests
- Running release tests
There are several tools that can help you run all of the steps above more smoothly, such as Jira, Azure DevOps, Redmine, Asana, etc. In this article, I want to focus on the two most important parts of manual testing–designing your tests and execution of those tests (#2 and #3, bolded above). We’ll look at how to properly set up your Azure DevOps tests to get the most out of your manual testing.
Perhaps in a future blog I can cover how do do the other 3 items above using DevOps.
Plan and Design the Tests
Imagine we have a project in Azure DevOps with some user stories in it. Let’s assume we would like to make sure we cover those user stories with tests before we start testing.
Test Plans
First of all we need a Test Plan. In Azure DevOps, a Test Plan looks more like a list of Test Suits and Test Cases versus the standard use of the term Test Plan with which you may be familiar. Head into the ‘Test Plans’ section and create a New Test Plan.
My preference is to do one Test Plan for each small project or phase of a bigger project (e.g MVP, Phase 2, Phase 3, etc.). However, Azure DevOps makes it quite tricky to work with tests from different test plans.
To get started tracking your tests, let’s look at how to set them up in Test Suites, Test Plans, and Test Cases.
Test Suites
There are 3 types of Test Suites in the Azure DevOps:
- Static suite
- Requirements based suite
- Query based suite
If you want to cover separate User Stories, I would recommend creating a Requirements Based test suite. This type of Test Suite makes a relationship 1-to-1 between a User Story and a Test Suite. Also, it gives you the opportunity to see your tests and Pass/Fail status on the Board for a particular User Story:
A Static suite is the best fits to general purposes, e.g if you want to cover by tests, already existing project, post-factum.
Query Based test suites will help with Sanity/Regression testing. You can sort existing test cases by Priority, Area Path etc. to form the best suite for your purposes.
Test Cases
Let’s add Test Cases to our Test Suites. There are couple of ways to add them to your Suite:
- Test Plan -> Test Suite -> Define
- Board -> Add Test
Make sure that each test case has a Priority and Area (functional zones of your app). It will help you to sort test cases for Sanity/Regression suits in the future. Also, it will give you the opportunity to see useful QA Statistics/Charts, regarding your test coverage.
Here are some of the most useful options for Test Cases and QA Statistics in Azure DevOps:
- Configurations – let you add Browsers, Devices to set them in Test Execution for Cross Browser, Cross Device testing.
- Parameters – let you set some static values e.g Credentials to Test/Prod environment, that you can use in Test Cases later.
Execute Tests and Report Bugs
Once our Test Suites, Plans, and Cases are ready, you’re ready to execute your tests and track the restules. Do do this, you can simply go to ‘Execute’ tab, set the configuration that you need, and Run your test execution:
While running your tests, make sure that bugs that was found are created inside of Execution pop up. This ensures that these bugs will have correct relationships to make your Charts more accurate in the future. If you don’t log bugs this way, you will need to set the relationships between your QA execution and the bug manually.
As a result, we will have a visualization of our Test Execution for your stakeholders or leadership team on the Board:
And on the default Progress Report:
Also, you can create your won custom Charts for Test Plans/Test Suits and for Bug tracking:
- Test Plan -> Chart
- Boards -> Queries -> Charts
Since Charts in Azure DevOps is quite a big topic, I will also review those feature in more detail in a future post.
Summary
With just a little work to make sure you have the proper setup in Azure DevOps projects, you can test and track the quality of your application more easily and accurately, and see how your application’s quality is progressing over time.
