As you know, traditional testing follows a fairly simple model in which there are test cases written beforehand for all the possible use cases and then, after the test cases are reviewed and approved, they are executed until they reach specific pass criteria.
Exploratory testing follows a different approach.
In exploratory testing, tests are designed by the testers simultaneously as they are executed. Exploratory testers do not start a test session with predefined, expected results. Instead, they compare the behavior of the application against what they might expect, based on experience, heuristics, and oracles. This technique has been shown to find more critical and versatile bugs than the more traditional, scripted form of testing.
Advantages of Exploratory Testing
You find more defects with exploratory testing because you have more latitude to try different types of tests while using your past experience and knowledge of the product or its domain. Scripted testing, on the other hand, limits you to only the steps outlined in test cases, which in turn limits your ability to vary test scenarios.
On one of my past projects, we were developing a video surveillance system and adding a feature to export video data and burn it to DVDs. During the first week of development, we performed extensive exploratory testing and found 62 bugs, of which 25 had critical and major severity. And when development was completed and all bugs were fixed and verified, the application did not require additional extensive testing and was ready for client acceptance. We had no test cases or any other documented ways to test, but we used knowledge about similar applications (DVD burners) and had great communication with developers.
The exploratory approach saved us a lot of time and effort, especially at the final stages of development.
Tour Testing Overview
Tour testing enhances pure exploratory testing by adding a structure and a specific theme to it. A tourism metaphor is used that treats exploration of software using the tools of a conventional tourist: organized tours, guidebooks, maps, and local information. This helps set goals that can then guide your decision-making during testing and uncover issues you may not see otherwise.
In terms of software testing, the software product you test is a city, where the tourists come, software features or modules are districts, and a tour is a plan of testing activities.
The tour includes the objectives and tasks to focus on during the testing. There is also a rule that a tour is usually limited to 4 hours.
Tours can be done at any time, but usually defining tours for the end game, when you think your new release I ready to ship, is the best option. Often this will give your team extra confidence that the most important features in your product work as expected.
Many proponents of exploratory testing have created their own sets of tours. In this article, I’ll be discussing the most popular one created by James Whittaker in the early 1990s.
Types of Tours
This is the place in the city where the business is done. In real life, this district is not very attractive for tourists, but in exploratory testing, this kind of tours addresses main application functionality and features that are designed to increase sales, the features with the highest business value. This is what the clients are buying your product for.
The guidebook tour is performed by reading the user manual or online help and following its advice. The goal is to try and execute each scenario described in the user documentation as faithfully as possible.
- instructions and screenshots provided by software documentation are incomplete or outdated
- misspellings, grammar errors, or awkward wordings in the documentation and online help
- the documentation itself is vague or lacks clarity
Testers performing this tour are guided by sales information from demos for the client or marketing videos.
- features described in the promotional booklets and brochures are absent
- information in advertising materials is inconsistent with the application functionality
- application has functional issues that affect its demonstration and use
Garbage Collector Tour
This is like a methodical spot check. The Garbage Collector’s tour is performed by choosing a goal (for example, all menu items, all error messages, all dialog boxes), and then visiting each one in the list by the shortest path possible. A good example of using this tour is a smoke test.
Issues revealed depend on the testing context. If it’s related to the text in the GUI, the issues will be mostly linguistic, and if it’s related to functionality, the issues will be functional.
The historic district represents legacy code and features that were introduced in earlier versions. Tours in the historic district are designed to test legacy code, functionality from the previous version of the application, and bug fixes. The latter are most important because bugs are “social creatures” and tend to cluster in one place.
Bad Neighborhood Tour
As bugs are found and reported, we can connect certain features with bug counts and can track where bugs are occurring on our product. Because bugs tend to congregate, revisiting buggy sections of the product is a tour worth taking.
Issues revealed by this tour are mostly functional.
During this tour, testers should identify older code and ensure it receives a fair share of testing attention.
- application crashes under higher screen resolutions or when run on newer OS versions
- the software does not follow modern standards and guidelines
Tourist needs some rest even on vacation. They visit entertainment districts for shows, not landmarks. Software applications are similar in this regard. For example, a business district for a word processor is its functionality to create and edit documents, add tables, charts, and drawings to them, but an entertainment district here would be to cast some visual beauty to the created documents, stylize them.
Tours through the entertainment district will visit supporting rather than mainline features.
Here we test the least used features and the ones that are the least attractive to users.
- non-critical functional issues
- functional areas or features that the users do not understand without help or hints
- usability issues
On the All-Nighter tour, we keep our application running without closing it. For example, we open browser tabs and don’t close them. We may connect to remote resources and never disconnect. This may reveal bugs like memory leaks or high CPU usage over time.
- performance issues (memory leaks, bad responsiveness etc.)
- data loss
- inability to restore working state after being suspended for some time
Each city has areas that attract tourists. There are many souvenir shops, restaurants, and other places to maximize the tourist experience and increase profits of local vendors. Here you can find fridge magnets and collectibles, try national cuisine, and immerse yourself into the city atmosphere.
Tours through the tourist district take several flavors. These can be special-purpose test scenarios or checklists with specific checks. These tours are not about making the software work, they are about visiting its functionality quickly.
This tour suggests collecting outputs from the software; and the more one collects, the better. The idea behind this tour is to go everywhere you can and document all the outputs you see.
Issues revealed are mostly functional but usually accessed through GUI.
During the Supermodel tour, the focus is not on functionality or real interaction. It’s only on the interface. Take the tour and watch the interface elements. Do they look good? Do they render properly, and is the performance good? As you make changes, does the GUI refresh properly? And so on.
Issues revealed are GUI- and UX-related.
Hotels are shelters for tourists. This is a place to escape from the hustle and bustle of popular spots and to take a little rest and relax.
A hotel district is for getting away from the primary functionality and popular features and test some of the secondary and supporting functions.
The idea behind the Rained-Out tour is to start operations and stop them. We may enter information to search for flights on a travel website only to cancel them after the search begins. We may print a document only to cancel it before the printing is complete. Every time a cancel button appears, we click it.
- critical and non-critical functional issues
- data integrity issues
- usability issues, especially when trying to cancel some flow
Couch Potato Tour
A Couch Potato tour means doing as little actual work as possible. This means accepting all default values, leaving input fields blank, filling in as little form data as possible, paging through screens without clicking any buttons or entering any data, and so forth. If there is any choice to go one way in the application or another, the couch potato always takes the path of least resistance. An example of great use for this tour is testing various application wizards.
Issues revealed are mostly related to usability and functionality.
These are unattractive places that a rare guide will tell you about. They are full of scammers and suspicious people, and it is best to bypass them.
In testing, it is such software areas that attract shifty users who strive to crash the system.
The Antisocial tour requires entering either the least likely inputs or known bad inputs. A tester on the Antisocial tour should find a much less meaningful input than expected from the user.
- crashes and other functional issues
- security vulnerabilities
- non-user-friendly error messages, unhandled exceptions (messages with stack trace)
- usability issues
Testers will enter the same input over and over. They will perform the same action over and over. Mostly, it’s about repetition. For example, order an item on a shopping site and then order it again to check if a multiple purchase discount applies.
Issues revealed can be various, but usually, they are non-critical functional or performance ones.
- “Exploratory Testing 3.0” – James M. Bach and Michael Bolton
- “Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design” – James A. Whittaker
- “Explore It!: Reduce Risk and Increase Confidence with Exploratory Testing” – Elisabeth Hendrickson