Francisco Fagalde, July 2, 2020
The Quality Assurance (QA) process during a project development can be done in many ways. In this article, we'll cover the best QA methods to provide comprehensive and effective test cases for 3 main testing practices, including: Regular Manual testing, Automation and Regressions.
As you may know, each tester has their own unique style (especially when designing test cases) but there are some basic procedures and best practices that should be followed.
Manual Testing
Manual testing is the most widely used practice in a project development. It is especially utilized in Adobe Experience Manager (AEM) projects as it's the currently the only way to properly test AEM author. This practice focuses mainly on functionality and specific requirements.
Manual testing can be divided in three main steps:
- Researching the requirements during development and designing test cases for it.
- Executing said test cases, validating all target areas of the implementation and reporting bugs if applicable.
- Periodically revisiting the implementation and verifying that it still works as expected after an integration or when changes are applied (mostly known as regression).
A good test case design is vital for all these instances, it is important to keep it clear and easy to understand, easy to update and easy to reuse. All test cases should be designed with pieces of information about the user story, a unique ID or name for each test case, preconditions, descriptions of the test scope, steps, expected results, actual results, and additional information/data sets if available.
In the example above, each step within a single test case has an expected result. This is useful during an execution to quickly identify at what moment of the execution the test failed.
It is normal to see test cases (including a single result option) at the end of all the steps. Including this small addition for each scenario allows the person in charge of the execution to report multiple bugs within a single test case and assign them directly to each action or triggered function.
It is also recommended to include a section for reported bugs for documentation and visibility purposes.
As mentioned previously, exclusively manual test cases can be written in any style as long as they are clear, easy to follow and easy to update. However, in cases where automation is part of the process, it is recommended to design test cases using Gherkin language (see automation section). This will provide a standardized structure that simplifies the entire process to the point where you will only need to copy and paste scenarios into the code.
For the results and reporting section it is also recommended to follow the same structure.
It is considered best practice to include the ID of the reported bugs on a failing step, as well as a link to the repository or management tool where the issue has been logged. This is critical so future testers can check the original requirements in the development tickets.
Regarding the Testing Data section, it’s important to be careful and only use it if required. This is important to avoid the pesticide paradox (a topic we cover during our live QA presentation at Evolve Conference) and don’t waste resources on an ineffective execution.
There are many tools that can be used for designing test cases including: Regular spreadsheets, Google Drive, TestRails, Zephyr and HP quality center. Deciding which tool is best for the job, mainly depends on each project's unique needs, budget and preference. Most of these tools, aside from regular spreadsheets, can be integrated with project management tools like Jira and Confluence which are highly customizable and allow the QA Analyst to include all the mentioned sections and more.
Automation
There are two methods for Automation testing. One is to create an automated tool from scratch and run quick validations on a regular basis, then provide a report (a link checker for a group of sites for example). The other option is to automate previous manual test cases for regression purposes on the publish side.
The first method is primarily based on your code and includes some best practices, such as:
Source of truth:
Tools that are automatically executed from a platform like Jenkins (typically link checkers), need an input file from where all the links are collected. On active projects it is of high importance to regularly compare this list vs a source of truth provided by the client.
External links:
Many of these sites contain links to external pages. It's recommended to include a method that collects all links within a page, then list them for the validation process (and for output information/reporting). Finally, ignore all reference links to external sites that are not within the project scope.
The second approach is usually a bit more complex because the preparation starts during the manual test case design.
Gherkin Language:
The Gherkin format is used to list the expected behavior of a functionality without going into too much detail (like in manual test cases). There are specific words in this format that can be used in the preconditions, steps and expected results. These words are then recognized by cucumber in your script allowing it to follow a specific structure.
Gherkin is commonly used to design test case templates that only require edit the specific data for each new scenario.
Gherkin words or commands:
Given: This command indicates the initial state of the application or functionality you wish to test
When: This command indicates the trigger action the tester or tool must perform in order to check the behavior
And: If needed, this is an additional action that must be performed after the initial trigger
Then: This command indicates the expected behavior after the previous actions are triggered.
But: This is commonly used as an extra validation step after the initial check. Mostly negative validations on a test case.
It's evident that manual test cases can be designed using this format and be perfectly executed following the instructions manually - but how do we use these commands for automated scripts?
Automated scripts using Gherkin:
The most used tool to automate and use Gherkin language as annotations in your code is Cucumber. In order to apply this, we need two main classes - the feature file and the steps class.
The feature file contains the basic information of the test and it’s easy to understand:
After this, each step is defined in step classes:
Regression
A good regression test plan outlines what and how testing will be performed to ensure that no functionality of the application and website(s is affected because of code changes.
Depending on the scope of the changes, testers can consider various regression testing approaches:
- Retest all
- The entire set of test cases are re-executed. This approach requires more time and resources compared to the other techniques.
- Regression test selection
- Test cases are selected from the test suite to be re-executed. The selection of the test cases is based on the code change in the application.
- Test case prioritization
- Test cases are placed into groups in order of high, medium, and low priority. Test cases with a higher priority are executed first, followed by those of a lower priority. The test case priority can be determined by several factors including the criticality and impact on the application and usage frequency covered by the test case.
- Hybrid
- This technique is a combination of the test case selection and test case prioritization approaches
The selection of the regression test suite is based on one or more of the following criteria:
- Test cases that cover the component/functionality where the changes have been done
- Test cases for the core functionality or feature of the application
- Functionalities that are frequently used
- Integration test cases that include all the major components/features
- Complex test cases
- High priority test cases should be included
- Test cases that frequently fail or recent testing defects were found
A regression test can combine manual and automated suites depending on the case. The methodologies explained above should be applied to both of them.
In Conclusion
The quality assurance process is a time demanding and necessary practice in every project. The more complex the development is, the more testing it will require. This means that early preparations and planning is vital to the successful launch of any new feature. For more information on testing best practices please visit our QA, UAT & GO LIVE presentation at EVOLVE19 by following this link: https://www.youtube.com/watch?v=TNS8Xec5JBQ&t
If you're in need of a QA tester (or a small army of them) equipped with unparalleled knowledge of Adobe Experience Manager, please reach out us- we're happy to help!