Tests and Automatic Tests: A Better Approach to AEM QA

At 3|SHARE, many of our clients that come to us for Adobe Experience Manager (AEM) Quality Assurance (QA) work seek help when they have an AEM project that’s about to be released. It’s a common thing that happens with many companies: they have an idea for their website or application, catch the enthusiasm bug, and rush at 1,000 miles per hour to build it. Then, as the project nears completion and is about to be released, they start running tests…and quickly realize they have a problem.

It’s a common approach, but it’s not the best one. Ideally, you should work through a QA cycle at the end of each sprint, taking an iterative approach. Why? Because getting to the end of a project only to discover that you have a defect list that’s 263 items long is the surest way to delay your release – not to mention, it’s costly. The solution is testing early, testing often, and leveraging smarter strategies that allow for better planning. Let’s take a closer look at a better way to handle tests and automatic AEM tests.

The Lifecycle of a Defect

If waiting until a project is about to be released before doing any AEM QA work is practically guaranteed to delay a release, why do companies wait until the last minute to perform any testing? In most cases, they believe that an iterative testing approach will slow them down, but in actuality, the reverse is true.

To understand how delayed QA can wreak havoc on your timeline, let’s take a look at the two possible lifespans of a defect. A defect identified during the development phase has a much shorter lifespan, resulting in lower costs to the organization:

1. One day, a developer creates a defect.
2. The more time a defect spends in the codebase, the more expensive it’s going to be to fix it.
3. If a developer discovers a defect through their own unit testing, the cost to the organization is limited to the time it took the developer to write the unit test.

On the other hand, a defect that remains in the codebase all the way to QA results in increased time and costs to rectify it:

1. If a defect remains in the codebase all the way through the development cycle and still exists by the time a project gets to UAT, that means the bug has left the development environment, and someone has deployed it to the QA/Stage server along with the rest of the code. 
2. In this case, a QA resource tests the software and discovers the bug.
3. The QA resource must then document the bug and send it back to development, where the developer must be able to replicate it, fix it, and then deploy and re-test, a process that takes much longer and results in increased costs.

A third – and most devastating – scenario is a defect that makes it all the way to production without being detected. In this case, you’d have big problems because a defect that could have been corrected weeks or months ago is now possibly impacting your revenue. If you have an e-commerce site, for instance, a defect is almost certainly impacting revenue – and someone has to realize that something is off (typically someone who notices that revenue is suddenly down). Even after the realization that something is amiss, someone has to research to find the root of the issue, and the process generally involves a lot more people and other resources, including management, who are almost certainly involved now that revenue is affected.

In short, the idea that you’ll save time by waiting until the end of a project to perform any testing is erroneous. By the time you get a fully functional app to your end users, bugs take longer and are more expensive to fix. And, the lower the level of code where the bug exists, the greater the cost to fix it.

What Should Organizations Be Testing Throughout Development?

There are three main types of testing for AEM applications: functional testing, automated testing, and load testing. Every project should utilize all three. AEM 6.3 provides tools for automating test cases such as Hobbes.js, a testing library written in JavaScript, and Calvin SDK, built on top of the Hobbes.js framework, which can be used to create automated tests for Adaptive Forms within AEM. These tools are most helpful, though, when you leverage them to test the right things, automate testing when possible, and test early and often.

Anything related to the user experience should be tested throughout the development cycle (and throughout the entire lifespan of the application), as well as any integrations with the backend system. Any outages to the backend should translate to a neutral user experience – at a minimum.

If revenue is impacted, it’s typically noticed within a matter of minutes (usually seconds), so any integrations with backend systems that are revenue channels should be a primary testing focus, all the time. For most organizations, the testing priorities look something like this:

1. Anything impacting revenue – revenue channels, integrations with backend systems affecting revenue, etc.
2. User experience – responsiveness of devices, how things actually render, etc.
3. Load – load testing is often diminished during development (because there’s no load on the system), so it becomes the most crucial thing to test right before delivery

Because most page load speed testing is performed without any actual load on the system, slowness often isn’t recognized until after the application is deployed to the real world. As is the case with code-level bugs, the earlier you detect page load speed issues, the easier and less expensive they are to rectify. Getting load testing going early is a key priority.

How the Pareto Principle Impacts Development

The Pareto Principle can be applied to just about anything, and software development and software quality is no exception. If you look at the total cost of ownership, for instance, about 20% of the cost is actually building an application, and the other 80% is maintaining it.

The earlier you can create a full suite of automated regression tests, the more value there is to your organization – and the less likely you’ll end up in a crunch right before your planned release. By building a suite of automated regression tests early, you can benefit throughout the entire lifespan of the application.

Automated AEM Tests vs. Manual Testing

The reality is that not everything can be automated. At 3|SHARE, we offer manual (functional) testing as well as automated and load testing. Ideally, goals are set at the beginning of the AEM project, such as aiming for 80% of test cases to be automated. That’s a goal that most organizations can achieve when they start building automated AEM test cases from the start.

The things that aren’t a good fit for automation are generally those that involve an existing manual process. Say an organization has a form submission process, and a person responds to those form submissions. In this case, you could automate the form submission, but not the response.

AEM test automation can also be more complex when you’re working with 3rd-party vendors. As an example, if an organization sends something to a vendor, and someone at the vendor’s organization has to do something with that information and then send it back, that’s a process that can’t be automated because you don’t have control over what that vendor does on their end.

Batch-type processes are also difficult to automate, and surprisingly, there are many real-world use cases that aren’t automated, even in the most robust industries. In the hotel industry, for instance, the backbones of most reservation systems are 30+ years old. The same is often true for companies in the financial industry, which also still performs batch processes. This scenario is more common in older industries, while newer industries may be able to automate practically everything simply because they don’t perform batch-type work.

Test Early, Test Often

The “test early, test often” mantra is preached repetitively for a reason. You should be testing at the end of each sprint to evaluate new functionality. At 3|SHARE, we prefer the cyclone-testing methodology:

• When the developer has completed the user story, they set up a cyclone-testing session with a QA resource.
• During this session, which is usually done via screen-sharing, the developer explains the user story in detail, including precisely what should be happening at every step. This is QA’s first opportunity to start asking questions and digging deeper.
• The QA resource asks questions, while the developer executes during the session. For example, if there’s a user login requirement that a username can’t exceed 16 characters, the QA may suggest trying to create an 18-character username to evaluate what happens.

There are several benefits to the cyclone-testing approach. The idea is to detect defects as early as possible, ultimately reducing costs. By leveraging cyclone-testing sessions, many bugs are detected and fixed quickly – often within two to three hours, making cyclone testing a huge cost-saver. Additionally, the process gives QA the foundational information to start making decisions about what can and can’t be automated. Then, they can build the automated AEM test suite and identify any manual regression tests that need to be performed.

By stepping back and looking at your testing requirements from an overall project perspective, you can better plan, not only in the timing sense but also for staffing. If you know that you’ll need to conduct regression testing several times throughout the year, for instance, and those regression tests are largely manual, you might decide to hire additional resources to manage that process.

Stuck in Testing Hell? 3|SHARE Can Help

At 3|SHARE, our entire focus is on Adobe technologies, and we’re the most knowledgable partner in the industry as a result. In fact, 46% of our experts are AEM certified. AEM is a specialized solution requiring a unique skill set, and 3|SHARE’s experts are continuing to build on those skills every day.

Get in touch with us today to discuss a QA review and learn more about how our in-house experts can help you save time and money with a better approach to testing and automatic AEM testing.