Challenges with automating Salesforce E2E testing

Q: What’s worse, doing your taxes or waiting in line at the DMV?

A: Testing Salesforce.

Your grumbling testers aren’t imagining things. If a million things can go wrong when you test Salesforce manually, you increase the number of things that can go wrong when you automate by an order of magnitude.

But what makes Salesforce testing difficult isn't specific technical aspects, such as the shadow DOM, SOQL, or any of the vast number of customizations Salesforce affords. While those are certainly the contributing factors/elements, it's the way those and other aspects of the Salesforce tech stack affect planning, building, running, and maintaining the automated tests that we're going to discuss today.

Planning

The first problem that black-box testers encounter is the sheer number of test cases that need to be identified and created. Salesforce apps are dynamic. What a user (or test) sees on the screen is affected by all of the controls interacting with all of the profile types interacting with the various record types. The number of combinations (test cases) can quickly spiral into the thousands for even basic applications and integrations.

Even more compatibility tests than traditional apps

Most testers are familiar with three forms of compatibility testing: third-party integration testing, cross-browser testing, and mobile testing. Salesforce comes with a few additional integration concerns:

• API Compatibility: Ensures APIs (REST, SOAP, Bulk, Streaming) continue to work as your application evolves.

• Lightning vs. Classic: Though legacy Classic components are slowly being phased out as applications refactor, there are still plenty of them around, usually sitting side by side with Lightning components, and those integration points need consistent compatibility regression.

ISVs, those who are developing plugins for Salesforce, have an even larger compatibility testing surface and need to tack on these additional compatibility test cases:

• Salesforce Clouds (e.g., Marketing, Sales, Service)

• Salesforce Editions (e.g., Essentials, Professional, Enterprise, Unlimited)
• AppExchange security review requirements
• Unmanaged and managed packages to ensure their application behaves correctly in both scenarios, accounting for differences in versioning, upgrades, and namespace management.

Susceptibility to coverage gaps

Testers typically plan automation for any app by first manually executing each test step-by-step to capture all the variants of a given flow and ensure your suite has optimal coverage. In Salesforce, manual testing gets tedious quickly when accounting for all the variations and permutations, and tedium increases the risk of missed coverage.

While not a technical challenge per se, another source of missed coverage happens when developers use features like Flow Builder and Process Builder, which hide business logic deep within Salesforce’s underlying structure which obscures it from testers.

Creating

Building automated tests for Salesforce apps is uniquely challenging for three reasons: locators, framework limitations, and test data management.

You may have no choice but to use suboptimal locators

The optimal locator in an automated test is a custom (and static) HTML element attribute, a “test ID,” so the test framework can always find what it is looking for regardless of changes to the DOM. Unfortunately, Salesforce dynamically generates the values for most IDs, so testers need to resort to other methods.

The second-best approach is to use CSS selector syntax. But, the significantly more complex DOM on any given Salesforce page challenges the limitations of CSS selectors.

This leaves us with the dreaded XPath option. XPaths are slower than CSS selectors, harder to read, and more brittle because they need to be tightly coupled to the structure of DOM. Any minor change to a page can cause an XPath to break. As you’ll read later, XPath’s damage isn’t limited to test creation

Some frameworks have limited support for asynchronous elements and shadow DOMs

Salesforce pages frequently incorporate dynamic and asynchronous elements. That’s not exciting. It’s standard practice these days, but not everyone has the luxury of updating their crusty, old test framework on a whim.

Older versions of some frameworks over-complicate dynamic element support (we’re looking at you, Selenium) or the shadow DOM (let alone Salesforce’s custom implementation). However, some frameworks, such as Playwright, make interacting with shadow DOMs and dynamic elements practically effortless. Switching frameworks is no small effort, but might be worth the upgrade if you’re heavily reliant on Salesforce as part of your tech stack.

Forced data integrity leads to suboptimal test data management strategies

As with everything else, Salesforce makes an already difficult task, test data management, even more challenging. Their complex data model forces data integrity by running validation rules before all modifications, so testers can’t simply query the database for records and update them to match the requirements of the test. In Salesforce, such updates would either be prohibited or cause unintended side effects. Instead, most automators use subsetted, sanitized data regularly migrated from a production snapshot.

The problem with relying on production data for testing (apart from the obvious security concerns) is that you can’t predict whether the data you need to make a particular test pass will be available in any given snapshot.

Any approach you use to prepare production data snapshots for use in testing comes with a drawback:

1. You can modify exported production data before it is imported to the test environment, but you risk type mismatches or foreign key constraints.
2. You can insert new data or modify existing data after the import but risk triggering a bunch of Salesforce rules and workflows that slow your application to crawl.
3. If you can guarantee that your application can support the transactions you need for testing without impacting your customers, you can run synthetic transactions in production just before exporting.
4. You can design a database that can be deployed entirely from scratch if you want to invest a huge amount of time and energy.

Running

The main issue with running automated e2e tests on Salesforce is the performance of the system under test (SUT) and the tests themselves.

The heavy DOM impacts locator performance

As we said in the previous section, Salesforce frequently leaves us no choice but to write our locators in XPath. As if that wasn’t a big enough hit to our test performance, many Salesforce pages have a heavy DOM structure, so locators are often deeply nested. The more paths XPath has to traverse, the longer the element takes to find, the slower the test takes to run, the greater the risk of flakes, and the later the new feature goes out.

Under-provisioned environments slow tests down

Most teams use Salesforce sandboxes and scratch orgs for testing. These environments are intentionally under-provisioned, which means that applications run slower, which means your tests take longer to complete.

API and Governor’s limits constrain test execution

Governor’s limits are restrictions on API calls, data storage, processing power, and other things that Salesforce uses to protect itself and its customers from the side effects of multi-tenant architecture when using Apex code. Salesforce also implements a separate set of API limits, which will throttle any tests that exceed them, likely causing the test to flake.

Multi-tenant activity can impact the performance of the SUT

While governor’s limits generally prevent Salesforce’s customers from competing for resources amongst themselves when executing Apex code, understanding Salesforce is a world unto itself. Although remote, another org’s activity can degrade your application’s performance, causing delays and flakes. Particularly in Salesforce’s intentionally under-provisioned sandbox environments and scratch orgs.

Test data can degrade performance, causing your app to slow and your tests to fail

Hopefully, it’s obvious that if you create a bunch of data on under-provisioned sandboxes or scratch orgs, your Salesforce application will slow down. There are other ways test data can negatively affect your test suite:

• If your test data has master-detail or lookup relationships, that data will take longer than usual to get into a state where your test can start without failing.
• SUTs with extensive sharing rules and security settings also slow down when dealing with test data.
• Inefficient test data design will do the same thing.

Unfortunately, due to the complexities of the Salesforce data model, it’s not always easy to design test data efficiently.

Maintenance

Because a Salesforce app has so many more moving parts than a standard web app, maintaining broken tests is absolutely Sisyphean.

Lack of control over updates can cause maintenance spikes

Salesforce itself only has three major updates a year and does an excellent job of publicizing them but they’ve adopted a continuous deployment model, meaning their developers can release bug fixes, security updates, and performance enhancements at any time.

But the bigger risk is plugins from the AppExchange. The ISVs who develop those applications can (and do) ship updates at any time with very little advanced notice (if any). These updates can force your testers to drop everything else, potentially even a release you have in progress.

Lack of control over the DOM increases maintenance

Remember how we told you that Salesforce gives automators limited locator choices due to its complex DOM and restrictions on adding custom elements to HTML attributes? Well, that also adds to the maintenance burden. XPath is a great tool, but it’s a drag to maintain convoluted XPath locators. Experienced testers on standard web apps know to use XPath as a last resort (for reasons mentioned in the section above). Unfortunately, when working with Salesforce apps, XPath is a first line of defense.

Now you know

We hope this article can help you and your team communicate the challenges of automating Salesforce testing.

At QA Wolf, we specialize in building automated black-box tests for any application stack, and Salesforce apps are in our wheelhouse. What’s more, our offering spans the phases of test automation, so as long as we are running your tests, you can let us focus on creating, running, and maintaining your tests while you focus on delivery. Give us a shout.

‍