How to run AB tests on VTEX e-commerce stores

AB testing is critical for optimizing e-commerce performance by comparing different versions of pages or features to see which performs better. Running AB tests on VTEX isn’t a new need: the platform has had a native split test tool for years, mainly to compare the performance between its versions (VTEX Legacy, VTEX IO, and VTEX Fast Store). But since VTEX officially deprecated it in 2023, brands have had to rethink their experimentation stack.

In this post, we’ll break down how the old system worked, the issues that led to its deprecation, the alternatives available today, and how to run fast, reliable, and scalable tests on VTEX using Croct.

How the deprecated VTEX’s native split test worked

VTEX’s native A/B tests were powered by the VTEX AB Tester app. The experiment setup and management were 100% dependent on developers, since it was done through VTEX CLI, as this official guide shows. The concept was simple:

1. You created a workspace with your variant changes (UI, layout, or functionality).
2. The app would split incoming traffic between the master (control) and your production workspace (variant).
3. Users were bucketed and remained in their assigned version for the test duration.
4. Results (sessions, conversions, revenue) were displayed in the Admin panel.

Tests had to be manually finished in VTEX Admin, and data was kept for 30 days, after which it was purged. The winning workspace could be promoted to master, making the changes live for all users.

Besides the high reliance on developers, there were known limitations. Some examples include miscalculated metrics (sessions and conversions sometimes report incorrect values) and inconsistent request routing (users may be shown different versions across requests).

For these and other reasons, VTEX deprecated the tool in 2023, leaving merchants to seek alternative solutions from third parties.

Third-party tools available for AB testing on VTEX

With the deprecation of native AB testing, most VTEX stores now rely on third-party tools. Today, most merchants run tests on VTEX via:

1. Client-side powered by visual editor tools like VWO, Convert, and Adobe Target.
2. Open-source and developer-friendly tools like GrowthBook and FlagSmith.
3. Headless, API-driven personalization and testing platforms like Optimizely and Croct.

Since most e-commerce stores are highly dependent on great performance and fast page loading time to maintain high conversion rates, integrating dynamic content on the server-side is the most recommended option, which leaves us with the second and third alternatives.

The problem with client-side powered by visual editor tools

If you’ve used a visual editor from a client-side testing platform, you're probably familiar with the flicker effect: the original content flashes before the variant loads. This happens because the DOM is being rewritten after the page renders, which:

• Hurts performance and lowers Core Web Vitals scores.
• Risks SEO penalties for inconsistent content.
• Can be blocked by ad-blockers.

The over-reliance on developers for feature flag tools

Developer-friendly tools like GrowthBook and FlagSmith are among the top options for ensuring excellent performance. Developers widely use feature flag tools to roll out and test changes gradually, making them useful for backend-controlled experimentation.

However, although it doesn't cause flickering or page speed issues, it:

• Requires a developer in every step of the process
• Can add complexity to the codebase
• Some tools can be expensive
• Usually doesn’t provide analytics

The flexibility of component CMS with dynamic content

Optimizely and Croct are among the top recommendations for VTEX because they’re built to be headless, fast, and integrate deeply without interfering with the platform’s rendering.

Through a component CMS, these tools offer AB testing and personalization directly within the content management system. They integrate seamlessly with VTEX and enable teams to manage experiments without incurring engineering overhead.

Some of the pros include:

• No flickering or performance issues
• Can be integrated with content workflows
• Only requires a one-time implementation
• Allows non-technical users to set up experiments
• Has integrations with the design system and helps ensure brand guidelines

How to use Croct with VTEX

Croct works as a headless API that delivers personalized and test content server-side or during hydration, avoiding flicker and preserving performance. You can integrate it into VTEX in two main ways, depending on the version of VTEX on which your e-commerce is built.

VTEX Legacy and VTEX Fast Store

For the oldest and newest VTEX versions, you should use Croct’s SDKs to directly track events and fetch dynamic content. If you use Next.js, this template can help you get up and running in less than a minute with just one command:

npx croct@latest use croct://starter/nextjs

VTEX IO

Since VTEX doesn't allow direct customization of its components, for VTEX IO, you'll need to duplicate your components and create custom versions of them.

Follow this VTEX guide or check this example for developing custom storefront components, and fetch your content from Croct’s API instead of hardcoding content or fetching it from VTEX CMS. This way, the content is already personalized when VTEX renders it.

Need any help?

To optimize your experiments and personalized journeys for better performance with Croct, create a free account and get started with this template or check the complete documentation.

Our team is highly experienced with VTEX e-commerce websites, so feel free to schedule a technical call with us. We'll ensure you have everything you need to integrate in just a few hours.