What’s New In Nosto: A/B Testing and Optimization

Retailers who are open to continuously creating and testing new things, and reinventing themselves, are the ones that tend to see the most success. In the context of continuous website optimization, facilitating changes justifiably requires some kind of testing tool to validate said changes.

Each test leads to new learnings, which fuels more ideas that can then be tested. This is the type of behavior that fosters a positive cycle of continuous improvement and growth.

The Challenges of A/B Testing —
and How Nosto is Helping Solve Them

Website testing and optimization can be a challenging task for many reasons:

Testing tools can be costly and/or difficult to use, which are two huge roadblockers for fast-growing businesses that typically don’t have the resources, knowledge, or time to dedicate purely to testing.
Traditional testing solutions typically don’t provide the sufficient tools or reporting needed for commerce professionals to fully understand how each variable affects consumer behavior. They lack the context retailers need to identify the underlying cause of changes in behavior patterns (i.e. identifying a sales uplift that is caused by the surfacing of a particular product range that a certain segment of customers has an affinity for) — context that would allow them to adjust their creative, merchandising or buy-in strategies in the proper direction.
Because of these complexities, a retailer is likely to de-prioritize – or even ignore – testing, which closes them off to a wealth of valuable insights and growth opportunities.

To solve for these challenges, we sought to make testing faster, more accessible and less laborious for retailers—without compromising the effectiveness that seasoned statisticians require.

By adding A/B Testing and Optimization to our product offering, we believe retailers can squeeze even more value out of personalization with Nosto. To differentiate our offering, we’ve also implemented advanced, retail-specific reporting capabilities that provide insight into how these tests impact a number of critical KPIs. Without further ado, let’s explore what retailers can now achieve with our latest release!

Merchandising Insights and Reporting for Commerce

The core component of Nosto’s testing solution is a segment-based reporting tool. This tool surfaces details about the behavior of different customer groups (known as segments), as well as compares each group’s behavior across test variants for deeper context.

For example, although a test of two banner variations may not indicate a clear winner within your general customer pool, one key customer segment (such as first time visitors) might indicate that they are much more likely to click one of the variations over the other. This insight helps fuel optimization for that specific segment more naturally—an advantage where regular testing solutions tend to fall short.

Merchandising Insights also gives retailers the capability to dive into brand and product purchase data based on test variations. For example, if visitors see a promotion for a particular brand, such as Nike, the assumption is that those visitors end up buying more products from that brand. Unlike typical testing tools, Merchandising Insights reveals valuable information beyond standard KPIs such as conversion rate or average order value: it reveals which specific products and brands were purchased following the behavior customers display when they are presented with one of the variations being tested. Therefore, merchants no longer have to assume that customers who see a branded promotion are buying products from the correlating brand because they can simply test that they are.

Continuous Optimization

Continuous Optimization is a feature that can be enabled for any test: It removes the need to monitor tests daily and reduces the risks related to poorly performing test variants.

This is achieved through the Continuous Optimization tool, automatically allocating more traffic to the highest performing variation of the test in real-time. With this tool, retailers are able to test even the most unconventional of variations when trying out new ideas without having to worry about losing too much revenue opportunity because of it. For example, if you have four different versions of a hero banner on the home page, and one of those four is a drastic departure from your typical creative, Continuous Optimization ensures that traffic will not be allocated to the risky banner if it is concluded that the banner is performing poorly early on.

In the context of personalization, this capability allows retailers to drill down on which variations of certain site elements speak to each of their customer segments without having to limit their testing activity out of fear of losing revenue.

Campaign and Experience Testing

The actual testing process, itself, is as simple as running a subject line test, and getting started can take a matter of seconds. In terms of methodologies, the tool supports most major testing methodologies; from A/B testing to A/B/n split testing and, of course, multivariate testing. Within the tool, tests are broken into two categories: Campaign Tests and Experience Tests.

Campaign tests are, by their nature, tactical tests where two variations of one element, which can be either a product recommendation or personalized content, are tested against each other. For example: Which product recommendation variation achieves the most clicks on a specific product page? These are implemented simply to find the optimal variation of each element for each segment.

Experience Tests, on the other hand, are when a merchant compares two or more variations of multiple different elements against each other to see how interactions across different elements and pages influence each other. These tests are more strategic in that the individual campaign tests within the experience will all need to inform each other for optimization of each possible journey consumers can take. This layer is what elevates the solution from an optimization tool to a personalization tool.

By being able to learn both which variations of specific elements inspire a segment to act and how certain series of variations inspire that same segment to continue acting, merchants are able to take customers on a highly personalized journey by optimizing which variation of each element they show them along the way.