As businesses aim to adapt and sharpen their skills in the digital-first world, they need new abilities to create business success. As customers interact with businesses in a real-time, always evolving competitive environment, they have to implement decisions with adaptive precision and run algorithmic experimentation at granular scale.
Experimentation is the technique carried out under a controlled environment to discover an unexplored effect or principle; to put forward or to establish a hypothesis, or to demonstrate a known principle.
Hence it is very important to create a culture of experimentation within any organization. Experimentation does imply that ideas bloom but also run the risk of some failure, and companies should be prepared for it. Without failing at some ideas, one will never be able to achieve that single great idea that might take your organization to the next level of growth.
Experimentation is an age-old technique to test a hypothesis. A good example of this is retail catalogs, which adopted an experiment to see if prices ending with $0.99, i.e. $7.99 and $8.99, would generate higher sales. To explore the effectiveness of this experimentation, retailers sent out varying product catalogs and, in some cases, left the prices unchanged, and in others, they changed the pricing either at the beginning or at the end of the catalog with $0.99. To their surprise, they found that having such experimentations resulted in an overall increase in sales.
Lee Hibbett, an Associate Professor of Marketing at Freed-Hardeman University, says it might appear ridiculous to price products one cent short of a dollar, but this trick of pricing has a psychological influence on customers. Hence as per Hibbett, because we read from left to right, the first digit of the price resonates well with us most often; hence the reason customers are more likely to buy a product for $7.99 than the same one for $8. This is famously called psychological pricing or charm pricing.
Retailers also use bundle pricing strategies, wherein organizations sell a set of goods with lower prices than they would have charged if the customer bought all of them independently. Most common examples are BOGO (buy one get one free), or buy two products and get another product at 20% off. Pursuing a bundle pricing strategy allows retailers to increase their profit by giving customers a discount.
To get to the optimum products that can be experimented as part of the bundle pricing strategy, retailers rely upon affinity analysis, a.k.a. market basket analysis. The main idea behind affinity analysis is to achieve insights by identifying which products are frequently purchased together. It helps retailers find patterns between purchases in orders, which can be used as a cross-selling opportunity. It can also be used to ascertain what products can go on discount. It’s needless to emphasize that this can apply to many other use cases, which can help increase sales and customer satisfaction.
Another strategy is anchor pricing, which allows retailers to make a product appear cheaper when it is put alongside another product. It is a technique to keep a buoyant pricing strategy. This is the very reason retailers put their own private labels or their own brands next to the market leader, as most often their own brand is cheaper than major label brands.
Price experiments are a technique used by retailers to gauge the association between demand and price change. Price experiments enable retailers to not only present the best price to their customers but also make sure they are rewarded well from a strategic business standpoint.
Tech companies have embraced the experimentation techniques more often than others. Online experiments are vital for ecommerce companies in the development of their web facing products. Given that they have a large user base, even tiny improvements can have a large impact on their profits. For such organizations online controlled experiments are crucial for assessing the impact of product changes in their businesses. They represent the best scientific design for establishing causal association between changes and their influence on observable user behavior.
This example demonstrates how important it is to assess the potential of fresh ideas. A Bing team member of Microsoft in 2012 suggested changing the way Bing search was displaying advertisement headlines, but his product manager considered it a low priority item. It was not until six months later that another team member launched a simple online controlled experiment of an A/B test to assess the impact of it. To everyone’s surprise, within a few hours it was delivering abnormally high revenue.
Similarly, Google’s 41 shades of blue experimentation demonstrates that small design decisions can have considerable impacts on user engagement, resulting in substantial positive engagement.
Digital marketing plays a pivotal role in generating measurable transactions that can be measured to calculate a return on investment. Amazon found that every 100ms of latency cost them 1% in sales. Similarly, a brokerage firm found that if they are 5ms behind their competition then they could lose $4 million in revenues per millisecond.
Thus, businesses are starting to appreciate how critical it is not only to run as many experiments concurrently — but also as cheaply — as possible. Now digital marketing plays a pivotal role in generating measurable transactions that can be measured to calculate a return on investment.
The other decision is the timeframe an experiment has to run for. In general, it is recommended to run an experiment for one to two weeks. Hence the treatment effect measured for such a time frame is called short-term impact. But there are cases when long-term impact is a lot different than short-term impact. For example, increasing the price of products on an ecommerce platform might increase revenue in the short term, but eventually act against long-term revenue because users might shift to some other site for better price.
Experiments might fail due to various reasons, for example the tactics being inadequate. If the experiment fails, then there could be a problem with the underlying experiment design, infrastructure, data or analysis of the result.
Leading global companies competent in algorithmic customer engagement are now powering digital strategies through their products and have always nurtured experimentation as a de facto standard. These companies have embedded experimentation into their organizational DNA, for all their deployment strategies, be they DevOps or MLOps adapts Blue/Green, Canary, A/B testing technique, while rolling out new feature functionalities.
They leverage multivariate testing to evaluate the impact made by their recommendations, merchandising rules and location of one or more placements on certain page(s) on their customers’ ecommerce platform. Likewise, they are also leveraging Contextual Multi-Armed Bandit to intelligently learn from multiple contexts like browsers, user segments, products and handheld devices, for true real-time decision-making supporting personalized recommendations in a seamless and automated fashion.
(This article has only focused on experimentation and how experiments influence decision making processes in the digital-first world and a few techniques that are adhered to in industry. It is in no way a full-fledged representation of various approaches that are followed across the industry since this is an evolving space.)
The article first published in RetailTouchPoints
