During a recent interview with an analyst working for a big multi-national retail store chain, I asked:
How do you check if promotion of a particular product had cannibalization effect on future Sales of the product or on other similar products?
The response I got was
By simply comparing Pre promotion Sales to Post promotion Sales.
This answer might fetch a good score quite a few Organizations and you might be surprised to see the percentage of analysts who give this answer. However, in any analytically evolved Organization, it will be considered incomplete, if not wrong!
In case, you think that this answer is right, think again!
This simplistic analysis works in steady state when there is no impact due to changes is external factors or factors you can not control for. However, this is hardly the case in any practical application. In today’s world, external factors can change overnight and the cost of making wrong conclusions can be hefty.
In this particular example, imagine what would happen if one of your competitor launches a better promotion for same product which causes your Sales to shrink by 50% in case of no promotion and 25% in case you are also running promotion. If you just do a simple pre vs. post comparison, you may end up concluding that the promotion caused your Sales to drop by 25%, while the reality is that it actually increased it by 50% (25% absolute increase on a base of 50%).
[stextbox id=”section”]Challenges with simple Pre vs. Post analysis?[/stextbox]
There are bunch of practical challenges you face in doing a simple Pre vs. Post comparison. How do you account for regulatory changes happening year on year (making it difficult to compare against base of previous years)? Or for Seasonal products (e.g. Ice-cream), how do adjust for sudden heat waves or temperature drops? What do you do when the entire Industry is expanding or shrinking?
In each of these scenarios, a Pre vs. Post analysis might lead you to a wrong conclusion. So, how do you arrive at effectiveness of Sales promotion?
A simple answer to this is performing tests or experiments in order to size the impact more accurately.
While testing is a subject which deserves multiple posts, I plan to cover simplest form of testing in this post. The simplest form of testing is popularly known as A / B testing or Champion / Challenger testing. This is your solution to the kinds of problems mentioned above in this post.
[stextbox id=”section”]So, what is A / B testing?[/stextbox]
Conceptually, it is a simple solution, where you change only the parameter / factor you want to test and run two alternates in parallel to find out which one works better. For example, to answer my initial question, we will find 2 sets of stores similar to each other, run promotion in one set and don’t run promotion in other set of stores. Once we run the test for required duration (to achieve statistical significance), we compare Pre vs. Post for both the sets of stores and find out if promotion has actually benefited the store or not?
The reading you get now will adjust itself for all external changes as long as the 2 sets of stores are similar. Isn’t it elegant?
[stextbox id=”section”]I understand A / B testing, where else can I use it?[/stextbox]
Applications of testing are limited by your thinking. It can help you in situations where coming out with a neat analytical solution is difficult or impossible. Following are just a few such examples:
- When you are in a situation where there is no right or wrong answer. How do you decide whether application form should be one page or multiple page? Should you suggest a particular credit card to your customers or let them select it themselves? Should you call a customer with 30 day default once a day or twice a day? All these can be run as tests to find out what works and what does not.
- When you want to measure impact of making a change. How does your application conversion change with change in layout of website? How does a new credit card collection strategy change the collection rate or default rate?
[stextbox id=”section”]It sounds simpler than it actually is[/stextbox]
In case you have started thinking that this sounds easy, let me warn you. It sounds easier than it actually is. Following are some precautions you need to keep in mind while doing these tests / experiments:
- The two scenarios you test should be different only on the parameter you want to test. For example, if you want to test Subject line of a communication to your customers, you need to make sure that only the subject line changes. Creative, content, delivery address, delivery timing across the two sets of population should remain same. If any of these change, then it might become difficult to understand which factor is responsible for what change. If you want to test multiple factors, i’ll discuss this in one of the future posts.
- The two sets of population should be chosen randomly. There should be no bias in the samples chosen. Normally it is a good idea to check distributions on 2 – 3 key parameters before rolling out the tests. If the test is to check whether making a particular offer to customers increases their chances of charge-offs, it makes sense to check whether the distribution of credit scores and vintage of the customers in two sets is similar. Again, if you don’t check for these distributions, you might end up drawing wrong conclusions.
- Make sure you have entire control on how are the two sets being treated. This might become a huge challenge especially if you are dealing with a cross-functional team. In one of the tests designed by a friend, he wanted to size the impact of a tele-calling intervention. So we designed a no-call test where a small group of customers were not called when their collection was due. The business unit responsible for collections took a decision to send additional communication to these customers with an intention to maximize collection without informing the analysts. Result – the test ended up under-estimating the impact of tele-calling and might have led us to believe that tele-calling has limited impact which was not the case!
- Monitor / Follow the test intensely for first few days. A lot of Organization may not have capability / systems to run multiple options in parallel. These arrangements are typically made outside the system or require a lot of change. Once the test goes live, you need to make sure it is actually what you intended and not otherwise. A healthy practice here is to include people with in project group in various sets and see if things are as per expectation.
[stextbox id=”section”]What next?[/stextbox]
Hopefully this post gives you a good overview of limitations of simple pre vs. post analysis and advantages of performing tests. Some time in future, I’ll cover multi-variate tests and Design of experiments. I’ll also share the best practices in designing a test.
For now, do let me know if you think you can apply this elegant concept in your respective areas and how can you benefit from it.
Image 2 credit: Maxymiser.com
If you like what you just read & want to continue your analytics learning, subscribe to our emails or like our facebook page
Kunal is a post graduate from IIT Bombay in Aerospace Engineering. He has spent more than 10 years in field of Data Science. His work experience ranges from mature markets like UK to a developing market like India. During this period he has lead teams of various sizes and has worked on various tools like SAS, SPSS, Qlikview, R, Python and Matlab.
