Glad to see the use of regression, bastion, logistic, multiple linear regression being used reverently for predicting customer behaviour. Pauly talked about affecting customer redemption from coupons, whether a customer goes to a store and how much they spend when they're in the store as this big kind of customer behavior that marketers are trying to influence using insights from the shoppers card data. Are there any other kinds of customer behaviour that marketers are trying to influence? Absolutely. I would say there's multiple different types of behaviors that retailers, marketers, are trying to understand and influence. If we go back to the transaction log data, and while we have a lot of. Individual data on customers, it's when you begin to group customers together, you can truly begin to understand that most important behaviour that we're trying to influence, and that's loyalty. And so, while we don't necessarily have everything a household may buy outside the retailer of interest, with very simple math and segmentation you can very quickly begin to paint a picture of individual households in terms of how frequently they're coming into your stores. You can begin to paint a picture in terms of their life style. When you sign up for a loyalty card, you don't tell us if you're having a baby. You don't tell us what ethnic group you belong to. You don't tell us how price sensitive you are. But we're able to use the information through algorithms, begin to distill those facts. So, if you're buying a lot of Hispanic oriented items, we know that you're most likely Hispanic. If you're beginning to go down the baby cure aisle and you've never been there before or we know that you're most likely beginning to expect a baby. And we talked a little bit about price sensitivity, but all of those are behaviors that need to be studied using sophisticated models or even simple models. Marketers really want to truly understand customers and experiment and try different programs to bring them those segments back into their stories and keep them loyal. And that's perhaps the most important thing that a marketer does is truly understand their customers by looking at the data through a different lens than they traditionally had been able to before the advent of the loyalty part. >> Bob, let me talk to you price elasticity. You mentioned that a few times now, and you've talked about how important is it to measure the effect of price on sales, price elasticity using regression models. And when you're doing things like this, there are lot of factors in the environment that need to be considered too. Things that are easily captured through shop a card data. But what advice would you give to someone trying to build a price elasticity model? On how to choose these external factors to include in the model from the multitude of options that are out there. >> So, that's a very good question, Raj. And determining what variables to include or to exclude in a model. It is complex. I think the best answer I can give is it will come with experience. So if you're first starting out, just know that you're going to get better over time. It's much like riding a bicycle. It takes time to build up that experience, but many different ways to address the problem. The first thing I would do is I would make sure that you would have a correlation matrix table of all the factors that you have at your disposal. And make sure that you're only going to truly include in your model things that are independent and that you're not going to be confusing the model by having factors that are really one and the same. But price elasticity is perhaps the most important measure in marketing. It's what marketers care about the most. It's one of the things that they can pull the lever on in terms of lowering or raising price. It's very easy to overestimate or under estimate price elasticity if you haven't factored in as much variables as you have. So you can underestimate price elasticity by not necessarily including factors such as merchandising and you could be reassigning factors that should have gone to another event or you could be overestimating it for the same reasons as well. Most retailers will have a lot of very good data but they don't necessarily have all the data that you may want. We use a saying in our business that don't make perfect be the enemy of good. So just make sure that you're looking at all available data, looking at that correlation matrix table, and ensuring that you're putting in things that make sense. So for example, if you're modeling a particular SKU, a product, you probably want to not just look at the information you have on that product, but perhaps some of the key competitors of that product. So you properly, to the best of your ability, assign the right price elasticity that you can come up with. And then ultimately, and we typically use a 5 or a 10% hold out sample. The best way to validate that your model is at least doing well is once you've finalized your model to apply it to that hold out sample to see how well it's stacking up. And that hold out sample is perhaps the strongest test that you have to ensure that you don't have any bias, that you haven't left to factor out or included a factor that you shouldn't have included. So it's a complex question, but it will come with experience. >> Paul, thank you for your time. This was great and I appreciate you giving us your time and I'm sure everyone appreciates learning from your real world experience. >> Raj, thank you for having me, it has been my pleasure. I wish everyone to continue learning. Analytics is fun, particularly in the marketing space. And I wish you all the best.
