Why Geo Holdout Tests Fail to Measure True Incrementality скачать в хорошем качестве

Why Geo Holdout Tests Fail to Measure True Incrementality

In this video you'll learn why geo holdout tests are being misused in marketing measurement, how the margin of error can make your ROAS vary between 0.5 and 5.5, and when these tests are actually appropriate to use. Constantine explains the critical difference between true AB testing and geo holdout methodologies, revealing why most companies are getting incorrect results despite spending significant money on these tests. #MarketingAnalytics #Incrementality #MarketingMeasurement #MarketingROI #DigitalMarketing #MarketingStrategy #DataDrivenMarketing #AdAttribution #MarketingMetrics #GeoHoldout Full episode: https://humansofmartech.com/2025/04/2... Marketing’s obsession with scientific-sounding measurement has birthed a monster: the geo holdout test. This expensive experiment promises to reveal your marketing’s true impact through the magic of “controlled testing.” Too bad it’s built on statistical quicksand that swallows marketing budgets while spitting out garbage insights. Constantine rips apart the foundational myth that geo holdout tests equal proper A/B testing. Real A/B tests randomize at the individual level – you take your entire audience, randomly split it, then randomly expose one group to your campaign. What happens in geo testing? Something wildly different: You cherry-pick regions you think behave similarly (goodbye, randomization) You construct a “synthetic control” (corporate-speak for “we made up a comparison group”) You pray external factors don’t torpedo your comparison “The primary quality of the proper A/B test is randomization. In geo holdout tests, we do not do randomization. We decide which states or which cities behave similarly. And then we build synthetic control.” In the messy real world, these tests crumble faster than cheap cookies. Constantine’s seen clients spectacularly botch implementation: “They have Facebook ads spending $50K a month and they shut down 50% of the states, but still the rest of the budget is automatically relocated to the control group.” You just accidentally pumped more money into your control regions! Your test is immediately worthless, but the pretty dashboard won’t tell you that. The statistical confidence intervals make these tests practically useless. Constantine breaks down the brutal math: “You’ve measured your incrementality as 5%, but usually for 5% incrementality with 5% minimal detectable effect, margin error is plus minus 4%. So your actual incrementality is between 1% to 9%.” When translated to ROI, you’ve proven your campaign is… somewhere between hemorrhaging money and wildly profitable. Congratulations on your $50K experiment that told you absolutely nothing actionable! We’re living in what Constantine calls “the dark era of marketing measurement.” The cookie apocalypse and tracking prevention have created fertile ground for measurement snake oil. “During such times when this is chaos, there are a lot of experts that appear on the market and they try to sell you like gold. While this gold is not gold, but no one knows.” The technical complexity creates perfect cover – CMOs lack the statistical background to spot the flaws, while data scientists often lack business context to see why their elegant models fail in practice.