Can you trust your marketing numbers?
Often not. Across 663 Facebook experiments, even the best non-experimental method overstated the effect of ads about five-fold, and eBay found its ads on its own brand name had no measurable short-term benefit. Ad effects are small next to the normal ups and downs of sales, so a single business can rarely prove what its ads return. Simple floors like promo codes help, but they undercount: most of the extra spending from emailed offers came without the code being used.
What to do
- Treat platform-reported conversions as estimates, not facts.
- Use promo codes and “how did you hear about us” as a floor, knowing they undercount.
- When you can, hold back a small group from a campaign and compare.
- Before paying for ads on your own name, try switching them off for a while.
What the research says
- Peer-reviewed or official
Consumer Heterogeneity and Paid Search Effectiveness
Blake, Nosko & Tadelis, Econometrica 2015
True returns from paid search were a fraction of the observational estimates. Brand-keyword ads had no measurable short-term benefit, and average returns on non-brand ads were negative.
What it means for you: Before paying for ads on your own business name, test switching them off: many customers would have come anyway.
Keep in mind: eBay, a very well-known brand. New and occasional customers were influenced by ads.
Read the sourceMethod and quote
A series of large-scale randomized field experiments at eBay.
“returns from paid search are a fraction of non-experimental estimates. As an extreme case, we show that brand keyword ads have no measurable short-term benefits.”
- Peer-reviewed or official
A Comparison of Approaches to Advertising Measurement
Gordon, Zettelmeyer, Bhargava & Chapsky, Marketing Science 2019
Standard observational methods often fail to recover the true effect of Facebook ads, even with rich data.
What it means for you: Platform-reported conversions are model estimates; holdout tests and promo-code counts are more trustworthy.
Keep in mind: US campaigns, with data from inside the ad platform.
Read the sourceMethod and quote
15 randomized lift studies at Facebook, compared against several observational models.
“Using data from 15 U.S. advertising experiments at Facebook comprising 500 million user-experiment observations and 1.6 billion ad impressions… The observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables.”
- Peer-reviewed or official
Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement
Gordon, Moakler & Zettelmeyer, Marketing Science 2023
Across 663 Facebook experiments, the best non-experimental method estimated a median lower-funnel lift of 24% when the true lift was 5%, roughly five times too high.
What it means for you: Even the best attribution models overstated ad effects about five-fold. Be sceptical of dashboard returns.
Keep in mind: Facebook campaigns, measured with richer data than any advertiser has.
Read the sourceMethod and quote
663 large-scale RCTs at Facebook with more than 5,000 user-level features. DML and propensity matching compared against experiments.
“The median RCT lifts are 29%, 18%, and 5% for the upper, middle, and lower funnel outcomes, respectively. Using DML (SPSM), the median lift by funnel is 83% (173%), 58% (176%), and 24% (64%), respectively, indicating significant relative measurement errors.”
- Peer-reviewed or official
The Unfavorable Economics of Measuring the Returns to Advertising
Lewis & Rao, QJE 2015
Across 25 large field experiments, the median confidence interval on ROI was over 100 percentage points wide. Ad effects are tiny relative to sales noise.
What it means for you: A single business can rarely prove ad returns statistically. Use simple floors such as promo codes and “how did you hear about us”.
Keep in mind: Large advertisers; for small businesses the noise is even worse.
Read the sourceMethod and quote
25 field experiments with major US retailers and brokerages, about $2.8M in ad spend.
“The median confidence interval on return on investment is over 100 percentage points wide.”
- Peer-reviewed or official
Do Targeted Discount Offers Serve as Advertising?
Sahni, Zou & Chintagunta, Management Science 2017
Emailed discount offers raised spending by $3.03 (+37.2%) during the offer window. 90% of the gain did not come through redemption of the offer, and the effect was larger for people who had not bought in the previous year.
What it means for you: Win-back offers work best on lapsed customers, and counting redeemed codes understates their effect.
Keep in mind: One US online platform, measured over a short promotion window.
Read the sourceMethod and quote
70 randomized email experiments at a large online ticket-resale platform, comparing recipients with randomized non-recipients.
“We find that the offers cause the average expenditure to increase significantly, by $3.03 (a 37.2% increase) during the promotion window. However, the redemption rate of these offers is low. Importantly, 90% of these gains are not through redemption of the offers… the effect of the offers is significantly higher among individuals who did not transact on the platform in the year before the offer was given.”
Sources checked as of October 25, 2026. Links go to the original publication.