Consider the following generalizations about marketing mix modeling (MMM) as a causal inference methodology:
MMM is a causal inference tool for estimating the impact your advertising budget level and allocation have on KPI. MMM-derived insights such as ROI and response curves have a clear causal interpretation, and the modeling methodology must be appropriate for this type of analysis.
The causal inference framework has important benefits, which are also critical components of any valid and interpretable MMM analysis:
ROI and other causal estimands are clearly defined using potential outcomes notation, which is both intuitive and mathematically rigorous.
Necessary assumptions can be determined and made transparent. All models require assumptions to provide valid estimates of the causal estimands.
It is common knowledge that randomized experiments are considered the ideal way to estimate causal effects. MMM, however, is an example of causal inference from observational data.
MMM has important advantages over experiments:
In the case of advertising, many experimental designs require individual user-level data that does not meet modern privacy standards. MMM uses observational data at an aggregate level that is privacy safe.
Experiments are often difficult to run due to cost and practicality. Observational data, on the other hand, is readily obtainable.
Experiments are typically designed to estimate one specific quantity. In advertising, for example, a geo experiment might be designed to estimate the ROAS of a specific channel such as TV. A causal inference model, such as MMM, can provide many insights such as ROI for every media channel, full response curves, and budget allocation without needing a complex and rigorous experimental design that might be impractical.
Testable and untestable assumptions
Because MMM is based on observational data, it requires statistical assumptions that are not necessary for most experiments. These assumptions can be categorized as untestable and testable.
Why do these assumptions matter from a practical standpoint? Multiple models can have good fit and predictive power yet provide different ROI and optimization results, therefore making it difficult to choose the best model.
Untestable assumptions
A condition known as conditional exchangeability is the main untestable assumption required for an MMM regression model to provide accurate causal inference results. This condition is untestable because there is no statistical way to determine its validity purely from observational data.
Generally, conditional exchangeability means that the control variable set both:
Includes all confounding variables, which are variables that causally affect both media execution and KPI, and
Excludes any mediator variables, which are variables that lie in the causal pathway between media and KPI
A causal graph can be used to justify the conditional exchangeability assumption. The causal graph must be constructed based on expert domain knowledge, as there is no statistical test to determine the correct graph structure purely from observational data.
In reality, the exchangeability assumption is never perfectly met. The classic principle applies that "all models are wrong but some are useful".
Testable assumptions
Testable assumptions include anything related to the mathematical structure of the model. Consider:
How are media effects represented in the model, including lagged effects and diminishing returns?
How are control variables modeled? Are nonlinear transformations required?
How are trend and seasonality modeled?
Testable assumptions can be evaluated to a certain extent by goodness of fit metrics, including prediction metrics such as out-of-sample R-squared. However:
Goodness of fit metrics don't give a complete picture of how good a model is for causal inference, and it is likely that the best model for causal inference is different from the best model for prediction.
The more candidate models you are comparing, the higher the risk of overfitting. For example, the best model is not the one that appears to have the best out-of-sample fit.
There is no threshold for R-squared or other metrics that makes a model good or bad. A model with 99% out-of-sample R-squared can still be a poor model for causal inference.
Conclusions
There is no absolute best solution to MMM, which follows from the fundamental principles of causal inference from observational data. We recommend that all MMM practitioners think critically about MMM within a causal inference framework, regardless of whether you use Meridian or any other solution. The mission of Meridian is to provide you with the utmost clarity about what your MMM is, how it works, and how you should interpret your results.