Meridian requires passing distributions for ROI calibration. Although setting custom priors using results from previous experiments is a sound approach, there are many nuances to consider before proceeding. For example:
The timing of the experiment in relation to the MMM time window: If the experiment was conducted either before or after the MMM time window, the results might not be directly applicable.
The duration of the experiment: An experiment of short duration might not effectively capture the long-term effects of the marketing effectiveness.
The complexity of the experiment: If the experiment involved a mixture of channels, the results might not provide clear insights into the performance of individual channels.
Estimand differences: The estimands used in experiments can differ from those used in the MMM. For example, the MMM counterfactual is zero spend, whereas some experiments might have a different counterfactual, such as reduced spend.
Population differences: The population targeted in the experiment might not be the same as the population considered in the MMM.
We recommend setting the custom priors based on your belief in the effectiveness of a channel. A prior belief can be informed by many things, including experiments or other reliable analyses. Use the strength in that prior belief to inform the standard deviation of the prior:
If you have a strong belief in the effectiveness of a channel, you can apply an adjustment factor to the standard deviation of the prior to reflect your confidence. For example, suppose you have conducted several experiments for a particular channel and all the experiments yielded similar ROI point estimates, or you have historical data from previous MMM analyses that support the effectiveness of this channel. In this case, you could set a smaller standard deviation for the prior so that the distribution won't vary widely. This tighter distribution indicates your strong confidence in the experimental results.
Conversely, the experiment might not necessarily translate to the MMM, considering some of the nuances listed earlier. In this case, you might choose to apply an adjustment factor to standard deviation of the prior distribution. For example, you could set a larger standard deviation for the prior, depending on your level of skepticism.
You should consider using the roi_calibration_period argument in
ModelSpec. For more information, see
Set the ROI calibration period.
When setting the prior, the LogNormal distribution is a common
one to use. The following sample code can be used to transform experiment's
mean and standard error to the LogNormal prior
distribution:
import numpy as np
def estimate_lognormal_dist(mean, std):
"""Reparameterization of lognormal distribution in terms of its mean and std."""
mu_log = np.log(mean) - 0.5 * np.log((std/mean)**2 + 1)
std_log = np.sqrt(np.log((std/mean)**2 + 1))
return [mu_log, std_log]
However, if the results from previous experiments are near zero, you should
consider whether your prior beliefs are accurately represented by a
non-negative distribution, such as the LogNormal distribution. We
highly recommend plotting the prior distribution to confirm it matches
your prior intuitions before proceeding with the analysis. The following
sample code shows how to get reparameterized LogNormal
parameters, define the distribution, and draw samples from it.
import tensorflow as tf
import tensorflow_probability as tfp
# Get reparameterized LogNormal distribution parameters
mu_log, std_log = estimate_lognormal_dist(mean, std)
mu_log = tf.convert_to_tensor(mu_log, dtype=tf.float32)
std_log = tf.convert_to_tensor(std_log, dtype=tf.float32)
# Define the LogNormal distribution
lognormal_dist = tfp.distributions.LogNormal(mu_log, std_log)
# Draw 10,000 samples
lognormal_samples = lognormal_dist.sample(10000).numpy()