The Benefits of Quasi-Experiments in Marketing [Research Methodology]

Quasi-experimental methods have been widely applied in marketing to explain changes in consumer behavior, firm behavior, and market-level outcomes. The purpose of quasi-experimental methods is to determine the presence of a causal relationship in the absence of experimental variation. A new Journal of Marketing article offers guidance on how to successfully conduct research in marketing with quasi-experiments to understand whether an action causally affects a marketing outcome.

As a vivid example, we describe a quasi-experiment that occurred when eBay shut down all the paid search advertising on Bing during a dispute with Microsoft, but lost little traffic. These quasi-experimental results inspired a follow-up field experiment where eBay randomized suspension of its branded paid search advertising and found results consistent with the quasi-experiment.
 
We begin by establishing various type of quasi-experimental variation at the individual, organizational, and market-levels. In each type, given the lack of an experiment, some individuals, companies, or markets receive an action or policy (i.e., treatment group) and some do not (i.e., control group). For example, some markets are affected by a new policy and some are not. The question is how the markets receiving the treatment would act if they had not received it (i.e., the counterfactual). The unobservability of the counterfactual means assumptions are required to ensure that differences (both observed and unobserved) are as untroubling as possible, thereby mimicking random assignment as closely as possible. 

We discuss how to structure an empirical strategy to identify a causal relationship using methods such as difference-in-differences, regression discontinuity, instrumental variables, propensity score matching, synthetic control, and selection bias correction. We emphasize the importance of clearly communicating identifying assumptions underlying the assertion of causality and establishing the generalizability of the findings.

We examine the following topics with the goal of helping researchers and analysts use quasi-experiments more effectively.

• Topic 1 – Research question: Do we care whether x causes y? The first and hardest stage in this process is identifying a question in which marketing scholars, managers, or policy makers actually care whether x causes y.
• Topic 2 – Data question: How to find data with quasi-experimental variation in x? Much of the work using quasi-experimental variations in marketing harnesses easily understood events such as contract changes; ecological issues such as the weather; geography; and macroeconomic, individual, organizational, and regulatory changes. The key is to consider why each of these sources of variation can approximate random assignment.
• Topic 3 – Identification strategy: Does x cause y to change? The researcher must first explain where the variation claimed to be exogenous comes from. Second, the researcher needs to demonstrate that the relationship between the variation and the outcome of interest is very likely driven by the relationship between x and y and not by some other factor.
• Topic 4 – Empirical analysis: How to estimate the effect of x on y? We discuss three different regression analysis frameworks using quasi-experiments: difference-in-differences, regression discontinuity, and instrumental variables.
• Topic 5 – Challenges to research question: What if variation in x is not exogenous? We outline three methods—propensity score matching, synthetic control methods, and selection bias correction—with steps to take when comparability between the control and treatment groups is violated.
• Topic 6 – Robustness: How robust is the effect of x on y? The idea here is to show that the sign, significance, and magnitude of the estimate remain broadly consistent across a vast range of possible models. A few examples of robustness checks include different controls, functional forms, choices of time periods, dependent variables, the size of the control group, and a placebo test.
• Topic 7 – Mechanism: Why does x cause y to change? Understanding how the process of the change unfolds adds insight that can drive new knowledge and stronger actions.
• Topic 8 – External validity: How generalizable is the effect of x on y? The external validity discussion in a paper should recognize the assumptions required for the analysis to capture the average treatment effect across the population of interest rather than a more local effect that is an artifact of the data sample or the source of quasi-experimental variation.
• Topic 9 – Apologies: What remains unproven and what are the caveats? Any identification strategy relies on assumptions that need to be explicit throughout the paper. While apologies do not mean all is forgiven, the objective should be to clarify the boundaries of claims made in the paper.

Read the full article.

From: Avi Goldfarb, Catherine Tucker, and Yanwen Wang, “Conducting Research in Marketing with Quasi-Experiments,” Journal of Marketing.

Go to the Journal of Marketing