Research Note: What Are Natural Experiments? Methods, Approaches, and Applications

I enjoy reading Craig et al. (2017) ‘s review article on Natural Experiments (An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research). In this post, I want to summarize its key points and attach some of my reflections about the development of causal inference.

This review article introduces what NEs are and what methods and approaches are available for NE data.

[Related Article: Causal Inference: An Indispensable Set of Techniques for Your Data Science Toolkit]

Typically, annual reviews, like this one by Craig et al., provide a quick look into the recent developments of the field in shape and into future directions.

It’s a great way of learning data science. Highly recommended!

What Is A NE?

According to the UK Medical Research Council, any event not under the control of a researcher that divides a population into exposed and unexposed groups.

Due to the lack of direct control of the assignment process, researchers have to rely on statistical tools to determine the causal effects of manipulating the variations in exposure to the treatment condition.

The key challenge with NEs is to rule out the possibility of selecting into the treatment group, which would be a violation of the ignobility assumption. The violation also makes the treatment and control groups non-comparable, and we can’t attribute the differences in the outcome variables to the presence of the intervention.

To address this issue, methodologists come up with Potential Outcomes Framework. POF stands for the outcomes that would occur if a person were exposed and not exposed to an intervention.

However, the tricky part is only one of those two outcomes is observable, and we have to rely on counterfactuals to infer the average outcomes among units.

If the assignment is random, as in Random Controlled Trials (RCTs), then the treatment and control groups are exchangeable. We can attribute the gap between these two groups to the presence of the intervention.

If the assignment is not random, as in NEs, then researchers have to rely on domain knowledge of the assignment mechanism and statistical methods to achieve conditionally exchangeable.

This is when qualitative research and domain knowledge kick into the equation and help us determine whether there is a causal story behind the assignment process.

In the real world, I’d say NEs have a broader scope of application than RCTs for practical and ethical reasons. So, it becomes critical to choose appropriate methods/designs to do causal inference for NE data, as suggested by Craig et al. (2017).

There are primarily eight techniques for doing so. I’ll include each method here with some research notes and links to real-life applications. Please refer to the original article (here) for fuller discussions of each technique.

Methods

1. Pre-and-Post Analysis. Personally, this would be my last resort if there is no better alternative available. Single case comparison without multiple data points. How can we control for confounders? Not an ideal choice for causal inference.
2. Regression Adjustment. It has a lot of applications when we try to make cases comparable.
3. Propensity Scores Matching. Good for observational data as well, but Gary King recently repudiates the idea of using PSM (here).
4. Difference-In-Differences. It’s a strong causal inference technique with a straightforward research ideology.
5. Interrupted Time Series. Strong causal method with multiple data entries. Arguably, the strongest quasi-experimental technique.
6. Synthetic Controls. This is a trendy method in the industry and academia, in which political scientists have made tremendous contributions to it. In brief, we could artificially create a weighted average of control groups to serve as a basis point if there are no cases in the control group that matches the treatment group. For example, we create an artificial control scenario using a weighted value of the other cases and compare the differences between these two groups. It is such an ingenious idea but with potential pitfalls, for which I’ll elaborate on another post.
7. Regress Discontinuity Design. Strong causal technique with great visual illustration.
8. Instrumental Variable. Technically speaking, IV contains a strong inferential power, but it’s rather difficult to find fit IVs. Thus, it has limited applications.

How To Make NEs Stronger In Causal Inference?

This review paper provides three solutions:

[Related Article: 5 Hands-on Skills Every Data Scientist Needs in 2020 – Coming to ODSC East 2020]

1. Incorporate qualitative components to understand the working mechanism. My two cents of view is we shall not forget the importance of qualitative research in an era of big data and machine learning. Understanding the process well enough, or domain knowledge, helps us come up with better correct statistical models.
2. A combination of multiple quantitative methods and visual checks for discontinuities in RDD and ITS. Visual inspections are crucial and straightforward for identifying irregularities. Use them more and wisely, if possible.
3. Introduce falsification/placebo tests to evaluate the plausibility of causal attribution. For example, we can use nonequivalent Dependent Variables to test for changes in outcomes that are not exposed to the intervention with the ones that exposed to the intervention. Here, the underlying idea is to cross-check results by using multiple DVs, a research idea widely used in social sciences.

Originally Posted Here