Conformal Inference for Counterfactual and Synthetic Controls

I’ve been reading Chernozhukov, Wüthrich, and Zhu (2021) on conformal inference for synthetic control. The basic idea is quite intuitive once viewed as a residual-based test.

1. The Core Problem: Measuring the Counterfactual

What was the causal impact of a treatment on a single treated unit (e.g., a state, city, or firm)? To measure this impact, we compare what actually happened after the treatment ($Y^I_{1t}$) against what would have happened if the treatment had never been implemented ($Y^N_{1t}$). This unobserved hypothetical scenario is called the counterfactual.

Researchers usually construct a proxy model ($\hat{P}^N_t$) for this counterfactual using untreated control groups or past historical trends—such as Synthetic Control (SC) or Difference-in-Differences (DID). However, calculating whether the observed treatment effect is statistically significant (i.e., not just random noise) is notoriously difficult when working with a single treated unit or a small number of pre-policy time periods ($T_0$).

2. What is Conformal Inference?

Conformal inference is a statistical framework that reframes causal policy evaluation as a counterfactual prediction and structural breaks testing problem.

Instead of relying on rigid parametric statistical assumptions or assuming the counterfactual model is perfectly specified, conformal inference asks an intuitive question:

Does the prediction error pattern after the treatment look abnormally large compared to the typical prediction errors before the treatment?

If the treatment had no effect, the prediction errors (residuals) in the post-treatment period should look like just another set of errors from the pre-treatment period. If the post-treatment errors are systematically larger or different, we can conclude that a structural break occurred—meaning the treatment likely had a true causal effect.

3. How Conformal Inference Works: Step-by-Step

Here is how the procedure operates in practice:

Step 1. Formulate a Sharp Null Hypothesis ($H_0$)

We postulate a specific trajectory for the treatment effect $\theta_0$ across the post-treatment period (for instance, the “zero-effect” null hypothesis: $H_0: \theta_t = 0$, for $t > T_0$).

Step 2. Impute the Counterfactual and Estimate Under the Null

By subtracting our hypothesized effect $\theta_0$ from the observed outcomes during the post-treatment period ($Y^N_{1t} = Y_{1t}^I - \theta_t^0$), we create an “as-if untreated” outcome series across all periods.

For example, under the “zero-effect” $H_0$, we have $$Y^N_{1t} = Y_{1t}^I - \theta_t^0 = Y_{1t} - 0 = Y_{1t}$$ In other words, if there was no treatment effect, the observed treated outcome is also the counterfactual untreated outcome.

We then estimate our counterfactual model across the entire dataset ($T_0$ pre-treatment + $T^*$ post-treatment periods) under this null assumption.

Why this matters: Estimating the model under the null guarantees exact finite-sample validity and prevents severe size distortions, especially when sample sizes ($T_0$) are small.

Step 3. Compute Prediction Residuals

We compute the prediction errors (residuals) across all periods: $$\hat{u}_t = Y^N_{1t} - \hat{P}^N_t,$$ where $Y^N_{1t}$ is the imputed counterfactual outcome and $\hat{P}^N_t$ is the model’s estimated proxy.

Step 4. Permute Residuals Across Time

Under the null hypothesis, the post-treatment error should follow the same underlying error distribution as the pre-treatment errors. We construct a reference distribution by permuting blocks of residuals across time—either using all possible permutations (if data points are independent/exchangeable) or moving block permutations (if data exhibits serial dependence). In practice, we often use moving block permutations in the panel data setting.

Step 5. Calculate a Test Statistic and $p$-Value

We calculate a test statistic $S(\hat{u})$ that measures the magnitude of post-treatment residuals.

$$ S(\hat{u})=S_q(\hat{u})=\left(\frac{1}{\sqrt{T_*}} \sum_{t=T_0+1}^T\left|\hat{u}_t\right|^q\right)^{1 / q} . $$ We often set $q = 1$.

The $p$-value is simply the proportion of permuted residual sets that generate a test statistic as large or larger than the observed test statistic.

Formally, the $p$-value is $\hat{p}=1-\hat{F}(S(\hat{u}))$, where

$$\hat{F}(x)=\frac{1}{|\Pi|} \sum_{\pi \in \Pi} 1\left\{S\left(\hat{u}_\pi\right) \lt x\right\}.$$ $\Pi$ denotes the set of permutations.

Here is a simple illustration for the step 4 and 5.

Conceptual picture

4. Why Is This Method a Major Advancement?

Conformal inference offers several key theoretical and practical benefits:

• Exact Finite-Sample Validity: It delivers reliable $p$-values even when the pre-treatment sample size ($T_0$) is small, avoiding the large-sample approximations that frequently break down in empirical case studies.

• Robustness Against Model Misspecification: Even if your underlying proxy model is misspecified or inconsistent, conformal inference remains valid as long as the estimator satisfies a basic stability condition (meaning small changes in individual data points do not cause drastic changes in predictions).

• Universal Compatibility: It works with almost any counterfactual estimator, including:

  • Difference-in-Differences and Synthetic Control
  • Constrained Lasso (a tuning-free method that unifies DID and SC)
  • Interactive Fixed Effects & Factor Models
  • Matrix Completion via nuclear norm regularization
  • Pure Time Series Models (such as AR or neural network models) and Fused Panel/Time Series Models.

5. A Real-World Example: Rhode Island Prostitution Decriminalization

To demonstrate the methodology, the paper re-analyzed an unexpected legal event in Rhode Island, where indoor prostitution was unexpectedly decriminalized between 2003 and 2009.

Using $T_0 = 19$ pre-treatment years (1985–2003) and $J = 50$ donor states as controls, the authors applied conformal inference across Difference-in-Differences, Synthetic Control, and Constrained Lasso. The test rejected the zero-effect null hypothesis ($p < 0.10$ across all estimators), providing rigorous evidence that indoor sex work decriminalization led to a statistically significant reduction in female gonorrhea incidence.

Reference

Chernozhukov, V., Wüthrich, K., & Zhu, Y. (2021). An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls. Journal of the American Statistical Association, 116(536), 1849–1864. https://doi.org/10.1080/01621459.2021.1920957

YouTube short intro: Conformal Prediction without Big Words