semiparametric efficiency | Chen Xing

A Road Map of Nonparametric Efficiency in Causal Inference

Mon, 16 Feb 2026 00:00:00 +0000

Causal inference promises to answer “what if” questions from observational data. But the standard approach — specify a parametric model, fit it, report the estimate — breaks down under misspecification. And misspecification is the norm, not the exception. This post draws from my notes on (Kennedy 2023). I walk through the core ideas of nonparametric efficiency theory — the framework that lets us pair flexible machine learning with rigorous inference. The central character is the Efficient Influence Function, a single mathematical object that tells us how to build estimators, why they work, and when we can trust their confidence intervals.

1. Motivation

First let’s keep the casual and statistical issues separate.

Causal to Statistical: The target causal parameter (e.g., ATE) can often be identified as a statistical functional of the observed data distribution.

$$\psi: \mathcal{P} \mapsto \mathbb{R}$$

Once identified, the causal problem becomes a pure functional estimation problem.

The Problem with Parametrics: Parametric models are likely misspecified, leading to biased estimates of the functional.

The Problem with Naive Nonparametrics: While we should use flexible machine learning (nonparametric) methods to avoid misspecification, simple “plug-in” estimators (fitting models and plugging predictions into the formula) fail because:

They are generally not $\sqrt{n}$-consistent (converge too slowly).
They do not yield valid Confidence Intervals.

The Solution: We need Efficiency Theory to understand the theoretical limit of performance and Influence Functions to construct estimators that reach that limit.

2. Efficiency Theory (Lower Bounds)

Now let’s build the “best” estimator.

The Benchmark: We want to find the best possible performance (lowest Mean Squared Error) any estimator can achieve.

Parametric Intuition: In parametric models, the Cramer-Rao (CR) lower bound sets this benchmark: no unbiased estimator can have a variance lower than the inverse Fisher information.

But now we are in the “nonparametric world”, how to exploit CR lower bound?

The “Submodel” Trick: To apply CR bounds to infinite-dimensional nonparametric models:

We define a Parametric Submodel: a smooth, one-dimensional path through the complex nonparametric space that passes through the true distribution.
Logic: If we cannot estimate the parameter well in this simple “parametric” slice, we certainly cannot do it in the full nonparametric model.
Result: The hardest submodel defines the Local Minimax Lower Bound.

3. The Unified Approach: Central Role of the EIF

The Efficient Influence Function (EIF) is the “master key.” It is the derivative term in a Distributional Taylor Expansion¹ (von Mises expansion):

Figure 1: Distributional Taylor Expansion

This single expansion solves all three statistical tasks:

A. How to construct the estimator? (The Recipe)

The expansion reveals that the bias of a naive plug-in estimator is roughly the expected value of the EIF.

Recipe: We construct a One-Step Estimator by taking the naive plug-in estimator and adding the empirical average of the estimated EIF to “de-bias” it.

$$ \hat{\psi}_{\text{one-step}} = \text{Plug-in} + \frac{1}{n}\sum \text{EIF}(\text{Data}) $$

Click to expand: Deconstructing the One-Step Estimator

The Intuition: A “Newton-Raphson” Correction

Think of it like Newton’s method in calculus: if you want to find the root of a function, you make an initial guess, calculate the derivative (slope) at that point, and use it to take a “step” closer to the true answer.

In this context:

The Plug-in is your initial “guess” at the parameter.
The EIF acts as the “derivative” (gradient) that tells you which direction to move to fix the error.
The Formula represents taking that single “step” to correct the bias of your initial guess.

Deconstructing the Formula

$$ \hat{\psi}_{\text{one-step}} = \underbrace{\psi(\hat{P})}_{\text{Plug-in}} + \underbrace{\frac{1}{n}\sum_{i=1}^n \phi(Z_i; \hat{P})}_{\text{Bias Correction}} $$

Part A: The Naive Plug-in $\psi(\hat{P})$

This is the estimate you get if you train your machine learning models (like regression or propensity scores), estimating the whole distribution $\hat{P}$, and simply “plug them in” to the formula for your parameter.

Why it fails: Flexible machine learning models trade bias for variance (regularization). They “smooth” the data to avoid overfitting. While this is good for predicting individual outcomes, it creates a first-order bias in the target parameter that does not vanish fast enough (slower than $1/\sqrt{n}$). If you stop here, your confidence intervals will be wrong.

Part B: The Bias Correction $\frac{1}{n}\sum \text{EIF}$

This term estimates the bias of the plug-in and removes it. The logic relies on the Distributional Taylor Expansion, which tells us that the error of the plug-in estimator is approximately equal to the negative expectation of the Influence Function:

$$ \psi(\hat{P}) - \psi(P_{\text{true}}) \approx -\mathbb{E}[\text{EIF}] $$

Since the error is roughly $-\mathbb{E}[\text{EIF}]$, we can “cancel out” this error by adding the empirical average of the EIF estimated from our data. Note: if our initial estimate $\hat{P}$ were perfect (the truth), the average of the EIF would be exactly zero. The fact that this term is not zero reflects the bias in our initial model that needs to be corrected.

Concrete Example: Average Treatment Effect (ATE)

To make this concrete, consider the ATE. This specific “One-Step” estimator is mathematically identical to the famous AIPW (Augmented Inverse Probability Weighting) or Doubly Robust estimator.

The Plug-in: You train a regression model to predict outcomes for treated vs. untreated groups. You calculate the average difference. The problem? If your regression is slightly wrong (which it always is), your effect estimate is biased.
The EIF Correction: You look at the residuals — the difference between what your model predicted and what actually happened, weighted by the propensity score (the probability of treatment). If your model consistently under-predicts outcomes for the treated group, the EIF term will be positive. Adding this average EIF to the plug-in “bumps” the estimate up, correcting the bias.

Summary

The “One-Step Estimator” recipe acknowledges that modern Machine Learning is great at learning patterns (the Plug-in) but bad at getting the total volume/area right (Bias). By adding the average of the Efficient Influence Function (the derivative), you explicitly calculate that missing “volume” and add it back in, ensuring the final estimate is unbiased and efficient.

B. How to analyze the estimator? (The Proof)

The expansion includes a Remainder Term ($R_2$). To prove the estimator is $\sqrt{n}$-consistent, we must show this remainder is negligible (second-order).

Sample Splitting: We need to use cross-fitting to handle the “empirical process term” (preventing overfitting when estimating nuisance parameters).

C. How to correct the bias? (Double Robustness)

The math of the EIF reveals that the error depends on the product of errors of the nuisance parameters (e.g., error in outcome regression $\times$ error in propensity score).

This yields Double Robustness: even if individual machine learning models converge slowly (e.g., $n^{-1/4}$), their product converges fast enough ($n^{-1/2}$) to allow for valid inference.

4. The Link Between EIF and Riesz Representer

The connection between the Efficient Influence Function (EIF) and the Riesz Representer (RR) is the bridge between theoretical statistics and practical machine learning.

The Deconstruction of the EIF

We already know the EIF is the “correction term” we add to a naive plug-in estimator to fix bias. The Riesz Representer is the key component inside that EIF.

For a vast class of problems (linear functionals), the EIF always takes this specific structure:

$$ \underbrace{\phi(Z)}_{\text{EIF}} = \underbrace{\psi(P)}_{\text{Plug-in}} + \underbrace{\alpha(X, W)}_{\text{Riesz Representer}} \times \underbrace{(Y - \mu(X, W))}_{\text{Outcome Residual}} $$

$\mu(X, W)$: The conditional expectation (the outcome model, e.g., regression of $Y$ on $X, W$).
$\alpha(X, W)$: The Riesz Representer. It tells us how much weight to give each residual to correct the bias.
The EIF says: “To fix bias, look at your regression errors ($Y - \mu$).”
The RR says: “Here is exactly how to weight those errors for this specific causal problem.”

Takeaway

Here is the takeaway. Nonparametric efficiency theory is not an abstraction separate from practice — it is the practice. The EIF tells us what to estimate, the Riesz Representer tells us how to weight the errors, and double robustness explains why the whole thing holds together even when our models are imperfect. If we use AIPW, DML, or any doubly robust estimator, we are already using these ideas. The theory simply makes explicit what those estimators are doing and why they deserve our trust.

Reference

Kennedy, Edward H. (2023), “Semiparametric doubly robust targeted double machine learning: a review.”

Williams, Nicholas T., Oliver J. Hines, and Kara E. Rudolph (2026), “Riesz representers for the rest of us.”

Chernozhukov, Victor, Whitney K. Newey, Victor Quintas-Martinez, and Vasilis Syrgkanis (2024), “Automatic debiased machine learning via riesz regression.”

Chernozhukov, Victor, Whitney K. Newey, and Rahul Singh (2022), “Automatic Debiased Machine Learning of Causal and Structural Effects,” Econometrica, 90 (3), 967–1027.

The Distributional Taylor Expansion is essentially the standard Taylor expansion applied at the distribution scale rather than the real-number scale. It is exactly equivalent to the concept of pathwise differentiability — both formalize the idea of taking a derivative of a statistical functional with respect to perturbations of the underlying distribution. ↩︎

AIPW vs. Residual-on-Residual regression: Non-Parametric Flexibility or Efficiency?

Wed, 11 Jun 2025 00:00:00 +0000

Introduction

Two powerful tools in causal inference are the Augmented Inverse Propensity Weighting (AIPW) estimator and the Residual-on-Residual regression estimator for partially linear models. Drawing from Wager’s notes (2024), this post breaks down how these estimators work, compares their strengths and weaknesses, and offers tips for when to use each.

Residual-on-Residual regression

The semiparamteric partially linear model (PLM) assumes the outcome $ Y $ can be written as:

$$ Y = \theta D + g(X) + \epsilon \tag{1}$$

Here, $ D $ is a binary treatment, $ \theta $ is the causal parameter of interest, $ g(X) $ is some unknown function of covariates $ X $, and $ \epsilon $ is random noise with $\E[\epsilon \mid D, X] = 0$. The residual-on-residual estimator (or Robinson (1988) estimator) isolates $ \theta $ in two steps:

Partialing out covariates: “Partialing out” $ X $ from $D$ and $Y$ using nonparametric regression and cross-fitting: $ \tilde{D} = D - \hat{\E}[D \mid X] $ and $ \tilde{Y} = Y - \hat{\E}[Y \mid X] $.
Estimate the effect: Use linear regression $ \tilde{Y} \sim \tilde{D} $ to get $ \hat{\theta}$.

However, the model is not fully general, because it imposes a parametric specification on the key component of interest. It imposes additivity in $g(X)$ and $D$.

Augmented Inverse Propensity Weighting (AIPW) Estimator

AIPW takes a fully non-parametric approach, aiming to estimate ATE, $ \tau = E[Y(1) - Y(0)] $, where $ Y(1) $ and $ Y(0) $ are potential outcomes under treatment and control.

$$ \hat{\tau}_{\text{AIPW}} = \frac{1}{n} \sum_{i=1}^n \left[ \hat{\mu}_1(X_i) - \hat{\mu}_0(X_i) + D_i \frac{Y_i - \hat{\mu}_1(X_i)}{\hat{e}(X_i)} - (1 - D_i) \frac{Y_i - \hat{\mu}_0(X_i)}{1 - \hat{e}(X_i)} \right] $$

Here, $ \hat{\mu}_1(X) $ and $ \hat{\mu}_0(X) $ are outcome regression estimators for treated and untreated units, and $ \hat{e}(X) $ is the estimator of propensity score. AIPW combines outcome modeling with inverse propensity weighting, making it doubly robust: it’s consistent if either the outcome model or propensity score is consistent.

The Key Difference: Non-Parametric vs. Partially Linear

AIPW is fully non-parametric, imposing no specific parametric form on the treatment effect, while residual-on-residual regression estimator assumes a partially linear structure

AIPW’s flexibility:
- AIPW does not assume a specific parametric form!
- AIPW is efficient in the generic non-parametric setting.
Residual-on-residual regression’s structure: What partially linear assumption buys us is that residual-on-residual estimators that exploit this constraint can have smaller variance than AIPW. In other words, adding this additional structure makes the residual-on-residual estimator more efficient than AIPW.

A risk of using the residual-on-residual estimator is that constant treatment effect model (1) may be misspecified.

Why This Matters

The choice between AIPW and residual-on-residual regression reflects a deeper trade-off in causal inference: flexibility versus efficiency. AIPW’s non-parametric nature makes it a Swiss Army knife for complex data, while PLM structure is like a precision tool—effective when conditions are right.

As Wager’s notes highlight:

Both AIPW and residual-on-residual regression are Neyman-orthogonal, making them robust to first‐stage errors.
However, their assumptions shape their performance. AIPW attains the lowest possible asymptotic variance for ATE under unconfoundedness. The residual-on-residual estimator, by imposing extra structure, can go beyond that bound when its structure is correct but at the cost of vulnerability to misspecification.

Conclusion

AIPW’s fully non-parametric approach offers robustness and flexibility, while residual-on-residual regression’s partially linear structure prioritizes efficiency when assumptions hold.

Reference

Wager, S. (2024). Causal inference: A statistical learning approach. https://web.stanford.edu/~swager/causal_inf_book.pdf

Big Picture of Debiased Machine Learning

Tue, 25 Mar 2025 00:00:00 +0000

Debiased machine learning (DML) is a generic recipe. The idea behind it is adding a correction term to the plug-in estimator of the functional, which leads to properties such as semi-parametric efﬁciency, double robustness, and Neyman orthogonality.

(Auto)-DML is a Method-of-Moments estimator

debiased/orthogonal moment scores
- Why it matters?
  - try to solve: model selection and/or regularization bias from ML learners (e.g. Lasso)
  - Neyman orthogonality: ensure the parameter of interest insentitive to first order perturbation of nuisance estimation
  - double robustness
  - asymptotic normality
- Key Idea: Debiasing is achieved by adding a correction term to the plug-in estimator of the functional
  - Three representations: $\theta = \mathbb{E}[m(W,g)] = \mathbb{E}[Y\alpha(W)] = \mathbb{E}[g(W)\alpha(W)]$, where
    - $g()$ is outcome regression;
    - $\alpha()$ is Rieze Representer (RR);
    - $m()$ is a continuous linear functional;
    - $W = (D, X)$ is data containing treatment $D$ and covariates $X$;
    - $Y(d)$ is potential outcome
  - Correct the residual using RR
    - $\mathbb{E}\{m(W,g) - \theta + \alpha(W)[Y-g(W)]\} = 0$
- How to construct orthogonal moment function?
  - orthogonal moment function = identifying moment function + first step influence function (FSIF)
  - identifying moment function: $m(W,g) - \theta$
    - involving outcome regression
  - FSIF: $\alpha(W)[Y-g(W)]$
    - correct the residual using Rieze Representer (RR)
    - Rieze Representer (RR)
      - In the case of ATE with binary treatment, RR are inverse propensity score terms
      - RR can be automatically characterized; NO NEED to know its analytical form
      - Can use random forests and NNet learners of RR
- Double Robustness
  - $\mathbb{E}[m(W ; g) -\theta_0 \left.+\alpha(W)(Y-g(W))\right] =-\mathbb{E}\left[\left(\alpha-\alpha_0\right)\left(g-g_0\right)\right]$
  - The score will be zero in expectation when either $\alpha(W) = \alpha_0(W)$ or $g(W) = g_0(W)$
Cross-fitting
- Why it matters?
  - Reduce overfitting bias

Orthogonal vs Non-orthogonal Learning

Sat, 22 Feb 2025 00:00:00 +0000

The following exercise is from CausalAIBook Notebook. The goal is to illustrate the difference between Neyman orthogonal vs non-orthogonal approach.

Setting

We compare the performance of the naive and orthogonal methods in a computational experiment where $p=n=100$, $\beta_j = 1/j^2$, $(\gamma_{DW})_j = 1/j^2$ and \[Y = 1 \cdot D + \beta' W + \epsilon_Y\]

where $W \sim N(0,I)$, $\epsilon_Y \sim N(0,1)$, and \[D = \gamma'_{DW} W + \tilde{D}\] where $\tilde{D} \sim N(0,1)/4$.

Note that: $\tilde{D}$ is the residual from the projection $D \sim W$.

The true treatment effect here is 1.
From the plots produced in this post (estimate minus ground truth), we show that the naive single-selection estimator is heavily biased (lack of Neyman orthogonality in its estimation strategy), while the orthogonal estimator based on partialling out, is approximately unbiased and Gaussian.

Simulation

library(hdm)
library(ggplot2)

# Initialize constants
B <- 10000  # Number of iterations
n <- 100  # Sample size
p <- 100  # Number of features

# Initialize arrays to store results
Naive <- rep(0, B)
Orthogonal <- rep(0, B)


lambdaYs <- rep(0, B)
lambdaDs <- rep(0, B)

for (i in 1:B) {
  # Generate parameters
  beta <- 1 / (1:p)^2
  gamma <- 1 / (1:p)^2

  # Generate covariates / random data
  X <- matrix(rnorm(n * p), n, p)
  D <- X %*% gamma + rnorm(n) / 4

  # Generate Y using DGP
  Y <- D + X %*% beta + rnorm(n)

  # Single selection method
  rlasso_result <- hdm::rlasso(Y ~ D + X)  # Fit lasso regression
  sx_ids <- which(rlasso_result$coef[-c(1, 2)] != 0)  # Selected covariates

  # Check if any Xs are selected
  if (sum(sx_ids) == 0) {
    Naive[i] <- lm(Y ~ D)$coef[2]  # Fit linear regression with only D if no Xs are selected
  } else {
    Naive[i] <- lm(Y ~ D + X[, sx_ids])$coef[2]  # Fit linear regression with selected X otherwise
  }

  # Partialling out / Double Lasso

  fitY <- hdm::rlasso(Y ~ X, post = TRUE)
  resY <- fitY$res

  fitD <- hdm::rlasso(D ~ X, post = TRUE)
  resD <- fitD$res

  Orthogonal[i] <- lm(resY ~ resD)$coef[2]  # Fit linear regression for residuals
}

Making a Nice Plot

# Specify ratio
img_width <- 15
img_height <- img_width / 2

# Create a data frame for the estimates
df <- data.frame(
  Method = rep(c("Naive", "Orthogonal"), each = B),
  Value = c(Naive - 1, Orthogonal - 1)
)

# Create the histogram using ggplot2
hist_plot <- ggplot(df, aes(x = Value, fill = Method)) +
  geom_histogram(binwidth = 0.1, color = "black", alpha = 0.7) +
  facet_wrap(~Method, scales = "fixed") +
  labs(
    title = "Distribution of Estimates (Centered around Ground Truth)",
    x = "Bias",
    y = "Frequency"
  ) +
  geom_vline(xintercept = 0, color = "red", linetype = "dashed") +
  scale_x_continuous(breaks = seq(-2, 1.5, 0.5)) +
  theme_minimal() +
  theme(
    plot.title = element_text(hjust = 0.5),  # Center the plot title
    strip.text = element_text(size = 10),  # Increase text size in facet labels
    legend.position = "none", # Remove the legend
    panel.grid.major = element_blank(),  # Make major grid lines invisible
    # panel.grid.minor = element_blank(),  # Make minor grid lines invisible
    strip.background = element_blank()  # Make the strip background transparent
  ) +
  theme(panel.spacing = unit(2, "lines"))  # Adjust the ratio to separate subplots wider

# Set a wider plot size
options(repr.plot.width = img_width, repr.plot.height = img_height)

# Display the histogram
print(hist_plot)

Conclusion: As we can see from the above bias plots (estimates minus the ground truth effect of 1), the double lasso procedure concentrates around zero whereas the naive estimator does not.

Reference

CausalAIBook Notebook

Notes on Semiparametric Models

Wed, 05 Feb 2025 00:00:00 +0000

Motivation

Semiparametric models contain both a finite-dimensional parameter of interest ($\theta$) and an infinite-dimensional nuisance parameter ($\eta$).
The goal is to estimate $\theta$ as efficiently as possible while filtering out the impact of $\eta$.

Key Concepts

1. Tangent Space

The tangent space consists of all possible local perturbations of the statistical model.
In parametric models, these directions are given by score functions (derivatives of the log-likelihood). In semiparametric models, the tangent space is typically an infinite-dimensional subspace of $L^2(P)$ (the space of square-integrable functions).

2. Nuisance Tangent Space ($\mathcal{T}_{\eta}$)

This is the subset of the full tangent space that corresponds to variations in the nuisance parameter $\eta$, while holding the parameter of interest $\theta$ fixed.
It represents all the directions in which the nuisance part of the model can change and potentially affect the estimation of $\theta$.

3. Orthogonal Complement of the Nuisance Tangent Space ($\mathcal{T}_{\eta}^\perp$)

Defined as:
$$ \mathcal{T}_{\eta}^\perp = \{ h \in L^2(P) : \langle h, g \rangle = 0 \quad \text{for all } g \in \mathcal{T}_{\eta} \} $$
where the inner product $\langle \cdot, \cdot \rangle$ is typically given by covariance (or Fisher information).
This space contains directions that are “free” of the influence of the nuisance parameter. In other words, any variation in this space does not get “contaminated” by changes in $\eta$.

Why It Matters?

Efficient Estimation

In semiparametric estimation, constructing an estimator with the smallest possible variance (i.e., achieving the efficiency bound) involves ensuring that its influence function lies in $\mathcal{T}_{\eta}^\perp$.
The influence function describes how an estimator responds to small changes in the data distribution.
By projecting any candidate influence function onto $\mathcal{T}_{\eta}^\perp$, one removes the component due to the nuisance parameter, yielding the efficient influence function.

Practical Implication

This separation allows us to focus on the parameter of interest while systematically “filtering out” nuisance effects, leading to more precise (optimal) estimators.

Summary

Nuisance Tangent Space ($\mathcal{T}_{\eta}$): Captures all the directions of change due to the nuisance parameter $\eta$.
Orthogonal Complement ($\mathcal{T}_{\eta}^\perp$): Contains directions free from nuisance effects, representing pure variations in $\theta$.
Efficient Influence Function: By projecting onto $\mathcal{T}_{\eta}^\perp$, one obtains an influence function that is optimal, meaning that the corresponding estimator achieves the semiparametric efficiency bound.

From Donsker Classes to Neyman Orthogonality: The Power of DML

Fri, 26 Apr 2024 00:00:00 +0000

Motivation & Intuition

In classical semiparametric theory, we want to estimate a low‑dimensional target parameter (say, a treatment effect) while controlling for high‑dimensional nuisance functions (like nonparametric regressions). However, in order to use the central limit theorem (CLT) and to characterize the asymptotic behavior, classical results require that the space of functions in which these nuisance functions lie is “small” in a technical sense. In particular, they must form a Donsker class — roughly speaking, a collection of functions whose complexity (measured via “entropy”) is bounded enough so that the empirical process converges to a Gaussian process. This condition, however, is too restrictive in modern applications where the nuisance functions are estimated by flexible machine learning methods (e.g., random forests, boosting, deep neural nets) that may come from very large, high‑dimensional spaces.

The double machine learning approach overcomes this problem by using Neyman orthogonal scores. The key idea is that the moment functions used to estimate the target parameter are constructed in such a way that small errors in estimating the nuisance functions have only a second-order effect on the final estimator. In plain language, even if your machine learning methods are “messy” or come from huge function classes (i.e. they do not satisfy the Donsker conditions), the estimation of your target parameter remains robust as long as the nuisance estimators converge at a certain rate. Moreover, the approach uses sample splitting (or cross‑fitting) to avoid overfitting biases.

Key Math Details

Summary

Flexibility in Nuisance Estimation: DML frees us from the need for nuisance estimators to lie in “small” Donsker classes. Thanks to Neyman orthogonality and cross-fitting, we can plug in flexible, machine-learning based nuisance estimates—even if they come from very rich function spaces—without contaminating the asymptotic distribution of the target parameter estimator.

Thus, the contribution of DML is in allowing the use of complex, modern ML methods to estimate nuisance functions while still obtaining valid inference for the target parameter, bypassing the traditional, more restrictive Donsker conditions.