Practical Aspects of Double Machine Learning
Fundamental Challenges in Causality
May 10, 2023
Motivation
What is Double/Debiased Machine Learning (DML)?
DML is a general framework for causal inference and estimation of causal parameters based on machine learning
Summarized in Chernozhukov et al. (2018)
Combines the strengths of machine learning and econometrics
What is Double/Debiased Machine Learning (DML)?
Obtain a DML estimate of the causal parameter with asymptotically valid confidence intervals
DML estimator has good theoretical statistical properties like √N rate of convergence, unbiasedness, approximate normality
DML can be generalized to other causal models and settings (multiple treatments, heterogeneous treatment effects, … )
Motivating Example
Partially linear regression model (PLR)
Y=Dθ0+g0(X)+ζ,E[ζ|D,X]=0,D=m0(X)+V,E[V|X]=0,
with
- Outcome variable Y
- Policy or treatment variable of interest D
- High-dimensional vector of confounding covariates X=(X1,…,Xp)
- Stochastic errors ζ and V
Introduction to Double Machine Learning
DML Key Ingredients
1. Neyman Orthogonality
Inference is based on a moment condition that satisfies the Neyman orthogonality condition ψ(W;θ,η) E[ψ(W;θ0,η0)]=0,
where W:=(Y,D,X,Z) and with θ0 being the unique solution that obeys the Neyman orthogonality condition ∂ηE[ψ(W;θ0,η]|η=η0=0.
∂η denotes the pathwise (Gateaux) derivative operator
DML Key Ingredients
1. Neyman Orthogonality
Neyman orthogonality ensures that the moment condition identifying θ0 is insensitive to small pertubations of the nuisance function η around η0
Using a Neyman-orthogonal score eliminates the first order biases arising from the replacement of η0 with a ML estimator ˆη0
PLR example: Partialling-out score function (cf. Section 5.3 , Appendix) ψ(⋅)=(Y−E[Y|X]−θ(D−E[D|X]))(D−E[D|X])
DML Key Ingredients
2. High-Quality Machine Learning Estimators
The nuisance parameters are estimated with high-quality (fast-enough converging) machine learning methods.
Different structural assumptions on η0 lead to the use of different machine-learning tools for estimating η0 Chernozhukov et al. (2018) (Section 3)
Rate requirements depend on the causal model and orthogonal score, e.g. (see Chernozhukov et al. (2018)),
- PLR, partialling out: ‖ˆm0−m0‖P,2×(‖ˆm0−m0‖P,2+‖ˆℓ0−ℓ0‖P,2)≤δNN−1/2
- IRM/DR score, ATE: ‖ˆm0−m0‖P,2×‖ˆℓ0−ℓ0‖P,2≤δNN−1/2
DML Key Ingredients
3. Sample Splitting
To avoid the biases arising from overfitting, a form of sample splitting is used at the stage of producing the estimator of the main parameter θ0.
Efficiency gains by using cross-fittng (swapping roles of samples for train / hold-out)
DML Key Ingredients
Main result in Chernozhukov et al. (2018)
There exist regularity conditions, such that the DML estimator ˜θ0 concentrates in a 1/√N-neighborhood of θ0 and the sampling error is approximately √N(˜θ0−θ0)∼N(0,σ2), with σ2:=J−20E(ψ2(W;θ0,η0)),J0=E(ψa(W;η0)).
Practical Aspects of Double Machine Learning
Practical Aspects of DML
Questions in Practice
Which learner should be used for estimation of the nuisance parameter η0?
- In PLR example (partialling out): η0=(ℓ0,m0)=(E[Y|X],E[D|X])
How should the hyperparameters of the learners be tuned? Which sample (splitting) to use for tuning?
Can AutoML frameworks be used? How well do they perform in practice?
Which causal model should be used?
Practical Aspects of DML
Approach
- Use various simulation settings and semi-synthetic benchmarks to elaborate role of practical choices
- Choice of learner and hyperparameters
- Parameter tuning
- Sample splitting
- Choice of causal models
- …
Goal
- Derive some guidance for applications of DML
Results
- First-stage error
- Estimation quality (MSE(ˆθ0), …)
- Emp. coverage
- Computation time
Motivating Example: Lasso Penalty
How do more/less accurate estimators for η0 affect the causal estimate, ˆθ?
Relationship of nuisance fit and estimation quality
Simulated example
- high-dimensional linear sparse model, based on Belloni, Chernozhukov, and Hansen (2014)
- n=100 and p=200
- continuous treatment variable D
Lasso learner for η0=(ℓ0,m0) evaluated over a grid of λ=(λℓ,λm) values (l1-penalty)
Motivating Example: Lasso Penalty
Motivating Example: Lasso Penalty
Motivating Example: Lasso Penalty
Motivating Example: Lasso Penalty
Simulation Results
Sample Splitting and Hyperparameter Tuning
Canditate splitting schemes
Simulation Settings
Settings considered for PLR Model
- Setting 1: High-dimensional sparse and linear DGP as of Belloni, Chernozhukov, and Hansen (2014)
- n=100, p=200
- n=400, p=200
- Setting 2: “Scenario 2” from ACIC 2019 data challenge
- PLR with nonlinear sparse specification for g0(X) and m0(X)
- n=1000, p=200
- Binary treatment D
More settings
- ACIC2019 all DGPs
- IHDP Benchmark
- IRM model (AIPW/DR score)
- …
Simulation Settings
Learners considered for PLR Model
scikit-learn’s cross-validated Lasso (Pedregosa et al. 2011)XGBoosttuned (Chen et al. 2015)XGBoostdefault- AutoML framework
flaml(Wang et al. 2021) - …
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Number of Folds for Cross-Fitting
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 2. ACIC (DGP 2)
Simulation Results: 2. ACIC (DGP 2)
Simulation Results: 2. ACIC (DGP 2)
Simulation Results: 2. ACIC (DGP 2)
Number of Folds for Cross-Fitting
Simulation Results: 2. ACIC (DGP 2)
Simulation Results: 2. ACIC (DGP 2)
Simulation Results: 2. ACIC (DGP 2)
flaml, DGP 2
Simulation Results: 2. ACIC (all DGPs)
flaml, all DGPs
Summary
Careful choice of learners and hyperparameter tuning is important for estimation quality for causal parameter θ0
Full-sample and on-fold tuning seem to outperform the split-data approach in terms of
- nuisance fit
- estimation quality of θ0
- coverage
On-fold tuning computationally more expensive
Lower (combined) first stage error found to be associated with better performance
Summary
- Higher number of folds for cross-fitting seem to improve the estimation performance, but at higher computational costs.
- more important in small samples
- can be evaluated empirically
- AutoML frameworks seem to work well in combination with DML
- Some frameworks suffer from some instabilities
Next steps
Choice of causal model / score
Scale and summarize results, more benchmarks
Evaluation of additional strategies:
- Ensemble
- Pick ‘best’ model
- 1-se Lasso
- …
Refined hyperparameter tuning
Appendix
DML Key Ingredients
1. Neyman Orthogonality
The inference is based on a score function ψ(W;θ,η) that satisfies E[ψ(W;θ0,η0)]=0,
where W:=(Y,D,X,Z) and with θ0 being the unique solution that obeys the Neyman orthogonality condition ∂ηE[ψ(W;θ0,η]|η=η0=0.
∂η denotes the pathwise (Gateaux) derivative operator
DML Key Ingredients
1. Neyman Orthogonality
Neyman orthogonality ensures that the moment condition identifying θ0 is insensitive to small pertubations of the nuisance function η around η0
Using a Neyman-orthogonal score eliminates the first order biases arising from the replacement of η0 with a ML estimator ˆη0
PLR example: Partialling-out score function (cf. Section 5.3 , Appendix) ψ(⋅)=(Y−E[Y|X]−θ(D−E[D|X]))(D−E[D|X])
Frisch-Waugh-Lovell Theorem
Solution to regularization bias: Orthogonalization
Remember the Frisch-Waugh-Lovell (FWL) Theorem in a linear regression model
Y=Dθ0+X′β+ε
θ0 can be consistently estimated by partialling out X, i.e,
OLS regression of Y on X: ˜β=(X′X)−1X′Y → Residuals ˆε
OLS regression of D on X: ˜γ=(X′X)−1X′D → Residuals ˆζ
Final OLS regression of ˆε on ˆζ
Orthogonalization: The idea of the FWL Theorem can be generalized to using ML estimators instead of OLS
Neyman Orthogonality
Naive approach
ψ(W,θ0,η)=(Y−Dθ0−g0(X))D
Regression adjustment score
η=g(X),η0=g0(X),
FWL partialling out
ψ(W,θ0,η0)=((Y−E[Y|X])−(D−E[D|X])θ0)(D−E[D|X])
Neyman-orthogonal score (Frisch-Waugh-Lovell)
η=(g(X),m(X)),η0=(g0(X),m0(X))=(E[Y∣X],E[D∣X])
Additional Results
Motivating Example: Lasso Penalty
Motivating Example: Lasso Penalty
Motivating Example: Lasso Penalty
Motivating Example: Lasso Penalty
Simulation Results: 1. BCH 2014
Number of Folds for Cross-Fitting
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 1. BCH 2014
Simulation Results: 2. ACIC (DGP 2)
rel. MSE(ˆθ)
Simulation Results: 2. ACIC (DGP 2)
Abs. bias
IHDP Benchmark
TODO
Resources
More resources
For a nontechnical introduction to DML: Bach et al. (2021)
Software implementation:
Paper draft to be uploaded at arxiv soon
::::
References
References
Practical Aspects of DML