Standard Errors & Inference

While the coefficients in our regression tell us what the estimated relationship is, our standard errors tell us how confident we are in that relationship. Getting standard errors right is critical: if they are wrong, then t-statistics, p-values, and confidence intervals are all wrong — even if the coefficient itself is unbiased. This lecture builds significantly on Michler and Josephson (2022), a comprehensive review of recent developments in inference for applied economics.

This lecture covers:

• Two competing sources of uncertainty: sampling-based vs. design-based
• Why default “robust” standard errors (HC1) can mislead in small samples
• Leverage-adjusted estimators: HC2, HC3, and Bell-McCaffrey
• Clustered standard errors and when they break down
• Bootstrap and wild cluster bootstrap methods
• Randomization inference
• Multiple hypothesis testing: FWER and FDR corrections

We’ll use the lifeexp dataset, which contains country-level life expectancy data with significant outliers (like the United States).

Two Sources of Uncertainty

Before diving into specific methods, it is essential to understand why we need standard errors in the first place. There are two fundamentally different sources of uncertainty in applied research:

In sampling-based inference, standard errors quantify how much the coefficient would vary across hypothetical repeated samples from the same population. In design-based inference, standard errors quantify how much the coefficient would vary across hypothetical repeated randomizations of treatment assignment.

Table 11.1: Sampling-Based Uncertainty (✔ is observed, ? is missing)

Table 11.2: Design-Based Uncertainty (✔ is observed, ? is missing)

This distinction matters because the appropriate method for calculating standard errors depends on the source of uncertainty. Most applied work in economics uses sampling-based inference, but design-based thinking (especially through randomization inference) has become increasingly important.

In practice, the two frameworks often give similar answers. But when samples are small, clusters are few, or the design is unusual, the choice can matter. It is good practice (though frequently not done) to clearly articulate the source of uncertainty in your data before selecting an inference method.

The Problem with Default “Robust” SEs

You have likely been taught to broadly append , robust to your regressions to correct for heteroskedasticity. In Stata, robust calculates standard errors using an algorithm historically known as HC1 (Heteroskedasticity-Consistent 1).

While HC1 works perfectly well asymptotically (in very large samples), it has a known flaw in smaller datasets: it is highly susceptible to leverage mismatch from influential observations. High-leverage observations — individual data points that sit far from the center of the data — can severely distort the HC1 variance estimate.

This flaw led to a major controversy in economics. In 2019, Alwyn Young published a highly cited paper in the Quarterly Journal of Economics claiming that conventional regression methods produced wildly inflated p-values in experimental data, fiercely advocating that the entire field abandon regression inference in favor of Randomization Inference.

However, researchers quickly dismantled his core claims. The team at the Data Colada blog (and others in the applied microeconometrics community) demonstrated that regression itself wasn’t broken — Young’s findings were merely an artifact of reliance on Stata’s outdated HC1 default. Simply switching the estimator from HC1 to HC2 (which mathematically adjusts the squared residuals using the “hat matrix” to downweight high-leverage outliers) completely stabilized the regressions and salvaged the p-values.

HC2 and HC3: Leverage-Adjusted Standard Errors

Whenever you have a small sample, you should generally avoid the basic robust command and instead instruct Stata to use HC2 or HC3.

• HC2 divides each squared residual by $(1 - h_{ii})$, where $h_{ii}$ is the leverage of observation $i$. This prevents high-leverage observations from artificially shrinking the variance estimate.
• HC3 goes further, dividing by $(1 - h_{ii})^2$. It is more conservative and performs better in very small samples, but can be overly cautious in moderate-sized datasets.

Let’s compare them:

* load dataset
	use             "$data/lifeexp.dta", clear
	
* stata's default robust (hc1)
	reg             lexp gnppc, robust
	eststo          hc1
	
* leverage-adjusted robust estimator (hc2)
	reg             lexp gnppc, vce(hc2)
	eststo          hc2

* more conservative adjustment (hc3)
	reg             lexp gnppc, vce(hc3)
	eststo          hc3

Notice that the coefficients are identical across all three columns — only the standard errors change. HC2 and HC3 produce larger standard errors than HC1, reflecting the leverage adjustment.

Bell-McCaffrey Standard Errors

Building on the HC2 framework, Bell and McCaffrey (2002) developed a small-sample bias correction that further adjusts the degrees of freedom used for inference. Their approach uses a modified t-distribution to account for the fact that, in small samples, the standard HC2 estimator is still slightly too liberal.

In Stata 18+, you can access these corrections directly:

* bell-mccaffrey adjusted standard errors
	reg             lexp gnppc, vce(hc2, dfadjust)

The dfadjust sub-option applies the Bell-McCaffrey degrees-of-freedom correction, producing more conservative p-values that better control the Type I error rate in small samples.

Do Exercise 5 - Robust Standard Errors

Bootstrapping

If we aren’t satisfied with asymptotic approximations of our standard errors (like the HC suite of estimators), we can calculate them entirely empirically through Bootstrapping.

Bootstrapping involves pulling thousands of random sub-samples (with replacement) from our own dataset — each drawn sample is the same size $N$ as the original, but will include duplicates of some observations and entirely skip others. We run the regression on each drawn sub-sample, then examine the distribution of the resulting coefficients. The standard deviation of those empirical coefficients across the thousands of draws is, by definition, the standard error.

* bootstrap standard errors using 1,000 empirical draws
	reg             lexp gnppc, vce(bootstrap, reps(1000) seed(123))
	eststo          boot

Bootstrap is especially useful when:

• You’re working with a complex estimator where the analytical formula for standard errors is unknown or unreliable
• You want a non-parametric check on your robust or clustered SEs
• Your data has unusual features (heavy tails, skewness) that may violate assumptions underlying analytical SEs

Do Exercise 6 - Bootstrap Comparisons

Clustered Standard Errors

Robust standard errors handle heteroskedasticity but still assume that errors are independent across observations. In many research settings, errors within groups are correlated:

• Households in the same village experience the same weather
• Students in the same classroom share a teacher
• Observations from the same individual over time (panel data)

When observations within a group share unobserved factors, their errors are correlated. If you ignore this clustering, standard errors tend to be too small — you think you have more independent information than you actually do.

In Stata, we cluster with vce(cluster varname):

* cluster standard errors by region
	reg             lexp gnppc, vce(cluster region)
	eststo          clust

How many clusters are enough?

Clustered standard errors rely intrinsically on asymptotic theory — they require a large number of independent clusters to be properly scaled. Michler and Josephson (2022) review the literature and note that the practical minimum is around 40 to 50 clusters. Below that threshold, cluster-robust standard errors tend to severely under-reject (i.e., produce p-values that are too small), leading to false discoveries.

Our lifeexp dataset has only 3 broad regions. This makes standard vce(cluster region) completely unreliable. The solution is the Wild Cluster Bootstrap, which we cover next.

Wild Cluster Bootstrap

As Michler and Josephson (2022) detail, the modern solution for inference with a small number of clusters is the Wild Cluster Bootstrap. Developed by Cameron, Gelbach, and Miller (2008), this method resamples residuals (rather than entire observations) within clusters, maintaining the dependence structure of the data while generating reliable p-values.

We deploy this in Stata using David Roodman’s boottest package. You will need to download this from github

	* install boottest
		net install boottest, replace ///
            from("https://raw.githubusercontent.com/droodman/boottest/master/")

Put the above code just after the install code for csdid in the projectdo.do file.

* run the basic clustered regression
	reg             lexp gnppc, vce(cluster region)
	
* post-estimation wild cluster bootstrap for the gnppc variable
	boottest        gnppc

The boottest output will show you a bootstrapped p-value and confidence interval. Compare the bootstrapped p-value to the original cluster-robust p-value: in our case, with only 3 clusters, the cluster-robust p-value is likely far too optimistic (small), while the bootstrapped p-value provides a more honest assessment.

Do Exercise 7 - Wild Cluster Bootstrap

Randomization Inference

If standard errors are theoretically messy, or if you only have a single treated cluster (preventing any form of cluster inference), we can bypass parametric inference entirely using Randomization Inference (RI), deployed via the ritest package.

The logic: instead of making assumptions about the distribution of the error term, RI asks a simpler question: “If treatment had no effect, how unusual would our observed coefficient be?” It answers this by randomly reassigning the treatment indicator thousands of times, computing a coefficient for each placebo assignment, and locating the actual coefficient within this empirical distribution.

* create an arbitrary treatment variable for demonstration
	gen             rich = (gnppc > 10000)
	
* run randomization inference, permuting 'rich' 1,000 times
	ritest          rich _b[rich], reps(1000) seed(123): ///
	                    reg lexp rich

If our actual coefficient is larger than 95% of the placebos, we empirically reject the null hypothesis at $p<0.05$. This matches the spirit of Alwyn Young’s (2019) suggestions but preserves the regression framework.

Randomization inference is particularly useful when:

• You have very few clusters
• The treatment assignment mechanism is known (e.g., a randomized experiment)
• You want a fully non-parametric p-value with minimal assumptions
• You have a single treated unit (e.g., a single state adopts a policy)

Do Exercise 8 - Randomization Inference

Multiple Hypothesis Testing

Everything above addresses inference for a single hypothesis. But applied research almost always involves testing multiple outcomes or subgroups simultaneously. When you run 20 regressions at $\alpha = 0.05$, you expect one spurious rejection even if every null hypothesis is true. Multiple hypothesis testing (MHT) corrections address this inflation of false discoveries.

There are two distinct frameworks for thinking about the problem:

FWER is more conservative — it guards against any false positive. FDR is more permissive and powerful — it allows some false positives as long as the overall share stays below a threshold (typically 5%). The right choice depends on the cost of a false discovery in your context.

FWER corrections

The simplest FWER correction is the Bonferroni adjustment: multiply each p-value by the number of hypotheses tested (or equivalently, divide $\alpha$ by the number of tests). If you test $m$ hypotheses at $\alpha = 0.05$, the Bonferroni threshold is $0.05/m$.

The Holm (1979) step-down procedure is a strict improvement over Bonferroni. It orders the p-values from smallest to largest and applies progressively less severe penalties. It controls the FWER at the same level as Bonferroni but rejects at least as many hypotheses:

1. Order p-values: $p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}$
2. For hypothesis $j$, the adjusted p-value is: \(\tilde{p}_{(j)} = \max_{k \leq j} \min\big((m - k + 1) \cdot p_{(k)},\ 1\big)\)
3. Reject all hypotheses whose adjusted p-value $\leq \alpha$

Both Bonferroni and Holm are analytical — they require no resampling and can be computed by hand.

For a more powerful resampling-based FWER correction, List, Shaikh, and Xu (2019) recommend the Westfall-Young step-down procedure, implemented in Stata via wyoung (by Julian Reif):

* install wyoung
	ssc install    wyoung, replace

* test the effect of gnppc on multiple outcomes simultaneously
	wyoung          lexp popgrowth safewater, ///
	                    cmd(reg OUTCOMEVAR gnppc) ///
	                    familyp(gnppc) ///
	                    reps(1000) seed(123)

The wyoung output reports both the original p-value and the Westfall-Young adjusted p-value for each outcome. The adjusted p-values account for the correlation structure across outcomes (via bootstrapping), making them less conservative than Bonferroni when outcomes are correlated.

FDR corrections: Anderson sharpened q-values

When FWER corrections are too conservative (as they often are in exploratory analyses with many outcomes), the FDR framework offers a middle ground. Anderson (2008) developed a sharpened q-value procedure based on Benjamini, Krieger, and Yekutieli (2006) that adapts to the number of true rejections in the data:

• A q-value of 0.05 means that among all hypotheses with q-values $\leq 0.05$, we expect no more than 5% to be false discoveries.
• Unlike Bonferroni, sharpened q-values can sometimes be smaller than the original p-values when many hypotheses are rejected — the procedure “rewards” you for having many true effects.

Anderson’s procedure is implemented via a standalone do-file (fdr_sharpened_qvalues.do) rather than a Stata package. It takes a variable of p-values as input:

* step 1: run your regressions and collect p-values
	local outcomes  lexp popgrowth safewater
	local i = 1

	foreach y of local outcomes {
	    reg         `y' gnppc, robust
	    local       p`i' = r(table)[4,1]
	    local       ++i
	}

* step 2: store p-values in a dataset
	clear
	set             obs 3
	gen             pval = .
	replace         pval = `p1' in 1
	replace         pval = `p2' in 2
	replace         pval = `p3' in 3

* step 3: run anderson's procedure
	do              "$code/fdr_sharpened_qvalues.do"

Which correction should you use?

In practice, pre-analysis plans in development economics typically specify Westfall-Young or Anderson q-values for primary outcomes and use FDR corrections for exploratory or secondary analyses.

Quick reference

Summary

• Standard errors quantify uncertainty; getting them wrong invalidates all inference
• Uncertainty can be sampling-based (random sample from population) or design-based (random treatment assignment)
• Default robust (HC1) is unreliable in small samples — use HC2 or HC3 instead
• Bell-McCaffrey corrections further improve small-sample inference via degrees-of-freedom adjustment
• Clustered SEs account for within-group correlation but require 40+ clusters for reliability
• When clusters are few, use the Wild Cluster Bootstrap (boottest)
• Randomization Inference provides fully non-parametric p-values by permuting treatment assignment
• Multiple Hypothesis Testing requires corrections when testing many outcomes — use FWER (Bonferroni, Holm, Westfall-Young) or FDR (Anderson sharpened q-values)