Multiple Testing with Many Covariates: My Master Thesis

My master thesis (Heidelberg, 2022) was written in Wolfgang Huber’s group at EMBL, closely supervised by him and by Nikolaos Ignatiadis, with Jan Johannes co-supervising at the university. The method it builds on is theirs, and the idea for the extension was Nikolaos’s. My part was the proofs, the code and the tests.

The method: IHW

Biology often tests millions of hypotheses at once, and the Benjamini–Hochberg procedure keeps the expected share of false discoveries, the false discovery rate (FDR), below a level $\alpha$. Often, though, each test comes with side information, a covariate, that hints at how likely it is to be a real effect. Independent Hypothesis Weighting (IHW), by Ignatiadis, Huber and co-authors, turns the covariate $X_i$ into a weight $W_i$ and rejects when

$$P_i \le W_i\, t,$$

so hypotheses with a promising covariate get an easier bar. The weights are learned on the other folds of the data, never on the p-value they apply to, and Ignatiadis and Huber (2021) proved that this controls the FDR asymptotically, and exactly for a censored variant.

To learn the weights, IHW cuts the covariate into bins by quantile. That works well for one covariate; with several, the bins multiply and each holds few hypotheses.

The extension: IHW-Forest

Nikolaos proposed letting a random forest draw the bins instead, and gave me a detailed description of it before I started. The trees regress an indicator of a large p-value on the covariates, so their leaves group hypotheses with a similar estimated share of true nulls, stay populated, and largely skip covariates that carry no signal.

With made-up hypotheses whose small p-values gather in one corner, the difference looks like this:

What I did

• Wrote out the proof that IHW controls the FDR asymptotically, following Ignatiadis and Huber (2021), with the steps a journal article leaves out.
• Implemented IHW-Forest in a fork of the IHW R package (pull request), with help from members of the group.
• Compared it with other methods in simulations, and revisited the histone QTL analysis of Ignatiadis and Huber (2021): about 16 billion tests, now with up to eleven covariates instead of one. In the three covariate sets both could handle, IHW-Forest made more discoveries than quantile binning, and it still ran on all eleven, where quantile binning could not. The code is in ihw-forest-paper.

The thesis

Open the PDF in a new tab