Lecture 7: Practical Issues in Data Analysis

Initial setup

Today

We’ll consider two common problems that complicate the interpretation of statistical analyses.

  • What happens when we ask many statistical questions using the same data?
  • How can we limit the number of misleading findings that result from repeated testing?
  • What can go wrong when some of the data we intended to observe are missing?
  • How does the reason data are missing affect what we can learn from the observed data?

These issues require us to think carefully about what our data can—and cannot—tell us.

Multiple testing

Suppose we conduct a randomized study comparing a treatment with a control group.

We have 5 different outcomes and a separate research question for each:

Does treatment affect this outcome?

For each outcome \(j\):

\[\large H_{0j}: \mu_{j1} - \mu_{j0} = 0, \qquad j=1,\ldots,5\]

where \(\mu_{j1}\) and \(\mu_{j0}\) are the population means for outcome \(j\) in the treatment and control groups.

Suppose all 5 null hypotheses are true — treatment has no effect on any of the outcomes.

For any one test with \(\alpha=0.05\), the probability of a Type I error is 5%.

But we are conducting 5 tests.

How likely are we to find at least one statistically significant effect and conclude that the treatment “works”?

Simulating the study

Let’s simulate this study under the assumption that all 5 null hypotheses are true.

For each outcome, we will generate data from the same population distribution for the treatment and control groups:

\[ \begin{align} Y_{11},Y_{10} &\sim N(25,3^2) \\ Y_{21},Y_{20} &\sim N(21,4^2) \\ Y_{31},Y_{30} &\sim N(17,2^2) \\ Y_{41},Y_{40} &\sim N(10,1.5^2) \\ Y_{51},Y_{50} &\sim N(31,2.5^2) \\ \end{align} \]

We will then test for a treatment effect separately for each of the 5 outcomes.

One simulated sample doesn’t tell us how often this happens. What if we repeat the sampling many times?

Let’s repeatedly sample many times . . .

In every simulated smaple:

  • all 5 null hypotheses are true
  • we test each outcome using \(\alpha=0.05\)
  • we record how many null hypotheses are rejected
  • In about 22% of samples, at least one of the 5 tests is statistically significant.

  • Each individual test still has a 5% Type I error rate.

  • But across the family of 5 tests, the probability of making at least one Type I error is much larger than 5%.

  • The probability of making at least one Type I error across the family is called the family-wise error rate (FWER):

\[\large \text{FWER} = P(\text{at least one Type I error})\]

For 5 independent tests, the probability that we make no Type I errors is

\[\large 0.95^{5}\approx0.773\]

so

\[ \large P(\text{at least one Type I error}) =1-0.95^{5} \approx 0.226. \]

Visualizing the problem

The figure shows 100 simulated samples. In each sample:

  • all 5 null hypotheses are true
  • each vertical line is a 95% confidence interval
  • a red interval does not include 0 and represents a Type I error
  • shaded columns identify studies with at least one Type I error

The expected Type 1 error rate with 5 outcomes is:

Controlling the family-wise error rate

We would like the probability of making at least one Type I error across all 5 tests to be no more than 5%.

Can we change the significance level for each individual test so that the FWER is controlled at 5%?

One simple approach is the Bonferroni correction.

Instead of testing each hypothesis using \(\alpha=0.05\), divide \(\alpha\) by the number of tests:

\[\large \alpha^*=\frac{\alpha}{m}\]

where \(m\) is the number of tests.

For our 5 outcomes:

\[\large \alpha^*=\frac{0.05}{5}=0.01\]

We now reject an individual null hypothesis only when \(p < 0.01\):

The Bonferroni correction controls the FWER because each individual test now has a Type I error probability of only 0.01:

\[\large \text{FWER} \leq 5(0.01) = 0.05\]

The \(\leq\) is important:

  • Bonferroni guarantees that the FWER is no greater than 5% — it does not have to be exactly 5%.
  • Unlike \(1-(0.95)^{5}\), Bonferroni does not require the tests to be independent.
  • This is useful because outcomes in the same study are often correlated.

But there is a tradeoff:

By making it harder to reject each null hypothesis, Bonferroni also makes it harder to detect real effects.

The Holm procedure

Suppose we conduct five tests and obtain these ordered p-values:

\[\large 0.003,\quad 0.009,\quad 0.015,\quad 0.031,\quad 0.20\]

With Bonferroni, every p-value is compared with

\[\large \frac{0.05}{5}=0.01.\]

Only the first two null hypotheses would be rejected.

Holm starts with the same threshold, but becomes progressively less stringent:

  • compare the smallest p-value with \(0.05/5\)
  • if rejected, compare the next with \(0.05/4\)
  • continue until a p-value fails to meet its threshold
p-value Holm threshold Decision
0.003 \(0.05/5=0.010\) Reject
0.009 \(0.05/4=0.0125\) Reject
0.015 \(0.05/3=0.0167\) Reject
0.031 \(0.05/2=0.025\) Stop
0.200 — Do not reject

Once a p-value fails to meet its threshold, the procedure stops.

Here, Bonferroni rejects 2 hypotheses; Holm rejects 3.

Holm controls the FWER just as Bonferroni does, but can have greater power.

Once the p-values have been adjusted, we compare them with the original significance level. In this case:

Reject the null hypothesis if the adjusted p-value is less than 0.05.

When all of the null hypotheses are true, Holm and Bonferroni have the same probability of making at least one rejection. Holm gains power after the first rejection by using progressively less stringent thresholds.

For controlling the FWER, Holm is generally preferred to Bonferroni because it provides the same error control while being at least as powerful.

Connecting back to ANOVA

In ANOVA, we may want to compare all pairs of group means.

With four groups, there are six pairwise comparisons:

\[\large \mu_1-\mu_2,\quad \mu_1-\mu_3,\quad \mu_1-\mu_4,\quad \mu_2-\mu_3,\quad \mu_2-\mu_4,\quad \mu_3-\mu_4\]

These six comparisons form another family of hypothesis tests, so we again need to consider the FWER.

  • Bonferroni and Holm are general-purpose methods for controlling the FWER.

  • The Tukey method is designed specifically for the situation in which we want to make all pairwise comparisons among a set of group means.

  • Rather than treating the comparisons as unrelated tests, Tukey takes advantage of their common structure.

  • Without going into the details, Tukey uses the studentized range distribution, which essentially asks:

If all the population means were equal, how far apart might the most separated sample means be simply because of sampling variability?

  • By taking advantage of the structure of all the pairwise comparisons, Tukey controls the FWER while avoiding the need to apply a general-purpose correction to each comparison separately.

When our goal is to compare all pairs of group means, Tukey is a natural choice. If we are interested in only a particular set of comparisons, a general method such as Holm may be more appropriate.

False discovery rate

So far, we have focused on the family-wise error rate:

FWER: How likely are we to make any false discoveries?

But there is another way to think about errors when we conduct many tests:

FDR: What is the expected proportion of our discoveries that will be false?

Suppose we are testing 10,000 genes to identify genes that may be associated with a disease.

Controlling the FWER means trying to keep the probability of making even one false discovery among the 10,000 tests below 5%.

Using Bonferroni, we would require

\[\large p < \frac{0.05}{10{,}000}=0.000005.\]

That is an extremely stringent threshold — and may be more stringent than we need.

Instead, we might be willing to tolerate some false discoveries, as long as they represent only a small proportion of all the discoveries we make.

False discovery proportion

Among the results we call statistically significant — our discoveries — some may be:

  • true discoveries — there really is an effect
  • false discoveries — Type I errors

For a particular sample, the false discovery proportion (FDP) is

\[\large \text{FDP} = \frac{\text{number of false discoveries}} {\text{total number of discoveries}}.\]

For example, if 8 of 100 discoveries are false:

\[\large \text{FDP}=\frac{8}{100}=0.08.\]

But FDP describes what happened in one particular sample.

The false discovery rate (FDR) is the expected FDP across repeated samples:

\[\large \text{FDR}=E(\text{FDP}).\]

This gives us two different ways of thinking about errors across repeated studies:

  • FWER: How likely are we to make any false discoveries?

  • FDR: What is the expected proportion of our discoveries that will be false?

Let’s see what happens if we do not adjust for multiple testing, and simply use (p<0.05) for every test.

What false discovery rate does this testing procedure produce?

Would the false discovery proportion always be about 39%?

Repeating this experiment

Now let’s repeat the sampling process many times.

For each sample, we calculate the false discovery proportion.

The false discovery rate is the expected FDP across repeated samples:

\[\large \text{FDR}=E(\text{FDP})\]

With no adjustment for multiple testing, the average FDP can be quite large.

Controlling the false discovery rate

Can we control the proportion of our discoveries that are false without being as stringent as controlling the probability of even one false discovery?

The Benjamini-Hochberg (BH) procedure is designed to do just that.

Let’s first see what it does before looking at how it works.

We have 1,000 tests:

  • 900 have no true population difference
  • 100 have a true population difference of 1

First, generate the data and calculate the p-value for each test:

How many discoveries would we make using the usual (p<0.05) criterion?

Now apply the Benjamini-Hochberg procedure:

Let’s look at some of the smallest original p-values and their BH-adjusted values:

Using BH, we call a result statistically significant when its BH-adjusted p-value is less than 0.05:

Now we can ask how many of these discoveries are true and how many are false:

Finally, the false discovery proportion in this particular sample is

The Benjamini-Hochberg procedure

The Benjamini-Hochberg (BH) procedure controls the false discovery rate.

Like Holm, BH begins by ordering the p-values from smallest to largest:

\[\large p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}.\]

Suppose we have five tests and want to control the FDR at 5%.

BH compares each ordered p-value with

\[\large \frac{i}{m}(0.05),\]

where \(i\) is the position of the p-value in the ordered list.

For five tests, the thresholds are:

\(i\) BH threshold
1 \((1/5)(0.05)=0.01\)
2 \((2/5)(0.05)=0.02\)
3 \((3/5)(0.05)=0.03\)
4 \((4/5)(0.05)=0.04\)
5 \((5/5)(0.05)=0.05\)

BH finds the largest p-value that satisfies its threshold:

\[\large p_{(i)} \leq \frac{i}{m}(0.05).\]

\(i\) p-value BH threshold
1 0.008 0.010 ✓
2 0.021 0.020
3 0.027 0.030 ✓
4 0.046 0.040
5 0.200 0.050

The largest p-value satisfying the criterion is \(p_{(3)}=0.027\), so BH rejects the first three null hypotheses.

Unlike Holm, BH does not stop when an individual p-value fails its threshold.

Because the BH threshold increases with \(i\), it can be less stringent than methods designed to prevent even a single false discovery.

Here, the largest p-value satisfying its BH threshold is the third:

\[ \large p_{(3)}=0.027 < \frac{3}{5}(0.05)=0.03. \]

BH therefore rejects the hypotheses corresponding to the three smallest p-values.

BH finds the largest \(i\) for which \(p_{(i)}\leq(i/m)\), and rejects the first \(i\) hypotheses.

Unlike FWER control, the goal is not to avoid every false discovery — it is to control the expected proportion of discoveries that are false.

Does BH actually control the FDR?

The FDP describes what happened in one particular sample. What happens if we generate many possible samples and apply BH each time?

Across repeated samples, the average FDP — the false discovery rate — is about 5% or less.

Controlling the FDR at 5% means keeping the expected false discovery proportion at or below 5% — not making exactly 5% of the discoveries false.

Missing data

Ideally, we would observe all of the measures we planned to collect for every participant.

But in real studies, data may be missing:

  • some participants may be missing the outcome
  • others may be missing baseline characteristics or other covariates
  • some may be missing both

When does missing data create a problem for our analysis?

Focusing on missing outcomes

To understand the basic issues, we’ll focus on a common situation:

  • baseline information is available for everyone
  • the outcome is missing for some participants

Suppose we are interested in estimating the average physical functioning score six months after enrollment in a population.

We have a representative sample of 1,000 participants.

For everyone, we measure physical functioning at baseline. Six months later, we attempt to measure physical functioning again.

Generating the complete data

Suppose baseline physical functioning is standardized, with mean 0 and standard deviation 1.

Participants with better physical functioning at baseline tend to have better physical functioning six months later.

We generate:

  • standardized baseline physical functioning \(X\)
  • six-month physical functioning \(Y\)
  • a population mean six-month score of 50
  • a 5-point higher expected six-month score for each 1-SD higher baseline score

\[\large X_i \sim N(0,1) \]

\[\large Y_i = 50 + 5X_i + \epsilon_i, \qquad \epsilon_i \sim N(0,10) \]

Because baseline physical functioning has a population mean of 0, the population mean six-month score is 50.

Our complete data

Baseline and six-month physical functioning

The dashed line shows the true population mean of 50.

The blue line shows the mean in our sample.

What happens when some of the six-month outcomes are missing?

What if some of the six-month outcomes disappear randomly?

Suppose each participant has a 30% chance of having a missing six-month physical functioning score.

The probability of missingness is unrelated to:

  • baseline physical functioning
  • six-month physical functioning

The blue line now shows the mean six-month physical functioning score among participants whose outcomes are observed.

Re-sampling repeatedly

Now let’s repeat the sampling many times.

In each simulated study, we estimate the population mean six-month physical functioning score using:

  • the complete data
  • the observed data after 30% of outcomes are missing

On average, both analyses recover the true population mean of 50.

But the estimates based on the incomplete data vary more from study to study:

Randomly losing outcomes costs us information and precision, but does not systematically shift the estimate of the population mean.

This type of missingness is called Missing Completely at Random (MCAR):

The probability that a value is missing does not depend on any observed or unobserved data.

What if missingness depends on something we observed?

Now suppose the probability that the six-month outcome is missing depends on the participant’s baseline physical functioning.

Participants with lower baseline physical functioning are more likely to have a missing six-month outcome.

Because baseline physical functioning is observed for everyone, we know something about why the outcome is missing.

Re-sampling

Because baseline physical functioning predicts six-month physical functioning, systematically losing participants with lower baseline scores may also change the mean outcome among those we observe.

When missingness depends on an observed variable that is related to the outcome, simply analyzing the observed outcomes can produce a biased estimate of the population mean.

This type of missingness is called Missing at Random (MAR):

The probability that a value is missing may depend on observed data, but — after accounting for those observed data — does not depend on the missing value itself.

Using the information we observed

Under MAR, we have information that helps explain why some outcomes are missing.

In our example:

  • baseline physical functioning is observed for everyone
  • baseline physical functioning predicts six-month physical functioning
  • baseline physical functioning also predicts whether the outcome is missing

Can we use baseline physical functioning to recover the population mean?

First, we estimate the relationship between baseline and six-month physical functioning using participants whose outcomes were observed:

We can then use this model to predict the six-month outcome for everyone in our sample based on their observed baseline physical functioning:

Instead of averaging only over the selected group with observed outcomes, we used information from the full sample to estimate the population mean.

This is one example of a model-based approach to handling missing data under MAR.

Imputation

Another approach is to fill in the missing values using information from the observed data.

This is called imputation.

A simple approach might replace a missing value with:

  • the mean of the observed values
  • a value predicted from other observed characteristics

But a single imputed value treats something we do not know as though we know it.

Multiple imputation recognizes that the missing value is uncertain.

Instead of filling in one value, we generate several plausible values, creating several completed datasets.

We analyze each dataset and combine the results, allowing the final uncertainty to reflect both:

  • sampling variability
  • uncertainty about the missing values

The validity of imputation still depends on assumptions about why the data are missing.

What if missingness depends on what we did not observe?

Now suppose the probability that a six-month physical functioning score is missing depends on the value of the six-month score itself.

Participants with lower six-month physical functioning are more likely to have a missing outcome.

Here, participants with lower six-month physical functioning are systematically more likely to be missing, inflating our estimate of the population mean.

But there is an important difference from our previous example:

For participants whose outcomes are missing, we do not observe the information that explains why they are missing.

Re-sampling

Let’s again repeat the sampling process many times.

When the probability of missingness depends on the missing outcome itself, the observed outcomes may systematically differ from the outcomes we did not observe.

This type of missingness is called Missing Not at Random (MNAR):

Even after accounting for the observed data, the probability that a value is missing depends on the value that is missing.

Can we tell whether data are MNAR?

Usually, we cannot determine from the observed data whether missingness is MAR or MNAR.

For participants with missing outcomes, we do not know what those outcomes would have been. Therefore, we cannot directly determine whether missingness depends on those unobserved values.

However, a MAR assumption may be more reasonable when we have good observed information about:

  • factors that predict whether data are missing
  • factors that predict the missing outcome

For example, if participants with poorer physical functioning are more likely to have missing outcomes, detailed measures of their physical functioning before the outcome is missing may help make an MAR assumption more plausible.

We cannot prove that data are MAR, but collecting and using variables that help explain both the outcome and missingness can make the assumption more reasonable.

Can observed information solve the problem?

We can try the same model-based approach we used under MAR:

Under MNAR, observed variables may help, but they cannot fully account for missingness that still depends on the unobserved value itself.

What can we do about MNAR?

Because we cannot know the missing outcomes, there is no purely data-driven solution to MNAR.

Instead, we can conduct sensitivity analyses:

Make different assumptions about the missing outcomes and ask whether our conclusions change.

More formal statistical methods for MNAR follow the same basic principle: they require assumptions about the data we did not observe.

Missing data: putting it together

  • MCAR: Missingness does not depend on observed or unobserved data.
    • Analyzing the observed data only may reduce precision; in our example, it does not systematically shift the estimate.
  • MAR: Missingness can depend on information we observed.
    • We can use that observed information to help account for the missing data.
  • MNAR: Missingness still depends on information we did not observe.
    • Observed information may help, but additional assumptions are needed to account fully for the missing data.

The consequences of missing data depend on why the data are missing — not just on how much data are missing.