If you run more than one test, your false positive rate will be higher than 5%.
Pitfall 3: multiple testing
If we compare survival of "treatment" and "control" within each of 18 subgroups, that's 18 comparisons.
If these comparisons were independent, the chance of at least one false positive would be…
Multiple testing
With 18 independent comparisons, we have 60% chance of at least 1 false positive.
Multiple testing
With 18 independent comparisons, we expect about 1 false positive.
Sources of multiple testing
a 2-year randomized trial where the efficacy of the treatment is evaluated by a Data Safety and Monitoring Board at 6 months, 1 year, and 18 months
Multiple looks at the data during sequential interim monitoring
a study where a walking test is administered at 1 months, 3 months, 6 months, and 1 year
Multiple time points for the outcome (repeated measures)
an observational study where the data analyst tests multiple different definitions for "moderate drinking" (e.g., 5 drinks per week, 1 drink per day, 1-2 drinks per day, etc.)
Multiple definitions for the exposures and outcomes
a randomized trial that tests the efficacy of an intervention in 20 subgroups based on prognostic factors
Subgroup analyses
an observational study with 40 dietary predictors or a trial with 4 randomization groups
Multiple predictors
a cohort study looking at the incidence of breast cancer, colon cancer, and lung cancer
Multiple outcomes
Example
Source
Results from Class survey…
My research question was to test whether or not being born on odd or even days predicted anything about your future.
I discovered that people who born on odd days wake up later and drink more alcohol than people born on even days; they also have a trend of doing more homework (p=.04, p<.01, p=.09).
Those born on odd days wake up 42 minutes later (7:48 vs. 7:06 am); drink 2.6 more drinks per week (1.1 vs. 3.7); and do 8 more hours of homework (22 hrs/week vs. 14).

上一页下一页