Statistical Inference II: Pitfalls of hypothesis testing; confidence intervals/effect sizes
Pitfall 1: over-emphasis on p-values
Statistical significance does not guarantee clinical significance.
Example: a study of about 60,000 heart attack patients found that those admitted to the hospital on weekdays had a significantly longer hospital stay than those admitted to the hospital on weekends (p<.03), but the magnitude of the difference was too small to be important: 7.4 days (weekday admits) vs. 7.2 days (weekend admits).
Ref: Kostis et al. N Engl J Med 2007;356:1099-109.
Pitfall 1: over-emphasis on p-values
Clinically unimportant effects may be statistically significant if a study is large (and therefore, has a small standard error and extreme precision).
Pay attention to effect sizes and confidence intervals (see end of this lecture).
Pitfall 2: association does not equal causation
Statistical significance does not imply a cause-effect relationship.
Interpret results in the context of the study design.
Pitfall 3: data dredging/multiple testing
In 1980, researchers at Duke randomized 1073 heart disease patients into two groups, but treated the groups equally.
Not surprisingly, there was no difference in survival.
Then they divided the patients into 18 subgroups based on prognostic factors.
In a subgroup of 397 patients (with three-vessel disease and an abnormal left ventricular contraction) survival of those in "group 1" was significantly different from survival of those in "group 2" (p<.025).
How could this be since there was no treatment
(Lee et al. "Clinical judgment and statistics: lessons from a simulated randomized trial in coronary artery disease," Circulation, 61: 508-515, 1980.)
Pitfall 3: multiple testing
The difference resulted from the combined effect of small imbalances in the subgroups
Multiple testing
A significance level of 0.05 means that your false positive rate for one test is 5%.

下一页