Question

A Methods and Statistics student, Chris, has an accountant for a mother. She has
been listening to Chris tell her about Type I errors and α = .05. She responds: “.05?
That means you’re making an error one time in twenty! If I made errors 5 percent of
the time I’d be out of a job so fast it would make your head spin! Those psychologists
need to shape up and set their α level to something more reasonable, say .001 or
.0001!” Is Chris’s mother’s interpretation that an alpha level of 5% means you will
commit errors 5% of the time correct? Explain (2 points), and in addition making it
clear what the advantages and/or disadvantages are of doing what Chris’s mother
suggests (3 points).

Answer 1

The significance level, also denoted as alpha or α, is the probability of rejecting the null hypothesis when it is true. For example, a significance level of 0.05 indicates a 5% risk of concluding that a difference exists when there is no actual difference.

Thus, Chris’s mother’s interpretation that an alpha level of 5% means you will commit errors 5% of the time is correct.

• Setting a significance level (before doing inference) has the advantage that Chris is not tempted to choose a cut-off on the basis of what he hopes is true.
• It has the disadvantage that it neglects that some p-values might best be considered borderline. This is one reason why it is important to report p-values when reporting results of hypothesis tests.