Alpha levels (sometimes just called “significance levels”) are
used in hypothesis tests. Usually, these tests are run with an
alpha level of .05 (5%), but other levels commonly used are .01 and
.10.
The significance level, also denoted as alpha or α, is the
probability of rejecting the null hypothesis when it is true. For
example, a significance level of 0.05 indicates a 5% risk of
concluding that a difference exists when there is no actual
difference.
The common level of significance and the corresponding
confidence level are given below:
The level of significance 0.10 is related to the 90% confidence
level.
The level of significance 0.05 is related to the 95% confidence
level.
The level of significance 0.01 is related to the 99% confidence
level.
The actual meaning of a p-value of 0.001 is that there is a one
in a thousand probability of finding the result that you did, ‘just
by chance alone’ i.e. the result that you found is very rare to
have occurred just by chance only.
In hypothesis testing, when your p-value is less than the alpha
level you selected (typically 0.05), you'd reject the null
hypothesis in favor of the alternative hypothesis.
Let's say we do a 2-sample t-test to assess the difference
between the mean strength of steel from two mills. The null
hypothesis says the two means are equal; the alternative hypothesis
states that they are not equal.
If we get a p-value of 0.02 and we're using 0.05 as our alpha
level, we would reject the hypothesis that the population means are
equal.
If the significance level was set at 0.20,this means that the
probability of making a Type I error is 80%, assuming that the null
hypothesis is true.
Type I error is an error that takes place when the outcome is a
rejection of null hypothesis which is, in fact, true. Type II error
occurs when the sample results in the acceptance of null
hypothesis, which is actually false.
When the null hypothesis is true and you reject it, you make a
type I error. The probability of making a type I error is α, which
is the level of significance you set for your hypothesis test. An α
of 0.05 indicates that you are willing to accept a 5% chance that
you are wrong when you reject the null hypothesis.
When the null hypothesis is false and you fail to reject it,
you make a type II error. The probability of making a type II error
is β, which depends on the power of the test.