Question

1. If you run a hypothesis test several times and the conclusion is always fail to reject, should you conclude the null hypothesis is correct?

PLEASE SHOW ALL WORK CLEARLY AND UNDERSTANDABLE THANK YOU

Answer 1

Say we run the simulation n times and the pvalue of every time is p with a standard deviation of s

Now we need to check how far is the confidence level of pvalue far away from the level of significance.

Say we get a pvalue CI as [a,b] and we are checking the conclusion at 5% level. Now if 0.05>>>b, then there is a high chance that we can think the null hypothesis could be correct. However, if we get something like 0.05 is approx b, we cannot say that the null hypothesis is always correct. It could be a case that our sampling has been biased till now.