Question

In the course of the thesis work, a student develops a new approach for the solution of a problem (here referred to as method B). The current state-of-the-art approach, method A, is well published in the literature and has been applied to a large standard problem set where its average performance was discovered to be (and published in the main paper by the developers as) 7 with a standard deviation of 3 across the different problems in the problem set. In addition to the publication, the developers of method A also provide their code for anyone to be able to experiment with and the student decides to pick a random set of 15 problems from the standard problem set and apply both methods to these problems, resulting in the following performance numbers for method A: {8, 3, 10, 8, 11, 4, 6, 4, 12, 4, 5, 10, 6, 2, 10}, and the following performance numbers for the student’s method B: {9, 5, 9, 10, 15, 4, 7, 4, 12, 7, 8, 10, 6, 4, 12}. Looking at this data, the student discovers that it seems that method B outperforms method A and sets out to prove this using significance testing with a two-tailed 5% significance threshold. Given that both published performance results as well as the student’s experimental results are available, a number of tests can be performed.

Evaluate the results in terms of the hypothesis that method B has a higher performance than method A. List all the steps (and formulas) involved in the test and what the result implies for the significance of the hypothesis.

Answer 1

Null Hypothesis, H₀:The mean performances of both the methods are the same (

Alternative Hypothesis,H_a:The mean perfomance of method B is higher that mean performance of method A

Since the two methods are applied on the same data, paired t test can be used

The test statistic is given as

The value of t is 3.4125

The critical value is 1.761

Since the calculated value is greater than the critical value , we reject the null hypothesis and conclude that method B has higher performance than method A at 5% level of significance