Question

Compare and contrast the following terms: test-retest reliability with inter-rater reliability. content validity with both predictive validity and construct validity. internal validity with external validity. Can a test be reliable without being valid? Can a test be valid without being reliable? Explain.

Answer 1

• Test-retest reliability is a measure of reliability obtained by administering the same test twice over a period of time to a group of individuals. The scores from Time 1 and Time 2 can then be correlated in order to evaluate the test for stability over time. Example: A test designed to assess student learning in psychology could be given to a group of students twice, with the second administration perhaps coming a week after the first. The obtained correlation coefficient would indicate the stability of the scores.
• An assessment or test of a person should give the same results whenever you apply the test.Test-retest reliability evaluates reliability across time.
• Inter-rater reliability is a measure of reliability used to assess the degree to which different judges or raters agree in their assessment decisions. Inter-rater reliability is useful because human observers will not necessarily interpret answers the same way; raters may disagree as to how well certain responses or material demonstrate knowledge of the construct or skill being assessed.
• When multiple people are giving assessments of some kind or are the subjects of some test, then similar people should lead to the same resulting scores. It can be used to calibrate people, for example those being used as observers in an experiment.Inter-rater reliability thus evaluates reliability across different people.
• Example: Inter-rater reliability might be employed when different judges are evaluating the degree to which art portfolios meet certain standards. Inter-rater reliability is especially useful when judgments can be considered relatively subjective. Thus, the use of this type of reliability would probably be more likely when evaluating artwork as opposed to math problems.
• Concurrent validity measures the test against a benchmark test and high correlation indicates that the test has strong criterion validity.
• This measures the relationship between measures made with existing tests. The existing tests is thus the criterion.For example a measure of creativity should correlate with existing measures of creativity.
• Predictive validity is a measure of how well a test predicts abilities. It involves testing a group of subjects for a certain construct and then comparing them with results obtained at some point in the future.
• This measures the extent to which a future level of a variable can be predicted from a current measurement. This includes correlation with measurements made with different instruments.
  For example, a political poll intends to measure future voting intent or College entry tests should have a high predictive validity with regard to final exam results.
• Content validity is the estimate of how much a measure represents every single element of a construct.Content validity occurs when the experiment provides adequate coverage of the subject being studied. This includes measuring the right things as well as having an adequate sample. Samples should be both large enough and be taken for appropriate target groups.
• The perfect question gives a complete measure of all aspects of what is being investigated. However in practice this is seldom likely, for example a simple addition does not test the whole of mathematical ability.Content validity is related very closely to good experimental design. A high content validity question covers more of what is sought.
• External validity occurs when the causal relationship discovered can be generalized to other people, times and contexts.Correct sampling will allow generalization and hence give external validity.
• Internal validity occurs when it can be concluded that there is a causal relationship between the variables being studied. A danger is that changes might be caused by other factors.It is related to the design of the experiment, such as in the use of random assignment of treatments.The less the confounding in an experiment, the higher is its internal validity.
• The basic difference between internal and external validity is that the former talks about the relationship between variables whereas the latter is concerned with the universality of the results.
• Internal validity is concerned with control of extraneous variable, whereas external validity stresses on the applicability of the outcome to the practical situations.
• Internal Validity either addresses or eliminates alternative explanation for the result. In contrast, external validity is used to generalize the outcome.
• If data are valid, they must be reliable. If people receive very different scores on a test every time they take it, the test is not likely to predict anything. However, if a test is reliable, that does not mean that it is valid. For example, we can measure strength of grip very reliably, but that does not make it a valid measure of intelligence or even of mechanical ability. Reliability is a necessary, but not sufficient, condition for validity.
• A test can be reliable without being valid. However, a test cannot be valid unless it is reliable.For example,If the scale is reliable it tells you the same weight every time you step on it as long as your weight has not actually changed. However, if the scale is not working properly, this number may not be your actual weight. If that is the case, this is an example of a scale that is reliable, or consistent, but not valid. For the scale to be valid and reliable, not only does it need to tell you the same weight every time you step on the scale, but it also has to measure your actual weight.