Unit Two: Understanding Validity, Reliability, and Classical Test Theory (CTT)
Return to: Introduction to Educational Assessment
Unit Overview
In this unit, we are going to talk about:
1, What is validity and reliability, and how to evaluate an assessment with these two definitions;
2, how Classic Test Theory (CTT) works.
Lesson Four: Validity and Reliability
First, please read these two articles
Validity
In the current conception, validity is the degree to which evidences and theories support the adequacy and appropriateness of interpreting test scores and making certain decision according to the test score (Messick, 1995; American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014). The process of gathering evidences to support the appropriateness of interpreting test scores and using the scores to make decision is test validation. Validation is the joint responsibility of both test developers and test-takers (American Educational Research Association et al., 2014). The current definition of validity is a unified concept, under the construct framework, which addresses both the score meaning and the score usage (Messick, 1995). The modern validity can be analyzed with six different aspects, which are content, process, structural, generalizability, external, and consequential aspects (Messick, 1995).
Reliability
Reliability is the overall consistency of a measure. A measure is said to have a high reliability if it produces similar results under consistent conditions. "It is the characteristic of a set of test scores that relates to the amount of random error from the measurement process that might be embedded in the scores. Scores that are highly reliable are accurate, reproducible, and consistent from one testing occasion to another. That is, if the testing process were repeated with a group of test takers, essentially the same results would be obtained (Reliability, 2018).
These two videos might help you understand validity and reliability better:
Reliability & Validity Explained
Lesson Five:Classical Test Theory (CTT)
Please read the article below
Classical Test Theory (CTT) has over 80 years history, whose name coming from the comparison with "modern test theory" (i.e. Item Response Theory).
The main statement of CTT is:
X (the raw score) = T (a true component) + E (a random error)
For example, a student took three tests (all of them are measuring 5th grade Math knowledge, and all of their full score is 100). The student got 88, 90, 94 on the three different tests. We can say the student is doing well, because all the scores are close to each other, all of them are not bad, and the student' s performance is stable. Also between 88 and 94, there might be a point, maybe 92, that reflects the student' s true Math ability. However, the scores vary, which might because the student felt good or sick on the test days.
Another example, a student took GRE 3 times, and his quantitative scores are 155,159,157. All three scores reflect this student' s ability and some random error, including not eating breakfast, being sick, something good happened before test, and broke up with girlfriend/boyfriend before test.
Related Resources
Introduction to Classical Test Theory
To move to the next unit, please click the link below:
Reference
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington D.C.: American Educational Research Association.
Classical test theory. (2018, May 01). Retrieved May 01, 2018, from https://en.wikipedia.org/wiki/Classical_test_theory
Kline, T. J. B. (2005). Classical test theory: assumptions, equations, limitations, and item analyses. Psychological Testing: A practical approach to design and evaluation, (pp. 91-106). Thousand Oaks, CA: Sage Publication
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American psychologist, 50(9), 741.
Reliability (statistics). (2018, May 01). Retrieved May 01, 2018, from https://en.wikipedia.org/wiki/Reliability_(statistics)
Traub, R. E., & Rowley, G. L. (1991). Understanding reliability. Educational measurement: Issues and practice, 10(1), 37-45.