This document describes a repeated-measures study conducted to measure the consistency of the Speaknow assessment in subsequent administrations. The results demonstrate that overall performance is consistent across administrations.
Test-retest reliability is one measure of an exam’s consistency. Assuming that language skills are fairly stable within a short period of time, one would expect that performance on the same test under the same conditions within a short time-frame would be similar.
This study was initially conducted to measure the efficacy of an instructional intervention by an education non-profit in Israel. The study was conducted with 64 students in 9th and 10th grades (ages 14-16) over a period of two months who completed two iterations of the Speaknow assessment. Nineteen exams were removed from the study due to difference of over 50% in the length of the responses, or failure to answer at least two questions.
The participants were all high school students enrolled in an instructional program designed to improve spoken English proficiency. Participation in the program was voluntary and subjects were not compensated for their participation.
Overall, there was moderate stability in the two test administrations. The intraclass correlation (3,1) was .74, lower 0.572 upper 0.848 with a power of 90%, p<.05.
Given that the students were involved in lessons specifically intended to improve their English, the results show strong consistency. 91% of the students had a change in their scores of less than a full level. Even with the differences between the two test administrations, the rates of stability and correlation provide positive evidence of stability of the test.
This study shows consistency of results across a large period of time, and with varying test conditions. It is expected that with more consistent testing conditions, the exam would show even greater consistency across administrations.
Tsagari, Dina & Banerjee, Jayanti. (2016). Handbook of Second Language Assessment.
Vilagut G. (2014) Test-Retest Reliability. In: Michalos A.C. (eds) Encyclopedia of Quality of Life and Well-Being Research. Springer, Dordrecht