The application of measurement theory to tests in mathematics: a study of the goodness-of-fit of rasch model to the alis mathematics test

Abstract

The scores provided by the International Test of Developed Ability (ITDA) have been used as an alternative baseline for comparing the progress of students in the A-level Information System (ALIS) project of U.K. The responses of 26,964 examinees to the mathematics items of ITDA in year 2000 were fitted by using the Rasch model. Five subject groups (the population, 2 gender groups and 2 ability groups) and 25 random samples (5 from each group) were generated from the responses of the examinees. The unconditional maximum likelihood estimates of the item difficulty and examinee ability parameters for various groups/samples were produced by the RASCAL program. The scatterplots among different sets of sample item difficulty parameters reflected that the feature of item and ability invariance was not preserved in the groups of extreme abilities. The assumptions of unidimensionality, equal item discrimination, zero guessing factor and non-speededness were generally not supported in the two ability groups. In particular, the result indicated that the ITDA Mathematical Test might be a speeded test. It was quite interesting in this study to see that the item difficulty parameters and examinee abilities estimated from the Classical Test Theory (CTT) and those from the Rasch model were very comparable and both frameworks exhibit more or less the same feature in terms of invariance. On the other hand, more items were "found" unfit by the CTT method than the Rasch approach indicating that the former looks more sensitive to the lack of fit than the latter. To study the effect of speededness, the analysis was repeated with the last 11 items (which has the highest omits) deleted. Disappointingly, the results showed no significant improvement. Further research on the fitness of data with speed incorporated into the estimation of ability level is recommended.