ERIC Number: EJ1026316
Record Type: Journal
Publication Date: 2014
Abstractor: As Provided
The Impact of Examinee Performance Information on Judges' Cut Scores in Modified Angoff Standard-Setting Exercises
Margolis, Melissa J.; Clauser, Brian E.
Educational Measurement: Issues and Practice, v33 n1 p15-22 Spr 2014
This research evaluated the impact of a common modification to Angoff standard-setting exercises: the provision of examinee performance data. Data from 18 independent standard-setting panels across three different medical licensing examinations were examined to investigate whether and how the provision of performance information impacted judgments and the resulting cut scores. Results varied by panel but in general indicated that both the variability among the panelists and the resulting cut scores were affected by the data. After the review of performance data, panelist variability generally decreased. In addition, for all panels and examinations pre- and post-data cut scores were significantly different. Investigation of the practical significance of the findings indicated that nontrivial fail rate changes were associated with the cut score changes for a majority of standard-setting exercises. This study is the first to provide a large-scale, systematic evaluation of the impact of a common standard setting practice, and the results can provide practitioners with insight into how the practice influences panelist variability and resulting cut scores.
Descriptors: Cutting Scores, Standard Setting (Scoring), Data, Licensing Examinations (Professions), Influences, Judges
Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: firstname.lastname@example.org; Web site: http://www.wiley.com/WileyCDA/
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Authoring Institution: N/A