Journal of Applied Measurement
A publication of the Department of Educational Psychology and Counseling
National Taiwan Normal University
Volume 24, Issue 1/2 (2023)
Special Issue: Interfacing Between Test Developers and Their Constituents
FOCUS ARTICLE
Clarifying Inputs and Outputs of Cognitive Assessments
William D. Schafer
University of Maryland
This paper is about actions testing programs can take to improve communications about cognitive tests with interested stakeholders, including what they measure and the uses of the results. Concern about testing in the United States has often centered around focusing too much on rote learning and narrowing the curriculum to only tested concepts and procedures, though more recently, there is concern as well about perceived bias in outcomes. I agree that acceptance of cognitive assessments will be more likely if we can improve public understanding of them and their usefulness in ways that go beyond the limited (although important) concerns about emphasis and fairness. But what can the assessment industry do to enhance transparency? I suggest here that (1) greater attention to and clarification about the domains of major standardized tests in relation to curriculum and (2) access to interactive devices to interpret their outcomes would help the public focus on what tests actually are and what they can do. I suggest improved understandings of what tests assess, how they assess, and what the results imply for both individuals and groups can move public conversation toward ways of meaningfully addressing (and studying) assessment concerns. I suggest (1) an approach to clarifying the domain of an assessment, useful for both test developers and examinees as well as other audiences, such as instructors; and (2) a web-based means for users to tailor contextualization of results for persons and for groups using both norm- and criterion-referenced information. Although these two concepts are presented only to convey feasibility, I suggest that using processes like them will foster better-focused tests and enable more effective use of the results.
Keywords: heuristics, domain description, test interpretation
Citation:
Schafer, W. D. (2023). Clarifying inputs and outputs of cognitive assessments. Journal of Applied Measurement, 24(1/2),1–8.
COMMENTARIES
A Review of Clarifying Inputs and Outputs of Cognitive Assessments
Peter Behuniak
Criterion Consulting
n/a
Citation:
Behuniak, P. (2023). A review of clarifying inputs and outputs of cognitive assessments. Journal of Applied Measurement, 24(1/2), 9–13.
Inputs and Outputs of Cognitive Assessment: Navigating the Complexities of Multiple Purposes and End-Users
Kit-Tai Hau
The Chinese University of Hong Kong
Leifeng Xiao
Shanghai Normal University
Luyang Guo
University of Macau
Schafer (2023) championed enhanced clarity in cognitive assessment domains and a comprehensive reporting system. While we resonate with these principles, we underscore the significance of less tangible, evolving higher-order abilities and innovative proficiencies. Furthermore, the objectives behind cognitive assessments and their reports vary across student, school, district, and country levels. Therefore, consolidating these diverse reports into a single system might not be ideal and optimal.
Citation:
Hau, K.-T., Xiao, L., & Guo, L. (2023). Discussant remarks on the pillars of measurement wisdom. Journal of Applied Measurement, 24(1/2), 14–18.
A Commentary on the Pillars of Measurement Wisdom by George Engelhard, Jr.
Fen-Lan Tseng
Research Center for Psychological and Educational Testing
National Taiwan Normal University
n/a
Citation:
Tseng, F.-L. (2023). A commentary on clarifying inputs and outputs of cognitive assessments. Journal of Applied Measurement, 24(1/2), 19–22.
Reactions to “Clarifying the Inputs and Outputs of Cognitive Assessments”
Nikolaus Bezruczko
The Chicago School
n/a
Citation:
Bezruczko, N. (2023). Reactions to “Clarifying the inputs and outputs of cognitive assessments”. Journal of Applied Measurement, 24(1/2), 23–38.
REJOINDER
Reactions to the Commentaries on Clarifying Inputs and Outputs of Cognitive Assessments
William D. Schafer
University of Maryland
n/a
Citation:
Schafer, W. D. (2023). Response to commentaries on clarifying inputs and outputs of cognitive assessments. Journal of Applied Measurement, 24(1/2), 39–43.
REGULAR ARTICLES
Impact of Violation of Equal Item Discrimination on Rasch Calibrations
Chunyan Liu
National Board of Medical Examiners
Wenli Ouyang
National Board of Medical Examiners
Raja Subhiyah
National Board of Medical Examiners
The Rasch model, a widely used item response theory (IRT) system, assumes equal discrimination of all items when estimating item difficulties and examinee proficiencies. However, to some extent, the impact of item misfit due to violations of the equal item discrimination assumption on Rasch calibrations remains unclear. In this simulation study, we assess the effects of balanced and systematic variation of item discrimination on Rasch difficulty estimates and Rasch model fit statistics. Our findings suggest that item misfit due to unequal item discrimination can negatively impact item difficulty estimates and INFIT/OUTFIT statistics for both misfitting and well-fitting items. Test developers may find our results useful for improving the accuracy of item difficulty estimates and, ultimately, of the estimated examinee proficiencies.
Keywords: Rasch model, item misfit, equal item discrimination
Citation:
Liu, C., Ouyang, W., & Subhiyah, R. (2023). Impact of violation of equal item discrimination on Rasch calibrations. Journal of Applied Measurement, 24(1/2), 44–57.
Examining Equivalence of Three Versions of Mathematics Tests in China’s National College Admission Examination Using a Single Group Design
Chunlian Jiang
University of Macau
Stella Yun Kim
University of North Carolina at Charlotte
Chuang Wang
University of North Carolina at Charlotte
Jincai Wang
Soochow University
The National College Admission Examination, also known as Gaokao, is the most competitive examination in China because students’ scores obtained in Gaokao are used as the only criterion to screen applicants for college admission. Chinese students’ scores in Gaokao are also accepted by many universities in the world. The one-syllabus-multiple-tests practice has been implemented since 1985, but not much has been explored as to the extent to which multiple tests are equivalent. This study attempts to examine the equivalence of three versions of Gaokao mathematics tests and illustrate the methodological procedure using a single group design with an item response theory (IRT) approach. The results indicated that the three versions were comparable in terms of content coverage; however, most items were found to be easy for the students so more challenging items are suggested to be included for distinguishing students with average and high mathematics competencies. Some differences were also noted in terms of differential item functioning analysis and the factor structure.
Keywords: test concordance, mathematics examinations, item response theory, single group design, Gaokao
Citation:
Jiang, C., Kim, S. Y., Wang, C., & Wang, J. (2023). Examining equivalence of three versions of mathematics tests in China’s national college admission examination using a single group design. Journal of Applied Measurement, 24(1/2), 58–87.