Journal of Applied Measurement

A publication of the Department of Educational Psychology and Counseling
National Taiwan Normal University

Journal of Applied Measurement Abstracts
Volume 22, Issue 3/4 (2021)

The Effects of Textual Borrowing Training on Rater Accuracy When Scoring Students’ Responses to an Integrated Writing Task

Kevin R. Raczynski
Kevin Raczynski & Associates
Jue Wang
The University of Science and Technology of China
George Engelhard, Jr.
Allan S. Cohen
The University of Georgia

Integrated writing (IW) tasks require students to incorporate information from provided source material into their written responses. While the growth of IW tasks has outpaced research on scoring challenges that raters working in this assessment context experience, several researchers have reported that raters in their studies struggled to agree about whether students demonstrated successful integration of source material in their written responses. One suggestion offered for meeting this challenge was to provide rater training on textual borrowing, a topic not covered in all training programs. We randomly assigned 17 middle school teachers to two training conditions to determine whether teachers who completed an augmented training protocol specific to textual borrowing would score students’ responses to an IW task more accurately than teachers who completed a comparison training protocol that did not include instruction on textual borrowing. After training, all teachers scored the same set of 30 benchmark essays. We compared the teachers’ scores to professional raters’ scores, dichotomized based on whether the scores matched, and then analyzed the resulting data using a FACETS model for accuracy. As a group, the teachers who completed the augmented training scored more accurately than the comparison group. Policy implications for scoring rubric design and rater training are discussed.

Keywords:Integrated writing, textual borrowing, rater training, rater accuracy

Citation:
Raczynski, K. R., Wang, J., Engelhard, G., Jr., & Cohen, A. S. (2022). The effects of textual borrowing training on rater accuracy when scoring students’ responses to an integrated writing task. Journal of Applied Measurement, 22(3/4), 161–190.

The Development and Validation of a Thinking Skills Assessment for Students With Disability Using Rasch Measurement Approaches

Toshiko Kamei
Masa Pavlovic
Assessment Research Centre, Melbourne Graduate School of Education,
The University of Melbourne, Melbourne, Australia

21st century skills such as thinking are gaining prominence in curricula internationally as fundamental to thrive and learn in an evolving global environment. This study developed and investigated the validity of a thinking skills assessment based on a learning progression of students with disability. It followed an established method (Griffin, 2007) previously used to develop assessments based on learning progressions of foundational learning skills of students with disability. An initial review of research and co-design process with teachers with expertise in teaching students with disability was used to develop a set of assessment items based on a hypothetical criterion-referenced framework of thinking skills. This was followed by empirical exploration of the developed thinking skills assessment through a Rasch partial credit model (Masters, 1982) analysis using student assessment data from a field trial of the thinking skills assessment items involving 864 students. SME review, person and item fit statistics, and reliability coefficients provided evidence to support the validity of the assessment for its intended purpose and supported arguments for a single, underlying construct. A thinking skills assessment based on a learning progression of school-age students with disability was derived, drawing on teacher interpretation and calibration of student assessment data. The resulting thinking skills assessment provided a practical tool that teachers can apply in the classroom to implement the teaching and learning of thinking skills to a cohort and level of learning previously not targeted.

Keywords: thinking skills, students with disability, learning progression, assessment

Citation:
Kamei, T., & Pavlovic, M. (2022). The development and validation of a thinking skills assessment for students with disability using Rasch measurement approaches. Journal of Applied Measurement, 22(3/4), 191–213.

Exploring the Impact of Open-Book Assessment on the Precision of Test-Taker and Item Estimates Using an Online Medical Knowledge Assessment

Stefanie A. Wind
Cheng Hua
The University of Alabama
Stefanie S. Sebok-Syer
Stanford University

Researchers concerned with the psychometric quality of assessments often examine indicators of model-data fit to inform the interpretation and use of parameter estimates for items and persons. Fit assessment techniques generally focus on overall model fit (i.e., global fit), as well as fit indicators for individual items and persons.
In this study, we demonstrate how one can also use item-level information from individual responses (e.g., the use of outside assistance) to explore the impact of such behaviors on model-data fit. This study’s purpose was to use data from an open-book format assessment to examine the impact of examinee help-seeking behavior on the psychometric quality of item and person estimates. Open-book assessment formats, where test-takers are permitted to consult outside resources while completing assessments, have become increasingly common across a wide range of disciplines and contexts. With our analysis, we illustrated an approach for evaluating model-data fit by combining residual analyses with test-takers’ self-reported information about their testtaking behavior. Our results indicated that the use of outside resources impacted the pattern of expected and unexpected responses differently for individual test-takers and individual items. Analysts can use these techniques across a variety of assessment contexts where information about test-taker behavior is available to inform assessment development, interpretation, and use; this includes: evaluating psychometric properties following pilot testing, item bank maintenance, evaluating results from operational exam administrations, and making decisions about assessment interpretation and use.

Keywords: Open-book assessment, psychometric properties, person fit, residual analysis

Citation:
Wind, S. A., Hua, C., & Sebok-Syer, S. S. (2022). Exploring the impact of open-book assessment on the precision of test-taker and item estimates using an online medical knowledge assessment. Journal of Applied Measurement, 22(3/4), 214–234.

Seeing Skill: Heuristic Reasoning When Forecasting Outcomes in Sports

Veronica U. Weser
Karen M. Schmidt
Gerald L. Clore
Dennis R. Proffitt
Department of Psychology, University of Virginia

Expert advantage in making judgments about the outcomes of sporting events is well-documented. It is not known, however, whether experts have an advantage in the absence of objective information, such as the current score or the relative skill of players. Participants viewed 5-second clips of modern Olympic fencing matches and selected the likely winners. Participants’ predictions were compared with the actual winners to compute accuracy. Study 1 revealed small but significant differences between the accuracy of experts and novices, but it was not clear what fencing behaviors informed participants’ judgments. Rasch modeling was used to select stimuli for Study 2, in which fencing-naïve participants rated the gracefulness, aggressiveness, and confidence of competitors before selecting winners. By using Rasch modeling to develop the stimuli set, fencing-naïve participants were able to identify winners at above chance rates. The results further indicate that in the absence of concrete information, competitiveness and confidence may be used as a heuristic for the selection of winning athletes.

Keywords: heuristics, Rasch model, video clips, fluency, sports

Citation:
Weser, V. U., Schmidt, K. M., Clore, G. L., & Proffitt, D. R. (2022). Seeing skill: Heuristic reasoning when forecasting outcomes in sports. Journal of Applied Measurement, 22(3/4), 235–248.

Psychometric Assessment of an Adapted Social Provisions Survey in a Sample of Adults With Prediabetes

Kathryn E. Wilson
Department of Kinesiology and Health, Georgia State University
Center for the Study of Stress, Trauma, and Resilience, Georgia State University
 Tzeyu L. Michaud
Department of Health Promotion, University of Nebraska Medical Center
Center for Reducing Health Disparities, University of Nebraska Medical Center
 Cynthia Castro Sweet
Omada Health, Inc., Medical Affairs, San Francisco, CA, USA
 Jeffrey A. Katula
Department of Health and Exercise Science, Wake Forest University
 Fabio A. Almeida
Fabiana A. Brito
Department of Health Promotion, University of Nebraska Medical Center
Center for Reducing Health Disparities, University of Nebraska Medical Center
 Robert Schwab
Department of Health Promotion, University of Nebraska Medical Center
Internal Medicine Division of General Medicine, University of Nebraska Medical Center
 Paul A. Estabrooks
Department of Health Promotion, University of Nebraska Medical Center
Center for Reducing Health Disparities, University of Nebraska Medical Center

The relevance of social support for weight management is not well documented in people with prediabetes. An important consideration is the adequate assessment of social provisions related to weight management. We aimed to assess the factor structure and measurement invariance of an adapted Social Provisions Scale specific to weight management (SPS-WM). Participants of a diabetes prevention trial (n = 599) completed a demographic survey and the SPS-WM. Confirmatory analyses tested the factor structure of the SPS-WM, and measurement invariance was assessed for gender, weight status, education level, and age. Removal of two items resulted in acceptable model fit, supporting six correlated factors for social provisions specific to weight management. Measurement invariance was supported across all subgroups. Results support score interpretations for these scales reflecting distinct components of social support specific to weight management in alignment with those of the original survey.

Keywords: social support, perceived support, diabetes prevention, factorial validity, measurement invariance

Citation:
Wilson, K. E., Michaud, T. L., Sweet, C. C., Katula, J. A., Almeida, F. A., Brito, F. A., Schwab, R., & Estabrooks, P. A. (2022). Psychometric assessment of an adapted social provisions survey in a sample of adults with prediabetes. Journal of Applied Measurement, 22(3/4), 249–269.

A Rasch Analysis of the Mindful Attention Awareness Scale—Children

Daniel Strissel
Neuro Educational Associates
Julia A. Ogg
Northern Illinois University

The Mindful Attention Awareness Scale—Children (MAAS-C) was developed using traditional psychometric methods to measure dispositional mindfulness in children. The MAAS-C is based on the MAAS, a highly cited mindfulness scale for adults. This study extended this effort by applying the Rasch model to investigate the psychometric properties of the MAAS-C. Evidence from Rasch Analyses conducted on the MAAS suggested local dependence of items and the need for modifications, including a rescoring algorithm. We aimed to examine how the MAAS-C performed when evaluated with Rasch analysis using a sample of 406 fifth- and sixth-grade children. Upon analysis, all 15 items on the MAAS-C worked in the same direction; the fit statistics fell within a range suitable for productive measurement; a PCAR of the residuals revealed an unpatterned distribution of residuals, and DIF was not found for any of the grouping variables. However, the items were not evenly distributed nor well-targeted for children with the highest levels of dispositional mindfulness. We demonstrated that the precision and item functioning of the MAAS-C could be improved by uniformly rescoring the response categories. Once rescored, the provided ordinal-to-interval conversion table (see Table 5) can be used to optimize scoring on the MAAS-C.

Keywords: dispositional mindfulness, Rasch analysis, rating scale

Citation:
Strissel, D., & Ogg, J. A. (2022). A Rasch analysis of the mindful attention awareness scale—Children. Journal of Applied Measurement, 22(3/4), 270–291.