Journal of Applied Measurement
A publication of the Department of Educational Psychology and Counseling
National Taiwan Normal University
Volume 26, Issue 1/2 (2025)
Remembering Thomas R. Knapp
William D. Schafer
University of Maryland
Hak Ping Tam
National Taiwan Normal University
n/a
Citation:
Schafer, W. D., & Tam, H. P. (2025). Remembering Thomas R. Knapp. Journal of Applied Measurement, 26(1/2), 1–3.
Detecting Unfavorable Experience in the Face of Digital Disruption: Using Rasch Measures in PLS-SEM Analysis
Nor Irvoni Mohd Ishar
Universiti Teknologi MARA
Zi Yan
The Education University of Hong Kong
Mohd Zali Mohd Nor
Newstar Agencies Sdn. Bhd.
T. Ramayah
Universiti Sains Malaysia
Rosmimah Mohd Roslin
UNITAR International University
Trevor G. Bond
n/a
In a modern digitized world, it has become increasingly difficult to meet consumer demands in a crowded market space. Providing exceptional services and experiences to discerning customers has become the key to gaining a competitive advantage. The experience economy now dictates the path for businesses wherein the goal is customer satisfaction and consequential loyalty. Yet, there are always unsatisfactory service experiences, and therefore, service providers are forced to contemplate the reasons. Based on a survey of 350 telco subscribers in Malaysia, this study seeks to understand the impact of adverse experiences on consumer behavior in the telecommunications sector. To demonstrate a hybrid data analysis method, the items identified as belonging to each construct were assessed for their fit to the requirements of the Rasch model. Subsequently, Partial Least Squares Structural Equation Modelling (PLS-SEM) was used to assess the convergent and discriminant validity of the reflective model before evaluating the structural model. The main results of the study show that unfavorable service experiences have a significant impact on dissatisfied customers’ response behavior. The present study’s contribution lies in the application of the Rasch analysis method to establish the measurement properties of each of the scales used in the study before person measures were imputed into the SEM analysis.
Keywords: social media complaining, digital disruption, Malaysia, PLS-SEM, Rasch model, unfavorable experience.
Citation:
Ishar, N. I. M., Yan, Z., Nor, M. Z. M., Ramayah, T., Roslin, R. M., & Bond, T. G. (2025). Detecting unfavorable experience in the face of digital disruption: Using Rasch measures in PLS-SEM analysis. Journal of Applied Measurement, 26(1/2), 4–20.
Development and Validation Study of the Preschool Mathematics Education Practices Survey
Toni May
SUNY-Binghamton University
Kristin Koskey
SUNY-Binghamton University
Kathleen Provinzano
SUNY-Binghamton University
Preschool mathematics skills have been shown to predict students’ success in later mathematics and other cognitive domains. Yet, studies of what preschool teachers teach in terms of mathematics in their classrooms are limited. When such research is conducted, observational methods are typically applied as effective self-report measures are not widely available for use with preschool teachers. This validation study provided multiple sources of validity evidence for the Preschool Mathematics Education Practices Survey (PS-MEPS), aligned with the Head Start Early Learning Outcomes Framework. Both qualitative and quantitative data sources from various interested parties and experts were employed in a rigorous design-based research process applying the Rasch model with findings used iteratively to inform the PS-MEPS development and validity evidence. Strong evidence for content, response processes, consequences of testing, and internal structure was found for using the PS-MEPS with diverse preschool teachers.
Keywords: Preschool mathematics, teaching preschool mathematics, instrument validation, validity evidence, Rasch
Citation:
May, T., Koskey, K., & Provinzano, K. (2025). Development and validation study of the preschool mathematics education practices survey. Journal of Applied Measurement, 26(1/2), 21–44.
Using Explanatory Rasch Models to Measure Attitudes Towards Censorship
George Engelhard
The University of Georgia
The purpose of this study is to illustrate the use of explanatory Rasch models to examine the psychometric quality of attitude scales. An attitude towards censorship scale is used to illustrate the application of generalized linear mixed models to estimate explanatory Rasch models. Censorship can be defined as the suppression of words, images, or ideas that are offensive to some people who succeed in imposing their personal political or moral values on others. The scale used for the illustration is the Attitude Toward Censorship Scale developed by Rosander and Thurstone (1931). Although this scale was created almost a century ago, the themes reflected in this scale remain remarkably relevant. A unique feature of explanatory Rasch models is that both item and person covariates can be included in the measurement model. Specifically, this study includes item valence (negative, neutral, or positive) as an item covariate, and two person covariates (gender and level of education). Information obtained from these covariates provides additional validity evidence that supports the interpretation of the meaning of the scores. There were 950 participants in the study. Item valence and gender were not significant predictors of the responses, while level of education emerged as an important predictor of responses on the Attitude Toward Censorship Scale. The implications for research, theory, and practice are discussed.
Keywords: Attitude measurement, Rasch measurement theory, generalized linear mixed models,
censorship, explanatory Rasch models
Citation:
Engelhard, G. (2025). Using explanatory Rasch models to measure attitudes towards censorship. Journal of Applied Measurement, 26(1/2), 45–66.
Many-Facet Rasch Designs: How Should Raters be Assigned to Examinees?
Christine E. DeMars
James Madison University
Yelisey A. Shapovalov
James Madison University
John D. Hathcoat
James Madison University
In many-facet Rasch measurement, raters should be connected, and there are multiple ways to connect raters. This simulation contains two studies. In one study, two raters scored each examinee. Each rater was either paired with many different raters a few times or repeatedly paired with only two other raters. The standard errors of both rater severity and examinee ability were higher when raters scored one examinee in common with many different raters compared to when they scored many examinees in common with two raters. However, the differences were small, especially for the standard error of examinee ability. In the next study, most examinees were scored by a single rater. Linking was accomplished either by assigning all raters to score the same small linking set of examinees or by assigning a subset of papers to pairs of raters, with each rater participating in multiple pairs to provide links. Slightly smaller standard errors were achieved when all raters scored a common linking set, compared to paired ratings, keeping the total number of ratings constant. Overall, the design made little difference, especially when examinees are double-scored.
Keywords: MFRM, Rasch, facets, raters
Citation:
DeMars, C. E., Shapovalov, Y. A., & Hathcoat, J. D. (2025). Many-facet Rasch designs: How should raters be assigned to examinees? Journal of Applied Measurement, 26(1/2), 67–86.
Semantic Factor Analysis Using LLM Transformers: No Respondents Needed
Rense Lange
Global Psytech, Kuala Lumpur
Haniza Yon
Global Psytech, Kuala Lumpur
Labiib Marzuki
Global Psytech, Kuala Lumpur
Two large language model (LLM) based transformers—MiniLM and distilBERT—were used to compute the semantic similarity between the texts of the fifty items of Temple University’s Big Five personality questionnaire, without administering these items to any test-takers. Overall, MiniLM transformer performed noticeably better than distilBERT in recovering the factor structure of the Big 5. When using MiniLM, all factors were recovered correctly when five factors were extracted and rotated based on the similarity of items’ text. When using ten factors, only two of the five factors’ cluster membership agreed perfectly with Temple’s item classification derived via standard factor analysis, Agreeableness being the most problematic. We expect that, given the rapid developments in LLM, research will advance rapidly, such that item misclassifications will disappear as transformers are becoming increasingly powerful and accurate. These findings suggest that tools can be developed to anticipate the traditional factor structure of sets of items without gathering any empirical data. The success of LLM may have important implications for measurement in the social sciences in general, while providing a new perspective on factor analysis. In particular, the results align with Wittgenstein’s notion of language “games,” suggesting that the factor structure of psychological
questionnaires represents shared linguistic practice rather than reality per se.
Keywords: Factor analysis, large language models, respondents
Citation:
Lange, R., Yon, H., & Marzuki, L. (2025). Semantic factor analysis using LLM transformers: No respondents needed. Journal of Applied Measurement, 26(1/2), 87–95.
Appendix A: https://reurl.cc/mpqD09
Appendix B: https://reurl.cc/M2Z8yp
Book Review: Constructing Measures: An Item Response Modeling Approach, Second Edition by Mark Wilson
Ya-Hui Su
National Chung Cheng University
n/a
Citation:
Su, Y.-H. (2025). Book review: Constructing measures: An item response modeling approach, second edition by Mark Wilson. Journal of Applied Measurement, 26(1/2), 96–99.