useR! 2024

Educational Large-Scale Assessment in Austria has undergone a major change in 2022, when the system was changed from one to multiple yearly tests for several subjects and grades (iKMPLUS). The major challenge for the test developers was the immediate feedback of test results to test takers and teachers. Keeping high psychometric standards, we used a mixture of pre-calibrating the test booklets for scaling and cohort-specific reference scores and automatically sourced R scripts for coding, analysing and reporting the test data. R was used in all these processes with TAM for IRT-scaling, dplyr and tidyr for convenient data wrangling, doParallel for handling the workload and R Markdown and ggplot2 for reporting and monitoring. In sum, our R-based process of reporting nation-wide test results, for primary and secondary school pupils, produces 850.000 reports every year for teachers, pupils and principals. Here, we want to present our main principles and experiences of reporting test results under an R-based programming approach with special emphasis on the underlying psychometric analyses, the subsequent automated generation of graphs and the process in general.