Test comparisons

Photo by Chris Liverani on Unsplash

Purpose

Comparing two tests of proficiency to identify whether they are interchangeable.

Approach

I used a range of methods from quantitative educational metrics analysis to identify whether the outcomes on two measures of English proficiency could be mapped to each other to obtain a concordance.

They key outcome was a report, profiling whether the tests appeared to be:

  1. Measuring the same construct
  2. Able to conform to the principle of equity, i.e. once two test forms were equated, it should not matter to the test-taker which form of the test is administered.
  3. Symmetrical, i.e. equating of X to Y must be the inverse of equating Y to X.
  4. Invariant by subpopulation, i.e. there would be no differences when equating by subpopulations of takers
  5. Providing the same inferences, i.e. designed for use to reach the same kinds of conclusions
  6. Applied to the same target population

Outcome

  • I provided advice to the client as to why “traditional” statistical tests, as suggested in a previous consultation by someone else, were not appropriate for this task, and practical actionable steps forward based on the analysis.

Key tools

  • R: Rmarkdown, tidyverse; Git + GitHub.