Analyze the quality of exams and question banks with two modern measurement approaches: Classical Test Theory (CTT) and Item Response Theory (IRT), on the ExamQR platform by TEX Vietnam.
Two internationally recognized methods integrated into ExamQR for a comprehensive analysis of question bank quality.
CTT analyzes items based on the observed scores of the whole group of candidates. Its statistics are sample-dependent – they change when the group of candidates changes.
IRT models the probability of a correct response as a function of the candidate's ability (θ). Item parameters are sample-invariant – more stable than CTT when comparing groups.
| Criterion | CTT | IRT (1PL/2PL/3PL) |
|---|---|---|
| Sample dependence | ✗ Highly dependent | ✓ Sample-invariant |
| Sample size requirement | ✓ N ≥ 30–50 | ✗ N ≥ 200–500 |
| Computational complexity | ✓ Simple | ✗ Complex (MLE/Bayesian) |
| Individual ability measurement | ~ Indirect, via total score | ✓ Direct (θ estimation) |
| Computerized adaptive testing (CAT) | ✗ Not supported | ✓ Foundation of CAT |
| Distractor analysis | ✓ Easy to perform | ~ Requires Nominal Response Model |
| Fit with ExamQR | ✓ Quick analysis, instructor reports | ✓ Item bank calibration, IRT-CAT |
| Statistic | Good | Acceptable | Reject |
|---|---|---|---|
| Difficulty (p) | 0.30 – 0.70 | 0.20–0.30 | 0.70–0.80 | <0.20 | >0.80 |
| Discrimination (D) | ≥ 0.30 | 0.20–0.29 | <0.20 |
| rpbis | ≥ 0.30 | 0.20–0.29 | <0.20 |
| Cronbach's α | ≥ 0.80 | 0.70–0.79 | <0.70 |
| Parameter | Good | Acceptable | Review |
|---|---|---|---|
| Difficulty (b) | −1.5 to 1.5 | ±1.5–2.5 | |b| > 2.5 |
| Discrimination (a) | 0.8 – 2.5 | 0.5–0.8 | < 0.5 |
| Guessing (c) | ≤ 0.20 | 0.20–0.25 | > 0.25 |
| Infit/Outfit MNSQ | 0.7 – 1.3 | 1.3–1.5 | > 1.5 |
Analysis of 20 multiple-choice items with N = 200 students. Data processed by the ExamQR Analytics module.
| Item | Content summary | Difficulty (p) | Discrimination (D) | rpbis | α if Deleted | Distractor A | Distractor B | Distractor C | Rating |
|---|
Item Q06 has low discrimination (D = 0.14). The option analysis shows that the keyed answer C* (the correct answer) attracts fewer high-ability candidates than distractor B – a sign of a problem with the item's content or wording.
| Group | Option A | Option B | Option C* ✓ | Option D | Omitted |
|---|---|---|---|---|---|
| High (n=50) | 18% | 32% | 36% | 12% | 2% |
| Middle (n=100) | 22% | 28% | 30% | 18% | 2% |
| Low (n=50) | 26% | 20% | 22% | 30% | 2% |
IRT parameters estimated by Maximum Likelihood Estimation (MLE) using the three-parameter logistic (3PL) model, with the ExamQR built-in analysis engine.
| Item | a (Discrimination) | b (Difficulty) | c (Guessing) | SE(b) | Infit MNSQ | Outfit MNSQ | IIF Peak | θ at Peak | Model Fit |
|---|
A comprehensive visualization of CTT and IRT statistics that helps instructors and assessment staff quickly identify items that need improvement.
| Item | CTT issue | IRT issue | Recommendation |
|---|---|---|---|
| Q06 | p=0.71, D=0.14, r=0.18 – Too easy, poor discrimination | a=0.41, c=0.22 – Low discrimination, high guessing | Rewrite completely |
| Q11 | p=0.81, D=0.10, r=0.12 – Too easy | b=−2.41, Outfit=1.67 – Poor model fit | Adjust difficulty |
| Q17 | p=0.19, D=0.18 – Too difficult | b=2.61, SE=0.42 – Unstable estimate | Simplify the item |
These items meet some criteria but have 1–2 statistics near the borderline. Keep them with minor adjustments:
A Cronbach's α = 0.782 indicates that the test has acceptable reliability (recommended threshold ≥ 0.70 for achievement tests, ≥ 0.80 for high-stakes tests). To reach α ≥ 0.80:
Based on an analysis of the 3 most recent exams, ExamQR found that items on the topic "Integration" tend to be unusually difficult (mean p = 0.31). We suggest strengthening learning resources for this chapter.
IRT results show that the current question bank provides the most information at ability levels θ ∈ [−0.5, 1.0], and lacks items in the θ > 1.5 region (top-performing students).
→ Recommendation: add 5–8 difficult items (b ∈ [1.5, 2.5]) for the next exam.