Getting started with mcqAnalysis
Overview
The mcqAnalysis package provides a unified toolkit
for classical test theory (CTT) item analysis of multiple-choice tests.
It computes item difficulty, item discrimination (point-biserial
correlation and upper-lower 27 percent discrimination index),
per-distractor analysis, and Haladyna’s distractor efficiency, and
packages the results into a tidy mcq_analysis object with
dedicated print, plot, and
apa_table methods.
This vignette walks through a complete item analysis using the
package’s example dataset.
Installation
Install the released version from CRAN:
install.packages("mcqAnalysis")
Or the development version from GitHub:
# install.packages("devtools")
devtools::install_github("Rafhq1403/mcqAnalysis")
Example data
The package ships with mcq_example, a simulated
200-student, 30-item, four-option multiple-choice test. The dataset is
constructed so that items span the full range of quality: easy items,
ideal medium-difficulty items, hard items with declining discrimination,
and two deliberately badly-written items with negative
discrimination.
data(mcq_example)
str(mcq_example, max.level = 1)
#> List of 2
#> $ responses: chr [1:200, 1:30] "D" "D" "D" "D" ...
#> ..- attr(*, "dimnames")=List of 2
#> $ key : Named chr [1:30] "D" "D" "D" "A" ...
#> ..- attr(*, "names")= chr [1:30] "item01" "item02" "item03" "item04" ...
mcq_example$key[1:6]
#> item01 item02 item03 item04 item05 item06
#> "D" "D" "D" "A" "D" "B"
head(mcq_example$responses[, 1:6])
#> item01 item02 item03 item04 item05 item06
#> student001 "D" "D" "D" "A" "D" "B"
#> student002 "D" "D" "D" "A" "A" "B"
#> student003 "D" "D" "C" "A" "D" "B"
#> student004 "D" "D" "D" "A" "D" "B"
#> student005 "D" "D" "D" "A" "D" "B"
#> student006 "D" "D" "D" "A" "D" "B"
Complete analysis with mcq_analysis()
The wrapper function mcq_analysis() runs every
item-level computation in a single call and returns an
mcq_analysis S3 object.
result <- mcq_analysis(mcq_example$responses, mcq_example$key)
result
#> Multiple-Choice Item Analysis
#> ------------------------------
#> Students: 200
#> Items: 30
#> Mean total score: 15.765 (SD = 6.342 )
#>
#> Item-level statistics:
#> item key difficulty point_biserial discrimination_index
#> item01 D 0.850 0.472 0.426
#> item02 D 0.860 0.440 0.370
#> item03 D 0.845 0.336 0.296
#> item04 A 0.840 0.515 0.537
#> item05 D 0.720 0.423 0.556
#> item06 B 0.695 0.465 0.593
#> item07 D 0.690 0.424 0.593
#> item08 A 0.660 0.484 0.611
#> item09 D 0.580 0.471 0.685
#> item10 C 0.565 0.486 0.722
#> item11 B 0.570 0.508 0.648
#> item12 A 0.550 0.467 0.667
#> item13 C 0.495 0.519 0.704
#> item14 B 0.505 0.486 0.741
#> item15 D 0.425 0.530 0.759
#> item16 A 0.395 0.431 0.630
#> item17 A 0.465 0.477 0.685
#> item18 D 0.420 0.458 0.704
#> item19 A 0.490 0.573 0.759
#> item20 D 0.440 0.522 0.741
#> item21 B 0.375 0.424 0.648
#> item22 B 0.325 0.454 0.611
#> item23 D 0.405 0.530 0.704
#> item24 D 0.395 0.525 0.741
#> item25 A 0.260 0.391 0.519
#> item26 C 0.380 0.290 0.389
#> item27 D 0.305 0.245 0.296
#> item28 A 0.300 0.195 0.259
#> item29 C 0.460 -0.396 -0.407
#> item30 D 0.500 -0.292 -0.296
#> distractor_efficiency
#> 2
#> 1
#> 2
#> 2
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 3
#> 1
#> 1
The default print method shows test-level summaries (number of
students, number of items, mean and SD of total scores) and an
item-level table with difficulty, point-biserial, discrimination index,
and distractor efficiency.
Visualizing item quality
The plot() method produces a difficulty-discrimination
scatter — the classical “item quality map” used for visually identifying
items that fall outside conventional adequacy cutoffs. By default, only
flagged items are labeled, keeping the plot legible when many items
cluster in the acceptable region.
Items in red are flagged because they violate at
least one of the default adequacy criteria: difficulty outside [0.30,
0.90] or discrimination below 0.30. Items 29 and 30 have
negative discrimination — high-ability students get them wrong
more often than low-ability students, indicating poorly written
distractors or a mis-keyed answer.
To label every item or to plot the upper-lower 27 percent
discrimination index instead of the point-biserial:
plot(result, label = "all")
plot(result, discrimination_metric = "discrimination_index")
Per-distractor analysis
For diagnosing specific problematic items, the
distractor_analysis() function returns a per-option
breakdown showing how each response option performed.
da <- distractor_analysis(mcq_example$responses, mcq_example$key)
head(da, 8)
#> item option is_key frequency proportion point_biserial
#> item01 item01 A FALSE 10 0.050 -0.2670906
#> item011 item01 B FALSE 12 0.060 -0.3234223
#> item012 item01 C FALSE 8 0.040 -0.2505524
#> item013 item01 D TRUE 170 0.850 0.5156325
#> item02 item02 A FALSE 10 0.050 -0.2525847
#> item021 item02 B FALSE 9 0.045 -0.2893215
#> item022 item02 C FALSE 9 0.045 -0.2550078
#> item023 item02 D TRUE 172 0.860 0.4838546
For each item-option combination, the output reports the option’s
selection frequency, the proportion of examinees choosing it, whether it
is the key, and its point-biserial correlation with the total test
score. The key should have a clearly positive point-biserial; each
distractor should have a non-trivial selection proportion and a negative
point-biserial.
Inspect a specific problematic item:
da[da$item == "item30", ]
#> item option is_key frequency proportion point_biserial
#> item30 item30 A FALSE 55 0.275 0.27246235
#> item301 item30 B FALSE 21 0.105 0.04108455
#> item302 item30 C FALSE 24 0.120 -0.07627404
#> item303 item30 D TRUE 100 0.500 -0.21893362
Distractor efficiency
distractor_efficiency() summarizes the per-option
analysis into a single integer per item: the count of functioning
distractors. A distractor is “functioning” if it is selected by at least
5 percent of examinees and has a negative point-biserial with the total
score (Haladyna & Downing, 1993).
de <- distractor_efficiency(mcq_example$responses, mcq_example$key)
de[1:10]
#> item01 item02 item03 item04 item05 item06 item07 item08 item09 item10
#> 2 1 2 2 3 3 3 3 3 3
For a four-option item, distractor efficiency ranges from 0 (no
functioning distractors — the item is essentially a two-option item) to
3 (all three distractors functioning — the item is performing at full
capacity).
Publication-ready output with apa_table()
The apa_table() method formats the item analysis as a
publication-ready APA-style table in data-frame, markdown, HTML, or
LaTeX form.
apa_table(result, format = "data.frame")[1:8, ]
#> Item Key Difficulty Point-biserial Discrimination D Distractor Efficiency
#> 1 item01 D 0.85 0.47 0.43 2
#> 2 item02 D 0.86 0.44 0.37 1
#> 3 item03 D 0.84 0.34 0.30 2
#> 4 item04 A 0.84 0.52 0.54 2
#> 5 item05 D 0.72 0.42 0.56 3
#> 6 item06 B 0.70 0.47 0.59 3
#> 7 item07 D 0.69 0.42 0.59 3
#> 8 item08 A 0.66 0.48 0.61 3
#> Difficulty Level Discrimination
#> 1 Moderate Excellent
#> 2 Moderate Excellent
#> 3 Moderate Good
#> 4 Moderate Excellent
#> 5 Moderate Excellent
#> 6 Moderate Excellent
#> 7 Moderate Excellent
#> 8 Moderate Excellent
The data-frame output includes interpretive columns based on
conventional CTT cutoffs (Ebel & Frisbie, 1991). For inclusion in an
R Markdown manuscript:
apa_table(result, format = "markdown")
Individual functions
If you do not need the full wrapper, each component statistic is
available as a standalone function:
item_difficulty(mcq_example$responses, mcq_example$key)[1:6]
#> item01 item02 item03 item04 item05 item06
#> 0.850 0.860 0.845 0.840 0.720 0.695
point_biserial(mcq_example$responses, mcq_example$key)[1:6]
#> item01 item02 item03 item04 item05 item06
#> 0.4723679 0.4401658 0.3356581 0.5153377 0.4228731 0.4652371
item_discrimination(mcq_example$responses, mcq_example$key,
method = "discrimination_index")[1:6]
#> item01 item02 item03 item04 item05 item06
#> 0.4259259 0.3703704 0.2962963 0.5370370 0.5555556 0.5925926
All functions share the same input convention: a matrix or data frame
of student responses (students in rows, items in columns) and a vector
of correct answers with one entry per item.
References
Ebel, R. L., & Frisbie, D. A. (1991). Essentials of
educational measurement (5th ed.). Prentice Hall.
Haladyna, T. M. (2004). Developing and validating multiple-choice
test items (3rd ed.). Lawrence Erlbaum Associates.
Haladyna, T. M., & Downing, S. M. (1993). How many options is
enough for a multiple-choice test item? Educational and
Psychological Measurement, 53(4), 999-1010.
Kelley, T. L. (1939). The selection of upper and lower groups for the
validation of test items. Journal of Educational Psychology,
30(1), 17-24.