Reporting the reliability of a composite score involves two separable decisions, following the framework of Kelley and Pornprasertmanit (2016): choosing the coefficient and choosing the interval.
The map for choosing the coefficient:
| Function | The composite | The claim being made |
|---|---|---|
reliability_alpha() |
unit-weighted sum | one factor with equal loadings (essential tau-equivalence) |
reliability_kr20() |
sum of 0/1 items | alpha’s claim, for binary items |
reliability_omega() (robust omega, the default) |
unit-weighted sum | one congeneric factor for the numerator only; the composite variance is estimated from the data |
reliability_omega(denominator = "model_implied") |
unit-weighted sum | one congeneric factor; the fitted model reproduces the composite variance |
reliability_omega_categorical() |
sum of observed ordinal categories | a probit threshold model links items to one factor for the numerator only; the total variance comes from the unrestricted polychoric correlations |
reliability_omega_categorical(denominator = "model_implied") |
sum of observed ordinal categories | a probit threshold model links items to one factor; the fitted model reproduces the polychoric correlations (Green & Yang, 2009) |
reliability_H() |
optimally weighted composite | one congeneric factor; a different composite, not a different assumption |
The vignette walks the table top to bottom on one running example, then turns to the interval.
A six-item congeneric scale with unequal loadings, which is the normal state of real items:
Coefficient alpha equals the population reliability when the items are essentially tau-equivalent, that is, when every item loads equally on the common factor. When loadings differ, alpha underestimates. On the running example the loadings range from .4 to .8, so alpha sits a little below omega:
res_alpha <- reliability_alpha(data = items, ci_method = "none")
res_omega <- reliability_omega(data = items,
denominator = "model_implied",
ci_method = "none")
c(alpha = res_alpha$value[res_alpha$term == "estimate"],
omega = res_omega$value[res_omega$term == "estimate"])
#> alpha omega
#> 0.7976157 0.8067366The gap here is modest because the loadings, though unequal, are of
similar magnitude. The gap grows when the tau-equivalence assumption
fails badly, and with real tests a common way for it to fail is in how
the composite is scored rather than in the tests themselves. Four of the
tests in the Holzinger and Swineford (1939) data
(holzinger_swineford) measure spatial ability: visual
perception, cubes, paper form board, and lozenges. Because the tests are
scored on different scales, their raw-score loadings on a common factor
are far from equal, and the unit-weighted sum of the raw scores, the
composite alpha and omega describe here, carries that inequality.
cfa_1() fits the single factor with the loadings free, the
congeneric model behind omega, and with the loadings constrained to be
equal, the essentially tau-equivalent model behind alpha:
spatial <- holzinger_swineford[, c("t1_visual_perception", "t2_cubes",
"t3_paper_form_board", "t4_lozenges")]
d_spatial <- descriptives(spatial)$descriptives
print(d_spatial[, c("variable", "mean", "sd")], digits = 3)
#> variable mean sd
#> 1 t1_visual_perception 29.6 7.00
#> 2 t2_cubes 24.4 4.71
#> 3 t3_paper_form_board 14.2 2.83
#> 4 t4_lozenges 18.0 9.05
congeneric <- cfa_1(data = spatial)
congeneric[startsWith(congeneric$term, "lambda") |
congeneric$term %in% c("chi_square", "df"),
c("syntax", "term", "estimate")]| syntax | term | estimate |
|---|---|---|
| f1 =~ t1_visual_perception | lambda_f1_1 | 4.728 |
| f1 =~ t2_cubes | lambda_f1_2 | 2.240 |
| f1 =~ t3_paper_form_board | lambda_f1_3 | 1.427 |
| f1 =~ t4_lozenges | lambda_f1_4 | 5.898 |
| chi_square | 1.855 | |
| df | 2 |
tau_equivalent <- cfa_1(data = spatial, equal_loading = TRUE)
tau_equivalent[tau_equivalent$term %in% c("chi_square", "df"),
c("term", "estimate")]| term | estimate |
|---|---|
| chi_square | 69.658 |
| df | 5 |
a_spatial <- reliability_alpha(data = spatial, ci_method = "none")
o_spatial <- reliability_omega(data = spatial,
denominator = "model_implied",
ci_method = "none")
est_spatial <- c(alpha = a_spatial$value[a_spatial$term == "estimate"],
omega = o_spatial$value[o_spatial$term == "estimate"])
est_spatial
#> alpha omega
#> 0.6220163 0.6794339The standard deviations of the four tests run from 2.83 (paper form board) to 9.05 (lozenges), and the raw-score loadings follow them. The congeneric model reproduces the covariances of the four tests closely, and constraining the loadings to be equal does not. For the sum of the raw scores, alpha falls 0.057 below omega, against 0.009 on the running example, so alpha here answers under an assumption this composite violates.
Tau-equivalence is a property of the composite as scored, not of the tests alone. Standardizing each test before summing gives every test the same variance and makes a different composite, the sum of the z-scores:
spatial_z <- as.data.frame(scale(spatial))
congeneric_z <- cfa_1(data = spatial_z)
congeneric_z[startsWith(congeneric_z$term, "lambda"),
c("syntax", "term", "estimate")]| syntax | term | estimate |
|---|---|---|
| f1 =~ t1_visual_perception | lambda_f1_1 | 0.675 |
| f1 =~ t2_cubes | lambda_f1_2 | 0.476 |
| f1 =~ t3_paper_form_board | lambda_f1_3 | 0.504 |
| f1 =~ t4_lozenges | lambda_f1_4 | 0.652 |
a_z <- reliability_alpha(data = spatial_z, ci_method = "none")
o_z <- reliability_omega(data = spatial_z,
denominator = "model_implied",
ci_method = "none")
est_z <- c(alpha = a_z$value[a_z$term == "estimate"],
omega = o_z$value[o_z$term == "estimate"])
est_z
#> alpha omega
#> 0.6643619 0.6695794The standardized loadings run from .476 to .675, a narrower spread than the running example’s .4 to .8, and on this composite alpha falls only 0.005 below omega. The sum of the raw scores and the sum of the standardized scores are different composites with different reliabilities, and a reported coefficient is a claim about the composite actually scored. The pattern generalizes: each coefficient is correct under its stated model, and the choice among them is a choice among models.
On the standardized spatial tests the gap all but closed, because their standardized loadings differ little. Items with equal variances can still differ widely in how strongly they reflect the factor, and then alpha falls well below omega. The items here are hypothetical, constructed to show the point: six items, each with population variance 1, three loading .9 on the factor and three loading .1, as in a scale where half of the items barely track the construct. The population alpha and omega follow from the loadings:
lambda_hyp <- c(0.9, 0.9, 0.9, 0.1, 0.1, 0.1)
k_hyp <- length(lambda_hyp)
Sigma_hyp <- tcrossprod(lambda_hyp) + diag(1 - lambda_hyp^2)
# Alpha from the item variances and the variance of the sum; omega as
# the share of the variance of the sum that the factor accounts for.
pop_hyp <- c(alpha = k_hyp / (k_hyp - 1) *
(1 - sum(diag(Sigma_hyp)) / sum(Sigma_hyp)),
omega = sum(lambda_hyp)^2 / sum(Sigma_hyp))
pop_hyp
#> alpha omega
#> 0.6258373 0.7177033Then 300 respondents are drawn from this population, and both coefficients are estimated as before. The draws reuse the running example’s seed, so the factor scores and the normal deviates behind the errors are those of the running example, and only the loadings differ:
# withr::with_seed() restores the generator state afterward, so drawing
# these items does not change the random numbers the later examples use.
N_hyp <- 300
items_hyp <- withr::with_seed(113, {
eta_hyp <- rnorm(N_hyp)
sweep(matrix(rep(eta_hyp, k_hyp), N_hyp, k_hyp), 2, lambda_hyp, `*`) +
matrix(rnorm(N_hyp * k_hyp), N_hyp, k_hyp) %*%
diag(sqrt(1 - lambda_hyp^2))
})
colnames(items_hyp) <- paste0("y", seq_len(k_hyp))
a_hyp <- reliability_alpha(data = items_hyp, ci_method = "none")
o_hyp <- reliability_omega(data = items_hyp,
denominator = "model_implied",
ci_method = "none")
est_hyp <- c(alpha = a_hyp$value[a_hyp$term == "estimate"],
omega = o_hyp$value[o_hyp$term == "estimate"])
est_hyp
#> alpha omega
#> 0.6516687 0.7383066In the population, alpha falls 0.092 below omega, and in the sample, 0.087, against 0.057 for the sum of the raw spatial scores and 0.009 on the running example, whose six items also have variance 1. These items already share one variance, so standardizing them leaves the gap in place: alpha falls below omega because the equal loadings it assumes are far from the loadings that generated the items.
Coefficient omega requires an estimate of the composite’s total
variance in its denominator, and reliability_omega() offers
two. The default, denominator = "observed" (robust omega),
estimates the composite variance directly from the data, which is
consistent for the actual variance of the scale score whether or not the
one-factor model is right. The alternative,
denominator = "model_implied", uses the total variance
reproduced by the fitted single-factor model, which is correct exactly
when the one-factor model reproduces the composite variance.
The coefficient was built for the following situation. Model misfit
is treated as a set of minor common factors, visible as residual
correlations: the scale is then not perfectly unidimensional, but a
single-factor approximation is retained. The coefficient isolates the
variance attributable to the general factor only, excluding the minor
factors, and expresses it relative to the observed variance of the
unweighted composite. Kelley and Pornprasertmanit (2016) named it
hierarchical omega after that hierarchical factor logic; upon
reflection, we would have named it observed omega or robust omega, and
DMAR uses robust omega. Two qualifications keep the name
accurate. The robustness is to misspecification of the total variance
only, because the numerator remains model based under either
denominator; and robust omega is distinct both from outlier-robust
estimation of alpha and omega (Zhang & Yuan, 2016) and from the
robust maximum-likelihood standard errors available through
ci_method.
When the one-factor model holds, the two agree, as on the running example:
o_mi <- reliability_omega(data = items,
denominator = "model_implied",
ci_method = "none")
o_ob <- reliability_omega(data = items, ci_method = "none")
c(model_implied = o_mi$value[o_mi$term == "estimate"],
observed = o_ob$value[o_ob$term == "estimate"])
#> model_implied observed
#> 0.8067366 0.8056993Now contaminate the scale with a nuisance doublet, a shared specific factor on the first two items, the sort of minor structure real item sets carry:
doublet <- rnorm(N)
items_d <- items
items_d[, 1] <- items[, 1] + 0.6 * doublet
items_d[, 2] <- items[, 2] + 0.6 * doublet
o_mi_d <- reliability_omega(data = items_d,
denominator = "model_implied",
ci_method = "none")
o_ob_d <- reliability_omega(data = items_d, ci_method = "none")
c(model_implied = o_mi_d$value[o_mi_d$term == "estimate"],
observed = o_ob_d$value[o_ob_d$term == "estimate"])
#> model_implied observed
#> 0.7846721 0.7693093The two now diverge, and only robust omega still answers the question
a reliability coefficient is supposed to answer, the proportion of the
variance of the composite you actually computed that is attributable to
the common factor. The divergence itself is diagnostic: a nontrivial gap
between the two denominators says the one-factor model is not
reproducing the composite variance, and the right response to a large
gap is to model the structure, not to choose a denominator. Note also
what the observed denominator does not fix: the fitted loadings absorb
part of the doublet under either setting, so checking the one-factor
model (for example with cfa_1()) remains part of reporting
any omega.
A small gap is not evidence of the reverse. A doublet or a method
factor shared by a few same-signed items inflates their loadings while
leaving the two denominators close, so the gap cannot certify the
numerator. The residual correlations of the single-factor fit are the
more direct check, and every omega carries them: the printed table names
the denominator and the largest residual correlation, and the full
matrix is in the residual_cor attribute.
| term | value |
|---|---|
| estimate | 0.769 |
| se | NA |
| ci_lower | NA |
| ci_upper | NA |
| conf_level | 0.950 |
| N | 300 |
| N_complete | 300 |
| n_items | 6 |
Denominator: observed composite variance (robust omega)
Largest residual correlation under the single-factor model: 0.228 (y1 with y2)
Here the largest residual sits on the doublet pair. The residuals detect local dependence more reliably than they locate it: when the shared variance sits on the items that dominate the factor, the single factor absorbs it and the misfit appears among the other items.
When items are ordered categories (Likert responses, symptom counts), the composite people score is a sum of those observed categories, and the category thresholds make the item-factor relationship nonlinear. Categorical omega (Green & Yang, 2009) fits the probit threshold model and returns the reliability of the categorical sum score on its own metric. The threshold configuration decides how much this matters relative to treating the items as continuous. First the benign case, every item cut at the same thresholds:
items_same <- apply(items, 2, function(x)
as.integer(cut(x, breaks = c(-Inf, -1, -0.2, 0.6, Inf))))
colnames(items_same) <- colnames(items)
oc_same <- reliability_omega_categorical(data = items_same, ci_method = "none")
o_same <- reliability_omega(data = items_same, ci_method = "none")
c(categorical_omega = oc_same$value[oc_same$term == "estimate"],
treated_as_continuous = o_same$value[o_same$term == "estimate"])
#> categorical_omega treated_as_continuous
#> 0.7820753 0.7755024With similar threshold patterns across items the two approaches nearly agree, which is precisely the condition under which Kelley and Pornprasertmanit (2016, Study 3) found the continuous treatment held its interval coverage. In real item sets the threshold patterns typically differ across items: some items skew one way, some the other. Cutting the same underlying responses with threshold patterns that differ across items:
breaks_hi <- c(-Inf, 0.5, 1.2, 1.9, Inf)
breaks_lo <- c(-Inf, -1.9, -1.2, -0.5, Inf)
breaks_by_item <- list(breaks_hi, breaks_hi, breaks_hi,
breaks_lo, breaks_lo, breaks_lo)
items_cat <- sapply(seq_len(6), function(j)
as.integer(cut(items[, j], breaks = breaks_by_item[[j]])))
colnames(items_cat) <- colnames(items)
oc <- reliability_omega_categorical(data = items_cat, ci_method = "none")
o_as_cont <- reliability_omega(data = items_cat, ci_method = "none")
c(categorical_omega = oc$value[oc$term == "estimate"],
treated_as_continuous = o_as_cont$value[o_as_cont$term == "estimate"])
#> categorical_omega treated_as_continuous
#> 0.7190046 0.6755575The two now diverge, here by about 0.04, with the continuous
treatment understating on this configuration. The direction and size of
the continuous treatment’s error depend on the threshold configuration,
which is exactly what makes it untrustworthy as a general practice; in
Study 3 its interval coverage failed as threshold patterns diverged
across items. Categorical omega is constructed on the metric of the
summed categories themselves, so it is the coefficient to report for
ordinal items, which is why reliability() routes
integer-coded, few-category items to
reliability_omega_categorical() automatically.
Categorical omega and robust omega share a design principle in Kelley and Pornprasertmanit (2016): in both, the total variance in the denominator is not taken from the fitted factor model. Robust omega estimates it from the sample covariances; categorical omega, as computed there and by default here, assembles it from the unrestricted polychoric correlations and the thresholds. In each case the fitted single factor supplies only the numerator, so the coefficient stays anchored to the composite that is actually scored even when the factor model is an approximation.
Green and Yang (2009) defined their coefficient with the other
denominator. Their appendix program takes “the reproduced (model-based)
correlation matrix” as the input to the denominator, so the original
categorical omega is the model implied version, and correlated errors
enter its total variance only when they are specified in the model.
denominator = "model_implied" reproduces it:
oc_mi <- reliability_omega_categorical(data = items_cat,
ci_method = "none",
denominator = "model_implied")
c(unrestricted = oc$value[oc$term == "estimate"],
model_implied = oc_mi$value[oc_mi$term == "estimate"])
#> unrestricted model_implied
#> 0.7190046 0.7171456With a single factor generating these items the two nearly agree, as
the continuous pair did on the running example; they part ways as the
model misfits, and the residual polychoric correlations in the
residual_cor attribute show where.
The unrestricted denominator asks more of the data than the model
implied one. Every unrestricted polychoric correlation must be
estimable, and when two items are skewed in opposite directions their
bivariate table can have empty cells that leave the correlation without
a finite standard error: every respondent below the top category of one
item sits in the bottom category of the other. In Green and Yang’s own
mixed-skew design (two items piled into the top category, two into the
bottom), most samples of 500 have such a pair. The default then stops
with an error naming the pair rather than filling the correlation with
an arbitrary value, and denominator = "model_implied"
remains available, since the fitted model gives that correlation no
weight and supplies its own value for it. It does so only while the
correlations the data do determine identify the loadings, which takes a
link from every item to the others, directly or through other items, and
at least one cycle of an odd number of items (three items whose three
correlations are all determined, for example). When every correlation
between the two top-piled items and the two bottom-piled ones is
undetermined, the determined ones fix only the products of the loadings
within each pair: the relative sign of the pairs is free, and the fit
can return loadings of opposite signs and a categorical omega near 0.05
against a population value of 0.716. Both denominators then stop with an
error naming the pairs and the unlinked blocks of items. A bootstrap of
the default meets the problem more often still: the few respondents in
the sparse cells are what determine the correlation, and a resample
often leaves some of them out, so in Green and Yang’s design at a sample
size of 1,000 about half the replications of a bootstrap of the default
are set aside even when the sample itself determines every correlation.
The printed bootstrap line and a warning count them, and the model
implied denominator keeps all but those whose remaining correlations
leave the loadings unidentified or whose solution is improper: 1 of 200
in a bootstrap of the help page’s complete data, of 1,000 respondents,
but a median of 42% in samples of 300 from the same design. When fewer
than half of the replications return a value, no interval is reported,
since the rest would be a minority of the resamples, chosen by whether
they returned one; a larger sample is the remedy. The help page of
reliability_omega_categorical() works through that
design.
Coefficient H (Hancock & Mueller, 2001) makes the same congeneric claim as omega but changes the composite: it is the reliability of the optimally weighted combination of the items, the best possible composite the items can form, rather than the unit-weighted sum. The two coincide when loadings are equal and diverge as loadings spread, because optimal weights exploit the stronger items:
| term | value |
|---|---|
| estimate | 0.847 |
| se | 0.014 |
| ci_lower | 0.817 |
| ci_upper | 0.873 |
| conf_level | 0.950 |
| N | 300 |
| N_complete | 300 |
| n_items | 6 |
c(omega = res_omega$value[res_omega$term == "estimate"],
H = res_H$value[res_H$term == "estimate"])
#> omega H
#> 0.8067366 0.8472086From raw data, reliability_H() fits the single-factor
model exactly as reliability_omega() does, and its interval
is a Wald interval on the logit scale whose standard error comes from
the delta method over the full covariance matrix of that fit’s parameter
estimates, robust maximum likelihood by default, as for omega. When the
factor belongs to a larger confirmatory or structural model, pass the
fitted model instead, reliability_H(fit, factor = "..."),
and the standard error is the one that model was fitted with. The form
reliability_H(loadings = , se_loadings = ) is not used
here. It builds the interval from the marginal standard errors of the
standardized loadings, which leave out the covariances among the loading
estimates; those estimates are correlated in any fitted factor model, so
its interval can be too narrow or too wide, and every such call gives a
warning saying so.
Report omega when people will score the scale by summing; report H
when scores will come from the factor model itself (or to see the
ceiling the item set could reach with optimal weighting). Reporting
both, with one sentence on which composite each describes, is often the
most informative choice under unequal loadings. Read H as a ceiling,
too, in the statistical sense: the sample H overestimates the population
coefficient, and a composite formed with weights estimated from the
sample is less reliable than H says, both markedly in samples of about
100 and barely beyond about 1,000 (Aguirre-Urreta, Rönkkö, &
McIntosh, 2019). A bootstrap interval for H
(ci_method = "bca", as for omega) also estimates that bias.
Reliability coefficients matter because researchers score and use
composites; an analysis conducted entirely within the SEM framework,
relating latent variables directly, has less need of them.
KR-20 completes the family as the binary-item special case of alpha,
provided for the classical test theory literature that names it
separately; reliability_kr20() and
reliability_alpha() agree on 0/1 data.
Choosing the interval is an empirical question, and it is the subject of Kelley and Pornprasertmanit (2016): across coefficients, sample sizes, loading patterns, distributions, and (for categorical items) threshold patterns, which interval procedures cover at their nominal rate. The DMAR defaults are their recommendations, with one refinement for omega described below the table:
| Coefficient | Default behavior | Recommended interval and basis |
|---|---|---|
| alpha | Bonett (2002) interval | well behaved among the closed forms under normality |
| omega, model implied | Wald interval with the robust ML standard error, its limits formed
on the logit scale ("mlr_logistic") |
the best performing closed form when item distributions deviate from normality; the limits are formed on the logit scale on the evidence of a later study |
| robust omega | point estimate; a message names the bootstrap call | percentile or BCa bootstrap, the pairing recommended when unidimensionality is approximate; no closed form exists for the observed denominator |
| categorical omega | point estimate; a message names the bootstrap call | BCa bootstrap, which held coverage across threshold patterns, category counts, and sample sizes |
The logit form for omega comes from a Monte Carlo study run for DMAR
(in the package’s source repository,
methodology/reliability_limits), which compared the two
forms of the Wald limits from the same standard error: the normal theory
maximum likelihood one for normal data and the Huber-White one of MLR
for nonnormal data. With normal data the limits on the scale of omega
missed mostly on one side, the population value falling below the lower
limit, and covered from .910 to .952; formed on the logit scale and
transformed back (Browne, 1982; Kelley, 2007), they covered from .939 to
.956 in every cell. With complete data the default takes lavaan’s
robust.sem standard error, which no cell used; it estimates
the same asymptotic variance as the Huber-White one.
fa_confirmatory() forms the limits of its omega row on the
logit scale too. ci_method = "mlr" forms the limits on the
scale of omega. The examples below name the default,
ci_method = "mlr_logistic", where they use it; leaving it
out gives the same interval.
Coefficient H was not among the coefficients Kelley and Pornprasertmanit (2016) compared. Its default is omega’s: a Wald interval on the logit scale with the robust ML standard error, the fast choice, with the percentile or BCa bootstrap the safer one for a reported analysis.
A bootstrap is never run unless it is requested. When one is
requested, B = 10000 replications is the default.
Every reliability function offers its interval methods through
ci_method (Feldt, Fisher, Bonett, Hakstian-Whalen, ML,
robust ML, ADF, their logistic transformed variants, and the bootstrap
family, as applicable). Except for coefficient H, the default is the
method with simulation evidence behind it. Two practical notes. The
bootstrap methods need raw data, not a covariance matrix, and take a
seed argument for reproducibility. And a bootstrap interval
on the categorical coefficient refits an ordinal factor model per
replicate, so it is the slowest interval in the family; the default
B = 10000 is a deliberate accuracy choice, worth the wait
for a final analysis and worth reducing while exploring.
# The default: robust omega, point estimate, with the message naming
# the bootstrap call.
reliability_omega(data = items)
#> Robust omega is reported without a confidence interval by default because its interval is bootstrap based. Request it with ci_method = "percentile" (or "bca"); B = 10000 replications is the default when you do.| term | value |
|---|---|
| estimate | 0.806 |
| se | NA |
| ci_lower | NA |
| ci_upper | NA |
| conf_level | 0.950 |
| N | 300 |
| N_complete | 300 |
| n_items | 6 |
Denominator: observed composite variance (robust omega)
Largest residual correlation under the single-factor model: -0.096 (y1 with y3)
# The recommended interval, requested explicitly (B reduced here to
# keep the vignette fast; the default is B = 10000).
reliability_omega(data = items, ci_method = "percentile",
B = 500, seed = 113)| term | value |
|---|---|
| estimate | 0.806 |
| se | 0.017 |
| ci_lower | 0.771 |
| ci_upper | 0.838 |
| conf_level | 0.950 |
| N | 300 |
| N_complete | 300 |
| n_items | 6 |
Denominator: observed composite variance (robust omega)
Largest residual correlation under the single-factor model: -0.096 (y1 with y3)
Bootstrap: 500 replications attempted, 500 returned a value
# The model implied denominator has a closed form and reports its
# robust ML Wald interval by default, with the limits formed on the
# logit scale. The call names that default, ci_method = "mlr_logistic";
# leaving it out gives the same interval.
reliability_omega(data = items, denominator = "model_implied",
ci_method = "mlr_logistic")| term | value |
|---|---|
| estimate | 0.807 |
| se | 0.017 |
| ci_lower | 0.772 |
| ci_upper | 0.837 |
| conf_level | 0.950 |
| N | 300 |
| N_complete | 300 |
| n_items | 6 |
Denominator: composite variance implied by the fitted model
Largest residual correlation under the single-factor model: -0.096 (y1 with y3)
The mlmr() vignette makes the case at length: listwise
deletion is unbiased only when values are missing completely at random,
and it is inefficient always. The same argument applies to a reliability
analysis, so reliability_alpha() and
reliability_omega() (and the reliability()
wrapper) take the same position mlmr() does. Two arguments
govern the treatment:
missing = c("fiml", "listwise", "two_stage"). The
default, "fiml", keeps every case with at least one
observed item, or with an auxiliary variable observed, and estimates by
full information maximum likelihood, consistent and efficient under the
weaker missing at random (MAR) assumption; "listwise"
analyzes the complete cases alone. Data in which no value is missing are
analyzed as they stand, with the same estimates FIML would give and the
complete data intervals, so the default analysis is FIML whenever an
item value is missing and the complete data analysis otherwise.
fa_confirmatory() differs: it fits complete data by FIML
too. The complete data analysis is used here because the closed-form
intervals of alpha and omega (Bonett’s among them) need it. Without
auxiliary variables a row with every item missing does not count as a
missing value, since listwise deletion and FIML both drop it. With
auxiliary variables, a case whose items are all missing still informs
the item moments through its auxiliaries, and setting it aside would
make the analysis conditional on having an observed item, which biases
the estimates when the items go missing together at random given the
auxiliaries.aux, a character vector naming auxiliary
variables: columns of data that are not part of the
composite but are correlated with the items or with the reasons values
are missing. They enter as saturated correlates (Graham, 2003):
correlated freely with each other and with every item’s residual, never
loading on the factor and never entering the composite. The measurement
model is undisturbed while FIML uses their information, and missingness
that depends on the auxiliary becomes MAR once the auxiliary is
conditioned on (Collins, Schafer, & Kam, 2001). Supplying
aux implies missing = "fiml".# Missingness on y2 that depends on an auxiliary z (MAR given z):
# listwise deletion is biased here, FIML with z is not.
z <- eta + rnorm(N, sd = 0.5)
d <- data.frame(items, z = z)
d$y2[runif(N) < plogis(-1 + 1.5 * as.numeric(scale(z)))] <- NA
# Listwise, asked for by name: the analysis drops the incomplete rows.
reliability_alpha(data = d[, paste0("y", 1:6)], missing = "listwise")| term | value |
|---|---|
| estimate | 0.762 |
| se | 0.025 |
| se_transformed | 0.106 |
| ci_lower | 0.707 |
| ci_upper | 0.807 |
| conf_level | 0.950 |
| N | 215 |
| N_complete | 215 |
| n_items | 6 |
# FIML with the auxiliary: every case with an observed item or an
# observed z contributes, and z informs the estimation.
reliability_alpha(data = d, aux = "z")| term | value |
|---|---|
| estimate | 0.796 |
| se | 0.019 |
| ci_lower | 0.759 |
| ci_upper | 0.833 |
| conf_level | 0.950 |
| N | 300 |
| N_complete | 215 |
| n_items | 6 |
The returned table always reports both N (the cases
used) and N_complete (the complete cases), so the cost of
listwise deletion is legible at a glance. The classical (analytic) alpha
under FIML is still the classical coefficient: the same formula, applied
to the FIML estimate of the item covariance matrix rather than the
complete-case one. Interval methods that cannot be made correct under
FIML (the complete-data closed forms, ADF, the profile likelihood) are
refused with an explanation rather than silently reverting to listwise
deletion; the delta method ("ml", "mlr") and
bootstrap intervals remain available. Users of MBESS may recognize the
aux idea from ci.reliability(), whose
implementation no longer runs on current semTools; DMAR implements the
saturated correlates model directly.
Bootstrap intervals require at least half of attempted replications and at least two defined estimates; AVE also retains a minimum of 100. A point estimate and counts remain available when an interval is withheld. These rules describe estimability and selection among resamples, not a guarantee of coverage. Defined improper values are retained under the stated policy and counted separately.
For categorical omega with multiple imputation, the default is 20 completed datasets inside each bootstrap replication. The point estimate has a separate adaptive rule that uses at least 20. More inner imputations reduce imputation noise at additional computational cost; the default of 20 does not rest on a calibration of coverage. A fraction of missing information estimated from a selected minority of replications is explicitly qualified and cannot certify precision merely by increasing the point’s number of imputations.
A planning sensitivity result takes the probability of obtaining an
interval within the target width, the mean width, and the coverage over
one denominator, the simulated studies that returned an interval. A
study that returns no interval is in none of them; the output states how
many studies were attempted, how many returned an interval, and how many
did not, and its simulation attribute gives Monte Carlo
limits for the width-target probability.
reliability() dispatches on type (or
auto-detects: ordered few-category items go to categorical omega,
otherwise omega) and forwards denominator and
ci_method (here the default of the model implied omega,
named); every family member returns the same tidy shape with
tidy() and glance() methods:
res <- reliability(data = items, type = "omega",
denominator = "model_implied",
ci_method = "mlr_logistic")
generics::tidy(res)
#> term estimate se ci_lower ci_upper conf_level wald_df
#> 1 omega 0.8067366 0.01652757 0.7722706 0.8370868 0.95 Inf
#> h1_diagnostic wald_calibration_note
#> 1 <NA> <NA>
generics::glance(res)
#> coefficient estimate se ci_lower ci_upper conf_level nobs n_items
#> 1 omega 0.8067366 0.01652757 0.7722706 0.8370868 0.95 300 6
#> ci_method wald_df h1_diagnostic wald_calibration_note
#> 1 mlr_logistic Inf <NA> <NA>Reliability estimation is also a design problem: a reliability
coefficient reported without a narrow interval is a weak claim.
ss_aipe_reliability() plans the sample size so the interval
for alpha or omega achieves a target width (with an
assurance probability if desired), which closes the loop
between this family and DMAR’s accuracy in parameter estimation (AIPE)
tradition. It plans for the coefficient this family estimates from the
raw item scores, so the population is used as a covariance matrix (a
correlation matrix plans for standardized items). Without an assurance
the size comes from a closed form for the expected width of the interval
type names: the normal theory variance of alpha (van Zyl,
Neudecker, & Nel, 2000), evaluated on the population covariance
matrix, for the parallel model and for the normal theory interval of the
tau equivalent model, and the delta method variance of the model implied
coefficient, the variance behind
reliability_omega(denominator = "model_implied", ci_method = "ml")
and
reliability_alpha(estimator = "model_implied", ci_method = "ml"),
for the factor analytic interval of the tau equivalent model and for the
congeneric model. With an assurance, a Monte Carlo search that starts
from that size finds the smallest one whose interval is no wider than
the target with the requested probability (Terry & Kelley, 2012); a
simulated study whose fit returns no interval counts as one whose
interval is too wide.
Reliability is a property of a test score in a particular population, not a property of the items in the abstract. A high alpha or omega does not mean:
Single-administration coefficients also book item-specific systematic variance as error, so all of the coefficients here are lower bounds on classical test-retest reliability. Use the family for what it is: an internal-consistency analysis conditional on a substantive theory of what the items mean.
?reliability, ?reliability_alpha,
?reliability_omega,
?reliability_omega_categorical,
?reliability_kr20, ?reliability_H.?ss_aipe_reliability for sample size planning.?cfa_1 for the one-factor CFA underlying the omega
family, and ?average_variance_extracted and
?htmt for the validity companions.Aguirre-Urreta, M. I., Rönkkö, M., & McIntosh, C. N. (2019). A cautionary note on the finite sample behavior of maximal reliability. Psychological Methods, 24, 236–252.
Bonett, D. G. (2002). Sample size requirements for testing and estimating coefficient alpha. Journal of Educational and Behavioral Statistics, 27, 335–340.
Browne, M. W. (1982). Covariance structures. In D. M. Hawkins (Ed.), Topics in applied multivariate analysis (pp. 72–141). Cambridge University Press.
Collins, L. M., Schafer, J. L., & Kam, C. M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330–351.
Graham, J. W. (2003). Adding missing-data-relevant variables to FIML-based structural equation models. Structural Equation Modeling, 10, 80–100.
Green, S. B., & Yang, Y. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74, 155–167.
Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. In R. Cudeck, S. du Toit, & D. Sorbom (Eds.), Structural equation modeling: Present and future (pp. 195–216). Scientific Software International.
Holzinger, K. J., & Swineford, F. (1939). A study in factor analysis: The stability of a bi-factor solution (Supplementary Educational Monographs, No. 48). University of Chicago Press.
Kelley, K. (2007). Confidence intervals for standardized effect sizes: Theory, application, and implementation. Journal of Statistical Software, 20(8), 1–24.
Kelley, K., & Pornprasertmanit, S. (2016). Confidence intervals for population reliability coefficients: Evaluation of methods, recommendations, and software for composite measures. Psychological Methods, 21, 69–92.
McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum Associates.
Terry, L. J., & Kelley, K. (2012). Sample size planning for composite reliability coefficients: Accuracy in parameter estimation via narrow confidence intervals. British Journal of Mathematical and Statistical Psychology, 65, 371–401.
van Zyl, J. M., Neudecker, H., & Nel, D. G. (2000). On the distribution of the maximum likelihood estimator of Cronbach’s alpha. Psychometrika, 65, 271–280.