| Title: | Automatic Correlation Method Selection Based on Variable Types |
| Version: | 1.2.0 |
| Description: | Detects variable types (continuous, count, binary, ordinal, categorical), checks outliers, normality, latent normality, and linearity, and selects the appropriate correlation method for each pair. Supports 18 methods: Pearson, Spearman, Kendall's tau, point-biserial, biserial, rank-biserial, phi, tetrachoric, Yule's Q, polychoric, polyserial, Goodman-Kruskal's gamma, the correlation ratio (eta-squared), Kruskal-Wallis eta-squared, Freeman's theta, Cramer's V, Tschuprow's T, and Theil's U, each with a confidence interval and p-value. Explains the selection rationale in the output, follows tidy data principles, and works in both interactive and scripted workflows. The methodology is described in Harshvardhan and Ranjan (2026) <doi:10.48550/arXiv.2607.22285>. |
| License: | GPL (≥ 3) |
| URL: | https://harshvardhaniimi.github.io/smartcor/, https://github.com/harshvardhaniimi/smartcor |
| BugReports: | https://github.com/harshvardhaniimi/smartcor/issues |
| Encoding: | UTF-8 |
| Depends: | R (≥ 3.6.0) |
| Imports: | cli (≥ 3.0.0), fNonlinear, generics, ggplot2 (≥ 3.4.0), graphics, grDevices, lmtest, Matrix, moments, MVN (≥ 6.0), mvtnorm, polycor, rlang (≥ 1.0.0), stats, tibble, utils, withr |
| Suggests: | effectsize, knitr, MASS, rmarkdown, rstatix, testthat (≥ 3.0.0) |
| VignetteBuilder: | knitr |
| Config/roxygen2/version: | 8.1.1 |
| RoxygenNote: | 7.3.3 |
| NeedsCompilation: | no |
| Packaged: | 2026-10-26 13:42:31 UTC; harshvardhan |
| Author: | M. Harshvardhan [aut, cre, cph], Pritam Ranjan [aut, cph] |
| Maintainer: | M. Harshvardhan <harshvardhan@aus.edu> |
| Repository: | CRAN |
| Date/Publication: | 2026-10-26 14:10:02 UTC |
smartcor: Automatic Correlation Method Selection Based on Variable Types
Description
Detects variable types (continuous, count, binary, ordinal, categorical), checks outliers, normality, latent normality, and linearity, and selects the appropriate correlation method for each pair. Supports 18 methods: Pearson, Spearman, Kendall's tau, point-biserial, biserial, rank-biserial, phi, tetrachoric, Yule's Q, polychoric, polyserial, Goodman-Kruskal's gamma, the correlation ratio (eta-squared), Kruskal-Wallis eta-squared, Freeman's theta, Cramer's V, Tschuprow's T, and Theil's U, each with a confidence interval and p-value. Explains the selection rationale in the output, follows tidy data principles, and works in both interactive and scripted workflows. The methodology is described in Harshvardhan and Ranjan (2026) doi:10.48550/arXiv.2607.22285.
Author(s)
Maintainer: M. Harshvardhan harshvardhan@aus.edu [copyright holder]
Authors:
Pritam Ranjan pritamr@iimidr.ac.in [copyright holder]
See Also
Useful links:
Report bugs at https://github.com/harshvardhaniimi/smartcor/issues
Which Methods Apply to a Pair of Variable Types
Description
Lists the methods smart_cor() can compute for two variable types.
Use it before forcing a method.
Usage
available_methods(type_x, type_y)
Arguments
type_x, type_y |
Character strings: |
Value
A tibble with columns method (code name) and label
(human-readable description).
Examples
path = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(path)
available_methods(detect_type(gss$degree), detect_type(gss$happy))
Run Every Applicable Method on One Pair
Description
Computes the correlation between x and y with every method that
applies to their type combination and lines the results up side by side.
Use it to check how much the method choice changes the estimate.
Usage
compare_methods(
x,
y,
x_type = NULL,
y_type = NULL,
assume_latent_normal = "auto",
alpha = 0.05,
ordinal_threshold = 10,
ignore_na = TRUE,
conf_level = 0.95,
bootstrap = "auto",
n_boot = 500,
n_perm = 500,
seed = NULL,
polyserial_ml = FALSE,
verbose = TRUE
)
Arguments
x, y |
Vectors of the same length. May be numeric, factor, character, or logical. |
x_type, y_type |
Optional character strings to override automatic
type detection. One of |
assume_latent_normal |
|
alpha |
Significance level for the diagnostics that decide the
|
ordinal_threshold |
Integer passed to |
ignore_na |
Logical. If |
conf_level |
Numeric between 0 and 1. Confidence level for every
interval in the table. Default: |
bootstrap |
Character or logical: |
n_boot |
Integer. Bootstrap replications per method. Default: |
n_perm |
Integer. Permutation shuffles for the Theil's U and
Freeman's theta p-values. Default: |
seed |
Optional integer seed for the bootstrap and permutation draws; the global RNG state is saved and restored. |
polyserial_ml |
Logical. Fit polyserial and biserial by full maximum
likelihood. Default: |
verbose |
Logical. If |
Details
The function computes every method listed by available_methods() for
the detected type pair. The recommended column marks the method that
smart_cor() would select under the assume_latent_normal setting,
so the recommended row always matches what smart_cor() and
smart_cormat() return.
When one variable is continuous or count and the other is categorical, only the Cramer's V row bins the continuous or count variable into quintiles; the two eta-squared measures use the raw values.
Methods that fail (a latent-variable optimiser that does not converge,
say) are kept in the output with estimate = NA.
Value
An object of class "smartcor_comparison": a list with elements:
resultsA
tibble::tibble()with columnsmethod,method_label,estimate,statistic,p.value,ci_lower,ci_upper,ci_method,ci_source, andrecommended(logical flag for the default method).x_nameName of the first variable.
y_nameName of the second variable.
x_typeDetected (or specified) type of
x.y_typeDetected (or specified) type of
y.n_completeNumber of complete observations used.
conf_levelConfidence level used for the intervals.
Examples
csv = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(csv)
# Two ordinal variables: Kendall, Spearman, gamma, and polychoric
compare_methods(gss$degree, gss$happy,
assume_latent_normal = FALSE, bootstrap = FALSE)
Flag Outliers with the Outer-Fence Rule
Description
Flags values beyond Tukey's outer fences, Q1 - k * IQR and
Q3 + k * IQR, with quartiles of type 7 (the R default). smart_cor()
uses this rule as the first diagnostic for every pair with a continuous or
count variable.
Usage
detect_outliers(x, k = 3)
## S3 method for class 'smartcor_outliers'
print(x, ...)
Arguments
x |
A numeric vector (continuous, count, or ordinal). An ordered factor is replaced by its integer codes. Missing values are dropped. |
k |
Fence multiplier. Default: |
... |
Ignored. |
Details
Values strictly below the lower fence or strictly above the upper fence
are outliers. When the IQR is 0 (more than half of the values are tied)
the rule is degenerate: nothing is flagged, degenerate is TRUE, and the
repeated values need separate inspection.
The skewness and excess kurtosis are moment estimators with divisor n, as used by the Jarque-Bera test.
Value
An object of class "smartcor_outliers": a list with
has_outliers (logical), outliers (the outlying values), index
(their positions in x), lower and upper (the fences), q1, q3,
iqr, degenerate (TRUE when the IQR is 0), skewness,
excess_kurtosis, and n (the number of non-missing values).
Examples
csv = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(csv)
detect_outliers(gss$coninc)
Detect the Statistical Type of a Variable
Description
Classifies a variable as continuous, binary, ordinal, count, or categorical from its R class and the number of distinct non-missing values. The rest of the package leans on this to pick correlation methods, but it is exported because the classification is useful on its own.
Usage
detect_type(x, ordinal_threshold = 10, detect_count = TRUE, verbose = FALSE)
Arguments
x |
A vector (numeric, factor, character, or logical). |
ordinal_threshold |
Integer. Numeric variables with this many or fewer
unique non-missing values are classified as ordinal. Default: |
detect_count |
Logical. If |
verbose |
Logical. If |
Details
The detection follows these rules, in order:
-
Any variable with exactly 2 unique values is
"binary", whatever its class (a two-level ordered factor included). -
Ordered factors with more than 2 levels are
"ordinal". -
Unordered factors and character vectors with more than 2 unique values are
"categorical". -
Logical vectors are classified as
"binary". -
Numeric vectors with
ordinal_threshold(10 by default) or fewer unique values are"ordinal". -
Numeric vectors that are non-negative integers with more than
ordinal_thresholdunique values are classified as"count". Disable by settingdetect_count = FALSE. All other numeric vectors are
"continuous".
The ordinal_threshold parameter controls the heuristic for numeric
variables: if a numeric variable has this many distinct values or fewer, it
is assumed to represent ordered categories (e.g., a 1–5 Likert scale
stored as integers). Set to 0 to disable this heuristic and treat all
numeric variables as continuous (or count, if non-negative
integer-valued).
Value
A single character string: "continuous", "binary", "ordinal",
"count", or "categorical".
Examples
path = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(path)
vapply(gss, detect_type, character(1))
ggplot2 Heatmaps for Smart Correlation Matrices
Description
Two heatmaps for smart_cormat() output. ggcor_heatmap() shows the
correlation values on a diverging scale centred at zero.
ggcor_method_heatmap() colours each cell by the selected method.
Usage
ggcor_heatmap(
x,
palette = "blue_red",
digits = 2,
title = NULL,
show_values = TRUE,
text_size = 3.5,
lower_only = FALSE,
...
)
ggcor_method_heatmap(
x,
title = NULL,
show_values = TRUE,
text_size = 3.5,
digits = 2,
...
)
Arguments
x |
A |
palette |
Character string selecting the colour palette.
For |
digits |
Integer. Number of decimal places in cell labels. Default: |
title |
Optional plot title. If |
show_values |
Logical. Show correlation values in cells? Default: |
text_size |
Numeric. Base text size for cell labels. Default: |
lower_only |
Logical. Show only the lower triangle? Default: |
... |
Currently ignored. |
Details
Both functions use the ggplot2 package.
Value
A ggplot object that can be customised further.
Examples
csv = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(csv)
mat = smart_cormat(
gss[, c("age", "coninc", "degree", "sex")],
assume_latent_normal = FALSE,
verbose = FALSE
)
# correlation heatmap
ggcor_heatmap(mat)
# lower triangle only
ggcor_heatmap(mat, lower_only = TRUE)
# method-selection heatmap
ggcor_method_heatmap(mat)
# customise further with ggplot2
ggcor_heatmap(mat) + ggplot2::labs(caption = "Frozen GSS 2024 extract")
Plot a Correlation Heatmap
Description
Draws an annotated heatmap of a smart correlation matrix with base R. Each cell is coloured by the correlation and labelled with its value.
Usage
## S3 method for class 'smartcormat'
plot(
x,
palette = "blue_red",
digits = 2,
title = NULL,
show_values = TRUE,
text_size = 1,
...
)
plot_cor_heatmap(
x,
palette = "blue_red",
digits = 2,
title = NULL,
show_values = TRUE,
text_size = 1,
...
)
Arguments
x |
A |
palette |
Character string selecting the colour palette. One of:
|
digits |
Integer. Number of decimal places shown in cell labels.
Default: |
title |
Optional character string for the plot title. If |
show_values |
Logical. Whether to display numeric values in each cell.
Default: |
text_size |
Numeric. Scaling factor for cell text size. Default: |
... |
Additional arguments (currently ignored). |
Details
The function works with any object that contains a correlations element
(matrix or data frame). This includes objects returned by smart_cormat()
and smart_cor_df().
Custom colour palettes can be supplied by passing a character vector of
colours to palette instead of a named preset. The vector should contain
at least 3 colours and will be interpolated to 100 steps.
Value
Invisibly returns the correlation matrix (numeric).
Examples
csv = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(csv)
# Plot a smart_cormat result
mat = smart_cormat(
gss[, c("age", "coninc", "degree", "sex")],
assume_latent_normal = FALSE,
verbose = FALSE
)
plot(mat)
# different palettes
plot(mat, palette = "purple_green")
plot(mat, palette = "viridis")
# Plot a smart_cor_df result
res = smart_cor_df(gss[, c("age", "coninc", "degree", "sex")],
assume_latent_normal = FALSE)
plot_cor_heatmap(res)
# custom colour vector
plot_cor_heatmap(res, palette = c("darkblue", "white", "darkred"))
Print a smartcor Object
Description
Shows the estimate, the method that was chosen and why, the inference payload, and one line for each diagnostic that ran (outer-fence rule, Mardia or Jarque-Bera, the LR latent-normality test, and the linearity test).
Usage
## S3 method for class 'smartcor'
print(x, ...)
Arguments
x |
A |
... |
Ignored. |
Value
Invisibly returns x.
Print a Method Comparison
Description
Print a Method Comparison
Usage
## S3 method for class 'smartcor_comparison'
print(x, digits = 4, ...)
Arguments
x |
A |
digits |
Integer. Number of decimal places. Default: |
... |
Ignored. |
Value
Invisibly returns x.
Print a Linearity Test
Description
Print a Linearity Test
Usage
## S3 method for class 'smartcor_lintest'
print(x, ...)
Arguments
x |
A |
... |
Ignored. |
Value
Invisibly returns x.
Print a Normality Test
Description
Print a Normality Test
Usage
## S3 method for class 'smartcor_normtest'
print(x, ...)
Arguments
x |
A |
... |
Ignored. |
Value
Invisibly returns x.
Print a smartcormat Object
Description
Prints the correlation matrix, the two-letter method codes for each cell, and a legend for the codes that appear.
Usage
## S3 method for class 'smartcormat'
print(x, digits = 2, show_methods = TRUE, ...)
Arguments
x |
A |
digits |
Integer. Number of significant figures. Default: |
show_methods |
Logical. Whether to print the method matrix. Default:
|
... |
Ignored. |
Value
Invisibly returns x.
Objects exported from other packages
Description
These objects are imported from other packages. Follow the links below to see their documentation.
- generics
Choose and Compute a Correlation
Description
Detects the types of x and y, selects a suitable correlation method,
and returns the estimate with its confidence interval and p-value.
Usage
smart_cor(
x,
y,
x_type = NULL,
y_type = NULL,
method = NULL,
assume_latent_normal = "auto",
ordinal_threshold = 10,
ignore_na = TRUE,
conf_level = 0.95,
alpha = 0.05,
bootstrap = "auto",
n_boot = 500,
n_perm = 500,
seed = NULL,
polyserial_ml = FALSE,
verbose = TRUE
)
Arguments
x, y |
Vectors of the same length. May be numeric, factor, character, or logical. |
x_type, y_type |
Optional character strings ( |
method |
Optional character string to force a specific method. One of
|
assume_latent_normal |
|
ordinal_threshold |
Integer passed to |
ignore_na |
Logical. If |
conf_level |
Numeric between 0 and 1. Confidence level for intervals.
Default: |
alpha |
Significance level for every diagnostic test (Mardia,
Jarque-Bera, the LR latent-normality test, and |
bootstrap |
Character or logical. Controls bootstrap inference:
|
n_boot |
Integer. Number of bootstrap replications when
bootstrapping. Default: |
n_perm |
Integer. Number of shuffles for the permutation p-values of
Theil's U and Freeman's theta. Default: |
seed |
Optional integer seed for the bootstrap and permutation
draws. The global |
polyserial_ml |
Logical. How polyserial and biserial correlations are
fitted. |
verbose |
Logical. If |
Details
The method selection follows Harshvardhan, M. and Ranjan, P. (2026),
"smartcor: Intelligent Correlation Method Selection for Mixed Variable
Types" (arXiv:2607.22285, doi:10.48550/arXiv.2607.22285); the package vignettes
(browseVignettes("smartcor")) cover the same material in more depth:
| Variable types | Default method | Alternative |
| Correlation methods (signed, from -1 to +1) | ||
| continuous + continuous | Pearson (linear, no outliers) | Spearman (outliers or nonlinear), Kendall's tau |
| continuous + binary | Biserial (if latent normality holds) | Point-biserial; rank-biserial (outliers or non-normal) |
| binary + binary | Pearson = phi | Tetrachoric (latent normal), Yule's Q (no latent continuum) |
| continuous + ordinal | Polyserial (if latent normality holds) | Kendall's tau (m <= 5) or Spearman |
| ordinal + ordinal | Polychoric (if latent normality holds) | Kendall's tau (m <= 5) or Spearman; Goodman-Kruskal gamma |
| binary + ordinal | Polychoric (if latent normality holds) | Rank-biserial; Kendall's tau, Spearman |
| Association measures (unsigned, from 0 to 1) | ||
| continuous + categorical | Correlation ratio \eta^2 | Kruskal-Wallis \eta^2_H (outliers or non-normal); Cramer's V (binned) |
| binary + categorical | Cramer's V | -- |
| ordinal + categorical | Freeman's theta | Cramer's V |
| categorical + categorical | Cramer's V | Theil's U, Tschuprow's T |
Count variables are treated as continuous, so each count combination follows the corresponding continuous row. m is the number of categories of the ordinal variable (the smaller of the two for an ordinal pair).
The selection starts from diagnostics run at significance level alpha:
The outer-fence rule (
detect_outliers(), Q1 - 3 IQR and Q3 + 3 IQR) is applied to each continuous variable of a continuous-continuous pair, to the continuous variable of a continuous-ordinal pair, and to the residuals around the group means for continuous-binary and continuous-categorical pairs. It is never applied to ordinal variables.Normality: Mardia's test (
MVN::mardia()) for two continuous variables; the Jarque-Bera test (moments::jarque.test()) on the continuous variable (continuous-ordinal) or on the within-group residuals (continuous-binary, continuous-categorical).Latent normality: the LR chi-square test of
test_bivariate_normality()on the (binned) contingency table.Linearity:
test_linearity()for two continuous variables.
A continuous-continuous pair gets Spearman when the outer fence flags
either variable, and otherwise follows test_linearity(), which runs its
diagnostics in both directions. When Pearson is selected and Mardia's test
rejects, the rationale says the interval and p-value are approximate.
For continuous-binary, continuous-ordinal, and continuous-categorical
pairs, a fence flag or a Jarque-Bera rejection selects the rank-based
method (rank-biserial, Kendall/Spearman, Kruskal-Wallis
\eta^2_H). Otherwise, and for binary-ordinal and ordinal
pairs, the latent-normality node decides between the latent-variable
method and the observed-scale one. A binary-binary pair under "auto"
gets phi: its 2 x 2 table is saturated (df = 0), so latent normality
cannot be tested. Inconclusive tests keep Pearson and report the reason.
The linearity decision does not depend on the order of x and y. An
explicit method bypasses all diagnostics. The tests that ran are
returned in diagnostics.
When a latent-variable estimator fails numerically (singular Hessian,
sparse table), the function falls back to Kendall's tau, Spearman, or
point-biserial and the returned method, method_label, and rationale
name the method actually used, not the one that failed.
One caveat on the point-biserial interval: it reuses the Fisher-z variance 1/(n - 3), which assumes bivariate normality; a binary margin cannot satisfy that exactly, so treat the interval as approximate.
Theil's U is asymmetric: method = "theils_u" computes U(y|x), the
proportion of the entropy in y explained by x. Swap the arguments
for the other direction. Its p-value, like that of Freeman's theta, comes
from a permutation test (n_perm shuffles), not from the bootstrap.
Kruskal-Wallis \eta^2_H, Freeman's theta, and Theil's U
have no analytic interval and get a percentile bootstrap under
bootstrap = "auto".
Value
An object of class "smartcor": a named list with elements
estimate, method, method_label, x_type, y_type, n,
n_complete, statistic, p.value, ci_lower, ci_upper,
conf_level, ci_method, ci_source, p_method, null_hypothesis,
p_interpretation, alternatives, rationale, normality_test (the
LR latent-normality test, when it ran), diagnostics, outlier_flag,
normality_p, latent_p, linearity_p, and linearity_test (when the
residual diagnostic rule ran). diagnostics is a list with outliers
(a detect_outliers() result, or a list of two for a
continuous-continuous pair), normality (Mardia or Jarque-Bera),
latent (the LR test), and linearity (the test_linearity()
result); entries that were not computed are NULL. The four summary
fields are NA when not computed. Has custom print.smartcor() and
tidy.smartcor() methods.
Examples
path = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(path)
# Continuous and binary variables: the diagnostics pick the method.
smart_cor(gss$coninc, gss$sex, verbose = FALSE)
# Two ordinal variables use Kendall's tau when latent normality is not assumed.
smart_cor(gss$degree, gss$happy,
assume_latent_normal = FALSE, verbose = FALSE)
# Force a method when the analysis calls for it.
smart_cor(gss$coninc, gss$age, method = "spearman", verbose = FALSE)
Correlation Matrix as Plain Data Frames
Description
A thin wrapper around smart_cormat() for pipeline work. It never
prompts, prints nothing, and returns two plain data frames (the estimates
and the method used for each pair) instead of an S3 object.
Usage
smart_cor_df(
data,
cols = NULL,
types = NULL,
assume_latent_normal = "auto",
ordinal_threshold = 10,
ignore_na = TRUE,
conf_level = 0.95,
alpha = 0.05,
bootstrap = "auto",
n_boot = 500,
n_perm = 500,
seed = NULL,
polyserial_ml = FALSE,
nearPD = FALSE
)
Arguments
data |
A data frame or tibble. |
cols |
Optional character vector of column names to include. Default: all columns. |
types |
Optional named list mapping column names to types
( |
assume_latent_normal |
|
ordinal_threshold |
Integer passed to |
ignore_na |
Logical. If |
conf_level |
Numeric between 0 and 1. Confidence level passed
through to each |
alpha |
Significance level for the diagnostic tests, passed
through to each |
bootstrap |
Character or logical, passed through: |
n_boot |
Integer. Bootstrap replications per pair. Default: |
n_perm |
Integer. Permutation shuffles for the Theil's U and
Freeman's theta p-values. Default: |
seed |
Optional integer seed for the bootstrap and permutation draws; the global RNG state is saved and restored. |
polyserial_ml |
Logical, passed to |
nearPD |
Logical, passed to |
Details
By default it uses assume_latent_normal = "auto", so each pair is
screened with the diagnostics described in smart_cor() and the method
follows them (a saturated 2 x 2 pair gets phi).
The two returned data frames share identical row and column names (the
variable names from data). Diagonal entries in correlations are 1;
diagonal entries in methods name the method a self-pair of that type
would get, matching smart_cormat().
Value
A named list with two elements:
correlationsA data frame of correlation estimates (numeric).
methodsA data frame of method codes (character) used for each pair.
Examples
csv = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(csv)
# Correlations and methods for four columns in the frozen GSS extract
result = smart_cor_df(
gss[, c("age", "coninc", "degree", "sex")],
assume_latent_normal = FALSE
)
result$correlations
result$methods
# Missing values are handled pairwise by default
example = gss[, c("age", "coninc", "sex")]
example$coninc[1:3] = NA
smart_cor_df(example, assume_latent_normal = FALSE)
Build a Mixed-Type Correlation Matrix
Description
Runs smart_cor() on every pair of columns and records both the estimate
and the method used for each pair.
Usage
smart_cormat(
data,
cols = NULL,
types = NULL,
assume_latent_normal = "auto",
ordinal_threshold = 10,
ignore_na = TRUE,
conf_level = 0.95,
alpha = 0.05,
bootstrap = "auto",
n_boot = 500,
n_perm = 500,
seed = NULL,
polyserial_ml = FALSE,
verbose = TRUE,
nearPD = FALSE
)
Arguments
data |
A data frame or tibble. All columns are used unless |
cols |
A character vector of column names to include. Default: all columns. |
types |
An optional named list mapping column names to types
( |
assume_latent_normal |
|
ordinal_threshold |
Integer passed to |
ignore_na |
Logical. If |
conf_level |
Numeric between 0 and 1. Confidence level passed to
each |
alpha |
Significance level for the diagnostic tests, passed to each
|
bootstrap |
Character or logical, passed to each |
n_boot |
Integer. Bootstrap replications per pair when
bootstrapping. Default: |
n_perm |
Integer. Permutation shuffles for the Theil's U and
Freeman's theta p-values. Default: |
seed |
Optional integer seed passed to each |
polyserial_ml |
Logical, passed to each |
verbose |
Logical. If |
nearPD |
Logical. If |
Details
For each pair of columns, smart_cor() detects variable types and
selects the appropriate method. The result is a symmetric matrix of
correlation/association measure estimates annotated with the method
used for each cell.
Since different methods are used for different pairs, the resulting
matrix may not be positive semi-definite. With nearPD = TRUE, the
completed matrix is adjusted using Matrix::nearPD(corr = TRUE).
The returned correlations are then positive definite with diagonal 1.
original_correlations retains the estimates before adjustment; methods,
details, and tidy.smartcormat() still describe those original estimates
and their inference. The adjustment does not refit the pairwise models.
A matrix containing missing or infinite entries cannot be adjusted.
Diagonal cells hold 1 by definition; the methods matrix records for each
diagonal cell the method that a self-pair of that type would get, so
every cell of the matrix is labelled the same way: "pearson" for
continuous and count variables, "phi", "tetrachoric", or "yules_q"
for binary variables (under assume_latent_normal = "auto", TRUE, or
FALSE), "polychoric" ("auto" or TRUE) or "kendall" (FALSE) for
ordinal variables, and "cramers_v" for categorical variables.
The stored details omit each pair's diagnostics list to keep the
object small; the summary fields outlier_flag, normality_p,
latent_p, and linearity_p remain, so tidy.smartcormat() reports
them.
Value
An object of class "smartcormat": a list with elements:
correlationsNumeric matrix of correlation estimates.
methodsCharacter matrix of method codes used for each pair.
typesNamed character vector of detected/specified variable types.
detailsList of
smart_cor()result objects (one per pair), without theirdiagnosticselement.original_correlationsMatrix before adjustment; present only when
nearPD = TRUE.nearPDAdjustment diagnostics (
normF,iterations,converged); present only whennearPD = TRUE.
Examples
path = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(path)
smart_cormat(
gss[, c("age", "coninc", "degree", "sex")],
assume_latent_normal = FALSE,
verbose = FALSE
)
# Override a detected type when the study design requires it.
smart_cormat(
gss[, c("age", "coninc", "degree")],
types = list(age = "continuous", degree = "ordinal"),
assume_latent_normal = FALSE,
verbose = FALSE
)
Test Bivariate Normality for a Pair of Variables
Description
Tests the normality assumption behind Pearson's inference and behind the latent-variable correlations (polychoric, polyserial, biserial). The test depends on the variable types:
- continuous + continuous
Mardia's multivariate skewness and kurtosis tests, via
MVN::mardia().- continuous + ordinal
Jarque-Bera on the continuous variable (
moments::jarque.test()), plus the LR chi-square threshold-model test after binning the continuous variable.- continuous + binary
Jarque-Bera on the residuals of the continuous variable around its group means, plus the LR chi-square test after binning (df = m1 - 2).
- ordinal or binary pairs
The LR chi-square test. A binary-binary table is saturated (df = 0) and cannot be tested.
Count variables are treated as continuous. Categorical variables are not supported.
Usage
test_bivariate_normality(
x,
y = NULL,
x_type = NULL,
y_type = NULL,
alpha = 0.05,
max_levels = 10L
)
Arguments
x, y |
Vectors of the same length. Alternatively, a pre-computed
contingency table (matrix or table object) can be passed as |
x_type, y_type |
Optional variable types ( |
alpha |
Significance level for the test. Default: |
max_levels |
Integer. Cap on the number of categories per variable
when building the contingency table from numeric vectors. A numeric
variable with more than |
Details
The LR chi-square test follows Joreskog (2005). The bivariate normal
threshold model is fitted with the two-step estimator (Olsson 1979) via
polycor::polychor(): the thresholds come from the marginal proportions
and the correlation is then estimated by one-dimensional maximum
likelihood given those thresholds. The likelihood-ratio statistic
G^2 = 2 \sum_{ij} O_{ij} \log(O_{ij} / E_{ij})
compares the observed counts O_{ij} with the expected counts
E_{ij} under the fitted model (cells with zero observed count
are skipped). The degrees of freedom equal
m_1 \times m_2 - m_1 - m_2, where m_1
and m_2 are the numbers of categories of the two variables. For a
2 \times 2 table the model is saturated (df = 0) and no test
is possible.
When several component tests are run, the overall test rejects if any
component rejects at alpha, and the overall p.value is the smallest
component p-value. The Python pysmartcor package computes the same
statistics, so the two implementations agree.
Value
An object of class "smartcor_normtest", a list with elements:
- statistic
The LR chi-square statistic.
- df
Degrees of freedom of the LR chi-square test.
- p.value
The overall p-value (smallest component p-value).
- rho
The two-step estimate of the polychoric correlation.
- row_thresholds
Estimated thresholds for the first variable.
- col_thresholds
Estimated thresholds for the second variable.
- n
Total number of observations.
- m1
Number of categories for the first variable.
- m2
Number of categories for the second variable.
- alpha
The significance level used.
- conclusion
One of
"reject","fail_to_reject", or"saturated".- test
Name of the test, e.g.
"Mardia","Jarque-Bera and LR chi-square", or"LR chi-square".- components
Data frame with one row per component test and columns
test,statistic,df, andp.value.- null_hypothesis
The null hypothesis in words.
- interpretation
One plain sentence stating the result.
statistic, df, rho, the thresholds, m1, and m2 describe the LR
chi-square component and are NA (or empty) when it was not run.
Examples
csv = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(csv)
# Check the latent-normality assumption for two ordinal variables
test_bivariate_normality(gss$degree, gss$happy)
# Mardia's test for two continuous variables
test_bivariate_normality(gss$age, gss$coninc, x_type = "continuous",
y_type = "continuous")
Test Linearity Using Ordered Regression Residuals
Description
Runs the sorted-residual diagnostics in both directions: y regressed on
x with the residuals ordered by x, and x regressed on y with the
residuals ordered by y. The relationship is treated as nonlinear only
when both directions reject, so the result does not depend on which
variable is passed first.
Usage
test_linearity(x, y, alpha = 0.05, ignore_na = TRUE)
Arguments
x, y |
Numeric vectors of the same length. |
alpha |
Significance level, strictly between 0 and 1. Default: |
ignore_na |
Remove incomplete pairs when |
Details
In each direction, incomplete pairs are removed, the regression is fitted,
and the residuals are ordered by the regressor. Residuals that share a
regressor value are averaged, so the order of tied rows does not matter.
Box-Pierce and Ljung-Box tests use lag 1. The BDS p-value is the smallest
of the p-values from fNonlinear::bdsTest(residuals, m = 3) across its
four default distance thresholds. Breusch-Pagan and RESET use the
defaults in the lmtest package.
A direction rejects linearity when at least one residual structure test
rejects and RESET rejects. Its p-value is max(min(p_bp, p_lb, p_bds), p_reset), the intersection-union p-value of that rule. Spearman is
selected only when both directions reject; the combined p.value is the
larger of the two directional p-values (p_y_on_x, p_x_on_y), so
p.value < alpha exactly when linearity is rejected. Otherwise Pearson is
selected, and the inference recommendation (standard or robust/bootstrap)
comes from the direction that does not reject; when both do not reject,
from the one with the larger p-value. The component p-values reported at
the top level belong to that direction (direction).
The regression model behind the test assumes errors independent of the regressor, which can hold in only one direction unless the pair is bivariate normal. Requiring both directions to reject keeps skewed margins from being read as nonlinearity.
linear = TRUE means this rule did not detect nonlinearity; it does not
establish linearity. Fewer than five distinct values, a constant variable,
an exact linear fit, or undefined diagnostics make a direction
inconclusive. If neither direction can be tested, or one rejects and the
other cannot be tested, the result is inconclusive (linear = NA) and
Pearson is retained. This function returns correlation estimates and an
inference recommendation; it does not compute confidence intervals or
correlation p-values.
Value
An object of class "smartcor_lintest": a list with p_bp,
p_lb, p_bds, p_het, p_reset (for the direction used),
p_y_on_x, p_x_on_y, direction, p.value (combined), alpha,
n_complete, structure_detected, homoskedastic, linear,
conclusion, decision, method, r_pearson, r_spearman,
r_kendall, r_final, reason, null_hypothesis, and
interpretation.
Examples
set.seed(42)
x = runif(100, 0, 2)
y = 2 + 3 * x + rnorm(100, sd = 0.5)
test_linearity(x, y)
Tidy a smartcor Object
Description
Lays a smart_cor() result out as a one-row tibble in the manner of the
broom package, so results drop straight into a dplyr pipeline.
Usage
## S3 method for class 'smartcor'
tidy(x, ...)
Arguments
x |
A |
... |
Ignored. |
Value
A one-row tibble::tibble() with columns covering identifiers
(var_x, var_y, x_type, y_type), the point estimate, the chosen
method (method, method_label, rationale), inference details
(statistic, p.value, p_method, null_hypothesis, p_interpretation,
ci_lower, ci_upper, conf_level, ci_method, ci_source), sample
sizes (n, n_complete), and diagnostic summaries (outlier_flag,
normality_p, latent_p, linearity_p; NA when not computed).
Examples
csv = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(csv)
res = smart_cor(gss$coninc, gss$age, verbose = FALSE)
tidy(res)
Tidy a smartcor Comparison
Description
Lays a compare_methods() result out as a tibble with one row per
candidate method, in the manner of the broom package.
Usage
## S3 method for class 'smartcor_comparison'
tidy(x, ...)
Arguments
x |
A |
... |
Ignored. |
Value
A tibble::tibble() with one row per method.
Tidy a smartcormat Object
Description
Turns a smart_cormat() result into a long tibble with one row per
variable pair (lower triangle only, no self-correlations).
If nearPD = TRUE was used, estimates and inference refer to the
original pairs, before the matrix adjustment.
Usage
## S3 method for class 'smartcormat'
tidy(x, ...)
Arguments
x |
A |
... |
Ignored. |
Value
A tibble::tibble() with the same columns as tidy.smartcor(),
one row per unique pair.
Examples
csv = system.file("extdata", "gss_2024_casestudy.csv", package = "smartcor")
gss = read.csv(csv)
mat = smart_cormat(
gss[, c("age", "coninc", "sex")],
assume_latent_normal = FALSE,
verbose = FALSE
)
tidy(mat)