Simulation based calibration for OncoBayes2

Tue Aug 4 19:33:43 2026

This report documents the results of a simulation based calibration (SBC) run for OncoBayes2. TODO

The calibration data presented here has been generated at and with the OncoBayes git version as:

## Created:  2026-08-04 09:41:25 UTC
## git hash: ab2f203cb91716f8b84c798cfd3f22aec3a75ad6
## MD5:      0ab6c545ab3bf1d299bbcf08facd3248

The MD5 hash of the calibration data file presented here must match the above listed MD5:

## /home/runner/work/OncoBayes2/OncoBayes2/inst/sbc/calibration.rds 
##                               "0ab6c545ab3bf1d299bbcf08facd3248"

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

  1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

  2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

  3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

  4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Model description TODO

The fake data simulation function returns … TODO. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Sampler Diagnostics Overview

data_scenario N total_divergent min_ess_bulk min_ess_tail max_Rhat total_large_Rhat min_lp_ess_bulk min_lp_ess_tail
combo2_EX 10000 1 268.345 80.350 1.019 0 393.622 535.898
combo2_EXNEX 10000 0 44.161 40.562 1.061 0 364.527 482.689
combo3_EXNEX 10000 0 63.058 31.794 1.037 0 388.331 452.627
log2bayes_EX 10000 0 950.458 747.960 1.023 0 308.346 538.114
log2bayes_EXNEX 10000 0 299.184 342.943 1.015 0 380.328 426.596

Large Rhat is defined as exceeding \(1.1\).

Sampler Adaptation & Performance Overview

ESS speed is in units of ESS per second.

\(\chi^2\) Statistic: Single-agent logistic regression, stratified

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 33.018 31 0.369
beta_group[A,I(log(drug_A/1)),log_slope] 44.422 31 0.056
beta_group[B,I(log(drug_A/1)),intercept] 31.251 31 0.454
beta_group[B,I(log(drug_A/1)),log_slope] 25.626 31 0.739
beta_group[C,I(log(drug_A/1)),intercept] 20.557 31 0.923
beta_group[C,I(log(drug_A/1)),log_slope] 42.534 31 0.081
beta_group[D,I(log(drug_A/1)),intercept] 27.635 31 0.640
beta_group[D,I(log(drug_A/1)),log_slope] 32.384 31 0.398
beta_group[E,I(log(drug_A/1)),intercept] 43.846 31 0.063
beta_group[E,I(log(drug_A/1)),log_slope] 29.952 31 0.520
beta_group[F,I(log(drug_A/1)),intercept] 25.235 31 0.757
beta_group[F,I(log(drug_A/1)),log_slope] 29.120 31 0.563
mu_log_beta[I(log(drug_A/1)),intercept] 29.222 31 0.558
mu_log_beta[I(log(drug_A/1)),log_slope] 39.622 31 0.138
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] 40.909 31 0.110
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] 29.530 31 0.542
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_intercept] 36.083 31 0.243
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_log_slope] 40.736 31 0.113

\(\chi^2\) Statistic: Single-agent logistic regression, EXchangeable/NonEXchangeable

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 35.078 31 0.281
beta_group[A,I(log(drug_A/1)),log_slope] 26.803 31 0.682
beta_group[B,I(log(drug_A/1)),intercept] 33.299 31 0.356
beta_group[B,I(log(drug_A/1)),log_slope] 31.046 31 0.464
beta_group[C,I(log(drug_A/1)),intercept] 29.485 31 0.544
beta_group[C,I(log(drug_A/1)),log_slope] 20.429 31 0.926
mu_log_beta[I(log(drug_A/1)),intercept] 27.219 31 0.661
mu_log_beta[I(log(drug_A/1)),log_slope] 27.706 31 0.636
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] 39.296 31 0.146
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] 43.622 31 0.066

\(\chi^2\) Statistic: Double combination, fully exchangeable, stratified

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 32.890 31 0.375
beta_group[A,I(log(drug_A/1)),log_slope] 27.872 31 0.628
beta_group[A,I(log(drug_B/2)),intercept] 30.810 31 0.476
beta_group[A,I(log(drug_B/2)),log_slope] 34.816 31 0.291
beta_group[B,I(log(drug_A/1)),intercept] 17.837 31 0.972
beta_group[B,I(log(drug_A/1)),log_slope] 31.616 31 0.436
beta_group[B,I(log(drug_B/2)),intercept] 44.909 31 0.051
beta_group[B,I(log(drug_B/2)),log_slope] 29.555 31 0.540
beta_group[C,I(log(drug_A/1)),intercept] 31.130 31 0.460
beta_group[C,I(log(drug_A/1)),log_slope] 26.586 31 0.693
beta_group[C,I(log(drug_B/2)),intercept] 44.755 31 0.052
beta_group[C,I(log(drug_B/2)),log_slope] 31.776 31 0.428
beta_group[D,I(log(drug_A/1)),intercept] 23.821 31 0.818
beta_group[D,I(log(drug_A/1)),log_slope] 30.893 31 0.472
beta_group[D,I(log(drug_B/2)),intercept] 38.554 31 0.165
beta_group[D,I(log(drug_B/2)),log_slope] 29.837 31 0.526
eta_group[A,I(drug_A/1 * drug_B/2)] 46.579 31 0.036
eta_group[B,I(drug_A/1 * drug_B/2)] 24.544 31 0.788
eta_group[C,I(drug_A/1 * drug_B/2)] 24.237 31 0.801
eta_group[D,I(drug_A/1 * drug_B/2)] 26.893 31 0.678
mu_eta[m[1]] 32.838 31 0.377
mu_log_beta[I(log(drug_A/1)),intercept] 23.277 31 0.839
mu_log_beta[I(log(drug_A/1)),log_slope] 16.723 31 0.983
mu_log_beta[I(log(drug_B/2)),intercept] 44.723 31 0.053
mu_log_beta[I(log(drug_B/2)),log_slope] 33.670 31 0.339
tau_eta[stratum_1,m[1]] 37.293 31 0.202
tau_eta[stratum_2,m[1]] 24.384 31 0.795
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] 32.480 31 0.394
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] 25.754 31 0.733
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_intercept] 21.933 31 0.885
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_log_slope] 34.157 31 0.318
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_intercept] 40.947 31 0.109
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_log_slope] 46.253 31 0.038
tau_log_beta[stratum_2,I(log(drug_B/2)),tau_intercept] 28.442 31 0.598
tau_log_beta[stratum_2,I(log(drug_B/2)),tau_log_slope] 46.573 31 0.036

\(\chi^2\) Statistic: Double combination, EXchangeable/NonEXchangeable

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 27.443 31 0.650
beta_group[A,I(log(drug_A/1)),log_slope] 20.768 31 0.918
beta_group[A,I(log(drug_B/2)),intercept] 44.250 31 0.058
beta_group[A,I(log(drug_B/2)),log_slope] 35.834 31 0.252
beta_group[B,I(log(drug_A/1)),intercept] 28.288 31 0.606
beta_group[B,I(log(drug_A/1)),log_slope] 24.058 31 0.808
beta_group[B,I(log(drug_B/2)),intercept] 37.997 31 0.181
beta_group[B,I(log(drug_B/2)),log_slope] 23.930 31 0.813
beta_group[C,I(log(drug_A/1)),intercept] 38.010 31 0.180
beta_group[C,I(log(drug_A/1)),log_slope] 32.358 31 0.400
beta_group[C,I(log(drug_B/2)),intercept] 41.901 31 0.092
beta_group[C,I(log(drug_B/2)),log_slope] 21.728 31 0.891
eta_group[A,I(drug_A/1 * drug_B/2)] 34.234 31 0.315
eta_group[B,I(drug_A/1 * drug_B/2)] 24.582 31 0.786
eta_group[C,I(drug_A/1 * drug_B/2)] 42.061 31 0.089
mu_eta[m[1]] 22.253 31 0.875
mu_log_beta[I(log(drug_A/1)),intercept] 37.248 31 0.204
mu_log_beta[I(log(drug_A/1)),log_slope] 37.158 31 0.206
mu_log_beta[I(log(drug_B/2)),intercept] 33.466 31 0.348
mu_log_beta[I(log(drug_B/2)),log_slope] 23.059 31 0.847
tau_eta[stratum_1,m[1]] 24.973 31 0.769
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] 33.581 31 0.343
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] 36.826 31 0.217
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_intercept] 22.912 31 0.852
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_log_slope] 22.214 31 0.876

\(\chi^2\) Statistic: Triple combination, EXchangeable/NonEXchangeable

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 40.410 31 0.120
beta_group[A,I(log(drug_A/1)),log_slope] 33.523 31 0.346
beta_group[A,I(log(drug_B/2)),intercept] 29.894 31 0.523
beta_group[A,I(log(drug_B/2)),log_slope] 24.954 31 0.770
beta_group[A,I(log(drug_C/4)),intercept] 48.832 31 0.022
beta_group[A,I(log(drug_C/4)),log_slope] 31.334 31 0.449
beta_group[B,I(log(drug_A/1)),intercept] 38.394 31 0.169
beta_group[B,I(log(drug_A/1)),log_slope] 36.205 31 0.239
beta_group[B,I(log(drug_B/2)),intercept] 38.438 31 0.168
beta_group[B,I(log(drug_B/2)),log_slope] 24.096 31 0.807
beta_group[B,I(log(drug_C/4)),intercept] 17.882 31 0.971
beta_group[B,I(log(drug_C/4)),log_slope] 24.602 31 0.785
beta_group[C,I(log(drug_A/1)),intercept] 33.050 31 0.367
beta_group[C,I(log(drug_A/1)),log_slope] 25.088 31 0.764
beta_group[C,I(log(drug_B/2)),intercept] 31.910 31 0.421
beta_group[C,I(log(drug_B/2)),log_slope] 30.970 31 0.468
beta_group[C,I(log(drug_C/4)),intercept] 49.069 31 0.021
beta_group[C,I(log(drug_C/4)),log_slope] 26.778 31 0.683
eta_group[A,I(drug_A/1 * drug_B/2 * drug_C/4)] 24.230 31 0.801
eta_group[A,I(drug_A/1 * drug_B/2)] 23.584 31 0.827
eta_group[A,I(drug_A/1 * drug_C/4)] 32.090 31 0.412
eta_group[A,I(drug_B/2 * drug_C/4)] 23.123 31 0.845
eta_group[B,I(drug_A/1 * drug_B/2 * drug_C/4)] 39.002 31 0.153
eta_group[B,I(drug_A/1 * drug_B/2)] 38.208 31 0.175
eta_group[B,I(drug_A/1 * drug_C/4)] 45.875 31 0.042
eta_group[B,I(drug_B/2 * drug_C/4)] 50.413 31 0.015
eta_group[C,I(drug_A/1 * drug_B/2 * drug_C/4)] 25.146 31 0.761
eta_group[C,I(drug_A/1 * drug_B/2)] 27.046 31 0.670
eta_group[C,I(drug_A/1 * drug_C/4)] 21.939 31 0.885
eta_group[C,I(drug_B/2 * drug_C/4)] 25.030 31 0.766
mu_eta[m[1]] 40.768 31 0.113
mu_eta[m[2]] 32.262 31 0.404
mu_eta[m[3]] 29.632 31 0.536
mu_eta[m[4]] 35.251 31 0.274
mu_log_beta[I(log(drug_A/1)),intercept] 26.086 31 0.717
mu_log_beta[I(log(drug_A/1)),log_slope] 39.686 31 0.136
mu_log_beta[I(log(drug_B/2)),intercept] 26.368 31 0.704
mu_log_beta[I(log(drug_B/2)),log_slope] 25.587 31 0.741
mu_log_beta[I(log(drug_C/4)),intercept] 23.795 31 0.819
mu_log_beta[I(log(drug_C/4)),log_slope] 32.160 31 0.409
tau_eta[stratum_1,m[1]] 34.886 31 0.288
tau_eta[stratum_1,m[2]] 29.830 31 0.526
tau_eta[stratum_1,m[3]] 38.227 31 0.174
tau_eta[stratum_1,m[4]] 18.739 31 0.959
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] 22.592 31 0.864
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] 37.050 31 0.210
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_intercept] 39.315 31 0.145
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_log_slope] 39.533 31 0.140
tau_log_beta[stratum_1,I(log(drug_C/4)),tau_intercept] 26.605 31 0.692
tau_log_beta[stratum_1,I(log(drug_C/4)),tau_log_slope] 29.888 31 0.523

Session Info

## R version 4.6.1 (2026-06-24)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.4 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## time zone: UTC
## tzcode source: system (glibc)
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
## [1] ggplot2_4.0.3    broom_1.0.13     tidyr_1.3.2      dplyr_1.2.1     
## [5] assertthat_0.2.1 knitr_1.51       here_1.0.2      
## 
## loaded via a namespace (and not attached):
##  [1] gtable_0.3.6       jsonlite_2.0.0     compiler_4.6.1     tidyselect_1.2.1  
##  [5] jquerylib_0.1.4    scales_1.4.0       yaml_2.3.12        fastmap_1.2.0     
##  [9] R6_2.6.1           generics_0.1.4     backports_1.5.1    tibble_3.3.1      
## [13] rprojroot_2.1.1    RColorBrewer_1.1-3 bslib_0.12.0       pillar_1.11.1     
## [17] rlang_1.3.0        cachem_1.1.0       xfun_0.60          S7_0.2.2          
## [21] sass_0.4.10        otel_0.2.0         cli_3.6.6          withr_3.0.3       
## [25] magrittr_2.0.5     digest_0.6.39      grid_4.6.1         lifecycle_1.0.5   
## [29] vctrs_0.7.3        evaluate_1.0.5     glue_1.8.1         farver_2.1.2      
## [33] rmarkdown_2.31     purrr_1.2.2        pkgconfig_2.0.3    htmltools_0.5.9