This report documents the results of a simulation based calibration (SBC) run for RBesT. The calibration data will be generated whenever relevant changes to the gMAP function were made. The calibration runs are performed for typical use cases of gMAP. These include the three likelihoods (binomial, gaussian & Poisson), a sparse (\(2\) trials) and dense (\(10\) trials) data situation and finally a run with a very/less conservative prior choice for between-trial heterogeniety parameter.

The calibration data presented here has been generated at and with the RBesT git version as:

The MD5 hash of the calibration data file presented here must match the above listed MD5:

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Hierarchical intercept only (random-effects intercept) model for binomial, gaussian and Poisson likelihood

Likelihood:

Binary \[y_i|\theta_{j} \sim \mbox{Bernoulli}(\theta_j), \] \[g(\theta) = \mbox{logit}(\theta)\]
Normal \[y_i|\theta_{j} \sim \mbox{Normal}(\theta_j, \sigma^2), \] \[g(\theta) = \theta\]
Poisson \[y_i|\theta_{j} \sim \mbox{Poisson}(\theta_j), \] \[g(\theta) = \log(\theta)\]

Hierarchical prior:

\[ g(\theta_j)|\mu,\tau \sim \mbox{Normal}(\mu, \tau^2)\]

\[\mu \sim \mbox{Normal}(m_\mu, s^2_\mu)\] \[\tau \sim \mbox{Normal}^+(0, s^2_\tau)\]

The fake data simulation function returns for binomial and Poisson data the sum of the responses while for normal the mean summary is used. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Sampler Diagnostics Overview

family	data_scenario	sd_tau	N	total_divergent	total_divergent_sim_fraction	min_ess	max_Rhat	total_large_Rhat	min_lp_ess_bulk	min_lp_ess_tail
binomial	dense	0.5	10000	330	0.025	213	1.012	0	517.597	512.380
binomial	dense	1.0	10000	434	0.029	128	1.021	0	558.833	545.517
binomial	sparse	0.5	10000	1059	0.060	61	1.046	0	144.899	59.269
binomial	sparse	1.0	10000	3402	0.114	5	1.367	1	42.591	55.864
gaussian	dense	0.5	10000	445	0.030	167	1.019	0	463.103	289.523
gaussian	dense	1.0	10000	426	0.024	83	1.027	0	354.980	467.374
gaussian	sparse	0.5	10000	20851	0.459	5	1.367	1	34.611	44.792
gaussian	sparse	1.0	10000	37103	0.402	4	1.669	1	8.742	40.653
poisson	dense	0.5	10000	843	0.060	184	1.026	0	421.486	394.058
poisson	dense	1.0	10000	2174	0.112	14	1.136	0	23.628	24.697
poisson	sparse	0.5	10000	21120	0.389	5	1.431	1	8.719	62.147
poisson	sparse	1.0	10000	37694	0.406	3	2.255	1	4.745	2.309

Note: Large Rhat is defined as exceeding 1.2.

Summary Statistics

\(\chi^2\) Statistic, \(\mu\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
1	dense	binomial	0.5	mu	65.587	63	0.387
13	dense	binomial	1	mu	45.734	63	0.950
25	dense	gaussian	0.5	mu	61.978	63	0.513
37	dense	gaussian	1	mu	73.920	63	0.163
49	dense	poisson	0.5	mu	43.904	63	0.968
61	dense	poisson	1	mu	68.506	63	0.296
73	sparse	binomial	0.5	mu	53.005	63	0.811
78	sparse	binomial	1	mu	62.016	63	0.511
83	sparse	gaussian	0.5	mu	56.922	63	0.691
88	sparse	gaussian	1	mu	50.099	63	0.881
93	sparse	poisson	0.5	mu	81.165	63	0.061
98	sparse	poisson	1	mu	73.165	63	0.179

\(\chi^2\) Statistic, \(\tau\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
2	dense	binomial	0.5	tau	74.035	63	0.161
14	dense	binomial	1	tau	56.730	63	0.698
26	dense	gaussian	0.5	tau	54.874	63	0.757
38	dense	gaussian	1	tau	72.883	63	0.185
50	dense	poisson	0.5	tau	74.982	63	0.143
62	dense	poisson	1	tau	76.045	63	0.125
74	sparse	binomial	0.5	tau	71.360	63	0.220
79	sparse	binomial	1	tau	65.190	63	0.400
84	sparse	gaussian	0.5	tau	62.950	63	0.478
89	sparse	gaussian	1	tau	48.166	63	0.916
94	sparse	poisson	0.5	tau	60.301	63	0.573
99	sparse	poisson	1	tau	60.275	63	0.574

\(\chi^2\) Statistic, group estimates \(\theta\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
3	dense	binomial	0.5	theta[10]	51.955	63	0.838
4	dense	binomial	0.5	theta[1]	77.453	63	0.104
5	dense	binomial	0.5	theta[2]	49.446	63	0.894
6	dense	binomial	0.5	theta[3]	65.485	63	0.391
7	dense	binomial	0.5	theta[4]	59.405	63	0.605
8	dense	binomial	0.5	theta[5]	47.693	63	0.924
9	dense	binomial	0.5	theta[6]	63.923	63	0.444
10	dense	binomial	0.5	theta[7]	55.642	63	0.733
11	dense	binomial	0.5	theta[8]	56.627	63	0.701
12	dense	binomial	0.5	theta[9]	70.515	63	0.241
15	dense	binomial	1	theta[10]	69.018	63	0.281
16	dense	binomial	1	theta[1]	72.666	63	0.190
17	dense	binomial	1	theta[2]	50.675	63	0.868
18	dense	binomial	1	theta[3]	45.274	63	0.955
19	dense	binomial	1	theta[4]	62.029	63	0.511
20	dense	binomial	1	theta[5]	92.390	63	0.009
21	dense	binomial	1	theta[6]	70.592	63	0.239
22	dense	binomial	1	theta[7]	51.584	63	0.847
23	dense	binomial	1	theta[8]	52.058	63	0.836
24	dense	binomial	1	theta[9]	48.589	63	0.909
27	dense	gaussian	0.5	theta[10]	59.110	63	0.616
28	dense	gaussian	0.5	theta[1]	80.435	63	0.068
29	dense	gaussian	0.5	theta[2]	67.046	63	0.340
30	dense	gaussian	0.5	theta[3]	86.131	63	0.028
31	dense	gaussian	0.5	theta[4]	65.344	63	0.395
32	dense	gaussian	0.5	theta[5]	60.582	63	0.563
33	dense	gaussian	0.5	theta[6]	65.869	63	0.378
34	dense	gaussian	0.5	theta[7]	84.314	63	0.038
35	dense	gaussian	0.5	theta[8]	53.632	63	0.794
36	dense	gaussian	0.5	theta[9]	44.442	63	0.963
39	dense	gaussian	1	theta[10]	100.390	63	0.002
40	dense	gaussian	1	theta[1]	56.755	63	0.697
41	dense	gaussian	1	theta[2]	48.870	63	0.904
42	dense	gaussian	1	theta[3]	48.013	63	0.919
43	dense	gaussian	1	theta[4]	71.488	63	0.217
44	dense	gaussian	1	theta[5]	58.086	63	0.652
45	dense	gaussian	1	theta[6]	48.410	63	0.912
46	dense	gaussian	1	theta[7]	51.008	63	0.861
47	dense	gaussian	1	theta[8]	69.018	63	0.281
48	dense	gaussian	1	theta[9]	66.035	63	0.372
51	dense	poisson	0.5	theta[10]	43.085	63	0.974
52	dense	poisson	0.5	theta[1]	84.390	63	0.037
53	dense	poisson	0.5	theta[2]	56.256	63	0.714
54	dense	poisson	0.5	theta[3]	75.763	63	0.130
55	dense	poisson	0.5	theta[4]	65.779	63	0.381
56	dense	poisson	0.5	theta[5]	63.040	63	0.475
57	dense	poisson	0.5	theta[6]	73.677	63	0.168
58	dense	poisson	0.5	theta[7]	55.782	63	0.729
59	dense	poisson	0.5	theta[8]	56.538	63	0.704
60	dense	poisson	0.5	theta[9]	56.512	63	0.705
63	dense	poisson	1	theta[10]	66.176	63	0.368
64	dense	poisson	1	theta[1]	49.920	63	0.884
65	dense	poisson	1	theta[2]	54.477	63	0.769
66	dense	poisson	1	theta[3]	63.386	63	0.463
67	dense	poisson	1	theta[4]	62.886	63	0.480
68	dense	poisson	1	theta[5]	87.181	63	0.024
69	dense	poisson	1	theta[6]	100.493	63	0.002
70	dense	poisson	1	theta[7]	56.768	63	0.697
71	dense	poisson	1	theta[8]	55.923	63	0.724
72	dense	poisson	1	theta[9]	71.578	63	0.215
75	sparse	binomial	0.5	theta[1]	61.043	63	0.546
76	sparse	binomial	0.5	theta[2]	48.179	63	0.916
77	sparse	binomial	0.5	theta[3]	55.731	63	0.730
80	sparse	binomial	1	theta[1]	64.576	63	0.421
81	sparse	binomial	1	theta[2]	60.518	63	0.565
82	sparse	binomial	1	theta[3]	67.264	63	0.333
85	sparse	gaussian	0.5	theta[1]	61.133	63	0.543
86	sparse	gaussian	0.5	theta[2]	44.954	63	0.958
87	sparse	gaussian	0.5	theta[3]	57.587	63	0.669
90	sparse	gaussian	1	theta[1]	68.134	63	0.307
91	sparse	gaussian	1	theta[2]	79.706	63	0.076
92	sparse	gaussian	1	theta[3]	64.678	63	0.418
95	sparse	poisson	0.5	theta[1]	63.654	63	0.453
96	sparse	poisson	0.5	theta[2]	73.997	63	0.162
97	sparse	poisson	0.5	theta[3]	76.762	63	0.114
100	sparse	poisson	1	theta[1]	71.680	63	0.212
101	sparse	poisson	1	theta[2]	58.099	63	0.651
102	sparse	poisson	1	theta[3]	59.277	63	0.610

Session Info

## R version 4.6.1 (2026-06-24)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.4 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## time zone: UTC
## tzcode source: system (glibc)
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] purrr_1.2.2         rstan_2.32.7        StanHeaders_2.32.10
##  [4] here_1.0.2          ggplot2_4.0.3       broom_1.0.13       
##  [7] tidyr_1.3.2         dplyr_1.2.1         assertthat_0.2.1   
## [10] knitr_1.51         
## 
## loaded via a namespace (and not attached):
##  [1] sass_0.4.10        generics_0.1.4     digest_0.6.39      magrittr_2.0.5    
##  [5] evaluate_1.0.5     grid_4.6.1         RColorBrewer_1.1-3 fastmap_1.2.0     
##  [9] rprojroot_2.1.1    jsonlite_2.0.0     pkgbuild_1.4.8     backports_1.5.1   
## [13] gridExtra_2.3.1    QuickJSR_1.10.0    scales_1.4.0       codetools_0.2-20  
## [17] jquerylib_0.1.4    cli_3.6.6          rlang_1.3.0        withr_3.0.3       
## [21] cachem_1.1.0       yaml_2.3.12        otel_0.2.0         parallel_4.6.1    
## [25] inline_0.3.21      vctrs_0.7.3        R6_2.6.1           matrixStats_1.5.0 
## [29] stats4_4.6.1       lifecycle_1.0.5    pkgconfig_2.0.3    RcppParallel_6.2.0
## [33] pillar_1.11.1      bslib_0.12.0       gtable_0.3.6       loo_2.10.1        
## [37] glue_1.8.1         Rcpp_1.1.2         xfun_0.60          tibble_3.3.1      
## [41] tidyselect_1.2.1   farver_2.1.2       htmltools_0.5.9    rmarkdown_2.31    
## [45] compiler_4.6.1     S7_0.2.2

Simulation based calibration for RBesT

Sebastian Weber

Tue Aug 4 12:38:36 2026