--- title: "Step intensities over arbitrary intervals" author: "TA Trikalinos" date: "2026-08-21" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Step intensities over arbitrary intervals} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} set.seed(20260821) knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) K <- 10^5 ``` ## What this vignette shows The vectorized samplers for piecewise constant intensity functions, `vdraw_sc_step_regular()` and its zero-truncated counterpart, assume that the rate matrix columns correspond to *equal-length* time intervals. `vdraw_sc_step()` and `vztdraw_sc_step()` drop this assumption: the interval bounds are passed in `time_breaks`, either once for all point processes or separately for each one. Sampling remains exact -- the inversion algorithm does not care whether the intervals are equally spaced. This matters in practice because piecewise constant rates are rarely published on equal-length intervals. The standard demographic and epidemiologic convention uses age bands $[0, 1), [1, 5)$, five-year bands up to age $85$, and a terminal open-ended band. With `vdraw_sc_step_regular()` one has to expand such rates to a fine regular grid (feasible here, with integer bounds, but wasteful; impossible exactly when the bounds are not commensurate). With `vdraw_sc_step()` the published bands are used as-is. We showcase the new functions and then do a small timing study against the regular-grid sampler. ## Simulation description Assume a population of $K = `r K`$ individuals indexed by $k \in [K] := \{1, \dots, K\}$. Each is subject to an age-specific disease incidence $\lambda_k(t) = w_k \, \lambda(t)$, where $\lambda(t)$ is a step function over the $M = 19$ standard age bands and $w_k$ is a person-specific multiplier (frailty) with $w_k \sim \textrm{Gamma}(5, 5)$, so that $E[w_k] = 1$. ```{r setup} library(nhppp) age_breaks <- c(0, 1, seq(5, 85, by = 5), 100) # M + 1 = 20 bounds lambda_bands <- c( 0.5, 0.3, 0.2, 0.3, 0.5, 0.8, 1.2, 2, 3.5, 6, 11, 20, 35, 60, 95, 140, 190, 235, 270 ) / 10^5 # rates per person-year, one per band w <- stats::rgamma(K, shape = 5, rate = 5) lambda_matrix <- outer(w, lambda_bands) # K x M ``` The expected number of events over a full life course is $w_k \int_0^{100} \lambda(t) \, dt = w_k \times `r round(sum(lambda_bands * diff(age_breaks)), 4)`$, so most people have no event and few have more than one. ## Sampling with shared interval bounds When all point processes share the same interval bounds, `time_breaks` is a vector with `ncol(lambda_matrix) + 1` increasing values. Each row of the returned matrix holds the (ordered) event times of one point process; rows with fewer events than the widest row are padded with `NA`. ```{r whole-range} Z <- vdraw_sc_step( lambda_matrix = lambda_matrix, time_breaks = age_breaks ) dim(Z) mean(!is.na(Z[, 1])) # proportion of people with at least one event ``` As with the other `vdraw_*()` functions, `Lambda_matrix` (cumulative intensities at the interval ends) can be given instead of `lambda_matrix`, and the truncation options work the same way: `report_first_K` / `report_last_K` report only the earliest or latest K events of a realization; `generate_at_least_K` / `generate_at_most_K` condition the sampled process on its number of events; and `budget_cap` caps the computational event budget. ### The sampled times follow the target intensity Pooled over people, the expected density of event ages is proportional to $\sum_k w_k \lambda(t) \propto \lambda(t)$ -- the frailties scale the counts, not the shape. The histogram of all simulated event ages should match $\lambda(t) / \int_0^{100}\lambda(u)\,du$, a step function. It does; there is no approximation at play. ```{r histogram, fig.alt="Histogram of simulated event ages overlaid with the normalized step intensity function. The two agree."} if (nchar(system.file(package = "ggplot2")) > 0) { library(ggplot2) t_events <- as.vector(Z) t_events <- t_events[!is.na(t_events)] dens_bands <- lambda_bands / sum(lambda_bands * diff(age_breaks)) step_df <- data.frame( age = age_breaks, density = c(dens_bands, dens_bands[length(dens_bands)]) ) ggplot() + geom_histogram( aes(x = t_events, y = after_stat(density)), breaks = 0:100, fill = "gray", color = "white" ) + geom_step( data = step_df, aes(x = age, y = density), direction = "hv", color = "red" ) + labs( x = "Age (years)", y = "Empirical and theoretical density", title = "All simulated event ages" ) } ``` ## Per-process interval bounds `time_breaks` can also be a matrix with one row of bounds per point process. The rows of `lambda_matrix` still refer to the same bands -- but each band is located at different times for different people. A natural example: simulate in *calendar time* rather than age. If person $k$ is born in year $B_k$, their age bands correspond to the calendar intervals $B_k + [0, 1), B_k + [1, 5), \dots$ We pass those directly. ```{r per-row-breaks} birth_year <- stats::runif(K, min = 1920, max = 2000) calendar_breaks <- outer(birth_year, age_breaks, FUN = "+") # K x (M + 1) Z_cal <- vdraw_sc_step( lambda_matrix = lambda_matrix, time_breaks = calendar_breaks ) range(Z_cal, na.rm = TRUE) # calendar years, not ages ``` Subtracting each person's birth year recovers event ages with the same distribution as before (different draws, same process): ```{r qq-per-row, fig.alt="QQ plot comparing event ages sampled on the age scale and on the calendar-time scale. The agreement is excellent."} age_at_event <- as.vector(Z_cal - birth_year) # birth_year recycles down columns qqplot( as.vector(Z), age_at_event, xlab = "Event ages, shared breaks", ylab = "Event ages, per-person calendar breaks" ) abline(0, 1, col = "red") ``` ## Subintervals and first events The rate information often spans a wider interval than the one to be sampled. With `t_min` and `t_max` (scalars or vectors), events are drawn only in `(t_min, t_max]`, without rebuilding the rate matrix. Here we draw each person's *first* event between study entry at age 40 and a person-specific exit age $T_{k1} \sim U(60, 100)$: ```{r subinterval} T1 <- stats::runif(K, min = 60, max = 100) t_first <- vdraw_sc_step( lambda_matrix = lambda_matrix, time_breaks = age_breaks, t_min = 40, t_max = T1, report_first_K = 1 ) summary(as.vector(t_first)) ``` ## Zero-truncated sampling `vztdraw_sc_step()` -- equivalently `vdraw_sc_step(generate_at_least_K = 1)` -- conditions on at least one event in the sampled interval. This is the natural tool when simulating people who are known to become cases, e.g., drawing an age at diagnosis for each of $K$ cases: ```{r zero-truncated} t_dx <- vztdraw_sc_step( lambda_matrix = lambda_matrix, time_breaks = age_breaks, t_min = 40, t_max = T1, report_first_K = 1, generate_at_least_K = 1 ) sum(is.na(t_dx)) # everyone has an event now ``` ## Timing study How much does the extra generality cost? On a regular grid the sampler maps a time to its interval with one division; on arbitrary bounds it needs a binary search over the breaks. And the band representation is more compact: $19$ columns instead of the $100$ one-year columns that `vdraw_sc_step_regular()` would need to represent the same step function exactly. (A one-year grid works here because the band bounds are integers. With non-commensurate bounds no regular grid is exact.) We compare four exact ways to sample the same process, on $K_2 = 10^4$ point processes: 1. `vdraw_sc_step()`, the $19$ bands, shared `time_breaks` vector; 2. `vdraw_sc_step()`, the $19$ bands, per-row `time_breaks` matrix (identical rows, to isolate the cost of per-row bounds); 3. `vdraw_sc_step()`, the same step function expanded to a one-year grid ($100$ columns), to isolate the cost of the binary search at equal width; 4. `vdraw_sc_step_regular()` on the one-year grid -- the only exact option before `vdraw_sc_step()` existed. First check that methods 3 and 4 are not just distributionally equal: on a regular grid the general kernel consumes the random-number stream exactly like the regular kernel, so the same seed gives the same draws (up to floating-point roundoff in the internal construction of the cumulative intensities). ```{r same-seed} K2 <- 10^4 lam_bands_2 <- lambda_matrix[1:K2, ] lam_grid_2 <- lam_bands_2[, rep(seq_along(lambda_bands), times = diff(age_breaks))] dim(lam_grid_2) set.seed(2026) Z_general <- vdraw_sc_step( lambda_matrix = lam_grid_2, time_breaks = 0:100 ) set.seed(2026) Z_regular <- vdraw_sc_step_regular( lambda_matrix = lam_grid_2, rate_matrix_t_min = 0, rate_matrix_t_max = 100 ) all.equal(Z_general, Z_regular, tolerance = 10^-12) ``` Now the timings: median elapsed seconds over `R = 10` replications. (The absolute numbers depend on the machine that rendered this document; the relative comparisons are what matter.) ```{r timing} median_elapsed <- function(f, R = 10) { stats::median( vapply(seq_len(R), function(i) system.time(f())[["elapsed"]], numeric(1)) ) } breaks_rowwise <- matrix(rep(age_breaks, each = K2), nrow = K2) timing <- data.frame( method = c( "vdraw_sc_step, bands, shared breaks", "vdraw_sc_step, bands, per-row breaks", "vdraw_sc_step, one-year grid", "vdraw_sc_step_regular, one-year grid" ), columns = c(19, 19, 100, 100), seconds = c( median_elapsed(function() { vdraw_sc_step(lambda_matrix = lam_bands_2, time_breaks = age_breaks) }), median_elapsed(function() { vdraw_sc_step(lambda_matrix = lam_bands_2, time_breaks = breaks_rowwise) }), median_elapsed(function() { vdraw_sc_step(lambda_matrix = lam_grid_2, time_breaks = 0:100) }), median_elapsed(function() { vdraw_sc_step_regular( lambda_matrix = lam_grid_2, rate_matrix_t_min = 0, rate_matrix_t_max = 100 ) }) ) ) knitr::kable(timing, digits = 4) ``` Three observations: * The dominant cost is the width of the rate matrix: both samplers spend most of their time constructing cumulative intensities column by column. Sampling the $19$ bands directly is a few times faster than either sampler on the expanded $100$-column grid -- and the rate matrix is about a fifth of the size. * The generality is not free. At equal width (methods 3 vs 4), the general kernel pays for the binary search over the breaks (instead of a division) and for reconstructing event times from the breaks; on our machines it takes roughly $1.5$ times as long as the regular kernel. When the grid really is regular, `vdraw_sc_step_regular()` remains the faster choice. * Per-row `time_breaks` (method 2) cost little more than a shared vector. Note that a shared vector (or, equivalently, a 1-row matrix) is *not* replicated internally: the C++ kernel reads the single row of bounds for every point process, so there is no hidden $K \times (M+1)$ allocation. ## Bibliography Trikalinos TA, Sereda Y. _nhppp: Simulating Nonhomogeneous Poisson Point Processes in R_. arXiv preprint arXiv:2402.00358. 2024 Feb 1. Trikalinos TA, Sereda Y. _The nhppp package for simulating non-homogeneous Poisson point processes in R_. PLoS One. 2024 Nov 21;19(11):e0311311. Since the publication of the paper, the syntax and options of the `nhppp` package have evolved. To reproduce the code in the paper, you have to install the version of `nhppp` used in the paper. Alternatively, take a look at the vignettes, which are written to work with the current package.