Wrap a user-supplied package resampling object with a design specification

design_resample() validates a user-supplied cv_resamples, nested_cv_resamples, or bootstrap_resamples object against a design spec and returns a design-aware resampling object. It computes the retained row universe from a design_spec() object and checks that the supplied split indices are compatible with those rows. It does not create new splits, fit a statistical model, or build a full-data design matrix.

Use design_fold() to materialize one validated split. Materialized CV and bootstrap splits fit their design schema on their analysis rows only and then encode assessment or out-of-bag rows with that analysis schema. Materialized nested-CV outer splits also return an inner design-aware CV object whose inner folds are materialized separately.

Usage

design_resample(
  spec,
  data,
  resamples,
  response_type = c("response", "ordinal_counts")
)

Arguments

spec: (design_spec)
Unfitted design specification returned by design_spec().
data: (data.frame)
Analysis data referenced by resamples. Objects that inherit from data.frame, including tibbles, are accepted.
resamples: (resamples)
A user-supplied cv_resamples, nested_cv_resamples, or bootstrap_resamples object. Its split indices must refer to rows of data.
response_type: (Scalar character: "response")
Response output mode used by design_fold(). Must be one of "response" or "ordinal_counts".

Value

A list of class c("design_<method>_resamples", "design_resamples") with elements:

spec: the original design_spec() object.
data: the original analysis data.
row_index: retained row identities used to validate split indices.
response_type: response representation used by materialized splits.
response_levels: declared response levels for ordinal-count output, or NULL.
resamples: the index-only resampling object.
splits: the named list of split indices from resamples. Each split carries a seed when the wrapped object carried one.

The first class is one of "design_cv_resamples", "design_nested_cv_resamples", or "design_bootstrap_resamples". Use design_fold() to convert a split into model-ready inputs.

Details

Workflow

A typical workflow starts with an index-only resampling object created by resample_cv(), resample_nested_cv(), resample_bootstrap(), or another package function that returns the same classes and split structure. Call design_resample() to attach a design_spec() object to those split indices. Then iterate over seq_along(resample$splits), materialize each split with design_fold(), fit the model on fold$analysis, and evaluate predictions on the assessment or OOB inputs supplied by that resampling method.

Use design_resample() when the split indices already exist. Use design_cv(), design_nested_cv(), or design_bootstrap() when the design-aware object should create the split indices.

Retained rows

The retained row universe is computed once before the supplied split indices are checked. The na_action value stored in spec controls whether rows with missing predictor or response values are kept, removed, or rejected. Weights and offsets are checked on retained rows using the same rules as design_fit().

All split indices must be contained in the design-retained row universe. If the design drops rows, create the resampling object on the retained rows or use one of the design-aware split constructors.

Validation

The supplied resampling object's n must equal nrow(data). Split indices must be positive integer vectors.

For CV splits, analysis and assessment must be non-empty, unique, and disjoint within every split.

For nested-CV splits, outer and inner analysis and assessment vectors must be non-empty, unique, and disjoint. Every inner analysis and inner assessment vector must be contained in the corresponding outer analysis vector.

For bootstrap splits, assessment must equal the retained row universe. oob must be set-equal to the retained-row complement of unique(analysis). Bootstrap analysis may contain duplicates. Ungrouped bootstrap analysis must have the same length as the retained row universe.

`response_type`

response_type = "response" returns the ordinary evaluated response in each materialized split. response_type = "ordinal_counts" returns weighted ordinal-count response matrices in the materialized y components supplied by the resampling method. Ordinal-count splits require a two-sided bare-symbol factor or ordered-factor response with non-missing retained responses. They also require weights when splits are materialized because the response matrix stores weighted counts.

The response type is fixed when the design-aware resampling object is created. Every materialized split uses the same response representation. Create a separate design-aware resampling object when a different response representation is needed.

Per-split seeds

Each split may carry a seed, a single positive integer for reproducible model fits on that split. Objects built by resample_cv(), resample_nested_cv(), and resample_bootstrap() already carry seeds. design_resample() preserves the seeds on the wrapped object and does not create or modify them, so design_fold() propagates whatever seed each split carries. See resample_cv() for the seeding recipe and its rationale.

Examples

#----------------------------------------------------------------------------
# Wrap existing CV splits and estimate prediction error
#----------------------------------------------------------------------------
library(bkmodel)

set.seed(1L)
analysis <- data.frame(
  patient_id = sprintf("P%02d", 1:24),
  age = round(rnorm(24, mean = 60, sd = 8)),
  treatment = factor(rep(c("control", "active"), 12)),
  stage = factor(rep(c("I", "II", "III"), each = 8)),
  exposure = runif(24, min = 0.5, max = 2),
  inverse_prob_weight = runif(24, min = 0.8, max = 1.2),
  chart_score = rnorm(24)
)
analysis$risk_group <- interaction(
  analysis$treatment,
  analysis$stage,
  drop = TRUE
)

analysis$outcome <- with(
  analysis,
  5 + 0.25 * age + 1.5 * (treatment == "active") +
    0.8 * (stage == "III") + log(exposure) + rnorm(24)
)

spec <- design_spec(
  outcome ~ .,
  weights = ~ inverse_prob_weight,
  offset = ~ log(exposure),
  case_id = ~ patient_id,
  exclude = ~ chart_score + risk_group,
  encoding = "dummy",
  intercept = TRUE,
  na_action = "na.omit",
  novel_levels = "fail"
)

set.seed(42L)
raw_cv <- resample_cv(
  analysis,
  outcome ~ age,
  n_folds = 4L,
  strata = ~ risk_group
)

cv <- design_resample(spec, analysis, raw_cv)

fold_rmse <- vapply(seq_along(cv$splits), function(i) {
  fold <- design_fold(cv, i)

  fit <- stats::lm.wfit(
    x = fold$analysis$x,
    y = fold$analysis$y,
    w = fold$analysis$weights,
    offset = fold$analysis$offset
  )

  predicted <- drop(fold$assessment$x %*% fit$coefficients)
  predicted <- predicted + fold$assessment$offset
  sqrt(mean((fold$assessment$y - predicted)^2))
}, numeric(1))

fold_rmse
#> [1] 1.1333054 0.9498026 0.8674357 0.5113645
mean(fold_rmse)
#> [1] 0.865477

#----------------------------------------------------------------------------
# Wrap existing bootstrap splits for out-of-bag scoring
#----------------------------------------------------------------------------
set.seed(7L)
raw_bs <- resample_bootstrap(
  analysis,
  outcome ~ age,
  n_boot = 10L,
  strata = ~ risk_group
)

bs <- design_resample(spec, analysis, raw_bs)
boot_fold <- design_fold(bs, 1L)
c(
  analysis_rows = length(boot_fold$analysis$row_index),
  oob_rows = length(boot_fold$oob$row_index)
)
#> analysis_rows      oob_rows 
#>            24             7