Materialize one fold of a design-aware resampling object

design_fold() returns model-ready inputs for one split of a design-aware resampling object. It fits the split-specific design schema on the split's analysis rows and encodes non-analysis rows with that fitted schema. It does not fit a statistical model.

For ordinary CV folds, the result contains analysis and assessment. For nested-CV outer folds, the result contains analysis, assessment, and inner. For bootstrap splits, the result contains analysis, assessment, and oob.

Usage

design_fold(resample, i)

Arguments

resample: (design_resamples)
A design-aware resampling object returned by design_cv(), design_nested_cv(), design_bootstrap(), or design_resample().
i: (integer or character scalar)
Split position or split name. Numeric values are coerced to integer with base::as.integer(). Character values are matched against names(resample$splits).

Value

A list of class c("design_<method>_fold", "design_fold"). The first class is one of "design_cv_fold", "design_nested_cv_fold", or "design_bootstrap_fold".

All returned folds contain:

id: split name.
seed: the split's single positive integer seed.
analysis: a "design_inputs" object for model fitting.
assessment: a "design_inputs" object for assessment.

Nested-CV outer folds also contain:

inner: a design-aware CV object restricted to the outer analysis rows.

Bootstrap folds also contain:

oob: a "design_inputs" object for out-of-bag assessment.

Details

Workflow

Use design_fold() after creating a design-aware resampling object with design_cv(), design_nested_cv(), design_bootstrap(), or design_resample(). Select a split by integer position or by split name. Fit the statistical model on fold$analysis. Evaluate predictions on fold$assessment, fold$oob, or an inner fold depending on the resampling method.

Design schemas

Each split's design schema is fitted on its analysis rows only. Assessment and OOB rows are encoded with that fitted schema. No assessment or OOB schema is fitted.

This prevents assessment and OOB rows from determining factor levels, dropped categorical terms, column order, or other fitted design-schema details. Fold-local novel levels are handled by the novel_levels value stored in the original design_spec() object.

Row identity

The row_index element in each returned design_inputs object refers to row numbers in the original data supplied to the resampling constructor or wrapper. For CV and nested-CV folds, analysis$row_index and assessment$row_index are disjoint. For bootstrap splits, analysis$row_index can contain duplicates and preserves bootstrap draw order. Bootstrap assessment$row_index is the full retained-row universe. Bootstrap oob$row_index contains rows not present in unique(analysis$row_index).

Assessment and OOB row identities are preserved even when fold-local novel levels encode to NA.

Nested CV

Materializing a nested-CV outer fold returns an inner design-aware CV object. Its splits are restricted to the outer analysis rows. Call design_fold(outer$inner, j) to materialize an inner fold for model selection. The outer fold's seed is used for the final refit, and each inner fold's seed is used for its candidate sweep.

Reproducible fits

Each fold carries the split's seed, so a fold is self-contained. Call set.seed(fold$seed) immediately before fitting a random-using model on fold$analysis to make the fit reproducible regardless of run order. Applying the seed sets the global random-number state. See resample_cv() for the seeding recipe and its rationale.

Response output

The response representation is fixed when the design-aware resampling object is created. With response_type = "response", materialized inputs contain the ordinary evaluated response. With response_type = "ordinal_counts", materialized y components contain weighted ordinal-count response matrices.

Examples

#----------------------------------------------------------------------------
# Materialize a CV fold and estimate assessment error
#----------------------------------------------------------------------------
library(bkmodel)

set.seed(1L)
analysis <- data.frame(
  patient_id = sprintf("P%02d", 1:24),
  age = round(rnorm(24, mean = 60, sd = 8)),
  treatment = factor(rep(c("control", "active"), 12)),
  stage = factor(rep(c("I", "II", "III"), each = 8)),
  exposure = runif(24, min = 0.5, max = 2),
  inverse_prob_weight = runif(24, min = 0.8, max = 1.2),
  chart_score = rnorm(24)
)
analysis$risk_group <- interaction(
  analysis$treatment,
  analysis$stage,
  drop = TRUE
)

analysis$outcome <- with(
  analysis,
  5 + 0.25 * age + 1.5 * (treatment == "active") +
    0.8 * (stage == "III") + log(exposure) + rnorm(24)
)

spec <- design_spec(
  outcome ~ .,
  weights = ~ inverse_prob_weight,
  offset = ~ log(exposure),
  case_id = ~ patient_id,
  exclude = ~ chart_score + risk_group,
  encoding = "dummy",
  intercept = TRUE,
  na_action = "na.omit",
  novel_levels = "fail"
)

set.seed(42L)
cv <- design_cv(
  spec,
  analysis,
  n_folds = 4L,
  strata = ~ risk_group
)

fold <- design_fold(cv, 1L)

fit <- stats::lm.wfit(
  x = fold$analysis$x,
  y = fold$analysis$y,
  w = fold$analysis$weights,
  offset = fold$analysis$offset
)

predicted <- drop(fold$assessment$x %*% fit$coefficients)
predicted <- predicted + fold$assessment$offset
sqrt(mean((fold$assessment$y - predicted)^2))
#> [1] 1.133305

#----------------------------------------------------------------------------
# Materialize an inner fold for nested model selection
#----------------------------------------------------------------------------
set.seed(7L)
ncv <- design_nested_cv(
  spec,
  analysis,
  n_outer_folds = 3L,
  n_inner_folds = 2L,
  strata = ~ risk_group
)

outer <- design_fold(ncv, 1L)
inner <- design_fold(outer$inner, 1L)
all(inner$analysis$row_index %in% outer$analysis$row_index)
#> [1] TRUE
all(inner$assessment$row_index %in% outer$analysis$row_index)
#> [1] TRUE

#----------------------------------------------------------------------------
# Materialize a bootstrap split for out-of-bag assessment
#----------------------------------------------------------------------------
set.seed(11L)
bs <- design_bootstrap(
  spec,
  analysis,
  n_boot = 10L,
  strata = ~ risk_group
)

boot_fold <- design_fold(bs, 1L)
c(
  analysis_rows = length(boot_fold$analysis$row_index),
  oob_rows = length(boot_fold$oob$row_index)
)
#> analysis_rows      oob_rows 
#>            24             9