Create a design-aware K-fold cross-validation plan

design_cv() creates a lazy design-aware resampling object for K-fold cross-validation. It computes the retained row universe from a design_spec() object and creates K-fold split indices on those rows. It does not fit a statistical model and does not build a full-data design matrix.

Use design_fold() to materialize one split. Each materialized fold fits its design schema on the analysis rows only and then encodes the assessment rows with that analysis schema.

Usage

design_cv(
  spec,
  data,
  n_folds = 5L,
  n_iters = 1L,
  strata = NULL,
  group = NULL,
  response_type = c("response", "ordinal_counts")
)

Arguments

spec: (design_spec)
Unfitted design specification returned by design_spec().
data: (data.frame)
Analysis data used to create folds. Objects that inherit from data.frame, including tibbles, are accepted.
n_folds: (numeric scalar: 5L; whole number in [2, unit_n])
Number of folds. Whole-number numeric values such as 5 and 5.0 are accepted and stored as integers. The value must be at least 2 and no larger than the number of sampling units.
n_iters: (numeric scalar: 1L; whole number in [1, .Machine$integer.max])
Number of independent K-fold partitions. Use values greater than one for repeated K-fold cross-validation.
strata: (formula: NULL)
Stratification specification. Use NULL for no stratification. Use ~ 1 to stratify by the raw response. Use a one-sided formula such as ~ treatment or ~ site + treatment to stratify by observed columns.
group: (formula: NULL)
Grouping specification. Use NULL for row-level resampling. Use a one-sided formula such as ~ patient_id to keep rows with the same group value together.
response_type: (Scalar character: "response")
Response output mode used by design_fold(). Must be one of "response" or "ordinal_counts".

Value

A list of class c("design_cv_resamples", "design_resamples") with elements:

spec: the original design_spec() object.
data: the original analysis data.
row_index: retained row identities used for split construction.
response_type: response representation used by materialized folds.
response_levels: declared response levels for ordinal-count output, or NULL.
resamples: the underlying index-only cv_resamples object.
splits: a named list of CV split indices.

Each split contains analysis, assessment, iter, fold, and seed. Use design_fold() to convert a split into model-ready inputs.

Details

Workflow

A typical validation workflow starts with design_spec() to declare the design rules. Call design_cv() after setting the random-number seed to create reproducible folds. Then iterate over seq_along(cv$splits), materialize each split with design_fold(), fit the model on fold$analysis, and evaluate predictions on fold$assessment.

Keeping split construction separate from fold materialization prevents assessment rows from determining factor levels, dropped categorical terms, column order, or other fitted design-schema details.

Retained rows

The retained row universe is computed once before folds are assigned. The na_action value stored in spec controls whether rows with missing predictor or response values are kept, removed, or rejected. Weights and offsets are checked on retained rows using the same rules as design_fit().

Each split stores row identities from the retained universe. When a fold is materialized, fold$analysis$row_index and fold$assessment$row_index refer to row numbers in the original data. Assessment row identities are preserved after split construction so prediction errors can be aligned back to the original data. Assessment predictors and responses are still encoded with the fold-specific analysis schema.

Splits

n_folds controls the number of folds in each K-fold partition. n_iters controls the number of independent K-fold partitions. The total number of splits is n_folds * n_iters.

Within each iteration, every retained row appears in exactly one assessment set. Across different iterations, the same row can appear in different assessment folds. Call base::set.seed() immediately before design_cv() for reproducible split assignment.

Each split carries a seed, a single positive integer for a reproducible model fit on that split. design_fold() copies this seed onto the materialized fold. See resample_cv() for the seeding recipe and its rationale.

Operational variables

group and strata are optional operational controls for assigning folds. They are evaluated on the retained rows after the design specification computes the resampling row universe.

If the design formula uses ., variables used only for group or strata must not enter the predictor set through . expansion. Put those variables in case_id, weights, offset, or exclude, or write an explicit predictor formula. Operational variables may also be named explicitly as predictors when that is the intended statistical design.

Response stratification

Use strata = NULL for unstratified folds. Use a one-sided formula such as strata = ~ treatment to balance folds by observed covariate levels. Use strata = ~ 1 to stratify by the raw response.

Response stratification evaluates the response before any response_type conversion. For ordinal-count workflows with rare levels, strata = ~ 1 can reduce the chance that an analysis fold has an all-zero declared-level response column. See resample_cv() for additional stratification and grouping details.

`response_type`

response_type = "response" returns the ordinary evaluated response in each materialized fold. response_type = "ordinal_counts" returns weighted ordinal-count response matrices in fold$analysis$y and fold$assessment$y. Ordinal-count folds require a two-sided bare-symbol factor or ordered-factor response with non-missing retained responses. They also require evaluated weights because the response matrix stores weighted counts.

The response type is fixed when the CV object is created. Every materialized fold uses the same response representation for analysis and assessment inputs. Create a separate design CV object when a different response representation is needed.

Examples

#----------------------------------------------------------------------------
# Estimate prediction error with stratified K-fold cross-validation
#----------------------------------------------------------------------------
library(bkmodel)

set.seed(1L)
analysis <- data.frame(
  patient_id = sprintf("P%02d", 1:24),
  age = round(rnorm(24, mean = 60, sd = 8)),
  treatment = factor(rep(c("control", "active"), 12)),
  stage = factor(rep(c("I", "II", "III"), each = 8)),
  exposure = runif(24, min = 0.5, max = 2),
  inverse_prob_weight = runif(24, min = 0.8, max = 1.2),
  chart_score = rnorm(24)
)
analysis$risk_group <- interaction(
  analysis$treatment,
  analysis$stage,
  drop = TRUE
)

analysis$outcome <- with(
  analysis,
  5 + 0.25 * age + 1.5 * (treatment == "active") +
    0.8 * (stage == "III") + log(exposure) + rnorm(24)
)

spec <- design_spec(
  outcome ~ .,
  weights = ~ inverse_prob_weight,
  offset = ~ log(exposure),
  case_id = ~ patient_id,
  exclude = ~ chart_score + risk_group,
  encoding = "dummy",
  intercept = TRUE,
  na_action = "na.omit",
  novel_levels = "fail"
)

set.seed(42L)
cv <- design_cv(
  spec,
  analysis,
  n_folds = 4L,
  strata = ~ risk_group
)

fold_rmse <- vapply(seq_along(cv$splits), function(i) {
  fold <- design_fold(cv, i)

  fit <- stats::lm.wfit(
    x = fold$analysis$x,
    y = fold$analysis$y,
    w = fold$analysis$weights,
    offset = fold$analysis$offset
  )

  predicted <- drop(fold$assessment$x %*% fit$coefficients)
  predicted <- predicted + fold$assessment$offset
  sqrt(mean((fold$assessment$y - predicted)^2))
}, numeric(1))

fold_rmse
#> [1] 1.1333054 0.9498026 0.8674357 0.5113645
mean(fold_rmse)
#> [1] 0.865477

#----------------------------------------------------------------------------
# Keep repeated patient observations together
#----------------------------------------------------------------------------
repeated <- data.frame(
  patient_id = rep(sprintf("P%02d", 1:12), each = 2),
  visit = rep(c("baseline", "follow_up"), times = 12),
  treatment = factor(rep(rep(c("control", "active"), each = 6), each = 2)),
  age = rep(round(rnorm(12, mean = 60, sd = 8)), each = 2),
  outcome = rnorm(24)
)

repeated_spec <- design_spec(
  outcome ~ age + treatment + visit,
  case_id = ~ patient_id,
  encoding = "dummy",
  intercept = TRUE
)

set.seed(7L)
grouped_cv <- design_cv(
  repeated_spec,
  repeated,
  n_folds = 4L,
  group = ~ patient_id,
  strata = ~ treatment
)

grouped_fold <- design_fold(grouped_cv, 1L)
intersect(grouped_fold$analysis$case_id, grouped_fold$assessment$case_id)
#> character(0)