Skip to contents

Create K-fold cross-validation resampling indices

resample_cv.Rd

Create an index-only resampling plan for K-fold cross-validation. Each split stores integer row positions from data for model fitting and model assessment. The function supports repeated K-fold cross-validation, stratified folds, and grouped folds.

Usage

resample_cv(
  data,
  formula,
  n_folds = 5L,
  n_iters = 1L,
  strata = NULL,
  group = NULL
)

Arguments

data

(data.frame)
Data set to resample. It must have at least one column and at least two rows.

formula

(formula)
Two-sided model formula. The left-hand side must be a single untransformed column name in data. Calls such as cbind(y1, y2), Surv(time, status), log(y), and factor(y) are rejected. The right-hand side is ignored when the resampling indices are created.

n_folds

(numeric scalar: 5L; whole number in [2, unit_n])
Number of folds. unit_n is nrow(data) when group is NULL and the number of unique groups otherwise. Whole-number numeric values such as 5 and 5.0 are accepted and stored as integers.

n_iters

(numeric scalar: 1L; whole number in [1, .Machine$integer.max])
Number of independent K-fold partitions. Use values greater than one for repeated K-fold cross-validation.

strata

(formula: NULL)
Stratification specification. Use NULL for no stratification. Use ~ 1 to stratify by the model outcome. Use a one-sided formula such as ~ a + b to stratify by the interaction of columns a and b.

group

(formula: NULL)
Grouping specification. Use NULL for row-level resampling. Use a one-sided formula such as ~ subject to keep rows with the same group value together. The right-hand side must be a single bare column name. Create composite group keys in data before calling resample_cv().

Value

An S3 list of class c("cv_resamples", "resamples") with elements:

  • call: the matched call.

  • method: "cv".

  • n: nrow(data).

  • formula: the input formula.

  • outcome_var: the resolved outcome column name.

  • strata: the input strata formula or NULL.

  • strata_vars: the resolved strata column names or NULL.

  • group: the input group formula or NULL.

  • group_vars: the resolved group column name or NULL.

  • n_groups: the number of unique groups when group is supplied, otherwise NULL.

  • n_folds: the resolved fold count.

  • n_iters: the resolved iteration count.

  • splits: a named list of split entries. Each split contains analysis, assessment, iter, fold, and seed.

Details

resample_cv() does not copy data. It returns row indices that can be used to subset data later. The predictor side of formula is not used to build the splits. It is accepted so the same formula can be passed to a model-fitting function.

Splits

Each element of splits contains:

  • analysis: integer row indices used to fit a model. These are all rows not in assessment. They are sorted in ascending order.

  • assessment: integer row indices used to assess the fitted model. They are sorted in ascending order.

  • iter: integer iteration number.

  • fold: integer fold number within iter.

  • seed: single positive integer for reproducible model fits on this split.

Within each iteration, the assessment indices partition seq_len(nrow(data)). A row appears in exactly one assessment set per iteration. Across different iterations, the same row can appear in different folds.

Fold assignment

Folds are assigned over sampling units. Rows are the sampling units when group is NULL. Groups are the sampling units when group is supplied.

Without stratification, fold sizes are balanced over the sampling units. The target fold sizes are floor(unit_n / n_folds), with unit_n %% n_folds folds one unit larger. The larger folds are selected at random.

When strata is supplied, strata are processed from smallest to largest, so rare strata are allocated first. Within each stratum, sampling units are shuffled and distributed across folds as evenly as possible. Per-stratum fold counts differ by at most one. Ties are broken with the current random-number generator state.

Stratification

Stratification is optional. Pass NULL for no stratification. strata = ~ 1 stratifies by the model outcome. A formula such as strata = ~ a + b stratifies by interaction(data$a, data$b, drop = TRUE). Variables referenced by strata must be columns of data. NA stratum values are not allowed. Empty factor levels are dropped. Stratification variables are treated as categorical. Pre-bin continuous variables before passing them to strata. This also applies when strata = ~ 1 and the model outcome is continuous.

Expressions in strata are not evaluated as transformations. Create transformed or composite strata columns in data before calling resample_cv().

Grouping

Grouping is optional. Pass NULL for ordinary row-level resampling. Pass a one-sided formula such as group = ~ subject to keep all rows from each subject together. The right-hand side of group must be a single bare column name in data. NA group values are not allowed. Empty factor levels are dropped. Rows that share a group value are kept together. All rows of any one group land entirely in one fold's assessment per iteration. Because groups are the sampling units, n_folds must be less than or equal to the number of unique groups. Fold balance is defined over groups, not rows. When group sizes differ, row counts across folds may differ by more than one.

When strata and group are both supplied, the resolved stratum vector must be constant within each group. If strata = ~ 1 is used with group, the outcome must be constant within each group.

Repeated K-fold

When n_iters > 1, the function creates n_iters independent K-fold partitions and concatenates the splits. The total number of splits is n_folds * n_iters. Each iteration advances the current random-number generator state.

Split Names

Split names are generated from the total number of splits. For fewer than 10 splits, names are split1, ..., split9. For 10 or more splits, names are zero-padded so lexical order matches numeric order.

Reproducibility

Call base::set.seed() immediately before resample_cv() for reproducible splits.

Split indices reproduce the fold rows, but a downstream model fit that uses randomness depends on the global random-number state when it runs, so it does not reproduce on its own. Each split therefore carries a seed, a single positive integer generated at construction. Set the split seed immediately before fitting that split to make the fit reproducible regardless of run order.

# Single fit per split.
set.seed(split$seed)
fit <- fit_model(split$analysis)

# Several candidate models on one analysis set, each reproducible alone.
set.seed(split$seed)
candidate_seeds <- sample.int(.Machine$integer.max, K, replace = FALSE)
for (k in seq_len(K)) {
  set.seed(candidate_seeds[k])
  fit_k <- fit_candidate_k(split$analysis)
}

Seeds are created from the global generator and the generator state is then restored, so split indices and the caller's random-number stream are unchanged. Reproducing a fit assumes the same base::RNGkind() on each run.

See also

resample_nested_cv(), resample_bootstrap(), base::set.seed()

Examples

#----------------------------------------------------------------------------
# resample_cv() examples
#----------------------------------------------------------------------------
library(bkmodel)

set.seed(1L)

data <- data.frame(
  outcome = c(rep("event", 12L), rep("none", 12L)),
  age = c(34L, 52L, 41L, 63L, 29L, 48L, 37L, 55L,
          46L, 31L, 60L, 43L, 39L, 50L, 28L, 44L,
          57L, 36L, 62L, 33L, 47L, 53L, 40L, 58L),
  clinic = rep(c("north", "south", "west"), each = 8L),
  patient = rep(seq_len(12L), each = 2L)
)

cv <- resample_cv(data, outcome ~ age, n_folds = 4L)
cv$splits[[1L]]$analysis
#>  [1]  3  5  6  8  9 10 12 13 15 16 17 18 19 20 21 22 23 24
cv$splits[[1L]]$assessment
#> [1]  1  2  4  7 11 14

# Keep outcome classes balanced across folds.
resample_cv(data, outcome ~ age, n_folds = 4L, strata = ~ outcome)
#> $call
#> resample_cv(data = data, formula = outcome ~ age, n_folds = 4L, 
#>     strata = ~outcome)
#> 
#> $method
#> [1] "cv"
#> 
#> $n
#> [1] 24
#> 
#> $formula
#> outcome ~ age
#> <environment: 0x5ed28e047c88>
#> 
#> $outcome_var
#> [1] "outcome"
#> 
#> $strata
#> ~outcome
#> <environment: 0x5ed28e047c88>
#> 
#> $strata_vars
#> [1] "outcome"
#> 
#> $group
#> NULL
#> 
#> $group_vars
#> NULL
#> 
#> $n_groups
#> NULL
#> 
#> $n_folds
#> [1] 4
#> 
#> $n_iters
#> [1] 1
#> 
#> $splits
#> $splits$split1
#> $splits$split1$analysis
#>  [1]  1  2  3  5  6  8  9 10 11 13 14 15 16 19 20 21 22 23
#> 
#> $splits$split1$assessment
#> [1]  4  7 12 17 18 24
#> 
#> $splits$split1$iter
#> [1] 1
#> 
#> $splits$split1$fold
#> [1] 1
#> 
#> $splits$split1$seed
#> [1] 1488999795
#> 
#> 
#> $splits$split2
#> $splits$split2$analysis
#>  [1]  3  4  5  6  7  9 10 11 12 14 15 16 17 18 19 21 23 24
#> 
#> $splits$split2$assessment
#> [1]  1  2  8 13 20 22
#> 
#> $splits$split2$iter
#> [1] 1
#> 
#> $splits$split2$fold
#> [1] 2
#> 
#> $splits$split2$seed
#> [1] 2045961320
#> 
#> 
#> $splits$split3
#> $splits$split3$analysis
#>  [1]  1  2  4  5  6  7  8  9 12 13 15 17 18 19 20 22 23 24
#> 
#> $splits$split3$assessment
#> [1]  3 10 11 14 16 21
#> 
#> $splits$split3$iter
#> [1] 1
#> 
#> $splits$split3$fold
#> [1] 3
#> 
#> $splits$split3$seed
#> [1] 1564828631
#> 
#> 
#> $splits$split4
#> $splits$split4$analysis
#>  [1]  1  2  3  4  7  8 10 11 12 13 14 16 17 18 20 21 22 24
#> 
#> $splits$split4$assessment
#> [1]  5  6  9 15 19 23
#> 
#> $splits$split4$iter
#> [1] 1
#> 
#> $splits$split4$fold
#> [1] 4
#> 
#> $splits$split4$seed
#> [1] 1191114220
#> 
#> 
#> 
#> attr(,"class")
#> [1] "cv_resamples" "resamples"   

# Repeat the full K-fold partition three times.
resample_cv(data, outcome ~ age, n_folds = 4L, n_iters = 3L)
#> $call
#> resample_cv(data = data, formula = outcome ~ age, n_folds = 4L, 
#>     n_iters = 3L)
#> 
#> $method
#> [1] "cv"
#> 
#> $n
#> [1] 24
#> 
#> $formula
#> outcome ~ age
#> <environment: 0x5ed28e047c88>
#> 
#> $outcome_var
#> [1] "outcome"
#> 
#> $strata
#> NULL
#> 
#> $strata_vars
#> NULL
#> 
#> $group
#> NULL
#> 
#> $group_vars
#> NULL
#> 
#> $n_groups
#> NULL
#> 
#> $n_folds
#> [1] 4
#> 
#> $n_iters
#> [1] 3
#> 
#> $splits
#> $splits$split01
#> $splits$split01$analysis
#>  [1]  2  3  4  5  7  9 10 11 13 14 15 16 17 18 20 21 22 24
#> 
#> $splits$split01$assessment
#> [1]  1  6  8 12 19 23
#> 
#> $splits$split01$iter
#> [1] 1
#> 
#> $splits$split01$fold
#> [1] 1
#> 
#> $splits$split01$seed
#> [1] 1450025315
#> 
#> 
#> $splits$split02
#> $splits$split02$analysis
#>  [1]  1  2  3  5  6  8  9 10 12 13 14 15 16 17 18 19 22 23
#> 
#> $splits$split02$assessment
#> [1]  4  7 11 20 21 24
#> 
#> $splits$split02$iter
#> [1] 1
#> 
#> $splits$split02$fold
#> [1] 2
#> 
#> $splits$split02$seed
#> [1] 1462950700
#> 
#> 
#> $splits$split03
#> $splits$split03$analysis
#>  [1]  1  2  3  4  5  6  7  8 10 11 12 14 19 20 21 22 23 24
#> 
#> $splits$split03$assessment
#> [1]  9 13 15 16 17 18
#> 
#> $splits$split03$iter
#> [1] 1
#> 
#> $splits$split03$fold
#> [1] 3
#> 
#> $splits$split03$seed
#> [1] 1680892265
#> 
#> 
#> $splits$split04
#> $splits$split04$analysis
#>  [1]  1  4  6  7  8  9 11 12 13 15 16 17 18 19 20 21 23 24
#> 
#> $splits$split04$assessment
#> [1]  2  3  5 10 14 22
#> 
#> $splits$split04$iter
#> [1] 1
#> 
#> $splits$split04$fold
#> [1] 4
#> 
#> $splits$split04$seed
#> [1] 1698407666
#> 
#> 
#> $splits$split05
#> $splits$split05$analysis
#>  [1]  2  3  4  5  7  8  9 10 11 14 15 18 19 20 21 22 23 24
#> 
#> $splits$split05$assessment
#> [1]  1  6 12 13 16 17
#> 
#> $splits$split05$iter
#> [1] 2
#> 
#> $splits$split05$fold
#> [1] 1
#> 
#> $splits$split05$seed
#> [1] 1035442934
#> 
#> 
#> $splits$split06
#> $splits$split06$analysis
#>  [1]  1  4  5  6  8 10 11 12 13 14 15 16 17 18 21 22 23 24
#> 
#> $splits$split06$assessment
#> [1]  2  3  7  9 19 20
#> 
#> $splits$split06$iter
#> [1] 2
#> 
#> $splits$split06$fold
#> [1] 2
#> 
#> $splits$split06$seed
#> [1] 1731218226
#> 
#> 
#> $splits$split07
#> $splits$split07$analysis
#>  [1]  1  2  3  5  6  7  8  9 12 13 14 16 17 18 19 20 21 22
#> 
#> $splits$split07$assessment
#> [1]  4 10 11 15 23 24
#> 
#> $splits$split07$iter
#> [1] 2
#> 
#> $splits$split07$fold
#> [1] 3
#> 
#> $splits$split07$seed
#> [1] 821486319
#> 
#> 
#> $splits$split08
#> $splits$split08$analysis
#>  [1]  1  2  3  4  6  7  9 10 11 12 13 15 16 17 19 20 23 24
#> 
#> $splits$split08$assessment
#> [1]  5  8 14 18 21 22
#> 
#> $splits$split08$iter
#> [1] 2
#> 
#> $splits$split08$fold
#> [1] 4
#> 
#> $splits$split08$seed
#> [1] 14344327
#> 
#> 
#> $splits$split09
#> $splits$split09$analysis
#>  [1]  1  2  3  4  5  6  8  9 11 13 14 17 18 19 20 22 23 24
#> 
#> $splits$split09$assessment
#> [1]  7 10 12 15 16 21
#> 
#> $splits$split09$iter
#> [1] 3
#> 
#> $splits$split09$fold
#> [1] 1
#> 
#> $splits$split09$seed
#> [1] 812565012
#> 
#> 
#> $splits$split10
#> $splits$split10$analysis
#>  [1]  2  4  5  6  7  9 10 12 13 14 15 16 17 18 19 20 21 24
#> 
#> $splits$split10$assessment
#> [1]  1  3  8 11 22 23
#> 
#> $splits$split10$iter
#> [1] 3
#> 
#> $splits$split10$fold
#> [1] 2
#> 
#> $splits$split10$seed
#> [1] 964227480
#> 
#> 
#> $splits$split11
#> $splits$split11$analysis
#>  [1]  1  2  3  7  8  9 10 11 12 14 15 16 18 19 21 22 23 24
#> 
#> $splits$split11$assessment
#> [1]  4  5  6 13 17 20
#> 
#> $splits$split11$iter
#> [1] 3
#> 
#> $splits$split11$fold
#> [1] 3
#> 
#> $splits$split11$seed
#> [1] 204650037
#> 
#> 
#> $splits$split12
#> $splits$split12$analysis
#>  [1]  1  3  4  5  6  7  8 10 11 12 13 15 16 17 20 21 22 23
#> 
#> $splits$split12$assessment
#> [1]  2  9 14 18 19 24
#> 
#> $splits$split12$iter
#> [1] 3
#> 
#> $splits$split12$fold
#> [1] 4
#> 
#> $splits$split12$seed
#> [1] 1670322643
#> 
#> 
#> 
#> attr(,"class")
#> [1] "cv_resamples" "resamples"   

# Keep repeated observations from the same patient together.
resample_cv(data, outcome ~ age, n_folds = 4L, group = ~ patient)
#> $call
#> resample_cv(data = data, formula = outcome ~ age, n_folds = 4L, 
#>     group = ~patient)
#> 
#> $method
#> [1] "cv"
#> 
#> $n
#> [1] 24
#> 
#> $formula
#> outcome ~ age
#> <environment: 0x5ed28e047c88>
#> 
#> $outcome_var
#> [1] "outcome"
#> 
#> $strata
#> NULL
#> 
#> $strata_vars
#> NULL
#> 
#> $group
#> ~patient
#> <environment: 0x5ed28e047c88>
#> 
#> $group_vars
#> [1] "patient"
#> 
#> $n_groups
#> [1] 12
#> 
#> $n_folds
#> [1] 4
#> 
#> $n_iters
#> [1] 1
#> 
#> $splits
#> $splits$split1
#> $splits$split1$analysis
#>  [1]  1  2  3  4  5  6  7  8  9 10 13 14 19 20 21 22 23 24
#> 
#> $splits$split1$assessment
#> [1] 11 12 15 16 17 18
#> 
#> $splits$split1$iter
#> [1] 1
#> 
#> $splits$split1$fold
#> [1] 1
#> 
#> $splits$split1$seed
#> [1] 1462456679
#> 
#> 
#> $splits$split2
#> $splits$split2$analysis
#>  [1]  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 23 24
#> 
#> $splits$split2$assessment
#> [1]  1  2 19 20 21 22
#> 
#> $splits$split2$iter
#> [1] 1
#> 
#> $splits$split2$fold
#> [1] 2
#> 
#> $splits$split2$seed
#> [1] 1214792828
#> 
#> 
#> $splits$split3
#> $splits$split3$analysis
#>  [1]  1  2  7  8  9 10 11 12 15 16 17 18 19 20 21 22 23 24
#> 
#> $splits$split3$assessment
#> [1]  3  4  5  6 13 14
#> 
#> $splits$split3$iter
#> [1] 1
#> 
#> $splits$split3$fold
#> [1] 3
#> 
#> $splits$split3$seed
#> [1] 939099213
#> 
#> 
#> $splits$split4
#> $splits$split4$analysis
#>  [1]  1  2  3  4  5  6 11 12 13 14 15 16 17 18 19 20 21 22
#> 
#> $splits$split4$assessment
#> [1]  7  8  9 10 23 24
#> 
#> $splits$split4$iter
#> [1] 1
#> 
#> $splits$split4$fold
#> [1] 4
#> 
#> $splits$split4$seed
#> [1] 1155083874
#> 
#> 
#> 
#> attr(,"class")
#> [1] "cv_resamples" "resamples"