Create nested K-fold cross-validation resampling indices

Create an index-only resampling plan for nested K-fold cross-validation. Each outer split stores integer row positions for final model assessment. Each outer split also contains inner cross-validation splits for model selection within the outer analysis set. The function supports repeated outer cross-validation, stratified folds, and grouped folds.

Usage

resample_nested_cv(
  data,
  formula,
  n_outer_folds = 5L,
  n_inner_folds = 5L,
  n_iters = 1L,
  strata = NULL,
  group = NULL
)

Arguments

data

(data.frame)
Data set to resample. It must have at least one column and at least two rows.

formula

(formula)
Two-sided model formula. The left-hand side must be a single untransformed column name in data. Calls such as cbind(y1, y2), Surv(time, status), log(y), and factor(y) are rejected. The right-hand side is ignored when the resampling indices are created.

n_outer_folds

(numeric scalar: 5L; whole number in [2, unit_n])
Number of outer folds. unit_n is nrow(data) when group is NULL and the number of unique groups otherwise. Whole-number numeric values such as 5 and 5.0 are accepted and stored as integers.

n_inner_folds

(numeric scalar: 5L; whole number in [2, unit_n - ceiling(unit_n / n_outer_folds)])
Number of inner folds. The upper bound ensures that every outer-analysis set has enough sampling units for inner cross-validation. Whole-number numeric values such as 5 and 5.0 are accepted and stored as integers.

n_iters

(numeric scalar: 1L; whole number in [1, .Machine$integer.max])
Number of independent outer K-fold partitions. Use values greater than one for repeated nested cross-validation.

strata

(formula: NULL)
Stratification specification. Use NULL for no stratification. Use ~ 1 to stratify by the model outcome. Use a one-sided formula such as ~ a + b to stratify by the interaction of columns a and b.

group

(formula: NULL)
Grouping specification. Use NULL for row-level resampling. Use a one-sided formula such as ~ subject to keep rows with the same group value together. The right-hand side must be a single bare column name. Create composite group keys in data before calling resample_nested_cv().

Value

An S3 list of class c("nested_cv_resamples", "resamples") with elements:

• call: the matched call.

• method: "nested_cv".

• n: nrow(data).

• formula: the input formula.

• outcome_var: the resolved outcome column name.

• strata: the input strata formula or NULL.

• strata_vars: the resolved strata column names or NULL.

• group: the input group formula or NULL.

• group_vars: the resolved group column name or NULL.

• n_groups: the number of unique groups when group is supplied, otherwise NULL.

• n_outer_folds: the resolved outer-fold count.

• n_inner_folds: the resolved inner-fold count.

• n_iters: the resolved iteration count.

• splits: a named list of outer split entries. Each outer split contains analysis, assessment, iter, outer_fold, seed, and inner. Each inner split contains analysis, assessment, iter, outer_fold, inner_fold, and seed.

Details

resample_nested_cv() does not copy data. It returns row indices that can be used to subset data later. The predictor side of formula is not used to build the splits. It is accepted so the same formula can be passed to a model-fitting function.

Splits

Each outer split separates the original data rows into:

• analysis: integer row indices used for model selection and final model fitting within that outer split. These are all rows not in the outer assessment. They are sorted in ascending order.

• assessment: integer row indices held out for final assessment of that outer split. They are sorted in ascending order.

• iter: integer iteration number.

• outer_fold: integer outer-fold number within iter.

• seed: single positive integer for reproducible model fits on this split.

• inner: a named list of inner cross-validation splits. Inner splits are created only from the outer analysis rows.

Each inner split contains:

• analysis: integer row indices used to fit a candidate model. They are restricted to the parent outer analysis set. They are sorted in ascending order.

• assessment: integer row indices used to compare candidate models. They are restricted to the parent outer analysis set. They are sorted in ascending order.

• iter: integer iteration number inherited from the parent outer split.

• outer_fold: integer outer-fold number inherited from the parent outer split.

• inner_fold: integer inner-fold number.

• seed: single positive integer for reproducible candidate model fits on this inner analysis set.

Within each iteration, the outer assessment indices partition seq_len(nrow(data)). Within each outer split, the inner assessment indices partition the parent outer analysis indices.

Fold assignment

Outer and inner folds use the same fold-assignment rules as resample_cv(). Rows are the sampling units when group is NULL. Groups are the sampling units when group is supplied.

Without stratification, fold sizes are balanced over the sampling units. When strata is supplied, strata are processed from smallest to largest, so rare strata are allocated first. Within each stratum, sampling units are shuffled and distributed across folds as evenly as possible. Per-stratum fold counts differ by at most one. Ties are broken with the current random-number generator state.

Stratification

Stratification is optional. Pass NULL for no stratification. strata = ~ 1 stratifies by the model outcome. A formula such as strata = ~ a + b stratifies by interaction(data$a, data$b, drop = TRUE). Variables referenced by strata must be columns of data. NA stratum values are not allowed. Empty factor levels are dropped. Stratification variables are treated as categorical. Pre-bin continuous variables before passing them to strata. This also applies when strata = ~ 1 and the model outcome is continuous.

Expressions in strata are not evaluated as transformations. Create transformed or composite strata columns in data before calling resample_nested_cv().

Grouping

Grouping is optional. Pass NULL for ordinary row-level resampling. Pass a one-sided formula such as group = ~ subject to keep all rows from each subject together. The right-hand side of group must be a single bare column name in data. NA group values are not allowed. Empty factor levels are dropped. Rows that share a group value are kept together in both outer and inner splits. No group is split between analysis and assessment.

Because groups are the sampling units, n_outer_folds must be less than or equal to the number of unique groups. n_inner_folds must be no larger than the smallest possible outer-analysis sampling-unit count. This upper bound is unit_n - ceiling(unit_n / n_outer_folds). Fold balance is defined over groups, not rows. When group sizes differ, row counts across folds may differ by more than one.

When strata and group are both supplied, the resolved stratum vector must be constant within each group. If strata = ~ 1 is used with group, the outcome must be constant within each group.

Repeated nested K-fold

When n_iters > 1, the function repeats the outer K-fold partitioning. Inner partitioning is generated separately for each outer split. The total number of outer splits is n_outer_folds * n_iters. Each iteration advances the current random-number generator state.

Split Names

Outer split names are generated from the total number of outer splits. For fewer than 10 outer splits, names are split1, ..., split9. For 10 or more outer splits, names are zero-padded so lexical order matches numeric order. Inner split names are generated separately within each outer split from n_inner_folds.

Reproducibility

Call base::set.seed() immediately before resample_nested_cv() for reproducible splits.

Split indices reproduce the fold rows, but a downstream model fit that uses randomness depends on the global random-number state when it runs, so it does not reproduce on its own. Each outer split carries a seed for its final refit, and each inner split carries a seed for its candidate sweep. Set the relevant seed immediately before fitting to make the fit reproducible regardless of run order.

# Inner candidate sweep on inner analysis set j of outer split i.
inner <- ncv$splits[[i]]$inner[[j]]
set.seed(inner$seed)
candidate_seeds <- sample.int(.Machine$integer.max, K, replace = FALSE)
for (k in seq_len(K)) {
  set.seed(candidate_seeds[k])
  fit_k <- fit_candidate_k(inner$analysis)
}

# Final refit on outer analysis set i.
set.seed(ncv$splits[[i]]$seed)
final <- fit_model(ncv$splits[[i]]$analysis)

Seeds are created from the global generator and the generator state is then restored, so split indices and the caller's random-number stream are unchanged. Reproducing a fit assumes the same base::RNGkind() on each run.

See also

resample_cv(), resample_bootstrap(), base::set.seed()

Examples

#----------------------------------------------------------------------------
# resample_nested_cv() examples
#----------------------------------------------------------------------------
library(bkmodel)

set.seed(1L)

data <- data.frame(
  outcome = c(rep("event", 12L), rep("none", 12L)),
  age = c(34L, 52L, 41L, 63L, 29L, 48L, 37L, 55L,
          46L, 31L, 60L, 43L, 39L, 50L, 28L, 44L,
          57L, 36L, 62L, 33L, 47L, 53L, 40L, 58L),
  clinic = rep(c("north", "south", "west"), each = 8L),
  patient = rep(seq_len(12L), each = 2L)
)

ncv <- resample_nested_cv(
  data,
  outcome ~ age,
  n_outer_folds = 4L,
  n_inner_folds = 3L
)
ncv$splits[[1L]]$assessment
#> [1]  1  2  4  7 11 14
ncv$splits[[1L]]$inner[[1L]]$assessment
#> [1]  3  6  8 10 18 21

# Keep outcome classes balanced in outer and inner folds.
resample_nested_cv(
  data,
  outcome ~ age,
  n_outer_folds = 4L,
  n_inner_folds = 3L,
  strata = ~ outcome
)
#> $call
#> resample_nested_cv(data = data, formula = outcome ~ age, n_outer_folds = 4L, 
#>     n_inner_folds = 3L, strata = ~outcome)
#> 
#> $method
#> [1] "nested_cv"
#> 
#> $n
#> [1] 24
#> 
#> $formula
#> outcome ~ age
#> <environment: 0x5ed28e4223f8>
#> 
#> $outcome_var
#> [1] "outcome"
#> 
#> $strata
#> ~outcome
#> <environment: 0x5ed28e4223f8>
#> 
#> $strata_vars
#> [1] "outcome"
#> 
#> $group
#> NULL
#> 
#> $group_vars
#> NULL
#> 
#> $n_groups
#> NULL
#> 
#> $n_outer_folds
#> [1] 4
#> 
#> $n_inner_folds
#> [1] 3
#> 
#> $n_iters
#> [1] 1
#> 
#> $splits
#> $splits$split1
#> $splits$split1$analysis
#>  [1]  1  2  5  6  7  8  9 10 11 14 15 16 17 18 19 21 22 24
#> 
#> $splits$split1$assessment
#> [1]  3  4 12 13 20 23
#> 
#> $splits$split1$iter
#> [1] 1
#> 
#> $splits$split1$outer_fold
#> [1] 1
#> 
#> $splits$split1$inner
#> $splits$split1$inner$split1
#> $splits$split1$inner$split1$analysis
#>  [1]  1  5  7  8 10 11 14 15 16 18 22 24
#> 
#> $splits$split1$inner$split1$assessment
#> [1]  2  6  9 17 19 21
#> 
#> $splits$split1$inner$split1$iter
#> [1] 1
#> 
#> $splits$split1$inner$split1$outer_fold
#> [1] 1
#> 
#> $splits$split1$inner$split1$inner_fold
#> [1] 1
#> 
#> $splits$split1$inner$split1$seed
#> [1] 1639459672
#> 
#> 
#> $splits$split1$inner$split2
#> $splits$split1$inner$split2$analysis
#>  [1]  2  5  6  7  9 11 17 18 19 21 22 24
#> 
#> $splits$split1$inner$split2$assessment
#> [1]  1  8 10 14 15 16
#> 
#> $splits$split1$inner$split2$iter
#> [1] 1
#> 
#> $splits$split1$inner$split2$outer_fold
#> [1] 1
#> 
#> $splits$split1$inner$split2$inner_fold
#> [1] 2
#> 
#> $splits$split1$inner$split2$seed
#> [1] 1282683189
#> 
#> 
#> $splits$split1$inner$split3
#> $splits$split1$inner$split3$analysis
#>  [1]  1  2  6  8  9 10 14 15 16 17 19 21
#> 
#> $splits$split1$inner$split3$assessment
#> [1]  5  7 11 18 22 24
#> 
#> $splits$split1$inner$split3$iter
#> [1] 1
#> 
#> $splits$split1$inner$split3$outer_fold
#> [1] 1
#> 
#> $splits$split1$inner$split3$inner_fold
#> [1] 3
#> 
#> $splits$split1$inner$split3$seed
#> [1] 1104490086
#> 
#> 
#> 
#> $splits$split1$seed
#> [1] 267197418
#> 
#> 
#> $splits$split2
#> $splits$split2$analysis
#>  [1]  2  3  4  6  8  9 10 11 12 13 14 15 16 17 20 22 23 24
#> 
#> $splits$split2$assessment
#> [1]  1  5  7 18 19 21
#> 
#> $splits$split2$iter
#> [1] 1
#> 
#> $splits$split2$outer_fold
#> [1] 2
#> 
#> $splits$split2$inner
#> $splits$split2$inner$split1
#> $splits$split2$inner$split1$analysis
#>  [1]  3  8  9 10 11 12 13 14 16 20 23 24
#> 
#> $splits$split2$inner$split1$assessment
#> [1]  2  4  6 15 17 22
#> 
#> $splits$split2$inner$split1$iter
#> [1] 1
#> 
#> $splits$split2$inner$split1$outer_fold
#> [1] 2
#> 
#> $splits$split2$inner$split1$inner_fold
#> [1] 1
#> 
#> $splits$split2$inner$split1$seed
#> [1] 119375614
#> 
#> 
#> $splits$split2$inner$split2
#> $splits$split2$inner$split2$analysis
#>  [1]  2  4  6  9 10 11 13 14 15 16 17 22
#> 
#> $splits$split2$inner$split2$assessment
#> [1]  3  8 12 20 23 24
#> 
#> $splits$split2$inner$split2$iter
#> [1] 1
#> 
#> $splits$split2$inner$split2$outer_fold
#> [1] 2
#> 
#> $splits$split2$inner$split2$inner_fold
#> [1] 2
#> 
#> $splits$split2$inner$split2$seed
#> [1] 1633443714
#> 
#> 
#> $splits$split2$inner$split3
#> $splits$split2$inner$split3$analysis
#>  [1]  2  3  4  6  8 12 15 17 20 22 23 24
#> 
#> $splits$split2$inner$split3$assessment
#> [1]  9 10 11 13 14 16
#> 
#> $splits$split2$inner$split3$iter
#> [1] 1
#> 
#> $splits$split2$inner$split3$outer_fold
#> [1] 2
#> 
#> $splits$split2$inner$split3$inner_fold
#> [1] 3
#> 
#> $splits$split2$inner$split3$seed
#> [1] 205988734
#> 
#> 
#> 
#> $splits$split2$seed
#> [1] 2050082128
#> 
#> 
#> $splits$split3
#> $splits$split3$analysis
#>  [1]  1  3  4  5  7  8  9 10 12 13 14 16 18 19 20 21 23 24
#> 
#> $splits$split3$assessment
#> [1]  2  6 11 15 17 22
#> 
#> $splits$split3$iter
#> [1] 1
#> 
#> $splits$split3$outer_fold
#> [1] 3
#> 
#> $splits$split3$inner
#> $splits$split3$inner$split1
#> $splits$split3$inner$split1$analysis
#>  [1]  1  3  4  8  9 12 13 14 18 19 23 24
#> 
#> $splits$split3$inner$split1$assessment
#> [1]  5  7 10 16 20 21
#> 
#> $splits$split3$inner$split1$iter
#> [1] 1
#> 
#> $splits$split3$inner$split1$outer_fold
#> [1] 3
#> 
#> $splits$split3$inner$split1$inner_fold
#> [1] 1
#> 
#> $splits$split3$inner$split1$seed
#> [1] 567883944
#> 
#> 
#> $splits$split3$inner$split2
#> $splits$split3$inner$split2$analysis
#>  [1]  3  4  5  7  8 10 13 16 18 20 21 23
#> 
#> $splits$split3$inner$split2$assessment
#> [1]  1  9 12 14 19 24
#> 
#> $splits$split3$inner$split2$iter
#> [1] 1
#> 
#> $splits$split3$inner$split2$outer_fold
#> [1] 3
#> 
#> $splits$split3$inner$split2$inner_fold
#> [1] 2
#> 
#> $splits$split3$inner$split2$seed
#> [1] 972300709
#> 
#> 
#> $splits$split3$inner$split3
#> $splits$split3$inner$split3$analysis
#>  [1]  1  5  7  9 10 12 14 16 19 20 21 24
#> 
#> $splits$split3$inner$split3$assessment
#> [1]  3  4  8 13 18 23
#> 
#> $splits$split3$inner$split3$iter
#> [1] 1
#> 
#> $splits$split3$inner$split3$outer_fold
#> [1] 3
#> 
#> $splits$split3$inner$split3$inner_fold
#> [1] 3
#> 
#> $splits$split3$inner$split3$seed
#> [1] 2068272056
#> 
#> 
#> 
#> $splits$split3$seed
#> [1] 1380788132
#> 
#> 
#> $splits$split4
#> $splits$split4$analysis
#>  [1]  1  2  3  4  5  6  7 11 12 13 15 17 18 19 20 21 22 23
#> 
#> $splits$split4$assessment
#> [1]  8  9 10 14 16 24
#> 
#> $splits$split4$iter
#> [1] 1
#> 
#> $splits$split4$outer_fold
#> [1] 4
#> 
#> $splits$split4$inner
#> $splits$split4$inner$split1
#> $splits$split4$inner$split1$analysis
#>  [1]  1  3  4  5  6 12 13 15 19 20 21 22
#> 
#> $splits$split4$inner$split1$assessment
#> [1]  2  7 11 17 18 23
#> 
#> $splits$split4$inner$split1$iter
#> [1] 1
#> 
#> $splits$split4$inner$split1$outer_fold
#> [1] 4
#> 
#> $splits$split4$inner$split1$inner_fold
#> [1] 1
#> 
#> $splits$split4$inner$split1$seed
#> [1] 425926257
#> 
#> 
#> $splits$split4$inner$split2
#> $splits$split4$inner$split2$analysis
#>  [1]  2  3  4  7 11 12 13 17 18 19 21 23
#> 
#> $splits$split4$inner$split2$assessment
#> [1]  1  5  6 15 20 22
#> 
#> $splits$split4$inner$split2$iter
#> [1] 1
#> 
#> $splits$split4$inner$split2$outer_fold
#> [1] 4
#> 
#> $splits$split4$inner$split2$inner_fold
#> [1] 2
#> 
#> $splits$split4$inner$split2$seed
#> [1] 216657380
#> 
#> 
#> $splits$split4$inner$split3
#> $splits$split4$inner$split3$analysis
#>  [1]  1  2  5  6  7 11 15 17 18 20 22 23
#> 
#> $splits$split4$inner$split3$assessment
#> [1]  3  4 12 13 19 21
#> 
#> $splits$split4$inner$split3$iter
#> [1] 1
#> 
#> $splits$split4$inner$split3$outer_fold
#> [1] 4
#> 
#> $splits$split4$inner$split3$inner_fold
#> [1] 3
#> 
#> $splits$split4$inner$split3$seed
#> [1] 746068041
#> 
#> 
#> 
#> $splits$split4$seed
#> [1] 29798004
#> 
#> 
#> 
#> attr(,"class")
#> [1] "nested_cv_resamples" "resamples"          

# Keep repeated observations from the same patient together.
resample_nested_cv(
  data,
  outcome ~ age,
  n_outer_folds = 4L,
  n_inner_folds = 3L,
  group = ~ patient
)
#> $call
#> resample_nested_cv(data = data, formula = outcome ~ age, n_outer_folds = 4L, 
#>     n_inner_folds = 3L, group = ~patient)
#> 
#> $method
#> [1] "nested_cv"
#> 
#> $n
#> [1] 24
#> 
#> $formula
#> outcome ~ age
#> <environment: 0x5ed28e4223f8>
#> 
#> $outcome_var
#> [1] "outcome"
#> 
#> $strata
#> NULL
#> 
#> $strata_vars
#> NULL
#> 
#> $group
#> ~patient
#> <environment: 0x5ed28e4223f8>
#> 
#> $group_vars
#> [1] "patient"
#> 
#> $n_groups
#> [1] 12
#> 
#> $n_outer_folds
#> [1] 4
#> 
#> $n_inner_folds
#> [1] 3
#> 
#> $n_iters
#> [1] 1
#> 
#> $splits
#> $splits$split1
#> $splits$split1$analysis
#>  [1]  3  4  5  6  7  8 13 14 15 16 17 18 19 20 21 22 23 24
#> 
#> $splits$split1$assessment
#> [1]  1  2  9 10 11 12
#> 
#> $splits$split1$iter
#> [1] 1
#> 
#> $splits$split1$outer_fold
#> [1] 1
#> 
#> $splits$split1$inner
#> $splits$split1$inner$split1
#> $splits$split1$inner$split1$analysis
#>  [1]  3  4  5  6  7  8 19 20 21 22 23 24
#> 
#> $splits$split1$inner$split1$assessment
#> [1] 13 14 15 16 17 18
#> 
#> $splits$split1$inner$split1$iter
#> [1] 1
#> 
#> $splits$split1$inner$split1$outer_fold
#> [1] 1
#> 
#> $splits$split1$inner$split1$inner_fold
#> [1] 1
#> 
#> $splits$split1$inner$split1$seed
#> [1] 1424497935
#> 
#> 
#> $splits$split1$inner$split2
#> $splits$split1$inner$split2$analysis
#>  [1]  5  6 13 14 15 16 17 18 21 22 23 24
#> 
#> $splits$split1$inner$split2$assessment
#> [1]  3  4  7  8 19 20
#> 
#> $splits$split1$inner$split2$iter
#> [1] 1
#> 
#> $splits$split1$inner$split2$outer_fold
#> [1] 1
#> 
#> $splits$split1$inner$split2$inner_fold
#> [1] 2
#> 
#> $splits$split1$inner$split2$seed
#> [1] 1989879223
#> 
#> 
#> $splits$split1$inner$split3
#> $splits$split1$inner$split3$analysis
#>  [1]  3  4  7  8 13 14 15 16 17 18 19 20
#> 
#> $splits$split1$inner$split3$assessment
#> [1]  5  6 21 22 23 24
#> 
#> $splits$split1$inner$split3$iter
#> [1] 1
#> 
#> $splits$split1$inner$split3$outer_fold
#> [1] 1
#> 
#> $splits$split1$inner$split3$inner_fold
#> [1] 3
#> 
#> $splits$split1$inner$split3$seed
#> [1] 616885468
#> 
#> 
#> 
#> $splits$split1$seed
#> [1] 1034275838
#> 
#> 
#> $splits$split2
#> $splits$split2$analysis
#>  [1]  1  2  3  4  7  8  9 10 11 12 13 14 15 16 17 18 23 24
#> 
#> $splits$split2$assessment
#> [1]  5  6 19 20 21 22
#> 
#> $splits$split2$iter
#> [1] 1
#> 
#> $splits$split2$outer_fold
#> [1] 2
#> 
#> $splits$split2$inner
#> $splits$split2$inner$split1
#> $splits$split2$inner$split1$analysis
#>  [1]  1  2  3  4 11 12 13 14 15 16 17 18
#> 
#> $splits$split2$inner$split1$assessment
#> [1]  7  8  9 10 23 24
#> 
#> $splits$split2$inner$split1$iter
#> [1] 1
#> 
#> $splits$split2$inner$split1$outer_fold
#> [1] 2
#> 
#> $splits$split2$inner$split1$inner_fold
#> [1] 1
#> 
#> $splits$split2$inner$split1$seed
#> [1] 1495967979
#> 
#> 
#> $splits$split2$inner$split2
#> $splits$split2$inner$split2$analysis
#>  [1]  1  2  7  8  9 10 13 14 17 18 23 24
#> 
#> $splits$split2$inner$split2$assessment
#> [1]  3  4 11 12 15 16
#> 
#> $splits$split2$inner$split2$iter
#> [1] 1
#> 
#> $splits$split2$inner$split2$outer_fold
#> [1] 2
#> 
#> $splits$split2$inner$split2$inner_fold
#> [1] 2
#> 
#> $splits$split2$inner$split2$seed
#> [1] 1339054593
#> 
#> 
#> $splits$split2$inner$split3
#> $splits$split2$inner$split3$analysis
#>  [1]  3  4  7  8  9 10 11 12 15 16 23 24
#> 
#> $splits$split2$inner$split3$assessment
#> [1]  1  2 13 14 17 18
#> 
#> $splits$split2$inner$split3$iter
#> [1] 1
#> 
#> $splits$split2$inner$split3$outer_fold
#> [1] 2
#> 
#> $splits$split2$inner$split3$inner_fold
#> [1] 3
#> 
#> $splits$split2$inner$split3$seed
#> [1] 635021485
#> 
#> 
#> 
#> $splits$split2$seed
#> [1] 30222245
#> 
#> 
#> $splits$split3
#> $splits$split3$analysis
#>  [1]  1  2  3  4  5  6  7  8  9 10 11 12 15 16 19 20 21 22
#> 
#> $splits$split3$assessment
#> [1] 13 14 17 18 23 24
#> 
#> $splits$split3$iter
#> [1] 1
#> 
#> $splits$split3$outer_fold
#> [1] 3
#> 
#> $splits$split3$inner
#> $splits$split3$inner$split1
#> $splits$split3$inner$split1$analysis
#>  [1]  1  2  5  6  7  8 15 16 19 20 21 22
#> 
#> $splits$split3$inner$split1$assessment
#> [1]  3  4  9 10 11 12
#> 
#> $splits$split3$inner$split1$iter
#> [1] 1
#> 
#> $splits$split3$inner$split1$outer_fold
#> [1] 3
#> 
#> $splits$split3$inner$split1$inner_fold
#> [1] 1
#> 
#> $splits$split3$inner$split1$seed
#> [1] 1708781975
#> 
#> 
#> $splits$split3$inner$split2
#> $splits$split3$inner$split2$analysis
#>  [1]  3  4  9 10 11 12 15 16 19 20 21 22
#> 
#> $splits$split3$inner$split2$assessment
#> [1] 1 2 5 6 7 8
#> 
#> $splits$split3$inner$split2$iter
#> [1] 1
#> 
#> $splits$split3$inner$split2$outer_fold
#> [1] 3
#> 
#> $splits$split3$inner$split2$inner_fold
#> [1] 2
#> 
#> $splits$split3$inner$split2$seed
#> [1] 960716496
#> 
#> 
#> $splits$split3$inner$split3
#> $splits$split3$inner$split3$analysis
#>  [1]  1  2  3  4  5  6  7  8  9 10 11 12
#> 
#> $splits$split3$inner$split3$assessment
#> [1] 15 16 19 20 21 22
#> 
#> $splits$split3$inner$split3$iter
#> [1] 1
#> 
#> $splits$split3$inner$split3$outer_fold
#> [1] 3
#> 
#> $splits$split3$inner$split3$inner_fold
#> [1] 3
#> 
#> $splits$split3$inner$split3$seed
#> [1] 1206911707
#> 
#> 
#> 
#> $splits$split3$seed
#> [1] 794883155
#> 
#> 
#> $splits$split4
#> $splits$split4$analysis
#>  [1]  1  2  5  6  9 10 11 12 13 14 17 18 19 20 21 22 23 24
#> 
#> $splits$split4$assessment
#> [1]  3  4  7  8 15 16
#> 
#> $splits$split4$iter
#> [1] 1
#> 
#> $splits$split4$outer_fold
#> [1] 4
#> 
#> $splits$split4$inner
#> $splits$split4$inner$split1
#> $splits$split4$inner$split1$analysis
#>  [1]  1  2  5  6  9 10 13 14 17 18 23 24
#> 
#> $splits$split4$inner$split1$assessment
#> [1] 11 12 19 20 21 22
#> 
#> $splits$split4$inner$split1$iter
#> [1] 1
#> 
#> $splits$split4$inner$split1$outer_fold
#> [1] 4
#> 
#> $splits$split4$inner$split1$inner_fold
#> [1] 1
#> 
#> $splits$split4$inner$split1$seed
#> [1] 1535033190
#> 
#> 
#> $splits$split4$inner$split2
#> $splits$split4$inner$split2$analysis
#>  [1]  5  6 11 12 13 14 17 18 19 20 21 22
#> 
#> $splits$split4$inner$split2$assessment
#> [1]  1  2  9 10 23 24
#> 
#> $splits$split4$inner$split2$iter
#> [1] 1
#> 
#> $splits$split4$inner$split2$outer_fold
#> [1] 4
#> 
#> $splits$split4$inner$split2$inner_fold
#> [1] 2
#> 
#> $splits$split4$inner$split2$seed
#> [1] 1258075316
#> 
#> 
#> $splits$split4$inner$split3
#> $splits$split4$inner$split3$analysis
#>  [1]  1  2  9 10 11 12 19 20 21 22 23 24
#> 
#> $splits$split4$inner$split3$assessment
#> [1]  5  6 13 14 17 18
#> 
#> $splits$split4$inner$split3$iter
#> [1] 1
#> 
#> $splits$split4$inner$split3$outer_fold
#> [1] 4
#> 
#> $splits$split4$inner$split3$inner_fold
#> [1] 3
#> 
#> $splits$split4$inner$split3$seed
#> [1] 2043641506
#> 
#> 
#> 
#> $splits$split4$seed
#> [1] 1891171943
#> 
#> 
#> 
#> attr(,"class")
#> [1] "nested_cv_resamples" "resamples"