Read large files by chunks — read

Have you ever needed to read a file larger than RAM? Have you ever needed only a subset of the data within that large file? I have just the function for you! Meet bkmisc::read_chunk(), it uses data.table::fread() as the backend file reader and allows you to pass a function to process each data chunk as it's being read in.

Usage

read_chunk(
  file,
  chunk_size,
  n_lines = NULL,
  fun = NULL,
  ...,
  sep = "auto",
  header = "auto",
  select = NULL,
  col_names = NULL,
  col_classes = NULL,
  na_strings = c(""),
  data.table = FALSE,
  message = TRUE
)

Arguments

file: String for file path.
chunk_size: Integer for size of chunk (number of rows).
n_lines: Integer for total number of lines to read. Usually the number of lines in the file; as long as the entire file, after processing, can fit into memory.
fun: A function that will be applied to each chunk as they're being read in. The first argument will be passed the chunk data.frame.
...: Other arguments passed to fun.
sep: String for separator between columns. See ?data.table::fread.
header: TRUE or FALSE. Does the first data line contain column names? See ?data.table::fread.
select: A character vector of column names or numeric vector of column positions to keep, drops the rest. See ?data.table::fread.
col_names: A character vector used to name the columns of the data. See ?data.table::fread argument col.names.
col_classes: A character vector (named [specific columns] or unnamed [all columns]) of column classes that correspond to columns in the data. See ?data.table::fread argument colClasses.
na_strings: A character vector of strings which will be used as NA values. See ?data.table::fread argument na.strings.
data.table: TRUE or FALSE. Return an object of class data.table or not.
message: TRUE or FALSE. If TRUE, will print timing and chunk position as read is progressing.

Value

data.frame

Details

Warning: If the file has column names, be sure to set header = TRUE. If no names, be sure to set header = FALSE. Otherwise, rows could be read in twice. Test on example below.

Examples


#----------------------------------------------------------------------------
# read_chunk() examples
#----------------------------------------------------------------------------
library(bkmisc)

# generate fake data
n_lines <- 100
df <- data.frame(var1 = seq_len(n_lines))
# save data to file
filepath <- tempfile(pattern = "read_chunk_example", fileext = ".csv")
write.csv(x = df, file = filepath, row.names = FALSE)
# read data
out <- read_chunk(
  file = filepath,
  chunk_size = 10,
  n_lines = n_lines,
  fun = function(x) {x[seq_len(nrow(x)) %% 2 == 0, , drop = FALSE]},
  header = TRUE
)
#> Reading chunk 1 of 10 - 2026-06-14 22:29:13.259838
#>   Processing chunk 1 of 10 - 2026-06-14 22:29:13.260807
#>   Finished chunk 1 of 10 - 2026-06-14 22:29:13.261291
#> Reading chunk 2 of 10 - 2026-06-14 22:29:13.261683
#>   Processing chunk 2 of 10 - 2026-06-14 22:29:13.262408
#>   Finished chunk 2 of 10 - 2026-06-14 22:29:13.262867
#> Reading chunk 3 of 10 - 2026-06-14 22:29:13.263242
#>   Processing chunk 3 of 10 - 2026-06-14 22:29:13.263954
#>   Finished chunk 3 of 10 - 2026-06-14 22:29:13.264399
#> Reading chunk 4 of 10 - 2026-06-14 22:29:13.26494
#>   Processing chunk 4 of 10 - 2026-06-14 22:29:13.265943
#>   Finished chunk 4 of 10 - 2026-06-14 22:29:13.26655
#> Reading chunk 5 of 10 - 2026-06-14 22:29:13.267078
#>   Processing chunk 5 of 10 - 2026-06-14 22:29:13.268078
#>   Finished chunk 5 of 10 - 2026-06-14 22:29:13.268691
#> Reading chunk 6 of 10 - 2026-06-14 22:29:13.269232
#>   Processing chunk 6 of 10 - 2026-06-14 22:29:13.27024
#>   Finished chunk 6 of 10 - 2026-06-14 22:29:13.270876
#> Reading chunk 7 of 10 - 2026-06-14 22:29:13.271447
#>   Processing chunk 7 of 10 - 2026-06-14 22:29:13.272209
#>   Finished chunk 7 of 10 - 2026-06-14 22:29:13.272661
#> Reading chunk 8 of 10 - 2026-06-14 22:29:13.273056
#>   Processing chunk 8 of 10 - 2026-06-14 22:29:13.273795
#>   Finished chunk 8 of 10 - 2026-06-14 22:29:13.274253
#> Reading chunk 9 of 10 - 2026-06-14 22:29:13.274644
#>   Processing chunk 9 of 10 - 2026-06-14 22:29:13.275381
#>   Finished chunk 9 of 10 - 2026-06-14 22:29:13.275844
#> Reading chunk 10 of 10 - 2026-06-14 22:29:13.276227
#>   Processing chunk 10 of 10 - 2026-06-14 22:29:13.276969
#>   Finished chunk 10 of 10 - 2026-06-14 22:29:13.277426
#> Binding rows into a single dataframe - 2026-06-14 22:29:13.277811
#> Finished - 2026-06-14 22:29:13.278447
out
#>    var1
#> 1     2
#> 2     4
#> 3     6
#> 4     8
#> 5    10
#> 6    12
#> 7    14
#> 8    16
#> 9    18
#> 10   20
#> 11   22
#> 12   24
#> 13   26
#> 14   28
#> 15   30
#> 16   32
#> 17   34
#> 18   36
#> 19   38
#> 20   40
#> 21   42
#> 22   44
#> 23   46
#> 24   48
#> 25   50
#> 26   52
#> 27   54
#> 28   56
#> 29   58
#> 30   60
#> 31   62
#> 32   64
#> 33   66
#> 34   68
#> 35   70
#> 36   72
#> 37   74
#> 38   76
#> 39   78
#> 40   80
#> 41   82
#> 42   84
#> 43   86
#> 44   88
#> 45   90
#> 46   92
#> 47   94
#> 48   96
#> 49   98
#> 50  100