Strategies for Processing Large Datasets

February 25, 2024 · 5 min read

If you have any questions, feel free to comment below. Click the block can copy the code.
And if you think it's helpful to you, just click on the ads which can support this site. Thanks!

In real-world problems, data analysts may face datasets with hundreds of thousands of records and hundreds of variables. Processing such large datasets requires a relatively large amount of computer memory, so use a 64-bit operating system and a device with relatively large memory whenever possible. Otherwise, data analysis may take too long or may even be impossible to carry out. In addition, effective strategies for processing data can greatly improve analysis efficiency.

1. Cleaning the Workspace #

To obtain as much memory space as possible during data analysis, it is recommended to clean the workspace first when starting any new analysis project.

# rm(list = ls(all = TRUE))

The function ls( ) is used to display objects in the current workspace. The parameter all defaults to FALSE; it is set to TRUE here to clear all objects, including hidden objects.

In addition, during data analysis, use the command rm(object1,object2, …) to promptly clear temporary objects and objects that are no longer needed.

2. Quickly Reading .csv Files #

.csv files take up little space and can be viewed and generated by Excel, so they are widely used to store data. The function read.csv( ) introduced earlier can read .csv files very conveniently. However, for large datasets, this function reads data too slowly and sometimes even reports errors. At this point, you can use the read_csv( ) function in the readr package or the fread( )^[1] function in the data.table package to read in data, with the latter reading faster (approximately twice as fast as the former).

The data.table package provides an advanced version of a data frame, greatly improving the speed of data processing. This package is particularly suitable for users who need to process large datasets (such as 1GB–100GB) in memory. However, the way this package operates differs considerably from other packages in R and requires a certain amount of time to learn.

3. Simulating a Large Dataset #

For ease of explanation, a large dataset containing 50,000 records and 200 variables is simulated below.

bigdata <- as.data.frame(matrix(rnorm(50000 * 200), ncol = 200))
# Two nested for loops and R's built-in constant letters (lowercase English letters) are used to name the 200 variables.
varnames <- NULL
# The outer loop constructs the first character of the variable names (a–t)
for (i in letters[1:20]) {
# The inner loop connects the numbers 1–10 to these letters respectively, using `_` as the separator.
  for (j in 1:10) {
  # The function paste( ) is used to concatenate strings.
    varnames <- c(varnames, paste(i, j, sep = "_"))
  }
}
names(bigdata) <- varnames
names(bigdata)

If you do not particularly want to use multiple loops^[2], you can consider^[3]:

# Unfortunately, apply here will cause extra spaces
# apply(expand.grid(1:20, letters[1:20]), 1, function(x) paste(x[2], x[1], sep="_")) 
# sprintf("%s_%s", expand.grid(1:10,letters[1:20])[,2],expand.grid(1:10,letters[1:20])[,1])

# Or
# as.vector(t(outer(letters[1:20], 1:10, paste, sep="_")))

4. Removing Unneeded Variables #

Before conducting a formal analysis, we need to remove variables that are temporarily not needed to reduce the burden on memory. The select family of functions in the dplyr package can be useful here, especially when these functions are used together with functions such as starts_with( ), ends_with( ), and contains( ) from the tidyselect package, which provides many conveniences.

First load these two packages:

library(dplyr)
library(tidyselect)

Next, examples illustrate how to use the select family of functions to select or remove variables.

subdata1 <- select(bigdata, starts_with("a"))
names(subdata1)
# 'a_1''a_2''a_3''a_4''a_5''a_6''a_7''a_8''a_9''a_10'
subdata2 <- select(bigdata, ends_with("2"))
names(subdata2)
#'a_2''b_2''c_2''d_2''e_2''f_2''g_2''h_2''i_2''j_2''k_2''l_2''m_2''n_2''o_2''p_2''q_2''r_2''s_2''t_2'

The functions starts_with( ) and ends_with( ) represent the prefix and suffix of variables, respectively. In the commands above, subdata1 selects all variables in the dataset that begin with a, while subdata2 selects all variables in the dataset that end with 2.

To select all variables that begin with a or b, you can use the following command:

# subdata3 <- select(bigdata, c(starts_with("a"), starts_with("b")))
subdata3 <- select_at(bigdata, vars(starts_with("a"), starts_with("b"))) # Note that the syntax is slightly different from select
names(subdata3)

To select all variables whose variable names contain certain characters, you can use the function contains( ). For example, to select all variables containing the character 1, you can enter the following command:

# subdata4 <- select(bigdata, c(contains("1")))
subdata4 <- select_at(bigdata, vars(contains("1")))
names(subdata4)

It should be noted that all variables ending in 10 also contain the character 1.

To remove certain variables, simply add a - sign before the functions starts_with( ), ends_with( ), and contains( ). For example, to remove variables ending in 1 or 5, you can use the following command:

# subdata5 <- select(bigdata, c(-contains("1"), -contains("5")))
subdata5 <- select_at(bigdata, vars(-contains("1"), -contains("5")))
names(subdata5)

5. Selecting a Random Sample from the Dataset #

Processing all records in a large dataset often reduces analysis efficiency. When writing code, you can extract only a portion of the records to test the program in order to optimize the code and eliminate bugs.

# The parameter size is used to specify the number of rows
sampledata1 <- sample_n(subdata5, size = 500)
nrow(sampledata1)
# The parameter size is used to specify the proportion of all rows.
sampledata2 <- sample_frac(subdata5, size = 0.02)
nrow(sampledata2)
# 500
# 1000

The functions sample_n( ) and sample_frac( ) are both used to randomly select a specified number of rows from a data frame. The parameter size in the former is used to specify the number of rows, while the parameter size in the latter is used to specify the proportion of all rows.

It should be explained that the strategies for processing large datasets discussed above apply only to processing GB-scale datasets. Regardless of which tool is used, processing TB- and PB-scale datasets is a challenge. Several packages in R can be used to process TB-scale datasets, such as RHIPE, RHadoop, and RevoScaleR. These packages have relatively steep learning curves and require some understanding of high-performance computing. If you have a need, you can explore them on your own; they are not introduced here.

sample_n() and sample_frac() are about to be retired. The package documentation recommends using slice_sample( ) instead. Its usage can be viewed here^[4].

# Process using slice_sample( )
sampledata1 <- slice_sample(subdata5, n = 500)
nrow(sampledata1)
sampledata2 <- slice_sample(subdata5, prop = 0.02)
nrow(sampledata2)