The rbind.pages function is used to combine a list of data frames into a single
data frame. This is often needed when working with a JSON API that limits the amount
of data per request. If we need more data than what fits in a single request, we need to
perform multiple requests that each retrieve a fragment of data, not unlike pages in a
book. In practice this is often implemented using a page parameter in the API. The
rbind.pages function can be used to combine these pages back into a single dataset.
Usage
rbind.pages(pages)
Arguments
pages
a list of data frames, each representing a page of data
Details
The rbind.pages function generalizes base::rbind and
plyr::rbind.fill with added support for nested data frames. Not each column
has to be present in each of the individual data frames; missing columns will be filled
up in NA values.
Examples
# Basic example
x <- data.frame(foo = rnorm(3), bar = c(TRUE, FALSE, TRUE))
y <- data.frame(foo = rnorm(2), col = c("blue", "red"))
rbind.pages(list(x, y))
## Not run:
baseurl <- "http://projects.propublica.org/nonprofits/api/v1/search.json"
pages <- list()
for(i in 0:20){
mydata <- fromJSON(paste0(baseurl, "?order=revenue&sort_order=desc&page=", i))
message("Retrieving page ", i)
pages[[i+1]] <- mydata$filings
}
filings <- rbind.pages(pages)
nrow(filings)
colnames(filings)
## End(Not run)