The stream_in and stream_out functions implement line-by-line processing
of JSON data over a connection, such as a socket, url, file or pipe. JSON
streaming requires the ndjson format, which slightly differs
from fromJSON and toJSON, see details.
a connection object. If the connection is not open,
stream_in and stream_out will automatically open
and later close (and destroy) the connection. See details.
handler
a custom function that is called on each page of JSON data. If not specified,
the default handler stores all pages and binds them into a single data frame that will be
returned by stream_in. See details.
pagesize
number of lines to read/write from/to the connection per iteration.
verbose
print some information on what is going on.
...
arguments for fromJSON and toJSON that
control JSON formatting/parsing where applicable. Use with caution.
x
object to be streamed out. Currently only data frames are supported.
Details
Because parsing huge JSON strings is difficult and inefficient, JSON streaming is done
using lines of minified JSON records, a.k.a. ndjson.
This is pretty standard: JSON databases such as dat
or MongoDB use the same format to import/export datasets. Note that this means that the
total stream combined is not valid JSON itself; only the individual lines are. Also note
that because line-breaks are used as separators, prettified JSON is not permitted: the
JSON lines must be minified. In this respect, the format is a bit different from
fromJSON and toJSON where all lines are part of a single JSON
structure with optional line breaks.
The handler is a callback function which is called for each page (batch) of
JSON data with exactly one argument (usually a data frame with pagesize rows).
If handler is missing or NULL, a default handler is used which stores all
intermediate pages of data, and at the very end binds all pages together into one single
data frame that is returned by stream_in. When a custom handler function
is specified, stream_in does not store any intermediate results and always returns
NULL. It is then up to the handler to process or store data pages.
A handler function that does not store intermediate results in memory (for
example by writing output to another connection) results in a pipeline that can process an
unlimited amount of data. See example.
If a connection is not opened yet, stream_in and stream_out
will automatically open and later close the connection. Because R destroys connections
when they are closed, they cannot be reused. To use a single connection for multiple
calls to stream_in or stream_out, it needs to be opened
beforehand. See example.
Value
The stream_out function always returns NULL.
When no custom handler is specified, stream_in returns a data frame of all pages binded together.
When a custom handler function is specified, stream_in always returns NULL.