ff

Package: ff
Version: 2.2-13
Date: 2012-03-29
Title: memory-efficient storage of large data on disk and fast access
functions
Author: Daniel Adler <dadler@uni-goettingen.de>,
Christian Gläser <christian_glaeser@gmx.de>,
Oleg Nenadic <onenadi@uni-goettingen.de>,
Jens Oehlschlägel <Jens.Oehlschlaegel@truecluster.com>,
Walter Zucchini <wzucchi@uni-goettingen.de>
Maintainer: Jens Oehlschlägel <Jens.Oehlschlaegel@truecluster.com>
Depends: R (>= 2.10.1), bit (>= 1.1-12), utils
Suggests: biglm
Description: The ff package provides data structures that are stored on
disk but behave (almost) as if they were in RAM by transparently
mapping only a section (pagesize) in main memory - the effective
virtual memory consumption per ff object. ff supports R's standard
atomic data types 'double', 'logical', 'raw' and 'integer' and
non-standard atomic types boolean (1 bit), quad (2 bit unsigned),
nibble (4 bit unsigned), byte (1 byte signed with NAs), ubyte (1 byte
unsigned), short (2 byte signed with NAs), ushort (2 byte unsigned),
single (4 byte float with NAs). For example 'quad' allows efficient
storage of genomic data as an 'A','T','G','C' factor. The unsigned
types support 'circular' arithmetic. There is also support for
close-to-atomic types 'factor', 'ordered', 'POSIXct', 'Date' and
custom close-to-atomic types.
ff not only has native C-support for vectors, matrices and arrays
with flexible dimorder (major column-order, major row-order and
generalizations for arrays). There is also a ffdf class not unlike
data.frames and import/export filters for csv files.
ff objects store raw data in binary flat files in native encoding,
and complement this with metadata stored in R as physical and virtual
attributes. ff objects have well-defined hybrid copying semantics,
which gives rise to certain performance improvements through
virtualization. ff objects can be stored and reopened across R
sessions. ff files can be shared by multiple ff R objects
(using different data en/de-coding schemes) in the same process
or from multiple R processes to exploit parallelism. A wide choice of
finalizer options allows to work with 'permanent' files as well as
creating/removing 'temporary' ff files completely transparent to the
user. On certain OS/Filesystem combinations, creating the ff files
works without notable delay thanks to using sparse file allocation.
Several access optimization techniques such as Hybrid Index
Preprocessing and Virtualization are implemented to achieve good
performance even with large datasets, for example virtual matrix
transpose without touching a single byte on disk. Further, to reduce
disk I/O, 'logicals' and non-standard data types get stored native and
compact on binary flat files i.e. logicals take up exactly 2 bits to
represent TRUE, FALSE and NA.
Beyond basic access functions, the ff package also provides
compatibility functions that facilitate writing code for ff and ram
objects and support for batch processing on ff objects (e.g. as.ram,
as.ff, ffapply). ff interfaces closely with functionality from package
'bit': chunked looping, fast bit operations and coercions between
different objects that can store subscript information ('bit',
'bitwhich', ff 'boolean', ri range index, hi hybrid index). This allows
to work interactively with selections of large datasets and quickly
modify selection criteria.
Further high-performance enhancements can be made available upon request.
License: GPL-2 | file LICENSE
LazyLoad: yes
ByteCompile: yes
Encoding: latin1
URL: http://ff.r-forge.r-project.org/
Packaged: 2014-04-07 21:16:45 UTC; root
NeedsCompilation: yes
Repository: CRAN
Date/Publication: 2014-04-09 09:54:20

Install log

* installing to library '/home/ddbj/local/lib64/R/library'
* installing *source* package 'ff' ...
** package 'ff' successfully unpacked and MD5 sums checked
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking sys/vfs.h usability... yes
checking sys/vfs.h presence... yes
checking for sys/vfs.h... yes
checking sys/mman.h usability... yes
checking sys/mman.h presence... yes
checking for sys/mman.h... yes
checking sys/file.h usability... yes
checking sys/file.h presence... yes
checking for sys/file.h... yes
checking for sys/stat.h... (cached) yes
checking for unistd.h... (cached) yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking sys/param.h usability... yes
checking sys/param.h presence... yes
checking for sys/param.h... yes
checking sys/mount.h usability... yes
checking sys/mount.h presence... yes
checking for sys/mount.h... yes
checking for struct statfs.f_iosize... no
checking sys/statfs.h usability... yes
checking sys/statfs.h presence... yes
checking for sys/statfs.h... yes
checking for struct statfs.f_iosize... (cached) no
checking sys/statvfs.h usability... yes
checking sys/statvfs.h presence... yes
checking for sys/statvfs.h... yes
checking for off_t... yes
checking for size_t... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for _LARGEFILE_SOURCE value needed for large files... no
checking for fseeko... yes
configure: creating ./config.status
config.status: creating src/ac_config.h
config.status: src/ac_config.h is unchanged
** libs
g++ -I/home/ddbj/local/lib64/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g -O2  -c Error.cpp -o Error.o
g++ -I/home/ddbj/local/lib64/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g -O2  -c FSInfo_statfs.cpp -o FSInfo_statfs.o
g++ -I/home/ddbj/local/lib64/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g -O2  -c FSInfo_win32.cpp -o FSInfo_win32.o
g++ -I/home/ddbj/local/lib64/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g -O2  -c MMapFileMapping.cpp -o MMapFileMapping.o
g++ -I/home/ddbj/local/lib64/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g -O2  -c Win32FileMapping.cpp -o Win32FileMapping.o
g++ -I/home/ddbj/local/lib64/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g -O2  -c ff.cpp -o ff.o
gcc -I/home/ddbj/local/lib64/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g -O2  -c ordermerge.c -o ordermerge.o
gcc -I/home/ddbj/local/lib64/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g -O2  -c r_ff.c -o r_ff.o
g++ -I/home/ddbj/local/lib64/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g -O2  -c r_file_resize.cpp -o r_file_resize.o
g++ -I/home/ddbj/local/lib64/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g -O2  -c utk_file_allocate_fseek.cpp -o utk_file_allocate_fseek.o
g++ -I/home/ddbj/local/lib64/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g -O2  -c utk_file_resize.cpp -o utk_file_resize.o
g++ -shared -L/home/ddbj/local/lib64/R/lib -L/usr/local/lib64 -o ff.so Error.o FSInfo_statfs.o FSInfo_win32.o MMapFileMapping.o Win32FileMapping.o ff.o ordermerge.o r_ff.o r_file_resize.o utk_file_allocate_fseek.o utk_file_resize.o -L/home/ddbj/local/lib64/R/lib -lR
installing to /home/ddbj/local/lib64/R/library/ff/libs
** R
** exec
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package 'ff'
    finding HTML links ... done
    CFUN                                    html  
    finding level-2 HTML links ... done

    Extract.ff                              html  
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/Extract.ff.rd:94: missing file link 'dimnames.ff'
    Extract.ffdf                            html  
    Forbidden_ffdf                          html  
    Internal_ffdf                           html  
    LimWarn                                 html  
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/LimWarn.rd:25: missing file link '[<-.ff'
    add                                     html  
    array2vector                            html  
    arrayIndex2vectorIndex                  html  
    as.ff.bit                               html  
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/as.ff.bit.rd:9: missing file link 'bit'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/as.ff.bit.rd:36: missing file link 'bit'
    as.ff                                   html  
    as.ffdf                                 html  
    as.hi                                   html  
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/as.hi.rd:79: missing file link 'poslength.hi'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/as.hi.rd:80: missing file link 'maxindex.hi'
    as.integer.hi                           html  
    as.vmode                                html  
    bigsample                               html  
    chunk.bit                               html  
    chunk.ffdf                              html  
    clone.ffdf                              html  
    clone                                   html  
    close.ff                                html  
    delete                                  html  
    dim.ff                                  html  
    dimnames.ff_array                       html  
    dimnames.ffdf                           html  
    dimorderCompatible                      html  
    dummy.dimnames                          html  
    ff                                      html  
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:78: missing file link 'delete.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:91: missing file link 'delete.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:97: missing file link 'delete.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:161: missing file link 'str.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:166: missing file link 'bit'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:184: missing file link 't.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:187: missing file link 'dimnames.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:198: missing file link '[[.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:199: missing file link '[[<-.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:204: missing file link '[.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:205: missing file link '[<-.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:206: missing file link 'swap.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:207: missing file link 'add.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:208: missing file link 'bigsample.ff'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ff.rd:213: missing file link 'delete.ff'
    ffapply                                 html  
    ffconform                               html  
    ffdf                                    html  
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/ffdf.rd:61: missing file link 'I'
    ffdfindexget                            html  
    ffdfsort                                html  
    ffdrop                                  html  
    ffindexget                              html  
    ffindexorder                            html  
    ffinfo                                  html  
    ffload                                  html  
    fforder                                 html  
    ffreturn                                html  
    ffsave                                  html  
    ffsort                                  html  
    ffsuitable                              html  
    ffxtensions                             html  
    file.resize                             html  
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/file.resize.rd:43: missing file link 'file.create'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/file.resize.rd:43: missing file link 'file.rename'
    filename                                html  
    finalize                                html  
    finalizer                               html  
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/finalizer.rd:27: missing file link 'delete.ff'
    fixdiag                                 html  
    geterror.ff                             html  
    getpagesize                             html  
    getset.ff                               html  
    hi                                      html  
    hiparse                                 html  
    is.ff                                   html  
    is.ffdf                                 html  
    is.open                                 html  
    is.readonly                             html  
    is.sorted                               html  
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/is.sorted.rd:19: missing file link 'intisasc'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/is.sorted.rd:31: missing file link 'intisasc'
    length.ff                               html  
    length.ffdf                             html  
    length.hi                               html  
    levels.ff                               html  
    matcomb                                 html  
    matprint                                html  
    maxffmode                               html  
    maxlength                               html  
    mismatch                                html  
    na.count                                html  
    names.ff                                html  
    nrowAssign                              html  
    open.ff                                 html  
    pagesize                                html  
    physical.ff                             html  
    physical.ffdf                           html  
    print.ff                                html  
    ram2ffcode                              html  
    ramattribs                              html  
    ramorder.default                        html  
    ramsort.default                         html  
    read.table.ffdf                         html  
    readwrite.ff                            html  
    regtest.fforder                         html  
    repnam                                  html  
    sortLevels                              html  
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/sortLevels.rd:39: missing file link 'rbind.data.frame'
    splitPathFile                           html  
    swap                                    html  
    symmIndex2vectorIndex                   html  
    symmetric                               html  
    unclass_-                               html  
    undim                                   html  
    unsort                                  html  
    update.ff                               html  
    vecprint                                html  
    vector.vmode                            html  
    vector2array                            html  
    vectorIndex2arrayIndex                  html  
    vmode.ffdf                              html  
    vmode                                   html  
    vt                                      html  
    vw                                      html  
    write.table.ffdf                        html  
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/write.table.ffdf.rd:82: missing file link 'write.csv'
Rd warning: /tmp/RtmpZExsyN/R.INSTALL46cae9e2b0b/ff/man/write.table.ffdf.rd:82: missing file link 'write.csv2'
** building package indices
** testing if installed package can be loaded
* DONE (ff)
Making 'packages.html' ... done