computeTf(channel, tableName, docId, textColumns, parser,
weighting = "normal", top = NULL, rankFunction = "rank", where = NULL,
idSep = "-", idNull = "(null)", stopwords = NULL, test = FALSE)
Arguments
channel
connection object as returned by odbcConnect
tableName
Aster table name
docId
vector with one or more column names comprising unique document id.
Values are concatenated with idSep. Database NULLs are replaced with
idNull string.
textColumns
one or more names of columns with text. Multiple coumn are
concatenated into single text field first.
parser
type of parser to use on text. For example, ngram(2) parser
generates 2-grams (ngrams of length 2), token(2) parser generates 2-word
combinations of terms within documents.
weighting
term frequency formula to compute the tf value. One of following:
'raw', 'bool', 'binary', 'log', 'augment', and
'normal' (default).
top
specifies threshold to cut off terms ranked below top value. If value
is greater than 0 then included top ranking terms only, otherwise all terms returned
(also see paramter rankFunction). Terms are always ordered by their term frequency (tf)
within each document. Filtered out terms have their rank ariphmetically greater than
threshold top (see details): term is more important the smaller value of its rank.
rankFunction
one of rownumber, rank, denserank, percentrank. Rank computed and
returned for each term within each document. function determines which SQL window function computes
term rank value (default rank corresponds to SQL RANK() window function).
When threshold top is greater than 0 ranking function used to limit number of
terms returned (see details).
where
specifies criteria to satisfy by the table rows before applying
computation. The criteria are expressed in the form of SQL predicates (inside
WHERE clause).
idSep
separator when concatenating 2 or more document id columns (see docId).
idNull
string to replace NULL value in document id columns.
stopwords
character vector with stop words. Removing stop words takes place in R after
results are computed and returned from Aster.
test
logical: if TRUE show what would be done, only (similar to parameter test in RODBC
functions sqlQuery and sqlSave).
Details
By default function computes and returns all terms. When large number of terms is expected then
use parameters top to limit number of terms returned by
filtering top ranked terms for each document. Thus if set top=1000 and there
is 100 documents then at least 100,000 terms (rows) will be returned. Result size could
exceed this number when other than rownumberrankFunction used:
rownumber applies a sequential row number, starting at 1, to each term in a document.
The tie-breaker behavior is as follows: Rows that compare as equal in the sort order will be
sorted arbitrarily within the scope of the tie, and all terms will be given unique row numbers.
rank function assigns the current row-count number as the terms's rank, provided the
term does not sort as equal (tie) with another term. The tie-breaker behavior is as follows:
terms that compare as equal in the sort order are sorted arbitrarily within the scope of the tie,
and the sorted-as-equal terms get the same rank number.
denserank behaves like the rank function, except that it never places
gaps in the rank sequence. The tie-breaker behavior is the same as that of RANK(), in that
the sorted-as-equal terms receive the same rank. With denserank, however, the next term after
the set of equally ranked terms gets a rank 1 higher than preceding tied terms.
percentrank assigns a relative rank to each term, using the formula:
(rank - 1) / (total rows - 1). The tie-breaker behavior is as follows: Terms that compare
as equal are sorted arbitrarily within the scope of the tie, and the sorted-as-equal rows
get the same percent rank number.
The ordering of the rows is always by their tf value within each document.
See Also
computeTfIdf, nGram, token
Examples
if(interactive()){
# initialize connection to Dallas database in Aster
conn = odbcDriverConnect(connection="driver={Aster ODBC Driver};
server=<dbhost>;port=2406;database=<dbname>;uid=<user>;pwd=<pw>")
# compute term-document-matrix of all 2-word Ngrams of Dallas police open crime reports
tdm1 = computeTf(channel=conn, tableName="public.dallaspoliceall", docId="offensestatus",
textColumns=c("offensedescription", "offensenarrative"),
parser=nGram(2),
where="offensestatus NOT IN ('System.Xml.XmlElement', 'C')")
# compute term-document-matrix of all 2-word combinations of Dallas police crime reports
# by time of day (4 documents corresponding to 4 parts of day)
tdm2 = computeTf(channel=conn, tableName="public.dallaspoliceall",
docId="(extract('hour' from offensestarttime)/6)::int%4",
textColumns=c("offensedescription", "offensenarrative"),
parser=token(2, punctuation="[-.,?\!:;~()]+", stopWords=TRUE),
where="offensenarrative IS NOT NULL")
# include only top 100 ranked 2-word ngrams for each offense status
# into resulting term-document-matrix using dense rank function
tdm3 = computeTf(channel=NULL, tableName="public.dallaspoliceall", docId="offensestatus",
textColumns=c("offensedescription", "offensenarrative"),
parser=nGram(2), top=100, rankFunction="denserank",
where="offensestatus NOT IN ('System.Xml.XmlElement', 'C')")
}
providing 
.