A formula that specifies outcome and treatment variables.
model.main
An optional formula that specifies pre-treatment covariates to be adjusted.
model.int
A formula specifying pre-treatment covariates to be interacted with treatment assignments when treat.type="single".
data
An optional data frame, list or environment (or object
coercible by 'as.data.frame' to a data frame) containing the
variables in the model. If not found in 'data', the
variables are taken from 'environment(formula)', typically
the environment from which 'FindIt' is called.
type
"binary" for a binary outcome variable, which needs to be integer class; "continuous" for a continuous outcome variable.
treat.type
"single" for interactions between a single treatment variable, which needs to be integer class, and multiple pre-treatment covariates specified with model.int; "multiple" is used when treatment-treatment interactions are of interest and treat is a matrix of multiple treatments.
nway
An argument passed to makeallway when treat.type="multiple". FindIt generates treatment-treatment interactions up to the order specified with this argument. In general, it is recommended to use the number of factorial treatments. The current version covers up to four way interactions.
search.lambdas
Whether to search for the tuning parameters for the LASSO constraints. If FALSE, lambdas must be supplied.
lambdas
Tuning parameters to be given to FindIt; only used if search.lambdas=FALSE.
make.twoway
If make.twoway=TRUE, all possible two-way interactions for the pre-treatment covariates specified in model.main and model.int are generated within FindIt. The default is set to be TRUE.
make.allway
If make.allway=TRUE, all possible treatment-treatment interactions for multiple treatments are generated when treat.type="multiple". Interactions of the order up to the value of nway is computed.
wts
An optional set of scaling weights. The default is 1.
scale.c
A set of weights for recaling the pre-treatment covariates; only used if make.twoway=FALSE. maketwoway is useful for generating these.
scale.int
A set of weights for recaling the covariates to be interacted with treatment variables ; only used if make.twoway=FALSE. maketwoway is useful for generating these.
fit.glmnet
Whether to fit using the coordinate descent method in glmnet (TRUE) or the regularization path method of LARS (FALSE).
make.reference
Whether to make a reference matrix to check which columns are dropped when makeallway=TRUE.
reference.main
If make.allway=FALSE and researchers generate a matrix of all possible interactions between factorial treatments, reference from makeallway function is better to be passed to FindIt through this argument.
threshold
An argument passed to makeallway when treat.type="multiple". Threshold to drop correlated columns when makeallway is used.
Details
Implements the alternating line search algorithm for estimating the tuning parameters, as described in Imai and Ratkovic (2013).
Value
coefs
A named vector of scaled coefficients
coefs.orig
A vector of coefficients on the original scale, if scale.c and scale.t was used
fit
Fitted values on an SVM scale
names.out
Names of the coefficients
y
A vector of observed outcomes
X.c
A matrix of pre-treatment covariates to be adjusted
X.t
A matrix of treatments and treatment-treatment interactions, or treatment-covariate interactions
GCV
GCV statistic at the minimum
ATE
When treat.type="single", the estimated ATE. When treat.type="multiple", the estimated treatment effect of each unique treatment combination
lambdas
Tuning parameters used for the fit
reference
When treat.type="multiple", after making all interaction terms, columns with no variation or columns perfectly correlated with one of other columns are automatically dropped. reference shows which columns are kept and dropped.
Author(s)
Naoki Egami, Marc Ratkovic and Kosuke Imai.
References
Imai, Kosuke and Marc Ratkovic. 2013. “Estimating Treatment Effect Heterogeneity in Randomized Program Evaluation.” Annals of Applied Statistics, Vol.7, No.1(March), pp. 443-470. http://imai.princeton.edu/research/files/svm.pdf
###################################################
## Example 1: Treatment-Covariate Interaction
###################################################
data(LaLonde)
## The model includes a treatment variable,
## nine covariates to be interacted with the treatment variable,
## and the same nine covariates to be adjusted.
## Not run:
## Run to find the LASSO parameters
F1 <-FindIt(model.treat= outcome ~ treat,
model.main= ~ age+educ+black+hisp+white+
marr+nodegr+log.re75+u75,
model.int= ~ age+educ+black+hisp+white+
marr+nodegr+log.re75+u75,
data = LaLonde,
type="binary",
treat.type="single")
## End(Not run)
## Fit with uncovered lambda parameters.
F1 <-FindIt(model.treat= outcome ~ treat,
model.main= ~ age+educ+black+hisp+white+
marr+nodegr+log.re75+u75,
model.int= ~ age+educ+black+hisp+white+
marr+nodegr+log.re75+u75,
data = LaLonde,
type="binary",
treat.type="single",
search.lambdas=FALSE,
lambdas = c(-3.8760,-4.0025) )
summary(F1)
## Returns all the estimated treatment effects.
pred1 <- predict(F1)
## Top10
head(pred1$data, n=10)
## Bottom 10
tail(pred1$data ,n=10)
## Visualize all the estimated treatment effects.
## Not run:
plot(pred1)
## End(Not run)
###################################################
## Example 2: Treatment-Treatment Interaction
###################################################
## Not run:
data(GerberGreen)
## The model includes four factorial treatments and
## all two, three, four-way interactions between them.
## Four pre-treatment covariates are adjusted.
## Run to search for lambdas.
F2<- FindIt(model.treat= voted98 ~ persngrp+phnscrpt+mailings+appeal,
nway=4,
model.main= ~ age+majorpty+vote96.1+vote96.0,
data = GerberGreen,
type="binary",
treat.type="multiple")
## Fit, given selected lambdas.
F2<- FindIt(model.treat= voted98 ~ persngrp+phnscrpt+mailings+appeal,
nway=4,
model.main= ~ age+majorpty+vote96.1+vote96.0,
data = GerberGreen,
type="binary",
treat.type="multiple",
search.lambdas=FALSE,
lambdas=c(-15.000,-6.237))
## Returns coefficient estimates.
summary(F2)
## Returns predicted values for unique treatment combinations.
pred2 <- predict(F2,unique=TRUE)
## Top 10
head(pred2$data, n=10)
## Bottom 10
tail(pred2$data, n=10)
## Visualize predicted values for each treatment combination.
plot(pred2)
## End(Not run)
###################################################
## Example 3: Conjoint Analysis. Causal Interaction.
###################################################
data(Immigration)
## Not run:
## The SVM classifier with a lasso constraint is estimated.
## The model includes all three-way and two-way interactions as
## well as main effects of five factorial treatments.
## Run to search for lambdas.
F.conjoint<- FindIt(model.treat= outcome ~
Education+Gender+Origin+Experience+Plans,
nway=3,
data = Immigration,
type="binary",
treat.type="multiple")
## Make the full factorial design matrix as the target population.
full <- full.FindIt(F.conjoint)
## Compute the predicted potential outcomes for the target population.
Unifdata <- predict(F.conjoint,newdata=full,sort=FALSE)$data
## End(Not run)
## load pre-computed F.conjoint and Unifdata
data(F.conjoint)
data(Unifdata)
## Compute AMTEs, ATCEs, and AMTIEs
## Range of each factor interaction.
## The range of the AMTEs for all factors
compare1 <- INT(F.conjoint,target.data=Unifdata,compare=TRUE,order=1)
compare1
## The range of the two-way AMTIEs for all two-way factor interactions
compare2 <- INT(F.conjoint,target.data=Unifdata,compare=TRUE,order=2)
compare2
## Not run:
## The range of the three-way AMTIEs for all three-way factor interactions
compare3 <- INT(F.conjoint,target.data=Unifdata,compare=TRUE,order=3)
compare3
## End(Not run)
## Compute AMTIEs within factor interactions.
## Origin x Experience
out.OP <- INT(F.conjoint,target.data=Unifdata,
column=c("Origin","Experience"),
base=c("India","No.job"), order=2)
out.OP
## Not run:
## Education x Gender x Origin
out.EGO <- INT(F.conjoint,target.data=Unifdata,
column=c("Education","Gender","Origin"),
base=c("No.formal","female","India"), order=3)
out.EGO
## End(Not run)
Results
R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> library(FindIt)
Loading required package: glmnet
Loading required package: Matrix
Loading required package: foreach
Loaded glmnet 2.0-5
Loading required package: lars
Loaded lars 1.2
> png(filename="/home/ddbj/snapshot/RGM3/R_CC/result/FindIt/FindIt.Rd_%03d_medium.png", width=480, height=480)
> ### Name: FindIt
> ### Title: FindIt for Estimating Heterogeneous Treatment Effects
> ### Aliases: FindIt summary.FindIt
>
> ### ** Examples
>
>
> ###################################################
> ## Example 1: Treatment-Covariate Interaction
> ###################################################
> data(LaLonde)
>
> ## The model includes a treatment variable,
> ## nine covariates to be interacted with the treatment variable,
> ## and the same nine covariates to be adjusted.
>
> ## Not run:
> ##D
> ##D ## Run to find the LASSO parameters
> ##D F1 <-FindIt(model.treat= outcome ~ treat,
> ##D model.main= ~ age+educ+black+hisp+white+
> ##D marr+nodegr+log.re75+u75,
> ##D model.int= ~ age+educ+black+hisp+white+
> ##D marr+nodegr+log.re75+u75,
> ##D data = LaLonde,
> ##D type="binary",
> ##D treat.type="single")
> ## End(Not run)
>
> ## Fit with uncovered lambda parameters.
> F1 <-FindIt(model.treat= outcome ~ treat,
+ model.main= ~ age+educ+black+hisp+white+
+ marr+nodegr+log.re75+u75,
+ model.int= ~ age+educ+black+hisp+white+
+ marr+nodegr+log.re75+u75,
+ data = LaLonde,
+ type="binary",
+ treat.type="single",
+ search.lambdas=FALSE,
+ lambdas = c(-3.8760,-4.0025) )
>
> summary(F1)
Call:
Treatment Model: outcome ~ treat
Main Model : ~age + educ + black + hisp + white + marr + nodegr + log.re75 +
u75
Interaction Covariates: ~age + educ + black + hisp + white + marr + nodegr + log.re75 +
u75
Treatment type: [1] "single"
Outcome type: [1] "binary"
ATE:
[1] 0.07903784
Coefficients:
Intercept black1 white1
0.164 -0.11 0.00577
age log.re75 hisp1:age
-0.0111 -0.0369 -0.0127
white1:nodegr1 marr1:age nodegr1:u751
-0.0557 0.00294 -0.00997
nodegr1:age nodegr1:educ u751:age
-0.00505 -0.00667 0.00351
age.2 log.re75.2 treat
0.00058 -0.0102 0.0692
treat:marr1 treat:black1:log.re75 treat:hisp1:log.re75
0.0495 0.00629 -0.0418
treat:white1:nodegr1 treat:marr1:educ treat:nodegr1:u751
0.181 0.0108 -0.0848
treat:u751:age treat:u751:educ treat:age.2
0.018 0.0403 -0.000232
treat:educ.2
0.00828
---------
Model Fit Statistics:
GCV:
Null: 0.986 Model: 625.985
Percent Misclassified:
Null: 0.43 Model: 0.32
Percent Improvement, vs. NULL: 26.75 %
Percent Outside Margin:
1.108 %, n = 8
>
> ## Returns all the estimated treatment effects.
> pred1 <- predict(F1)
> ## Top10
> head(pred1$data, n=10)
Treatment.effect outcome treatment age educ black hisp white marr nodegr
25 0.4828246 0 0 31 4 0 0 1 0 1
719 0.4657111 1 1 31 4 1 0 0 0 1
263 0.4350411 0 0 33 5 0 1 0 0 1
503 0.3979228 1 1 28 15 1 0 0 0 0
255 0.3850236 1 0 30 14 1 0 0 1 0
434 0.3708778 1 1 22 16 1 0 0 0 0
168 0.3679018 1 0 32 12 0 1 0 1 0
174 0.3674185 0 0 50 10 0 1 0 0 1
53 0.3619136 1 0 28 11 0 1 0 1 1
186 0.3605496 1 0 34 13 1 0 0 1 0
log.re75 u75
25 9.278092 0
719 8.299286 0
263 0.000000 1
503 0.000000 1
255 0.000000 1
434 0.000000 1
168 0.000000 1
174 0.000000 1
53 0.000000 1
186 0.000000 1
> ## Bottom 10
> tail(pred1$data ,n=10)
Treatment.effect outcome treatment age educ black hisp white marr nodegr
235 -0.1391499 1 0 29 12 0 0 1 0 0
266 -0.1392401 1 0 50 8 1 0 0 1 1
107 -0.1445337 1 0 29 9 0 1 0 0 1
554 -0.1492456 0 1 29 10 0 1 0 0 1
264 -0.1565843 0 0 31 12 0 0 1 0 0
548 -0.1576406 0 1 31 12 0 0 1 0 0
595 -0.1880842 0 1 34 11 0 1 0 0 1
302 -0.1945496 0 0 36 11 0 1 0 0 1
245 -0.2029329 0 0 27 12 0 1 0 0 0
268 -0.2140965 0 0 36 12 0 1 0 0 0
log.re75 u75
235 9.409273 0
266 8.636042 0
107 9.698982 0
554 9.088286 0
264 6.206576 0
548 7.867489 0
595 8.442685 0
302 8.026497 0
245 10.098273 0
268 9.350189 0
>
> ## Visualize all the estimated treatment effects.
> ## Not run:
> ##D plot(pred1)
> ## End(Not run)
>
> ###################################################
> ## Example 2: Treatment-Treatment Interaction
> ###################################################
>
> ## Not run:
> ##D data(GerberGreen)
> ##D
> ##D ## The model includes four factorial treatments and
> ##D ## all two, three, four-way interactions between them.
> ##D ## Four pre-treatment covariates are adjusted.
> ##D
> ##D ## Run to search for lambdas.
> ##D F2<- FindIt(model.treat= voted98 ~ persngrp+phnscrpt+mailings+appeal,
> ##D nway=4,
> ##D model.main= ~ age+majorpty+vote96.1+vote96.0,
> ##D data = GerberGreen,
> ##D type="binary",
> ##D treat.type="multiple")
> ##D
> ##D ## Fit, given selected lambdas.
> ##D F2<- FindIt(model.treat= voted98 ~ persngrp+phnscrpt+mailings+appeal,
> ##D nway=4,
> ##D model.main= ~ age+majorpty+vote96.1+vote96.0,
> ##D data = GerberGreen,
> ##D type="binary",
> ##D treat.type="multiple",
> ##D search.lambdas=FALSE,
> ##D lambdas=c(-15.000,-6.237))
> ##D
> ##D ## Returns coefficient estimates.
> ##D summary(F2)
> ##D
> ##D ## Returns predicted values for unique treatment combinations.
> ##D pred2 <- predict(F2,unique=TRUE)
> ##D ## Top 10
> ##D head(pred2$data, n=10)
> ##D ## Bottom 10
> ##D tail(pred2$data, n=10)
> ##D
> ##D ## Visualize predicted values for each treatment combination.
> ##D plot(pred2)
> ## End(Not run)
>
> ###################################################
> ## Example 3: Conjoint Analysis. Causal Interaction.
> ###################################################
> data(Immigration)
>
> ## Not run:
> ##D
> ##D ## The SVM classifier with a lasso constraint is estimated.
> ##D ## The model includes all three-way and two-way interactions as
> ##D ## well as main effects of five factorial treatments.
> ##D
> ##D ## Run to search for lambdas.
> ##D F.conjoint<- FindIt(model.treat= outcome ~
> ##D Education+Gender+Origin+Experience+Plans,
> ##D nway=3,
> ##D data = Immigration,
> ##D type="binary",
> ##D treat.type="multiple")
> ##D
> ##D ## Make the full factorial design matrix as the target population.
> ##D full <- full.FindIt(F.conjoint)
> ##D
> ##D ## Compute the predicted potential outcomes for the target population.
> ##D Unifdata <- predict(F.conjoint,newdata=full,sort=FALSE)$data
> ## End(Not run)
> ## load pre-computed F.conjoint and Unifdata
> data(F.conjoint)
> data(Unifdata)
>
> ## Compute AMTEs, ATCEs, and AMTIEs
>
> ## Range of each factor interaction.
> ## The range of the AMTEs for all factors
> compare1 <- INT(F.conjoint,target.data=Unifdata,compare=TRUE,order=1)
[1] "Using Data representing the target population."
[1] "Range of Marginal Effects"
> compare1
Education Gender Origin Experience Plans
0.197383880 0.007739259 0.126250464 0.087042921 0.267419190
>
> ## The range of the two-way AMTIEs for all two-way factor interactions
> compare2 <- INT(F.conjoint,target.data=Unifdata,compare=TRUE,order=2)
[1] "Using Data representing the target population."
> compare2
V1 V2 range.TIE
8 Origin Experience 0.09497208
3 Education Experience 0.08462559
9 Origin Plans 0.07603530
2 Education Origin 0.06145049
10 Experience Plans 0.05552839
4 Education Plans 0.04373174
1 Education Gender 0.04204575
7 Gender Plans 0.03679968
5 Gender Origin 0.03343086
6 Gender Experience 0.01627208
>
> ## Not run:
> ##D ## The range of the three-way AMTIEs for all three-way factor interactions
> ##D compare3 <- INT(F.conjoint,target.data=Unifdata,compare=TRUE,order=3)
> ##D compare3
> ## End(Not run)
>
> ## Compute AMTIEs within factor interactions.
> ## Origin x Experience
> out.OP <- INT(F.conjoint,target.data=Unifdata,
+ column=c("Origin","Experience"),
+ base=c("India","No.job"), order=2)
[1] "Using Data representing the target population."
> out.OP
$`Range of AMTIE`
[1] 0.09497208
$AMTIE
AMTIE Origin Experience
24 0.05781929 Philippines Three.Five
39 0.02278772 China One.Two
27 0.01900762 Mexico Three.Five
5 0.01166398 Somalia No.job
15 0.01130097 Somalia Morethan.five
33 0.00783490 Sudan One.Two
3 0.00337015 Sudan No.job
32 0.00242682 Iraq One.Two
13 0.00192642 Sudan Morethan.five
25 0.00187452 Somalia Three.Five
36 0.00095525 France One.Two
38 0.00095525 Poland One.Two
40 0.00095525 Germany One.Two
2 0.00043286 Iraq No.job
1 0.00000000 India No.job
11 -0.00036302 India Morethan.five
6 -0.00103870 France No.job
8 -0.00103870 Poland No.job
10 -0.00103870 Germany No.job
9 -0.00116432 China No.job
16 -0.00140171 France Morethan.five
18 -0.00140171 Poland Morethan.five
20 -0.00140171 Germany Morethan.five
19 -0.00152733 China Morethan.five
31 -0.00216084 India One.Two
12 -0.00581640 Iraq Morethan.five
37 -0.00790280 Mexico One.Two
22 -0.00935660 Iraq Three.Five
21 -0.00978947 India Three.Five
17 -0.01025977 Mexico Morethan.five
26 -0.01082816 France Three.Five
28 -0.01082816 Poland Three.Five
30 -0.01082816 Germany Three.Five
7 -0.01315837 Mexico No.job
34 -0.01948207 Philippines One.Two
14 -0.02183903 Philippines Morethan.five
23 -0.02544479 Sudan Three.Five
4 -0.02881151 Philippines No.job
29 -0.03240939 China Three.Five
35 -0.03715279 Somalia One.Two
$ATCE
ATCE AMTIE Origin Experience
24 0.154402938 0.05781929 Philippines Three.Five
20 0.121031247 -0.00140171 Germany Morethan.five
18 0.118807677 -0.00140171 Poland Morethan.five
30 0.100724439 -0.01082816 Germany Three.Five
27 0.099890810 0.01900762 Mexico Three.Five
28 0.098500870 -0.01082816 Poland Three.Five
40 0.087542873 0.00095525 Germany One.Two
11 0.086679906 -0.00036302 India Morethan.five
14 0.085624970 -0.02183903 Philippines Morethan.five
38 0.085319304 0.00095525 Poland One.Two
16 0.085159887 -0.00140171 France Morethan.five
19 0.082613227 -0.00152733 China Morethan.five
17 0.081503781 -0.01025977 Mexico Morethan.five
13 0.079817504 0.00192642 Sudan Morethan.five
39 0.071082942 0.02278772 China One.Two
21 0.066373098 -0.00978947 India Three.Five
26 0.064853079 -0.01082816 France Three.Five
34 0.052136597 -0.01948207 Philippines One.Two
36 0.051671514 0.00095525 France One.Two
33 0.049880642 0.00783490 Sudan One.Two
31 0.049036743 -0.00216084 India One.Two
37 0.048015407 -0.00790280 Mexico One.Two
15 0.046922401 0.01130097 Somalia Morethan.five
23 0.041565940 -0.02544479 Sudan Three.Five
29 0.040850817 -0.03240939 China Three.Five
10 0.034351341 -0.00103870 Germany No.job
8 0.032127772 -0.00103870 Poland No.job
25 0.026615593 0.00187452 Somalia Three.Five
1 0.000000000 0.00000000 India No.job
6 -0.001520019 -0.00103870 France No.job
9 -0.004066679 -0.00116432 China No.job
3 -0.005781686 0.00337015 Sudan No.job
4 -0.008390429 -0.02881151 Philippines No.job
7 -0.008437747 -0.01315837 Mexico No.job
12 -0.009633901 -0.00581640 Iraq Morethan.five
22 -0.024054464 -0.00935660 Iraq Three.Five
32 -0.037236030 0.00242682 Iraq One.Two
35 -0.037376699 -0.03715279 Somalia One.Two
5 -0.039757505 0.01166398 Somalia No.job
2 -0.090427562 0.00043286 Iraq No.job
$`Sum of AMTEs`
Sum of AMTEs AMTIE Origin Experience
20 0.1224329595 -0.00140171 Germany Morethan.five
18 0.1202093902 -0.00140171 Poland Morethan.five
30 0.1115526016 -0.01082816 Germany Three.Five
28 0.1093290323 -0.01082816 Poland Three.Five
14 0.1074640034 -0.02183903 Philippines Morethan.five
24 0.0965836454 0.05781929 Philippines Three.Five
17 0.0917635473 -0.01025977 Mexico Morethan.five
11 0.0870429213 -0.00036302 India Morethan.five
40 0.0865876190 0.00095525 Germany One.Two
16 0.0865615998 -0.00140171 France Morethan.five
38 0.0843640497 0.00095525 Poland One.Two
19 0.0841405612 -0.00152733 China Morethan.five
27 0.0808831893 0.01900762 Mexico Three.Five
13 0.0778910846 0.00192642 Sudan Morethan.five
21 0.0761625634 -0.00978947 India Three.Five
26 0.0756812418 -0.01082816 France Three.Five
29 0.0732602032 -0.03240939 China Three.Five
34 0.0716186628 -0.01948207 Philippines One.Two
23 0.0670107266 -0.02544479 Sudan Three.Five
37 0.0559182067 -0.00790280 Mexico One.Two
31 0.0511975808 -0.00216084 India One.Two
36 0.0507162592 0.00095525 France One.Two
39 0.0482952206 0.02278772 China One.Two
33 0.0420457440 0.00783490 Sudan One.Two
15 0.0356214320 0.01130097 Somalia Morethan.five
10 0.0353900382 -0.00103870 Germany No.job
8 0.0331664689 -0.00103870 Poland No.job
25 0.0247410741 0.00187452 Somalia Three.Five
4 0.0204210820 -0.02881151 Philippines No.job
7 0.0047206259 -0.01315837 Mexico No.job
1 0.0000000000 0.00000000 India No.job
35 -0.0002239085 -0.03715279 Somalia One.Two
6 -0.0004813215 -0.00103870 France No.job
9 -0.0029023601 -0.00116432 China No.job
12 -0.0038175047 -0.00581640 Iraq Morethan.five
3 -0.0091518367 0.00337015 Sudan No.job
22 -0.0146978627 -0.00935660 Iraq Three.Five
32 -0.0396628453 0.00242682 Iraq One.Two
5 -0.0514214893 0.01166398 Somalia No.job
2 -0.0908604261 0.00043286 Iraq No.job
>
> ## Not run:
> ##D ## Education x Gender x Origin
> ##D out.EGO <- INT(F.conjoint,target.data=Unifdata,
> ##D column=c("Education","Gender","Origin"),
> ##D base=c("No.formal","female","India"), order=3)
> ##D out.EGO
> ## End(Not run)
>
>
>
>
>
>
> dev.off()
null device
1
>