R Graphical Manual

Browse All

Last data update: 2014.03.03

R: Display ROC curve and related AUC statistic, or...

ROCtest

R Documentation

Display ROC curve and related AUC statistic, or sensitivity-specificity plot of glm with binomial family.

Description

Provides two options following the glm() function with binomial family. 1: Senstivity-specificity plot with optimal cut point statistic 2: ROC plot with Area Under Curve (AUC) statistic

Usage

ROCtest(model= model, fold=10, type=c("ROC","Sensitivity"))

Arguments

`model`	model name
`fold`	number of k-folds
`type`	type of plot

Format

x: The function has three arguments: modelname, folds, type of plot

Details

ROCtest is a post-estimation function for logistic regression, following the use of glm(). Options to display a sensitivity-specificity plot or ROC curve are available.

Value

plot

Note

ROCtest() must be loaded into memory in order to be effectve. As a function in LOGIT, it is immediately available to a user.

Author(s)

Rafael de Souza, ELTE, Hungary, Joseph M. Hilbe, Arizona State University.

References

Hilbe, Joseph M. (2016), Practical Guide to Logistic Regression, Chapman & Hall/CRC. Hilbe, Joseph M. (2009), Logistic Regression Models, Chapman & Hall/CRC.

Examples

library(MASS)
 library(LOGIT)
 data(R84)
 R84$cage <- R84$age - mean(R84$age)
 R84$cdoc <- R84$docvis - mean(R84$docvis)
 mylogit <- glm(outwork ~ cdoc + female + kids + cage + factor(edlevel),
 family=binomial, data=R84)
 summary(mylogit)
 ROCtest(mylogit, fold=10, type="Sensitivity")
 ROCtest(mylogit, fold=10, type="ROC")

Results


R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(LOGIT)
> png(filename="/home/ddbj/snapshot/RGM3/R_CC/result/LOGIT/ROCtest.Rd_%03d_medium.png", width=480, height=480)
> ### Name: ROCtest
> ### Title: Display ROC curve and related AUC statistic, or
> ###   sensitivity-specificity plot of glm with binomial family.
> ### Aliases: ROCtest
> ### Keywords: models
> 
> ### ** Examples
> 
> library(MASS)
>  library(LOGIT)
>  data(R84)
>  R84$cage <- R84$age - mean(R84$age)
>  R84$cdoc <- R84$docvis - mean(R84$docvis)
>  mylogit <- glm(outwork ~ cdoc + female + kids + cage + factor(edlevel),
+  family=binomial, data=R84)
>  summary(mylogit)

Call:
glm(formula = outwork ~ cdoc + female + kids + cage + factor(edlevel), 
    family = binomial, data = R84)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.3615  -0.6807  -0.4320   0.8541   2.8220  

Coefficients:
                  Estimate Std. Error z value Pr(>|z|)    
(Intercept)      -2.022806   0.085790 -23.579  < 2e-16 ***
cdoc              0.024480   0.006255   3.914 9.08e-05 ***
female            2.268467   0.084039  26.993  < 2e-16 ***
kids              0.373053   0.090586   4.118 3.82e-05 ***
cage              0.054355   0.004207  12.921  < 2e-16 ***
factor(edlevel)2  0.009293   0.173172   0.054 0.957201    
factor(edlevel)3  0.478016   0.153786   3.108 0.001882 ** 
factor(edlevel)4 -0.830282   0.214336  -3.874 0.000107 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 5091.1  on 3873  degrees of freedom
Residual deviance: 3889.9  on 3866  degrees of freedom
AIC: 3905.9

Number of Fisher Scoring iterations: 4

>  ROCtest(mylogit, fold=10, type="Sensitivity")
$plot

$Observed
   [1] 1 0 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0
  [38] 0 0 0 1 1 0 1 1 0 0 0 0 1 1 1 0 1 0 0 0 1 0 0 1 1 0 0 1 0 0 0 1 0 0 0 1 1
  [75] 1 0 1 0 0 0 1 0 1 0 0 1 0 1 1 0 1 0 0 0 1 1 0 0 0 1 0 0 0 0 1 1 1 0 1 1 0
 [112] 0 1 0 0 1 1 1 0 0 0 0 1 0 1 1 1 1 0 1 1 0 1 0 0 1 0 0 1 0 1 1 0 0 1 0 1 1
 [149] 0 1 1 0 0 0 0 0 1 0 0 0 1 1 0 0 1 0 1 1 0 1 0 0 1 0 0 1 0 0 1 1 0 0 0 0 0
 [186] 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 1 1 0 0 1 1 0 1 0 1 1 0 1 0 1 0 1 1
 [223] 1 0 0 1 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 1 1 0 1 1 0 0 1 0 0 1 1 0 1
 [260] 0 1 0 0 1 1 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 1 0 0 1 0 0 0 0 0
 [297] 0 0 0 1 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 0 1 1 1 1 0 0 0
 [334] 0 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 1 1 1 0 0 0 0 1 0 0 0 0 0 0 0 1 1 1 1 0
 [371] 1 0 1 0 0 0 1 0 0 0 1 0 0 1 1 0 1 0 0 1 0 0 0 0 1 1 0 1 0 0 0 0 1 1 1 0 0
 [408] 1 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 1 1 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 1 1 1 0
 [445] 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 1 0 1 1 0 0 0 0 1 1 1 1 0 1 0 1 0 1 1 0
 [482] 1 1 0 0 0 0 0 1 0 1 0 1 0 1 0 1 1 1 0 1 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0 0
 [519] 0 1 1 1 1 0 0 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0 0 0 1 1 1 0 0 0 1 0 1 1 0 1 1
 [556] 0 0 0 1 0 1 1 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 0 1 1 0 0 1 0 1 1 0 1 1 0 0
 [593] 0 0 0 0 0 1 0 0 1 1 1 1 1 0 1 1 1 0 0 1 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1 0 1
 [630] 0 1 1 1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0 1 1 0 1 0 1 1 0 0 1 1 0 1 1 1 1
 [667] 0 0 1 1 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 1 0 0
 [704] 0 0 0 0 1 1 1 1 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 0 1 1 0 1 0 0 1 1 0 1 0 0
 [741] 1 0 0 0 0 1 1 1 0 0 1 1 0 0 0 0 1 0 1 0 1 1 1 0 0 0 0 0 0 1 0 1 0 0 0 0 0
 [778] 0 0 0 0 1 0 0 0 1 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0
 [815] 1 1 0 0 1 1 1 1 1 1 1 1 1 0 1 0 0 0 1 0 0 1 0 1 1 1 0 0 1 1 1 0 0 0 0 1 1
 [852] 1 1 0 1 0 0 0 1 0 0 1 1 1 0 0 0 1 0 1 1 0 1 1 0 0 0 0 0 0 0 0 0 1 0 0 1 0
 [889] 0 1 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 0 1 0 0 1 1 1 0 0 0 0 1 1 0 1 1 0 0 0
 [926] 1 0 1 0 0 0 0 1 1 0 1 0 0 1 0 0 0 0 1 1 1 0 1 1 1 1 0 0 1 1 0 1 0 0 0 1 1
 [963] 0 0 1 0 0 0 1 1 0 1 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 1 0 0 1 0 1 0
[1000] 0 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 1 1 0 1 0 1 1 0 0 1 1 0 0 0 0 1 0 0 0
[1037] 0 1 0 0 0 1 1 0 0 0 0 1 0 0 0 0 1 1 1 0 0 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0
[1074] 0 0 1 0 1 0 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0
[1111] 0 0 1 0 0 0 0 0 0 0 1 1 0 0 1 0 1 0 1 0 1 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 1
[1148] 0 0 1 1 0 0 1 1 0 1 1 1 1 0 0 0 1 1 1 0 0 1 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0
[1185] 0 0 1 0 1 0 1 1 0 0 1 0 0 0 1 1 1 0 1 1 1 1 0 1 0 1 0 0 0 1 0 1 0 1 0 0 0
[1222] 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 1 0 0 0 0 0 1 1
[1259] 0 0 0 1 0 0 0 0 0 1 0 0 1 0 0 1 1 0 0 1 1 0 1 1 0 1 1 0 0 0 1 0 0 0 0 1 0
[1296] 0 1 0 1 0 0 0 0 0 0 0 0 1 0 1 1 0 1 1 1 0 0 0 1 1 1 0 0 0 1 0 1 0 0 0 1 0
[1333] 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 1 0 1 0 0 0 1 1 1 0 0
[1370] 1 1 0 0 0 1 0 1 0 1 0 1 0 0 1 0 0 0 0 0 1 0 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1
[1407] 0 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 1 1 0 0 1 1 1 0 1 0 0 0 1 0 0 0 0 0 1
[1444] 1 0 0 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1 1 1 1 0 0 0 1 0
[1481] 1 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 1 0 1 0 1 0 1 0 0 0 0 0 0 1 1 1 1 1 0 0 0
[1518] 0 1 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 1 1
[1555] 0 0 0 0 1 0 0 1 0 0 1 1 0 1 1 1 0 0 1 1 0 1 1 0 0 0 0 1 0 0 0 0 0 0 0 1 0
[1592] 1 0 0 0 0 0 0 1 1 1 0 1 0 0 0 0 0 0 1 1 0 0 1 0 0 0 0 0 1 0 1 1 0 0 0 0 0
[1629] 1 0 0 1 0 0 0 0 1 1 1 1 0 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 1 0 0 0 0 1
[1666] 1 1 1 1 1 0 1 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 1 1 0 1 0 0 1 0 0 1 0
[1703] 0 1 1 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1 0 1 0 1 0 0 0 0 1 0 0
[1740] 0 0 1 0 0 1 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 1 0 1 0 0 1 0 0 1 1 0 1 1
[1777] 0 0 1 1 1 1 0 1 0 1 0 0 0 1 1 0 0 1 0 1 0 1 0 1 0 1 0 0 0 1 1 1 1 0 0 0 0
[1814] 1 0 0 0 0 1 0 0 1 0 0 0 0 0 0 1 1 1 0 1 0 0 0 1 0 1 0 0 0 1 0 0 0 1 1 0 1
[1851] 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 0 1 1 1 1 0 0 1 0 0 1 1
[1888] 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0
[1925] 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 1 0 0 1 0 1 0 1 0 0 0 1 0 0
[1962] 1 1 1 0 0 0 1 0 0 0 0 0 0 1 1 1 0 0 1 1 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 1 1
[1999] 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 0 0 0 1 0 0 0 1 1 1 0
[2036] 0 0 1 0 0 1 1 1 1 1 0 1 1 0 0 0 1 0 1 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0
[2073] 0 1 0 1 1 0 1 0 0 1 0 0 0 0 1 0 0 1 1 0 1 0 0 0 1 0 1 1 1 0 0 0 0 0 0 0 0
[2110] 1 1 1 1 0 0 0 0 0 1 0 0 0 1 0 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0
[2147] 0 0 1 0 0 1 1 1 1 0 1 0 0 0 0 1 1 0 0 1 0 1 0 1 0 1 0 1 0 0 0 1 1 0 1 1 1
[2184] 0 0 0 1 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0 1 1 0 0 0 0 0 0 0 0 0 0
[2221] 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 1 1 0 1 0 0 1 0 1 1 1 1 1 0
[2258] 1 0 1 0 1 1 0 1 0 0 0 1 1 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 1 0 1 0 0 0 0
[2295] 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 0 0 0
[2332] 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 1 1 1 1 0 0 1 0 0 0 1
[2369] 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 1 1 0 0 1 0 0 0 0 1 0 0 0 0 1 1 0 1 1 0
[2406] 1 0 0 0 0 1 0 1 1 0 1 0 0 1 1 0 0 1 0 0 0 0 0 1 1 0 1 1 0 0 0 1 0 1 1 0 0
[2443] 1 1 0 1 0 1 0 1 0 0 0 1 0 0 0 1 0 1 1 1 1 0 0 0 0 1 1 0 0 1 0 0 0 0 1 0 0
[2480] 0 1 1 1 1 0 0 0 1 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 1 1 1 0 0
[2517] 0 1 1 0 1 0 1 1 1 0 0 1 1 0 1 1 0 0 1 0 0 0 0 0 1 1 0 1 1 0 1 0 0 1 0 0 0
[2554] 0 0 1 1 0 0 1 0 0 0 0 0 1 0 1 1 0 1 1 1 1 1 0 0 1 0 1 1 0 0 1 0 0 0 1 0 0
[2591] 1 0 1 0 1 1 0 0 0 1 1 0 0 0 0 1 0 1 0 1 1 0 0 0 0 1 0 0 1 0 1 1 0 0 1 0 0
[2628] 0 0 0 1 0 0 1 0 0 0 1 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 1 0 0 0 1 0 1 0 0 0 1
[2665] 1 0 0 0 0 1 0 0 1 0 0 1 0 1 1 1 1 0 0 1 0 1 1 0 0 1 1 1 0 0 0 0 0 0 1 0 1
[2702] 0 0 1 0 0 0 1 1 1 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0
[2739] 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 1 0 0 0 0 0 1 0 1 1 0 0 0 0 1 1
[2776] 0 1 0 0 0 0 1 0 0 1 0 1 0 1 0 1 1 0 1 0 0 1 0 0 0 1 0 1 0 0 0 0 1 1 0 1 1
[2813] 1 0 0 0 0 0 1 1 0 0 0 1 0 1 0 0 0 0 1 0 0 1 0 0 0 1 1 1 0 1 1 1 1 0 0 0 1
[2850] 0 1 1 1 0 1 0 0 0 1 0 0 0 0 0 0 1 0 1 1 0 1 0 0 1 0 1 0 0 1 1 1 1 0 0 0 0
[2887] 0 0 0 1 0 1 1 0 1 0 0 1 0 1 1 1 1 0 1 0 1 0 0 0 0 0 1 0 1 1 0 0 1 1 1 0 1
[2924] 1 1 0 0 1 0 1 1 0 1 0 0 0 0 0 1 0 1 1 1 1 1 1 0 0 0 1 0 0 0 0 0 1 1 1 0 0
[2961] 1 1 1 0 0 1 1 1 0 1 1 1 0 1 1 1 0 1 0 0 0 0 1 0 1 1 0 1 0 1 1 0 1 0 0 1 1
[2998] 0 0 1 1 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 1 1 0 1 0 0 0 0 1 0 0
[3035] 0 1 1 0 0 0 0 1 1 0 1 0 1 1 0 0 1 1 0 1 1 1 0 1 0 0 0 0 0 1 0 1 0 1 0 0 0
[3072] 0 0 0 0 0 1 0 1 1 1 1 0 0 0 1 0 1 1 0 1 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0
[3109] 0 1 0 1 1 1 1 1 0 1 1 1 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 1 1
[3146] 0 1 1 0 1 1 1 1 1 0 0 1 0 0 1 0 1 0 0 1 1 0 1 1 0 1 1 0 0 0 0 1 0 1 1 0 0
[3183] 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 1 0 0 0 1 0 1 0 0 0 0 1 0 1 1 1 0 0 1 1
[3220] 0 0 0 0 1 0 1 0 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 1 0 1 1 0 0
[3257] 0 0 1 1 0 0 1 0 1 1 0 1 1 0 0 1 1 0 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 1 0
[3294] 0 1 1 1 0 0 0 0 1 1 1 0 0 1 1 1 1 0 1 0 0 0 0 0 0 0 1 0 0 0 1 1 1 0 0 0 1
[3331] 1 1 1 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 1 0 1 0
[3368] 1 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1
[3405] 0 1 0 0 1 1 0 1 0 1 0 0 1 1 1 1 0 0 1 0 0 0 0 1 0 0 0 0 1 1 0 1 0 1 1 1 1
[3442] 1 0 0 1 0 1 1 1 0 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 1 1
[3479] 0 0 1 1 1 1 1 1 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 1 1 1 0
[3516] 0 0 0 1 0 1 1 1 1 0 1 1 1 0 1 1 1 1 1 0 0 0 0 1 0 0 0 1 0 0 1 0 0 1 0 0 1
[3553] 1 0 1 0 0 0 0 0 1 0 0 1 0 0 1 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 1
[3590] 0 0 0 0 0 0 1 0 1 0 1 0 1 1 1 0 0 0 1 0 1 1 0 0 0 1 0 1 1 1 1 0 0 0 1 0 0
[3627] 1 1 1 0 0 0 1 1 1 1 1 1 1 0 0 1 0 0 0 0 0 1 0 1 1 1 0 1 1 0 0 1 1 0 1 0 1
[3664] 0 0 0 1 1 0 0 0 0 1 0 0 0 1 1 1 1 0 1 0 0 0 0 0 0 1 1 0 1 0 1 1 0 1 0 0 1
[3701] 0 1 1 1 0 1 0 0 1 0 1 1 1 0 0 0 0 1 1 0 1 0 0 0 1 1 1 0 0 0 0 0 0 1 0 1 0
[3738] 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 1 0
[3775] 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 0 1 0 1 1 0 0 0 1 0 0 0 0 0
[3812] 0 0 1 0 0 0 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 0 0 0 1 1 0 1 0 1 1 0 0 0 0 0 1
[3849] 0 0 1 1 0 0 0 0 1 1 0 0 0 0 1 0 0 1 1 1 0 0 1 1 1 0
Levels: 0 1

$Predicted
   [1] 0 0 1 1 0 1 0 0 1 1 1 1 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 1 0 0 1 1
  [38] 1 0 0 1 1 0 0 1 0 1 0 0 1 0 1 1 1 1 0 0 1 1 0 1 1 0 1 1 0 1 1 1 0 0 0 1 1
  [75] 1 1 1 0 0 0 0 0 1 1 0 1 0 1 1 0 1 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 1 0
 [112] 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 0 0 1 1 1 0 1 0 1 0 1 0 0 0 1 0 0 1 1 1 1
 [149] 0 1 1 1 1 0 0 0 1 1 1 0 1 1 0 1 1 1 0 1 0 1 0 0 0 0 0 1 1 0 1 1 0 0 0 0 1
 [186] 1 0 0 1 1 0 0 1 0 1 1 1 0 0 0 0 1 1 0 1 1 1 0 1 1 0 1 0 1 1 1 1 0 1 0 1 1
 [223] 1 1 0 1 1 0 0 1 1 1 0 1 1 1 1 0 1 0 1 0 0 1 0 1 1 0 1 0 0 1 1 0 1 1 0 0 1
 [260] 0 1 0 0 1 1 0 1 1 0 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 1 0 0 1 1 0 1 1 0
 [297] 0 1 0 1 1 0 0 1 1 0 1 0 0 1 1 0 0 1 1 1 0 0 1 0 1 1 0 0 1 0 1 0 1 1 0 1 0
 [334] 0 0 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 1 1 0 1 0 1 0 0 0 1 1 1 1 0
 [371] 1 0 1 1 0 0 1 1 0 0 1 1 1 0 1 1 1 1 0 1 1 0 0 0 0 1 1 1 0 0 1 0 1 1 1 0 0
 [408] 0 0 1 1 0 0 0 1 1 1 1 1 0 0 0 0 0 1 1 1 0 0 0 0 0 1 0 1 1 1 1 0 0 0 1 1 0
 [445] 0 0 0 1 0 1 0 0 0 1 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 0 1 1 1 1 1 0 1 0 1 1 0
 [482] 1 0 0 0 1 0 0 1 1 1 1 1 1 1 0 1 1 1 0 1 0 0 1 1 1 0 0 0 0 1 1 0 1 0 0 1 0
 [519] 0 1 1 1 1 0 0 0 0 1 1 0 1 1 0 0 1 1 1 0 0 1 0 1 1 1 1 0 0 1 1 0 1 1 1 1 1
 [556] 0 0 0 0 0 1 1 0 0 1 0 1 0 1 0 1 0 0 0 0 0 0 1 0 1 1 0 0 1 0 1 1 0 1 1 0 0
 [593] 0 0 1 1 0 1 0 0 1 1 1 1 1 0 1 1 1 0 1 1 1 1 0 1 1 1 1 1 0 1 1 1 0 1 1 1 1
 [630] 0 1 1 1 0 0 0 1 0 1 0 1 0 1 0 0 0 1 0 0 0 1 1 1 1 0 1 1 0 0 1 0 0 1 0 1 1
 [667] 1 1 1 1 0 0 0 0 1 1 0 0 0 1 0 1 1 0 1 1 0 1 1 0 1 0 1 0 1 1 1 0 0 1 1 0 1
 [704] 1 0 0 0 1 1 1 1 0 1 1 0 0 0 1 0 0 1 0 0 1 0 0 0 0 1 1 0 1 0 0 0 1 0 1 0 0
 [741] 1 0 1 0 1 1 1 1 1 0 1 1 0 0 0 1 1 0 1 0 1 1 1 0 0 0 0 0 0 1 1 1 0 0 1 1 0
 [778] 0 0 1 0 1 1 1 1 1 0 1 1 1 1 1 1 0 1 0 0 1 0 0 0 1 0 0 0 0 1 1 0 0 0 1 0 0
 [815] 0 1 1 1 1 1 1 1 1 1 0 0 1 1 1 0 0 1 1 1 0 1 0 1 1 1 1 1 1 1 1 0 0 0 0 1 0
 [852] 1 0 0 1 1 1 0 1 0 0 1 0 1 0 0 1 1 0 1 1 0 1 1 0 1 0 0 1 0 0 1 1 1 0 1 1 1
 [889] 0 1 1 1 1 0 1 0 0 0 1 1 0 0 1 0 1 1 1 1 1 0 1 1 0 0 0 1 0 1 1 0 0 1 0 0 1
 [926] 1 0 1 0 0 0 0 1 1 0 1 1 1 1 0 0 1 0 1 1 1 0 1 1 1 1 1 0 1 1 0 1 0 1 0 1 1
 [963] 0 1 1 0 1 1 1 1 1 0 1 1 0 1 0 0 0 0 1 0 1 0 0 0 1 1 1 0 0 1 0 0 1 0 0 0 1
[1000] 0 0 1 0 1 0 0 1 1 0 0 1 0 1 1 0 0 1 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
[1037] 0 1 0 0 1 1 1 1 0 0 0 1 0 1 0 1 1 1 1 0 0 0 1 1 1 1 0 1 0 0 0 1 1 0 1 1 1
[1074] 0 0 1 0 1 0 1 0 1 0 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 0 1 0 0 1 1 0 0 0 0 1 0
[1111] 0 0 1 0 1 0 1 0 0 0 0 1 1 1 1 1 1 0 1 0 1 0 0 0 0 1 1 1 1 1 0 1 1 1 0 0 1
[1148] 0 0 1 1 1 0 1 1 0 1 1 1 1 1 1 1 0 0 1 0 1 0 1 0 0 0 0 1 0 1 1 0 0 1 1 0 0
[1185] 0 0 1 0 1 1 1 1 1 0 1 0 1 0 1 1 1 0 1 0 1 0 0 1 0 1 0 1 1 1 0 1 0 1 0 0 0
[1222] 0 0 0 0 0 1 1 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 1 1 1 1 0 1 0 0 0 0 0 1 1
[1259] 1 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 1 0 1 1 0 0 0 1 0 0 1 0 1 0
[1296] 1 1 1 1 0 1 0 0 1 1 0 0 1 0 1 1 1 1 1 1 0 0 0 1 1 1 0 0 1 1 0 1 0 0 1 1 1
[1333] 0 1 0 0 1 0 1 0 0 0 0 0 1 1 0 1 0 1 0 0 0 1 0 1 0 1 1 0 0 0 0 1 1 1 0 0 0
[1370] 1 1 0 0 0 1 0 0 1 1 0 1 0 1 1 0 0 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1
[1407] 0 1 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 1 1 1 0 0 1 1 0 1 0 1 0 1 1 0 1 1 1 1
[1444] 1 0 0 0 1 0 1 0 1 1 1 1 0 1 1 0 0 0 1 0 1 1 0 1 1 1 0 1 1 0 1 1 0 0 0 1 0
[1481] 1 0 0 0 0 0 1 0 0 1 0 0 0 1 1 0 1 0 1 0 1 0 1 1 0 1 0 0 0 1 1 1 1 1 0 1 0
[1518] 1 1 1 0 0 0 0 1 0 0 0 1 1 0 1 0 0 1 1 1 0 0 0 1 1 1 0 1 0 0 1 1 1 0 0 1 1
[1555] 0 1 0 1 1 0 1 1 0 0 1 1 1 1 1 1 0 0 1 1 0 1 1 0 0 0 0 1 0 1 1 1 1 0 0 0 1
[1592] 1 0 0 1 0 0 0 1 0 1 1 1 0 1 0 0 1 1 0 1 1 0 1 0 1 1 0 0 1 1 0 1 1 0 0 0 1
[1629] 1 0 0 1 0 0 0 1 1 0 1 1 0 0 1 1 0 1 0 0 1 1 1 1 0 0 1 0 1 0 0 1 0 0 0 0 1
[1666] 1 1 1 0 1 1 1 1 1 0 1 0 0 1 0 1 1 0 1 0 1 1 1 0 0 1 1 1 0 0 1 0 1 0 0 0 0
[1703] 0 1 1 1 0 0 0 0 1 0 0 0 0 0 1 1 0 1 0 1 1 1 1 0 1 0 0 1 0 1 1 0 1 0 0 1 1
[1740] 0 1 1 0 0 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 0 1 1 1 0 1 1 0 0 1 0 1 1 1 0 1 1
[1777] 0 0 1 1 1 0 1 1 0 1 0 0 0 1 1 0 0 1 0 1 0 1 0 1 0 1 1 0 0 1 0 1 1 1 0 0 0
[1814] 1 0 0 0 1 1 0 0 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 0 1 1 1 1 1 1 0 0 0 1 1 1 1
[1851] 1 1 0 1 0 0 1 1 1 1 0 0 0 0 0 0 0 0 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1
[1888] 1 0 0 0 0 0 0 1 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 1 0 0 1 0 1 0 0 1 0 1 0 0
[1925] 0 0 0 0 1 0 0 0 0 0 0 1 0 0 1 1 1 1 0 0 1 0 1 1 0 0 1 0 1 0 1 1 0 1 1 1 0
[1962] 1 1 1 0 1 0 1 1 0 1 1 1 1 1 1 1 1 0 1 1 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 1 1
[1999] 1 1 1 0 1 0 0 1 0 1 0 1 1 0 1 0 1 0 1 0 1 0 0 1 0 0 0 1 1 1 1 1 0 1 1 0 0
[2036] 1 1 1 0 0 1 1 1 0 1 1 1 1 0 0 0 1 0 1 1 0 1 0 0 0 0 1 1 0 1 0 0 1 0 0 0 1
[2073] 1 1 0 1 1 0 1 0 1 0 0 0 0 0 1 1 0 0 1 0 1 0 1 0 1 0 1 1 0 0 0 1 0 0 0 0 0
[2110] 1 1 1 1 0 0 0 0 0 1 0 0 0 1 1 1 0 0 1 1 0 0 0 1 0 0 0 0 0 1 1 1 1 0 1 1 0
[2147] 0 0 1 0 0 1 1 1 1 0 1 0 0 0 1 1 0 0 1 1 0 1 1 1 0 1 0 1 1 0 0 1 1 0 1 0 1
[2184] 1 0 0 1 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0 0 0 1 1 0 1 1 0 1 1 1 1 0 1 0 0 0
[2221] 1 1 0 1 0 1 1 0 0 1 0 0 0 0 0 1 0 0 0 1 1 1 0 1 1 1 1 1 1 1 0 1 1 1 1 1 0
[2258] 1 0 1 0 1 1 0 1 0 0 0 1 0 0 0 1 1 1 0 0 1 1 1 0 1 1 0 0 1 1 0 1 1 0 1 0 1
[2295] 1 0 0 1 0 0 1 0 0 0 0 0 0 1 0 0 1 1 1 0 0 1 0 1 1 0 1 1 0 1 0 1 0 0 0 0 0
[2332] 1 1 0 1 0 0 0 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 0 1 1 1 1 0 0 1 0 0 0 1
[2369] 0 0 1 0 0 1 0 1 1 1 1 0 1 1 0 1 0 1 1 1 0 1 0 0 0 1 1 0 1 0 0 1 1 0 1 1 1
[2406] 1 0 1 0 0 1 1 1 1 0 1 0 1 1 1 0 0 1 0 1 1 1 0 1 1 0 1 1 0 0 0 1 0 0 1 0 0
[2443] 1 1 1 1 0 1 1 1 1 0 0 1 1 0 0 1 1 1 1 1 1 0 0 0 1 1 1 1 0 1 0 0 1 1 1 1 0
[2480] 1 1 1 1 1 0 0 0 1 0 0 0 0 0 1 0 0 1 1 0 1 0 0 0 0 1 0 0 0 0 0 0 1 1 1 0 0
[2517] 0 1 1 0 0 0 0 0 1 0 0 1 1 1 1 1 0 0 1 0 0 0 1 0 0 1 0 1 1 0 1 0 0 1 0 1 1
[2554] 0 0 1 1 0 0 1 0 0 1 0 0 1 0 1 1 1 0 0 1 1 0 0 0 1 0 0 1 0 0 1 0 1 0 1 1 1
[2591] 1 0 1 0 1 1 0 1 0 1 1 0 0 0 1 1 0 1 1 1 0 0 1 0 0 1 0 1 1 1 1 1 1 0 1 1 0
[2628] 1 0 0 1 0 1 1 1 0 1 1 1 1 0 1 1 0 1 1 0 1 1 1 0 1 1 1 0 1 1 1 1 1 0 1 0 1
[2665] 1 1 0 1 1 0 0 1 1 0 0 0 1 1 1 1 1 0 0 1 0 1 1 1 0 1 0 1 1 0 1 1 1 1 1 1 1
[2702] 0 0 0 0 1 1 1 0 1 0 0 0 1 0 1 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 0 1 0 0
[2739] 0 1 1 1 0 1 0 0 1 0 0 0 0 1 0 0 0 1 0 1 1 1 1 0 0 0 1 1 0 1 1 1 0 0 0 0 0
[2776] 1 1 0 0 1 0 0 0 0 1 1 1 0 1 1 1 1 0 1 0 0 1 0 0 0 1 1 1 0 0 0 1 1 1 0 1 1
[2813] 0 0 0 0 0 1 0 1 1 1 0 1 1 1 0 0 0 1 1 1 1 1 0 1 0 1 0 1 0 1 1 1 1 1 0 0 1
[2850] 1 1 1 1 0 1 1 0 0 1 1 0 0 0 1 0 1 0 1 1 0 1 0 0 0 1 1 0 0 1 1 1 1 0 1 0 0
[2887] 0 0 0 1 0 0 1 1 0 0 0 1 1 1 1 1 1 0 1 0 1 0 0 0 0 0 0 0 1 1 1 0 1 1 1 0 1
[2924] 0 1 1 0 1 0 0 1 1 1 0 1 0 0 1 1 0 1 1 1 1 1 1 0 0 0 1 0 0 0 0 1 1 0 1 0 0
[2961] 1 1 1 1 0 1 1 0 1 1 0 0 0 1 1 1 0 1 0 0 1 1 1 0 1 1 0 1 0 1 1 0 1 0 0 1 1
[2998] 0 0 1 1 1 1 1 1 0 0 0 1 0 1 1 1 1 0 0 0 1 0 1 1 1 1 1 1 1 0 0 0 1 0 1 0 0
[3035] 1 1 1 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 0 1 1 1 0 1 0 1 0 1 0 1 1 1 0 0 0 0 0
[3072] 0 1 0 1 0 1 0 1 1 1 1 0 1 0 1 1 1 1 1 1 0 1 1 0 1 0 0 0 0 1 1 1 0 0 0 1 1
[3109] 1 0 1 1 0 0 1 1 1 1 1 1 0 0 1 0 1 1 1 0 0 0 0 0 1 1 0 0 1 0 1 0 0 0 1 0 1
[3146] 1 1 1 0 1 1 1 1 1 0 0 1 0 0 1 1 1 1 0 1 1 0 0 1 0 1 0 0 0 0 0 1 0 1 1 0 0
[3183] 0 0 0 1 1 1 0 1 0 1 0 0 1 0 1 1 1 1 1 1 1 1 0 1 0 1 0 1 0 0 1 1 1 0 1 1 1
[3220] 0 0 0 0 1 0 0 0 0 1 1 0 1 0 0 0 0 1 1 1 0 1 0 0 0 1 0 0 0 1 0 1 0 1 1 0 1
[3257] 0 0 1 1 0 1 1 0 1 1 1 1 1 0 1 1 1 1 1 1 0 1 0 1 0 1 0 1 0 1 1 1 1 1 0 1 0
[3294] 0 1 1 1 1 0 0 0 1 0 1 0 0 1 1 1 1 0 0 1 0 0 0 0 0 0 1 0 0 0 1 1 1 0 1 0 1
[3331] 1 1 1 0 0 0 0 0 1 0 1 0 1 1 0 1 1 0 1 1 1 0 1 0 0 0 0 0 1 0 1 1 0 1 0 1 0
[3368] 0 1 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1 0 0 0 0 1 1 1 1 1 0 0 1
[3405] 0 1 0 0 1 1 0 1 0 1 1 0 1 1 1 1 0 0 1 0 0 0 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1
[3442] 1 0 1 1 0 1 0 1 0 0 0 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 0 0 0 1 1 1 1 1 1
[3479] 0 1 1 1 1 1 1 0 0 0 1 0 1 1 0 1 0 1 1 0 1 0 0 0 0 0 1 0 0 1 0 0 1 1 1 1 1
[3516] 1 1 1 1 1 1 1 1 1 0 0 1 1 0 1 1 1 1 1 0 0 0 1 1 0 1 0 1 1 1 1 1 0 1 1 0 1
[3553] 0 0 1 1 0 0 0 1 1 1 0 0 1 0 1 1 0 1 1 1 0 1 0 0 1 0 0 1 1 1 1 0 1 1 1 1 1
[3590] 0 1 0 0 0 0 1 0 1 0 0 1 1 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 1 1 1 1 1 0 1 0 1
[3627] 1 1 1 0 0 0 1 1 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 1 0 1 1 0 0 1 1 1 1 1 0
[3664] 0 0 0 1 1 1 0 1 1 1 1 0 0 1 1 0 0 0 1 0 0 0 0 0 0 1 1 1 1 1 1 1 0 1 0 0 1
[3701] 1 1 1 0 0 1 0 1 1 0 1 0 1 0 1 1 0 1 1 0 1 0 0 0 1 1 1 1 0 0 0 0 0 1 1 1 0
[3738] 1 0 1 1 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 0 0 0 0 1 1 1 1 0 1 1 0 1 1 0
[3775] 0 0 1 1 0 0 0 0 1 0 1 1 1 1 0 1 0 1 0 0 0 1 0 0 1 0 1 1 0 1 1 0 0 1 0 0 0
[3812] 0 1 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 0 0 0 1 1 0 1 1 1 1 1 0 1 0 1 1
[3849] 0 1 1 1 1 0 0 1 1 1 1 0 1 1 1 0 1 1 1 1 1 0 1 1 1 0
Levels: 0 1

$cut
[1] 0.3814174

>  ROCtest(mylogit, fold=10, type="ROC")
$plot

$Observed
   [1] 1 0 0 1 0 0 1 1 1 1 0 1 1 0 0 0 0 0 0 1 0 1 0 1 0 1 0 0 0 0 1 0 0 0 0 1 1
  [38] 1 0 1 1 0 1 0 1 1 0 1 0 0 1 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 0 1 1 0 0
  [75] 0 0 1 0 0 0 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 1 0 1 0 1 1 0 0 1 1 0 0 1 0 0
 [112] 0 0 0 0 0 0 0 1 1 0 0 1 1 0 0 0 1 0 1 0 0 1 0 0 1 0 0 1 0 1 1 1 0 0 0 1 0
 [149] 0 1 0 0 0 0 0 0 1 0 1 1 1 1 1 0 1 0 1 1 1 0 1 0 0 0 0 1 0 0 1 0 0 0 0 1 0
 [186] 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 1 0 1 0 0 0 0 0 0 1 1 1 1 0 1 0 0 1 1 0 1 1
 [223] 1 1 0 0 0 0 1 1 0 1 1 0 1 0 0 0 0 1 0 1 1 1 0 0 1 0 0 0 0 1 0 0 0 1 1 0 1
 [260] 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 1 0 0 0 1 0 1 1 0 0 0 0 0 0 1
 [297] 1 0 1 0 0 1 0 0 0 0 1 1 0 0 1 1 1 0 1 1 0 0 1 0 1 0 0 1 1 0 0 0 1 0 0 1 0
 [334] 1 1 0 0 1 1 0 1 0 0 0 0 0 1 0 1 0 1 0 1 0 1 1 1 0 1 0 1 0 1 0 1 0 0 0 0 0
 [371] 0 0 0 1 0 0 0 0 1 1 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 1
 [408] 0 0 0 1 0 1 0 0 0 1 1 0 1 0 0 0 0 0 1 1 1 0 0 1 1 0 0 1 0 0 0 0 0 0 0 0 1
 [445] 0 1 0 1 0 0 0 0 0 1 0 0 0 1 1 0 0 1 1 0 0 1 0 1 1 0 1 0 1 0 1 0 1 0 0 1 1
 [482] 1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0
 [519] 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 1 0 0 1 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0
 [556] 0 0 0 1 0 0 1 1 0 0 0 0 1 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0
 [593] 1 1 0 0 0 1 0 1 0 0 1 0 1 1 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 0 0
 [630] 0 0 0 1 1 1 1 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 0 0 1
 [667] 0 1 1 0 0 1 1 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 1 0 0 1 0
 [704] 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 1 0 0 1 0 0 1 0 1 0 0 1 1 0
 [741] 0 0 0 0 1 0 0 1 1 0 0 0 0 1 1 0 1 0 0 1 1 0 1 1 0 0 0 1 1 0 0 0 0 0 1 1 0
 [778] 1 0 0 1 1 0 0 1 1 0 0 1 0 0 0 0 0 0 1 1 1 0 1 1 0 0 1 0 0 1 1 0 0 1 0 0 1
 [815] 1 1 1 0 0 1 0 0 1 0 0 1 1 1 1 0 1 0 0 0 0 0 1 1 1 0 0 0 1 1 1 0 0 1 1 0 1
 [852] 1 0 1 1 0 1 1 0 1 0 0 0 1 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 0 0 0 1 1 0 0 0 0
 [889] 1 1 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 0 1 1 0 0 1 1 0 1 0 0
 [926] 0 0 1 0 0 1 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 1 1 0 0 1 0 1 1 1
 [963] 1 1 1 0 0 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 0 1 1 0 1 0 0 1 1 1 1 0 0 0
[1000] 0 0 0 1 1 1 0 1 1 0 0 0 1 1 1 0 1 0 0 1 0 0 1 0 1 1 1 1 0 0 1 1 1 1 0 0 0
[1037] 1 0 0 0 1 0 0 0 0 0 1 0 0 1 0 1 1 1 0 0 1 1 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0
[1074] 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0
[1111] 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1 1 1 0 0 0 1 1 0 0 0 0 0 0
[1148] 1 0 0 0 0 1 0 1 0 0 1 1 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 1 0 0 1 0 0 0
[1185] 1 0 0 0 1 0 0 1 0 0 1 1 1 0 1 1 0 0 0 0 1 1 1 1 1 0 0 0 1 1 0 1 0 1 0 0 0
[1222] 0 0 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 1 1 0 0 0 0 1 1 0 1 0 1 0
[1259] 0 1 0 0 1 1 0 1 1 0 0 1 1 0 0 0 1 1 1 1 0 0 1 0 0 0 1 0 1 0 1 0 1 0 1 0 0
[1296] 0 1 0 0 0 0 0 0 1 0 0 1 0 0 0 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 1 0 1 1 0 0 0
[1333] 1 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 1 1 0 1 0 1 1 0 0 0
[1370] 1 1 1 0 1 1 0 1 0 1 1 0 1 1 0 1 0 0 0 0 1 0 1 0 0 1 0 1 0 0 0 1 0 1 0 1 1
[1407] 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 1 0 0 1 0 1 0 0 1 1 1 1 0 0 1
[1444] 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 0 1 1 0 1 1 0 0 0 0 0 0 1 0 1
[1481] 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 1 0 1 1 0 1 0 0 1
[1518] 1 1 1 0 1 0 0 0 0 0 0 1 0 0 1 1 0 1 1 0 1 0 0 1 1 1 0 0 1 0 1 1 1 1 0 1 0
[1555] 0 0 0 0 0 0 0 0 1 0 1 0 0 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0
[1592] 0 1 0 0 0 1 0 1 1 1 1 0 0 0 1 1 1 0 0 0 0 0 1 0 1 0 0 0 0 1 0 1 1 1 0 1 1
[1629] 0 0 0 0 0 1 1 1 0 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 1 1 0 0 0 0 1 0 1 1 0
[1666] 0 1 0 1 0 1 1 0 1 0 0 0 0 1 0 1 1 1 0 0 1 0 0 1 0 1 1 0 0 0 1 1 0 0 1 0 1
[1703] 1 0 1 0 0 0 0 1 1 1 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 1 0 1 0 0 0 1 1 1 1 0
[1740] 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 1 0 1 0 1 1 0 0 1 0 1 1 0 0
[1777] 1 0 0 1 1 0 0 0 1 0 0 0 1 0 0 1 1 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0 1
[1814] 0 1 0 0 0 0 0 0 1 1 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0
[1851] 0 1 1 0 1 0 0 0 0 0 0 0 1 1 0 1 1 1 1 1 0 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0
[1888] 0 0 1 0 0 1 1 0 1 1 0 1 0 0 0 1 0 0 1 1 1 0 0 1 0 0 1 0 0 0 1 1 0 0 0 0 0
[1925] 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0
[1962] 1 1 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 1 1 0 0 0 1 1 1 0
[1999] 0 1 1 0 0 0 1 0 0 1 0 0 0 1 1 1 1 1 1 1 0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 1
[2036] 0 1 1 1 1 0 0 1 0 0 0 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 1 0 0 1 1 1 0 0
[2073] 0 1 1 1 0 0 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 0 1 1 0 0 0 1 1 0 1 0 1 1 1 0 0
[2110] 1 0 0 0 1 1 0 1 1 0 1 0 0 1 1 1 1 0 1 1 0 1 0 1 0 1 1 0 0 0 1 1 1 0 0 1 1
[2147] 0 1 1 0 0 0 0 0 0 0 1 0 0 1 0 1 1 1 0 0 1 1 0 0 0 0 0 0 1 0 0 1 0 0 1 1 0
[2184] 1 0 0 1 0 1 1 1 0 0 0 1 0 0 0 0 0 0 1 1 1 0 0 1 1 0 1 0 0 0 0 1 0 0 0 0 0
[2221] 0 0 1 0 1 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 1 1 0 1 1 0 0 0
[2258] 0 1 1 1 0 0 1 1 0 0 0 1 0 1 1 1 1 1 0 0 0 0 1 0 0 1 1 0 1 0 0 0 1 1 0 1 1
[2295] 1 1 0 0 0 1 1 1 0 0 1 0 1 1 1 1 0 1 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0 0 0 0 0
[2332] 0 1 0 0 0 0 1 0 0 0 1 1 0 0 0 0 1 1 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 1 0
[2369] 1 0 0 0 1 1 1 1 0 1 0 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 1 0 0 0 1 0 1 0 0 0
[2406] 0 1 1 1 1 1 0 0 1 0 0 0 0 0 0 0 0 1 1 1 0 1 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0
[2443] 1 1 1 1 1 0 1 1 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 1 0 0 1 0 0
[2480] 1 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 0 1 1 0 0 1 0 0 1 1 0 0 1 0 0 0 0 0
[2517] 0 1 1 1 1 0 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 1 1 1 0 1 0
[2554] 0 1 0 1 1 1 0 1 0 0 0 0 1 0 0 1 0 0 1 1 0 0 0 1 1 1 0 0 0 1 0 0 0 1 1 0 0
[2591] 0 1 0 1 0 1 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 1 0
[2628] 1 0 1 0 1 1 1 1 0 1 0 0 1 1 0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 1 0 0 1 0
[2665] 0 0 0 0 0 0 0 1 1 1 0 1 1 1 1 0 0 0 1 0 0 0 1 1 0 0 1 1 1 1 0 1 0 1 0 0 0
[2702] 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 1 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 1
[2739] 0 0 0 0 0 0 1 1 0 0 0 1 0 1 0 1 0 1 0 1 1 0 1 1 1 1 0 0 0 0 0 1 0 0 1 0 0
[2776] 0 1 0 0 0 1 0 0 0 0 0 0 1 1 1 0 0 0 0 1 1 1 0 0 0 1 1 0 0 0 0 1 1 0 0 0 0
[2813] 1 0 0 0 1 1 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 1 0 1 0 1 0 0 1 0 1 1 1 0
[2850] 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 1 0 0 0 1 0 1 0 0 0 0 1 1 0 0 0 0 1 1 1 0 0
[2887] 1 0 0 0 0 1 1 1 0 1 1 0 0 1 0 0 1 0 1 1 0 0 0 1 1 1 0 0 0 0 0 0 1 0 0 1 1
[2924] 0 0 1 1 1 0 0 1 0 0 0 1 0 0 0 1 1 0 0 1 1 1 1 0 1 1 0 0 0 0 0 0 1 0 0 0 0
[2961] 1 1 1 0 0 1 1 1 0 0 0 0 0 1 1 0 1 0 1 1 1 1 0 0 1 1 0 0 1 0 1 0 0 0 1 1 1
[2998] 1 0 0 0 1 0 1 0 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0 0 1 1 0 0 1 0 0 0 0 0 0 0
[3035] 0 0 0 0 0 0 1 1 0 0 0 0 0 1 0 1 0 0 0 0 1 0 0 1 1 0 0 0 0 1 0 1 1 1 1 1 0
[3072] 0 0 0 0 1 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0
[3109] 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 1 1 0 0
[3146] 1 1 0 1 0 0 0 0 1 0 1 1 1 0 0 0 0 1 1 0 0 0 0 0 1 0 1 0 0 1 1 0 0 0 0 0 0
[3183] 1 0 1 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 1 1 0 1 1 0 0 1 1 1 1 0
[3220] 0 1 0 0 0 1 0 1 1 1 0 0 1 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 1 0 1 1 0 0 1 0 0
[3257] 0 1 1 0 0 0 0 1 1 0 1 0 1 0 1 0 0 0 0 0 1 0 0 1 0 1 0 0 1 0 1 1 1 1 0 1 1
[3294] 0 1 1 0 0 1 0 1 1 1 0 1 0 1 1 1 0 0 1 1 0 1 0 0 0 1 0 0 1 1 0 0 1 0 1 1 0
[3331] 1 0 0 1 0 0 0 0 1 1 0 0 1 1 1 1 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0
[3368] 1 1 0 0 1 0 0 0 0 1 1 0 0 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 1 0 1 1 1
[3405] 0 0 1 1 0 1 0 0 0 1 0 1 1 1 0 1 1 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0
[3442] 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 0 1 1 0 1 1
[3479] 1 0 0 0 1 1 0 0 1 1 0 1 0 0 1 0 1 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0 0
[3516] 0 0 1 1 1 1 0 0 1 0 0 0 1 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0
[3553] 1 0 1 1 0 1 1 0 0 0 0 0 0 1 1 1 0 1 0 0 0 1 1 1 0 1 0 1 0 0 1 1 0 1 0 0 0
[3590] 0 0 1 0 1 0 1 0 1 0 0 1 1 1 1 0 0 0 1 1 0 0 0 1 1 0 1 1 1 1 1 0 0 0 1 1 0
[3627] 0 0 1 1 0 1 0 0 1 1 1 1 0 1 1 0 0 1 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 1
[3664] 0 0 0 0 0 0 0 0 1 1 0 0 0 1 1 1 1 0 0 0 1 1 1 1 1 0 0 0 0 0 1 0 1 0 0 1 1
[3701] 0 1 1 1 1 0 1 1 0 1 0 0 0 1 0 0 0 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 1 0 0 0 0
[3738] 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 1 0 0 1 0 1 0
[3775] 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0
[3812] 0 0 0 1 0 1 0 0 0 1 0 1 1 1 0 0 0 0 0 1 0 1 1 0 0 0 1 1 0 0 0 0 0 1 0 1 0
[3849] 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 0 1 0 0
Levels: 0 1

$Predicted
   [1] 0 1 1 1 0 0 0 1 1 1 0 1 1 0 0 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 0 0 1 0 1 1
  [38] 1 1 1 1 0 1 0 0 0 1 1 0 0 1 0 1 1 0 1 1 0 0 0 1 0 1 0 1 0 1 1 0 1 1 1 0 0
  [75] 1 0 1 0 1 0 1 0 0 0 0 1 1 0 0 1 1 0 0 0 1 0 1 1 1 0 1 1 0 0 1 1 0 0 1 1 0
 [112] 0 0 0 1 0 1 1 1 1 0 0 1 1 0 1 0 1 0 1 0 0 1 0 1 1 1 0 1 0 1 1 0 1 1 1 1 0
 [149] 0 1 0 0 0 0 0 1 1 0 1 1 1 1 1 0 1 1 1 1 1 0 1 1 0 0 1 1 1 0 1 0 1 1 0 1 0
 [186] 1 0 0 0 1 0 0 0 1 1 1 1 1 0 0 1 0 1 0 1 0 0 0 1 1 1 1 1 1 1 1 0 0 1 1 1 1
 [223] 0 1 0 0 0 0 1 1 0 1 1 0 1 0 0 1 0 1 0 0 1 0 0 1 0 0 0 0 0 1 0 1 0 1 1 1 1
 [260] 0 0 0 1 1 1 1 1 0 0 0 1 0 0 1 1 0 1 1 1 1 1 1 0 0 1 1 1 1 1 0 1 0 1 0 1 1
 [297] 1 0 1 0 1 1 0 1 0 0 1 1 0 0 1 1 1 0 0 0 1 0 1 1 1 0 0 1 1 0 1 1 1 1 0 1 1
 [334] 1 1 0 0 1 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 1 1 1 0 0 1 1 1 0 1 1 1 1 0 1 0 0
 [371] 0 0 1 0 1 0 0 0 1 1 0 1 1 1 1 1 0 0 0 0 1 0 0 0 0 0 1 0 1 1 1 0 0 1 0 1 0
 [408] 0 0 0 1 0 1 1 1 0 1 1 1 1 0 0 1 1 1 1 0 1 0 1 1 1 0 0 1 0 0 1 0 0 1 1 1 1
 [445] 1 1 0 1 0 1 1 0 0 1 1 0 1 0 1 1 0 1 1 0 0 1 0 1 1 0 0 0 1 0 1 0 1 0 0 0 1
 [482] 1 1 0 0 1 1 1 0 1 0 1 1 0 1 0 1 0 0 0 0 1 0 0 1 0 0 0 0 0 1 0 1 0 0 1 0 0
 [519] 1 0 0 0 0 0 1 0 0 1 1 0 0 1 1 0 0 0 1 0 0 1 0 0 0 1 1 1 1 0 0 1 1 0 1 0 0
 [556] 0 0 0 1 0 0 1 0 0 0 0 0 1 1 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 0 0 0 1 0 1
 [593] 1 1 0 0 1 1 0 1 0 0 1 1 0 1 1 1 0 0 1 0 0 1 1 0 0 1 0 0 0 1 1 1 1 0 0 0 0
 [630] 1 1 0 1 1 1 1 1 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 1 1 0 0 0 0 1 1 0 1 1 0 0 1
 [667] 0 1 1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 0 1 0 1 0 0 1 0 1 1 1 1 0 1 1 1 0 0 1 0
 [704] 0 0 0 0 0 1 0 0 0 0 0 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 1 0 1 1 0 1 0 1 1 0 0
 [741] 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1 0 1 0 0 1 1 0 1 1 0 1 0 1 1 1 1 1 0 1 1 0 0
 [778] 0 0 0 1 1 1 0 1 1 1 0 0 1 0 1 0 0 0 1 1 1 1 1 1 0 0 1 0 0 1 1 0 0 1 0 1 0
 [815] 0 0 1 1 0 1 1 0 1 0 0 1 1 1 1 1 1 0 1 1 0 0 1 1 1 1 1 0 0 1 1 0 0 1 1 1 1
 [852] 1 1 1 1 0 1 1 0 1 0 0 1 1 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1
 [889] 1 1 0 0 1 0 0 1 0 0 0 0 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 1 0 1 1 0 1 0 1
 [926] 0 0 1 0 0 1 0 0 1 1