Visualization for Model Selection/Validation

Description

Function for plotting the cross-validated tuning profiles of a PRSP object. It uses the user's choice of statistics among the Log Hazard Ratio (LHR), Log-Rank Test (LRT) or Concordance Error Rate (CER) as a function of the model tuning parameter, that is, the optimal number of peeling steps of the peeling sequence (inner loop of our PRSP algorithm).

Usage

  plot_profile(object,
               main = NULL, 
               xlab = "Peeling Steps", 
               ylab = "Mean Profiles",
               add.sd = TRUE, 
               add.legend = TRUE, 
               add.profiles = TRUE,
               pch = 20, 
               col = 1, 
               lty = 1, 
               lwd = 2, 
               cex = 2,
               device = NULL, 
               file = "Profile Plot", 
               path=getwd(), 
               horizontal = FALSE, 
               width = 8.5, 
               height = 5.0, ...)

Arguments

Details

Model tuning is done by applying the optimization criterion defined by the user's choice of specific statistic. The goal is to find the optimal value of the number of steps by maximization of LHR or LRT, or minimization of CER.

Currently, this is done internally for visualization purposes, but it will ultimately offer the option to be done interactively with the end-user as well for parameter choosing/model selection.

Value

Invisible. None. Displays the plot(s) on the specified device.

Note

End-user plotting function.

Author(s)

• "Jean-Eudes Dazard, Ph.D." jxd101@case.edu

• "Michael Choe, M.D." mjc206@case.edu

• "Michael LeBlanc, Ph.D." mleblanc@fhcrc.org

• "Alberto Santana, MBA." ahs4@case.edu

Maintainer: "Jean-Eudes Dazard, Ph.D." jxd101@case.edu

Acknowledgments: This project was partially funded by the National Institutes of Health NIH - National Cancer Institute (R01-CA160593) to J-E. Dazard and J.S. Rao.

References

• Dazard J-E., Choe M., LeBlanc M. and Rao J.S. (2015). "Cross-validation and Peeling Strategies for Survival Bump Hunting using Recursive Peeling Methods." Statistical Analysis and Data Mining (in press).

• Dazard J-E., Choe M., LeBlanc M. and Rao J.S. (2014). "Cross-Validation of Survival Bump Hunting by Recursive Peeling Methods." In JSM Proceedings, Survival Methods for Risk Estimation/Prediction Section. Boston, MA, USA. American Statistical Association IMS - JSM, p. 3366-3380.

• Dazard J-E., Choe M., LeBlanc M. and Rao J.S. (2015). "R package PRIMsrc: Bump Hunting by Patient Rule Induction Method for Survival, Regression and Classification." In JSM Proceedings, Statistical Programmers and Analysts Section. Seattle, WA, USA. American Statistical Association IMS - JSM, (in press).

• Dazard J-E. and J.S. Rao (2010). "Local Sparse Bump Hunting." J. Comp Graph. Statistics, 19(4):900-92.

Examples

#===================================================
# Loading the library and its dependencies
#===================================================
library("PRIMsrc")

#=================================================================================
# Simulated dataset #1 (n=250, p=3)
# Non Replicated Combined Cross-Validation (RCCV)
# Peeling criterion = LRT
# Optimization criterion = LRT
#=================================================================================
CVCOMB.synt1 <- sbh(dataset = Synthetic.1, 
                    cvtype = "combined", cvcriterion = "lrt",
                    B = 1, K = 5, 
                    vs = TRUE, cpv = FALSE, 
                    decimals = 2, probval = 0.5, 
                    arg = "beta=0.05,
                           alpha=0.1,
                           minn=10,
                           L=NULL,
                           peelcriterion="lr"",
                    parallel = FALSE, conf = NULL, seed = 123)

plot_profile(object = CVCOMB.synt1, 
             main = "Cross-validated tuning profiles for model #1",
             xlab = "Peeling Steps", ylab = "Mean Profiles",
             pch=20, col="black", lty=1, lwd=2, cex=2,
             add.sd = TRUE, add.legend = TRUE, add.profiles = TRUE,
             device = NULL, file = "Profile Plot", path=getwd(),
             horizontal = FALSE, width = 8.5, height = 5.0)

Results

R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(PRIMsrc)
Loading required package: parallel
Loading required package: survival
Loading required package: Hmisc
Loading required package: lattice
Loading required package: Formula
Loading required package: ggplot2

Attaching package: 'Hmisc'

The following objects are masked from 'package:base':

    format.pval, round.POSIXt, trunc.POSIXt, units

Loading required package: glmnet
Loading required package: Matrix
Loading required package: foreach
Loaded glmnet 2.0-5

Loading required package: MASS
PRIMsrc 0.6.3
Type PRIMsrc.news() to see new features, changes, and bug fixes

> png(filename="/home/ddbj/snapshot/RGM3/R_CC/result/PRIMsrc/plot_profile.Rd_%03d_medium.png", width=480, height=480)
> ### Name: plot_profile
> ### Title: Visualization for Model Selection/Validation
> ### Aliases: plot_profile
> ### Keywords: Exploratory Survival/Risk Analysis Survival/Risk Estimation &
> ###   Prediction Non-Parametric Method Cross-Validation Bump Hunting
> ###   Rule-Induction Method
> 
> ### ** Examples
> 
> #===================================================
> # Loading the library and its dependencies
> #===================================================
> library("PRIMsrc")
> 
> #=================================================================================
> # Simulated dataset #1 (n=250, p=3)
> # Non Replicated Combined Cross-Validation (RCCV)
> # Peeling criterion = LRT
> # Optimization criterion = LRT
> #=================================================================================
> CVCOMB.synt1 <- sbh(dataset = Synthetic.1, 
+                     cvtype = "combined", cvcriterion = "lrt",
+                     B = 1, K = 5, 
+                     vs = TRUE, cpv = FALSE, 
+                     decimals = 2, probval = 0.5, 
+                     arg = "beta=0.05,
+                            alpha=0.1,
+                            minn=10,
+                            L=NULL,
+                            peelcriterion="lr"",
+                     parallel = FALSE, conf = NULL, seed = 123)

Survival dataset provided.

Requested single 5-fold cross-validation without replications 
Cross-validation technique:  COMBINED 
Cross-validation criterion:  LRT 
Variable pre-selection: TRUE 
Computation of permutation p-values: FALSE 
Peeling criterion:  LRT 
Parallelization: FALSE 

Pre-selection of covariates and determination of directions of peeling... 
Pre-selected covariates:
X1 X2 X3 
 1  2  3 
Directions of peeling at each step of pre-selected covariates:
X1 X2 X3 
 1 -1 -1 
Fitting and cross-validating the Survival Bump Hunting model using the PRSP algorithm ... 
replicate : 1
seed : 123
Fold : 1
Fold : 2
Fold : 3
Fold : 4
Fold : 5
Success! 1 (replicated) cross-validation(s) has(ve) completed 
Generating cross-validated optimal peeling lengths from all replicates ...
Generating cross-validated box memberships at each step ...
Generating cross-validated box rules for the pre-selected covariates at each step ...
Generating cross-validated modal trace values of covariate usage at each step ...
Covariates used for peeling at each step, based on covariate trace modal values:
X1 X2 
 1  2 
Generating cross-validated box statistics at each step ...
Finished!
> 
> plot_profile(object = CVCOMB.synt1, 
+              main = "Cross-validated tuning profiles for model #1",
+              xlab = "Peeling Steps", ylab = "Mean Profiles",
+              pch=20, col="black", lty=1, lwd=2, cex=2,
+              add.sd = TRUE, add.legend = TRUE, add.profiles = TRUE,
+              device = NULL, file = "Profile Plot", path=getwd(),
+              horizontal = FALSE, width = 8.5, height = 5.0)
Device:  2 
> 
> 
> 
> 
> 
> dev.off()
null device 
          1 
>