DEEP Software



Download and Install the DEEP R package

The DEEP R package is hosted on the Columbia University Technology Ventures site. In order to download the DEEP package, you need to have the Devtools package installed


The primary software package for DEEP is called Integrated DEEP, and it:
  • Includes the DEEP Survey Tools and the DEEP Analysis Tools (for R).
  • Supports Qualtrics and LimeSurvey.
  • Can be inserted into any point in your existing survey.
  • Can be customized for either DEEP Time or DEEP Risk.
  • Can have up to 12 questions.
If you need to use DEEP on a platform other than Qualtrics or LimeSurvey, or need the DEEP 20-question version.


Run the following code in R to install the dependencies and the DEEP package:
# Install dependencies
packageDependencies <- c("dplyr", "XML", "jsonlite", "MSBVAR", "MCMCpack", "httr", "RCurl")
install.packages(packageDependencies)

# install DEEP package
install.packages("devtools")
devtools::install_github("ColumbiaCDS/DEEPAnalysis")

# Load DEEP
library(DEEP)

How it works





Add DEEP to your survey



Data analysis

Download your data

Qualtrics
  1. Click the "View Results" tab and view your survey.
  2. Click the "Download Data" button.
  3. Select CSV and download.

LimeSurvey
  1. Go to your survey. On the menu bar, click "Responses and Statistics" with the pie chart icon.
  2. Then, click the "Export results to application" on the menu with the database icon.
  3. For the format, select "CSV" (not "CSV with all answers part").
  4. Click "Export Data".

You'll see that all of the DEEP data is compacted into a single column. This is normal. The DEEP R package transforms the data to the proper format.

Analyzing your data

Before starting, open your R Console and make sure the DEEP package is loaded.  

Examples
  • There are 2 ways to run the analysis: using the aggregated function deepRun() or run the two steps separately (run deepTransform() first, then run deepRiskHB or deepTimeHB). Below are examples of what the R code to start the data analysis should look like. 
  • To understand the functions and output, read the description below.

1. Use Aggregated Function

# Risk
deepRun(DEEPtype = "risk",WD = "C:/YourUser/Documents/OutputFolder/", file_path = "C:/YourUser/Documents/OutputFolder/
LS_DEEP_RISK8_sample.csv
", filter_by = "DeepRisk8"
)

# Time
deepRun(DEEPtype = "time",WD = "C:/YourUser/Documents/OutputFolder/", file_path = "C:/YourUser/Documents/OutputFolder/
LS_DEEP_TIME8_sample.csv
", filter_by = "DEEPTIME8"
)

2. Use Separate Functions
If for any reason that the above code doesn't work, you can try the following R code:

# Risk
deepTransform("time", WD = 
"C:/YourUser/Documents/OutputFolder/"
, file_path = 
"C:/YourUser/Documents/OutputFolder/
LS_DEEP_RISK8_sample.csv
", 
filter_by = "DeepRisk8")
deepRiskHB()

# Time
deepTransform("time", WD = 
"C:/YourUser/Documents/OutputFolder/"
, file_path = 
"C:/YourUser/Documents/OutputFolder/
LS_DEEP_TIME8_sample.csv
", 
filter_by = "DEEPTIME8")
deepTimeHB()


deepRun(DEEPtype, WD, file_path, filter_by, collaborate)
  • Description: An aggregated function to run analysis. It first calls DEEP Transformation (deepTransform) and then runs Hierarchical Bayes algorithm (deepRiskHB/deepTimeHB).
  • Preparation: Have the path of your csv data file downloaded from LimeSurvey or Qualtrics as the input. Here we provide an example file (LS_DEEP_RISK8_sample.csv) for you to familiarize the process.
  • Arguments: Same to deepTransform().
  • Output: Everyting from deepTransform and deepRiskHB/deepTimeHB.


deepTransform(DEEPtype, WD, file_path, filter_by, collaborate)
  • Description: Transforms JSON DEEP output into a readable dataframe
  • Preparation: Have the path of your csv data file downloaded from LimeSurvey or Qualtrics as the input. Here we provide an example file (LS_DEEP_RISK8_sample.csv) for you to familiarize the process.
  • Arguments
    1. DEEPtype: Refers to the type of DEEP question you used in your analysis. It accepts either "risk" or "time" (not case sensitive).If you have more than one DEEP question, please specify the DEEPtype for the question in the "filter_by" field.
    2. WD: Refers to the working directory. This is the folder/path where all the DEEP output will be exported. If you have multiple DEEP questions we recommend you create a folder for each question since any output file with the same name will be overwritten during analysis.  
    3. file_path: Refers to the path of the file you exported from LimeSurvey or Qualtrics. If you do not specify a file_path, a Graphical User Interface will appear to allow you to manually select the file from you computer. 
    4. filter_by: Used to specify the "question identifier" ("question name" used in Qualtrics or LimeSurvey) in case you have multiple DEEP surveys and/or more than one DEEP type in the Qualtrics/Limesurvey output. The DEEP package can only analyze a single DEEP question at once. Failure to specify the question identifier will produce an error if more than one DEEP question is found. This is not case-sensitive. 
  • Output
    • The first 3 files on the list (bold) are what you need for the next step (deepRiskHB or deepTimeHB).
      RiskTimeDescription
      adaptive_questions_risk_1.csvadaptive_questions_time_1.csvcontains the path of adaptive questions
      choices_risk_1.csvchoices_time_1.csvcontains participants choices (should be either 1 or 2)
      serials_risk_1.csvserials_time_1.csvcontains the Qualtrics or Limesurvey ID matched with the serial assigned by the R script
      Qtiming_DEEPrisk.csvQtiming_DEEPtime.csvcontains the response time for each question as columns
      serials_respid.csvserials_respid.csvcontains the Qualtrics or Limesurvey ID matched with the serial assigned by the R script
      Unshrunken_Parameter_Estimates.csvUnshrunken_Parameter_Estimates.csvwhich contains the pre-Hierarchical Bayes estimates.
      V1= Unshrunken_beta (present bias), V2 = Unshrunken_daily_discount_rate (Note: The old wiki page called this parameter delta, which is a misnomer). 


deepRiskHB(iterUntilSave, iterSaved) and deepTimeHB(iterUntil, iterSaved)
  • Description: Estimation of Risk/Time Preference Parameters Using Hierarchical Markov Chain Mote Carlo Bayesian Framework
  • Preparation: Have the following files in your working directory (for example, setwd("C:/YourUser/Documents/Risk")). They are the output of deepTransform() function.
    • RiskTime Description 
      adaptive_question_risk_1.csvadaptive_question_time_1.csvcontains the path of adaptive questions
      choices_risk_1.csvchoices_time_1.csvcontains participants choices (should be either 1 or 2)
      serial_risk_1.csvserial_time_1.csvcontains the Qualtrics or Limesurvey ID matched with the serial assigned by the R script
  • Arguments: The total number of iteration is the sum of the two arguments
    1. iterUntilSave: number of iteration before starting to save the result, default to 50000 in deepRiskHB and 10000 in deepTimeHB (recommended).
    2. iterSaved: number of iteration saved, default to 50000 in deepRiskHB and 40000 in deepTimeHB (recommended).
  • Output:
    •  Risk Time Description
      gambles1_risk_adaptive_r.csv

      		gambles1_time_adaptive_r.csv

      		contains the information of the chosen option (gamble1).
      Risk:       V1 and V3= the outcome of gamble 1, V2 and V4 = the probability of the corresponding outcome.
      Time: V1 = the outcome of gamble 1, V2 = delay of time (V2 = 0, immediate payment)
      gambles2_risk_adaptive_r.csvgambles2_time_adaptive_r.csv contains the information of the chosen option (gamble2).
      Risk:       V1 and V3= the outcome of gamble 2, V2 and V4 = the probability of the corresponding outcome.
      Time: V1 = the outcome of gamble 2, V2 = delay of time (V2 = 0, immediate payment)
      alphaconv_risk_q8_R0_1_r.csv
      		alphaconv_time_q8_R0_1_r.csvshows the likelihood that alpha estimates are converging
      D_risk_q8_R0_1_r.csvD_time_q8_R0_1_r.csvcontains the estimates of D
      delta_risk_q8_R0_1_r.csvdelta_time_q8_R0_1_r.csvcontains the estimates of response error
      deltaconv_risk_q8_R0_1_r.csv
      		deltaconv_time_q8_R0_1_r.csvshows the likelihood that delta estimates are converging
      geofit_risk_q8_R0_1_r.csv contains a measure of fit for each respondent; not critical
      likeconv_risk_q8_R0_1_r.csv
      		likeconv_time_q8_R0_1_r.csvshows the likelihood that likelihood estimate is converging
      theta_risk_q8_R0_1_r.csvtheta_time_q8_R0_1_r.csvshows how well the model fit your data (A higher theta suggests the model fit your preferences well, whereas a lower theta suggests the model was unable to fit your preferences well)
      wts_risk_q8_R0_1_r.csvwts_time_q8_R0_1_r.csvcontains the actual estimates.
      Risk: V1= alpha(Distortion of probability), V2 = sigma(diminishing sensitivity/curvature of value function), V3= lambda (loss aversion)
      Time: V1= beta(present bias), V2 = daily_discount_rate 
    • "q8" implies you used the 8-question survey. If you used a different a survey with X number of questions, it would say "wts_time_qX_R0_1_r.csv"
  • Error Handling:
    • Error in colMeans(likeconv[(round(nit1/10) + 1):((nit1 + nit2)/10), ]) : 'x' must be an array of at least two dimensions
    • Error in likeconv[(round(nit1/10) + 1):((nit1 + nit2)/10), ] : subscript out of bounds
    • If you came across with any of the above error message, you were using a too small number of iteration; it could be solved if you increase your number of iteration to be a reasonable size.


Please remember that the DEEP license requires you to use cite the paper below for purposes of publication: Toubia, O., Johnson, E., Evgeniou, T., & Delquie, P. (2013) 'Dynamic experiments for estimating preferences: An adaptive method of eliciting time and risk parameters.' Management Science, 59(3), 613-640.

Understanding the Output

*** UNDER CONSTRUCTION ***

For DEEPTime:
If the script ran correctly, it would have produced a number of csv files (18 files). 
The ones you should look for in DEEP Time are:
  1.  "alphaconvqh_q8_R0_r.csv" 
  2.  "deltaconvqh_q8_R0_r.csv"
  3.  "likeconvqh_q8_R0_r.csv"

For DEEPRisk:
If the script ran correctly, it would have produced a number of csv files (19 files). 
The ones you should look for in DEEP Time are:
  1.  "alphaconvv_risk_q8_R0_r.csv" 
  2.  "deltaconvv_risk_q8_R0_r.csv"
  3.  "likeconvv_risk_q8_R0_r.csv"



*** UNDER CONSTRUCTION ***

Getting help

If you have any issues or have comments with DEEP, we're here to help. Email both brian@decisionsciences.columbia.edu and shannon@decisionsciences.columbia.edu.