DEEP Software



Download the DEEP R package

The DEEP R package is hosted on the Columbia University Technology Ventures site. In order to download the DEEP package, you need to have the Devtools package installed


The primary software package for DEEP is called Integrated DEEP, and it:
  • Includes the DEEP Survey Tools and the DEEP Analysis Tools (for R).
  • Supports Qualtrics and LimeSurvey.
  • Can be inserted into any point in your existing survey.
  • Can be customized for either DEEP Time or DEEP Risk.
  • Can have up to 12 questions.
If you need to use DEEP on a platform other than Qualtrics or LimeSurvey, or need the DEEP 20-question version.

How it works





Add DEEP to your survey



Data analysis

Download your data

Qualtrics
  1. Click the "View Results" tab and view your survey.
  2. Click the "Download Data" button.
  3. Select CSV and download.

LimeSurvey
  1. Go to your survey. On the menu bar, click "Responses and Statistics" with the pie chart icon.
  2. Then, click the "Export results to application" on the menu with the database icon.
  3. For the format, select "CSV" (not "CSV with all answers part").
  4. Click "Export Data".

You'll see that all of the DEEP data is compacted into a single column. This is normal. The DEEP R package transforms the data to the proper format.

Analyzing your data

Before starting, open your R Console and make sure the DEEP package is loaded.  

The DEEP package takes four parameters in order to properly run the analysis. Read the description below to learn what you need to know before running the analysis:
  • filter_by: Used to specify the "question identifier" ("question name" used in Qualtrics or LimeSurvey) in case you have multiple DEEP surveys and/or more than one DEEP type in the Qualtrics/Limesurvey output. The DEEP package can only analyze a single DEEP question at once. Failure to specify the question identifier will produce an error if more than one DEEP question is found. This is not case-sensitive. 
  • DEEPtype: Refers to the type of DEEP question you used in your analysis. It accepts either "risk" or "time" (not case sensitive).If you have more than one DEEP question, please specify the DEEPtype for the question in the "filter_by" field.
  • WD: Refers to the working directory. This is the folder/path where all the DEEP output will be exported. If you have multiple DEEP questions we recommend you create a folder for each question since any output file with the same name will be overwritten during analysis.   
  • file_path: Refers to the path of the file you exported from LimeSurvey or Qualtrics. If you do not specify a file_path, a Graphical User Interface will appear to allow you to manually select the file from you computer. 
Below are two examples of what the R code to start the data analysis should look like. 

deepRun(DEEPtype = "risk", WD = "C:/YourUser/Documents/OutputFolder/Risk", file_path = "C:/YourUser/Documents/OutputFolder/mySurvey.xml", filter_by = "RiskP")

deepRun(DEEPtype = "time", WD = "C:/YourUser/Documents/OutputFolder/Time", file_path = "C:/YourUser/Documents/OutputFolder/mySurvey.csv", filter_by = "DEEPtimeQ")

If for any reason that the above code doesn't work, you can try the following R code:
Risk: 
deepTransform("risk", WD = "C:/YourUser/Documents/OutputFolder/Risk", file_path = "C:/YourUser/Documents/OutputFolder/mySurvey.xml"
)
deepRiskHB()

Time:
deepTransform("time", WD = 
"C:/YourUser/Documents/OutputFolder/Time"
, file_path= 
"C:/YourUser/Documents/OutputFolder/mySurvey.csv"
)
deepTimeHB()



deepRiskHB(iterUntilSave, iterSaved) and deepTimeHB(iterUntil, iterSaved)
  • Description: Estimation of Risk/Time Preference Parameters Using Hierarchical Markov Chain Mote Carlo Bayesian Framework
  • Preparation: Have the following files in your working directory (for example, setwd("C:/YourUser/Documents/Risk"))
    • RiskTime Description 
      adaptive_question_risk_1.csvadaptive_question_time_1.csvcontains the path of adaptive questions
      choices_risk_1.csvchoices_time_1.csvcontains participants choices (should be either 1 or 2)
      serial_risk_1.csvserial_time_1.csvcontains the Qualtrics or Limesurvey ID matched with the serial assigned by the R script
  • Arguments: The total number of iteration is the sum of the two arguments
    1. iterUntilSave: default to 50000 in deepRiskHB and 10000 in deepTimeHB
    2. iterSaved: default to 50000 in deepRiskHB and 40000 in deepTimeHB
  • Output:
    •  Risk Time Description
      gambles1_risk_adaptive_r.csv

      gambles1_time_adaptive_r.csv

      contains the information of the chosen option (gamble1).
      Risk: 
      V1 and V3= the outcome of gamble 1, V2 and V4 = the probability of the corresponding outcome.
      Time: V1 = the outcome of gamble 1, V2 = delay of time (V2 = 0, immediate payment)
      gambles2_risk_adaptive_r.csvgambles2_time_adaptive_r.csv contains the information of the chosen option (gamble2).
      Risk: 
      V1 and V3= the outcome of gamble 2, V2 and V4 = the probability of the corresponding outcome.
      Time: V1 = the outcome of gamble 2, V2 = delay of time (V2 = 0, immediate payment)
      alphaconv_risk_q8_R0_1_r.csv
      alphaconv_time_q8_R0_1_r.csvshows the likelihood that alpha estimates are converging
      D_risk_q8_R0_1_r.csvD_time_q8_R0_1_r.csvcontains the estimates of D
      delta_risk_q8_R0_1_r.csvdelta_time_q8_R0_1_r.csvcontains the estimates of response error
      deltaconv_risk_q8_R0_1_r.csv
      deltaconv_time_q8_R0_1_r.csvshows the likelihood that delta estimates are converging
      geofit_risk_q8_R0_1_r.csv  
      likeconv_risk_q8_R0_1_r.csv
      likeconv_time_q8_R0_1_r.csvshows the likelihood that likelihood estimate is converging
      theta_risk_q8_R0_1_r.csvtheta_time_q8_R0_1_r.csvshows how well the model fit your data (A higher theta suggests the model fit your preferences well, whereas a lower theta suggests the model was unable to fit your preferences well)
      wts_risk_q8_R0_1_r.csvwts_time_q8_R0_1_r.csvcontains the actual estimates.
      Risk: V1= alpha(Distortion of probability), V2 = sigma(diminishing sensitivity/curvature of value function), V3= lambda (loss aversion)
      Time: V1= beta(present bias), V2 = daily_discount_rate 
    • "q8" implies you used the 8-question survey. If you used a different a survey with X number of questions, it would say "wts_time_qX_R0_1_r.csv"
  • Error message:
    • Error in colMeans(likeconv[(round(nit1/10) + 1):((nit1 + nit2)/10), ]) : 'x' must be an array of at least two dimensions
    • Error in likeconv[(round(nit1/10) + 1):((nit1 + nit2)/10), ] : subscript out of bounds


Please remember that the DEEP license requires you to use cite the paper below for purposes of publication: Toubia, O., Johnson, E., Evgeniou, T., & Delquie, P. (2013) 'Dynamic experiments for estimating preferences: An adaptive method of eliciting time and risk parameters.' Management Science, 59(3), 613-640.

Understanding the Output

*** UNDER CONSTRUCTION ***

For DEEPTime:
If the script ran correctly, it would have produced a number of csv files (18 files). 
The ones you should look for in DEEP Time are:
  1.  "adaptive_question_time_1.csv" which contains the path of adaptive questions
  2.  "alphaconv_time_q8_R0_1_r.csv" which shows the likelihood that alpha estimates are converging
  3.  "alphaconvqh_q8_R0_r.csv" 
  4.  "choices_time_1.csv" which contains participants choices (should be either 1 or 2)
  5.  "D_time_q8_R0_1_r.csv" which contains the estimates of D
  6.  "delta_time_q8_R0_1.csv" which contains the estimates of response error
  7.  "deltaconv_time_q8_R0_1_r.csv" which shows the likelihood that delta estimates are converging
  8.  "deltaconvqh_q8_R0_r.csv"
  9.  "gambles1_time_adptive_1_r.csv" which contains the information of the chosen option (gamble1). V1 = the outcome of gamble 1, V2 = delay of time (V2 = 0, immediate payment)
  10.  "gambles2_time_adptive_1_r.csv" which contains the information of the other option (gamble 2). V1 = the outcome of gamble 2, V2 = delay of time (V2 = 0, immediate payment)
  11.  "likeconv_time_q8_R0_1_r.csv" which shows the likelihood that likelihood estimate is converging
  12.  "likeconvqh_q8_R0_r.csv"
  13.  "Qtime_DEEPtime.csv" which contains the response time for each question as columns
  14.  "serials_respid.csv" which contains the Qualtrics or Limesurvey ID matched with the serial assigned by the R script.  
  15.  "serial_time_1.csv" which contains serial numbers (Make sure that there are NO values with NA in the data file)
  16.  "theta_time_q8_R0_1_r.csv" which shows how well the model fit your data (A higher theta suggests the model fit your preferences well, whereas a lower theta suggests the model was unable to fit your preferences well)
  17.  "Unshrunken_Parameter_Estimates.csv" which contains the pre-Hierarchical Bayes estimates.You will have 2 columns.  V1= Unshrunken_beta(present bias), V2 = Unshrunken_daily_discount_rate (Note: The old wiki page called this parameter delta, which is a misnomer). 
  18.  "wts_time_q8_R0_1_r.csv" which contains the actual estimates. You will have 2 columns.  V1= beta(present bias), V2 = daily_discount_rate 

For DEEPRisk:
If the script ran correctly, it would have produced a number of csv files (19 files). 
The ones you should look for in DEEP Time are:
  1.  "adaptive_question_risk_1.csv" which contains the path of adaptive questions
  2.  "alphaconv_risk_q8_R0_1_r.csv" which shows the likelihood that alpha estimates are converging
  3.  "alphaconvv_risk_q8_R0_r.csv" 
  4.  "choices_risk_1.csv" which contains participants choices (should be either 1 or 2)
  5.  "D_risk_q8_R0_1_r.csv" which contains the estimates of D
  6.  "delta_risk_q8_R0_1.csv" which contains the estimates of response error
  7.  "deltaconv_risk_q8_R0_1_r.csv" which shows the likelihood that delta estimates are converging
  8.  "deltaconvv_risk_q8_R0_r.csv"
  9.  "gambles1_risk_adptive_1_r.csv" which contains the information of the chosen option (gamble1). V1 and V3= the outcome of gamble 1, V2 and V4 = the probability of the corresponding outcome.  
  10.  "gambles2_risk_adptive_1_r.csv" which contains the information of the other option (gamble 2).  V1 and V3= the outcome of gamble 1, V2 and V4 = the probability of the corresponding outcome.  
  11. "geofit_risk_q8_R0_1_r.csv"
  12.  "likeconv_risk_q8_R0_1_r.csv" which shows the likelihood that likelihood estimate is converging
  13.  "likeconvv_risk_q8_R0_r.csv"
  14.  "Qtime_DEEPrisk.csv" which contains the response time for each question as columns
  15.  "serials_respid.csv" which contains the Qualtrics or Limesurvey ID matched with the serial assigned by the R script.  
  16.  "serial_risk_1.csv" which contains serial numbers (Make sure that there are NO values with NA in the data file)
  17.  "theta_risk_q8_R0_1_r.csv" which shows how well the model fit your data (A higher theta suggests the model fit your preferences well, whereas a lower theta suggests the model was unable to fit your preferences well)
  18.  "Unshrunken_Parameter_Estimates.csv" which contains the pre-Hierarchical Bayes estimates.You will have 2 columns.  V1= Unshrunken_beta(present bias), V2 = Unshrunken_daily_discount_rate (Note: The old wiki page called this parameter delta, which is a misnomer). 
  19.  "wts_risk_q8_R0_1_r.csv" which contains the actual estimates. You will have 3 columns.  V1= alpha(Distortion of probability), V2 = sigma(diminishing sensitivity/curvature of value function), V3= lambda (loss aversion)
(Note that q8 implies you used the 8-question survey. If you used a different a survey with X number of questions, it would say "wts_time_qX_R0_1_r.csv")


*** UNDER CONSTRUCTION ***

Getting help

If you have any issues or have comments with DEEP, we're here to help. Email both brian@decisionsciences.columbia.edu and shannon@decisionsciences.columbia.edu.