class: center middle main-title section-title-3 # In-person<br>session 13 .class-info[ **April 14, 2022** .light[PMAP 8521: Program evaluation<br> Andrew Young School of Policy Studies ] ] --- name: outline class: title title-inv-8 # Plan for today -- .box-1.medium[Exam and final project stuff] -- .box-5.medium[Control variables and confounding] -- .box-3.medium[Ethics + pre-analysis plans] -- .box-2.medium[Sharing your stuff] -- .box-6.medium[Simulating data] --- layout: false name: exam class: center middle section-title section-title-1 animated fadeIn # Exam and<br>final project stuff --- layout: false name: controls-confounding class: center middle section-title section-title-5 animated fadeIn # Control variables<br>and confounding --- layout: true class: middle --- .box-5.large[Do we really not need to<br>interpret every coefficient?!] --- .box-5.less-medium[Why do we teach stats with the idea of throwing<br>in a bunch of control variables if that's<br>really not the best way to evaluate impact?] .box-5.less-medium[What is the point of using control variables<br>if they aren't going to be interpreted?] .box-5.less-medium[How do we know which coefficients we<br>need to ultimately worry about and interpret?] --- .box-5.large[Why did we control for things<br>in the RCT section of problem set 8?] --- .box-5.large[How can you NOT have unobserved confounders unless you're doing a laboratory study where you can control everything?] --- .box-5.medium[Design-based inference seems easier since<br>there's no unobserved confounding?<br><br>Why don't we just do that all the time?] ??? Yeah, but you have to discover weird situations. Like WWII submarine thing! - https://braddelong.substack.com/p/podcast-hexapodia-is-e-key-insight-331 - Matthew Suandi, "Promoting to Opportunity: Evidence and Implications from the U.S. Submarine Service" - https://are.berkeley.edu/sites/are.berkeley.edu/files/job-candidates/paper/MatthewSuandi_JMP_Nov_21_Berkeley.pdf --- .box-5.less-medium[You said that choosing between<br>model-based inference versus design-based<br>inference is based on the situation that we are in.<br><br>Which one of the approaches do you think<br>will be used the most in the real world?] --- .box-5.large[Do we not need to use a DAG<br>if we use one of<br>the design-based methods?] --- .center[ <figure> <img src="img/13-class/design-based-dags.png" alt="Design-based DAGs" title="Design-based DAGs" width="80%"> </figure> ] --- layout: false name: ethics class: center middle section-title section-title-3 animated fadeIn # Ethics +<br>pre-analysis plans --- layout: true class: middle --- .box-3.medium[Where can one publish their assumptions before running the numbers in research?] .center.float-left[[OSF.io](https://www.cos.io/initiatives/prereg) &ensp; [AsPredicted.org](https://aspredicted.org/)] --- .box-3[Do people really provide all their code and analysis in advance?<br>What if you realize you made a mistake<br>or want to explore something later?] .box-3[What if we have additional variables to investigate that we missed out during our simulation of fake data and pre-registration/pre-analysis stage? Can we add them once they have been registered?] .box-3[Does pre-registering include interactions<br>or quadratics you want to test?] --- .center[[Standard operating procedures](https://github.com/acoppock/Green-Lab-SOP)] .center[—] .center[[Departures from preregistered plan](https://osf.io/x5w7h/wiki/02%20Departures%20from%20Preregistered%20Plan/)] .center[—] .center[[Example](https://www.andrewheiss.com/research/articles/chaudhry-heiss-ngos-philanthropy/)] --- .box-3.medium[Should you share the synthetic data<br>as part of a preanalysis plan?] --- .box-3.medium[Does pre-registration kill<br>creativity and "ah-ha" moments?] .center[[Example of confirmatory vs.<br>exploratory preregistration](https://osf.io/hsbyd)] --- layout: false name: sharing class: center middle section-title section-title-2 animated fadeIn # Sharing your stuff --- class: middle .box-2.large[What can we put<br>on our resumes now?] --- class: middle .box-2.large[Can I really just post R stuff<br>on a website or on Twitter?] .box-2.large[It seems like that's what<br>the experts do, not me!] .center.small[[(Public work)](https://evalsp22.classes.andrewheiss.com/slides/14-slides.html#133)] --- class: title title-2 # Websites for sharing R stuff .box-inv-2[Super easiest (but least flexible): [RPubs](https://rpubs.com/)] -- .box-inv-2[Easiest (but less flexible): [R Markdown websites](https://rmarkdown.rstudio.com/lesson-13.html)] .center.small[[Example](https://stats.andrewheiss.com/who-cares-about-crackdowns/); [example](https://talks.andrewheiss.com/2021-seacen/)] -- .box-inv-2.sp-before[Medium-est: [Distill](https://rstudio.github.io/distill/)] .center.small[[Example](https://www.pipinghotdata.com/)] -- .box-inv-2.sp-before[Hardest (but most flexible): [Blogdown](https://pkgs.rstudio.com/blogdown/)] .center.small[[Example](https://evalsp22.classes.andrewheiss.com/); [example](https://www.andrewheiss.com/)] --- layout: false name: fake-data class: center middle section-title section-title-6 animated fadeIn # Simulating data