class: center middle main-title section-title-3 # In-person<br>session 3 .class-info[ **January 27, 2022** .light[PMAP 8521: Program evaluation<br> Andrew Young School of Policy Studies ] ] --- name: outline class: title title-inv-8 # Plan for today -- .box-4.medium.sp-after-half[Regression and R stuff] -- .box-1.medium.sp-after-half[Logic models] --- layout: false name: regression class: center middle section-title section-title-4 animated fadeIn # Regression and R stuff --- class: middle .box-4.large[Big ol' example of running,<br>interpreting, and exploring<br>some regression models] --- class: middle .box-4.huge[andhs.co/live] --- class: middle .box-4.large[Side-by-side regression tables] --- <table style="font-size:0.5em;border-bottom: 0; width: auto !important; margin-left: auto; margin-right: auto;" class="table"> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:center;"> Model 1 </th> <th style="text-align:center;"> Model 2 </th> <th style="text-align:center;"> Model 3 </th> <th style="text-align:center;"> Model 4 </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> (Intercept) </td> <td style="text-align:center;"> 362.307 </td> <td style="text-align:center;"> −5780.831*** </td> <td style="text-align:center;"> −5736.897*** </td> <td style="text-align:center;"> −5433.534*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (283.345) </td> <td style="text-align:center;"> (305.815) </td> <td style="text-align:center;"> (307.959) </td> <td style="text-align:center;"> (286.558) </td> </tr> <tr> <td style="text-align:left;"> bill_length_mm </td> <td style="text-align:center;"> 87.415*** </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 6.047 </td> <td style="text-align:center;"> −5.201 </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (6.402) </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (5.180) </td> <td style="text-align:center;"> (4.860) </td> </tr> <tr> <td style="text-align:left;"> flipper_length_mm </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 49.686*** </td> <td style="text-align:center;"> 48.145*** </td> <td style="text-align:center;"> 48.209*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (1.518) </td> <td style="text-align:center;"> (2.011) </td> <td style="text-align:center;"> (1.841) </td> </tr> <tr> <td style="text-align:left;"> sexmale </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 358.631*** </td> </tr> <tr> <td style="text-align:left;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> (41.572) </td> </tr> <tr> <td style="text-align:left;"> Num.Obs. </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 333 </td> </tr> <tr> <td style="text-align:left;"> R2 </td> <td style="text-align:center;"> 0.354 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.760 </td> <td style="text-align:center;"> 0.807 </td> </tr> <tr> <td style="text-align:left;"> R2 Adj. </td> <td style="text-align:center;"> 0.352 </td> <td style="text-align:center;"> 0.758 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.805 </td> </tr> <tr> <td style="text-align:left;"> AIC </td> <td style="text-align:center;"> 5400.0 </td> <td style="text-align:center;"> 5062.9 </td> <td style="text-align:center;"> 5063.5 </td> <td style="text-align:center;"> 4863.3 </td> </tr> <tr> <td style="text-align:left;"> BIC </td> <td style="text-align:center;"> 5411.5 </td> <td style="text-align:center;"> 5074.4 </td> <td style="text-align:center;"> 5078.8 </td> <td style="text-align:center;"> 4882.4 </td> </tr> <tr> <td style="text-align:left;"> Log.Lik. </td> <td style="text-align:center;"> −2696.987 </td> <td style="text-align:center;"> −2528.427 </td> <td style="text-align:center;"> −2527.741 </td> <td style="text-align:center;"> −2426.664 </td> </tr> <tr> <td style="text-align:left;"> F </td> <td style="text-align:center;"> 186.443 </td> <td style="text-align:center;"> 1070.745 </td> <td style="text-align:center;"> 536.626 </td> <td style="text-align:center;"> 457.118 </td> </tr> </tbody> <tfoot><tr><td style="padding: 0; " colspan="100%">  + p &lt; 0.1, * p &lt; 0.05, ** p &lt; 0.01, *** p &lt; 0.001</td></tr></tfoot> </table> --- <table style="NAborder-bottom: 0; width: auto !important; margin-left: auto; margin-right: auto;" class="table"> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:center;"> Model 1 </th> <th style="text-align:center;"> Model 2 </th> <th style="text-align:center;"> Model 3 </th> <th style="text-align:center;"> Model 4 </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> (Intercept) </td> <td style="text-align:center;"> 362.307 </td> <td style="text-align:center;"> −5780.831*** </td> <td style="text-align:center;"> −5736.897*** </td> <td style="text-align:center;"> −5433.534*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (283.345) </td> <td style="text-align:center;"> (305.815) </td> <td style="text-align:center;"> (307.959) </td> <td style="text-align:center;"> (286.558) </td> </tr> <tr> <td style="text-align:left;"> bill_length_mm </td> <td style="text-align:center;"> 87.415*** </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 6.047 </td> <td style="text-align:center;"> −5.201 </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (6.402) </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (5.180) </td> <td style="text-align:center;"> (4.860) </td> </tr> <tr> <td style="text-align:left;"> flipper_length_mm </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 49.686*** </td> <td style="text-align:center;"> 48.145*** </td> <td style="text-align:center;"> 48.209*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (1.518) </td> <td style="text-align:center;"> (2.011) </td> <td style="text-align:center;"> (1.841) </td> </tr> <tr> <td style="text-align:left;"> sexmale </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 358.631*** </td> </tr> <tr> <td style="text-align:left;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> (41.572) </td> </tr> <tr> <td style="text-align:left;"> Num.Obs. </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 333 </td> </tr> <tr> <td style="text-align:left;"> R2 </td> <td style="text-align:center;"> 0.354 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.760 </td> <td style="text-align:center;"> 0.807 </td> </tr> <tr> <td style="text-align:left;"> R2 Adj. </td> <td style="text-align:center;"> 0.352 </td> <td style="text-align:center;"> 0.758 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.805 </td> </tr> <tr> <td style="text-align:left;"> AIC </td> <td style="text-align:center;"> 5400.0 </td> <td style="text-align:center;"> 5062.9 </td> <td style="text-align:center;"> 5063.5 </td> <td style="text-align:center;"> 4863.3 </td> </tr> <tr> <td style="text-align:left;"> BIC </td> <td style="text-align:center;"> 5411.5 </td> <td style="text-align:center;"> 5074.4 </td> <td style="text-align:center;"> 5078.8 </td> <td style="text-align:center;"> 4882.4 </td> </tr> <tr> <td style="text-align:left;"> Log.Lik. </td> <td style="text-align:center;"> −2696.987 </td> <td style="text-align:center;"> −2528.427 </td> <td style="text-align:center;"> −2527.741 </td> <td style="text-align:center;"> −2426.664 </td> </tr> <tr> <td style="text-align:left;"> F </td> <td style="text-align:center;"> 186.443 </td> <td style="text-align:center;"> 1070.745 </td> <td style="text-align:center;"> 536.626 </td> <td style="text-align:center;"> 457.118 </td> </tr> </tbody> <tfoot><tr><td style="padding: 0; " colspan="100%">  + p &lt; 0.1, * p &lt; 0.05, ** p &lt; 0.01, *** p &lt; 0.001</td></tr></tfoot> </table> --- class: middle .box-4.large[See full documentation and<br>examples for `modelsummary()` [here](https://vincentarelbundock.github.io/modelsummary/articles/modelsummary.html)] --- layout: true class: middle --- .box-4.large[Make nicer tables when<br>knitting with `kable()`] .box-inv-4[(Or even fancier tables with [kableExtra](https://cran.r-project.org/web/packages/kableExtra/vignettes/awesome_table_in_html.html)!)] --- .box-4.large[Style suggestions] --- .box-4.large[Typing math] --- .box-4.large[Cleaner chunk output] --- .box-4.large[Figure resizing] --- .box-4.large[Non-English characters in PDFs] --- layout: false name: logic-models class: center middle section-title section-title-1 animated fadeIn # Logic models --- layout: true class: middle --- .box-1.large[Logic models as managerial tools] --- .box-1.large[Inputs vs. Activities vs.<br>Outputs vs. Outcomes] --- .box-1.large[Impact theory vs. logic model] --- layout: false class: title title-1 # Impact theory .box-inv-1[Ensure that the theory linking activities to the outcomes is sound!] .center[ <figure> <img src="img/03/psd-impact-theory-arrow.png" alt="PSD impact theory" title="PSD impact theory" width="100%"> </figure> ] --- .center[ <figure> <img src="img/01/psd-logic-model.png" alt="PSD logic model" title="PSD logic model" width="70%"> </figure> ] --- layout: false class: title title-1 # MPA/MPP at GSU .pull-left[ <figure> <img src="img/03/mpp-mission@4x.png" alt="MPP mission" title="MPP mission" width="130%"> </figure> ] .pull-right[ <figure> <img src="img/03/mpa-mission@4x.png" alt="MPA mission" title="MPA mission" width="100%"> </figure> ] ??? Inputs: - Students - Curriculum - Money (grants, tuition) - AYSPS itself - Faculty, staff - Technology (iCollege, PAWS) - Infrastructure - Transportation - State regulations - National accreditation standards - Faculty research Activities: - Classes - Group work - Exams - Internships - Job fairs - Commencement - Studying - Tutoring - Office hours - Advising - Attendance Outputs: - Assignments and projects - Grades - Degree - Jobs - Awards - Network - Debt Outcomes: - Leadership - AYS national ranking - Public service motivation - Critical thinking - Efficient workforce - Income - Better society --- layout: true class: middle --- .box-1.medium[Isn't it best to always<br>have an articulated theory?] .box-1.medium[Should implicit theory and articulated theory<br>be the same thing in most cases?] --- .box-1.medium[How much does this evaluation stuff cost?] .box-1.medium[Can you do scaled-down versions<br>of these evaluations?] --- .box-1.medium[What if a program exists already<br>and doesn't have a logic model?] .box-1.medium[What if a program exists already and doesn't have baseline data (or any data!)?] --- .box-1.medium[How can programs evaluate their final<br>outcomes if they are not measurable?<br>Do they find a different outcome<br>that is easier to measure?] .box-1.medium[Why would they pick final outcomes<br>that can't be measured?] --- .box-1.medium[What should you do if you find that your theory of change (or logic model in general) is wrong in the middle of the program? Is it ethical to stop or readjust?] --- layout: true class: title title-1 --- # Outcomes and programs .box-1.medium[Outcome variable] .box-inv-1.sp-after-half.SMALL[Thing you're measuring] -- .box-1.medium[Outcome change] .box-inv-1.sp-after-half.SMALL[∆ in thing you're measuring over time] -- .box-1.medium[Program effect] .box-inv-1.SMALL[∆ in thing you're measuring over time *because of* the program] --- # Outcomes and programs <figure> <img src="img/04/program-effect.png" alt="Outcomes and program effect" title="Outcomes and program effect" width="100%"> </figure>