In-person
session 3

January 27, 2022

PMAP 8521: Program evaluation
Andrew Young School of Policy Studies

1 / 29

Plan for today2 / 29

Plan for today

Regression and R stuff

2 / 29

Plan for today

Regression and R stuff

Logic models

2 / 29

Regression and R stuff3 / 29

Big ol' example of running,
interpreting, and exploring
some regression models

4 / 29

andhs.co/live

5 / 29

Side-by-side regression tables

6 / 29

 
    Model 1 
    Model 2 
    Model 3 
    Model 4 
  
    (Intercept) 
    362.307 
    −5780.831*** 
    −5736.897*** 
    −5433.534*** 
  
    (283.345) 
    (305.815) 
    (307.959) 
    (286.558) 
  
    bill_length_mm 
    87.415*** 
     
    6.047 
    −5.201 
  
    (6.402) 
     
    (5.180) 
    (4.860) 
  
    flipper_length_mm 
     
    49.686*** 
    48.145*** 
    48.209*** 
  
    (1.518) 
    (2.011) 
    (1.841) 
  
    sexmale 
     
    358.631*** 
  
    (41.572) 
  
    Num.Obs. 
    342 
    342 
    342 
    333 
  
    R2 
    0.354 
    0.759 
    0.760 
    0.807 
  
    R2 Adj. 
    0.352 
    0.758 
    0.759 
    0.805 
  
    AIC 
    5400.0 
    5062.9 
    5063.5 
    4863.3 
  
    BIC 
    5411.5 
    5074.4 
    5078.8 
    4882.4 
  
    Log.Lik. 
    −2696.987 
    −2528.427 
    −2527.741 
    −2426.664 
  
    F 
    186.443 
    1070.745 
    536.626 
    457.118 
  
 + p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001

7 / 29

	Model 1	Model 2	Model 3	Model 4
(Intercept)	362.307	−5780.831***	−5736.897***	−5433.534***
	(283.345)	(305.815)	(307.959)	(286.558)
bill_length_mm	87.415***		6.047	−5.201
	(6.402)		(5.180)	(4.860)
flipper_length_mm		49.686***	48.145***	48.209***
		(1.518)	(2.011)	(1.841)
sexmale				358.631***
				(41.572)
Num.Obs.	342	342	342	333
R2	0.354	0.759	0.760	0.807
R2 Adj.	0.352	0.758	0.759	0.805
AIC	5400.0	5062.9	5063.5	4863.3
BIC	5411.5	5074.4	5078.8	4882.4
Log.Lik.	−2696.987	−2528.427	−2527.741	−2426.664
F	186.443	1070.745	536.626	457.118
+ p < 0.1, * p < 0.05, p < 0.01, * p < 0.001

 
    Model 1 
    Model 2 
    Model 3 
    Model 4 
  
    (Intercept) 
    362.307 
    −5780.831*** 
    −5736.897*** 
    −5433.534*** 
  
    (283.345) 
    (305.815) 
    (307.959) 
    (286.558) 
  
    bill_length_mm 
    87.415*** 
     
    6.047 
    −5.201 
  
    (6.402) 
     
    (5.180) 
    (4.860) 
  
    flipper_length_mm 
     
    49.686*** 
    48.145*** 
    48.209*** 
  
    (1.518) 
    (2.011) 
    (1.841) 
  
    sexmale 
     
    358.631*** 
  
    (41.572) 
  
    Num.Obs. 
    342 
    342 
    342 
    333 
  
    R2 
    0.354 
    0.759 
    0.760 
    0.807 
  
    R2 Adj. 
    0.352 
    0.758 
    0.759 
    0.805 
  
    AIC 
    5400.0 
    5062.9 
    5063.5 
    4863.3 
  
    BIC 
    5411.5 
    5074.4 
    5078.8 
    4882.4 
  
    Log.Lik. 
    −2696.987 
    −2528.427 
    −2527.741 
    −2426.664 
  
    F 
    186.443 
    1070.745 
    536.626 
    457.118 
  
 + p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001

8 / 29

	Model 1	Model 2	Model 3	Model 4
(Intercept)	362.307	−5780.831***	−5736.897***	−5433.534***
	(283.345)	(305.815)	(307.959)	(286.558)
bill_length_mm	87.415***		6.047	−5.201
	(6.402)		(5.180)	(4.860)
flipper_length_mm		49.686***	48.145***	48.209***
		(1.518)	(2.011)	(1.841)
sexmale				358.631***
				(41.572)
Num.Obs.	342	342	342	333
R2	0.354	0.759	0.760	0.807
R2 Adj.	0.352	0.758	0.759	0.805
AIC	5400.0	5062.9	5063.5	4863.3
BIC	5411.5	5074.4	5078.8	4882.4
Log.Lik.	−2696.987	−2528.427	−2527.741	−2426.664
F	186.443	1070.745	536.626	457.118
+ p < 0.1, * p < 0.05, p < 0.01, * p < 0.001

See full documentation and
examples for modelsummary() here

9 / 29

Make nicer tables when
knitting with kable()

(Or even fancier tables with kableExtra!)

10 / 29

Style suggestions

11 / 29

Typing math

12 / 29

Cleaner chunk output

13 / 29

Figure resizing

14 / 29

Non-English characters in PDFs

15 / 29

Logic models16 / 29

Logic models as managerial tools

17 / 29

Inputs vs. Activities vs.
Outputs vs. Outcomes

18 / 29

Impact theory vs. logic model

19 / 29

Impact theory

Ensure that the theory linking activities to the outcomes is sound!

20 / 29

21 / 29

MPA/MPP at GSU

22 / 29

Inputs:

Students
Curriculum
Money (grants, tuition)
AYSPS itself
Faculty, staff
Technology (iCollege, PAWS)
Infrastructure
Transportation
State regulations
National accreditation standards
Faculty research

Activities:

Classes
Group work
Exams
Internships
Job fairs
Commencement
Studying
Tutoring
Office hours
Advising
Attendance

Outputs:

Assignments and projects
Grades
Degree
Jobs
Awards
Network
Debt

Outcomes:

Leadership
AYS national ranking
Public service motivation
Critical thinking
Efficient workforce
Income
Better society

Isn't it best to always
have an articulated theory?

Should implicit theory and articulated theory
be the same thing in most cases?

23 / 29

How much does this evaluation stuff cost?

Can you do scaled-down versions
of these evaluations?

24 / 29

What if a program exists already
and doesn't have a logic model?

What if a program exists already and doesn't have baseline data (or any data!)?

25 / 29

How can programs evaluate their final
outcomes if they are not measurable?
Do they find a different outcome
that is easier to measure?

Why would they pick final outcomes
that can't be measured?

26 / 29

What should you do if you find that your theory of change (or logic model in general) is wrong in the middle of the program? Is it ethical to stop or readjust?

27 / 29

Outcomes and programs

Outcome variable

Thing you're measuring

28 / 29

Outcomes and programs

Outcome variable

Thing you're measuring

Outcome change

∆ in thing you're measuring over time

28 / 29

Outcomes and programs

Outcome variable

Thing you're measuring

Outcome change

∆ in thing you're measuring over time

Program effect

∆ in thing you're measuring over time because of the program

28 / 29

Outcomes and programs

29 / 29

Help

Keyboard shortcuts

↑, ←, Pg Up, k

Go to previous slide

↓, →, Pg Dn, Space, j

Go to next slide

Home

Go to first slide

End

Go to last slide

Number + Return

Go to specific slide

b / m / f

Toggle blackout / mirrored / fullscreen mode

Clone slideshow

Toggle presenter mode

Restart the presentation timer

?, h

Toggle this help

Tile View: Overview of Slides

In-person
session 3

January 27, 2022

PMAP 8521: Program evaluation
Andrew Young School of Policy Studies

1 / 29

Plan for today2 / 29

Plan for today

Regression and R stuff

2 / 29

Plan for today

Regression and R stuff

Logic models

2 / 29

Regression and R stuff3 / 29

Big ol' example of running,
interpreting, and exploring
some regression models

4 / 29

andhs.co/live

5 / 29

Side-by-side regression tables

6 / 29

 
    Model 1 
    Model 2 
    Model 3 
    Model 4 
  
    (Intercept) 
    362.307 
    −5780.831*** 
    −5736.897*** 
    −5433.534*** 
  
    (283.345) 
    (305.815) 
    (307.959) 
    (286.558) 
  
    bill_length_mm 
    87.415*** 
     
    6.047 
    −5.201 
  
    (6.402) 
     
    (5.180) 
    (4.860) 
  
    flipper_length_mm 
     
    49.686*** 
    48.145*** 
    48.209*** 
  
    (1.518) 
    (2.011) 
    (1.841) 
  
    sexmale 
     
    358.631*** 
  
    (41.572) 
  
    Num.Obs. 
    342 
    342 
    342 
    333 
  
    R2 
    0.354 
    0.759 
    0.760 
    0.807 
  
    R2 Adj. 
    0.352 
    0.758 
    0.759 
    0.805 
  
    AIC 
    5400.0 
    5062.9 
    5063.5 
    4863.3 
  
    BIC 
    5411.5 
    5074.4 
    5078.8 
    4882.4 
  
    Log.Lik. 
    −2696.987 
    −2528.427 
    −2527.741 
    −2426.664 
  
    F 
    186.443 
    1070.745 
    536.626 
    457.118 
  
 + p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001

7 / 29

	Model 1	Model 2	Model 3	Model 4
(Intercept)	362.307	−5780.831***	−5736.897***	−5433.534***
	(283.345)	(305.815)	(307.959)	(286.558)
bill_length_mm	87.415***		6.047	−5.201
	(6.402)		(5.180)	(4.860)
flipper_length_mm		49.686***	48.145***	48.209***
		(1.518)	(2.011)	(1.841)
sexmale				358.631***
				(41.572)
Num.Obs.	342	342	342	333
R2	0.354	0.759	0.760	0.807
R2 Adj.	0.352	0.758	0.759	0.805
AIC	5400.0	5062.9	5063.5	4863.3
BIC	5411.5	5074.4	5078.8	4882.4
Log.Lik.	−2696.987	−2528.427	−2527.741	−2426.664
F	186.443	1070.745	536.626	457.118
+ p < 0.1, * p < 0.05, p < 0.01, * p < 0.001

 
    Model 1 
    Model 2 
    Model 3 
    Model 4 
  
    (Intercept) 
    362.307 
    −5780.831*** 
    −5736.897*** 
    −5433.534*** 
  
    (283.345) 
    (305.815) 
    (307.959) 
    (286.558) 
  
    bill_length_mm 
    87.415*** 
     
    6.047 
    −5.201 
  
    (6.402) 
     
    (5.180) 
    (4.860) 
  
    flipper_length_mm 
     
    49.686*** 
    48.145*** 
    48.209*** 
  
    (1.518) 
    (2.011) 
    (1.841) 
  
    sexmale 
     
    358.631*** 
  
    (41.572) 
  
    Num.Obs. 
    342 
    342 
    342 
    333 
  
    R2 
    0.354 
    0.759 
    0.760 
    0.807 
  
    R2 Adj. 
    0.352 
    0.758 
    0.759 
    0.805 
  
    AIC 
    5400.0 
    5062.9 
    5063.5 
    4863.3 
  
    BIC 
    5411.5 
    5074.4 
    5078.8 
    4882.4 
  
    Log.Lik. 
    −2696.987 
    −2528.427 
    −2527.741 
    −2426.664 
  
    F 
    186.443 
    1070.745 
    536.626 
    457.118 
  
 + p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001

8 / 29

	Model 1	Model 2	Model 3	Model 4
(Intercept)	362.307	−5780.831***	−5736.897***	−5433.534***
	(283.345)	(305.815)	(307.959)	(286.558)
bill_length_mm	87.415***		6.047	−5.201
	(6.402)		(5.180)	(4.860)
flipper_length_mm		49.686***	48.145***	48.209***
		(1.518)	(2.011)	(1.841)
sexmale				358.631***
				(41.572)
Num.Obs.	342	342	342	333
R2	0.354	0.759	0.760	0.807
R2 Adj.	0.352	0.758	0.759	0.805
AIC	5400.0	5062.9	5063.5	4863.3
BIC	5411.5	5074.4	5078.8	4882.4
Log.Lik.	−2696.987	−2528.427	−2527.741	−2426.664
F	186.443	1070.745	536.626	457.118
+ p < 0.1, * p < 0.05, p < 0.01, * p < 0.001

See full documentation and
examples for modelsummary() here

9 / 29

Make nicer tables when
knitting with kable()

(Or even fancier tables with kableExtra!)

10 / 29

Style suggestions

11 / 29

Typing math

12 / 29

Cleaner chunk output

13 / 29

Figure resizing

14 / 29

Non-English characters in PDFs

15 / 29

Logic models16 / 29

Logic models as managerial tools

17 / 29

Inputs vs. Activities vs.
Outputs vs. Outcomes

18 / 29

Impact theory vs. logic model

19 / 29

Impact theory

Ensure that the theory linking activities to the outcomes is sound!

20 / 29

21 / 29

MPA/MPP at GSU

22 / 29

Inputs:

Students
Curriculum
Money (grants, tuition)
AYSPS itself
Faculty, staff
Technology (iCollege, PAWS)
Infrastructure
Transportation
State regulations
National accreditation standards
Faculty research

Activities:

Classes
Group work
Exams
Internships
Job fairs
Commencement
Studying
Tutoring
Office hours
Advising
Attendance

Outputs:

Assignments and projects
Grades
Degree
Jobs
Awards
Network
Debt

Outcomes:

Leadership
AYS national ranking
Public service motivation
Critical thinking
Efficient workforce
Income
Better society

Isn't it best to always
have an articulated theory?

Should implicit theory and articulated theory
be the same thing in most cases?

23 / 29

How much does this evaluation stuff cost?

Can you do scaled-down versions
of these evaluations?

24 / 29

What if a program exists already
and doesn't have a logic model?

What if a program exists already and doesn't have baseline data (or any data!)?

25 / 29

How can programs evaluate their final
outcomes if they are not measurable?
Do they find a different outcome
that is easier to measure?

Why would they pick final outcomes
that can't be measured?

26 / 29

What should you do if you find that your theory of change (or logic model in general) is wrong in the middle of the program? Is it ethical to stop or readjust?

27 / 29

Outcomes and programs

Outcome variable

Thing you're measuring

28 / 29

Outcomes and programs

Outcome variable

Thing you're measuring

Outcome change

∆ in thing you're measuring over time

28 / 29

Outcomes and programs

Outcome variable

Thing you're measuring

Outcome change

∆ in thing you're measuring over time

Program effect

∆ in thing you're measuring over time because of the program

28 / 29

Outcomes and programs

29 / 29

In-personsession 3

Plan for today

Plan for today

Plan for today

Regression and R stuff

Logic models

Impact theory

MPA/MPP at GSU

Outcomes and programs

Outcomes and programs

Outcomes and programs

Outcomes and programs

Plan for today

Help

Class 3: Regression and logic models

In-personsession 3

Plan for today

Plan for today

Plan for today

Regression and R stuff

Logic models

Impact theory

MPA/MPP at GSU

Outcomes and programs

Outcomes and programs

Outcomes and programs

Outcomes and programs

In-person
session 3

In-person
session 3