Being Leslie Knope: Part 1

Parks and Recreation was one of the best shows in recent memory to grace our television screens. It was funny, poignant, and an inside peek into the far-from-glamorous world of municipal bureaucracy, albeit comedically exaggerated. Which was alright with me. A veteran of state government internships and survivor of multiple red tape mummifications, the true value of Parks and Rec was not the writing, acting, or receiving; but the moments when policies driven feverishly by the incomparable Leslie Knope won. It gave all of us who work in public policy hope that one day we too would become Knopes in our own department, rather than nopes who saw far too many of their good ideas skewered and left to bleed out on a conference room floor. Of course the comedy in P&R came when those glorious policies failed, either at the hand of a public forum or in implementation as everything backfired. We've all been there Knope.

This is the first installation in a series of posts I am doing to help me with preparing for my weekly MPA class in program evaluation. You will get a policy-backed explanation of the Parks Department's best and worst ideas from someone with two degrees in environmental policy and I will get to binge watch my favorite show of all time and call it studying? Win. Win. Win. And off we go.

The first policy I will be tackling is the Pawnee River cleanup. Program evaluation is a five-step cycle which includes program evaluation, problem definition, policy formulation, policy adoption, and policy implementation. I'll bullet these steps to make the identification easier. 

Problem definition: the Pawnee River needs to be cleaned up

Policy formulation: workers will be hired using grant funding to clean up the river

Policy adoption: once we get the grant money, a bill will be written to begin cleanup

Policy implementation: the cleanup of the river will begin

Program evaluation: temporal analysis of river cleanliness

That's it in a nutshell. Unfortunately I have five more pages of notes so we're about to do some digging, Pawnee-style. 

There are common components in the evaluation process that loosely determine the impact and validity of a program. They are listed below in the scope of the river cleanup:

Theoretical foundation: the theoretical underpinning is the river is dirty and we need to clean it up

Operationalization: the main indicator of this program's impact is the cleanliness of the river

Measurement: sample data will be collected to gauge water quality and aesthetic value will be measured by sporadic sweeps of the river bank

Impact analysis: public feedback, recreational use, and wildlife vitality will be the principle measures of impact

Policy judgement: If the indicators reach a certain level then the program is worth maintaining

Evaluation is considered the “systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy.” With this, there are key elements that help to define policies and programs: systematic assessment, focuses on processes and outcomes, standards for comparison, and the practical purpose of a program. 

For the river cleanup these elements are easy to define. A systematic assessment would be both quantitative (water quality data) and qualitative (aesthetic appeal). The focus on the process and outcomes would be related to how the river was cleaned up and the end result based on data. A standard for comparison would be an equally polluted river funded by federal grant money and the purpose would be obvious.

Focusing on these evaluation efforts, we can begin to discuss the outcomes of the river cleanup based on cause-and-effect questions, descriptive questions, process question, and normative questions, or the questions based on an ideal level of river pollution. 

Evaluation is different than basic research. For instance, a basic research question would seek to understand the level of pollution in the river before and after. A policy evaluation must measure many more things, including utility, program-derived questions, judgmental quality, action settings after the program is complete, role conflicts, publication intent, and allegiance. 

There are also boilerplate questions to consider when implementing a program. Before she hatched the plan to cleanup Pawnee River, these may have been the questions floating around in Leslie Knope's mind:

What are the stakes of the program? A healthier river for all and a quality of life improvement for the citizens of Pawnee.
Does the program impact a large number of people? It will, seeing as the river comprises a large area in Pawnee.
Is the program innovative? Not necessarily, but the river has been neglected and is thus polluted.
Can the program be replicated in other settings? Absolutely.
Has the program been evaluated before? Most likely, but given the town's history not in Pawnee.
Can the results of the program evaluation be used to inform the policy process? Yes.
Is the program strategically relevant? Yes.

So here is the crux of the program evaluation process: the logic model. A logic model is a description of how a program will achieve its intended result. This step is critical in visualizing the ideal outcome for a program and any pitfalls which may await it. The logic model is a three-step process:

1. Implementation - What are the inputs, outputs, and activities in the program? The input into the program will be a federal riverbed preservation grant which will help volunteers and staff clean up the Pawnee River. The major output of the program will be lower levels of pollution in the river and an increased sense of civic pride. Activities will be related to the grant writing and cleaning processes.
2. Results - What are the outcomes and final outcomes of the program? A cleaner Pawnee River and the precedent to keep it that way.
3. Assumptions and Risks - The major assumption Leslie had was the receipt of federal funds to help her clean the river up. She also took the risk that, in the middle of her turbulent tenure as city councilor, her constituency would supper her cleanup efforts.

I'll spare you the actual logic model, but you can see how it works.

After developing a logic model, the next step a trained civil servant like Leslie would have done would be to have developed hypotheses (if-then statements) about the model. Questions like "if we receive federal funding, the cleanup will be finished six months ahead of schedule" and "if we use activated charcoal, then the amount of x pollutant will be reduced." Things like that. The main thing we are looking for in a program is causal inference, or the impact that a program (P), has on an interest (Y). Essentially, if P can be shown to have a positive relation to Y then the program was worth implementing (O = (Y|P = 1) - (Y|P = 0)). This is shown hilariously in such cases as Leslie's bailout of the Pawnee Video Dome (funding (P) brought the Video Dome (Y) out of bankruptcy (positive relationship) but led it to become an adult video store and backlash from Ron and some citizens) and the implementation of a soda tax in Pawnee (the tax (P) led to lower levels of obesity and diabetes (Y) but (unintended consequence alert!) led to public backlash and her eventual recall). Policymakers like Leslie must weigh the hope for a positive causal inference against counterfactual evidence, or what would have happened if a program was never implemented. In the case of the Video Dome, the store would have gone bankrupt, an easy counterfactual. But in the case of the soda tax, would levels of obesity and diabetes have gone down without the tax? It seems obvious, but policymakers can never take outcomes for granted or assume based on past data.



So now we arrive at the end of our first lesson on how to Leslie Knope program evaluations: measuring the impact of a program. There are 5 measures that can be used when evaluating a program: program outcomes, unintended consequences, interim measures of progress, program processes, and program inputs. As a program, cleaning up the Pawnee River is an easy measure. The inputs and processes are defined and interim measures can be analyzed very easily as well.

In public policy today there is a push toward a results- and/or outcomes-based accountability for policymakers who are feeling the heat from their constituency, a 24-hour media cycle, and budget constraints. If you have ever witnessed a public forum in an episode of Parks and Rec, it is simply easier to articulate concrete outcomes to a frothing mass than to micromanage small deadlines and have to explain each and every one. Policymakers must be able to be transparent, sound honest, and reach a mutually-agreed upon bottom line within the collective attention span of militant action groups, overly concerned citizens, and people who showed up just to pick a fight. Rarely if ever does a policymaker accomplish this, but just as Leslie is the definition of the uber-bureaucrat we strive to be, the ideal forum is out there.

Stay tuned for Part 2 of Being Leslie Knope: Policymaking From Pawnee to Program Evaluation 1001.