Gayblack Canadian Man

Foreign Policy Analysis
Highlights: Experiments in Public Management Research: Challenges and Contributions

Highlights: Experiments in Public Management Research: Challenges and Contributions


(upbeat music) – So this is what experimental studies do. They involve an intervention
or a manipulation that tries to change X in order to influence Y and then, if you can kinda
make Y change by changing X, you’ve shown Cause and Effect. This kind of evidence, when you’re not manipulating X, is just relying on the assumption that X may be causing Y. One thing you can see though, right away, is that sometimes manipulating something like government performance is not easy. (laughing) But it’s important to realize that, in modern experiments, intervention alone is not enough. So, the basic set-up is
you have a condition, you intervene, and then you observe a
change in the condition. In any kind of before
and after comparison, you don’t know what would have happened. Conditions change, things improve, sometimes if you just
leave something alone it will change anyway, right? There’s a condition,
there’s an intervention to change the condition. The condition changes, but you also have to know
what would have happened, had the intervention not occurred and that’s what’s referred
to as a counterfactual, but the problem is how do
you observe a counterfactual? So, how would we observe Obama had he not been president, right? The solution is the randomized
comparative experiment, where you have a control group that represents the counterfactual. You apply an intervention
and you observe an outcome, but then you have another group that resembles the treatment group that you also observe, but you don’t apply the treatment and the control group mimics
what would have happened to the treatment group had
it not gotten the treatment. And that control group is
formed by random assignment, which guarantees statistical equivalents on both observed and unobserved variables. Many of the limitations
of observational studies have to do with the fact that there could be important variables that you can’t observe
or you can’t measure. Experiments can be used to check on and try to replicate or test results that come from more survey-based analysis. A study I proposed on a
change in job satisfaction in the public sector after
the 9/11 terrorist attack. My hypothesis was this, basically that the 9/11 terrorist attacks, this is the presidential approval rating before and after the terrorist attacks. After the attacks, the
approval ratings shot up, but this outpouring of positive feelings about government might influence the way government workers
felt about their work and also that the 9/11
attacks might have changed the meaning of public service. Workers would actually be
more satisfied with their job after the 9/11 attacks
and before or relative to the private sector. So, I use the private
sector as a control group. So, this is the mean job satisfaction and you can see for the public sector it shot up after 9/11. So, in the private sector, it declined. There was a recession
after the terrorist attack. So, it’s not by any means
a randomized experiment, but you can see it
still has the same logic because there was an event, an exogenous event, presumably it may have affected one group more than the other and it bares it out. Experiment can contribute a lot to the theory testing and development, especially demonstrating cause
and effect relationships. They have limitations. They’re often artificial. They focus on things
that can be manipulated. Not everything in the
world can be manipulated. And there are ethical constraints. Observational studies remain important. Qualitative studies remain important. A mixed methods approach,
I think, is often needed. I don’t think the field should
be entirely experimental anymore than it should be
entirely regression analysis or should be entirely qualitative
or historical analysis. (upbeat music)

Leave a Reply

Your email address will not be published. Required fields are marked *