The Perils of Doing Evaluation

One of the things I love to do is program evaluation. I really like evaluation because through it I am helping community organizations clarify their goals and objectives and making the case for why their work is important. I always approach evaluation in this way--as a positive process that can translate into a better program and more funding.

But, it isn't always a positive experience for me. Oftentimes, evaluators are viewed as threatening--because evaluation is so often tied to funding. This is especially true when a funding agency hires me to "go in" and do an evaluation. I am often viewed in these cases as the enemy.

At other times, the process of collecting data for an evaluation goes very smoothly. It is only after a report is submitted that sparks fly. Example: I was hired to do an evaluation of a program, and had done so since the program was founded--about 5 years. The program got moved to a new department at the organization. The new supervisor of the program didn't like the content of one of the reports that I submitted. So, that person didn't renew my contract. As the person put it, "We terminated your contract because your report was unprofessional."

Here's the rub: this person has labeled me as unprofessional (whatever that means), is telling people that I don't know what I am doing, and that my work is so poor that the funding agency told them to hire a new evaluator! All of which is untrue.

I expect that people might not be happy about the results of an evaluation or might not want to cooperate because there is something they find threatening about the process. But I didn't think that this person would be so disgruntled that they would out and out lie about me!!

So, how do I deal with the fact that this person is sullying my reputation as an evaluator? How do you do a good job as an evaluator, but at the same time not piss people off? You probably can't. I probably can't--and that is the risk of doing a good job. But how am I supposed to deal with the fallout from this person's comments? This person is in a high level position. Even though they have a reputation for not working well with other people, this person has influence. How do I defend my record as an evaluator?

And what lessons do I take from this into the next job I get as an evaluator?

I suppose lesson one is that evaluation is a political process. I don't know if understanding this would have helped me in the current case, but it is something that I didn't really understand until this happened. People often want evaluators to only report what they want to hear--the good stuff. So, some negotiation of difficult terrain is often necessary.

The other lesson I've learned is that it is really important to document everything. I tend to do that anyway, but that has really hit home for me. I saved every e-mail that I sent this person and documented phone calls and conversations that I had with this person, even though they were very limited at the beginning of the year, and then non-existent as it got closer to the time when my report was due to the funder. This person did not return repeated requests for a meeting time to get their feedback on what other people involved with the program were telling me. Since this person claims that I didn't talk to the "right people," I have proof that the attempt was made to do so.

I think the other lesson is that I should use the weight of the funder when necessary to force cooperation. When my request for a meeting was repeatedly ignored, I should have talked to the funder and explained what was happening. At least in this case, the funder would know up front that there was a problem.

Finally, I really appreciate that I have mentors who do this kind of work, have lots more experience than me, and are willing to discuss issues like this with me. I hope to talk with them in the next few days about how to navigate the situation I've found myself in.

I post an update after that meeting.