How I structure an evaluation engagement

Most evaluation plans I’ve inherited from previous evaluators are written backwards. They start with the methods — surveys, focus groups, administrative data — and treat the question of what the report needs to change as something to mention in a closing paragraph. By then it’s usually too late: the data is already structured around what was easiest to collect, not around what would actually move a decision.

The CDC Framework for Program Evaluation gets this right in principle. Engage stakeholders, describe the program, focus the design, gather credible evidence, justify conclusions, ensure use. The trouble is that the framework is silent on the decision rules that operationalize each step — and without those rules, “engage stakeholders” quietly becomes “send them the draft for review” and “ensure use” quietly becomes “hope they read it.”

What follows is the working method I bring to engagements. It’s not a manifesto. It’s a set of small commitments, each of which has a name and a place in the plan, so that I and the people who hire me can hold each other to them. I’ve found that the engagements where the method holds best are the ones where the funder, the program staff, and the residents who participate all read the same evaluation plan and recognize their part of the bargain in it.

i.Name the primary user before naming the methods

Patton’s utilization-focused frame insists on one thing above all: identify the named primary user, the actual human whose decision the report exists to inform, and design backwards from there. In practice this is harder than it sounds. Programs often have many stakeholders, and there is real political pressure to name them all and call it “participatory.” That conflates two different things.

My rule: every engagement I take has, in writing, a single named primary user (or, in larger programs, a small named user group) who can complete the sentence: “When I read the report, I will use it to decide ___.” If the named user can’t finish that sentence, we don’t collect data yet. We redesign the question.

If the named user can’t tell me which decision the report will inform, we redesign the question before we collect data.

This sounds austere. In practice it’s clarifying. A coalition director who initially asks for “an outcome evaluation” will, when pressed, often realize that what they actually need is evidence to defend a budget line at the next state hearing — and that need produces a very different evaluation than “outcomes.” Naming the user surfaces the decision; naming the decision rewrites the design.

ii.Apply the three equity questions to every row of the plan

Equity-centered evaluation, in too many engagements, lives in an appendix. The methods are designed for population averages; the disaggregation happens at the end; and a section gets added titled something like “Equity considerations.” The problem isn’t the section. The problem is that by the time it’s being written, the design has already foreclosed most of the equity questions worth asking.

I work with three equity questions, and they ride every row of the evaluation plan from the start:

• Who benefits? Not in aggregate — in the specific cuts the program is supposed to serve. Whose outcomes is the program theory of change actually claiming to move?
• Who is missing from the data? Not just “hard-to-reach populations” in the abstract, but: which residents’ experiences will be invisible in this dataset, and what will that invisibility cost the conclusions we draw?
• Whose voice shapes the narrative? Who decides what the findings mean, in what order, and which framings reach the funder? This is usually the most consequential equity question, and the easiest one to leave unanswered.

These three questions go in the evaluation plan, in the methods, in the analysis, and in the reporting cadence. They do not live in a closing paragraph. When a quarterly review meeting opens, we open by asking them again, in that order. The discipline of repetition is what keeps them from becoming decorative.

iii.Scope participation honestly — symbol vs. authority

Participatory evaluation is the term we use for a wide range of practices that are not always the same thing. At one end, residents and program staff actually share authority over evaluation decisions: which questions to ask, how to interpret findings, what to do about them. At the other end, residents review a draft someone else wrote, and their comments are folded in if they’re convenient.

Both can be called participatory. Only one is. I think evaluators owe their partners more honesty about which they’re offering.

My approach: every engagement I write makes explicit, in plain language, which decisions are co-owned with community partners and which are mine to make. Co-owned decisions are listed by name — usually questions of which findings get foregrounded, how stories get told, what counts as “good enough” evidence. Mine include methodological choices that require evaluation training to make well, the consistency of analytic frameworks across sites, and the integrity of what the report says is true.

Naming the line lets a partner push on it. I’ve had resident leadership councils rewrite my list, moving items from my column into the co-owned column, and I’ve agreed and moved them. That’s the conversation I want. The conversation I don’t want is a vague claim of partnership that quietly excludes residents from the decisions that matter most.

iv.Let the method follow the question, not the inventory

Evaluators have inventories of methods we know well, and there is a temptation to apply the ones we like. Surveys, because we have a good template; focus groups, because we have a trusted facilitator; admin data analysis, because the dataset is sitting there. None of these is wrong. They become wrong when they precede the question.

Once the named primary user has finished the sentence about what the report will inform, the question of methods becomes a much smaller question. Often a single, well-designed cohort interview series will do more for the decision than a 30-question survey would. Sometimes the right move is to integrate with an existing administrative dataset and avoid burdening residents with new data collection at all. Sometimes the question requires both, and the methods need to be designed to triangulate rather than parallel each other.

The CDC Framework treats this as “focusing the evaluation design.” In practice I think of it more simply: what is the smallest, lightest data infrastructure that can credibly answer the named user’s named question? The answer is almost always smaller than the proposal first imagined.

v.Differentiate the reporting cadence by primary user

One of the most common failures of evaluation engagements is that everybody gets the same report. Funders get a 60-page document; coalition staff get the same 60-page document; residents get a glossy two-pager that strips out the analysis. None of those audiences is well served, and resentment compounds: the funder feels that the report is bloated, staff feel it’s academic, residents feel it’s condescending.

Better, in my experience, is to design the reporting cadence around each named user up front:

• For the funder or program officer: a quarterly brief written to the structure of the federal or state reporting template, so it can be lifted directly into their work product.
• For coalition leadership: a narrative-and-numbers internal memo, monthly, focused on what is working, what is breaking, and what to try next. Written for a leader who has 20 minutes to read it before a board meeting.
• For residents and partner CBOs: a dashboard or short bilingual summary that translates findings into the form residents can use to advocate, organize, or hold the program accountable. Not a watered-down version of the staff memo — a different document that serves a different need.

This costs more time than producing one report. It also dramatically increases the chance that the evaluation actually changes anything. If the funder needs different evidence than the coalition does, then producing one document for both means producing one document for neither.

vi.Be explicit about what I won’t do

A method is partly defined by what it refuses. There are a small number of engagements I decline, even when the budget is right, because I know in advance the conditions for the method to hold won’t be met:

• Engagements where the program team can’t name a primary user who can describe the decision the report will inform. This is usually a sign the evaluation is being commissioned for compliance reasons rather than learning ones, and a different kind of evaluator is the right fit.
• Engagements where the funder requires a predetermined finding. Sometimes this is explicit; more often it’s implicit in a scope that asks the evaluation to “document impact” rather than “test whether impact occurred.”
• Engagements where participation with community partners is presented as central but the budget allocates no time or compensation for residents to actually participate. Authentic participation costs money. If it’s not in the budget, it’s not in the engagement.

I name these in scoping conversations rather than after a contract is signed. It saves everyone time and produces, occasionally, the much better outcome of a redesigned engagement that fits all three constraints from the start.

vii.What this all sits on

None of this is original. The CDC Framework, the utilization-focused tradition Patton has been refining for forty years, the equity-evaluation literature of the last decade — the field has been here before, repeatedly, and the work I do is downstream of that literature.

What I’ve tried to do is convert ideas that often live as principles into decision rules with a place in the plan. A principle a partner agrees with is good. A decision rule a partner can hold me to is better. The difference between them is what I think it means to take utilization-focused, participatory, equity-centered evaluation seriously in 2026.

If you’re considering an engagement and any of this resonates — or doesn’t, and you want to push back — I’d welcome the conversation.