Naming the question is the work

A few weeks ago I sat in a room at Brown while five speakers thanked each other before Jaron Lanier took the stage. The chair of physics introduced the deputy dean, the deputy dean introduced a Nobel-laureate’s seminar series, the seminar series introduced a former student who introduced Lanier — who had never gone to graduate school, and who began by thanking Brown for not being the other Ivy. By the time Lanier started speaking, the room had already done most of his work for him: it had named the chain of relations the talk depended on, in public, in order.

Late in the evening, in response to a Gen-Z student who asked whether AGI was bullshit, Lanier said something that has been with me since. He told the student, more or less kindly, that he had not asked a question. The room laughed. Then Lanier explained why it mattered. Real power relies on the ability to formulate the problems. The schooling we get from teachers, from parents, from large language models, all rewards the production of correct answers to questions that have already been posed. The harder skill, and the more political one, is the act of posing the question in the first place. Power, he said, lies in the ability to define the problem — and, in defining the problem, to set the paradigm through which the world will be examined.

I have been thinking about that line for two weeks. It is, almost word for word, what Michael Quinn Patton has been saying about evaluation for forty years.

Patton, in plainer clothes

The utilization-focused tradition has a single load-bearing rule: identify the named primary user, the actual human whose decision the report exists to inform, and design backwards from there. Most evaluation plans I’ve inherited from previous evaluators do exactly the opposite. They start with the methods — surveys, focus groups, administrative data — and treat the question of what the report needs to change as something to mention in a closing paragraph.

By then the design has already foreclosed most of the questions worth asking. The data is structured around what was easiest to collect, not around what would actually move a decision. The methods, having arrived first, decide what counts as a finding. The report comes out, the funder thanks everyone for the report, and the program continues to do whatever it was going to do.

The methods, having arrived first, decide what counts as a finding.

What Lanier was naming on stage at Brown, and what Patton has been naming in the evaluation literature since the late 1970s, is the same operation in different vocabularies. The political event in any inquiry is the moment the question gets formed. Everything downstream — the methods, the analysis, the report, the headline number — is the working out of an answer to a question that someone, somewhere, decided to pose. That someone has the power. The rest of us are doing math.

The question I ask before I take the contract

The decision rule I try to hold to in my own engagements is small and unglamorous. Every evaluation I take has, in writing, a single named primary user (or, in larger programs, a small named user group) who can complete the sentence:

“When I read the report, I will use it to decide ___.”

If the named user can’t finish that sentence, we don’t collect data yet. We redesign the question.

This sounds austere. In practice it is the most useful five minutes of any engagement. A coalition director who initially asks for “an outcome evaluation” will, when pressed, often realize that what they actually need is evidence to defend a budget line at the next state hearing — and that need produces a very different evaluation than “outcomes.” The named decision rewrites the design. The data collection gets smaller, the analytic question gets sharper, the report gets shorter and more readable. Nothing has been lost; the engagement has been put on the question that the named user can act on.

What participation looks like, after the question

Once the question is named, the conversation about who else gets to shape it becomes possible. This is where I think a lot of evaluation engagements go quietly wrong. Participation gets discussed as if it were a constant — a posture the engagement either has or doesn’t. In practice it is a set of decisions, each of which can be co-owned with community partners or made by the evaluator alone, and naming which is which is the partnership.

Some questions belong to the funder, who is paying for an answer to a specific decision and is entitled to the answer. Some belong to residents, whose lives are the program’s subject and who have authority over how their experience gets framed. Some belong to me, because they are methodological choices that require evaluation training to make well. The work of an honest engagement is to write that division of authority down, in plain language, before any data is collected, and to invite the partners to argue with it.

I have had resident leadership councils rewrite my list, moving items from my column into the co-owned column, and I have agreed and moved them. That is the conversation I want. The conversation I don’t want is a vague claim of partnership that quietly excludes residents from the decisions that matter most.

The room at Brown, again

I keep coming back to that opening at Brown. Five speakers, in order, naming the relations the evening depended on. It looked at first like a scaffold, the kind of academic pre-amble you wait through before the speaker shows up. By the end of the evening it looked more like the most honest thing in the room. The chain was there whether anyone said it aloud or not; saying it aloud changed what the talk was capable of becoming.

Evaluation, on a much smaller scale, has the same property. The chain — whose decision the report is for, who is paying, whose data this is, who decides what the findings mean — is there whether we name it or not. The engagements I am proudest of are the ones where I named it first, in writing, before anything else got built. The engagements I learned the most from are the ones where I didn’t.

I am still figuring out, in a sober way, how much of this is method and how much of it is something more like a practice. For now I will say only that the question is the work. The methods are how we keep our promises to it.