Twenty Years of the Same Question

In the spring of 2004 I spent weeks on the University of Memphis campus running college students through an experiment about animated agents — walking them through the training exercises one at a time, sitting with each of them afterward for long interview sessions, and screening and scoring the whole thing with the rigor the work demanded.

This was not a class project. It was a joint study run by Rhodes College and the University of Memphis, part psychology and part software engineering, supporting the doctoral research of Dr. Joyce Kim and her associates. I was one of the undergraduates on the team running the sessions, and most of that work happened on the Memphis campus.

The system was iSTART, a reading trainer built around characters who taught you how to read scientific text. Merlin, Dr. Julie, Genie. They introduced strategies — monitoring your own comprehension, paraphrasing, predicting, elaborating, bridging between sentences — and then watched you try. You'd read a passage about thunderstorms, type your own explanation of it, and the agent would respond to what you actually wrote.

Our question was blunt, and it's right there in the title of the study: Are Animated Agents Effective or Superfluous? One hundred and eighty students from Rhodes and the University of Memphis worked through different versions of the trainer — the full agent version, agent and text, voice and text, text alone, voice alone — because we wanted to know whether the character earned its place, or whether the same instruction in plain text did the same work.

The preliminary answer: preference followed the complexity of the condition, with the agent trainer on top, and the agents mattered most for students who had no prior exposure to the strategies. The character helped the people who didn't yet know what they were supposed to be doing, and it mattered less to everyone else.

Twenty-two years later I design and build these systems for a living — the strategy, the architecture, and lately the agents themselves. Which is exactly why I want to be careful about what that experiment does and doesn't mean.

What I'm not claiming

I didn't build Merlin. iSTART came out of a research program with a real team behind it, and my part was the unglamorous middle: running the sessions, doing the interviews, scoring the results alongside the other undergraduates on the team. The findings I described were preliminary, and the reading I'm about to draw from them is mine, looking back — not something the study set out to prove.

I also didn't predict anything. Nobody in that lab was thinking about transformer models, least of all me. The agents on that screen were scripted characters with animation and recorded voice, not systems that could interpret an arbitrary request.

So this is not a claim of priority. It is closer to a claim of stubbornness. The thing I was already curious about in 2004 — whether a system that acts on your behalf should show itself, and what it owes you when it does — turned out to be the thing that matters most now, and I did not arrive at it because the bandwagon came past. I was watching students react to it, one interview at a time, before some of the people building these systems were born, which is a strange thing to be able to say.

The loop that survived

Look at what iSTART actually did, once you strip the animation away.

The agent said what it was about to do. You did the thinking, in your own words. Then the system asked which strategy you'd just used — so your reasoning became visible instead of assumed — and it responded to what you had actually written rather than what it wished you'd written.

Almost none of that system made it to the present. The animated body is gone, and honestly our own results half-predicted that, since the advantage concentrated among people who were new to the material. The recorded voice is gone, along with the 3D lobby and the speech bubbles. What survived is the conduct: announce, let the person work, surface the reasoning, respond to what's real.

I build now with agents that have no face at all — the one I work with all day is text in a terminal — and it still has to do those four things or the work falls apart.

What changed is the stakes

Merlin was semi-autonomous inside a reading passage. He could be wrong about your paraphrase, interrupt at a bad moment, bore you. The worst case was twenty wasted minutes, and the passage was still sitting there when you came back.

The agents I build and govern now book, cancel, refund, merge, deploy, and move money. Some of what they do cannot be undone at all. The same four moves still apply, but they're carrying weight they never carried in a reading trainer, and that weight is what turns a design preference into a governance problem. It's why the questions I work on have shifted from should the agent appear to what is the agent allowed to do, when does it stop and ask, and what can you take back — which is the subject of Designing Agent Conduct and the rest of that series.

The other half of it

There's a second thread, from a year earlier.

In 2003 I presented a poster at the Stanford Undergraduate Psychology Conference about people living with spinal cord injury: whether peer mentoring changed their actual participation in life — work, mobility, social integration, economic self-sufficiency — and whether self-monitoring went with it. Fifty-six participants, all injured at least five years earlier. The mentored group scored higher on four measures of participation, and higher on self-monitoring.

The part that stayed with me wasn't the intervention so much as the measurement problem: whether a thing we believed helped people actually showed up in the life they lived afterward.

That's the same instinct behind the way I build and measure systems now. Not whether the system seems good, but whether the person's experience of it can be measured, and whether it held. The Observable Experience is that question with software in the subject position.

Why this matters to me

Everyone in this field right now has a 2023 origin story. Mine is older and considerably less glamorous: a lab, a reading trainer with a cartoon wizard in it, a stack of interview transcripts, and a question about whether the wizard was worth having. I've been working on that question ever since, in research, then in product and design, and now in the systems themselves.

The last twenty years didn't really change the question so much as raise the price of getting it wrong. In 2004 an agent that overstepped wasted your afternoon; today it spends your money, sends your message, or takes an action you can't reverse, which is why "does the agent show itself" has become "what is it allowed to do, and what do you still control."

As for the question on that study — effective or superfluous — the honest answer took twenty-two years to arrive. The character was the superfluous part. The conduct turned out to be the whole job.

I've been asking whether the agent earns its place since before it could answer.