Twenty Years of the Same Question

In April 2004 I spent an afternoon at a poster in the Frazier Jelke lobby at Rhodes College, explaining an experiment about animated agents.

The system was iSTART, a reading trainer built around characters who taught you how to read scientific text. Merlin, Dr. Julie, Genie. They introduced strategies — monitoring your own comprehension, paraphrasing, predicting, elaborating, bridging between sentences — and then watched you try. You'd read a passage about thunderstorms, type your own explanation of it, and the agent would respond to what you actually wrote.

Our question was blunt, and it's in the title of the poster: Are Animated Agents Effective or Superfluous? One hundred and eighty students from Rhodes and the University of Memphis worked through different versions. The full agent trainer. Agent and text. Voice and text. Text only. Voice only. We wanted to know whether the character earned its place, or whether the same instruction in plain text did the same work.

The preliminary answer: preference followed the complexity of the condition, with the agent trainer on top, and the agents mattered most for students who had no prior exposure to the strategies. In other words, the character helped the people who didn't yet know what they were supposed to be doing, and mattered less to everyone else.

Twenty-two years later I design and build these systems for a living — the strategy, the architecture, and the agents themselves. I want to be careful about what that does and doesn't mean.

What I'm not claiming

I didn't build Merlin. iSTART was a research system, and I was one of twelve student collaborators on a study evaluating it, with Dr. Joyce Kim as our faculty mentor. The results I just described were preliminary, presented at an undergraduate symposium, by an undergraduate.

I also didn't predict anything. Nobody at that poster session was thinking about transformer models. The agents on that screen were scripted characters with animation and recorded voice, not systems that could interpret an arbitrary request.

So this isn't a claim of priority. It's a claim about attention. The thing I was already curious about in 2004 — whether a system that acts on your behalf should show itself, and what it owes you when it does — turned out to be the thing that matters most now. I didn't arrive at it because the bandwagon came past.

The loop that survived

Look at what iSTART actually did, with the animation stripped away.

The agent said what it was about to do. You did the thinking, in your own words. The system asked you which strategy you'd just used, so your reasoning became visible rather than assumed. Then it responded to what you had actually said, not to what it wished you'd said.

That loop survived. Nothing else did.

The animated body didn't survive — and honestly, our own results half-predicted that, since the advantage concentrated among people new to the material. The recorded voice didn't survive. The 3D lobby and the speech bubbles didn't survive. What survived is the conduct: announce, let the person work, surface the reasoning, respond to what's real.

I build now with agents that have no face at all. The one I work with all day is text in a terminal. It still has to do those four things or the work falls apart.

What changed is the stakes

There's one difference between then and now, and it's the whole reason the field is hard.

Merlin was semi-autonomous inside a reading passage. He could be wrong about your paraphrase. He could interrupt at a bad moment. He could bore you. The worst case was twenty wasted minutes, and the passage was still sitting there when you came back.

The agents I build and govern now book, cancel, refund, merge, deploy, and move money. Some of what they do cannot be undone at all. The same four moves still apply, but they're carrying weight they never carried in a reading trainer, and that weight is what turns a design preference into a governance problem. It's why the questions I work on have shifted from should the agent appear to what is the agent allowed to do, when does it stop and ask, and what can you take back — the subject of Designing Agent Conduct and the rest of that series.

The interface question got easier. The consequence question got harder.

The other half of it

There's a second thread, from a year earlier.

In 2003 I presented a poster at the Stanford Undergraduate Psychology Conference about people living with spinal cord injury: whether peer mentoring changed their actual participation in life — work, mobility, social integration, economic self-sufficiency — and whether self-monitoring went with it. Fifty-six participants, all injured at least five years earlier. The mentored group scored higher on four measures of participation, and higher on self-monitoring.

What I cared about there wasn't the intervention. It was the measurement: whether a thing we believed helped people actually showed up in the life they lived afterward.

That's the same instinct behind the way I build and measure systems now. Not whether the system seems good, but whether the person's experience of it can be measured, and whether it held. The Observable Experience is that question with software in the subject position.

Why this matters to me

Everyone in this field right now has a 2023 origin story. Mine is older, quieter, and considerably less glamorous: a poster session, a reading trainer with a cartoon wizard in it, and a question about whether the wizard was worth having. I have been working on that question ever since — in research, then in product and design, and now in the systems themselves.

I don't think the last twenty years changed my question. They changed how much it costs to get it wrong. In 2004 an agent that overstepped wasted your afternoon. Today it spends your money, sends your message, or takes an action you can't reverse — which is why "does the agent show itself" has become "what is it allowed to do, and what do you still control."

The character was always superfluous. The conduct never was.

I've been asking whether the agent earns its place since before it could answer.