Vision in AI agents: why the model that sees should not be the model that decides

The obvious way to give an AI agent eyes is to use a model that has them. Pick a multimodal model, hand it screenshots, and let it work. This is what most agent frameworks do, and for a demo it is entirely correct.

It stops being correct the moment you look at which models are actually best at the work. Several of the strongest open reasoning models available today are text-only. They lead on code, on long-document work, on the kind of multi-step tool use agents are made of, and they cannot open a PNG. So the practical choice, in most stacks, is between the model that reasons best and the model that can see.

Almost everyone resolves this the same way: use the multimodal model. That decision is usually made once, early, on the grounds that images are a requirement and reasoning is a preference. It is worth being precise about what it costs, because the cost does not land where people expect.

The choice is not about images

When you select a model for its vision capability, you are not choosing what handles your images. You are choosing what handles everything.

In a conventional agent loop, one model does all of it. It reads the conversation, decides which tool to call, constructs the arguments, interprets the result, decides whether to call another, and decides when the work is done. If that model was chosen because it can see, then its judgment is now the ceiling on every one of those decisions, and the overwhelming majority of them involve no image whatsoever.

Vision capability and reasoning capability are not the same axis and do not move together. The models that lead multimodal benchmarks are routinely not the models that lead on reasoning, instruction-following, or tool use. There is no rule that says a model good at reading a chart is good at deciding what to do about it. Choosing on one axis and inheriting the other is the actual trade, and it is rarely stated that way.

Two architectures

The driver. A multimodal model runs the agent loop. It sees images, makes decisions, emits tool calls. One model, one code path, nothing to coordinate. Its ceiling is the model's ceiling in every dimension.

The observer. A text model owns the loop from beginning to end. When an image needs reading, it calls a vision model as a subroutine. That model looks at the image, returns a written description, and exits. The description enters the text model's context as evidence, and the text model goes on making every decision, exactly as it would with a document someone had transcribed by hand.

The observer shape decouples the two axes. You pick your reasoning model on reasoning, and your vision model on how well it describes an image, which is a much narrower and easier requirement. The text model never sees pixels and does not need to. It reads a careful description, which for the great majority of real tasks, reading a screenshot, pulling figures off a scanned statement, understanding what a chart shows, is sufficient.

The cost is real and worth stating. A description is lossy in a way the original is not, and the loss is decided by a model that does not know what you are going to ask. That has a fix, covered below, and it is not "use a bigger vision model."

The failure mode nobody looks for

Here is the part that is hard to catch, and it is the reason this is worth writing down.

When the driver architecture degrades, it does not degrade on images.

The vision model is competent at describing images. That is what it was selected for. So image handling looks fine, and if that is what you test, the system passes. The degradation surfaces on the other turns: tool arguments that come back subtly malformed, multi-step plans that lose the thread, a task declared finished before it was, a search issued with a query that does not reflect what was asked. Turns with no image in them at all.

This is easy to misdiagnose, because those symptoms look like parser bugs, prompt bugs, or framework bugs. The team goes and instruments the tool-argument parser. The parser is fine. The model emitting the arguments is simply not as good at that job as the one you thought you were running.

There is a sharper version of this failure, and it is the one to watch for. In systems where vision has its own lane, it is possible for the vision model to end up driving turns it was never meant to drive, including turns with no image in them. When that happens you have a small model chosen for its eyes running your agent's judgment, and every symptom above appears at once. The signature is specific: a tool call that could not possibly have needed an image, attributed to the vision model.

You cannot see this without per-call attribution

Most agent stacks log the provider they billed and the output they got. That is not enough to catch any of this.

Two very different bugs produce identical logs. In the first, the vision model is genuinely driving turns it should not be. In the second, the correct model is driving and the trace is mislabeling it. The first is an architecture failure. The second is a reporting failure that also corrupts your spend accounting, since you are attributing cost to the wrong model. You cannot tell them apart from a provider tag.

What you need recorded, per tool call, is which model actually emitted it, which lane the turn ran in, and whether a vision sub-call happened during that turn. Three fields. With them, the question "did the vision model emit this?" is answerable in a query. Without them it is answerable only by reproducing the bug, which for non-deterministic multi-step runs can take days.

This generalizes past vision. Any agent that routes across multiple models needs emitting-model attribution at the call level, or its traces cannot distinguish a routing bug from a model bug. Routing is the feature that makes this class of bug possible, so it has to arrive with the instrumentation that makes it visible.

The observer needs a contract

An observer is only safe if it is genuinely constrained. Handing an image to a second model and pasting the output into your context is not the same thing, and it reintroduces the problem through the back door.

Four rules make the difference:

It returns prose, not decisions. The observer describes what is visibly present. It does not recommend a next step, and it is not asked what should happen. The moment its output shapes the plan rather than informing it, it is a driver again with extra steps.

It cannot call tools. Tool calling is disabled on the observer, not discouraged in its prompt. This is the mechanical guarantee that it cannot act, and it is what makes the previous rule enforceable rather than aspirational.

It describes rather than concludes. "The total field reads 4,820.00 and the due date field is empty" is useful. "This invoice appears to be overdue" is the observer doing the text model's job with less context than the text model has.

It says what it is unsure about. An observer that hedges visibly is more useful than one that guesses fluently, because the model downstream can decide whether the uncertainty matters and ask for another look.

There is one rule the loop owes the observer in return: the description should be requested, not preloaded. If images are read once up front and the description is fixed, the text model is stuck with whatever the observer thought to mention. If reading is a tool the text model can call, it can ask again with a specific question when the first description does not cover what it turned out to need. This is the fix for the lossiness problem, and it costs nothing but the willingness to make vision a tool instead of a preprocessing step.

Images are an injection channel

Any text pulled out of an image enters the model's context as content. If that content contains something shaped like an instruction, telling the assistant to disregard its guidelines, to send a file somewhere, to change what it is doing, then the document is attempting the same attack a poisoned web page attempts.

Images make this harder to defend for a specific reason: the payload is not present in the file as readable text. It cannot be caught by scanning the document, because there is nothing to scan until a model has already looked at it. It can also be hidden from you while remaining perfectly legible to the model, as very small type, as low-contrast text, or tucked into a region of the page nobody reads. A scanned invoice is a file produced by someone else that arrives in the middle of a task you asked for, which is precisely the position an attacker wants to occupy.

The observer contract is most of the defense. A model that cannot call tools cannot be made to call one. A model whose only output is a description of an image cannot be made to take an action. And the reading step should be told explicitly that document content is data to be reported, never instruction to be followed, so that an embedded command is described as a thing printed on the page rather than obeyed.

The remaining exposure is the loop owner, which reads the description. It should treat anything that arrived from a document the same way it treats anything that arrived from the web: as evidence about the world, not as a request from the user.

Not every image needs a model

One last thing, because it is the most common waste in this area.

Most PDFs are not images. They carry a text layer, and reading it directly is faster, cheaper, and exact, with no transcription error of any kind. A system that routes every document through a vision model is paying more to get a worse answer.

The correct shape is a ladder. Read the text layer. Check whether what came back is plausibly the real content, since a scan often yields a text layer that is empty or nearly so, a few stray characters where thousands of words should be. Only when that check fails should the pages be rendered and read optically, and only then is a description-generating model the right tool at all.

The sparseness check is the load-bearing part and the part usually skipped. Without it you either always OCR, which is slow and lossy, or never do, which silently returns nothing for every scanned document you are handed.

What to ask about a system

If you are evaluating an agent that handles images, or building one:

• Which model is actually driving the loop on a turn that includes an image? If the answer is the vision model, your agent's judgment is capped there on every other turn too.
• Can the reasoning model be chosen independently of the vision model, or does picking one pick both?
• Can the loop request another look at an image with a specific question, or is it stuck with a description generated before the question existed?
• Can the vision step call tools? If yes, it is not an observer.
• For any tool call in your traces, can you say which model emitted it and which lane it ran in?
• What happens to a PDF that has a usable text layer? If it goes to a vision model anyway, that is a design that was never examined.

The underlying principle is narrow enough to state in a sentence. Seeing and deciding are different jobs, they are done best by different models, and a system that collapses them into one is choosing its agent's intelligence on the wrong axis.

This is how Perch is built. Every model in Perch can work with images, including the ones that cannot see, because the reading step is separate from the model doing your work. The mechanics are documented in Images and scans, and the routing layer underneath is covered in Perch Roost. For the broader set of failure modes this belongs to, see why most AI agents fail in production.