Home / DPA Insights / AI Engineering
The Agent That Argued Itself Out of a Job
Use an agent to discover the workflow, then retire it into deterministic code.
An agent built on Recursive Language Models found a sound procedure for reading dense tables off scanned engineering drawings: find the band, confirm the page orientation, cut the table at the tick marks, read the tiles, check against engineering invariants, re-read whatever fails. Every reliability fix moved decisions from the model into code. By the end, the agent was receiving a pre-solved problem and making one tool call, and the reasoning tokens were still being billed. Written as ordinary Python with no agent at all, accuracy was unchanged, the vision work was byte-for-byte identical and the reasoning half of the bill disappeared. Agentic reasoning pays when the path is unknown on every run. Use an agent to discover the workflow, then retire it into deterministic code.
We faced a difficult extraction problem: reading dense tabular data off scanned engineering drawings, a hundreds of survey readings per sheet, printed as a table beneath a profile graph. No text layer. Just pixels.
Instead of passing the whole image to a single model and hoping for the best, we wanted something that could plan its own path through the data. Recursive Language Models appeared to be the right shape for it. Rather than pushing an enormous payload through a single prompt, an RLM keeps the data outside the model and lets it write code to inspect only what it needs. We gave the agent a small toolkit to locate the drawing’s data band, read a strip of it, zoom into a region and let it plan its own way through the sheet.
Watching the agent work
The first runs were instructive. Watching the agent work showed us what a sound reading procedure looked like: find the band, confirm the page orientation, cut the table at the tick marks so adjacent columns can’t swap, read the tiles, check the output against known engineering invariants, re-read whatever fails.
- Find the band
- Confirm the page orientation
- Cut the table at the tick marks so adjacent columns can’t swap
- Read the tiles
- Check the output against known engineering invariants
- Re-read whatever fails
“We would not have arrived at that sequence as quickly by reasoning about it on a whiteboard.”
Every fix took autonomy away
Then we started fixing the failures, and every fix took autonomy away. The agent spent expensive passes guessing page rotation, so we computed the orientation in code and handed it over with an instruction not to re-check it. Extracted rows flowing back through its context inflated reasoning tokens and truncated the output, so we routed the table through a side channel and instructed the agent to submit an empty list. Free-form planning produced inconsistent tile boundaries between attempts, so we cached them.
By the end, our “agent” was receiving a pre-solved problem, making one tool call, and returning nothing of substance. The reasoning tokens were still being billed.
Ordinary Python, with no agent at all
So we wrote those same calls as ordinary Python, with no agent at all. Accuracy was unchanged. The vision work was byte-for-byte identical and the reasoning half of the bill disappeared, along with a sandbox runtime dependency we had been carrying purely to host the loop.
| With the agent still running | Ordinary Python | |
|---|---|---|
| What it received | A pre-solved problem, then one tool call | The same calls, written down |
| Accuracy | Unchanged | Unchanged |
| Vision work | Still billed | Byte-for-byte identical |
| Reasoning tokens | Still billed | The reasoning half of the bill disappeared |
| Extra cost | A sandbox runtime dependency to host the loop | No agent at all |
Where agency earns its cost
What workedThe RLM as a discovery instrument. It found the recipe.
What didn’tKeeping it in production once the recipe was known. Every reliability fix moved decisions from the model into code, and we continued paying for a planner that no longer planned.
The lesson we took forward is about where agency earns its cost. Agentic reasoning pays when the path is unknown on every run. Ours was unknown once. After that, the honest engineering decision was to write the procedure down.
We now treat this as a deliberate phase: use an agent to discover the workflow, then retire it into deterministic code. The experiment wasn’t the failure. Leaving it running would have been.
Suyash Shrivastava
Senior Vice President - Engineering
Is your agent still planning?
If you run an agent in production, we can look at which decisions it still makes and which have already moved into code, before you pay for another round of reasoning tokens.
Book a working sessionFrequently asked questions
What is a Recursive Language Model (RLM)?
Rather than pushing an enormous payload through a single prompt, an RLM keeps the data outside the model and lets it write code to inspect only what it needs. In this project the agent had a small toolkit to locate the drawing’s data band, read a strip of it and zoom into a region.
When is an AI agent worth its cost?
Agentic reasoning pays when the path is unknown on every run. Here the path was unknown once. After the agent found the recipe, the honest engineering decision was to write the procedure down.
Why replace a working AI agent with deterministic code?
Every reliability fix moved decisions from the model into code. The agent ended up receiving a pre-solved problem, making one tool call and returning nothing of substance, while the reasoning tokens were still being billed. Written as ordinary Python, accuracy was unchanged and the reasoning half of the bill disappeared, along with a sandbox runtime dependency.
How do you read tables from scanned engineering drawings with no text layer?
The procedure the agent found was: find the band, confirm the page orientation, cut the table at the tick marks so adjacent columns can’t swap, read the tiles, check the output against known engineering invariants, re-read whatever fails.
Was the agent experiment a failure?
No. The RLM worked as a discovery instrument: it found the recipe. The experiment wasn’t the failure. Leaving it running would have been.