Loading header...

Home / DPA Insights / AI Engineering

The Agent That Argued Itself Out of a Job

Use an agent to discover the workflow, then retire it into deterministic code.

The short version

An agent built on Recursive Language Models found a sound procedure for reading dense tables off scanned engineering drawings: find the band, confirm the page orientation, cut the table at the tick marks, read the tiles, check against engineering invariants, re-read whatever fails. Every reliability fix moved decisions from the model into code. By the end, the agent was receiving a pre-solved problem and making one tool call, and the reasoning tokens were still being billed. Written as ordinary Python with no agent at all, accuracy was unchanged, the vision work was byte-for-byte identical and the reasoning half of the bill disappeared. Agentic reasoning pays when the path is unknown on every run. Use an agent to discover the workflow, then retire it into deterministic code.

From this buildAfter the recipe was known
6 steps in the recipe the agent found: band, orientation, tick marks, tiles, invariants, re-read
1 tool call left once every reliability fix had moved into code
0 agent in the Python rewrite. Accuracy unchanged; the reasoning half of the bill disappeared

We faced a difficult extraction problem: reading dense tabular data off scanned engineering drawings, a hundreds of survey readings per sheet, printed as a table beneath a profile graph. No text layer. Just pixels.

Instead of passing the whole image to a single model and hoping for the best, we wanted something that could plan its own path through the data. Recursive Language Models appeared to be the right shape for it. Rather than pushing an enormous payload through a single prompt, an RLM keeps the data outside the model and lets it write code to inspect only what it needs. We gave the agent a small toolkit to locate the drawing’s data band, read a strip of it, zoom into a region and let it plan its own way through the sheet.

Watching the agent work

The first runs were instructive. Watching the agent work showed us what a sound reading procedure looked like: find the band, confirm the page orientation, cut the table at the tick marks so adjacent columns can’t swap, read the tiles, check the output against known engineering invariants, re-read whatever fails.

  • Find the band
  • Confirm the page orientation
  • Cut the table at the tick marks so adjacent columns can’t swap
  • Read the tiles
  • Check the output against known engineering invariants
  • Re-read whatever fails

“We would not have arrived at that sequence as quickly by reasoning about it on a whiteboard.”

Every fix took autonomy away

Then we started fixing the failures, and every fix took autonomy away. The agent spent expensive passes guessing page rotation, so we computed the orientation in code and handed it over with an instruction not to re-check it. Extracted rows flowing back through its context inflated reasoning tokens and truncated the output, so we routed the table through a side channel and instructed the agent to submit an empty list. Free-form planning produced inconsistent tile boundaries between attempts, so we cached them.

By the end, our “agent” was receiving a pre-solved problem, making one tool call, and returning nothing of substance. The reasoning tokens were still being billed.

Ordinary Python, with no agent at all

So we wrote those same calls as ordinary Python, with no agent at all. Accuracy was unchanged. The vision work was byte-for-byte identical and the reasoning half of the bill disappeared, along with a sandbox runtime dependency we had been carrying purely to host the loop.

Table 01 / After the recipe was known
With the agent still running Ordinary Python
What it received A pre-solved problem, then one tool call The same calls, written down
Accuracy Unchanged Unchanged
Vision work Still billed Byte-for-byte identical
Reasoning tokens Still billed The reasoning half of the bill disappeared
Extra cost A sandbox runtime dependency to host the loop No agent at all

Where agency earns its cost

What workedThe RLM as a discovery instrument. It found the recipe.

What didn’tKeeping it in production once the recipe was known. Every reliability fix moved decisions from the model into code, and we continued paying for a planner that no longer planned.

The lesson we took forward is about where agency earns its cost. Agentic reasoning pays when the path is unknown on every run. Ours was unknown once. After that, the honest engineering decision was to write the procedure down.

We now treat this as a deliberate phase: use an agent to discover the workflow, then retire it into deterministic code. The experiment wasn’t the failure. Leaving it running would have been.

Suyash Shrivastava
Senior Vice President - Engineering

Is your agent still planning?

If you run an agent in production, we can look at which decisions it still makes and which have already moved into code, before you pay for another round of reasoning tokens.

Book a working session

Frequently asked questions

What is a Recursive Language Model (RLM)?

Rather than pushing an enormous payload through a single prompt, an RLM keeps the data outside the model and lets it write code to inspect only what it needs. In this project the agent had a small toolkit to locate the drawing’s data band, read a strip of it and zoom into a region.

When is an AI agent worth its cost?

Agentic reasoning pays when the path is unknown on every run. Here the path was unknown once. After the agent found the recipe, the honest engineering decision was to write the procedure down.

Why replace a working AI agent with deterministic code?

Every reliability fix moved decisions from the model into code. The agent ended up receiving a pre-solved problem, making one tool call and returning nothing of substance, while the reasoning tokens were still being billed. Written as ordinary Python, accuracy was unchanged and the reasoning half of the bill disappeared, along with a sandbox runtime dependency.

How do you read tables from scanned engineering drawings with no text layer?

The procedure the agent found was: find the band, confirm the page orientation, cut the table at the tick marks so adjacent columns can’t swap, read the tiles, check the output against known engineering invariants, re-read whatever fails.

Was the agent experiment a failure?

No. The RLM worked as a discovery instrument: it found the recipe. The experiment wasn’t the failure. Leaving it running would have been.

Suyash Shrivastava
Suyash Shrivastava

Senior Vice President - Engineering,

Decimal Point Analytics Pvt Ltd

Suyash Shrivastava is Senior Vice President - Engineering at Decimal Point Analytics. This article is about using an agent to discover a workflow, then retiring it into deterministic code.