One day someone typed a question about revenue into an accounting application's chat field. The answer was a SQL query. The whole query — table names, conditions, everything — right in front of the customer.
The easy explanation is that the model is bad. That was not the cause.
The system gives the model tools: functions it can call to fetch data. The model does not see the database. It sees a list of tools and their descriptions, and if it needs numbers, it is supposed to call a tool.
A tool call travels through the API as a structured field — separate from text. The response has two fields:
{
"choices": [{
"message": {
"content": null, ← text answer
"tool_calls": [{ ← structured call
"function": { "name": "fetch_numbers", "arguments": "{...}" }
}]
},
"finish_reason": "tool_calls"
}]
}
When the model uses the tool_calls field, the application sees
the call and can execute it. The user sees nothing — only the final answer.
But the model can also do something else: write the same thing
as text in the content field.
{
"choices": [{
"message": {
"content": "[TOOL_CALL]{\"tool\":\"fetch_numbers\",\"args\":{...}}[/TOOL_CALL]",
"tool_calls": null ← no call
},
"finish_reason": "stop"
}]
}
Then the application sees only text. No function to call. And if the
application displays the content field as-is — as a chat
window does — the user sees the raw text.
Measured in this system:
finish_reason: stop, no tool_calls key at all[TOOL_CALL] as text in the content fieldThe last point is what makes this hard. The system did not fail. HTTP 200, valid JSON, a valid response. The only strange thing was its content — and nothing validates the correctness of content.
The exact shape of a tool call is an API contract, not a model feature. The model is trained to produce both structured calls and text, and which one comes out depends on what is in the prompt and which API version is used.
Three common causes, in this case probably all three:
If the system prompt says "you have available a function
fetch_numbers which you call to get the numbers", the model
may interpret that it should write the call. A structured call
requires the tools be passed in the tools parameter, not as
text in the prompt.
When the prompt contains a thousand lines of rules, the
tools definition becomes a relatively small part of the
context. The longer the prompt, the more likely the model imitates the
textual form it sees rather than the structure.
By default the model may choose: call a tool or answer directly. If it judges it cannot, it may "show effort" by writing the call as text. This is especially common when the model does not know the table structure — it invents table names, and an invented table reveals that the guess failed.
In this case the model invented tables that do not exist:invoices,company_id,amount. The real ones wereentries,sales_invoices,purchase_invoices,chart_of_accounts.
Most APIs support tool_choice: "required", which forces the
model to call some tool. It works — but breaks normal conversation. When
the user says "hi", the model cannot answer, because it must call a tool.
Usable only when all traffic is tool-driven. Not in a chat window.
Find the [TOOL_CALL] block in the response and interpret it.
This is tempting because it "fixes" the problem visibly. But it is also a
contract you have to live with: the model's textual format is not
documented, not versioned and not guaranteed. It can change in the next
model update.
Accept that the model sometimes leaks, and clean it away before the user sees it. Do not try to fix the model's behaviour — fix the result.
This is what we did, because it is the only option that does not depend on the model's internal logic. It is not beautiful, but it is deterministic: the same input always produces the same output.
The format gate is not one rule but four, and each corresponds to a different kind of leak:
| Leak | What is removed |
|---|---|
[TOOL_CALL]...[/TOOL_CALL] | The whole block, start to end |
| A code fence containing SQL or JSON | The whole fence with its contents |
At line start, [SQL] or a bare SELECT | That line |
| At line start, "Searching…", "Checking…", "I will use the tool…" | That line |
The fourth line is the one I did not understand at first. Once the blocks are removed, what often remains is a promise of what the model was supposed to do:
"Searching for revenue in the accounting of Company 2."
This is neither an answer nor an error. It is a description of intent left behind when the actual call was removed. To the user it looks like the system is about to do something but never gets there.
The first version of the cleaning used a rule: if more than half the response was removed and fewer than 40 characters remained, reject the whole response. A reasonable-looking threshold.
It broke a correct answer:
Result: 1,121,280 EUR ← 19 characters, but entirely correct
A short answer is not a bad answer. The fix was simple but not obvious: a number alone is enough content. If the remaining text contains a number, it is data. Pure meta-text contains no number.
This is the detail no documentation tells you, and that you learn only by seeing a wrong rejection.
If you are building the same: put the gate in the response path, not in the prompt. A prompt is a wish, a gate is a check. And record what the gate removed — that number says more than any alert.
Todenta does this as part of its AI governance layer. Read what else it does →
Todenta · Lennoware Oy · Home · One page · Privacy policy