For the past few weeks, I’ve been building and experimenting with AI agents.
Not just chatbots, but agents that search the web, modify files, interact with Linux terminals, schedule cron jobs, orchestrate MCP tools, and automate real work.
Like many people, I initially evaluated models using the usual metrics:
- Benchmark scores
- Coding ability
- Reasoning capability
- Context window
- Cost
- Speed
Those metrics are useful.
But after spending enough time with autonomous agents, I’ve come to believe they don’t measure the thing I care about most.
Operational reliability.
The morning that changed my mind
Recently I was using Mimo 2.5 as the reasoning model behind my Hermes agent.
Within the span of about an hour, I encountered three separate incidents that completely changed how I think AI models should be evaluated.
None of them were catastrophic.
None of them involved complex reasoning.
Yet all three exposed behaviors that would concern me far more than a low benchmark score.
Incident 1 — A timezone that wasn’t
My Linux server reported:
Time zone: Asia/Manila (PST, +0800)
The model interpreted PST as Pacific Standard Time instead of Philippine Standard Time.
It ignored the authoritative information immediately beside it:
Asia/Manila+0800
From that single incorrect assumption, it concluded:
- my server was running in Pacific Time,
- cron jobs would execute 16 hours differently,
- reminders would trigger at unexpected times,
- I should reconfigure the server’s timezone.
The interesting part wasn’t the mistake.
The interesting part was how confidently the rest of the reasoning grew from that incorrect assumption.
Incident 2 — Correcting me before checking
Later we were discussing Wimbledon.
The model stated that Barbora Krejčíková was the defending champion.
I questioned it.
Instead of checking the web—despite having web search available—it confidently explained why I was mistaken.
Only after I supplied the official Wimbledon website did it search, verify the facts, and admit that I had been right.
The problem wasn’t that it got a tennis fact wrong.
The problem was the sequence.
Instead of:
Verify → Respond
it followed:
Respond → Verify
For sports trivia, that’s annoying.
For an autonomous agent managing infrastructure or financial systems, that’s a much bigger concern.
Incident 3 — A source that never said it
Curious about what had happened, I asked the model what search result originally led it to believe Krejčíková was the defending champion.
It replied that an ESPN article contained the phrase:
“guarantees new Wimbledon champion”
I asked for the link.
It found the ESPN article.
I opened it.
The phrase wasn’t there.
When I pointed this out, the model admitted that it had conflated multiple sources and incorrectly attributed the wording to ESPN.
Again, this wasn’t merely a factual mistake.
It was a failure of source attribution.
None of these incidents were extraordinary
Every frontier model makes mistakes.
I’ve seen factual errors from multiple models.
That’s not the point.
What caught my attention was that all three incidents followed the same behavioral pattern.
The model tended to:
- make an early assumption,
- express it confidently,
- build additional reasoning on top of it,
- verify only after being challenged.
That pattern is much more important than whether the original mistake involved Linux, sports, or something else.
Is this a Mimo 2.5 problem?
Not necessarily.
These incidents all occurred while I was using Mimo 2.5, so it’s fair to document them as observations from that model.
However, I don’t think it’s intellectually honest to conclude from a handful of incidents that Mimo 2.5 is categorically worse than every competing model.
I’ve seen similar behaviors—though not always with the same frequency—from other modern LLMs.
The takeaway isn’t:
“Mimo 2.5 is bad.”
The takeaway is:
Operational reliability deserves to be evaluated independently from intelligence.
Those are two different qualities.
Benchmarks don’t measure this
Traditional benchmarks answer questions like:
- Can the model solve this problem?
- Can it write this code?
- Can it answer this question?
What they rarely measure is:
- Does the model know when it might be wrong?
- Does it verify before disagreeing?
- Does it use available tools appropriately?
- Does it recognize authoritative evidence?
- Does it revise its beliefs when new information appears?
- Does it attribute information to the correct source?
Those questions matter far more once an AI starts acting autonomously.
From bug log to incident database
These incidents convinced me to stop keeping ad hoc notes.
Instead, I’m building an AI Agent Reliability Database.
Rather than recording isolated mistakes, I’m documenting failure patterns.
Examples include:
- Anchored on ambiguous terminology
- Corrected the user before verification
- Failed to verify assumptions
- Ignored authoritative evidence
- Preferred internal memory over available tools
- Incorrect source attribution
- Overconfidence despite uncertainty
Each incident links to one or more reusable patterns.
That way, if a future model exhibits the same behavior, I can compare it objectively rather than relying on memory or anecdotes.
Engineering can mitigate many of these problems
One unexpected lesson is that many reliability issues aren’t solved by changing models.
They’re solved by improving the surrounding system.
For example:
- Pass IANA timezone identifiers (
Asia/Manila) instead of ambiguous abbreviations likePST. - Provide structured data instead of free-form text.
- Require verification before contradicting the user.
- Let deterministic software perform scheduling and time calculations.
- Prefer authoritative sources whenever tools are available.
Good orchestration reduces the opportunity for language models to make reasoning mistakes.
A different way to evaluate AI
I’ve become less interested in asking:
“Which model is the smartest?”
Instead, I’m asking:
- Which model is the easiest to trust?
- Which one verifies before acting?
- Which one admits uncertainty appropriately?
- Which one recovers gracefully after making a mistake?
- Which one behaves predictably in autonomous workflows?
Those aren’t benchmark questions.
But I suspect they’ll become some of the most important questions as AI agents move from demonstrations to production systems.
And ironically, I have Mimo 2.5 to thank for making me realize that.