2026-09-29
Allorc-Agent vs Hermes-Agent: Memory That Holds, Not Fades
Most AI agents forget. We measured ours across months of conversation -- and it remembered the small things.
Ask an agent what you said last week and watch what happens. It searches, it paraphrases, it summarizes -- and somewhere between your words and its answer, the details disappear. The dinosaur exhibit your kids loved becomes "family activities." The exact week you went camping becomes "sometime in June." A fact you mentioned once, six months ago, never comes back at all.
That is not a prompt problem. It is a memory problem: what gets captured, what stays retrievable, and what survives the trip into the final answer.
We built Allorc's agent around durable, subject-level memory -- then measured it on one of the hardest public long-conversation benchmarks there is.
How we measure long-term memory
We evaluate on LoCoMo-200: a filtered, text-verifiable subset of the public LoCoMo benchmark (Maharana et al., ACL 2024). Each evaluation replays 10 conversations -- 272 sessions, 5,882 turns -- and then asks 200 questions that can only be answered from what the agent retained:
- 100 multi-hop (facts that must be combined across sessions and speakers)
- 70 temporal (dates, order, durations, plans versus outcomes)
- 10 single-hop (direct recall of a stated fact)
- 20 adversarial (questions whose premise is not in the conversation; the correct behavior is to say so, not to invent an answer)
An LLM-as-judge scores every answer 1 or 0: 1 only when every required fact is present and supported. Missing one item, adding one unsupported detail, or presenting a plan as something that already happened scores 0.
We compare against Hermes-agent -- a strong baseline that keeps the whole conversation in its session context with compression. It is a fair, capable reference point for what "read the entire history every time" can do.
Both agents ran on GLM 5.3 Flash with a low reasoning setting: Allorc-Agent V1 and Hermes-Agent v2026.8.16, each using its native memory architecture.
The dataset, runner, and judge are public -- the same ones used for the numbers below: Allorc-official/Locomo-200. Bring your own agent and score it on the same questions.
More than twice the score, on the same 200 questions
LLM-as-judge score on LoCoMo-200
Binary 1/0 judging by a frontier LLM across 200 questions; higher is better
Allorc-agent scores 70.5 out of 100 versus 32.0 for Hermes-agent, a strong baseline that keeps the whole conversation in session context with compression.
By category, the picture is even clearer:
| Category | Allorc-agent | Hermes-agent |
|---|---|---|
| Multi-hop (facts across sessions) | 62 % | 20 % |
| Temporal (dates, order, plans vs done) | 80 % | 49 % |
| Single-hop (direct recall) | 80 % | 40 % |
| Adversarial (refuse false premises) | 70 % | 30 % |
The biggest gap is exactly where long-term memory usually breaks: questions that need facts from two or more sessions, and questions that require knowing when to say "I don't have that."
It answers in about two turns -- not twenty
Work per question
Model turns and retrieval calls, by question category
Model turns per question
Retrieval calls per question
The agent spends about 1.7 model turns and less than one retrieval call per question on average -- consulting the record when a question needs it, not by default.
Good memory is not "send everything, every time." Across the benchmark the agent answers in about 1.7 model turns, calling retrieval less than once per question on average. Many questions are answered straight from what the agent already holds; retrieval kicks in when a question needs it -- and it pays off most on adversarial questions, where consulting the record instead of guessing is the difference between a grounded answer and a confident invention.
The proof is in the old facts
Recall across evidence age
Sessions between the evidence and the question asking for it
Pass rate by evidence age: Allorc recall stays between 63.9% and 79.5% from recent evidence to 16+ sessions back, while Hermes-agent never exceeds 38.3%.
This is the number we care about most. We grouped every question by how far back in the conversation its evidence sat -- from the most recent sessions to 16+ sessions back -- and measured recall in each group:
| Evidence age (sessions back) | 0–5 | 6–10 | 11–15 | 16+ |
|---|---|---|---|---|
| Allorc-agent | 63.9 % | 67.5 % | 79.5 % | 71.7 % |
| Hermes-agent | 24.6 % | 30.0 % | 35.9 % | 38.3 % |
For Hermes-agent, memory is weak everywhere and stays weak. For Allorc, recall does not fade with age: facts from fifteen sessions ago come back as reliably as facts from yesterday. That is the difference between an agent that reads your conversation and one that remembers it.
Why it works
Three design choices make long-term memory reliable:
- Capture the small things. Every durable detail becomes an atomic fact -- a feeling, a quantity, a preference, a date, a plan -- filed under its subject (a person, a company, a project), not buried inside a session summary.
- Never let stored facts fall out of view. The complete remembered set for a conversation is put in front of the agent on every question, and it is protected: our runtime never trims stored memory away to make room. Older facts are not "compressed out."
- Answer with discipline. The agent enumerates every fact that answers the question, keeps dates and quantities exactly as stated, separates plans from completed events, never adds a detail the record does not support, and refuses a false premise instead of inventing around it.
What these four question types look like in real work
The benchmark's categories are not academic -- they are the shapes of the questions your customers and teammates ask every day:
| Benchmark category | The everyday question | What it demands |
|---|---|---|
| Multi-hop | "What did we agree across our last few threads?" | Combining facts from different conversations, people, and months |
| Temporal | "When was that due -- and did it happen?" | Resolving dates and keeping plans separate from outcomes |
| Single-hop | "What's the billing contact on file?" | Exact recall of a stated detail |
| Adversarial | "Did anyone promise a refund?" | Checking the record and saying "nothing on file" instead of guessing |
A memory system that shines on direct recall but guesses under pressure is not ready for customers. One that cannot combine two threads makes your team repeat itself. That is why we report all four categories -- and why the multi-hop and adversarial numbers are the ones to watch.
What this means for you
If your customers talk to your agent over weeks and months -- in support threads, project channels, or email -- the difference between 70.5 and 32.0 is the difference between "let me look that up" and "here is what you told me in March." Long-term memory is not a feature; it is the foundation that makes every other capability useful.
Benchmark: LoCoMo-200, a filtered subset of LoCoMo (Maharana et al., "Evaluating Very Long-Term Conversational Memory of LLM Agents", ACL 2024). Dataset, runner, and judge: Allorc-official/Locomo-200. Judging: LLM-as-judge, binary 1/0 semantic evaluation by a frontier LLM. Setup: Allorc-Agent V1 · Hermes-Agent v2026.8.16 · GLM 5.3 Flash, low reasoning. Baseline: Hermes-agent, a single-session agent with context + compression memory.