Benchmark index
Benchmark results
Oh measures its memory on public benchmarks and publishes each study’s protocol, cost, and limits in the repository. This page lists every completed comparison with its headline scores and date. Scores are the share of questions an answering model answered correctly from the memory each system prepared. AI agents ran these studies, and no person or outside group has audited them.
LongMemEval-S: all 500 questions, three runs
| System | Score |
|---|---|
| Oh reading pipeline | 93.07% |
| First pass only | 91.20% |
| Oh semantic retrieval | 88.87% |
| BM25 retrieval | 86.13% |
GPT-5 mini answered each question three times from each system’s memory, and GPT-4o graded the answers with LongMemEval’s own prompts. The pipeline reads every message the user wrote plus the assistant replies retrieval ranks highest, within 180,000 bytes; the two retrieval arms keep their top 100 turns within 96,000 bytes. The pipeline’s log builder, date rules, and re-read rules are not part of the Oh package, and they were written after studying all 500 questions, so its score is in-sample. On the measure fixed before the run, questions answered correctly in at least two of three runs, Oh semantic retrieval leads BM25 by 2.8 points with a 95% interval from 0.0 to 5.6, which does not rule out a tie.
Oh, Supermemory, and BM25 on 60 LongMemEval-S questions
| System | Score |
|---|---|
| Supermemory | 75.00% |
| Oh default SDK search | 71.67% |
| BM25 keyword retrieval | 68.33% |
All three systems received the same histories, the same single query per question, at most 20 returned results, and the same GPT-4o reader and judge. Supermemory stored one document per session, as its published method does; Oh and BM25 stored single turns. The result separates no pair: Oh minus Supermemory is −3.33 points with a 95% interval of −13.33 to +6.67, and Oh minus BM25 is +3.33 points (−1.67 to +8.33).
CloneMem: the default SDK search
| System | Score |
|---|---|
| Oh default SDK search, local reranker | 80.59% |
| Oh semantic retrieval | 69.86% |
GPT-4o mini picked an answer from each question’s options three times, reading the top 10 results within 96 KiB; a pick counted when it matched the correct option. The candidate is the real Oh.search route with the local reranker, and the control is the same store’s semantic search. The gain is +10.73 points with a 95% bootstrap interval of +4.57 to +16.89. Both personas improved, but the questions come from two personas already used in development, and no other memory framework ran.
CloneMem: the local reranker on reserved personas
| System | Score |
|---|---|
| Local reranker over the lexical-semantic union | 77.82% |
| Vector retrieval | 70.54% |
A pinned local Qwen3 reranker reordered the union of lexical and semantic candidates, returning the top 10 under the same answering model, scoring, and reading limit as the SDK study. The gain is +7.28 points with a 95% persona-cluster interval of +4.61 to +10.27. The seven personas were held back from development but had earlier project exposure, and a retry rule was added after two failed campaign attempts. The reranker ships in Oh as the opt-in mode: "rerank".
LoCoMo: packing context with matched turns
| System | Score |
|---|---|
| Fixed windows | 78.11% |
| Query-aware packing | 77.33% |
Filling each question’s context with the nearby turns that best match it raised evidence-turn recall from 88.93% to 90.08% across 1,586 questions, but judged accuracy fell on the 300-question sample: the difference is −0.78 points with a 95% interval of −4.64 to +2.34. GPT-4o mini answered three times per method from the same top 20 vector matches within 12,000 bytes. The variant stayed an experimental adapter, not a change to Oh’s default search.
LoCoMo: all 1,540 scored questions
| System, 24 KB | GPT-5 mini | GPT-5 nano |
|---|---|---|
| Oh semantic | 84.4% | 81.0% |
| BM25 window | 81.6% | 78.1% |
The study, its question list, and its scope were sealed by digest before the first reader call. It covers the four scored categories and excludes the 446 adversarial questions; a gpt-4o-mini judge graded each answer once, and all 11,065 calls completed for $11.82. Paired on the same questions, Oh semantic beat BM25 window 126 to 83 with mini and 169 to 124 with nano. Every conversation had been exposed to earlier studies, and the record gives no confidence interval.
LongMemEval-S: single-run comparisons
| System, 96 KB | GPT-5 nano | GPT-5 mini |
|---|---|---|
| Oh semantic | 75.8% | 89.8% |
| BM25 window | 75.6% | 85.4% |
| Full history | 71.0% | not run |
Each reader answered all 500 questions once over the same frozen contexts: the top 100 turns within 96,000 bytes. With GPT-5 nano, full history trailed both retrieval arms and cost about five times as much per answer. These runs kept the dataset’s original session identifiers, and an audit later found they can carry answer-label wording, with an unmeasured effect on scores. Later studies replace the identifiers; these numbers stand as historical measurements.
LongMemEval-S under the MemEval harness
| System | Reader | Score |
|---|---|---|
| Oh semantic, 96 KB | gpt-4.1 | 61.8% |
| Oh semantic, 96 KB | gpt-4.1-mini | 60.8% |
MemEval is a third-party harness whose protocol lowers every system’s score here: it drops the question date, uses a short generic answer prompt, and caps answers at 50 tokens. On the first 30 questions of the same sample, its Mem0 open-source adapter scored 43.3% against Oh’s 60.0% with the same reader; 30 questions support no precise margin. The run needed three local harness patches, which the record lists.
Every study, passed and failed
The repository’s benchmark README lists every study, including the ones that failed their pre-set rules and the ones a later document replaced, and gives the dataset pins and commands needed to reproduce the measurements. No study measures an agent writing or updating its own memory during work, or resuming a task from memory.