Oh vs Supermemory

Head to head on LongMemEval-S

On September 24, 2026, this repository ran a matched pilot: the same 60 LongMemEval-S questions went to Supermemory, Oh’s default SDK search, and a BM25 control, each limited to 20 results, answered by the same GPT-4o reader and graded by the same GPT-4o judge under LongMemEval’s own prompts.

On the conservative measure, which counts a failed or unrun cell as zero, Supermemory answered 75.00% correctly, Oh 71.67%, and BM25 68.33%. On the comparison fixed before the run, the Oh − Supermemory difference is −3.33 points with a 95% interval from −13.33 to +6.67: 60 questions, all used earlier in Oh’s development, cannot separate them.

Supermemory’s edge concentrated in knowledge-update, where it answered nine of ten to Oh’s five and where two of Oh’s three ungraded cells also sit. Oh led on single-session-preference (five to four) and single-session-user (nine to eight); the other three types were ties. Supermemory completed all 60 cases; Oh and BM25 completed 57 each after two retrieval failures and one cell left unrun.

The arms differed by design. Supermemory stored one document per session, its published LongMemBench method, and searched with one fixed profile (hybrid mode, top 20, rerank on); Oh and BM25 stored single turns. Supermemory’s search answered in a median of about one second as a hosted API call, while Oh’s median was 12.5 seconds including local reranker inference on the benchmark machine, a development measurement rather than a hosted-service comparison. On the ingestion side, Supermemory’s asynchronous pipeline took a median 61.5 minutes to readiness per case, up to 141 minutes, while Oh indexed each case in a disposable local store. The Supermemory side spent about $43 of Pro-plan credits across the runs, and the shared reader and judge cost $2.28.

• Full result and limits
• Numbers as JSON

What Supermemory runs for you

Supermemory hosts the memory store, the ingestion pipeline, and the retrieval stack. Container tags partition memories by user, project, or any string you choose: each tag maps to its own vector namespace, and API keys can be scoped to specific tags, so a customer’s agent cannot read another tenant’s data. Connectors sync Google Drive, Gmail, Notion, OneDrive, GitHub, Granola, and crawled web pages into the same memory space on webhooks or a schedule.

Pricing is metered in SM tokens at the same rates on every plan, billed monthly with credits that refresh each month, checked September 26, 2026: a free tier with $5 of credits, Pro at $19 a month with $20, Max at $100 with $130 and the Gmail connector, and Scale at $399 with $600 plus the S3 and web crawler connectors, unlimited seats, SOC 2 and HIPAA compliance, and a self-hosted deployment option. Enterprise is custom.

A self-hosted build exists. Supermemory local is a free, open-source single binary that speaks the same API and runs on your own model key, or fully offline on a local model. Its documentation keeps connectors, the MCP server, and the platform’s tuned extraction models on the hosted platform.

What Oh keeps local

Oh writes each record to a SQLite file on your machine: oh put stores a record, oh get reads it back, oh search finds it, and oh verify replays the append-only log of every change and recomputes each record’s SHA-256 digest. There is no account, no credits meter, and no hosted Oh service; Oh is MIT licensed and free.

Each record carries dependency links to the records it rests on, so a fact traces back to the edition, passage, or table it was read from instead of arriving as bare memory text. Keyword search needs no model; semantic search is optional and runs on a local model by default, with a hosted Cloudflare profile you can configure. Operation-level sync to a libSQL database you control is likewise opt-in, and it sends operations rather than database pages.

Choosing between them

• Supermemory when your product needs per-end-user memory with managed ingestion, connectors, scoped keys, and someone else running the retrieval stack, at metered prices.
• Oh when the memory is your agent’s own working knowledge: local by default, every fact linked to its sources, and no account or per-call cost.

The pilot did not separate them on answer quality, so if memory quality for your workload decides the choice, run both on your own questions. The pilot’s protocol, adapters, and cost ledger are checked into the repository.

Limits of this evidence

• 60 exposed questions The sample is previously exposed development questions, run once; the interval describes resampling within this sample only.
• Unpinned models The GPT-4o reader and judge ran as gateway aliases, so a returned model name does not identify an immutable snapshot.
• Uneven granularity Supermemory indexed session documents while Oh and BM25 indexed turns; the run is not a claim about Supermemory’s defaults, its best configuration, or its published Recall@20 result.
• Asymmetric latency Oh’s search time includes local reranker inference on the benchmark machine, and Supermemory’s readiness wait reflects its asynchronous pipeline under a 16-case concurrent load; neither is a like-for-like hosted comparison.
• Agent-run AI agents ran this study, and no person or outside group has audited it.
• Dated claims Supermemory’s pricing, connector list, and self-hosting notes come from its own pages, checked September 26, 2026, and can change.

Sources

• Memory framework pilot result The matched run’s protocol, costs, failure cells, and limits.
• Pilot data as JSON The checked-in numbers behind every figure on this page.
• Supermemory pricing Plans and credit terms checked September 26, 2026.
• Supermemory connectors Connector list and sync behavior checked September 26, 2026.
• Supermemory container tags Per-tag isolation and scoped keys checked September 26, 2026.
• Supermemory self-hosting The local binary and what stays Platform-only, checked September 26, 2026.
• Oh README Install, CLI commands, and what Oh stores.
• Oh ontology specification Records, digests, the operation log, and memory composition.