Skip to content

Add /benchmarks page: LoCoMo & LongMemEval results vs. mem0 - #226

Closed
jy7lsna wants to merge 2 commits into
smaramwbc:mainfrom
jy7lsna:feature/benchmarks-page
Closed

Add /benchmarks page: LoCoMo & LongMemEval results vs. mem0#226
jy7lsna wants to merge 2 commits into
smaramwbc:mainfrom
jy7lsna:feature/benchmarks-page

Conversation

@jy7lsna

@jy7lsna jy7lsna commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Description

Adds a new /benchmarks page presenting Statewave's LoCoMo and LongMemEval results head-to-head against mem0 (cloud and OSS), run on mem0's own eval harness with gpt-4o as a shared answerer/judge across all three backends. The page includes an interactive scoreboard, methodology writeup, copy-pasteable reproduction steps per backend, an explicit scope section (what is and isn't claimed), a governance comparison table, and an FAQ. Wired into routing, nav, footer, and the SEO/sitemap route table so it's discoverable and indexable.

Related Issue

Closes #

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to change)
  • 📚 Documentation update
  • 🔧 Maintenance (refactoring, dependencies, CI, etc.)
  • 🧪 Test improvement

Changes Made

  • Add src/pages/BenchmarksPage.tsx: scoreboard (with axis zoom/full toggle and series isolation), methodology pipeline diagram, per-backend reproduce steps, scope/claims section, governance comparison matrix, and FAQ
  • Wire /benchmarks into src/App.tsx (route), src/components/Navbar.tsx (nav link), src/components/Footer.tsx (footer link), and src/lib/seo-meta.ts (title/description/og:type/sitemap entry)
  • Add --series-statewave / --series-mem0-cloud / --series-mem0-oss categorical chart color tokens to src/index.css for light and dark themes, validated for CVD contrast, plus a .section-glow--soft utility for sections carrying saturated chart hues
  • Add an iconOnly variant to src/components/CodeCopyButton.tsx for dense terminal-panel headers

Testing

  • Unit tests pass locally
  • Integration tests pass locally
  • Manual testing completed
  • New tests added for new functionality

Test Commands Run

npm run typecheck   # clean
npm run lint        # clean
npm test             # 321 passed, 2 skipped (26 files)
npm run build        # client + SSR + server build; /benchmarks prerendered and present in sitemap.xml

Checklist

  • My code follows the project's coding standards
  • I have performed a self-review of my code
  • I have commented my code where necessary
  • I have updated documentation as needed
  • My changes generate no new warnings or errors
  • I have added tests that prove my fix/feature works
  • All existing tests pass
  • I have checked for breaking changes

Screenshots / Recordings
image
image
image
image
image
image
image
image
image
image

@vercel

vercel Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

@jy7lsna is attempting to deploy a commit to the Statewave Team on Vercel.

A member of the Team first needs to authorize it.

@jy7lsna
jy7lsna force-pushed the feature/benchmarks-page branch from 7a26a87 to 49b63db Compare August 3, 2026 14:00
Adds a detailed, reproducible benchmark writeup (scoreboard, methodology,
run-it steps, scope, governance comparison, FAQ) run on mem0's own eval
harness, plus the categorical chart palette and icon-only CodeCopyButton
variant it needed.
… the closing CTA

The "Diff the fork against upstream" button compared
statewave-memory-benchmarks against itself. Extract UPSTREAM_DIFF_URL so it
compares against mem0ai/memory-benchmarks, the repo it was forked from.

Also brings the closing CTA in line with the site's cta-card treatment.
smaramwbc pushed a commit that referenced this pull request Aug 10, 2026
Four page and content additions, landed together:

- /benchmarks — LoCoMo and LongMemEval results for Statewave, mem0 cloud and
  mem0 OSS, run in-harness with a shared gpt-4o judge; numbers trace to the
  statewave-memory-benchmarks repo.
- /vs/mem0 — comparison page: deterministic, token-bounded context assembly
  with policy enforcement and receipts vs. ranked similarity retrieval.
- /use-cases/grounded-shop-assistant — use-case page for the grounded shop
  assistant reference build (links smaramwbc/statewave-grounded-shop-assistant).
- Custom header and OG images for all five blog posts.

Combined from PRs #226, #228, #235 and #239 (see those PRs for the full
per-change history and discussion).
@smaramwbc

Copy link
Copy Markdown
Owner

Shipped 🎉 — this landed on main via #237 (squashed as 5d76012) together with #228, #235 and #239. The /benchmarks page is live with your numbers exactly as written — we verified every figure traces to statewave-memory-benchmarks before merging. Thanks for the careful, source-backed work; closing since the content is merged.

@smaramwbc smaramwbc closed this Aug 10, 2026
@smaramwbc smaramwbc mentioned this pull request Aug 10, 2026
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants