Problem
packages/nooa-memory/src/nooa_memory/README.md cites examples/memory_bench/ as the source of its benchmark results, but that directory is not in the repository. It is absent at every relative depth, so this is not a broken-path bug — there is nothing to point at.
Five references:
| Line |
Reference |
Claim made |
| 8 |
examples/memory_bench/ |
"Runnable demos + measured results" |
| 303 |
examples/memory_bench/reflecting.py (make_llm_reasoner) and longmemeval.py (make_llm_reconciler) |
"for working implementations" |
| 400–408 |
examples/memory_bench/ |
"Measured with real gpt-5.4 + text-embedding-3-large", followed by a results table |
| 449 |
examples/memory_bench/ |
"for runnable benchmarks and measured behavior" |
The table at 400–408 publishes quantified results, e.g.:
| `longmemeval.py` | LongMemEval + reconsolidation | memory **+70%**, reflection neutral (+0%) |
So the README states a measured improvement, names the script that produced it and the model it was run against, and points readers at a directory they cannot open. A reader who wants to check the +70% figure, or copy make_llm_reasoner as the docs suggest, has nothing to go to.
Verification
$ find . -type d -name memory_bench # nothing
$ ls examples/
README.md advanced arc_agi_3 assets benchmarks cybergym
nooa-slides-prototype.ipynb quickstart util
What does exist for memory: examples/quickstart/12_memory.py and examples/advanced/memory.py. Neither is memory_bench, and neither contains make_llm_reasoner or make_llm_reconciler.
examples/benchmarks/ exists but holds the Harbor bench-agent adapter (bench_agent.py, harbor_adapter.py, harbor_minimal.yaml), not memory benchmarks.
Why it seems worth fixing rather than ignoring
nooa-memory is a published PyPI distribution, and this README is its main documentation. The reproducibility claim is the load-bearing part: "brain-inspired long-term memory" with a +70% number attached is exactly the kind of assertion a reader will want to run themselves before adopting it. Right now they cannot, and there is no note saying why.
I would guess memory_bench/ exists internally and did not make it into the open-source export — the fact that the README names two specific modules and two specific factory functions inside them reads like documentation written against real code rather than aspirationally.
Possible resolutions
Ordered by what I would guess is most likely, but this is a maintainer call and I do not want to guess in a PR:
- Publish
examples/memory_bench/. Best outcome if it exists and is publishable — the claims become checkable and the docs need no change.
- Point the references at what does exist (
examples/quickstart/12_memory.py, examples/advanced/memory.py) and mark the results table as measured with a harness that is not currently published, so the numbers stay but stop implying a directory the reader can open.
- Drop the references and the table if the harness is not going to be published.
Happy to send the PR for (2) or (3) — both are small and docs-only. I have not opened one because picking between them decides whether published numbers keep an implied source, which is not mine to decide.
Separately, #103 fixes the one link in this file that is just a wrong relative depth (12_memory.py, which does exist). That one is independent of this.
🤖🤖🤖
Problem
packages/nooa-memory/src/nooa_memory/README.mdcitesexamples/memory_bench/as the source of its benchmark results, but that directory is not in the repository. It is absent at every relative depth, so this is not a broken-path bug — there is nothing to point at.Five references:
examples/memory_bench/examples/memory_bench/reflecting.py(make_llm_reasoner) andlongmemeval.py(make_llm_reconciler)examples/memory_bench/examples/memory_bench/The table at 400–408 publishes quantified results, e.g.:
So the README states a measured improvement, names the script that produced it and the model it was run against, and points readers at a directory they cannot open. A reader who wants to check the +70% figure, or copy
make_llm_reasoneras the docs suggest, has nothing to go to.Verification
What does exist for memory:
examples/quickstart/12_memory.pyandexamples/advanced/memory.py. Neither ismemory_bench, and neither containsmake_llm_reasonerormake_llm_reconciler.examples/benchmarks/exists but holds the Harbor bench-agent adapter (bench_agent.py,harbor_adapter.py,harbor_minimal.yaml), not memory benchmarks.Why it seems worth fixing rather than ignoring
nooa-memoryis a published PyPI distribution, and this README is its main documentation. The reproducibility claim is the load-bearing part: "brain-inspired long-term memory" with a +70% number attached is exactly the kind of assertion a reader will want to run themselves before adopting it. Right now they cannot, and there is no note saying why.I would guess
memory_bench/exists internally and did not make it into the open-source export — the fact that the README names two specific modules and two specific factory functions inside them reads like documentation written against real code rather than aspirationally.Possible resolutions
Ordered by what I would guess is most likely, but this is a maintainer call and I do not want to guess in a PR:
examples/memory_bench/. Best outcome if it exists and is publishable — the claims become checkable and the docs need no change.examples/quickstart/12_memory.py,examples/advanced/memory.py) and mark the results table as measured with a harness that is not currently published, so the numbers stay but stop implying a directory the reader can open.Happy to send the PR for (2) or (3) — both are small and docs-only. I have not opened one because picking between them decides whether published numbers keep an implied source, which is not mine to decide.
Separately, #103 fixes the one link in this file that is just a wrong relative depth (
12_memory.py, which does exist). That one is independent of this.🤖🤖🤖