gui: Extractors sits under "COMMON — BOTH METHODS" but datanodes is hard-capped at 8 lanes

[LeyckerS]

Found by running the program, not by reading it — the operator set Extractors 32 and observed that datanodes was far slower than fuckingfast on the same batch, and suspected the "common" settings were not really common. They were right.

What the code does

moon_extract.py:878:

DN_LANES = max(1, min(int(os.environ.get("MOON_DN_LANES", "3") or 3), 8))

Datanodes extraction runs through _lane_queue, which holds exactly DN_LANES permits. The docstring at moon_extract.py:1229 is explicit: "Blocks (awaits, does not spin) once DN_LANES pages are already open — excess workers queue here instead of piling more tabs."

So datanodes extraction is capped at 8 concurrent pages, and only 3 unless raised. Setting Extractors to 32 gives datanodes no additional concurrency at all: the other 24+ workers simply block on the lane queue.

For fuckingfast — pure HTTP, no lanes — all 32 workers really do run.

Why this is a bug and not just documentation

The GUI panel is titled "COMMON · BOTH METHODS" and Extractors is the first slider in it, with "rec. 16" next to it. That tells the operator the value applies to both providers. It does not. The provider-specific control for datanodes is Pages (dn_pages), which lives in a different panel.

The measured consequence, same 82-link batch: fuckingfast finished in 4m02s, datanodes took 6m09s for 72 links (10 were genuinely dead). The operator reasonably concluded something was structurally wrong with the settings.

What needs deciding

Either:

1. Move Extractors out of the COMMON panel, or annotate it — something like "datanodes is limited to Pages lanes regardless of this value", or
2. Clamp what the slider offers when it cannot be honoured, so the number shown is the number that runs, or
3. Let dn_pages follow Extractors where the 8-lane ceiling allows it, so one control means one thing

Option 1 is the smallest and the least likely to break anything. Whoever takes this should say which they picked and why.

Related to #45, which is the same class of problem on the CLI side: a control whose name promises something it does not do.

Files

• web/index.html — the panel markup and its heading
• web/app.js — the slider wiring
• moon_extract.py:878 — where the ceiling is set

[LeyckerS]

Measured, on two runs of the same 82-link batch. This is no longer just a reading of the code.

	fuckingfast	datanodes
Wall clock	242.5s	369.9s
Session speed	172.0 MB/s	101.8 MB/s
Median per-file speed	4.2 MB/s	26.2 MB/s
Median download time	120.6s	20.4s
Files in flight, average	≈40	≈4

Both runs had dl_streams = 48 and browsers = 32.

The in-flight figure is derived from the report: files × median download time ÷ wall clock. fuckingfast keeps around 40 of the 48 download slots busy. Datanodes keeps about 4.

That is the lane ceiling made visible. Each datanodes file downloads six times faster individually — 26.2 against 4.2 MB/s, because it has the pipe nearly to itself — and the run still takes half again as long, because extraction can only feed 8 lanes and 40+ download slots sit idle waiting for links.

So the operator's original complaint was exactly right: raising Extractors to 32 did nothing for datanodes, and the panel that says "COMMON — BOTH METHODS" is why they expected it to.

Whoever takes this now has a number to beat: datanodes should be able to use more than 4 of its 48 download slots.

[LeyckerS]

Controlled measurement from the CLI: same 5 datanodes links, same machine, only MOON_DN_LANES changed.

	3 lanes (default)	8 lanes (the ceiling)
Wall clock	54s	44s
Session speed	48.5 MB/s	58.9 MB/s
Peak aggregate	183.6 MB/s	224.8 MB/s
Files downloading at peak	3	5
Extraction rounds	2 (3 links, then 2)	1 (all 5)

Raising the lane count removed the second extraction round: 19% faster overall, 21% more throughput, purely from letting more links extract at once. That is the mechanism this issue describes, confirmed rather than inferred.

But there is a second cost the lane count does not touch, and it is the bigger one here. In both runs the first download began at roughly 00:26. Lanes decide how many links are extracted per round; they do nothing about how long a round takes. That ~26 seconds is the fixed price of a datanodes extraction — Chrome start, page load, Turnstile — and on a 5-link batch it is 59% of the entire run.

So the ceiling has two separate consequences:

1. Batches larger than the lane count pay the fixed cost repeatedly, once per round. That is what this issue is about, and raising the ceiling helps.
2. Every datanodes batch, however small, pays it at least once. Raising the ceiling does not help there at all.

Worth keeping both in mind when deciding what "fixed" means — an operator who sets Extractors to 32 is expecting the first, and will still be surprised by the second.

Measured on the CLI, which cannot set lanes at all except through the MOON_DN_LANES environment variable.