Load full task file tree in one request, inline small contents (#998)

* perf: load task file tree and contents in one round trip

The task-definition file pane was slow because every interaction paid a
full browser -> Next.js -> backend -> S3 round trip: one request for the
root listing, one more per folder expand, and one per file click (archive-
backed tasks get no per-file presigned URLs, so content always took the
proxy path too).

- Frontend: fetch the listing recursively once on open and build the full
  nested tree client-side. Folder expands are now instant; the lazy
  per-prefix loader is kept only as a fallback for non-recursive
  responses. The public-share auto-select now prefers instruction.md so
  the nested tree doesn't default to a file inside environment/.
- Backend: archive-backed recursive listings inline small UTF-8 file
  contents (100KB/file, 4MB total caps) straight from the already-loaded
  tarball, so the tree and most file bodies arrive in a single response
  and file clicks render with no further requests. Binary and oversize
  members still go through the per-file endpoint. Inlining copies the
  member dicts so the per-process archive cache stays pristine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJwwqogxTKMCswnP8f4H7N

* perf: drop lazy directory fallback, inline all layouts, add skeleton

Follow-up to the one-round-trip listing change:

- Remove the per-directory lazy-load fallback entirely (loadDirectory,
  loadingDirs, isLoaded, updateTree, buildNodesFromListing). Task trees
  are shallow; the single recursive request is the only load path now,
  and folder expands are pure UI state.
- Inline small file contents server-side for the expanded per-file and
  plain S3 layouts too (concurrent bounded GETs next to S3), matching
  the archive path, so every task listing returns the tree and the file
  bodies in one response. Best-effort: binary/oversize/failed fetches
  fall back to the per-file URL.
- Replace the loading spinners with a whole-pane skeleton that mirrors
  the sidebar + content layout while the listing request is in flight.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJwwqogxTKMCswnP8f4H7N

* fix: clear stale loading flag when selecting a file with cached content

Switching from a file with an in-flight content fetch to one with
inlined/cached content left fileContentLoading stuck true: the cancelled
fetch skips its own reset and the cached-content early return never
cleared it, so the content pane showed the skeleton indefinitely.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJwwqogxTKMCswnP8f4H7N

* perf: stream task file listings — tree first, contents as they load

The one-shot listing blocked on the whole content fan-out before the
client saw anything. Listings can now stream NDJSON (stream=1): a
listing chunk goes out as soon as the tree is known, then per-file
content chunks follow as bodies load, shallowest files first. The panel
paints the tree off the first chunk and patches contents in behind it;
a chunk for the file currently on screen renders immediately.

- storage: parse task tarballs once (single gzip decompress) extracting
  member metadata and inline-eligible text bodies in the same pass, and
  cache (bytes, members, texts) per archive — warm listings and file
  reads do zero decompression. This also fixes the double decompress per
  listing flagged by review. stream_task_files() yields the bare tree
  then contents from the cached texts (archive) or an as-completed
  concurrent S3 fan-out (expanded/plain layouts).
- API: stream=1 on the authed, public-share, and self-hosted task files
  endpoints returns NDJSON via a shared response helper; non-stream
  responses are unchanged.
- Next.js proxies pass NDJSON bodies through instead of buffering.
- Frontend requests stream=1 and consumes the stream progressively;
  endpoints that don't stream (trial files) ignore the param and the
  plain-JSON path still applies.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJwwqogxTKMCswnP8f4H7N

* fix: address stream review findings

- task_expand_handler unpacked _load_task_archive's old 2-tuple; every
  TASK_EXPAND job hitting an archive raised ValueError. Unpack the new
  (bytes, members, texts) shape (and align the test fake, which is why
  the suite missed it).
- make_task_files_ndjson_response now awaits the first chunk before
  constructing the StreamingResponse, so listing-time failures (task not
  found, storage errors) return real HTTP error statuses instead of
  silently truncating an already-started 200 stream.
- Frontend: a stream failure after the tree has painted no longer
  replaces the tree with an error state, and a failed dedicated content
  fetch no longer overwrites a body the stream already delivered.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJwwqogxTKMCswnP8f4H7N

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: pfbyjy

backend/api/routers/tasks.py

@@ -67,6 +67,8 @@
     ensure_experiment_public,
     get_task_file_content_s3,
     list_task_files_s3,
+    make_task_files_ndjson_response,
+    stream_task_files_s3,
 )
 from oddish.core.idempotency import (
     IdempotencyReplay,
@@ -1636,11 +1638,17 @@ async def list_task_files(
         True, description="Include presigned URLs for direct S3 access"
     ),
     version: int | None = Query(None, description="Task version number"),
-) -> dict:
+    stream: bool = Query(
+        False,
+        description="Stream NDJSON: the file tree first, then file contents",
+    ),
+):
     """List all files in a task's S3 directory.
 
     When presign=True (default), includes presigned URLs for each file,
     allowing clients to fetch content directly from S3 without additional API calls.
+    With stream=True the response is NDJSON: a listing chunk as soon as the
+    tree is known, then per-file content chunks as they load.
     """
     auth.require_scope(APIKeyScope.READ)
 
@@ -1654,6 +1662,19 @@ async def list_task_files(
         if version is None and task.current_version:
             version = task.current_version.version
 
+    if stream:
+        return await make_task_files_ndjson_response(
+            stream_task_files_s3(
+                task_id=task_id,
+                prefix=prefix,
+                recursive=recursive,
+                limit=limit,
+                cursor=cursor,
+                presign=presign,
+                version=version,
+            )
+        )
+
     return await list_task_files_s3(
         task_id=task_id,
         prefix=prefix,

frontend/src/app/api/public/experiments/[token]/tasks/[task_id]/files/route.ts

@@ -20,10 +20,24 @@ export async function GET(
       return NextResponse.json(error, { status: res.status });
     }
 
+    const cacheControl = "public, max-age=600, stale-while-revalidate=60";
+
+    // Streamed listings (stream=1) are NDJSON — pass the body through so the
+    // client can paint the tree before the file contents finish loading.
+    const contentType = res.headers.get("content-type") ?? "";
+    if (contentType.includes("application/x-ndjson")) {
+      return new NextResponse(res.body, {
+        headers: {
+          "Content-Type": "application/x-ndjson",
+          "Cache-Control": cacheControl,
+        },
+      });
+    }
+
     const data = await res.json();
     return NextResponse.json(data, {
       headers: {
-        "Cache-Control": "public, max-age=600, stale-while-revalidate=60",
+        "Cache-Control": cacheControl,
       },
     });
   } catch (error) {

frontend/src/app/api/tasks/[task_id]/files/route.ts

@@ -27,13 +27,26 @@ export async function GET(
       return NextResponse.json(error, { status: res.status });
     }
 
-    const data = await res.json();
+    // Presigned URLs expire in 15 min, so cache for 10 min to be safe.
+    // This allows the browser to reuse the listing without hitting the backend.
+    const cacheControl = "private, max-age=600, stale-while-revalidate=60";
+
+    // Streamed listings (stream=1) are NDJSON — pass the body through so the
+    // client can paint the tree before the file contents finish loading.
+    const contentType = res.headers.get("content-type") ?? "";
+    if (contentType.includes("application/x-ndjson")) {
+      return new NextResponse(res.body, {
+        headers: {
+          "Content-Type": "application/x-ndjson",
+          "Cache-Control": cacheControl,
+        },
+      });
+    }
 
-    // Presigned URLs expire in 15 min, so cache for 10 min to be safe
-    // This allows the browser to reuse the listing without hitting the backend
+    const data = await res.json();
     return NextResponse.json(data, {
       headers: {
-        "Cache-Control": "private, max-age=600, stale-while-revalidate=60",
+        "Cache-Control": cacheControl,
       },
     });
   } catch (error) {