Skip to content

banana_eval_dataset.md

Latest commit

 

History

History
276 lines (199 loc) · 8.62 KB

banana_eval_dataset.md

File metadata and controls

276 lines (199 loc) · 8.62 KB
Raw

Banana Evaluation Dataset

This document describes the banana bowl LeRobot datasets used for Eval 1 and Eval 2 fine-tuning experiments.

Purpose

The banana task has one fixed demonstrated motion per target position:

rslxcvg/banana_blue   target position: left
rslxcvg/banana_red1   target position: center
rslxcvg/banana_green  target position: right

The original collection layout is:

left bowl:   blue
center bowl: red
right bowl:  green

The color-swap builder renders additional color layouts while keeping the same state/action trajectory. Prompt text must be generated from the rendered bowl layout, not from the source dataset name.

LeRobot Schema

The builder preserves the source LeRobot SO follower schema:

observation.images.front  front RGB video, 480x640
observation.state         float32[6], current calibrated follower joints
action                    float32[6], calibrated absolute joint action
task                      language instruction string
task_index                integer index into meta/tasks.parquet

Only observation.images.front and task are changed by the augmentation. observation.state and action are copied from the source episode unchanged.

Builder

Use:

.venv/bin/python data_processing/augment_banana_lerobot.py \
    --output-repo-id rslxcvg/banana_act_position_targetcolor_v2 \
    --root outputs/lerobot/banana_act_position_targetcolor_v2 \
    --mode position_target_colors \
    --prompt-mode eval_mixed \
    --auto-trim-static-start \
    --overwrite

Add --push-to-hub --private only after the local validation checks pass.

Do not use older local outputs unless they were rebuilt after the eval prompt and static-start trim changes:

outputs/lerobot/banana_act_position_targetcolor_v1
outputs/lerobot/banana_act_eval_prompts_v1

banana_act_position_targetcolor_v1 was observed as stale/partial locally: its metadata reported 5 episodes and 1 task, and its parquet data file was unreadable.

--auto-trim-static-start removes source episode start pre-roll by finding the first sustained robot-state and image-motion onset. This filters table/init segments without applying a blunt fixed crop to episodes that start active.

Augmentation Modes

--mode position_target_colors

Creates three rendered color layouts per source episode. The demonstrated target-position bowl is rendered once as red, once as green, and once as blue. This is the right mode for Eval 1 and Eval 2 language-conditioned VLA training.

--mode labelsafe_colorpos

Keeps the demonstrated target color fixed and swaps only distractor bowls. This is useful for label-safe visual color robustness, but it does not create the full Eval 1/Eval 2 language prompt mix by itself.

Prompt Modes

--prompt-mode eval_mixed is the default and recommended VLA setting. For each rendered episode it writes one copy for each prompt bucket:

direct_color
position_ordinal
relative_color
exclusion

--prompt-mode position keeps one legacy position-only prompt per rendering. Use this only for debugging position robustness.

--prompt-mode eval_sampled writes one deterministic prompt from one eval bucket per rendering.

--prompt-mode all_eval writes every template in every bucket.

Prompt Semantics

For each rendered episode:

colors_by_position = (left_color, center_color, right_color)
target_position = left | center | right
target_color = colors_by_position[target_position]

All text is derived from colors_by_position and target_position.

Example for source rslxcvg/banana_blue with target position left:

rendered layout: left red, center blue, right green
target bowl:     left red

direct_color:      Move the banana to the red colored bowl.
position_ordinal:  Put the banana into the left bowl from the robot perspective.
relative_color:    Put the banana into the bowl on the left of the blue bowl from the robot perspective.
exclusion:         Put the banana into the bowl that is not green and not blue.

Example for source rslxcvg/banana_red1 with target position center:

rendered layout: left blue, center green, right red
target bowl:     center green

direct_color:      Put the banana in the green bowl.
position_ordinal:  Put the banana into the 2nd bowl from the left from the robot perspective.
relative_color:    Put the banana into the bowl between the blue bowl and the red bowl.
exclusion:         Put the banana into the bowl that is not red and not blue.

Expected Counts

Let N be the total number of source episodes across the three source repos.

For --mode position_target_colors --prompt-mode eval_mixed:

renderings per source episode: 3
prompt copies per rendering:   4
output episodes:               N * 12

If each source repo has 20 episodes, the expected full output is:

source episodes: 60
output episodes: 720

Task strings are deduplicated in meta/tasks.parquet, so the number of unique tasks can be smaller than the number of output episodes.

Static-start trimming changes total frame counts. It should not change the expected output episode count.

Verified Smoke Dataset

The local smoke dataset inspected during review was:

outputs/lerobot/tmp_banana_prompt_smoke

It was built from one rslxcvg/banana_blue episode with:

mode:        position_target_colors
prompt mode: eval_mixed
episodes:    12
frames:      5568
unique tasks: 10

The visual audit found no prompt/image mismatches in that smoke dataset.

Validation

Run prompt checks:

.venv/bin/python data_processing/check_banana_prompt_templates.py

Run color-swap regression checks, including the real banana data case:

.venv/bin/python data_processing/check_banana_color_swap.py --real-data

After building a full dataset, inspect metadata:

.venv/bin/python - <<'PY'
import json
from pathlib import Path
import pandas as pd

root = Path("outputs/lerobot/banana_act_position_targetcolor_v2")
info = json.loads((root / "meta/info.json").read_text())
tasks = pd.read_parquet(root / "meta/tasks.parquet")
episodes = pd.read_parquet(root / "meta/episodes/chunk-000/file-000.parquet")

print(info["total_episodes"], info["total_frames"], info["total_tasks"])
print(tasks.reset_index().head(20).to_string(index=False))
print(episodes[["episode_index", "tasks", "length"]].head(20).to_string(index=False))
PY

Then visually inspect representative encoded frames from videos/observation.images.front. Every direct-color, relative-color, and exclusion prompt must resolve to the same rendered target bowl.

Also inspect the first seconds of representative videos. Episodes should start near the reach, not with a long static table/init pose.

Training Notes

MolmoAct2 and other language-conditioned VLAs can use the task strings.

For MolmoAct2 runs that expect the older SO-101 calibration convention, use the converted dataset copy instead of patching transforms inside the trainer:

.venv/bin/python data_processing/convert_banana_molmo_compat.py \
    --source-root outputs/lerobot/banana_act_position_targetcolor_v2 \
    --output-root outputs/lerobot/banana_act_position_targetcolor_v2_molmo_compat \
    --output-repo-id rslxcvg/banana_act_position_targetcolor_v2_molmo_compat \
    --overwrite

Add --push-to-hub after local validation. The converter uploads with .cache/** ignored and syncs the LeRobot v3.0 tag to main, which matters because LeRobot defaults to loading datasets from the codebase-version tag.

The converter leaves videos, task text, frame indices, episode indices, and joint order unchanged. It applies:

observation.state[shoulder_lift.pos] = 90 - observation.state[shoulder_lift.pos]
observation.state[elbow_flex.pos] = observation.state[elbow_flex.pos] + 90
action[shoulder_lift.pos] = 90 - action[shoulder_lift.pos]
action[elbow_flex.pos] = action[elbow_flex.pos] + 90

It then recomputes meta/stats.json and the per-episode numeric stats in meta/episodes.

Vanilla ACT training in this repo does not condition on text. For ACT, prompt copies are duplicated visual/action samples rather than language augmentation. Use ACT runs mainly as trajectory and visual sanity checks unless the policy is modified to consume text.

Known Risks

The prompt generator uses the intended rendered layout. If color swapping fails in the image, the prompt can still be internally correct while mismatching the pixels. Always run both prompt checks and visual checks before publishing or training.

The current README examples should pass --mode labelsafe_colorpos explicitly when building label-safe distractor-only datasets. Otherwise the builder defaults to position_target_colors.