FAQ

FAQ & Design Decisions

Why itemized bullets instead of a single prompt? Localization. The Generator references bullet ids, so the Curator edits only the relevant ones. Full-prompt rewriting is exactly what triggers context collapse.

Why is the merge deterministic (non-LLM)? Safety and cost. A mechanical merge can never accidentally summarize the whole context into oblivion, multiple deltas can merge in parallel, and adaptation cost scales with the delta size, not the full context.

Why a separate Reflector? Separating evaluation/insight-extraction from curation improves context quality (paper §4.6 ablation). One model judging and rewriting in a single pass is where detail gets lost.

Does a longer playbook mean higher serving cost? Not linearly. The paper reports 91.8% of input tokens served from KV cache during evaluation, cutting billed input cost ~82.6% vs. counting raw tokens.

When is ACE not worth it? Tasks solvable by a short, fixed instruction (e.g. HotPotQA-style retrieve-and-synthesize, or fixed-strategy games). ACE shines when success needs detailed domain knowledge, complex tool use, or environment-specific strategies.

How do I cap playbook growth? Grow-and-refine de-dupes per concept; tune dedup_threshold, harmful_margin, and choose proactive (refine_every) vs. lazy (lazy_refine_token_budget). See Configuration.

Does ACE need ground-truth labels? No. It adapts with labels or from label-free execution signals via a feedback_fn. The label-free path is what makes it work for live agents. See Custom Tasks and Feedback.

Can I use a non-OpenAI model? Yes — implement the two-method LLM protocol (complete, complete_json) and pass it to ACE(...). The SimulatedLLM shows the minimal shape.

Offline or online — which do I use? Offline = multi-epoch optimization over a train split (system-prompt optimization). Online = sequential test-time adaptation. They compose: warm-start online from an offline playbook (the paper's strongest AppWorld config).