Custom Tasks and Feedback

Custom Tasks and Feedback

Two extension points make ACE general-purpose: bring your own Task and your own feedback. No ground-truth labels are required.

from ace import ACE, Feedback, Sample, Task, OpenAILLM

my_task = Task(name="my-domain", samples=[Sample(id="1", question="...")],
               evaluate=lambda pred, s: my_score(pred, s))

def my_feedback(sample, generation) -> Feedback:
    # plug in execution signals, a reward fn, or an LLM judge — your call
    ok = run_my_checks(generation.answer)
    return Feedback(correct=ok, signal="tests passed" if ok else "tests FAILED")

ace = ACE(OpenAILLM(model="gpt-4o-mini"))
ace.adapt_online(my_task, feedback_fn=my_feedback)   # learns from YOUR signals

See examples/05_custom_task.py (runs offline) and cookbook recipes 03–04.

---

Feedback regimes

ACE adapts with or without ground-truth labels. The label-free path is what makes it work for live agents (the paper's headline AppWorld result).

Regime	How to use it	Example
Labeled	provide sample.answer; use_labels=True (default)	offline system-prompt optimization on a train split
Custom hook	pass feedback_fn(sample, generation) → Feedback	code tests pass/fail, API status, a reward model
Pure label-free	signal=... via the hook, no ground_truth	live agent learning from environment responses

flowchart TD
    ST["ACE.step(sample, feedback)"] --> D{"feedback_fn provided?"}
    D -- yes --> CF["Caller computes Feedback (execution signal · reward · LLM-judge)"]
    D -- no --> L{"labels available? (use_labels and sample.answer)"}
    L -- yes --> GT["grade vs task.evaluate(): Feedback(correct, ground_truth)"]
    L -- no --> NS["Feedback(correct=None): nothing reliable to learn from"]
    CF --> R["Reflector"]
    GT --> R
    NS --> R

Loading

ACE is robust under rich feedback but can degrade without reliable signals — the quality of your feedback_fn matters.

---

Other extension points

• New backend — implement the two-method LLM protocol (complete, complete_json) and pass it to ACE(...).
• New agent framework — mirror ace/integrations/openai_agents.py: inject playbook.render() into the system prompt and call ace.step(...) with the captured trajectory.
• Curation mode — by default the Curator calls the LLM to propose ADD/UPDATE/REMOVE edits, with a deterministic ADD-only fallback that never drops a distilled lesson. Set ACEConfig(curator_use_llm=False) to force the deterministic path.
• Semantic dedupe — pass embedder=make_openai_embedder() (or any batched embedding callable) to ACE(...) for embedding-based de-duplication.

See Configuration for the full set of knobs.