Feat/othello#1190
Open
wanderingweights wants to merge 7 commits into
Open
Conversation
Pure, shared building blocks for the env-tournament memory system: a fixed-slot memory store and OpenAI-compatible tool schemas/dispatch that both the validator eval harness and the training rollout will use. - core/models/pvp_models.py: MemoryArea/MemoryOp enums, MemoryConfig and the MemorySlotEdit/GameActionArgs argument models. - core/pvp/memory.py: SlotMemory — fixed slots, per-slot token budget, rewrite (tail-truncate) / append (FIFO front-evict), total no-op-on-bad- input semantics, injectable TokenCounter. - core/pvp/tools.py: tool schemas and dispatch generated from the MemoryArea x MemoryOp product; slot fields range-constrained for grammar. - tests/test_pvp_memory.py: 36 tests covering truncation, eviction, bad-op no-ops, generative tool set, dispatch and arg coercion. Placed in core/ (not validator/evaluation/) so the training rollout can share the identical implementation without depending on eval code.
Each turn is now an agentic loop: the model is given the game state plus its memory slots and a tool set, may call memory tools (whose results are fed back), and ends the turn by committing a legal move via game_action. The conversation is rebuilt fresh every turn — the only state carried across turns is SlotMemory, never a growing transcript, so context stays bounded. Models (core/models/pvp_models.py): - ToolCall (arguments: dict[str, JsonScalar]), ToolSchema/FunctionSchema envelope, ChatMessage extended with tool_calls/tool_call_id + to_openai() wire conversion, ChatResult.tool_calls, ChatRole.TOOL, ChatFn(config, messages, tools), MemoryArea.persists_across_games. Logic: - core/pvp/tools.py returns ToolSchema models; core/pvp/memory.py gains reset(). - chat.py passes tools + tool_choice=auto and parses tool_calls back into ToolCall, serialising messages/tools via their models. - bot.py rewritten: reconstruct-per-turn context, memory tools + game_action, illegal move -> error result -> retry, bad memory op -> no-op, bounded by PVP_MAX_INNER_STEPS + per-turn SIGALRM. working resets / long_term persists. - constants: PVP_TURN_TIMEOUT_SECONDS 5 -> 90 (sized for the loop) plus loop and slot config. Tests: new test_pvp_bot_loop.py for the loop; test_pvp_bot.py reduced to strip-think + timeout/forfeit integration; memory and coverage-gap tests updated to the tool-calling protocol. Full PvP suite green. Cross-game persistence and the reflection turn are wired in Phase 3; bots currently use fresh per-game memories.
game_runner now carries one long-term memory per player across all games of a matchup, so each side builds an opponent model over the series; working memory stays fresh per game. After each game both surviving bots run a single-shot reflection (same memory tools, no game_action) to consolidate long-term memory; a forfeiting bot is skipped. - bot.py: add LLMBot.reflect() — best-effort, single generation, swallows errors since the game is already decided. Factor the memory render into a shared block used by both the turn and reflection prompts. - game_runner.py: per-matchup long_term_a/long_term_b threaded into each game's bot via fresh working + persistent long-term; reflect both players post-game. - tests: TestReflection (bot-level) and test_pvp_persistence.py (a marker written in game 1's reflection appears in a later game's prompt). Still using the whitespace token counter; the tokenizer-backed counter is a follow-up. Persisting per-hotkey long-term memory to the DB is Phase 4.
…atch runners The shared system_prompt_template still told the model to "respond with ONLY the action id (a single number)", which contradicts the tool-calling loop (the bot separately instructs it to call game_action). Remove that Output Format block — the bot's tool guidance now stands alone — and add a test asserting the system prompt no longer demands a bare action id. Dev/eval tooling for driving the harness without a GPU: - scripts/pvp_play.py: manual stepper — render the exact per-turn prompt, apply tool calls by hand, advance the game (state persisted to /tmp). - scripts/pvp_anthropic_match.py: a ChatFn adapter bridging the harness to the Anthropic Messages API with tool use, so two Claude models can play a matchup. Validated end-to-end: cross-game opponent memory drove a real exploit.
…eline
Model prep (a minimal image shipping only core/ + trainer/model_prep/) needs to
run the eval harness to compute a format-aligned baseline, so the shared harness
moves out of validator/ into core/pvp/ — the same shared-code split already used
for memory/tools.
Harness -> core/pvp/ (zero validator imports, verified):
- bot.py, agents.py, chat.py, scoring.py moved; game_eval.py holds the shared
_AGENT_REGISTRY + timeout/forfeit evaluate_bots wrapper; constants.py holds the
harness PVP_* constants. validator/core/constants.py re-exports them and
validator/evaluation/pvp/{bot,agents,chat,scoring}.py are back-compat shims, so
existing imports keep working.
- game_runner.py keeps the eval-only matchup orchestration, importing from core.
Tokenizer-backed memory budgets:
- core/pvp/tokenizer_counter.py: HFTokenCounter + load_token_counter (whitespace
fallback for non-repo ids / load failures). game_runner builds a per-player
counter so slot budgets are real model tokens, not whitespace words.
In-harness MCTS baseline:
- core/pvp/baseline.py: run_mcts_baseline plays the model (tool-calling LLMBot)
vs a pyspiel MCTSBot, same format as eval, no external MCTS-API server.
- trainer/model_prep/env_stats.py now computes baseline stats via run_mcts_baseline
instead of POSTing to the external /evaluate server.
- model-prep.dockerfile gains open-spiel + openai (harness runtime deps).
Tests: test_pvp_tokenizer.py, test_pvp_baseline.py; existing tests updated for the
new module paths. Full PvP suite (150) green.
- game_runner no longer re-exports _evaluate_with_timeout / _forfeit_returns; the two tests and pvp_play import them from core.pvp.game_eval directly. - Remove now-unused ENV_EVAL_TASK_TIMEOUT / CONSECUTIVE_FAILURE_LIMIT from env_stats (the external-server episode loop they belonged to is gone). - pvp_play imports LLMBot from core.pvp.bot. pyflakes clean across the changed surface; full PvP suite (150) green.
Wire othello (OpenSpiel) as a PvP environment across eval, scoring, and training paths: - EnvironmentName.OTHELLO + ENVIRONMENT_CONFIGS entry (task_id 400M-499M, PVP, MCTS baseline payload) - OthelloAgent + a setup_initial_state hook on BaseGameAgent. Othello is deterministic with no chance nodes, so the hook applies 2-6 seeded random opening plies: same seed reproduces the same start, different seeds diverge. Paired base/swapped instances share a seed, preserving seat fairness. - Register in _AGENT_REGISTRY; othello_rules in pvp_game_prompts.yml - Call setup_initial_state in game_runner and baseline after new_initial_state - Reference trainer rollout fn (dockerfiles/environment_functions/othello.py) - text_trainer: drive rollout_func wiring off eval_type == PVP instead of a hardcoded game tuple, so othello (and future PvP games) are covered - Add othello to FALLBACK_ENV_IMAGES for consistency Cleanup: drop the dead BaseGameAgent.generate_user_prompt (the runtime uses LLMBot._user_prompt) and its test; OthelloAgent relies on the base format_state rather than re-overriding observation_string.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.