This documents how the game_learning model on the Florian branch differs
from the model on main. The short version: main is a single-attacker,
graph-based, linear-probability game; Florian is an edge-free,
multi-attacker game with the memoryless exponential compromise model and
HMM belief filter from Datar & Dujardin ("Adaptive Learning for Moving Target
Defence", CoDIT 2025), plus a coordination bonus and a fictitious-play trainer.
| Aspect | main (old) |
Florian (new) |
|---|---|---|
| Topology | graph G = (V, E) |
edge-free: n independent nodes (graph optional, labels only) |
| Attackers | one attacker | K attackers (num_attackers) |
| Attacker action | subset A ∈ {0,1}^n, max_attack_nodes budget |
each attacker probes one node or idles; aggregated to per-node probe counts ρ |
| Compromise prob. | q_v = β_v (constant, linear) |
q_v = 1 − e^{−α_v·ρ_v} (exponential, memoryless) |
| Coordination | none | more simultaneous probes on a node ⇒ higher success |
| Observation | binary detected probes Y ⊆ A |
per-node detected count ~ Binomial(ρ_v, 1−ν) |
| Obs. likelihood | `(1−ν)^{ | Y |
| Belief filter | sum over attacker subsets A |
sum over joint probe-count vectors ρ |
| Defender | PPO over belief | PPO over belief + threshold-over-belief policy |
| Attacker learning | fixed policy only | fixed or REINFORCE softmax (fictitious play) |
| Config keys | beta, max_attack_nodes |
alpha, num_attackers, control_reward, attacker_idle_probability |
main framed the game on a graph G = (V, E). On Florian there are no edges:
each node is an independent attack/defence surface, and both the transition and
observation kernels factorise over nodes. GameConfig now takes num_nodes
directly (a networkx graph is still accepted but used only to label/lay out
nodes for plots, never for dynamics). experiment_config.build_graph returns an
edgeless graph for path/empty types.
main had a single attacker emitting a subset A ∈ {0,1}^n (with a
max_attack_nodes budget). Florian has K independent attackers; each probes
at most one node per step or stays idle. The environment aggregates their
choices into a per-node probe-count vector ρ, where ρ_v = how many
attackers hit node v this step. AttackerEnsemble bundles the attackers and
exposes both a sampler and the exact joint distribution over ρ.
This is the core behavioural change.
main (linear, constant per node):
q_v = 0 if v reimaged
q_v = 1 if undefended and already compromised
q_v = β_v if undefended, clean, and attacked
q_v = 0 otherwise
Florian (paper's memoryless exponential):
q_v = 0 if v reimaged (reset to clean)
q_v = 1 if undefended, compromised (stays compromised)
q_v = 1 − e^{−α_v · ρ_v} if undefended, clean, probed ρ_v times
q_v = 0 if ρ_v = 0
Because ρ_v sums the simultaneous probes, attackers coordinating on the same
node are strictly more likely to compromise it than the same attackers spread
across nodes — 1 − e^{−α·2} > 1 − e^{−α·1}. With one attacker and α chosen so
1 − e^{−α} = β, this reduces to main's single-probe behaviour, so the new
model is a strict generalisation. (The exponential form is memoryless, matching
the paper's 1 − e^{−α(ρ+1)} derivation.)
main observed a binary set Y ⊆ A of detected probes, with no false
positives and per-probe miss probability ν:
L(Y | A) = (1−ν)^{|Y|} · ν^{|A|−|Y|}, Y ⊆ A ; else 0
Florian observes a per-node detected count. With ρ_v probes on node v,
the number detected is Binomial(ρ_v, 1−ν), so:
L(Y | ρ) = ∏_v C(ρ_v, Y_v) · (1−ν)^{Y_v} · ν^{ρ_v − Y_v}, Y_v ≤ ρ_v ; else 0
The binomial coefficient matters: it weights how plausible each hidden probe
count is given the detections — essential now that ρ_v can exceed 1.
Both versions keep an exact Bayes filter over all 2^n hidden states. The
difference is what the inner sum ranges over:
main: sum over attacker subsets A ∈ {0,1}^n.
Florian: sum over joint probe-count vectors ρ, weighted by the attacker
ensemble's exact joint_action_distribution(state, defense):
b'(s') ∝ Σ_s b(s) Σ_ρ π_A(ρ | s, D) · L(Y | ρ) · K^{ρ,D}(s' | s)
Keeping attacker policies closed-form (uniform or softmax) is what keeps this filter exact.
New on Florian:
FocusedAttackerPolicy— softmax-over-nodes attacker with learnable logits (sharper logits ⇒ attackers pile onto one node ⇒ exploit the coordination bonus).ThresholdDefenderPolicy— reimage nodes whose belief marginal exceeds a threshold, up to budget. This is the threshold-in-belief structure the paper proves optimal, generalised to independent nodes.fictitious_play.py— rotating best-response training forD, A_1, …, A_K(defender threshold via line search; each attacker via REINFORCE), realising the paper's Algorithm 1 with several attackers, torch-free.evaluation.py— roll out any defenderact_fnand report return / average compromised nodes; baseline builders for no-defense, random, and threshold.
[env] keys renamed/added (and experiment_config.py updated to match):
- removed:
beta,max_attack_nodes - added:
alpha,num_attackers,control_reward,attacker_idle_probability
GameConfig likewise: beta → alpha, dropped max_attack_nodes, added
num_attackers, control_reward, attacker_idle_probability; graph is now
optional and num_nodes is the primary way to size the game.
main reported a single attacker reward. Florian reports per-attacker
rewards (info["attacker_rewards"]) plus a team total (info["attacker_reward"]).
Defender reward is control_reward · (n − compromised) − defender_cost · |D|;
each attacker earns control_reward · compromised − attacker_cost · (probed?).
- Rewritten:
belief.py,policies.py,env.py,__init__.py,visualization.py,tests/test_belief.py, the examples. - New:
fictitious_play.py,evaluation.py,examples/fictitious_play_demo.py,examples/evaluate.py. - Adapted from
main:experiment_config.py,configs/path_graph_7.toml,README.md(re-pointed to the new schema/model). - Not merged from
main(changed on both branches, left as theFlorianversions):env.py,examples/train_ppo.py,examples/visualize_ppo.py.