July 6, 2026, ~24 hours in. A self-play RL agent for OpenFront.io, built on the real game engine. See also: DESIGN.md (the living spec: intent table, obs schema, head layout), README (how to run everything).
One PPO run is live: ppo_v3, trained from scratch on curriculum v2, now at stage 4
(30 bots, three maps) with a 60-70% rolling win rate. Its rival ppo_v2c, warm-started
from v2b weights, was retired Jul 6 evening after the fresh run overtook it by a full curriculum
stage. Both produced genuine engine wins on stage 0 (1v1 vs one nation on Onion),
the first time the win condition has actually fired, partly because the curriculum now makes wins
reachable and partly because win detection was silently broken until today (bugs).
A parallel experiment is training a behavior-cloning warm start from archived human games.
The full observe→act→reward loop, a 7-map win-gated curriculum, restart-proof cloud training, and a
complete visualization suite (real-client replay videos with a live model-debug overlay, plus live play
against the agent) are in place.
On the observation side, the encoder bake-off (AE v3.1) is concluded:
benchmarking the bot-trained AE on real human games exposed a 16-point border-accuracy gap, and
parallel ablations found the fix: latent resolution, not channel count. Final standings on
uniform-crop eval: d8 (64ch @ 1/8) at 89.3% human / 96.1% bot borders, and the compact d8c32
(32ch @ 1/8) at 88.2% / 95.5%, one point behind at half the policy input. d8c32 cleared the
pre-registered ≥88% bar, so it is the encoder for PPO v4. The channel-count control (c96) was
pruned unfinished: its host kept dying and the resolution result had already answered the question.
First deployment-style eval of ppo_v3 (fixed-seed local games, sampled actions):
2 wins / 2 losses on the first four stage-4 games, consistent with its training roll-win.
Jul 7 ~00:45: v4.2 deployed mid-run. Watching v4 replays exposed the phantom-boat exploit
(43-78% of decisions were boats the engine silently discarded - see bugs).
v4.2 fixes it three ways (honest masks, valid-tile snapping, wasted-intent penalty) and resumed
ppo_v4 from its own checkpoint - no retrain needed, since no tensor shapes changed.
The new episode/wasted curve opened at ~85-93 discarded intents per episode
(stage 4), the measured size of the exploit; the penalty now prices those at noop-minus-w_waste.
Watch items: wasted falling, boat's action-mix share normalizing, and a transient roll-win dip
while the value head absorbs the reward shift (resumed at update 260, roll-win 0.28).
Numbers: 375k bot + 420k human full-state snapshots · AE border accuracy 71.8% → 89.3% (human) · 5.9M-param policy · 11 curriculum stages · first 2 engine wins Jul 6.
What is running, what is done, and what is stale. All RL/BC still loads frozen
ae_v3.pt until the v3.1 winner is wired into rl/obs.py.
| run | status | notes |
|---|---|---|
ppo_v4 | active (v4.2 code) | The full v4 stack from scratch: d8c32 encoder at 1/8, learned spawns, local owner crop,
eval loop. Stages 0→3 in ~2h, stage 4 by ~3.6h despite two restarts (OOM at minibatch 128,
then the throughput redeploys). v4.1 async collector since Jul 6 ~11pm; v4.2 (honest
boat/expand/nuke masks, valid-tile snapping, wasted-intent penalty - the phantom-boat fix)
resumed from the same checkpoint Jul 7 ~00:45, no retrain: no tensor shapes changed, only
mask semantics and a small reward term. Expect a transient dip while the value head absorbs
the reward shift; watch episode/wasted fall and the boat share of the action mix
normalize. Checkpoint on HF as ppo_v4/policy.pt. |
bc_v4 | active | Feedforward BC on the full 291-game cache-bc dataset (265 train / 26 holdout) with spawn
supervision. ~56 ex/s GPU-bound at batch 24 x accum 4. Uploads bc.pt +
bc_best.pt to HF. |
bc_seq_v4 | active | The temporal transformer experiment (--seq 8), judged properly this time: same cache-bc data as bc_v4, A100-80GB pod (bc3), batch 8 x accum 4 effective 32. Pod bootstraps from the prebuilt cache-bc tars on HF (no raw download, no replay, no prefeaturize). |
ppo_v3 | retired Jul 6 night | Curriculum v2 from scratch. Stage 4 (30 bots, Pangaea/Caucasus/BlackSea) by Jul 6 evening, rolling win 0.6-0.7, roll-score 0.93. Local eval: 2W/2L over four stage-4 games, wins by outright conquest at ticks ~10-15k. Pod terminated after final checkpoint (update 632) uploaded to HF; its TB event files died with the pod. |
ppo_v2c | retired Jul 6 | Same curriculum v2, warm-started from v2b weights. Stalled at stage 3 (win rate
0.07-0.17, entropy stuck near 8) while from-scratch v3 sailed past to stage 4. Final
checkpoint (update 502, 771k steps) on HF as ppo_v2c/policy.pt. See
lessons: warm-starting from a stale reward/curriculum hurt. |
bc_v0_pilot | paused Jul 6 eve | Outcome-conditioned BC on partial human sidecars (~50 games). Stopped at step 3,850
(~185k samples) to fix the data pipeline: at ~34 ex/s the run was CPU-bound and the loss
curve was flat-noisy around 8-11 with acted-step accuracy ~0.6, too little throughput to
call it. Checkpoint on HF as bc_v0_pilot/bc.pt; resumes after the featurize
cache lands. |
bc_seq_v1 | paused Jul 6 eve | Temporal BC (--seq 8). Lost hours OOM crash-looping at batch 24 (a seq-8 window multiplies activation memory 8x; one 19.9 GiB allocation on a 24 GiB card), then ~450 steps at batch 8 before the pipeline pause: too early to read. Only 11 games had sidecars on that pod. No checkpoint yet (died before first save interval). |
ae_v31_d8c32 | done, chosen for v4 | Compact v3.1: 32ch @ 1/8 latent. Uniform eval 88.2% human / 95.5% bot borders,
one point behind d8 at half the latent footprint. On HF as ae_v31_d8c32.pt. |
ae_v31_d8 | done | v3.1 @ 1/8 latent, 64ch. Best absolute border accuracy: 89.3% human / 96.1% bot. On HF
as ae_v31_d8.pt. Kept as the quality ceiling; d8c32 preferred for policy input
size. |
ae_v31 | done | Full v3.1 decoder/loss stack @ 1/16. Finished 40k steps; human borders 78.5% (worse than mixed-data v3 retrain). Superseded by d8. |
ae_v31_c96 | pruned | Channel-count control (96ch @ 1/16). Killed twice by an oversubscribed host (load 435 on 384 cores) and abandoned: d8 vs v3.1-at-1/16 already isolated resolution as the variable that matters, so the control's answer would not change any decision. |
ae_v3_mix | partial | Homelab bot+human retrain, OOM at step 22.7k. Proved data helps (80.1% human borders) but run didn't finish. |
ppo_v2b, v1 curriculum | stale | Reached stage 4 on 90-player Pangaea with top-third placement but 0 wins. Curriculum, reward, and win counting all changed since; checkpoint only useful as warm start for v2c, not as a progress metric. |
ppo_smoke, ppo_pod1, etc. | archived | Early plumbing runs. No longer representative. |
Is PPO actually improving? Yes, but the signal was hard to read. Placement-gated curriculum v1 looked like progress (roll-score ~0.65 on Pangaea) while the agent never won. Curriculum v2 reset both runs to 1v1, win detection was broken until 12:06, and both policies now win stage 0 in replay. By evening v3 had cleanly won the head-to-head: stage 4 with 60-70% wins vs v2c stuck at stage 3 with 7-17%, which is why v2c was retired. Neither lineage has been trained with the new encoder or BC warm start yet.
boat on 43-78% of decisions (p(boat) up to 0.79, p(noop) 0.01) with nearly all of
them silently discarded by the engine - the cap is 3 transports in flight. Root cause: a
whole class of actions was "legal" per the mask but a guaranteed no-op at execution, and a
discarded intent was reward-identical to noop with occasional upside - a free lottery ticket
the policy learned to farm (ppo_v3's version was 61% build on random land). Fixed
everywhere at once: masks now match engine state (canBoat = boat slot free + own
shore, canExpand = neutral land actually borders, hasSilo now
respects silo cooldown + spawn immunity), the translator snaps tile picks to tiles that can
work (boats never target own/ally land, builds only own territory, ports own shore), and
whatever still slips through is counted by the bridge (wasted, exact engine
calls per intent) and penalized at W_WASTE=0.01 per discarded intent, so noop strictly
dominates doomed actions. New episode/wasted TB curve. Tagged v4.2 and
hot-deployed to the running ppo_v4 pod at ~00:45: trainer killed, repo synced,
resumed from the same checkpoint (update 260, stage 4, win-window intact) - resume-safe
because nothing about the network changed. First v4.2 episodes measured the exploit at
~85-93 wasted intents per episode.encode_grids), which also fixed a stale-fallout bug (dynamic fallout was riding
inside the per-episode cached terrain tensor). Raw 64x64 local owner-crop bypass. Learned
spawn placement end to end: spawn action + legal-region mask in PPO, spawn-phase snapshots +
labels in replay sidecars (formatVersion 2) for BC. BC prefeaturize cache
(scripts/prefeaturize_bc.py): 1.45 ms/sample single-threaded vs ~15-20 ms before.
PPO v4 fixes: entropy anneal, stage LR warmdown, persisted win-gate window, periodic
fixed-seed greedy eval. Pod supervisors got commit assertions and crash-loop backoff.The core architectural finding so far. Three iterations:
The lesson: a one-bit fact reconstructed at 95% is strictly worse than reading the bit. Autoencoders are for high-dimensional state; never make exact small state fight the map for latent capacity.
AE details that held up: border-weighted cross-entropy (borders blur first and matter most), owner relabeling to static per-game spawn slots (any player count, fixed channels), fully-convolutional training on random crops (one model, any map size), and rarity-weighted BCE detection for structures: count regression collapses to all-zeros on 99.9%-empty grids.
The trigger: overall tile accuracy saturates near 99% for every model (water and player interiors are easy), so the honest metric is border-tile accuracy, and it wasn't in the training logs at all. Benchmarking the bot-trained v3 on the newly replayed human games exposed a 16-point domain gap: 87.5% border accuracy on bot data, 71.8% on human. Human territories are gnarly (naval invasions, enclaves, 50+ player fronts); nation bots grow blobs.
Two fixes ran in sequence. First, retraining v3 unchanged on a bot+human mix recovered the domain gap (80.1% human), so data was part of the problem. Then v3.1 attacked the architecture with seven changes at once plus two ablations to find the real constraint:
Result: resolution is the constraint, not capacity. Uniform-crop eval, 256 samples:
| model | latent | border (human) | border (bot) |
|---|---|---|---|
| v3 bot-only | 64ch @ 1/16 | 71.8% | 87.5% |
| v3 on bot+human mix | 64ch @ 1/16 | 80.1% | 86.8% |
| v3.1, all fixes | 64ch @ 1/16 | 78.5% | 90.7% |
| v3.1 @ 1/8 res | 64ch @ 1/8 | 89.3% | 96.1% |
At the same 1/16 resolution, the v3.1 decoder/loss work helped bot borders but couldn't beat plain mixed-data retraining on human borders. Borders are high-frequency spatial detail, and a 64-number vector summarizing a 16x16 patch simply cannot store where a ragged front cuts through. Halving the patch to 8x8 bought ~9 points on both domains. Structure detection stayed at precision/recall 1.0 throughout. The catch: the 1/8 latent is 4x the policy input; a 1/8 x 32ch run (2x baseline) is in flight to see if the win survives halved channels, alongside the 96ch control.
Complementary hedge (decided): the policy will also receive a raw local owner-map crop around its own territory, exact borders where the agent acts most, latent for global context. The latent doesn't have to be pixel-perfect everywhere.
Full intent surface from day one, legality masking only (never curricular). Factorized masked heads over a shared conv trunk. One flat softmax is impossible (tile arguments alone are millions of options):
| head | chooses |
|---|---|
| action type | noop, attack, expand, boat, build, launch_nuke, alliance req/reject/break, donate gold/troops, embargo, retreat |
| player pointer | target player slot |
| tile-region pointer | a 16x16 map region; the bridge snaps to the best legal tile inside it |
| build / nuke type | structure or warhead class |
| quantity | troop fraction bucket |
Masks come from exact engine legality calls each step; the policy can only pick what the engine would accept. Where a tile argument can still fail at execution (boat destination unreachable by water, build site too close to another structure), the bridge counts the discarded intent and the env charges w_waste for it - the mask keeps the policy honest, the penalty keeps the engine honest. Full intent table in DESIGN.md.
Iterated from raw land occupancy to a composite strength index: 0.45 x land + 0.25 x military + 0.30 x economy (shares). Land alone mis-scores legit strategies like tiny-island gold-stacking.
w_str x strength x timeweight, timeweight 0.5→1.0 over the first 8000 ticks.
Strong late is worth double strong early, and it doesn't net to zero on death like pure deltasw_place x place^-0.7, power law: 1st ≈ w_place, 2nd ≈ 0.62x, 10th ≈ 0.2x;
the dead rank behind everyone still alivew_win = 30 (doubled when the curriculum became win-gated)w_waste = 0.01 per intent the engine silently discarded
(doomed boat, invalid build site, expand with no neutral border): without a price these are
noop-with-upside and the policy farms them (see bugs)Eleven stages over seven maps (Onion → Pangaea → Caucasus → BlackSea → BetweenTwoSeas → World → Asia). Three principles, all added after watching v1 fail to progress:
Advancement is win-gated: rolling win rate over the last 40 on-stage episodes must exceed 0.5. The agent must be winning more often than not, not merely surviving.
josh-freeman/openfront-rl
(HF mischievers/openfront-rl-agent,
"v19b") is the other PPO agent on this engine we're aware of, and a lot of our scaffolding follows
from what they figured out first. The main fork is observation and tile choice: they keep a compact
scalar view (~96 floats) and route spatial decisions through hand-coded heuristics (closest border
tile, pickBuildTile()); we put map structure in the observation stack and learn tile
selection with a spatial pointer (the bridge still snaps to legal tiles). Different tradeoffs: compact
obs plus heuristics is simpler to train and deploy; richer spatial input is heavier but can in
principle learn geography, chokepoints, and nuke aim.
| AlphaFront | openfront-ai | |
|---|---|---|
| observation | ~96 floats (16 player scalars + 16 neighbors × 5); no map | frozen spatial AE latent + ego planes + entity tokens + raw bypass |
| actions | 17 types, 16 neighbor targets, 5 fractions; no diplomacy | full intent surface incl. diplomacy; factorized heads, engine masks |
| tile choice | heuristics (pickBuildTile(), closest border, etc.) |
learned spatial pointer; bridge snaps within the chosen region |
| network | MLP 512-512-256, ~0.5M params | CNN + token transformer + pointers, 5.9M |
| curriculum | 10 win-rate-gated stages, LR warmdown | 11 win-gated stages, 7 real maps, rehearsal (gating idea borrowed from them) |
| human data | not in scope | archive replay pipeline + behavior cloning |
| live play | Puppeteer bot driving a browser | native websocket client, no browser |
| results | 100% vs Easy/2 opp (20 games); survival plateaus ~35–47% mid-curriculum | first engine wins on stage 0; climb in progress |
What they showed works here, and we adopted: PPO converges on this engine at ~10 ticks/decision (we chose the same cadence independently); win-rate-gated advancement is viable. Still on our borrow list from their setup: LR warmdown on stage transitions; tiny flat generated maps as an even softer stage 0; a mid-game "beat all bots" gate so advancement doesn't wait on long endgames. On our side we've added a human-replay pipeline for behavior cloning: extra imitation signal that's awkward to fold into a purely scalar observation.
GPU util was poor and env stepping was the bottleneck:
--compile flag.8a0da05). The
single main thread was serializing fp32 copies, minibatch collation, and GPU compute; observations
now ride as fp16, transfers go through pinned buffers, and a worker thread collates the next
minibatch during the current backward. 3.6x on stage-3 maps (~590 → ~2100 game-ticks/s).b8c331c). A collector
thread builds rollout k+1 (env stepping + encode/act on a snapshot policy) while the main thread
runs the PPO update on rollout k. One rollout of staleness; old_logp stays anchored
to the acting snapshot so the clipped ratio math is unchanged. The former ~35s rollout phase now
hides inside the update phase (perf/rollout_wait_s tracks what's left).Generated by scripts/make_progress_graphs.py from the surviving TB event files and
BC jsonl logs. ppo_v3's curves died with its pod (lesson: archive events before terminating);
it appears as annotated reference lines from the devlog record.
Every performance fix this project has shipped was some flavor of "get work off the single Python thread" (VecEnv processes, GPU featurization, pinned prefetch, the v4.1 async collector). Two structural moves land the endgame:
9f262a6). The two stable, allocation-heavy
loops are now Rust with the GIL fully released: cache-bc frame decode (zstd + split) and the
pad+stack collate that runs per PPO minibatch (2.6x single-threaded, and it no longer blocks
other threads at all - which is what the prefetch thread actually needed).
rl/native.py keeps numpy fallbacks so nothing requires the toolchain; pod
scripts build the wheel opportunistically at boot. Formats are frozen (CACHE_FORMAT=1,
obs v4), which is what makes these safe to freeze in a compiled language.Round two (8aa7fbd): the whole BC sampler is Rust now. The DataLoader
process workers turned out to be a regression in production - 16 workers sat idle while the
main process choked unpickling ~50MB raw batches off the IPC queues (stall 300s per 50-step
window on bc2). The fix was to delete the abstraction, not tune it:
sample_batch(n) does everything rl/bc_data.py did
per sample (zstd frame/entity/step decode, serde JSON, the full obs-v4 featurization,
label→head targets, legality forcing) in Rust, rayon across step draws, GIL held only
to hand back numpy dicts. Laptop: 639 → 2631 samples/s vs the Python path; parity
proven exactly (every array, 11.8k samples) by scripts/test_ofrs_parity.py.
sample_windows(batch, k) draws a whole seq micro-batch in parallel for the
temporal run. The trainer feeds from a plain thread pool again - no processes, no pickle.ofrs::unpack_arrays. Numpy fallback
speaks the same protocol.np.stack calls now go
through ofrs::stack_f32/f16 (parallel copy, GIL released), so the PPO prefetch
thread's collate no longer serializes against the collector.expandable_segments after an OOM at update 267
(fragmentation after 1135s uptime, not the new code path).Determinism is the superpower: the bridge mirrors the client's createGameRunner() init
exactly (same PseudoRandom ID stream, no spawn timer), so agent games saved as GameRecords replay
bit-identically in the real client, and verify_record.ts proves it by re-simulating
from intents alone.
rl/watch.py: one greedy episode → flat debug WebM + GameRecord + a
.debug.json sidecar (every decision: action, probs, value).scripts/render_client_replay.py: headless Chromium replays the record in the
actual game client → WebM with real graphics. Agent renders as AGENT with the gold spawn
ring, crown when first, "You Won!" modal; camera starts fit to the whole map.scripts/serve_replay.py: archive-API shim so a normal browser replays agent
records interactively.RlDebugOverlay.ts): in-client panel synced to the sim tick: chosen
action, value, probability bars, recent-actions log. Works in rendered videos, browser replays,
and live matches; auto-probes the agent's feed on localhost:8988, zero setup.rl/play.py + bridge/play.ts: the agent joins a real lobby over the
live websocket protocol; you fight it while watching it think.patches/client-replay-tooling.patch: all client hooks as an uncommitted-submodule
patch, auto-reapplied if the submodule resets.vec.py checked winner[1] == "Agent"
(username); the engine emits the clientID ("AGENTRL1"). Every "the agent never wins"
observation before Jul 6 12:06 was partly measurement error. Found only by watching a replay where
the agent visibly won while the stats said otherwise. Validate the reward path end-to-end with a
game you can see.BlackSea → blackSea); dirs are fully lowercase. Fine on Mac, ENOENT on Linux.--stage 4;
resume took max(cli, checkpoint), so a manually reset checkpoint kept snapping back, and a
still-running trainer kept re-saving its old stage over the reset. Checkpoint stage is now
authoritative; kill the whole process group before editing checkpoints.troops > 100; the engine additionally requires a free boat slot (max 3), an own
shore to launch from, and a non-own/non-ally destination with a reachable shore - and
silently discards the intent otherwise, with troops never deducted. A discarded boat
was therefore reward-identical to noop but occasionally landed and grabbed land, so the
policy-gradient math correctly learned to spam it: 43-78% of decisions in v4 stage 2-3
replays, p(noop) driven to 0.01. Same shape in expand (fizzles without a neutral border),
build (61% of one v3 episode's decisions, snapped to random land the engine rejected), and
nukes (silo cooldown/spawn immunity ignored by the mask). The bridge comment said "illegal
tile picks become no-ops the policy learns to avoid" - but there was no signal to learn from:
the failure and the noop pay identically. Fix: honest masks + valid-tile snapping + a
per-discarded-intent penalty (Jul 7 timeline entry). Caught by computing the action mix from
replay debug sidecars, not from any training curve.::1 while headless Chromium
resolves localhost to IPv4 (force --host 127.0.0.1); the client rejects software
WebGL (needs --use-angle=metal --enable-gpu --ignore-gpu-blocklist).ppo_v2c resumed v2b weights (trained under
the old placement reward and pre-fix win counting) onto the new curriculum; ppo_v3
started from scratch on the same code. In one day the fresh run reached stage 4 at 60-70% wins
while v2c stalled at stage 3 at 7-17% with entropy stuck near 8, exploring without committing.
When the reward or curriculum changes materially, retrain; a checkpoint tuned for the old
objective is a local optimum in the new one.git pull had silently failed on an untracked file. A restart loop needs two cheap
guards: back off (or change something, or alert) when the child dies instantly twice in a row,
and assert the deployed commit matches origin/master at launch.Homelab for datagen and some training; cloud GPUs for long PPO and AE runs.
scripts/pod_train.sh supervises RL jobs (idempotent deploy, auto-restart). Datasets are
prefeaturized once; checkpoints live on Hugging Face:
openfront-snapshots,
openfront-human-games
(285 hash-verified replayed human games + raw intent records),
openfront-tile-autoencoder.
Ops scar tissue from the AE push, mostly self-inflicted: the homelab mixed run was host-OOM-killed at
step 22.7k (CPU evals + HF upload + 6 DataLoader workers on one box); remote trainers hit
ulimit -n with 16 workers memory-mapping 459 game caches; a billing outage restarted
cloud machines mid-run (tmux gone; the interrupted ablations were relaunched from scratch since AdamW
state isn't checkpointed, only weights).
Git history was scrubbed once with git filter-repo after large blobs snuck in; data dirs
are gitignored now.
Three arms, two launched together (Jul 6 eve), the third conditional:
ppo_v4): from scratch, stage 0. Baseline and control;
also the only arm learning spawn placement without supervision.bc_v4): feedforward pilot, 60k steps on the cache-bc
pipeline. At ~1.5ms/sample CPU the run is GPU-bound; verdict lands in hours, not days.
Saves bc_best.pt on holdout improvement (acted-action + tile accuracy - the
honest heads; plain action accuracy is inflated by forced-legality noops).ppo_v4_bcinit, conditional): launched when B plateaus.
rl/ppo.py --init loads the shared Policy weights and folds the winner-bucket
conditioning into the head biases - verified exact (max output diff ~1e-7 across all heads).
Same hypers/envs/pod class as A; judged at equal PPO wall-clock on the
eval/win curves and time-to-stage-N.Gates. B is learning iff holdout tile_region accuracy climbs and holdout
loss falls (10k steps ≈ 1M samples ≈ 5x the entire old pilot's data; flat there = dead, kill the
pod, A continues regardless). B has plateaued (launch C) when holdout metrics are flat across two
consecutive eval windows (~10k steps) or the cosine tail begins. C's first ~30min will look bad
while the untrained value head catches up - judge after the critic settles. If C's entropy climbs
back toward ~7 in the first hour, the BC prior got washed out: add a KL-to-prior regularizer
(AlphaStar-style) before rerunning.
Watch items. BC first log line should read several hundred ex/s (under ~100 = cache
problem, stop). Late-curriculum OOM lever: MINIBATCH=64 env var (1/8 grids cost ~4x
v3's conv activations at World/Asia). Rollout buffer RAM ~20GB at Asia-size grids with 96x32;
halve --rollout if the pod is RAM-tight. Fixed-seed eval pauses training for minutes
at late stages (~2% overhead at eval-every 300). Old v1 sidecars carry no spawn
labels; re-replay with BC=1 REBC=1 datagen/replay_all.sh feeds the next BC run.
Deliberately skipped: BC value-head pretraining on placements (scale-mismatched with PPO returns, muddies A-vs-C), fp16 rollout buffers (RAM insurance we may not need), latent precompute to disk (at 1/8 a latent is ~2.4MB/snapshot ≈ 200GB corpus - the GPU-batched encode from the raw cache IS that idea, done right).
--seq BCPolicy
exists but has only ever run at starvation throughput. With cache-bc feeding it, run
seq=8 vs feedforward at equal samples on full holdout; if seq wins, that's the
evidence that buys the PPO LSTM.eval/* curves), win-gate window persisted in the checkpoint, entropy anneal
(0.01 → 0.002 over 4k updates) and stage LR warmdown (x0.85 per advance). v4 trains from
scratch - the 1/8 grid, new action ("spawn"), and 43ch observation change the MDP and the
tensor shapes anyway.--bc --rebc (formatVersion 2) carry spawn labels; old v1 sidecars still train
everything else. Re-replay the archive on a pod to unlock BC spawn supervision everywhere.