openfront-ai devlog

Self-play RL on OpenFront.io, real engine, ~two days in. DESIGN.md · README

Status snapshot (July 6th evening)

One PPO run is live: ppo_v3, trained from scratch on curriculum v2, now at stage 4 (30 bots, three maps) with a 60-70% rolling win rate. Its rival ppo_v2c, warm-started from v2b weights, was retired Jul 6 evening after the fresh run overtook it by a full curriculum stage. Both produced genuine engine wins on stage 0 (1v1 vs one nation on Onion), the first time the win condition has actually fired, partly because the curriculum now makes wins reachable and partly because win detection was silently broken until today (bugs). A parallel experiment is training a behavior-cloning warm start from archived human games. The full observe→act→reward loop, a 7-map win-gated curriculum, restart-proof cloud training, and a complete visualization suite (real-client replay videos with a live model-debug overlay, plus live play against the agent) are in place.

On the observation side, the encoder bake-off (AE v3.1) is concluded: benchmarking the bot-trained AE on real human games exposed a 16-point border-accuracy gap, and parallel ablations found the fix: latent resolution, not channel count. Final standings on uniform-crop eval: d8 (64ch @ 1/8) at 89.3% human / 96.1% bot borders, and the compact d8c32 (32ch @ 1/8) at 88.2% / 95.5%, one point behind at half the policy input. d8c32 cleared the pre-registered ≥88% bar, so it is the encoder for PPO v4. The channel-count control (c96) was pruned unfinished: its host kept dying and the resolution result had already answered the question. First deployment-style eval of ppo_v3 (fixed-seed local games, sampled actions): 2 wins / 2 losses on the first four stage-4 games, consistent with its training roll-win.

Jul 7 ~00:45: v4.2 deployed mid-run. Watching v4 replays exposed the phantom-boat exploit (43-78% of decisions were boats the engine silently discarded - see bugs). v4.2 fixes it three ways (honest masks, valid-tile snapping, wasted-intent penalty) and resumed ppo_v4 from its own checkpoint - no retrain needed, since no tensor shapes changed. The new episode/wasted curve opened at ~85-93 discarded intents per episode (stage 4), the measured size of the exploit; the penalty now prices those at noop-minus-w_waste. Watch items: wasted falling, boat's action-mix share normalizing, and a transient roll-win dip while the value head absorbs the reward shift (resumed at update 260, roll-win 0.28).

Numbers: 375k bot + 420k human full-state snapshots · AE border accuracy 71.8% → 89.3% (human) · 5.9M-param policy · 11 curriculum stages · first 2 engine wins Jul 6.

Run ledger

What is running, what is done, and what is stale. All RL/BC still loads frozen ae_v3.pt until the v3.1 winner is wired into rl/obs.py.

Is PPO actually improving? Yes, but the signal was hard to read. Placement-gated curriculum v1 looked like progress (roll-score ~0.65 on Pangaea) while the agent never won. Curriculum v2 reset both runs to 1v1, win detection was broken until 12:06, and both policies now win stage 0 in replay. By evening v3 had cleanly won the head-to-head: stage 4 with 60-70% wins vs v2c stuck at stage 3 with 7-17%, which is why v2c was retired. Neither lineage has been trained with the new encoder or BC warm start yet.

July 5th through July 7th, 2026

Complete chronological record (newest entries at bottom of list).

Timeline

• Jul 5 eve: headless datagen + tile AE v1. TS runner boots the real engine in Node, plays bot games, dumps packed tile grids + full entity state every 10 ticks.
• Jul 5 night: AE v2: one unified latent over ALL state. Spatial recon excellent; exact facts not (alliance pairs F1 0.67, troop ordering 0.81 after four tuning runs).
• Jul 5 23:14: AE v3: spatial-only + bypass split. The pivotal correction. 1.17M params, 64ch latent per 16x16 region, structures at P/R 1.0 through the latent.
• Jul 5 23:25 → Jul 6 00:36: RL scaffold → vectorized PPO → curriculum v1. Engine bridge over stdio, obs builder on the frozen AE, factorized masked heads, strength-index reward.
• Jul 6 morning: human-game pipeline (fetch archive, hash-verified deterministic replay to snapshots, HF upload) and restart-proof cloud training (full-state checkpoints, supervised restarts, HF checkpoint sync). Training hosts became disposable.
• Jul 6 10:27–11:32: GameRecord replays (bit-identical in the real client) and the live-lobby agent client: fight the policy from a normal browser.
• Jul 6 11:07–11:19: curriculum v2: map pools, 25% rehearsal, win-gated advancement, 1v1 → 1v3 start for dense wins.
• Jul 6 11:40: behavior cloning from human games (placement-conditioned; optional temporal transformer).
• Jul 6 12:06: win-detection bug found and fixed. Every win-rate stat before this is invalid.
• Jul 6 12:20: training speed overhaul: per-batch padding, pipelined rollout, bf16, binary IPC, per-stage decision cadence.
• Jul 6 13:04–13:39: real-client replay videos (actual game graphics, agent crowned as AGENT) and the MODEL debug overlay everywhere: rendered videos, browser replays, live matches.
• Jul 6 aft: AE reckoning: benchmarked the bot-trained encoder on human games (border accuracy 71.8% vs 87.5% on bot data), retrained on a bot+human mix, then designed and ran the v3.1 ablation matrix in parallel. Latent resolution won. New encoder + benchmark table uploaded to HF.
• Jul 7 early: the phantom-boat audit. Replay sidecars showed the agent picking boat on 43-78% of decisions (p(boat) up to 0.79, p(noop) 0.01) with nearly all of them silently discarded by the engine - the cap is 3 transports in flight. Root cause: a whole class of actions was "legal" per the mask but a guaranteed no-op at execution, and a discarded intent was reward-identical to noop with occasional upside - a free lottery ticket the policy learned to farm (ppo_v3's version was 61% build on random land). Fixed everywhere at once: masks now match engine state (canBoat = boat slot free + own shore, canExpand = neutral land actually borders, hasSilo now respects silo cooldown + spawn immunity), the translator snaps tile picks to tiles that can work (boats never target own/ally land, builds only own territory, ports own shore), and whatever still slips through is counted by the bridge (wasted, exact engine calls per intent) and penalized at W_WASTE=0.01 per discarded intent, so noop strictly dominates doomed actions. New episode/wasted TB curve. Tagged v4.2 and hot-deployed to the running ppo_v4 pod at ~00:45: trainer killed, repo synced, resumed from the same checkpoint (update 260, stage 4, win-window intact) - resume-safe because nothing about the network changed. First v4.2 episodes measured the exploit at ~85-93 wasted intents per episode.
• Jul 6 eve: v4: the whole policy stack moved to 1/8 resolution (d8c32 encoder, 8x8 tile regions, 150x250 grid budget) - the clean cut, no space-to-depth shims. GPU-side featurization (ego pooling, fallout unpack, AE encode, local crop batched in encode_grids), which also fixed a stale-fallout bug (dynamic fallout was riding inside the per-episode cached terrain tensor). Raw 64x64 local owner-crop bypass. Learned spawn placement end to end: spawn action + legal-region mask in PPO, spawn-phase snapshots + labels in replay sidecars (formatVersion 2) for BC. BC prefeaturize cache (scripts/prefeaturize_bc.py): 1.45 ms/sample single-threaded vs ~15-20 ms before. PPO v4 fixes: entropy anneal, stage LR warmdown, persisted win-gate window, periodic fixed-seed greedy eval. Pod supervisors got commit assertions and crash-loop backoff.

July 6th, 2026 - architecture and curriculum

Observation stack, reward, curriculum design (written this day).

Observation: compress the map, bypass the rest

The core architectural finding so far. Three iterations:

• v1: AE over tile ownership + terrain. Worked; too narrow.
• v2: one unified AE over all state (tiles, players, units, diplomacy). Spatial recon excellent, but tiny exact facts fought the bottleneck: alliance pairs peaked at F1 0.67 and troop ordering 0.81 across four tuning attempts (loss weights, pooled-vector capacity, per-slot latents, graph-to-encoder).
• v3 (current): compress only what is big: the map. Tile accuracy 99.4% (border-tile 87–99% by map), structures precision/recall 1.0 per class. Everything small and exact bypasses the latent raw: pairwise diplomacy, per-player scalars, transient units (nukes in flight with impact points), attack aggregates, legality masks.

The lesson: a one-bit fact reconstructed at 95% is strictly worse than reading the bit. Autoencoders are for high-dimensional state; never make exact small state fight the map for latent capacity.

AE details that held up: border-weighted cross-entropy (borders blur first and matter most), owner relabeling to static per-game spawn slots (any player count, fixed channels), fully-convolutional training on random crops (one model, any map size), and rarity-weighted BCE detection for structures: count regression collapses to all-zeros on 99.9%-empty grids.

AE v3.1: the border-accuracy push

The trigger: overall tile accuracy saturates near 99% for every model (water and player interiors are easy), so the honest metric is border-tile accuracy, and it wasn't in the training logs at all. Benchmarking the bot-trained v3 on the newly replayed human games exposed a 16-point domain gap: 87.5% border accuracy on bot data, 71.8% on human. Human territories are gnarly (naval invasions, enclaves, 50+ player fronts); nation bots grow blobs.

Two fixes ran in sequence. First, retraining v3 unchanged on a bot+human mix recovered the domain gap (80.1% human), so data was part of the problem. Then v3.1 attacked the architecture with seven changes at once plus two ablations to find the real constraint:

• decoder conditioned on static terrain (land + magnitude, never dynamic fallout), side information the policy gets for free, so the latent stops re-encoding coastlines
• nearest-upsample + 3x3 convs replacing ConvTranspose (no checkerboard), plus a full-res 3x3 refinement block before the classifier (was a bare 1x1, zero neighborhood context)
• border-density rejection sampling for training crops (uniform crops waste most gradient on ocean), focal loss on hard pixels, warmup+cosine LR, and native border-accuracy logging
• ablations: latent channels 64→96 at 1/16 res, vs. latent grid at 1/8 res

Result: resolution is the constraint, not capacity. Uniform-crop eval, 256 samples:

At the same 1/16 resolution, the v3.1 decoder/loss work helped bot borders but couldn't beat plain mixed-data retraining on human borders. Borders are high-frequency spatial detail, and a 64-number vector summarizing a 16x16 patch simply cannot store where a ragged front cuts through. Halving the patch to 8x8 bought ~9 points on both domains. Structure detection stayed at precision/recall 1.0 throughout. The catch: the 1/8 latent is 4x the policy input; a 1/8 x 32ch run (2x baseline) is in flight to see if the win survives halved channels, alongside the 96ch control.

Complementary hedge (decided): the policy will also receive a raw local owner-map crop around its own territory, exact borders where the agent acts most, latent for global context. The latent doesn't have to be pixel-perfect everywhere.

Action space

Full intent surface from day one, legality masking only (never curricular). Factorized masked heads over a shared conv trunk. One flat softmax is impossible (tile arguments alone are millions of options):

Masks come from exact engine legality calls each step; the policy can only pick what the engine would accept. Where a tile argument can still fail at execution (boat destination unreachable by water, build site too close to another structure), the bridge counts the discarded intent and the env charges w_waste for it - the mask keeps the policy honest, the penalty keeps the engine honest. Full intent table in DESIGN.md.

Reward

Iterated from raw land occupancy to a composite strength index: 0.45 x land + 0.25 x military + 0.30 x economy (shares). Land alone mis-scores legit strategies like tiny-island gold-stacking.

• dense: w_str x strength x timeweight, timeweight 0.5→1.0 over the first 8000 ticks. Strong late is worth double strong early, and it doesn't net to zero on death like pure deltas
• delta term for immediate credit (v4: symmetric 5.0; v5: gain 5.0 / loss 6.5)
• terminal: w_place x place^-pow (v4: pow=0.7; v5: pow=1.5), power law: 1st ≈ w_place, 2nd much lower at v5; the dead rank behind everyone still alive
• win bonus w_win = 30 (doubled when the curriculum became win-gated)
• small flat death penalty
• wasted-intent penalty w_waste = 0.01 per intent the engine silently discarded (doomed boat, invalid build site, expand with no neutral border): without a price these are noop-with-upside and the policy farms them (see bugs)

Curriculum

Eleven stages over seven maps (Onion → Pangaea → Caucasus → BlackSea → BetweenTwoSeas → World → Asia). Three principles, all added after watching v1 fail to progress:

1. Dense win signal first. Stage 0 is 1v1 vs a single nation on Onion; stage 1 is 1v3. Only then tribe bots and full rosters. Before this the win bonus essentially never fired.
2. Map pools, not single maps. The map is sampled per episode, so no overfitting one layout.
3. Rehearsal against forgetting. 25% of episodes replay an earlier stage's map at the current bots/difficulty. Old maps come back harder. Rehearsal trains but doesn't count toward advancement.

Advancement is win-gated: rolling win rate over the last 40 on-stage episodes must exceed 0.5. The agent must be winning more often than not, not merely surviving.

Prior art: AlphaFront

josh-freeman/openfront-rl (HF mischievers/openfront-rl-agent, "v19b") is the other PPO agent on this engine we're aware of, and a lot of our scaffolding follows from what they figured out first. The main fork is observation and tile choice: they keep a compact scalar view (~96 floats) and route spatial decisions through hand-coded heuristics (closest border tile, pickBuildTile()); we put map structure in the observation stack and learn tile selection with a spatial pointer (the bridge still snaps to legal tiles). Different tradeoffs: compact obs plus heuristics is simpler to train and deploy; richer spatial input is heavier but can in principle learn geography, chokepoints, and nuke aim.

What they showed works here, and we adopted: PPO converges on this engine at ~10 ticks/decision (we chose the same cadence independently); win-rate-gated advancement is viable. Still on our borrow list from their setup: LR warmdown on stage transitions; tiny flat generated maps as an even softer stage 0; a mid-game "beat all bots" gate so advancement doesn't wait on long endgames. On our side we've added a human-replay pipeline for behavior cloning: extra imitation signal that's awkward to fold into a purely scalar observation.

July 6th, 2026 - evening

Throughput engineering, progress graphs.

Performance

GPU util was poor and env stepping was the bottleneck:

• Per-batch dynamic grid padding. Everything was padded to the largest map (75x125 regions); Onion is 32x32, ~9x wasted conv compute on small maps. Obs stay native-size; collate pads to the batch max; pointer heads convert global↔local region indices.
• Double-buffered rollout. Env stepping (CPU/Node) and inference (GPU) were serial; now one env group steps while the other is on the GPU.
• bf16 autocast + TF32 + fused AdamW, plus a --compile flag.
• Binary tile IPC. Tile grids crossed the bridge as gzip+base64 JSON every step; now a JSON header + raw bytes on stdout.
• Per-stage decision cadence. Early stages decide every 15 engine ticks instead of 10.
• fp16 rollout storage + pinned staging + prefetch (Jul 6, 8a0da05). The single main thread was serializing fp32 copies, minibatch collation, and GPU compute; observations now ride as fp16, transfers go through pinned buffers, and a worker thread collates the next minibatch during the current backward. 3.6x on stage-3 maps (~590 → ~2100 game-ticks/s).
• v4.1 async rollout/update overlap (Jul 6 night, b8c331c). A collector thread builds rollout k+1 (env stepping + encode/act on a snapshot policy) while the main thread runs the PPO update on rollout k. One rollout of staleness; old_logp stays anchored to the acting snapshot so the clipped ratio math is unchanged. The former ~35s rollout phase now hides inside the update phase (perf/rollout_wait_s tracks what's left).

Progress graphs (Jul 6 night)

Generated by scripts/make_progress_graphs.py from the surviving TB event files and BC jsonl logs. ppo_v3's curves died with its pod (lesson: archive events before terminating); it appears as annotated reference lines from the devlog record.

• Curriculum: ppo_v4 cleared stages 0→3 in ~2h from scratch and reached stage 4 at ~3.6h - territory v2c never reached and roughly v3's pace despite v4 carrying the 1/8-resolution stack, learned spawns, and two mid-run restarts.
• Throughput: the per-update tps window decays as the curriculum moves to bigger maps (Node engine tick cost), so compare like-for-like: at stage 3-4 the fp16+prefetch fix took sustained tps from ~590 to ~2100, and v4.1 hides the rollout phase entirely.
• BC: the cache killed the data bottleneck (~1.5ms/sample vs 15-20ms), which moved the limiter to the GPU: bc_v4 runs ~56 ex/s at batch 24 x accum 4 on the 5090 (vs the pilot's 34 ex/s where the GPU sat idle waiting on gzip+JSON). Loss/accuracy now measure the model, not the pipeline. Loss is not yet trending down - that judgment (and the temporal transformer comparison) needs the full 60k-step run.

July 6th, 2026 - late night

Phantom-boat fix and the Rust port (~midnight).

Escaping the GIL: Rust hot paths + free-threaded Python (Jul 7 early am)

Every performance fix this project has shipped was some flavor of "get work off the single Python thread" (VecEnv processes, GPU featurization, pinned prefetch, the v4.1 async collector). Two structural moves land the endgame:

• rust/ofrs (PyO3 + rayon, 9f262a6). The two stable, allocation-heavy loops are now Rust with the GIL fully released: cache-bc frame decode (zstd + split) and the pad+stack collate that runs per PPO minibatch (2.6x single-threaded, and it no longer blocks other threads at all - which is what the prefetch thread actually needed). rl/native.py keeps numpy fallbacks so nothing requires the toolchain; pod scripts build the wheel opportunistically at boot. Formats are frozen (CACHE_FORMAT=1, obs v4), which is what makes these safe to freeze in a compiled language.
• Free-threaded CPython works, today, on our stack. torch dropped 3.13t but ships cp314t wheels (2.10+, CPU/CUDA/macOS); numpy and zstandard have cp314t wheels; orjson doesn't (we already fall back to stdlib json). Verified locally on 3.14.5t: torch 2.11 imports with the GIL genuinely off, rl.bc trains, and the BC sampler scales 322 → 1104 samples/s at 4 threads (vs 356 → 598 with the GIL - 1.7x from thread parallelism the GIL was eating). Two caveats from the bench: threads must own their samplers (shared hot objects ping refcounts across cores), and 8 threads regressed on the laptop (memory bandwidth), so pods should tune worker count.
• Migration stance: pods stay on the image's GIL python while runs are hot; the RunPod images pin torch to the driver. Next fresh pod generation moves to 3.14t (cu130 cp314t wheels exist), at which point thread-based feeding replaces the DataLoader-process workaround and the PPO collector/updater threads stop time-slicing.

Round two (8aa7fbd): the whole BC sampler is Rust now. The DataLoader process workers turned out to be a regression in production - 16 workers sat idle while the main process choked unpickling ~50MB raw batches off the IPC queues (stall 300s per 50-step window on bc2). The fix was to delete the abstraction, not tune it:

• ofrs::Sampler - sample_batch(n) does everything rl/bc_data.py did per sample (zstd frame/entity/step decode, serde JSON, the full obs-v4 featurization, label→head targets, legality forcing) in Rust, rayon across step draws, GIL held only to hand back numpy dicts. Laptop: 639 → 2631 samples/s vs the Python path; parity proven exactly (every array, 11.8k samples) by scripts/test_ofrs_parity.py. sample_windows(batch, k) draws a whole seq micro-batch in parallel for the temporal run. The trainer feeds from a plain thread pool again - no processes, no pickle.
• vec.py IPC off pickle - env workers ship a binary frame (u32 header-len ++ json ++ raw array buffers); the parent parses with ofrs::unpack_arrays. Numpy fallback speaks the same protocol.
• collate fully native - the ~8 small per-key np.stack calls now go through ofrs::stack_f32/f16 (parallel copy, GIL released), so the PPO prefetch thread's collate no longer serializes against the collector.
• Measured on the pods: bc2 14.9 → ~105 ex/s sustained (GPU 36% → ~90%, OOM-tuned to micro-batch 24 x accum 4); bc3 seq 4.5 → ~10.5 ex/s (window draws remain rejection-sampling-bound; next lever is a smarter end-picker, not more Rust); rl4 restarted clean on the new IPC with expandable_segments after an OOM at update 267 (fragmentation after 1135s uptime, not the new code path).

July 7th, 2026

Overnight check-in through the morning session.

Overnight results (Jul 7 morning)

• bc_v4 (bc2): step 6.5k → 17k overnight (~1M examples at ~95-105 ex/s). Holdout: acted-action 0.74-0.75, player 0.65, quantity 0.62, build 0.67-0.79, nuke 0.75; best-checkpoint metric climbed to 0.406. tile_region stays ~0-0.07 - the ~37k-class tile head is the clear weak spot, not new tonight. Known issue: throughput decays over hours (100 → 25 ex/s, RSS 23GB, stall climbing) and a trainer restart fully restores it - smells like slow accumulation in the feed path; restart-on-decay is the mitigation until it's profiled.
• bc_seq_v4 (bc3): ~3k steps at ~10 ex/s. First holdout evals: acted-action ~0.50, player 0.59-0.60. Behind the feedforward run at the same wall-clock, as expected at 5x less data; the temporal-vs-feedforward call needs matched sample counts, not matched hours.
• ppo_v4 (rl4): reached update 433 / 1.35M steps, stage 4. Headline concern: rolling win rate fell 0.28 → 0.00 over ~120 updates while placement score held 0.72-0.84 and entropy slid 9.2 → 2.8. The agent consistently places 2nd and has stopped converting to wins - looks like entropy collapse into a safe-second strategy (the win bonus vanishes from its experience, so nothing pushes it over the top). The binary-IPC deploy is exonerated: a pod-side round-trip probe preserves the win bool, and the decline started before the deploy. Candidate levers for the next session: entropy-coef floor or schedule, and the v5 asymmetric strength shaping.

PPO v5: breaking the safe-second attractor (Jul 7)

ppo_v4 was killed at update 433 (final checkpoint archived to HF) and ppo_v5 launched on the same pod with three MDP/optimizer changes aimed squarely at the collapse, plus a BC warm start:

• Adaptive entropy floor (--ent-floor 3.5): a multiplicative controller on the entropy coef (x1.3 per update when measured entropy is under the floor, capped at x30, hysteresis at 1.4x floor). v4's entropy slid 9.2 → 2.8 at a barely-annealed coef of 0.009 - the schedule alone cannot prevent collapse, only a feedback loop can. The scale persists through checkpoints and multiplies the existing anneal.
• Sharper placement curve: PLACE_POW 0.7 → 1.5. Terminal rewards for 1st/2nd/3rd go from 45/9.2/6.9 to 45/5.3/2.9 - second place lost 42% of its value, so "safe second" is no longer a comfortable optimum even before the win bonus enters experience.
• Loss-aversion delta shaping (v5 wishlist): W_DELTA splits into GAIN 5.0 / LOSS 6.5. v4 games showed hyper-expansion then collapse (103k tiles at t8001, dead by t10711); symmetric shaping priced a lost tile the same as a gained one, so defense was never worth learning.
• BC warm start: INIT_BC=bc_v4 pulls bc_best.pt (holdout 0.406) from HF, loads the 70 shared Policy tensors and folds the winner-bucket conditioning into the head biases.

Early read (30 min in): stage 0 cleared in ~25 updates, stage 2 by update 64 - v4 needed hours for the same ground. Entropy holds ~4.8 with the floor controller live (ecoef oscillating 0.010-0.048 as the controller breathes). Deliberately NOT in v5: the LSTM/temporal policy - bc3's seq-vs-feedforward comparison is the evidence gate for that, and it hasn't reached matched sample counts yet.

bc2 decay hunt + encode as the real feed cost (Jul 7)

• Instrumentation first: bc.py now logs per-phase feed seconds (smp/enc/col) plus CUDA allocated/reserved every 50 steps. First readings overturned the "sampler accumulation" theory: smp ~0s (the Rust sampler is essentially free), col ~5s, enc 50-65s per 50-step window - the frozen-AE encode is the entire feed cost, and it shares the GPU with the trainer.
• bf16 autocast on the frozen AE encode (rl/obs.py): free conv throughput - the latents already ship as fp16 to PPO. bc2 recovered to ~85-93 ex/s; bc3's seq run is fully encode-bound (enc 153s of a 163s window, 8 frames per sample) and sits at GPU ~95%: the next real lever there is caching AE latents in cache-bc (the encoder is finally frozen at d8c32, so the cache won't be invalidated - v4's reason to skip it is gone).
• Decay watch - caught in the act (30 min after restart): bc2 slid 93 → 35 ex/s again, and the instrumentation pins it: enc grew 49 → 135s/window while smp stayed 0.0 and col ~9s; CUDA reserved is flat at 27.5GB; GPU util 31% with clocks at 2872/3090MHz, 43°C, 202W. So it is NOT fragmentation, NOT thermal throttle, NOT the sampler - something CPU-side inside encode_batch (the single-threaded np.stack H2D staging of full-res owner grids) slows as the process ages. RSS was already 11.7GB at 34 min uptime (23GB overnight): prime suspect is glibc heap/arena churn from the hundreds of MB of short-lived numpy buffers per batch - consistent with a restart fully fixing it. Experiment running: MALLOC_ARENA_MAX=2 on bc2. Real fix if confirmed: have ofrs return pre-stacked batch arrays so the Python-side memcpy churn disappears.

v5.1 denser rewards + bc2 decay root-caused (Jul 7 midday)

ppo_v5 status at the time: not stuck so much as slow - roll-win crept 0.00 → 0.17 and roll-score 0.29 → 0.85 on stage 4 by update 260, but the gate needs >0.5 wins, and the run was also OOM-crash-looping every ~1h (CUDA OOM in group_norm/conv2d at 96 envs on the stage-4 map pool; the supervisor restarts and resumes, so progress survives but momentum doesn't). Two changes:

• v5.1: structures join the strength blend. The old blend (land/mil/eco 45/25/30) had a real mis-incentive: buying a building read as a strength loss - your gold share drops and the building counts nowhere, so the loss-averse delta actively punished construction. Now K_BUILD=0.15 adds a structure-value share (blend 40/20/25/15): each player's finished structures are valued by in-game cost ceilings (City/Port/Factory/Silo 1.0, DefensePost 0.25, SAM 3.0, × level) and shared over living players' totals. Completing a building now pays an immediate W_DELTA_GAIN spike, losing one to a nuke costs W_DELTA_LOSS, and holding a built-up economy earns dense reward every step - the reward is denser exactly where the action space was silent.
• bc2 decay: root-caused and fixed fundamentally. The faulthandler dumps caught the prefetch thread in np.stack at rl/obs.py:449 - the terrain_static stack, ~460MB of fresh float32 per batch (19MB/sample × 24), freed immediately. Allocations that size go through glibc's mmap path, so every batch paid mmap+munmap page faults; on a host with badly fragmented physical memory (65M compaction stalls, 115B failed page migrations in vmstat) the fault path itself degrades - a fresh-allocation memcpy microbenchmark on the degraded box ran ~300MB/s cold vs 1.6GB/s warm. That's why restarts fixed it (fresh page tables, warm buffers) and MALLOC_ARENA_MAX didn't (arenas don't govern mmap'd allocations). Fix: encode_grids now stacks the four big full-res arrays into persistent per-thread staging buffers (np.stack(out=...)), keyed by array name+shape and grown to the largest batch seen - the pages fault once and stay warm, so there is nothing left to decay.
• Per-stage win gates + curriculum re-grounding (afternoon). Stage-4 telemetry showed the safe-second signature reforming under v5.1: roll-score climbed 0.29 → 0.84 and saturated while roll-win stalled at 0.17 - the agent reliably places 2nd but doesn't close out games. Diagnosis: the flat 0.5 gate let the BC-warm-started run clear stages 0-3 at 21/40 wins each, so it never mastered the close-out kill where wins are cheap (1v1 Onion). The gate is now per-stage - 0.9 / 0.8 / 0.75 / 0.65 / 0.55 for stages 0-4, tapering to 0.3 by stage 10 where winning is genuinely hard - and ppo_v5's checkpoint was reset to stage 0 (weights and optimizer kept, win window cleared): it must now re-earn the curriculum under dominance gates, with rehearsal keeping the old maps in the mix.

Reward MDP reference (Jul 7)

The dense + terminal reward is fully specified in rl/curriculum.py and applied in rl/vec.py. These panels are generated from the live constants by scripts/make_reward_graphs.py (re-run after any tweak):

• Terminal panel: v5's sharper PLACE_POW is the main anti-safe-second lever - 2nd place drops from 9.2 to 5.3 before any win bonus. The win bonus (+30) still dominates a outright win (45 total for 1st+win vs 5.3 for 2nd-only), but the agent has to actually win to see it; placement alone now pushes harder for the top.
• Dense panel: most steps are small (strength ~0.1-0.3 at timeweight 1.0 gives ~0.002-0.006/step); the delta term is where expansion and collapse show up (±0.05 strength swing ≈ ±0.25-0.33 per step). The heatmap is the tuning surface for "how aggressive should expansion be?" Loss-aversion tilts the agent toward defending a lead.
• Scenario panel: synthetic strength traces (not real games) show total return under the old vs new MDP. Safe-second timeout pays similarly in dense terms but v5 cuts the terminal; expand-then-collapse gets punished twice (asymmetric deltas on the way down + worse placement). Waste spam shows why W_WASTE exists: without it, doomed boats are free lottery tickets.
• Strength panel: the composite index is not raw land - eco islands and military rushes can score honestly. Tuning K_LAND/K_MIL/K_ECO shifts which strategies the dense signal rewards mid-game.

July 7th, 2026 - evening

v5.1 freeze postmortem, the OOM root cause, temporal-BC verdict, v6 plan.

ppo_v5.1 frozen: the MDP worked, the crashes ate the wins

ppo_v5 is frozen (run killed tonight; final snapshot at update 550, global_step 1,725,216, stage 4, roll-win ~0.25 against the 0.55 gate - it advanced from ~530 during shutdown). The verdict on v5.1 is that the reward and gate design did their job: after the curriculum reset it re-earned stages 0-4 rapidly under the dominance gates, the structure-value reward (K_BUILD) is visibly expressed in play (see the behavior analysis), and it cleared the stage-5 win gate twice. Both advances were lost to CUDA OOM crashes before a checkpoint captured them - the run kept dying at exactly the moment it succeeded. The frozen weights become the v6 warm start.

OOM root cause: collate padding, not fragmentation (36a8efb)

• The real cause. NOT fragmentation - expandable_segments was already on. collate() pads every update minibatch to its largest grid, so one BetweenTwoSeas sample (132x223 latent cells) padded all 128 samples to 29k cells, and loss.backward() needed multi-GiB activation allocations - precisely at stage-5 entry, where the big maps first appear. The crash wasn't random; it was structural to advancing.
• The fix: zero padding waste. Update minibatches are now split by native grid shape, with a 1.6M latent-cell budget per forward/backward (MAX_UPD_PIX in rl/ppo.py); sub-batch losses are weighted by sample fraction so the accumulated gradients are numerically identical to the padded batch. Result: GPU 30.2 → ~25GB, throughput ~353 → ~430 game-ticks/s, zero OOMs after deploy.
• Checkpoint on curriculum advance (same commit): a stage advance now flags an immediate save at the end of the in-flight update instead of waiting out the 10-update cadence. A cleared gate can no longer be lost to whatever kills the process next.

The "always 20% attacks" mystery: resolved by measurement

Replays kept reading as if the agent always attacks with ~20% of its troops. New quantity-mix telemetry (TB + analyzer) settles it: the 5-bucket quantity head ({5,10,25,50,100}%) is alive and differentiated, favoring 10% and 25% attacks, with 100% almost never sampled - and a stream of 10-25% attacks reads as "about 20%". Coincidentally 20% is the engine default when troops is null, but our intents always carry explicit counts. Nothing was broken; the eyeball estimate landed on the mixture mean.

Behavior analysis: where the policy actually is

scripts/analyze_policy_behavior.py, run on the rl4 GPU against the update-490 checkpoint, 4 sampled episodes per stage:

• Stage 3 win 3/4, stage 4 win 3/4. The winning pattern: expand early, mass troops, then snowball 200k → 900k tiles with streams of 10-25% boat+attack intents while building continuously. One stage-4 win ended with 59 cities, 21 ports, 6 silos - the structure reward is clearly expressed.
• Stage 5: 0/4. Dies tick 3,400-5,200 before establishing any economy (zero structures in 3 of the 4 deaths), even on rehearsal Onion. The wall is the Easy → Medium difficulty jump, not map size.

bc_seq_v4 killed + the BC slow decay finally fixed (9c030f7)

• Temporal BC loses, decisively. bc_seq_v4 (temporal BC, seq=8) killed on the bc3 A100: holdout action accuracy 0.48-0.60 vs the feedforward bc_v4's 0.65-0.70, at 8x the encode cost per sample; the A100 sat at ~0% utilization (~10 ex/s). Lesson: temporal BC is only worth revisiting after AE-latent caching makes sequence windows nearly free.
• BC slow-decay root cause, one level deeper. glibc malloc's dynamic mmap threshold caps at 32MB while batch buffers are 50-80MB, so every batch paid mmap/munmap plus zero-page faults; bc_v4 decayed 88 → 12 ex/s over ~5h, with the stall moving to collate after the earlier obs.py staging fix. Fix: MALLOC_MMAP_THRESHOLD_=256MB and MALLOC_TRIM_THRESHOLD_=256MB in both pod launchers; bc_v4 restarted at ~100 ex/s.

Next: v6 (plan written)

Highlights of the written plan:

• Action space 14 → 21, covering every mechanical human intent: upgrade_structure, move_warship, cancel_boat, delete_unit, embargo_stop, target_player, alliance_extension. Warship joins BUILD_TYPES.
• Nuke head 3 → 5: Atom/Hydrogen × arc direction + MIRV. The arc flips the parabolic flight path for SAM evasion; MIRV ignores it.
• Quantity head becomes a scalar: a 0-1 Beta distribution replaces the 5-bucket softmax.
• BC sidecar re-replay for new-action labels.
• Extend-and-copy warm start from frozen ppo_v5.
• Per-update state.json to make the rolling win window restart-proof.
• AE-latent RAM-LRU cache to make BC ~10x faster: encode is 90% of BC wall time, re-computing a frozen deterministic function.

July 8th, 2026 - daytime

v6 first night, the honest cache verdict, the v6 label pipeline, and the 100x scaling plan.

ppo_v6 launched: stages 0-3 in one night, then the stage-4 wall

ppo_v6 went live on rl4 at ~11pm Jul 7, warm-started from frozen v5 via --init-extend: 62 tensors copied verbatim, the action head extended 14 → 21, build 6 → 7, nuke 3 → 5 (new logits initialized near zero so existing behavior is preserved), and the Beta quantity head fresh - the one component with no v5 ancestor. It cleared stages 0-3 overnight under the dominance gates (0.8 / 0.75 / 0.65 win) and hit stage 4 by morning. There it sits at the wall: update ~743, roll-win 0.10 against the 0.55 gate, roll-score 0.88 - it dominates games but is weak at converting them, the same safe-second pattern v5.1 showed at this stage. Entropy settled 8.9 → 7.6 over the night with the floor controller live - no collapse, the extended heads are being explored.

Live-play stack verified against v6: scripts/play_live.sh works end-to-end with the new action space - the agent joins a real lobby and plays full games.

Behavior analysis: a genuinely new playstyle on night one

scripts/analyze_policy_behavior.py against the update-190 checkpoint, 4 sampled episodes per stage: wins 100 / 75 / 75 / 25 / 0% on stages 0-4. The interesting part is not the win curve, it's that v6 is not playing like v5.1:

• Boat-dominated: boats are 54-73% of actions across stages (v5.1 mixed boats and land attacks much more evenly), and cancel_boat - a brand-new action - is already used as an active hedge, up to 14% of actions at stage 4: launch, reassess, recall.
• Warships are real: 13% of builds, with move_warship following coherently after them rather than firing at random.
• Build mix shifted: Defense Posts (48%) + Factories (22%) dominate, where v5.1 leaned cities/ports - consistent with the loss-averse deltas finally having a defensive action vocabulary to express themselves through.
• The Beta quantity head is healthy and differentiated on night one: boats commit 20-50% of troops, attacks 10-30%, with contextual shifts rather than one fixed fraction - the scalar head is doing on day one what the 5-bucket softmax took v5 its whole life to approximate (the "always 20%" mystery).
• Nukes fully dormant: 0 launches, 1 silo built all night. The 5-way nuke head has yet to enter the policy's experience; worth watching once economies get bigger.

Boat-churn and the outcome-only reward rule

The boat numbers above are not a playstyle, they're an exploit. 54-73% boats with cancel_boat at 14% of stage-4 actions is launch/recall churn: the agent sends boats out and calls them home in loops, and the roll-score 0.88 / win 0.10 split is the signature of learned busywork rather than closing out games. The rule this hardens into (rl/curriculum.py header): punish illegal or wasted actions, reward outcomes (state), never individual actions. Any reward attached to an action invites pairs of legal actions that cancel each other out and farm the shaping instead of the game.

The audit found no explicit per-action bonus to delete - the MDP was already outcome-based (strength blend + deltas + terminal placement/win) plus the W_WASTE illegality tax, and no-ops were never penalized. The churn reward was hiding in measurement: the strength blend's military share read player.troops(), the home pool only, and the engine deducts troops from the pool the moment a boat or attack launches, refunding them (minus the 25% retreat malus) on recall. So "send boat" scored as a strength loss and "cancel_boat" as a gain - a per-action reward pair in disguise - and boats doubled as a free troop bank, since troops parked on them dodge the maxTroops regen throttle. Fix: strengths() now counts troops in the field (aboard transports, committed to outgoing attacks) toward the military share. Launching and recalling are reward-neutral state moves; only real losses - combat deaths, the retreat malus - move the blend. W_WASTE still prices intents the engine silently discards (a cancel_boat with no boat in flight stays wasted), constants and the reward-MDP table are otherwise unchanged. Lands on rl5 at its next restart from master; watch the boat share of the action mix and the win conversion at stage 4.

The honest cache verdict: promised 10x, delivered ~2x

The v6 plan promised the AE-latent cache would make BC ~10x faster, on the theory that encode was 90% of wall time. Live numbers: bc2 peaked at 128-145 ex/s at 33-48% cache hits, bc_v6 runs 110-118 ex/s at 57-66% hits - call it ~2x over the ~60-100 ex/s pre-cache baselines. The profiler found why: at BATCH=96 the GPU fwd/bwd step itself caps at ~196 ex/s, because conv FLOPs scale with the padded batch-max grid. Encode was only part of the story; the model step was always going to cap throughput near there, and the cache merely exposed it.

• Fixes landed for the next restart (63d6e1a): disk-persisted z-cache (restarts stay warm instead of re-encoding from zero), a GPU-resident hit path (cache hits never round-trip through host memory), and area-bucketed batches (batchmates share grid sizes, so padding waste collapses - the collate-padding lesson again, this time for throughput instead of OOM).
• In flight today: bf16 autocast + torch.compile per shape bucket + dynamic area-based batch sizing (256+ effective) in rl/bc.py. Expected compound on top of the ~196 ex/s cap: ~500-650 ex/s ≈ 5-6x. Measured by a 500-step A/B on bc3 after its natural supervisor restart.
• Lesson: verify the ceiling before promising the multiplier - the second bottleneck is always waiting behind the first. "Encode is 90% of wall time" was true and still didn't make 10x true, because removing it just promoted the model step to the new ceiling.

The v6 label pipeline: 303 games re-replayed, 8h → ~2.5h

The seven new v6 actions need BC labels, which means a full re-replay of the human archive: 303 games to formatVersion 3 sidecars. First estimate was ~8 hours - because the replay driver ran 16 workers each batching 4 games, leaving a 128-core box ~95% idle. Fixed to 64 per-game workers plus a restart-safe --rebc skip that ignores already-refreshed games (d1ff455): ~2.5h wall for the full archive.

• One casualty: an EIO crash on the mfs network volume during prefeaturize. Recovered incrementally thanks to the skip logic - 212 games refreshed in the second pass, 0 empty sidecars. The cache-bc-v6 tars (29GB) are on HF, so no future pod ever replays this archive again.
• bc_v6 is the consumer: fine-tuning on bc3 from bc_v4's step-55k checkpoint via its own --init-extend (e6245da - the PPO head-extension machinery reused for BC). Step ~30k by midday, holdout evals already at bc_v4-final levels, and the Beta quantity target trains to MAE 0.096 - the scalar labels are cleaner than the old bucket argmax ever was.

The scaling plan: BC to ~5x, PPO to 10x then 100x

Written today after profiling both trainers. Where the time actually goes: on BC the feed is solved and the GPU step is the ceiling (above); on PPO the rollout (52s) already hides inside the update (82s) via the async collector, and the engine is coasting (384 cores, wait 0.0s) - so scaling means scaling the learner first, then actors. PPO currently does ~400 game-ticks/s at stage 4 on one 32GB GPU; target ~4,000 (10x), stretch ~40,000 (100x). Three phases, each shippable alone:

• Phase A - learner code wins (days, ~1.5-2x, no new hardware): bf16 autocast + torch.compile on the update path, epochs 4 → 2 with minibatch retuning (PPO tolerates this at our batch sizes; KL/clip-frac logging added so the gate is measurable).
• Phase B - one fat node (a week, ~8-15x cumulative): DDP the update across a 4-8x H100/A100 pod over the existing minibatch loop (the pixel-budget sub-batching already shards cleanly), scale envs 96 → 384+ with rollout inference sharded across the same GPUs. No algorithm change - pure scale, de-risking Phase C.
• Phase C - async multi-node actor-learner (2-4 weeks, the 100x path): evolve, not rewrite - rl/actor.py (extracted rollout + bridge + encode, shipping trajectories as zstd fp16 latents, ~1-2MB/step, version-tagged), rl/learner.py (the existing update loop + staleness handling: PPO's own ratio clipping + a hard version cutoff first, V-trace only if measured staleness demands it), rl/transport.py (ZMQ push/pull, no broker). Actors stateless and spot-friendly. Cost sketch: 4x H100 learner + 4 cheap actor pods ≈ $16-20/hr chasing ~30-60x. Gated on v6.1 running stably for ~a week.

Executing now: v6.1 = Phase A + Phase B together, launched as one new run on a 4x H100 pod (rl5) resuming ppo_v6's weights - same action space, direct resume, stage continuity preserved. See the run ledger for the verification gates.

v7 direction (parallel, non-blocking): foveated two-stream observation. Uniform /16 resolution was rejected by the old ablation - border accuracy 89.3% at /8 vs 78.5% at /16, "resolution is the constraint" - but that measured uniform resolution. Strategy-level context doesn't need pixel borders: v7 pairs a coarse /16 global stream (4x fewer cells on the maps that dominate late training) with a fine /8 stream restricted to own territory + the border band, extending the existing exact-borders local-crop machinery in rl/obs.py. Near tile targets resolve on the fine stream; far targets (nukes, cross-map boats) tolerate coarse regions. It cuts update AND rollout AND (later) transport cost - the single biggest code lever - but it changes the observation space and trunk input, so it needs an AE /16 fine-tune and an eval-parity gate (stage 4-6 win rate within noise) first. That's why it's v7's headline change, deliberately NOT in v6.1.

Rejected outright: feeding raw full-res maps into the policy (no AE) - it multiplies the measured bottleneck (per-latent-cell update FLOPs) ~64x, and weight pruning doesn't reduce dense conv time on GPUs. Submanifold sparse conv over active cells stays a stretch option (~3-5x ceiling, heavy complexity), unscheduled unless post-foveation profiling still shows dead compute.

Naming note (resolved late Jul 8): the "v7" tag briefly meant two different things - the foveation idea above, and the same-day full-state observation expansion in the next section. Since no v7 run had launched, the scope was merged rather than split: v7 = obs expansion + foveation, one version, one from-scratch retrain. Both changes break every checkpoint anyway, so shipping them separately would have cost two retrains and two BC-cache formats for no benefit. See the amended v7 scope.

July 8th, 2026 - evening: browser-side live play

ONNX in the browser, headless Chromium agent, MODEL overlay restored, spawn re-roll bug.

Live play on the homelab showcase no longer needs a server GPU/CPU model process. The pipeline:

• scripts/export_onnx.py exports the frozen AE encoder + policy to ONNX (dynamic map axes; TransformerEncoder nested-tensor / MHA fast-path disabled so the trace matches PyTorch numerically).
• openfront/src/client/webbot/: TypeScript ports of observation featurization (obsCore.ts / features.ts), categorical+Beta sampling, and intent translation, plus an onnxruntime-web session. Inference runs in a dedicated Web Worker so the main-thread WebSocket ping loop stays alive (~5s forward pass on World would otherwise trip the server's 60s ping timeout).
• URL trigger: ?webbot=<gameID> short-circuits the normal UI (Main.ts) and joins the lobby as AgentRL. Bot identity is isolated to sessionStorage so a human tab and a bot tab in the same browser don't fight over player_persistent_id.
• scripts/webbot_launcher.py: headless Chromium (Playwright) opens that URL; showcase_hub.py's /play launches it instead of rl.play. A small HTTP sidecar on --debug-port serves window.__webbotDebug so the existing MODEL overlay keeps working.
• Deploy: forked djmango/OpenFrontIO (upstream submodule we can't push to), pinned via openfront.commit + Dockerfile remote URL, rebuilt the showcase image on the homelab.

First live bug (spawn re-roll loop). During the spawn phase the action mask forced spawn every decision tick. Engine SpawnExecution treats every spawn intent as a re-roll: relinquish all tiles, then re-place. The bot teleported every ~10 ticks until a pick landed where getSpawnTiles returned empty (SpawnExecution: cannot spawn AgentRL), leaving it with 0 tiles → isAlive()===false → game over. Fix: gate spawn on !player.hasSpawned() (noop while waiting out the spawn timer), plus a translator guard that drops spawn intents once the agent already owns tiles. Python rl.play / PPO never hit this the same way because training episodes typically place once and the spawn phase ends quickly under the curriculum's start delay.

July 8th, 2026 - evening

BC verdict + freeze, playable v6_bc baseline, and v7 full-state observation expansion (plan, implementation, BC-regen ops saga).

BC was never really great - freeze, kill, keep a playable prior

Honest read across the whole BC line (bc_v1…bc_v6, plus the killed temporal bc_seq_v4):

• Ceiling is low. Feedforward holdout action accuracy saturates around 0.60-0.70 and then stops moving. bc_v6 spent ~35k steps (of a planned 60k) moving action acc from 0.60 → 0.62 and loss from 8.41 → 7.55 - two percentage points of action accuracy for a full A100 day. Temporal BC was worse (0.48-0.60) at 8x the encode cost. The human archive is small (~300 games), noisy, and heavily no-op / UI-default (quantity bucket 25% dominates because the client slider defaults there), so the prior learns "looks like a human clicking around" more than "wins games."
• As a PPO warm start it helped once, then stopped being the bottleneck. ppo_v5 cleared early curriculum stages fast off bc_v4/bc_best, which is the real win - BC buys a legal-action prior and a non-random spawn, not a mid-game strategy. Every later plateau (safe-second on stage 4, boat-churn, passivity after the fielded-troops fix) was a reward / curriculum / obs problem, not a "BC wasn't accurate enough" problem. Grinding BC from 0.62 → 0.65 would not have moved those.
• Infra cost dominated the learning signal. The z-cache was supposed to be 10x and delivered ~2x (cache verdict); glibc mmap churn, CUDA OOM on padded large-map batches, and finally an EDQUOT loop (disk z-cache budget 450GB on a 120GB volume - mfs reports cluster free space in df, so the quota is invisible until writes fail; 732b6c5 defaults Z_CACHE_GB=60) ate more calendar time than the accuracy curve justified.
• v7 makes the current BC weights unloadable anyway. The evening's obs expansion moves C_GRID 43→89 / P_FEAT 12→21 / N_SCALARS 8→11 (v7 scope), so there is no warm-start path from bc_v6 into v7.

What we did: froze bc_v6 at step 35,600 (ckpt already on HF as bc_v6/), terminated the bc3 A100 pod (fleet = rl5 only), and published a playable pure-BC baseline:

• v6_bc = bc_v6/bc_best.pt folded into a v6-shape Policy with the winner placement bucket (same fold as ppo --init). On HF as v6_bc/policy.pt.
• Play: bash scripts/play_live.sh --run-name v6_bc (now the script default). Or short PPO: python -m rl.ppo --init runs/rl/v6_bc/policy.pt … on a v6-shape checkout (pre-v7 C_GRID=43) - not against the v7 obs tree.
• No bc_v7; see the moratorium below.

BC moratorium (decided late Jul 8): no further BC runs until further notice. The trigger for the moratorium is honesty about evidence: the one claimed BC win - ppo_v5 clearing early curriculum fast off a BC warm start - was never isolated from the reward/curriculum changes that shipped alongside it. Before any bc_v7 is even considered, the existing BC artifact has to prove it does something useful. The unfreeze experiment is cheap and fully specified: on a v6-shape checkout, run two short PPO starts under identical seeds/config - ppo --init runs/rl/v6_bc/policy.pt vs from-scratch - and compare wall-clock to clear stages 0-3 and stage-4 roll-win at a fixed step budget. If the warm start doesn't show a clear, repeatable advantage, BC stays a frozen toolbox item indefinitely. Consequences now: ppo_v7 launches from scratch with no BC warm start; the homelab archive re-replay keeps running (raw sidecars are cheap, shape-agnostic, and useful for eval/analysis regardless), but the BC-specific prefeaturize + cache-bc/ upload steps are deferred, not queued.

Net: BC stays in the toolbox as a cheap prior hypothesis, not as a research track. Next accuracy wins come from obs (v7, now including foveation), reward (outcome-only / fielded troops), and scale (rl5 DDP) - not from another week of imitating 300 human games.

Closing the remaining blind spots (v7 part 1: full-state expansion)

Scope note (amended late Jul 8): v7 is now two changes shipping as one version - this full-state expansion (implemented, below) and the foveated two-stream resolution (designed, to build). Both break every checkpoint shape, so merging them costs nothing and saves a second from-scratch retrain.

v6's transient/entity streams omitted several things a human player sees that only started to matter once the bots got good enough to punish blindness to them. A plan was written and then implemented end to end in one session:

• Other players' land attacks - previously only boat/nuke arrivals showed up on the grid; ground invasions between two bots were invisible. New attack-front planes (TR_ATTACK_SRC, TR_ATTACK_RETREAT) rasterize every active attack's origin, any player, with log(troops) intensity, plus whether it's retreating.
• Unit ownership on the map - every transient unit plane is now ego-relative-split (own/ally/enemy) using the same clut ego-classing the static ownership grid already used, instead of one undifferentiated presence plane per unit type.
• Troop counts on boats, via intensity-encoded (not just binary) Transport/Warship/Train planes.
• Alliance-expiry timers - a countdown scalar per ally, ticks-to-expire normalized over a 3000-tick window.
• New unit types: SAM Missile, MIRV Warhead, Train, each a first-class transient plane; Missile Silo/SAM Launcher cooldown planes; an active-train-station flag on City/Port/Factory (rail network stays lightweight - no separate graph/edge representation, just the flag and the unit plane).
• Trade-ship destinations, per-unit health/cooldown, tile defense bonus (decoded from the engine's raw tile-state bit, exposed both as a region-averaged grid channel and a 5th local-crop plane alongside the existing exact-borders bypass), target-painting (a marked flag, mirrors the existing embargo-by-self column), troop/gold income rates, and the doomsday clock (in-clock flag + ticks, both per-player and a global clock-enabled scalar).

Architecture: every addition goes through the existing bypass, none through the AE. The frozen spatial AE (ae/model_v3.py) compresses only the map (ownership + terrain + static structures); everything above is either small-and-exact (player features, global scalars) or already-transient (unit planes), so it rides the same raw-channel/vector paths v3 established for nukes-in-flight and diplomacy. Consequence: this part needs no AE retrain, but no BC/PPO checkpoint survives - every shape constant moved (N_TRANSIENT 8→53, C_GRID 43→89, P_FEAT 12→21, N_SCALARS 8→11, N_LOCAL 4→5), so v7 is a from-scratch retrain, not a warm start (PPO only - BC is on moratorium). Note the foveation half of v7 does need an AE fine-tune; see below.

Engine surface + backward-compatible replay

obsCore.ts (live play) and datagen/common.ts (replay) both gained the same new fields, kept in parity, plus a hasFn feature-detection helper in datagen/common.ts: replay_all.sh checks out the archive's exact historical engine commit per bucket before replaying, and several of the new APIs (maxHealth(), troopIncreaseRate(), inDoomsdayClock(), hasTrainStation()) postdate some of those commits. Calling a method that doesn't exist yet at that commit would crash the replay instead of just omitting the field, so every new accessor is guarded and defaults sanely for old commits. The same defensive pattern runs on the read side: rl/obs.py pulls every new field with .get() + a zero/null default, so training keeps working against not-yet-re-replayed archive data during the migration window. The cached-frame format bumped (CACHE_FORMAT 1→2, adds a packed defense-bonus plane) with a stale-cache rebuild guard in scripts/prefeaturize_bc.py, and the Rust fast-path sampler (rust/ofrs/src/sampler.rs) stays frozen at the pre-v7 schema - rl/bc_data.py compares its compiled-in (P_FEAT, N_TRANSIENT, N_SCALARS) against the live Python constants and falls back to the pure-Python sampler on mismatch, so training is always correct, just slower until the Rust side is ported to match.

Foveated two-stream resolution (v7 part 2: designed, to build)

Decision (late Jul 8): the foveated observation is part of v7, not deferred to a later version. The original plan (see the daytime scaling entry) treated foveation as a follow-on because it changes the observation space and trunk input - but the full-state expansion above already breaks every checkpoint shape and forces a from-scratch retrain. Deferring foveation would mean a second from-scratch retrain and a second BC-cache format a few weeks later for zero benefit. Nothing is wasted by merging now: no v7 run has launched, the v7 code push is still held back, and prefeaturize hasn't started (and is deferred anyway under the BC moratorium).

Why it's worth it. The v3.1 ablation rejected uniform /16 latents - border accuracy 78.5% at /16 vs 89.3% at /8, "resolution is the constraint" - but that measured uniform resolution everywhere. Strategy-level context doesn't need pixel borders: what needs /8 precision is own territory and the contested border band; what's three map-widths away only needs to exist as a region. Foveation keeps the /8 fidelity exactly where the ablation said it matters and pays /16 everywhere else. On the giant maps that dominate late curriculum stages this is ~4x fewer global latent cells, and it cuts update AND rollout AND (later, Phase C) transport cost - the single biggest code lever identified in the scaling plan, bigger than anything left in Phase A.

Design - two grid streams replacing the single /8 stream:

• Coarse global stream, /16 over the full map: AE latents at /16 plus the same transient/bypass channel stack as today, region-averaged to /16. Every tile of the map is represented; nothing is ever fully invisible.
• Fine stream, /8, restricted to own territory + the border band: extends the existing exact-borders local-crop machinery in rl/obs.py (the same philosophy that already gives the policy pixel-exact borders and the 5-plane local crop). Carries the full C_GRID=89 channel stack at /8, plus one coverage-mask channel so the trunk can tell "empty" from "outside the fovea". Cells outside coverage are zeroed and masked out of any fine-stream pointer logits.
• Tile pointer heads go two-level: near targets (attacks, expansion, structure placement - the actions the ablation showed need precision) resolve on the fine stream; far targets (nuke aim points, cross-map boat destinations) resolve on the coarse stream, where a /16 region is already the honest precision of those actions. Concretely: the tile head picks a coarse cell first, then refines within the fine stream when the picked cell has fine coverage - one softmax over coarse cells, one masked refinement over the covered fine cells. Exact head wiring is the main open implementation question; the mask plumbing (rl/env.py legality masks per region) must follow whichever wiring wins.

What it requires:

• AE /16 fine-tune. The frozen encoder (d8c32) is /8-native. The coarse stream needs a /16 head - fine-tuned from d8c32 (a downsampling stage + short fine-tune, not a from-scratch AE), trained on the same bot+human mix. Quality gate below.
• Obs plumbing in rl/obs.py + the bridge: emit both streams; the fine stream's bounding box / coverage set is derived from ownership + border dilation, recomputed per decision step (it already is, for local crops). Bridge serialization gains the coarse stream; step payload shrinks on big maps.
• Trunk/conv input change: two spatial encoders (share conv weights where shapes allow) pooled and concatenated - the same fusion pattern as today, one more stream.

Gates before ppo_v7 launches on this:

• AE quality gate: the fine stream keeps d8c32's measured quality by construction (same encoder, same /8). The /16 fine-tune must hit region-level fidelity - structure precision/recall ~1.0 and ownership-region accuracy comparable to ae_v3_mix's /16 numbers - but is explicitly NOT held to the 89.3% border bar; borders live on the fine stream now. Benchmark table goes to HF next to the d8 one.
• Early-curriculum sanity: the original plan's eval-parity gate (swap obs under a trained policy, stage 4-6 win rate within noise) can't run - v7 is from scratch, there is no trained v7-shape policy to swap under. The honest replacement: ppo_v7 must clear stages 0-3 in wall-clock comparable to the v6 lineage (which did it in ~2h on 1 GPU), and per-update cost on the big-map stages must come in measurably under v6.1's. If either fails, the fallback is running v7 with the fine stream covering the whole map (= v7 obs at uniform /8, foveation off via config), which must remain a supported degenerate case.

Still rejected (unchanged from the daytime entry): raw full-res maps into the policy with no AE (~64x the measured per-latent-cell update FLOPs; pruning doesn't reduce dense conv time on GPUs), and submanifold sparse conv stays an unscheduled stretch option (~3-5x ceiling, heavy complexity) unless post-foveation profiling still shows dead compute.

v7's first mid-run fix: coarse tile picks only searched a quarter of their cell

Symptom (Jul 8 late): ppo_v7 on stage 1 showed boat-heavy noise again - lots of boats the engine discards, no boat-and-cancel play. Familiar smell (the v4 phantom-boat exploit, the v6 boat-churn), but the mechanism this time was new, and it was a genuine bug in the foveation change, not a reward hole.

Root cause: in the two-level tile head, only precision actions (REFINE_TILE: spawn/build/upgrade/cancel_boat/delete) refine down to the fine /8 grid. Boat, nuke, and warship targets stop at the coarse /16 pick, and _coarse_local_to_global maps that cell to its top-left /8 region - which the translator then treated as the whole target, searching just that 8x8-tile quadrant for a valid destination. Three quarters of every coarse cell was unreachable by boat, ever. Picks whose valid tiles sat in the other quadrants either snapped to nothing (translator-side wasted penalty, noop-equivalent) or forced the policy toward whatever cells happened to have targets in the lucky quadrant. The policy cannot learn its way around this: the information that a cell's target sits in the wrong quadrant isn't in the action space.

Fix (translator-only, no tensor shapes touched): region_tile gains a span parameter; boat/nuke/warship snaps search the full 2x2 region block (span=2), matching the /16 granularity the head actually decided at. Refine actions keep exact /8 snapping. Bonus fix while in there: boat_tile now prefers shoreline candidates (the engine resolves destinations via targetTransportTile → closestShore(owner, dst, 50), so a shore pick always resolves; inland only works when the owner has shore within 50 tiles by land), falling back to any valid tile. Fewer boats die at canBuild. The remaining engine-side reject (water-component reachability) stays priced by the wasted-intent penalty, and translator-failed snaps were already counted as wasted in vec.py - that incentive hole was closed in v4.2 and stayed closed.

Deploy: v4.2-style hot patch - repo synced on the pod, trainer killed, relaunched by the crash loop with --resume from the last checkpoint. Watch items: episode/wasted falling, boat share of the action mix normalizing, stage-1 roll-win resuming its climb.

Jul 9 audit: the entropy controller was the real stage-1 killer

A systematic pass over the training paths after the boat fix found the actual reason stage-1 roll-win decayed to 0.00 - and it wasn't boats. The adaptive entropy-floor controller was pegged at its 30x cap and structurally unable to come down. The log tells it cleanly: at updates 17-22 the policy started winning stage 0 (roll-win 0.40 → 0.85), which naturally dropped entropy to 3.3-3.5 - just under the 3.5 floor. The controller read commitment as collapse and ratcheted the coefficient 1.3x/update, 0.013 → 0.298 in 13 updates. The huge bonus then held entropy at 3.7-3.9: above the floor, but below the floor×1.4 = 4.9 release threshold - a dead band the bonus itself kept entropy inside, so the scale never decayed. For 100+ updates the entropy term (~1.1) dwarfed the policy gradient (~0.01-0.03); the optimizer was actively keeping the policy random.

Root causes and fixes (all in rl/ppo.py): (1) dead band: the scale now decays slowly (/1.05 per update) anywhere above the floor, not just above floor×1.4; (2) cap 30 → 5 (ENT_SCALE_MAX), and a resumed scale is clamped to the new cap so the pegged run recovers on redeploy; (3) no startup grace for fresh runs: the spawn-heavy-rollout grace only applied to resumes - now unconditional; (4) floor 3.5 → 2.5: 3.5 was tuned for the pre-v7 single full-grid tile softmax. v7's two-level head is structurally lower-entropy (coarse has 1/4 the cells, refinement ≤ ln 4), and a winning stage-0 policy sat at 3.3-3.5 - the old floor actively fought commitment.

Also shipped in the same pass:

• Update-path throughput: _fine_to_coarse_mask ran a per-sample Python loop with a GPU-syncing nonzero() inside every evaluate() minibatch (~24k syncs/update; upd 70-100s vs roll 14s). Replaced with a batched scatter.
• act/evaluate logprob consistency: when the fine mask falls back (no legal fine cells anywhere), the sampled fine cell can live outside the sampled coarse cell; act() now scores the coarse factor of the cell evaluate() will reconstruct (identical in all non-degenerate cases). Verified by 200-trial sampled parity test.
• Per-action coarse content masks: new coarse_has_land/water obs keys prune coarse tile picks - boats/nukes can no longer point at pure-ocean /16 cells, warship moves at landlocked ones. Less doomed-pick space to learn away by penalty.
• Webbot parity: the browser translator got the same span-2 + shore-preference fix (openfront submodule), since master's webbot serves v7 exports.

Env fleet auto-sizing (same night): post-fix GPU util read 20-40% on the 4x H200s with CPU load 17/128 - the box was barely working. Rollout runs on a collector thread fully overlapped with the update, so with roll ~16s vs upd ~68s the 384 envs sat idle ~75% of every cycle. The trainer now writes a suggested_envs hint into state.json every update (current envs scaled by the EMA'd upd/roll ratio, damped to ≤2.5x per restart, capped at 8/core and 2048), and pod_train.sh ENVS=auto re-reads it at every relaunch - restarts are the only point the fleet can resize, so the loop self-tunes across them. MINIBATCH=auto scales with the fleet (ROLLOUT×ENVS/12) so optimizer steps/update stay constant and bigger fleets mean bigger kernels, not more per-step overhead. ppo_v7 relaunched at envs 768 / minibatch 2048 (cold default; history takes over from here).

The real update-time culprit (found while scaling): more envs alone made updates WORSE (68s → 160s at 768) - v7's grid_fine is a per-sample coverage crop whose shape drifts every step, and the update's exact-shape sub-batch grouping (fine for the old per-map grids) splintered each minibatch into hundreds of single-digit sub-batches: the update was kernel-launch overhead, not compute. Fine dims now bucket up to multiples of 8 (collate zero-pads within the bucket, policy masks padded cells): >20x fewer sub-batches for bounded padding waste. A latent np.split abort also surfaced (auto minibatch rarely divides B_total evenly) - now array_split. Result at envs 1024 / minibatch 2730: update 68s → ~33s, game-ticks/s 2,650 → ~10,800 (4x), GPU util 20-40% → 36-72% at 380-460W. Rollout (~48s) is now the long pole; the auto-sizer will trim the fleet toward balance on future restarts.

BC-regen ops saga: three infra bugs before the first game replayed

Re-replaying the human archive (303 games, three engine-commit buckets) to backfill the new BC fields should have been routine - the same datagen/replay_all.sh pipeline as the v6 label regen. Getting it running cleanly surfaced three unrelated environment bugs:

• Submodule remote mismatch. .gitmodules points the openfront submodule at the djmango/OpenFrontIO fork, but the working repo's local origin had long since been repointed at the upstream openfrontio/OpenFrontIO - and at least one archived game's engine commit only exists on upstream's v32 branch, not the fork. A fresh clone (on the homelab) follows .gitmodules literally, so git checkout <historical-commit> failed with "pathspec did not match any file(s)", silently leaving the submodule on the wrong commit - the replay then ran the wrong engine version against an old game and desynced at tick 90. Fix: git remote set-url origin to the upstream URL on every fresh checkout of this repo, matching what the primary dev machine already runs. Worth eventually reconciling .gitmodules itself so new clones don't repeat this.
• macOS AppleDouble junk survives tar transfer. Copying the untracked webbot/ directories (needed because the submodule's working tree loses them on every historical checkout) via tar/scp from a Mac carried ._* resource-fork shadow files into the Linux submodule tree - harmless to the TypeScript build but noisy in git status; swept with a plain find -name '._*' -delete.
• The homelab's login shell is fish, not bash. Bash heredocs and && chains piped through a bare ssh host 'command' run under the remote login shell, so anything beyond trivial one-liners needs ssh host bash -s <<'EOF' ... EOF (stdin piped to an explicit bash) instead - the more common ssh host bash -c "..." also breaks past a couple of nesting levels of quoting once tmux is in the mix.

Earlier in the session, before settling on the homelab, the same job was attempted in an isolated git worktree on the primary dev machine to avoid touching the main repo's in-progress submodule state; that surfaced a fourth, homelab-irrelevant issue - the local Shell tool's background-process backgrounding has a hard ~15-18 minute lifetime independent of &/disown, which doesn't matter once the job runs inside a detached tmux session on a persistent remote host instead.

Once past those three bugs: a single-game validation replay passed clean (hash-verified, 10,751 ticks), and the full backfill launched in a tmux session on the homelab (20 vCPU, 47GB RAM - replay_all.sh defaults its worker count to nproc, one game per worker). Steady-state load ~18-25/20 cores. In progress as of this entry: bucket 1 of 3, single-digit desync failures expected and tolerated (the pipeline skips a failed game rather than blocking the batch) - one seen so far out of the first ~16 games, consistent with the occasional non-deterministic edge case rather than a systemic issue.

Found two GPU pods already running; kept one

Checking readiness for the BC/PPO retrain surfaced openfront-bc3 (1x A100) and openfront-rl5 (4x H100, ppo_v61) both still billing. bc3 was mid-restart into a torch.compile / EDQUOT crash loop on the stale bc_v6 run - already low-value (flat learning curve, see BC verdict) and about to be obsoleted by the v7 shape change regardless, so it was terminated after archiving bc_v6/ + publishing the playable v6_bc prior. rl5 is mid-flight on real progress (ppo_v61, update ~976, stage 4) and shares no state with the v7 retrain, so it stays running untouched - stopping a live, valuable, unrelated run to save ~$12/hr would be the wrong trade. General lesson for this kind of work: check what's already running and billing before assuming a clean slate - the pod list doesn't show up in any code diff.

Remaining before ppo_v7 can start (amended late Jul 8 for the merged scope + BC moratorium): the path no longer runs through BC data at all. In order: (1) implement the foveated two-stream obs (coarse /16 stream, fine-stream coverage mask, two-level tile head, bridge serialization); (2) run the AE /16 fine-tune off d8c32 and pass its quality gate; (3) push the combined v7 code to origin/master - push still deliberately held back, because pod_train.sh/pod_bc.sh bootstrap with git reset --hard origin/master and rl5's next restart would load v7 shapes against a v6.1 checkpoint and crash; the push therefore coincides with deciding rl5's fate (freeze ppo_v61 or pin its pod to the pre-v7 commit); (4) launch ppo_v7 from scratch, no BC warm start. The homelab archive re-replay keeps running to completion (the refreshed maps//bc/ sidecars get uploaded to djmango/openfront-human-games for archival/eval value), but prefeaturize + cache-bc/ are deferred under the moratorium and no longer gate anything. Until the push: local master stays at v7 for the homelab regen; rl5 stays on the last pushed commit.

GPU utilization push: 20% → 72%, then killed to move the hot path to Rust

Starting point: ppo_v7 on the 4x H200 pod was pegged at 20-40% GPU (140-195W of a ~700W card) with only 384 envs and CPU load ~13/128. The rollout/update loop is sequential and only the update touches the GPU heavily, so an idle CPU phase (rollout) was starving the accelerator every cycle.

What actually moved the needle, in order:

1. Env fleet auto-sizing. Rollout runs on a collector thread fully overlapped with the update; envs idle whenever roll < upd. The trainer now writes a suggested_envs hint into state.json every update (EMA'd upd/roll ratio, damped ≤2.5x per restart, capped 8/core), and pod_train.sh ENVS=auto/MINIBATCH=auto adopt it at every crash-loop restart.
2. Sub-batch bucketing. Scaling envs alone made updates worse (68s → 160s): v7's grid_fine is a per-sample coverage crop whose shape drifts every step, and the update's exact-shape grouping splintered each minibatch into hundreds of single-digit sub-batches - pure kernel-launch overhead. Bucketing fine dims to multiples of 8 cut that >20x.
3. Crash-loop unhang. A rank that dies by signal (SIGSEGV/SIGABRT) left orphaned env worker processes holding the log pipe open, so the restart loop's tee never saw EOF and the run sat dead in "restarting" indefinitely - this is what actually ended the session (see below). Fixed with a direct redirect + explicit reap, but not in time to save this run.
4. Batched coverage/content masks. Stack-sampled the main rank's Python stacks (RunPod drops CAP_SYS_PTRACE, so py-spy can't attach; used faulthandler + SIGUSR1 instead) and found _fine_coverage running .any()/nonzero()/int() per sample - 5+ GPU syncs and 3 host transfers each, ~2.5k syncs per act batch at 256 envs/rank. Rewrote as one batched kernel per op plus a host-side box search, verified bit-identical against the old path.

Result: envs 384 → 1024, update time 68s → ~25-30s, game-ticks/s 2,650 → ~10,000-13,000 (4-5x), GPU utilization 20-40% → peaks of 72% (avg still 12-50%, noisy across the 4 ranks). Never reached the 90% target.

Why it plateaued below 90%, and why that's a dead end in Python: even after killing every measured Python-side sync, the rollout half of the loop is still bound by VecEnv's Node.js game-engine subprocesses talking to the trainer over pipes - that's wall-clock the GPU cannot touch no matter how the featurizer or update batching are tuned. Every fix so far has been "find the next accidental sync/serialization in the Python glue"; there's no more slack to cut on that side, and each fix has been getting smaller (auto-sizing and bucketing were 2-4x each, batched coverage was incremental). The single-GPU control run (NPROC=1) that was supposed to establish the true non-DDP baseline never got readings before the pod's tmux session died with the rank's SIGSEGV and the crash loop hung (bug #3 above) - so the DDP-vs-single-GPU question is still open, but at this point it's moot: the environment step/serialize path itself, not DDP or the featurizer, is the actual ceiling, and that only moves by rewriting rl/vec.py's engine bridge (currently Node.js subprocesses + JSON/pipe IPC) in Rust with a real shared-memory or FFI hot path instead of pipes. This is the plan going forward - see the game engine's existing rust/ofrs native-path precedent (escaping the GIL) - rather than continuing to squeeze the Python side.

Run status: ppo_v7 killed at update 355 (stage 1, step 5.32M, roll-win 0.25). Final checkpoint + state.json synced to djmango/openfront-rl on HF before teardown - resumable once the Rust-backed vec env lands. The 4x H200 pod is being torn down; no GPU spend while the port is in progress.

July 9th-10th, 2026 - the Rust port: trainer, native engine, and the utilization ceiling that moved

PPO trainer ported to Rust/tch, then the game engine itself went native in-process. IPC is dead. Sustained GPU util 55% → 82% on a single A100, engine ticks 12-14k → 130-150k/s standalone. And one honest learning-signal finding that matters more than any of the speed numbers.

The trainer port (rust/oftrain, tch-rs/libtorch)

Full port of the PPO training path (rl/ppo.py + rl/vec.py + rl/obs.py + curriculum + translator + policy) to a Rust cargo workspace: ofcore (Python-free featurization/curriculum/translate), oftrain (bridge client, threaded vec-env - one OS thread per env, no GIL so no multiprocessing/pickle - tch policy net, PPO loop, checkpointing, nvidia-smi-polling util sampler). BC warm-start and AE training explicitly out of scope; policy sees raw pooled grid planes instead of an AE latent for now.

Two libtorch linking footguns worth never re-deriving: (1) tch 0.24 is hard-pinned to torch 2.11.0 - and not just by version string: its C++ shim calls ATen ops that don't exist in older torch headers, so it fails to compile (not link) against a pod's stock torch 2.8. Fix both locally and on pods: dedicated venv with pinned torch==2.11.0 (CUDA pods: +cu128 wheel), LIBTORCH pointed at it. (2) The big one: CUDA silently not linked. Nothing in our Rust references a torch_cuda symbol directly (kernels self-register via static initializers), so the linker's --as-needed drops libtorch_cuda.so entirely - Device::Cuda(0) "works" and then every real op panics. Known upstream (tch-rs #907/#1015). Fix: oftrain/build.rs re-asserts -Wl,--no-as-needed -ltorch_cuda -lc10_cuda from the binary crate; readelf -d to verify. Plus the returning NVRTC cameo: lgamma (Beta quantity head) needs libnvrtc-builtins from the separate nvidia pip package on LD_LIBRARY_PATH.

Rounds 1-3 on RunPod: the utilization number that lied, then the one that didn't

First A100 runs looked great - "98-100% GPU util" - until scaling to 4 and 8 GPUs exposed that those were instantaneous snapshots at the logging moment, not time-averaged utilization. The honest metric (min_mean_util, cumulative time-average across the run) said 36-41%: the GPU burst to 100% during the update and sat idle through the whole CPU/IPC-bound rollout. Same trap the Python ppo_v7 push fell into from the other side. Fixes that actually moved it: pipelined actor/learner (collect rollout N+1 on background threads with a frozen actor copy while the learner trains on N - classic one-step-lag IMPALA-style overlap, done with scoped threads and disjoint actor/learner struct borrows), GPU-resident minibatching (build full batch tensors once, index_select on-device per minibatch), flattened gradient sync for multi-GPU (manual replica averaging, "poor man's DDP" - 4-GPU scaled ~5x over 1-GPU). Result: 66-68% sustained, 330 → 560 decisions/s, and a measured dead end: fwdbwd was ~1s per 512-sample minibatch of genuine saturated compute, and the rollout side was pinned to the wall-clock of Node.js subprocesses talking JSON over pipes - the exact same ceiling Python hit. One instrumentation bug worth remembering: timing both phases from the same join point made collect_s == train_s on every line - time the inner work directly, and let the join cost land on the phase that actually waits.

Going native: the engine moves in-process

Instead of optimizing the pipe protocol (shared memory/FFI to the same Node engine - real work, throwaway destination), the call was to go all-in on the native Rust engine port (rust/engine, the rust-ofrs-fast workstream): the environment becomes a plain Rust object (RlSession mirroring bridge/env.ts's reset/step/countWasted), and oftrain grows a GameEngine trait with two backends - Node (the old bridge, kept for A/B) and Native (in-process, zero IPC). Porting the RL surface meant porting the full observation head (obsCore.ts entities/legality field-for-field), the TS percentage-based FFA win check (80% of total map tiles, checked every 10 ticks - not last-alive), missing intent executions (targetPlayer), and spawn-phase termination for singleplayer. One Rust wart: Game held a raw pointer for the execution list, making it !Send; documented unsafe impl Send after confirming single-thread ownership per session.

The outcome-parity campaign: hash parity was the wrong bar

Exact hash parity (every tick, every record, bit-identical) was costing more than it bought for RL purposes - what training needs is outcome parity: same winner, similar terminal conditions. New gate: replay all 78 archived human-game records through both engines, compare winner identity, terminal tick (±20%), winner land share (±10%); target 55/78 (70%). The TS side became a cached oracle (one-time expensive TS replay per record, resumable per-record shards, parallel child processes, fingerprinted by record-set hash + parity commit) so iterating on the native side never re-pays the TS cost. Then a long divergence-fixing grind, one root cause per commit, ~19 landed: win-condition semantics (FFA ordering, team aggregation, timers, bot suppression), random-boat target filtering and troop caps, nation defense-post sampling bounds, alliance-triggered attack retreats returning all troops, relation clamping to TS bounds + per-tick decay toward neutral + cross-alliance-only boosts, reciprocal alliance requests canceling in-flight nukes, manual boat retreat execution, island invasion target selection, city rail-connectivity placement scoring, and the warship suite (construction, patrol, shell targeting/flight, damaged/docked healing lifecycle, trade piracy + post-piracy patrol replan). The full-78 gate run is grinding in a pinned worktree as of this writing; per-cluster spot checks pass.

Native hot path: 4.9x single-thread, ~10x the old IPC ceiling

Standalone benchmark (rust/engine/examples/bench_rl_session.rs, no policy, no IPC): raw RlSession stepping went 9.3k → 45k ticks/s single-threaded, 51-58k → 130-152k ticks/s across 8 threads. The embarrassing hot spot: a scalar per-tile loop re-encoding the full tile plane to bytes on every step, eaten immediately by a decode back to u16 on the trainer side - ~78% of step time paying for a serialization format whose only consumer no longer exists. Fixes: memcpy-fast-path encoding (unit-tested byte-identical), Value::Array built straight from iterators (no intermediate Vecs), and a zero-copy tile_state() -> &[u16] accessor so the native backend skips bytes entirely. Correctness pinned by bit-identical obs JSON on fixed seeds before/after, and parity tests failing identically (same tick, same hash) on base vs. optimized - pre-existing oracle staleness, not regression. A rayon batched-stepping path exists behind a parallel feature but plain thread-sharding measured 20-40% faster - memory-bandwidth-bound, not scheduler-bound.

A100 validation: 82% sustained, and the law the sweep taught us

Fresh 1x A100 SXM pod, --engine native, 64 envs x 32 rollout, full policy: 81-82% sustained time-averaged utilization (the honest metric), flat through a 150-update / ~30min stability run, zero crashes, ~173 steps/s, collect_s ≈ train_s - collection and training genuinely overlapped and balanced. Every config swept told the same story from a different angle: 96 envs → 76-78% (collection scales linearly, no fixed overhead to amortize); 128 envs + foveation → 58-64% despite fitting more envs (cheaper obs = less GPU work = balance tips back to collection-bound); 192 envs + small policy (GC=128/BLOCKS=2) → highest raw throughput of anything tried (~285 steps/s) at the lowest utilization (37%); 256 envs full-res → OOM (the 250x150x63 obs batch alone is ~77GB at that scale - real ceiling, not fragmentation). The law: in this pipelined architecture, utilization tracks the ratio of GPU-bound train compute to CPU-bound collect compute - anything that makes the network cheaper buys throughput and costs utilization. Chasing the util percentage with a bigger network would be goodharting; decisions/s/$ is the metric that matters, and the small-policy config wins it by a mile if it learns as well per sample. Also landed flag-gated and GPU-verified: --amp (manual bf16 - tch has no bf16 autocast; halves activation memory, needs a clean same-minibatch A/B for speed), --foveate (real 48x48 crop centered on own-tile centroid, fully vectorized coordinate math, two broadcasting bugs caught by a dedicated unit test), --gc/--blocks, --pinned-h2d. Confirmed unreachable in tch-rs 0.24 and correctly skipped, not hacked: channels-last, fused/foreach optimizer, CUDA graphs.

The finding that outranks the speed work: stage 0 never wins

New OFTRAIN_DEBUG_EPISODES diagnostic dumped per-episode results and exposed that every episode in every config runs to tick-limit truncation: place 1/2, ~50 tiles, won=false, so the curriculum win-window never fills and training is permanently stuck at stage 0. Not an engine bug and not a PPO-sign bug (win-check math, entropy bonus sign, grad clipping all audited correct, hyperparameters match rl/ppo.py): the FFA win condition requires 80% of all map tiles, and the policy plateaus around half the map, stops expanding into neutral land, and its entropy collapses from ~7 to <0.05 within 2-5 updates on this low-variance, nearly-solved 2-player stage - then never recovers across 150 updates. Inherited characteristic of the ported curriculum/reward design (ent_coef=0.01 is rl/ppo.py's own default), but at Rust speeds it's now the blocker: the pipeline can push samples 10x faster at a policy that isn't going anywhere. Fix candidates: higher/slower-annealing entropy coefficient for early stages, reward shaping that keeps paying for expansion into neutral land after the opponent is matched, or stage-0 win semantics that are actually reachable (e.g. relative-strength or opponent-elimination based).

Jul 10, ~2am: the stall investigated and root-caused - three bugs, none of them the theory

The "curriculum/exploration tuning problem" framing above didn't survive contact with one question: the Python trainer won stage 0 on this exact curriculum, so what's actually different? A node-vs-native A/B (same Rust trainer, 12 CPU updates each, per-episode diagnostics on) plus a line-by-line diff against rl/ppo.py found three concrete divergences, in ascending order of embarrassment:

1. Episode budget: 3000 ticks vs Python's 15000. The Rust CLI default silently shipped at 3000 max_episode_ticks; Python's is 15000. Every "plateaued at half the map" episode was truncated at a fifth of the budget the Python wins were earned under. The 80%-ownership win was never reachable, entirely independent of policy quality. Fixed: default now 15000.
2. The entropy-floor controller was never ported. rl/ppo.py --ent-floor (default 2.5, on by default!) scales the entropy coef up 1.3x/update (cap 5x) whenever measured entropy dips below the floor, with hysteresis and startup grace. The Rust port had only the linear anneal - so the collapse to ~0 nats that Python's controller exists specifically to catch just... ran to completion, every run. Fixed: controller ported faithfully (including the "decay anywhere above the floor" rule that keeps a transient spike from trapping the scale), driven by minibatch-mean entropy like Python rather than a last-minibatch snapshot.
3. The native engine never left the spawn phase. The big one, and the answer to "is it our bridge?": in TS singleplayer, GameRunner adds no spawn timer - the spawn phase ends when SpawnExecution sees the human place a spawn (endSpawnPhase()). The native SpawnExecution port dropped exactly that call (the replay path has a timer, so replays/parity never noticed), so every native RL episode spent its entire life in the spawn phase: legality returned {actions: {}} (policy entropy pinned at ln-uniform ~6.9 over everything, Beta quantity head literally never active), vecenv's random-spawn fallback fired every step (which is also why native episodes burned ~2x ticks per decision), and no real game - no PlayerExecutions, no win check - ever ran. The tell in retrospect: node-engine logs showed ent≈2-3.8, entq≈-0.1 while native showed ent=6.916, entq=+0.000 from the very first update. Every native-engine GPU number from yesterday (the 82% included) was measured on spawn-phase noise. The throughput conclusions survive (the compute mix barely changes); the learning conclusions were never about learning at all. Fixed: execute_player_spawn reports success and singleplayer+human spawns end the phase, matching TS.

Post-fix A/B, same 12-update CPU config: native now behaves like node in kind - entropy 3.2 vs 3.8, quantity head active on both (entq -0.33 vs -0.10), episodes at the same tick cadence, mean reward 13.9 vs 10.9 and 1st-place 17/24 vs 10/24 (native's nations look somewhat softer than node's - the outcome gate, still grinding through all 78 records in its pinned worktree, is the instrument that quantifies exactly this). Neither leg wins inside 2000-tick test episodes with 12 updates of a tiny CPU net - wins need the real budget and a real training run, which is the very next thing to spend a pod on. Engine unit tests: identical pass/fail (47/35, all failures pre-existing missing-fixture paths) before and after the spawn fix.

Lesson worth writing down: all three bugs were silent defaults and omissions in a port, not logic errors - the code that existed was correct, the problem was code and config that didn't exist. Port audits should diff the flag surface and controller inventory against the original, not just the math that made it across.

Why the gate took 4 hours (it shouldn't have) and what the numbers actually mean

outcome_gate was strictly sequential with zero progress output until the whole 78-record report printed at the end - a run silently ran for ~4 hours with no way to tell if it was almost done or hung, and there was no per-record timeout at all. Root cause wasn't the engine (still 130-150k ticks/s standalone) - these 78 records are real archived multiplayer games, uniformly 400 bots + ~125 human players each (every single one - checked the full set), individually taking 100-1150s to fully replay tick-by-tick. Fixed: outcome_gate now runs records across a worker-thread pool (--jobs, defaults to core count) with a per-record wall-clock cap via a channel recv_timeout (--record-timeout-seconds) and streams N/total record -> category (Ns) progress to stderr as each one finishes. Also uploaded the cached TS oracle to djmango/openfront-human-games on HF (outcome-oracle/<commit>.json) and wired an automatic fetch into run_outcome_gate.sh - the ~1hr+ TS-engine regeneration only has to happen once, ever, per parity commit, not per machine.

The result, and why it's not the alarm it looks like: every record checked so far is missing_winner - not "close but outside tolerance," wildly different final states. One example (Two Lakes, Team mode, 400 bots): TS's real game ended at tick 7231 with Team Red at 95.05% land share; replaying the exact same human intents natively at the same tick leaves Red at 14.8%, with unaffiliated "Bot" entities alone holding 36.4% of the map - a different game, not a near-miss. But every one of the 78 records uses 400 bots, and the RL curriculum never goes above 150 (stages run 0, 0, 5, 10, 30, 30, 50, 80, 80, 120, 150). Bot AI acts autonomously each tick (it's not in the recorded intent stream), so any small per-bot decision difference between native and TS compounds every tick across however many bots are alive - at 400 independent agents over 5,000-13,000 ticks that compounds into a completely different map, while at the curriculum's actual bot counts the same per-bot error has far less surface area to snowball on. This test set answers "does native match TS at the most extreme bot density OpenFront supports," not "does native match TS at the bot counts the curriculum actually trains on" - it's the wrong instrument for the question that matters for training, and a 0%-ish pass rate on it shouldn't be read as "training is broken."

What it does establish, cleanly: bot AI has a real, likely systemic behavioral gap between native and TS (worth its own investigation - see next steps), and the existing 78-record human- game archive can't validate curriculum-relevant parity because it has no low-bot-count games at all. Both are now tracked as their own next-step items rather than blocking anything.

Jul 10, afternoon: all six next steps, done in parallel - the stall is fixed, native is default, here's what's real and what's not

Two subagents (curriculum-representative parity + bot-AI tick-level investigation), three A100 pods (stall-fix validation, model-size A/B, clean AMP A/B), one 4-GPU sanity check. Every item landed real evidence, not vibes.

1-2. Curriculum-representative parity + bot-AI root cause (done together, they compound)

First pass at curriculum-bot-count parity (0/5/10/30/50/80/120/150, `datagen/gen_curriculum_parity.ts`, `scripts/run_curriculum_parity_gate.sh`) used the same 4500-tick/6-min-timer budget as the archive and landed 22/40 (55%) with every failure a narrow-margin wrong_winner in a crowded field, not a systemic miss - a completely different failure mode from the 400-bot catastrophe. But that generator had every game hit the exact same max_timer cutoff at tick ~3910 regardless of bot count - i.e. every "outcome" was a coin-flip snapshot of an unfinished race, not a genuine win. Reran with a much longer horizon (20000 ticks / 40min timer) so games can actually resolve: 0/5/10 bots (the curriculum's first 5-6 stages) now hit real, complete dominance in both engines (land shares 0.93-1.0, not snapshots) - 0 bots 2/2 exact match, 5/10 bots near-misses where both engines agree on a genuine winner and near-identical land share, just occasionally disagreeing on which nation among several that converge to ~100% simultaneously. 30+ bots still shows real winner-identity divergence even at full dominance.

In parallel, a tick-level side-by-side trace (new `tick_dump` binary + TS dump script) at ~50 bots found the actual bug: first divergence appeared at tick 303 - two nations with byte-identical troop counts landed 1 tile apart, pointing at conquest-order/PRNG mechanics rather than a formula error. Root cause: GameMap::for_each_neighbor4 iterated neighbors west/east/north/south instead of TS's north/south/west/east - silently shifting which specific tile received which PRNG draw during every border-tile conquest-priority computation, in every attack, everywhere in the engine, for as long as it's existed. Fixed, plus two related missing is_impassable filters in attack/bot-AI decision code (one of which was a real invariant violation - impassable tiles could become "owned"). Effect: first divergence moved from tick ~303 to ~1497 (5x later, and everything in between is now byte-identical), total tile-attribution error at tick 6000 dropped 46% (756k -> 410k). This is now on master and is what the parity numbers above already reflect.

Bottom line: curriculum-relevant parity (bots 0-10, stages 0-3ish) is good - real wins, right magnitude, occasional coin-flip on exact winner identity in tight races. Higher bot counts (30+) still have a real, if much smaller than before, systemic gap worth continued attention as training reaches those stages, but nothing here blocks starting or continuing training now.

3. Stall-fix validation - confirmed, end to end, on a real A100

Single-GPU pod, --engine native, full 15000-tick budget, entropy floor on, all three 2026-07-10 fixes in place. Result: entropy holds 2-5 nats the whole run (entropy-floor controller visibly engaging - ecoef climbs to 0.02-0.05 when entropy dips, decays back down otherwise), the quantity head stays active (entq nonzero throughout, never pinned at 0.000), win rate climbs into the 80-100% range in bursts, and the curriculum advanced twice - stage 0 (Onion, 0 bots) → stage 1 → stage 2 (Onion+Pangaea, 5 bots) - within ~2200s / 166 updates. This is the thing that was completely impossible before yesterday's fixes (every episode timed out at 1/5 budget with zero win-rate signal, forever). The run stopped cleanly sometime after update 166 with no error in the log and the GPU idle/released - not a crash we could find a cause for, but not before it had already proven the thing we needed proven.

4. Model size re-asked with learning curves, not just throughput

Full policy (11.2M params, GC=256/BLOCKS=4) vs small (2.6M params, GC=128/BLOCKS=2), identical 64-env/rollout-32 config, run in parallel on separate pods for the same wall-clock. Small is genuinely faster per step (~280-380 steps/s vs full's ~150-160, matching yesterday's throughput numbers) but its learning curve was rockier: it dipped to a 0% win-rate window partway through (full policy dipped too, recovered faster) before eventually also reaching stage 2. Both networks are capable of learning this task; full policy got there first and more smoothly at this wall-clock budget despite the throughput disadvantage. Verdict: not the clean "smaller = more learning/hour" win hypothesized - model size trades throughput for stability/sample-efficiency here, and the net effect on learning-per-hour needs a longer, cleaner run (both to full convergence, same seed) to call definitively. Foveation's case is similarly still open - same throughput-vs-capacity tradeoff logic applies.

5. Clean AMP A/B

Same config, same minibatch count (8), only --amp flipped, 8 updates each, on a fresh pod so nothing else could confound it. fwdbwd_s: 0.505s -> 0.318s per minibatch (37% faster), steps/s 154.9 -> 217.3 (+40%), no NaN, losses in the same healthy range as the non-AMP run. This is a clean, real win - worth turning on by default for future training once the model-size question above is settled (AMP's memory savings also directly relax the env-count ceiling that caused the 256-env OOM on 2026-07-09).

6. Native the default; multi-GPU confirmed working

--engine native is now the default (--engine node still available as the parity-testing fallback) - justified by items 1-3 above: curriculum-relevant parity is real, the stall is fixed, and native is still ~10x faster ticking regardless. Also ran a 4-GPU sanity check (256 total envs, native engine, all fixes in place): shards scale correctly across all 4 GPUs (81-100% per-GPU during compute), ~520-550 steps/s (~3.4x single-GPU), 64% sustained min_mean_util (lower than single-GPU's 76-82%, matching the known per-shard-thread-overhead pattern from earlier multi-GPU work), and it also advanced the curriculum (stage 0 -> 1) before being torn down. Nothing broke moving to multi-GPU with today's changes.

Remaining next steps, in order of expected value

1. Kick off a real training run. Everything that was blocking one is resolved: native is default, the stall is fixed and validated, parity is good at the bot counts that matter early, AMP is a clean win. Turn on --amp, pick full policy (safer choice per the model-size A/B above) or small policy (if willing to accept rockier-but-faster), and let it run for real instead of 150-400-update smoke tests - single GPU first, watch the curriculum advance past stage 2, *then* scale.
2. Settle model size / foveation with a longer, cleaner A/B. Today's runs showed both sizes can learn but weren't run to convergence or matched by seed - re-run full vs small (and foveated) to the same wall-clock or update budget with a fixed seed and compare updates-to-stage-N, not just steps/s.
3. Investigate the pod1 validation run's silent stop (stopped cleanly after update 166, no error in the log, GPU released) - probably benign (pod inactivity policy, disconnect) but worth confirming before relying on multi-hour unattended runs.
4. Keep chipping at bot-AI parity for 30+ bots as training reaches those curriculum stages - the neighbor-order fix closed most of the gap but a smaller residual (troop-level, first appearing ~tick 1498 in the tick-level trace) remains undiagnosed.
5. Profile the collection phase now that engine ticking is ~10x too fast to be the bottleneck: collection scaled linearly with env count, so the cost is per-step obs decode/featurize/act - the next 76→90%+ util move is either actor/learner CUDA stream separation or moving featurization onto the GPU, not more engine speed.
6. Scale to 8 GPUs once a real run is underway and stable on 1-4 - the plumbing is confirmed working (this session's 4-GPU sanity check advanced the curriculum too), just hasn't been pushed further yet.

Jul 10, 2026 - ppo_v8 launch plan (8x A100, written now, not started)

Everything upstream of "press go" is validated: native engine, stall fixes, curriculum-relevant parity, AMP. This is the concrete runbook for the first real (not smoke-test) run, sized for 8 GPUs. Explicitly not executed yet - writing it down first so launch is a checklist, not a series of decisions made live on an $12/hr clock.

Prerequisites - done as of Jul 10, afternoon

1. Checkpoint resume: implemented. --resume <path> loads the VarStore weights plus a <ckpt>.state.json sidecar (curriculum stage, entropy-floor scale, learning rate, total env steps, win-rate window, update counter) written atomically (tmp+rename, matching rl/ppo.py's policy.pt.tmp pattern) alongside every checkpoint. Every save also refreshes a fixed-name latest.ot/.state.json pointer so a wrapper never has to parse filenames. Known, documented gap: AdamW's momentum/ variance state is not restored - tch-rs 0.24 exposes no optimizer state_dict save/load at all (checked the crate source directly) - so it rebuilds over the first few dozen post-resume updates. Deliberate, not silent.
   Smoke-tested end to end: short run → checkpoint → resume confirms the update counter and decisions_total continue (not reset to 0), and a synthetic mid-curriculum state (stage=2, ent_scale=3.5, lr_now=1.5e-4) injected into the sidecar correctly reaches the training loop (ecoef in the log reflects the resumed scale exactly: 0.01 × 3.5 = 0.035).
2. HF sync + restart-loop wrapper: implemented (scripts/pod_train_v8.sh, the Rust-flavored pod_train.sh equivalent) - bootstraps repo/toolchain/CUDA-libtorch-venv/ build, verifies libtorch_cuda.so is actually linked before spending GPU-hours, seeds --resume from a local-or-HF checkpoint, crash-loops with backoff, and background-syncs latest.ot/.state.json to djmango/openfront-rl every 10 minutes.
3. Found and fixed a real crash bug while smoke-testing resume - see below, this is likely what actually killed the pod1 validation run.
4. Model-size/foveation decision: still open (unchanged from the earlier next-steps list) - the plan below defaults to the safe choice (full policy, no foveation) pending a longer seed-matched A/B, exactly as before.

The bug that (probably) actually killed pod1's validation run

Testing --resume against a synthetic stage-2 state (maps=[Onion, Pangaea], matching exactly what pod1 had just advanced to before it went quiet) reproduced an immediate panic: obs 2 grid shape mismatch (v1 requires uniform batch shape). batch.rs's build_obs assumed every env in a batch shares one grid resolution - true only for single-map stages with no rehearsal draw. Every curriculum stage past stage 1 samples multiple maps of different native sizes, and even single-map early stages mix in differently-sized past-stage maps via rehearsal sampling - so a mixed-shape batch isn't a rare edge case, it's the guaranteed outcome of training past stage 1 for more than a few minutes. This fully explains why the run "stopped cleanly with no error" right after the stage 0 → 1 → 2 advance logged.

Fix: pad every item to the batch's max (gh, gw), top-left aligned, zero-fill. grid_valid (previously always an all-ones placeholder - the code comment already said so) is now 1 only over each item's real region. This needed no architecture change: policy.rs already multiplies grid_valid into every tile-legality mask and the foveation coverage math, so padded cells were already correctly excludable from action selection by design - only the batcher itself needed to stop assuming uniformity. Same-shape batches (still the common case for early stages) take an unchanged fast path. New regression test (mixed_shape_batch_pads_instead_of_panicking) pins down padding placement and the grid_valid mask precisely. Stress-tested with a real 20-update run at stage 3 (Pangaea+Caucasus, naturally sampling both) - no crash, no NaN.

A second finding while investigating: the sizing math only covers early stages

The "64 envs/GPU" ceiling from 2026-07-09 was established entirely on Onion (the smallest map, 512x512 native → 64x64 after /REGION alignment). Checked the other curriculum maps' actual dimensions: Asia is 2000x1200 → 250x150 - exactly GW_MAX/GH_MAX, and 9.2x more grid cells than Onion. Stage 6 onward (bots=50+) mixes in World/Asia; since the padding fix above makes batches size themselves to the largest map actually sampled, a batch containing even one Asia-scale episode costs ~9x the raw grid memory of an all-Onion batch - and the intermediate conv-tower activations (GC=256 channels vs 63 input channels) scale by the same spatial factor on top of a channel multiplier, so the realistic multiplier is higher than 9.2x, not lower. The 64-envs/GPU figure is only proven safe through roughly stage 5; expect to need fewer envs (or foveation, which caps the fine-grid cost regardless of map size) once training reaches stage 6+. This is exactly the kind of thing checkpoint resume now makes cheap to handle reactively: if stage 6+ OOMs, relaunch with a smaller --num-envs and it resumes from the last synced checkpoint with nothing lost - no need to get this perfectly right preemptively. Added as an explicit watch item in the monitoring section below.

Config decisions and why

Sizing for 8 GPUs

Single-GPU ceiling found 2026-07-09: 64 envs x 32 rollout (2048 samples/update) fits at 46% GPU memory and sustains 81-82% utilization; 256 envs at the same config OOMs (~77GB just for the observation batch tensor, before activations). AMP roughly halves activation memory but the raw observation-batch storage is the dominant cost and isn't shrunk by AMP alone, so don't scale envs up to compensate for AMP's savings without re-testing for OOM first - keep per-GPU envs at the proven-safe 64 for the initial launch.

The multi-GPU architecture is data-parallel replicas (each GPU shard runs its own independent copy of the policy + its own 64 envs, gradients averaged across shards every update - "poor man's DDP", no cross-GPU memory sharing), so per-GPU memory pressure at 8 GPUs is identical to 1 GPU. Confirmed working at 4 GPUs on 2026-07-10 (256 total envs, ~520-550 steps/s, 64% sustained util, curriculum advanced). Launch config:

--engine native --num-envs 64 --num-gpus 8 --rollout-len 32 --epochs 2 --minibatches 8 \
--amp --pinned-h2d --stage 0 --device cuda:0 --ckpt-every 50 --log-every 1 \
--resume <latest checkpoint if any>

= 512 total envs across 8 shards. Expect steps/s roughly 2x the 4-GPU number (~1000-1100) if scaling stays similarly sub-linear (4-GPU was ~3.4x single-GPU, not the full 4x - per-shard-thread overhead); sustained util likely similar-or-slightly-lower than 4-GPU's 64% (more shards = more overhead), which is fine per the established "utilization% isn't the metric, decisions/s/$ is" rule from 2026-07-09's sweep.

Launch runbook (once prereqs are done)

1. Provision: runpodctl create pod --gpuType "NVIDIA A100-SXM4-80GB" --gpuCount 8 --containerDiskSize 60 --volumeSize 200 --vcpu 64 --mem 500 --startSSH --ports "22/tcp" --secureCloud (bump vcpu/mem/volume vs the single/4-GPU pods used this week - 512 envs means ~512+ Rust OS threads plus whatever the native engine itself spawns per session).
2. Everything else - repo clone, Rust toolchain, the CUDA-linked libtorch venv (must be exactly torch 2.11.0 - tch 0.24's C++ shim calls ATen ops that don't exist in the pod's stock torch 2.8, fails to compile not just link), build, the readelf CUDA-link check, resume-seed from HF, and the crash-loop - is now one command:

   RUN_NAME=ppo_v8 NUM_GPUS=8 bash scripts/pod_train_v8.sh

   or as the pod's start command:

   bash -c "curl -fsSL https://raw.githubusercontent.com/djmango/openfront-ai/master/scripts/pod_train_v8.sh | RUN_NAME=ppo_v8 NUM_GPUS=8 bash"

3. Set EXTRA_ARGS to override the frozen config decisions above if deliberately deviating (default is --amp --pinned-h2d); OFTRAIN_DEBUG_EPISODES=1 isn't wired into the wrapper (it's a stderr firehose at 512 envs) - for the first stretch, tail /tmp/train_ppo_v8.log and run the binary manually with that env var set for a short supervised burst before trusting the unattended wrapper.

Monitoring, gates, and stop conditions

• Every checkpoint (--ckpt-every): confirm it actually synced to HF (the wrapper's background loop logs [hf-sync] synced latest checkpoint every HF_SYNC_INTERVAL_SECONDS) - a checkpoint that only exists on the pod's local disk isn't a safety net.
• Approaching stage 6 (bots=50, World/Asia enter the map pool): watch GPU memory% closely for an OOM as batches start sizing themselves to Asia-scale (~9x Onion's grid cells, see above) - if it OOMs, just relaunch with a smaller NUM_ENVS; the wrapper's crash-loop + resume means this is a config tweak and a restart, not a lost run.
• Curriculum stage index: the single clearest learning-progress signal - it should be monotonically increasing over hours, not stuck. Stuck for a long time at any one stage (no fixed threshold yet - use judgment, but "stuck" means many multiples of the 2026-07-10 stage-0->2 pace of ~2200s/166 updates on 1 GPU, adjusted for the 8-GPU throughput multiplier) is a stop-and-investigate signal, not a push-through signal - it was a real bug the last three times a stage looked stuck.
• Entropy (ent in the log): should stay in the low single digits, not collapse toward 0 and stay there for many updates (the pre-fix failure mode) - if it does despite the floor controller, that's a stop-and-investigate signal, not something to wait out.
• ecoef: watching it spike (entropy-floor engaging) and decay back down is healthy; pinned at the 5.0x cap for a long stretch means the floor controller is fighting a losing battle and something upstream (LR, clip range, reward scale) may need attention.
• min_mean_util / steps/s: sanity-check against the sizing section's expectations, but per the 2026-07-09 finding, don't chase a utilization number for its own sake - decisions/s and curriculum-stage progress are what matter.
• Cost: 8x A100 SXM ballparks ~$12/hr on RunPod (matches the 8-GPU pod pricing from 2026-07-09's session) - budget/time-box the run and check in on progress rather than leaving it fully unattended for the first day, until the resume/crash-recovery path has actually been proven under a real crash, not just the smoke test in the prerequisites.

Reference and open threads

Tooling, bugs, lessons, infra, recipes, backlog.

Visualization & tooling

Determinism is the superpower: the bridge mirrors the client's createGameRunner() init exactly (same PseudoRandom ID stream, no spawn timer), so agent games saved as GameRecords replay bit-identically in the real client, and verify_record.ts proves it by re-simulating from intents alone.

• rl/watch.py: one greedy episode → flat debug WebM + GameRecord + a .debug.json sidecar (every decision: action, probs, value).
• scripts/render_client_replay.py: headless Chromium replays the record in the actual game client → WebM with real graphics. Agent renders as AGENT with the gold spawn ring, crown when first, "You Won!" modal; camera starts fit to the whole map.
• scripts/serve_replay.py: archive-API shim so a normal browser replays agent records interactively.
• MODEL overlay (RlDebugOverlay.ts): in-client panel synced to the sim tick: chosen action, value, probability bars, recent-actions log. Works in rendered videos, browser replays, and live matches; auto-probes the agent's feed on localhost:8988, zero setup.
• rl/play.py + bridge/play.ts: the agent joins a real lobby over the live websocket protocol; you fight it while watching it think.
• patches/client-replay-tooling.patch: all client hooks as an uncommitted-submodule patch, auto-reapplied if the submodule resets.

Bugs

• Wins were never counted. vec.py checked winner[1] == "Agent" (username); the engine emits the clientID ("AGENTRL1"). Every "the agent never wins" observation before Jul 6 12:06 was partly measurement error. Found only by watching a replay where the agent visibly won while the stats said otherwise. Validate the reward path end-to-end with a game you can see.
• Case-insensitive macOS hid a Linux crash. Map dir lookup lowercased only the first char (BlackSea → blackSea); dirs are fully lowercase. Fine on Mac, ENOENT on Linux.
• Resume logic fought the supervisor. Supervisor passed a stale --stage 4; resume took max(cli, checkpoint), so a manually reset checkpoint kept snapping back, and a still-running trainer kept re-saving its old stage over the reset. Checkpoint stage is now authoritative; kill the whole process group before editing checkpoints.
• Cropped-map index crash. Transport/nuke target coords weren't bounds-checked against the 16px-cropped grid; a boat targeting BlackSea's trimmed edge strip crashed the obs builder.
• Phantom boats: "legal" actions that always no-op. The boat mask was just troops > 100; the engine additionally requires a free boat slot (max 3), an own shore to launch from, and a non-own/non-ally destination with a reachable shore - and silently discards the intent otherwise, with troops never deducted. A discarded boat was therefore reward-identical to noop but occasionally landed and grabbed land, so the policy-gradient math correctly learned to spam it: 43-78% of decisions in v4 stage 2-3 replays, p(noop) driven to 0.01. Same shape in expand (fizzles without a neutral border), build (61% of one v3 episode's decisions, snapped to random land the engine rejected), and nukes (silo cooldown/spawn immunity ignored by the mask). The bridge comment said "illegal tile picks become no-ops the policy learns to avoid" - but there was no signal to learn from: the failure and the noop pay identically. Fix: honest masks + valid-tile snapping + a per-discarded-intent penalty (Jul 7 timeline entry). Caught by computing the action mix from replay debug sidecars, not from any training curve.
• Headless-browser quirks. Vite binds IPv6 ::1 while headless Chromium resolves localhost to IPv4 (force --host 127.0.0.1); the client rejects software WebGL (needs --use-angle=metal --enable-gpu --ignore-gpu-blocklist).

Lessons

• Only compress what's big. The AE is for the map; exact small state rides the bypass. One decision dissolved four v2 loss-weight tuning runs the same evening.
• Detection beats regression on sparse targets. Structure counts under MSE collapse to zero; rarity-weighted BCE occupancy hits P/R 1.0.
• Make wins reachable before making them valuable. A win bonus is useless if the setting never produces wins. 1v1 → 1v3 staging turned the win signal from theoretical to dense.
• Warm starts inherit stale habits. ppo_v2c resumed v2b weights (trained under the old placement reward and pre-fix win counting) onto the new curriculum; ppo_v3 started from scratch on the same code. In one day the fresh run reached stage 4 at 60-70% wins while v2c stalled at stage 3 at 7-17% with entropy stuck near 8, exploring without committing. When the reward or curriculum changes materially, retrain; a checkpoint tuned for the old objective is a local optimum in the new one.
• Rehearse or forget. Revisit cleared maps at current difficulty for a fixed fraction of episodes.
• Exploit determinism everywhere. Bit-identical replay gives verification, visualization, and live mirroring from one mechanism.
• Pad to the batch, not the maximum. Most of a "GPU too slow" problem was wasted convolution.
• Watch the agent play. The replay tooling caught the win-detection bug and the camera problems; curves never would have.
• An action that can silently fail is a free lottery ticket. If a "legal" action's failure mode is exactly a noop (no cost, no signal) and its success mode has upside, the policy will rationally spam it forever and it will look like pathological behavior. Every action needs one of: an honest mask, or a price on the discarded intent. "The policy will learn to avoid illegal picks" is only true if avoiding them pays better than trying them.
• The loss curve lies; log the metric you care about. The mixed-retrain loss looked flat for hours while border accuracy was still climbing. Overall tile accuracy is 99% saturated and borders are a sliver of pixels. Border accuracy cost one line to log natively (the mask already existed for the loss weighting).
• Benchmark on the distribution you'll deploy on. The encoder looked done at 87% border accuracy until it met real human games and dropped 16 points. Bot data underrepresents exactly what humans do: allying, betraying, invading by sea.
• Spatial precision can't be bought with channels. +50% latent channels does far less for borders than halving the patch each latent vector must summarize. And don't make the latent re-encode static side-information (terrain) the downstream consumer already has.
• Treat training hosts as disposable. Checkpoint everything, sync to HF, restore on boot, supervise with auto-restart. Every ops incident so far violated some part of this.
• Auto-restart is not auto-recover. The temporal-BC supervisor relaunched an OOM-crashing trainer into the same wall for hours, and a separate pod ran stale code all day because its git pull had silently failed on an untracked file. A restart loop needs two cheap guards: back off (or change something, or alert) when the child dies instantly twice in a row, and assert the deployed commit matches origin/master at launch.
• Sequence models multiply memory before they multiply quality. A seq-8 BC window is 8 full observations through the frozen encoder per sample: activation memory scales with window length, so batch size must scale down by the same factor. Budget it before launch instead of discovering it as a 19.9 GiB allocation on a 24 GiB card.

Infrastructure

Homelab for datagen and some training; cloud GPUs for long PPO and AE runs. scripts/pod_train.sh supervises RL jobs (idempotent deploy, auto-restart). Datasets are prefeaturized once; checkpoints live on Hugging Face: openfront-snapshots, openfront-human-games (285 hash-verified replayed human games + raw intent records), openfront-tile-autoencoder. Ops scar tissue from the AE push, mostly self-inflicted: the homelab mixed run was host-OOM-killed at step 22.7k (CPU evals + HF upload + 6 DataLoader workers on one box); remote trainers hit ulimit -n with 16 workers memory-mapping 459 game caches; a billing outage restarted cloud machines mid-run (tmux gone; the interrupted ablations were relaunched from scratch since AdamW state isn't checkpointed, only weights). Git history was scrubbed once with git filter-repo after large blobs snuck in; data dirs are gitignored now.

v4 training recipe & decision gates

Three arms, two launched together (Jul 6 eve), the third conditional:

• Arm A - pure RL (ppo_v4): from scratch, stage 0. Baseline and control; also the only arm learning spawn placement without supervision.
• Arm B - BC (bc_v4): feedforward pilot, 60k steps on the cache-bc pipeline. At ~1.5ms/sample CPU the run is GPU-bound; verdict lands in hours, not days. Saves bc_best.pt on holdout improvement (acted-action + tile accuracy - the honest heads; plain action accuracy is inflated by forced-legality noops).
• Arm C - BC→RL (ppo_v4_bcinit, conditional): launched when B plateaus. rl/ppo.py --init loads the shared Policy weights and folds the winner-bucket conditioning into the head biases - verified exact (max output diff ~1e-7 across all heads). Same hypers/envs/pod class as A; judged at equal PPO wall-clock on the eval/win curves and time-to-stage-N.

Gates. B is learning iff holdout tile_region accuracy climbs and holdout loss falls (10k steps ≈ 1M samples ≈ 5x the entire old pilot's data; flat there = dead, kill the pod, A continues regardless). B has plateaued (launch C) when holdout metrics are flat across two consecutive eval windows (~10k steps) or the cosine tail begins. C's first ~30min will look bad while the untrained value head catches up - judge after the critic settles. If C's entropy climbs back toward ~7 in the first hour, the BC prior got washed out: add a KL-to-prior regularizer (AlphaStar-style) before rerunning.

Watch items. BC first log line should read several hundred ex/s (under ~100 = cache problem, stop). Late-curriculum OOM lever: MINIBATCH=64 env var (1/8 grids cost ~4x v3's conv activations at World/Asia). Rollout buffer RAM ~20GB at Asia-size grids with 96x32; halve --rollout if the pod is RAM-tight. Fixed-seed eval pauses training for minutes at late stages (~2% overhead at eval-every 300). Old v1 sidecars carry no spawn labels; re-replay with BC=1 REBC=1 datagen/replay_all.sh feeds the next BC run.

Deliberately skipped: BC value-head pretraining on placements (scale-mismatched with PPO returns, muddies A-vs-C), fp16 rollout buffers (RAM insurance we may not need), latent precompute to disk (at 1/8 a latent is ~2.4MB/snapshot ≈ 200GB corpus - the GPU-batched encode from the raw cache IS that idea, done right).

Open threads

• v5 wishlist (drafted Jul 6 night, from watching v4 launch day):
  • Memory. The policy is purely reactive - one snapshot per decision, no recurrence. It can't see trends (massing neighbors, receding attacks) or its own commitments (attack/retreat flip-flopping in the overlays). Add an LSTM between trunk and heads for PPO; hidden states ride the rollout buffer through resets.
  • BC temporal transformer, judged properly. The --seq BCPolicy exists but has only ever run at starvation throughput. With cache-bc feeding it, run seq=8 vs feedforward at equal samples on full holdout; if seq wins, that's the evidence that buys the PPO LSTM.
  • Reward MDP audit + asymmetric strength shaping. v4 games show the shape: hyper-aggressive expansion, big holdings, then collapse (s2 seed 7: 103k tiles at t8001, dead by t10711). W_DELTA treats gains and losses symmetrically; weight losses heavier than gains (loss-aversion shaping) so defense is learned before stage 4 punishes its absence. While in there: audit timeweight and the noop-heavy action mix (35% noop at stage 2) against the actual returns.
  • Async rollout/update overlap. Post-fix the phases are still sequential (~33s rollout with GPU half idle, ~48s update with 46 cores idle). Overlapping them is another ~1.5-2x. Same surgery should batch the two act-groups' encodes.
• PPO v4 wishlist: shipped (Jul 6 eve). Learned spawn placement (PPO + BC), d8c32 encoder at full 1/8 with the local owner-crop bypass, periodic fixed-seed greedy eval (eval/* curves), win-gate window persisted in the checkpoint, entropy anneal (0.01 → 0.002 over 4k updates) and stage LR warmdown (x0.85 per advance). v4 trains from scratch - the 1/8 grid, new action ("spawn"), and 43ch observation change the MDP and the tensor shapes anyway.
• Spawn supervision coverage: only sidecars regenerated with --bc --rebc (formatVersion 2) carry spawn labels; old v1 sidecars still train everything else. Re-replay the archive on a pod to unlock BC spawn supervision everywhere.
• BC warm start: does initializing PPO from human imitation beat the 1v1 curriculum bootstrap, or compound with it? Formalized Jul 8 as the BC moratorium's unfreeze experiment: v6-shape A/B, --init v6_bc vs from scratch, identical seeds/config. Until it runs and wins, no new BC runs.
• Warm-start refreshes: future same-architecture encoder updates on new data should resume from the current checkpoint (and checkpoint optimizer state, see billing-restart scar) instead of 40k fresh steps.
• Curriculum progression: with wins now measured correctly, watch whether stages clear; tune WIN_AT / WINDOW if stage 0 saturates or stalls.
• Remaining action heads: upgrade_structure, delete_unit, move_warship, cancel_attack/boat are schema'd but not all wired.
• Self-play league: past-policy opponents replacing scripted bots once the agent clears the bot curriculum.