All projects/case study

PokeGen3-RL

RL agent that plays Gen 3 Pokémon battles by reading emulator memory directly. A Lua bridge inside mGBA drives the game; a Gymnasium env handles masking, rewards and an overworld autopilot.

PythonPyTorchLuaGymnasiummGBA

Overview

A reinforcement-learning agent that plays Gen 3 (Emerald) battles by reading emulator memory directly. No screen pixels: the game state is read straight out of RAM and keypresses are driven by a Lua script inside mGBA.

Actions
18
Observation
431 floats
Battler slots
4 (doubles-ready)
Transport
TCP, line-based

The agent only ever plays the battle. Everything between fights (walking to grass, healing at a Pokémon Center, clicking through level-up and evolution screens) is scripted in the bridge, so training runs unattended for as long as the emulator stays up.

Architecture

Two processes talk over localhost. The Lua bridge owns the emulator; the Python env owns the learning loop.

process layout
  mGBA (Lua)                              Python
┌───────────────────────┐   :8888   ┌──────────────────────┐
│ battle_bridge.lua     │◄────────► │ env.py (Gymnasium)   │──► PPO
│  · RAM reads          │           │  · observation, mask │
│  · phase machine      │           │  · reward shaping    │
│  · key-press driver   │   :8889   └──────────────────────┘
│  · overworld nav      │◄───────── address finder (RAM dumps)
└───────────────────────┘
CommandWhat it does
GET_STATEOne comma-separated line: 4 battlers, both parties, party levels, weather, flags
STATUSCurrent phase and the result of the last task (OK / FAILED / ABORTED)
ACTION:nQueue move / switch n; replies ACK or REJECT:reason
THROW[:ball]Open the bag and throw a ball (wild, single battles only)
NAV:HUNT | HEALScripted overworld walk to encounter tiles or the Pokémon Center
INVENTORY / BOX / DEXRead-only catch observability: bag, PC, Pokédex owned bits
AUTO:ON | OFFKill switch (also the L button in the emulator)

The env never sleeps for a fixed time. The bridge derives a phase from callback2, the per-battler controller function and the battle outcome (NOT_IN_BATTLE, CHOOSE_ACTION, FORCED_SWITCH, BUSY, WAITING, BATTLE_OVER), and the env polls STATUS until the game is actually asking for input. Animations and text boxes are just WAITING.

Reading the game

emerald_config.lua is the single address map. Everything else reads through it, which is what made the address bugs in the deep dive cheap to fix.

SymbolAddressUsed for
gBattleMons[4]0x020240844 × 0x58 bytes: HP, PP, moves, types, status, stat stages, ability
gPlayerParty / gEnemyParty0x020244EC / 0x020247446 × 0x64 bytes, encrypted substructures
gBattleTypeFlags0x02022FECdouble / trainer bits
gBattleOutcome0x0202433A0 = ongoing; won, lost, caught, forfeited
gBattleWeather0x020243CCrain / sand / sun / hail masks
gMain.callback20x030022C4which screen the game is on
gBattlerControllerFuncs0x03005D60is the player waiting on action, move or target input

Party Pokémon store species, item and moves in four 12-byte substructures that are XOR-encrypted and shuffled by personality value. The bridge decrypts them in Lua, so the env gets the real moveset of every bench Pokémon and of the whole enemy party.

battle_bridge.lua · decryptPartySlot
local key   = pid ~ emu:read32(base + 4)     -- personality xor OT id
local order = block_orders[(pid % 24) + 1]   -- 24 possible substructure orders
-- growth block: species + item, attacks block: the four moves
local g0 = emu:read32(growthAddr) ~ key
species, item = g0 & 0xFFFF, g0 >> 16

Observation & actions

A flat 431-float vector and 18 discrete actions, with a legality mask so the policy never wastes a turn on something the game will refuse.

PartSizeContents
4 battlers4 × 47HP, PP, move id/type/power/accuracy/class/priority, effectiveness vs target, status, stat stages, species, ability
Weather1none / rain / sand / sun / hail
Bench12 × 19both parties: HP, species, item, move types, effectiveness vs current enemy, power, accuracy
Extras14party levels, level gap, min/mean level, balls left, wild flag, species already owned, PC training stock
  • 0–11 · move (4) × target (enemy left, enemy right, partner)
  • 12–16 · switch to bench slot 1–5 (slot 0 is always the active Pokémon)
  • 17 · throw a ball (wild battles only)

Masking removes: moves with no PP, damaging moves the target is immune to (0× effectiveness), switches while trapped (Shadow Tag, Arena Trap, Magnet Pull, wrap, Mean Look, with the ability exceptions), bench switches after the per-battle cap, and throws with no balls or a trainer on the other side. The 14 extras were appended after the original 417 values, so older checkpoints could be extended instead of retrained.

Input driver

The model emits an action id; the Lua side turns that into the exact button presses the game needs.

  • Tasks · move, switch (voluntary and forced) and throw are small state machines with a cooldown between taps and a 1200-frame timeout.
  • Menus · The action and move menus are 2×2 grids (bit 0 = column, bit 1 = row), so gridStep() picks LEFT/RIGHT/UP/DOWN from the live cursor address instead of counting presses.
  • Stuck guard · navTap() fails a task when the same key at the same cursor value repeats more than five times.
  • Prompts · Evolution scenes, "forget a move?", "stop learning?", level-up boxes and "use next Pokémon?" are answered by opcode, so no human is needed between fights.
  • Watchdog · If the battle script sits on one opcode and controller for 600 frames, it presses B.

Reward design

Win or lose stays dominant at ±100. Shaping is sized so a whole fight of it adds up to roughly 10–30.

TermValueWhy
Damage50 × (dealt − taken)Summed over team HP fractions, so switching never changes the reward by itself
Win / loss+100 / −100Dominates all shaping
Win HP bonusup to +20Share of team HP kept versus battle start
KO±5Per enemy faint (+) and per own faint (−)
Turn cost−1A one-turn kill beats a three-turn kill; stalling is never free
Voluntary switch−3, escalating3 → 4.5 → 6 → 7.5, hard cap of 4 bench switches; forced and refused switches are free
Level growth100 × Σ ln(level) gainA level on an L30 Pokémon pays about 3× a level on an L88 one
Ball used−1Per ball spent in the fight
Scale× 0.01Keeps the critic loss from being dominated by outcome bonuses

Switching to a lower-level bench member is free while the lead is 15 or more levels above the enemy. That is exactly the situation where the bench needs the XP, and it is what stops one Pokémon from outgrowing the rest of the party.

Catching

Catching started as a scripted policy and became a learned action once the reward could tell a useful catch from ball spam.

  • Action · THROW_ACTION (17) is masked outside wild battles, with no balls, or after 4 throws in one battle.
  • Value · A training candidate (at least 12 levels below the party average) pays +80, anything else +10. Once the PC already holds 6 candidates, more are worth 25% as much.
  • Bonuses · +15 for a species not in the Pokédex yet, +10 when the enemy was under 50% HP (Gen 3 catch rate rises as HP falls).
  • Bridge · The throw task opens the bag, switches to the Balls pocket, scrolls to the chosen ball by reading gBagPosition, then presses Use, and retries if the bag stays open.

The old CatchPolicy is still in the repo behind SCRIPTED_CATCH = False, which makes it easy to compare a hand-written rule against what the model learns.

Overworld autopilot

Scripted, never learned, and not part of the observation. Its only job is to put the agent in front of the next battle.

  • Hunt · Plans a path over the live map grid to a wild-encounter tile, using the game's own encounter flag so grass, long grass, ash, caves and sand all count.
  • Heal · Goes to a Pokémon Center first when team HP is under 35% or fewer than two Pokémon can still fight. Healing before a wipe beats a whiteout, which costs money and respawns somewhere else.
  • Terrain · Warps, ledges and directional walls are classified from metatile behaviour. Cliffs are one-way, so the planner checks both the tile being left and the tile being entered.
  • Nurse · Found by sprite id, so it works in any Center without hardcoded coordinates.
  • Recovery · The env retries five times with a delay, then pauses 30 seconds and runs the whole round again, so a stuck cutscene never ends the run.

Deep dive

Bugs that cost real time, with what the symptom looked like, the actual cause, and the fix.

  • Symptom

    Entropy fell fast and the agent picked the same move every turn, whatever the matchup.

    Diagnosis

    Training against trivially easy, underleveled wild encounters gave no differentiating reward signal. Any action won, so nothing was worth learning.

    Fix

    Switched to level-matched opponents. Policy distillation on real battle logs confirmed genuinely branching, type-aware decisions afterward.