PokeGen3-RL
RL agent that plays Gen 3 Pokémon battles by reading emulator memory directly. A Lua bridge inside mGBA drives the game; a Gymnasium env handles masking, rewards and an overworld autopilot.
Overview
A reinforcement-learning agent that plays Gen 3 (Emerald) battles by reading emulator memory directly. No screen pixels: the game state is read straight out of RAM and keypresses are driven by a Lua script inside mGBA.
- Actions
- 18
- Observation
- 431 floats
- Battler slots
- 4 (doubles-ready)
- Transport
- TCP, line-based
The agent only ever plays the battle. Everything between fights (walking to grass, healing at a Pokémon Center, clicking through level-up and evolution screens) is scripted in the bridge, so training runs unattended for as long as the emulator stays up.
Architecture
Two processes talk over localhost. The Lua bridge owns the emulator; the Python env owns the learning loop.
mGBA (Lua) Python
┌───────────────────────┐ :8888 ┌──────────────────────┐
│ battle_bridge.lua │◄────────► │ env.py (Gymnasium) │──► PPO
│ · RAM reads │ │ · observation, mask │
│ · phase machine │ │ · reward shaping │
│ · key-press driver │ :8889 └──────────────────────┘
│ · overworld nav │◄───────── address finder (RAM dumps)
└───────────────────────┘| Command | What it does |
|---|---|
| GET_STATE | One comma-separated line: 4 battlers, both parties, party levels, weather, flags |
| STATUS | Current phase and the result of the last task (OK / FAILED / ABORTED) |
| ACTION:n | Queue move / switch n; replies ACK or REJECT:reason |
| THROW[:ball] | Open the bag and throw a ball (wild, single battles only) |
| NAV:HUNT | HEAL | Scripted overworld walk to encounter tiles or the Pokémon Center |
| INVENTORY / BOX / DEX | Read-only catch observability: bag, PC, Pokédex owned bits |
| AUTO:ON | OFF | Kill switch (also the L button in the emulator) |
The env never sleeps for a fixed time. The bridge derives a phase from callback2, the per-battler controller function and the battle outcome (NOT_IN_BATTLE, CHOOSE_ACTION, FORCED_SWITCH, BUSY, WAITING, BATTLE_OVER), and the env polls STATUS until the game is actually asking for input. Animations and text boxes are just WAITING.
Reading the game
emerald_config.lua is the single address map. Everything else reads through it, which is what made the address bugs in the deep dive cheap to fix.
| Symbol | Address | Used for |
|---|---|---|
| gBattleMons[4] | 0x02024084 | 4 × 0x58 bytes: HP, PP, moves, types, status, stat stages, ability |
| gPlayerParty / gEnemyParty | 0x020244EC / 0x02024744 | 6 × 0x64 bytes, encrypted substructures |
| gBattleTypeFlags | 0x02022FEC | double / trainer bits |
| gBattleOutcome | 0x0202433A | 0 = ongoing; won, lost, caught, forfeited |
| gBattleWeather | 0x020243CC | rain / sand / sun / hail masks |
| gMain.callback2 | 0x030022C4 | which screen the game is on |
| gBattlerControllerFuncs | 0x03005D60 | is the player waiting on action, move or target input |
Party Pokémon store species, item and moves in four 12-byte substructures that are XOR-encrypted and shuffled by personality value. The bridge decrypts them in Lua, so the env gets the real moveset of every bench Pokémon and of the whole enemy party.
local key = pid ~ emu:read32(base + 4) -- personality xor OT id
local order = block_orders[(pid % 24) + 1] -- 24 possible substructure orders
-- growth block: species + item, attacks block: the four moves
local g0 = emu:read32(growthAddr) ~ key
species, item = g0 & 0xFFFF, g0 >> 16Observation & actions
A flat 431-float vector and 18 discrete actions, with a legality mask so the policy never wastes a turn on something the game will refuse.
| Part | Size | Contents |
|---|---|---|
| 4 battlers | 4 × 47 | HP, PP, move id/type/power/accuracy/class/priority, effectiveness vs target, status, stat stages, species, ability |
| Weather | 1 | none / rain / sand / sun / hail |
| Bench | 12 × 19 | both parties: HP, species, item, move types, effectiveness vs current enemy, power, accuracy |
| Extras | 14 | party levels, level gap, min/mean level, balls left, wild flag, species already owned, PC training stock |
- 0–11 · move (4) × target (enemy left, enemy right, partner)
- 12–16 · switch to bench slot 1–5 (slot 0 is always the active Pokémon)
- 17 · throw a ball (wild battles only)
Masking removes: moves with no PP, damaging moves the target is immune to (0× effectiveness), switches while trapped (Shadow Tag, Arena Trap, Magnet Pull, wrap, Mean Look, with the ability exceptions), bench switches after the per-battle cap, and throws with no balls or a trainer on the other side. The 14 extras were appended after the original 417 values, so older checkpoints could be extended instead of retrained.
Input driver
The model emits an action id; the Lua side turns that into the exact button presses the game needs.
- Tasks · move, switch (voluntary and forced) and throw are small state machines with a cooldown between taps and a 1200-frame timeout.
- Menus · The action and move menus are 2×2 grids (bit 0 = column, bit 1 = row), so gridStep() picks LEFT/RIGHT/UP/DOWN from the live cursor address instead of counting presses.
- Stuck guard · navTap() fails a task when the same key at the same cursor value repeats more than five times.
- Prompts · Evolution scenes, "forget a move?", "stop learning?", level-up boxes and "use next Pokémon?" are answered by opcode, so no human is needed between fights.
- Watchdog · If the battle script sits on one opcode and controller for 600 frames, it presses B.
Reward design
Win or lose stays dominant at ±100. Shaping is sized so a whole fight of it adds up to roughly 10–30.
| Term | Value | Why |
|---|---|---|
| Damage | 50 × (dealt − taken) | Summed over team HP fractions, so switching never changes the reward by itself |
| Win / loss | +100 / −100 | Dominates all shaping |
| Win HP bonus | up to +20 | Share of team HP kept versus battle start |
| KO | ±5 | Per enemy faint (+) and per own faint (−) |
| Turn cost | −1 | A one-turn kill beats a three-turn kill; stalling is never free |
| Voluntary switch | −3, escalating | 3 → 4.5 → 6 → 7.5, hard cap of 4 bench switches; forced and refused switches are free |
| Level growth | 100 × Σ ln(level) gain | A level on an L30 Pokémon pays about 3× a level on an L88 one |
| Ball used | −1 | Per ball spent in the fight |
| Scale | × 0.01 | Keeps the critic loss from being dominated by outcome bonuses |
Switching to a lower-level bench member is free while the lead is 15 or more levels above the enemy. That is exactly the situation where the bench needs the XP, and it is what stops one Pokémon from outgrowing the rest of the party.
Catching
Catching started as a scripted policy and became a learned action once the reward could tell a useful catch from ball spam.
- Action · THROW_ACTION (17) is masked outside wild battles, with no balls, or after 4 throws in one battle.
- Value · A training candidate (at least 12 levels below the party average) pays +80, anything else +10. Once the PC already holds 6 candidates, more are worth 25% as much.
- Bonuses · +15 for a species not in the Pokédex yet, +10 when the enemy was under 50% HP (Gen 3 catch rate rises as HP falls).
- Bridge · The throw task opens the bag, switches to the Balls pocket, scrolls to the chosen ball by reading gBagPosition, then presses Use, and retries if the bag stays open.
The old CatchPolicy is still in the repo behind SCRIPTED_CATCH = False, which makes it easy to compare a hand-written rule against what the model learns.
Overworld autopilot
Scripted, never learned, and not part of the observation. Its only job is to put the agent in front of the next battle.
- Hunt · Plans a path over the live map grid to a wild-encounter tile, using the game's own encounter flag so grass, long grass, ash, caves and sand all count.
- Heal · Goes to a Pokémon Center first when team HP is under 35% or fewer than two Pokémon can still fight. Healing before a wipe beats a whiteout, which costs money and respawns somewhere else.
- Terrain · Warps, ledges and directional walls are classified from metatile behaviour. Cliffs are one-way, so the planner checks both the tile being left and the tile being entered.
- Nurse · Found by sprite id, so it works in any Center without hardcoded coordinates.
- Recovery · The env retries five times with a delay, then pauses 30 seconds and runs the whole round again, so a stuck cutscene never ends the run.
Deep dive
Bugs that cost real time, with what the symptom looked like, the actual cause, and the fix.
Symptom
Entropy fell fast and the agent picked the same move every turn, whatever the matchup.
Diagnosis
Training against trivially easy, underleveled wild encounters gave no differentiating reward signal. Any action won, so nothing was worth learning.
Fix
Switched to level-matched opponents. Policy distillation on real battle logs confirmed genuinely branching, type-aware decisions afterward.