A tabular Q-learning agent on the 4×4 gridworld. Nothing is trained when this page loads — the agent starts knowing nothing, and every frame you play is one more episode it actually lives through. Press Play, or step with −1 / +1, and watch value spread backwards from the goal. The How it works tab explains the world, thealgorithm and what to look for.
Every number the agent knows. Hover or tab to a cell for its value and how many times it has been tried.
| state | up | down | left | right |
|---|
Shading is maxa Q(s,a); the arrow is the action the agent would take. A dot means it has no opinion yet.
Q-value / V(s)
A 4×4 grid. The agent starts in the top-left corner and is trying to reach the goal in the bottom-right. One square is a wall it cannot enter; one is a pit that ends the episode badly.
| START(0,0) | (0,1) | (0,2) | (0,3) |
| (1,0) | WALL(1,1) | (1,2) | PIT−10 |
| (2,0) | (2,1) | (2,2) | (2,3) |
| (3,0) | (3,1) | (3,2) | GOAL+10 |
Four actions are available everywhere: up, down, left, right. The rules:
| Event | Reward | Episode |
|---|---|---|
| Ordinary move | −1 | continues |
| Walk into a wall or off the edge | −1 | continues (you don't move, but you still pay) |
| Enter the goal | +10 | ends |
| Enter the pit | −10 | ends |
That −1 per step is doing real work. Without it, any path that eventually reaches the goal would be worth the same as the shortest one, and the agent would have no reason to hurry. It's what turns "find the goal" into "find the goal quickly".
The agent is never told any of this. It doesn't get the map, the location of the goal, the rules of the wall, or the fact that a pit exists. All it ever observes is a stream of four-part experiences: I was in s, I did a, I received r, I ended up in s′. Everything in the table on the Simulation tab was inferred from that stream alone.
One episode is a single run: drop the agent at the start, let it act until it stops. Episodes end in three ways, but only two of them are real:
How endings shift as the agent learns, measured over the default 1000-episode run:
| Episodes | Goal | Pit | Ran out of steps |
|---|---|---|---|
| 1–50 | 42% | 42% | 16% |
| 51–200 | 80% | 20% | 0% |
| 201–600 | 96% | 5% | 0% |
| 601–1000 | 99% | 1% | 0% |
The median episode across the whole run is 6 steps — the optimal path. Early on, episodes of 40+ steps are common; all eight truncations happen in the first fifty episodes.
Termination and truncation are not the same thing, and conflating
them is a classic and quiet bug. At the goal there genuinely is no future, so
the value of "what comes next" is zero. At step 50 the future still exists —
zeroing it there would teach the agent that being mid-journey is worthless.
In this code the done flag comes only from the environment, so
truncation never sets it and the last update bootstraps normally. Gymnasium
splits the flag into terminated and truncated
precisely because so many implementations get this wrong.
| Term | Symbol | Here |
|---|---|---|
| State | s | which square the agent is on |
| Action | a | up / down / left / right |
| Reward | r | −1, −10 or +10 for one step |
| Next state | s′ | where that action landed it |
| Policy | π | the rule for choosing actions — what we want |
| Return | G | total discounted reward from here to the end |
The prime in s′ just means "one step later". a′ is
the odd one out: it is not the action the agent takes next, it is a bound
variable ranging over the four actions available in s′, used
only to take a maximum. In the code the whole expression
maxa′ Q(s′,a′) is simply max(Q[nextState])
— the index is never named, exactly like the i in a max over a
list.
A number in a table. 16 states × 4 actions = 64 numbers, and that table is the agent. Nothing else is stored.
Q(s,a) means: if I am in state s and take action a, and then
behave well from there on, how much total reward should I expect over the rest
of the episode?
Take Q((2,2), down) = 8.50 from a trained run. That step pays
−1, not 8.5. The 8.5 is the whole remaining journey — −1 now, then −1,
then +10 at the end, each discounted. One number summarising an entire future.
So r is the fact and Q is the judgement.
Having Q makes acting trivial: read the four numbers for your square and
take the biggest. No search, no lookahead, no simulating futures. All the
thinking was done during training and compressed into 64 numbers. The related
quantity V(s) = maxa Q(s,a) is "how good is it to be
here, assuming I then play well" — that's what shades the world map.
Four numbers per square rather than one matters. Look at (2,3)
in a trained run: down = +9.95 (one step from the goal) but
up = −6.51 (one step into the pit). It is simultaneously a great
square and a dangerous one. A single value per square could never say that.
All of Q-learning is one line, applied after every single step:
Read it as new estimate = old estimate + a small step toward something better. In code:
const bestNext = done ? 0 : maxQ(q, s2); const target = r + gamma * bestNext; q[s*4 + a] += alpha * (target - q[s*4 + a]);
r — the reward actually received. The only piece of hard
ground truth in the whole equation; everything else is estimate.maxa′ Q(s′,a′) — look at the four actions
available where you landed and take the best one's value. "How good is the
situation I've moved into, assuming I keep playing well."γ — the future arrives later, so discount it.target = r + γ·max Q(s′,a′) — the closest thing to a
correct answer we can build. It's better than what's in the table
because one real observed reward has been substituted in for one layer of
guessing.α — don't overreact. Move 10% of the way, not all of it.max, and not "the action I actually took next"That single choice is what makes Q-learning off-policy. The agent
might be exploring — about to take a random action straight into a wall — but
the update still assumes optimal play from s′ onward. So the table
converges toward the value of the best policy even while the agent
behaves badly to gather data. Replace maxa′ Q(s′,a′)
with Q(s′, a_actually_taken) and you get SARSA, which instead
learns the value of the exploring policy, mistakes included.
Square (3,2), one step left of the goal, action right.
Every time the agent takes it: reward +10, episode ends, so the
bootstrap term is zero and the target is a flat 10. Starting from
zero, each visit closes 10% of the remaining gap:
visit: 1 2 3 4 5 ... 10 ... 50 Q: 1.00 1.90 2.71 3.44 4.10 ... 6.51 ... 9.95 Q_n = 10 · (1 − 0.9ⁿ)
Now (2,2), one square further out, where the reward is always
−1. Early on it learns nothing useful: the agent steps down,
collects −1, and looks up max Q[(3,2)], which is still 0, so the
target is a pessimistic −1. Only once (3,2) has climbed does the
target become −1 + 0.95 × 10 = 8.5:
Q[(3,2)] : 0.00 1.00 1.90 2.71 ... 6.51 ... 9.95
target : -1.00 -0.05 0.81 1.57 ... 5.19 ... 8.45
↑ ↑
"this square is bad" "this square is great"
Same r in every column. The target climbs purely because the
downstream estimate improved. This is the whole mechanism: value seeps
backwards from the goal, one square per generation. It's why the start
square — six updates from the only source of truth — is always the last to
learn anything.
← means assign: read the right side, overwrite the left. It's
+=, not algebra, which is why having Q(s,a) on both
sides is no more paradoxical than count = count + 1. If it were a
real equation you could cancel and conclude the TD error is always zero — the
arrow is exactly what forbids that move.
The genuine self-reference is on the right: the target is built out of
Q too, so we're learning a guess from a guess. That isn't circular
for the same reason recursion isn't — there's a base case. At a terminal state
there is no future, max Q(s′,a′) is replaced by a hard
0, and the target collapses to pure observed reward.
Q((3,2), right) is anchored to +10 by reality alone.
Everything else eventually hangs off that anchor.
Once the updates stop moving, you're left with a real equation with Q on both sides:
Compare x = cos(x): also self-referential, also perfectly
well-posed. Iterate from anywhere — 0 → 1 → 0.540 → 0.858 → 0.654 → … → 0.739 —
and you converge on the one value that satisfies it. Q-learning does the same
to a whole table at once, using sampled transitions instead of exact
arithmetic.
γ is what guarantees it. Each sweep multiplies any remaining error by at most γ, so error shrinks geometrically and cannot orbit or blow up. This is the contraction-mapping property. Set γ = 1 in a world with no terminal state and the guarantee evaporates along with the convergence.
A satisfying check: in a converged run the start square reads
3.21, and hand-computing the discounted return of the optimal six-step
path — −1 −0.95 −0.95² −0.95³ −0.95⁴ + 0.95⁵·10 — gives
3.2134. The table isn't approximately right, it's exactly right.
| Knob | Default | What it controls |
|---|---|---|
| α learning rate | 0.1 | How far to move toward each new target. Too high and it thrashes on noise; too low and it crawls. |
| γ discount | 0.95 | How much a reward one step later is worth now. At 0 the agent is purely greedy; near 1 it plans far ahead. |
| ε exploration | 1.0 → 0.05 | Probability of ignoring what it knows and acting at random. Decays by ×0.995 each episode. |
ε is the exploration/exploitation compromise made concrete. Without the random component the agent locks onto the first mediocre path it stumbles across and never discovers the better one sitting next to it. With too much of it, it never commits. Starting at 1.0 and annealing to 0.05 means: be adventurous while you know nothing, decisive once you do.
In a deterministic world like this one you could set α = 1.0 and jump straight to the target every time. It would still work, just noisily. Small α matters in a stochastic world, where r is a random variable and the update needs to average over samples rather than let the latest one overwrite everything. Formal convergence proofs require α to decay toward zero for exactly this reason.
The Q-table shows all 64 numbers — every state as a row, every action as a column. This is the complete contents of the agent's mind.
The world map shows the same data compressed twice over: the shading
is maxa Q(s,a) and the arrow is
argmaxa Q(s,a). Both throw information away — the
Q-table knows not just what to do but how much worse the alternatives
are, which is often the more interesting part. A · instead of
an arrow means all four actions are still tied, so the agent has no opinion
yet.
Colour is a diverging scale: blue positive, red negative, neutral gray at zero. The domain is fixed at ±10 and never auto-scales, so a given shade means the same value in every frame and in both panels — an auto-scaling scale would make an untrained table look as confident as a trained one. Every value is also printed, so nothing is encoded by colour alone.
Hovering a cell also reports how many times that state-action has been tried, which is often more revealing than the value itself.
The status chip reads LEARNING when the playhead is at the frontier — the agent is living that episode for the first time — and REPLAYING when you've rewound behind it and are watching saved snapshots. Rewind moves the playhead only; it never makes the agent unlearn. Retrain is what discards the agent and starts a new one from zero.
Q((3,2), right) still
converges to 10.00 — one step from the goal, discounting can't touch it. But
the start square collapses from +3.21 to about −1.63: six steps away, a
myopic agent cannot see the goal at all. The map becomes a picture of how far
the agent's horizon reaches.Values are only correct where the agent actually goes. Compare a trained table against the true optimal values and squares on the settled route are exact, while squares off it can be wildly wrong — the one beside the pit often sits near −0.5 when its true value is about +4.4. Once ε decays the agent stops visiting it, and its estimate simply freezes at nonsense. The policy is fine, because it only needs correct values along the route it takes. But if the world changed, the agent would be confidently wrong about most of the map. That's the exploration/exploitation trade-off presenting its bill.
Tabular Q-learning needs one entry per state-action pair. That's 64 here, which is why it converges in seconds. Chess has more positions than atoms in the observable universe; a camera image has effectively infinite states. The table has to be replaced by a function approximator — typically a neural network — which is the step from Q-learning to DQN. The update rule barely changes; what changes is that the target now moves for a second reason (the network shifts under you), which is why DQN freezes a copy of the network to compute targets from.
This world is deterministic and fully observed. The same action from the same square always does the same thing, and the agent always knows exactly where it is. Relax either — add a 20% chance of slipping sideways, or hide the agent's position behind a noisy sensor — and the problem gets substantially harder. Slipping alone is a good exercise: the agent learns to give the pit a much wider berth, because the value of a square near it now includes the chance of being blown in.