Step by Step
S
State — the current situation
A snapshot representing everything relevant about the current situation the agent must make a decision in.
Example: in a game, the state might be the current board position and score.
P
Policy — the strategy mapping states to actions
The learned strategy that decides which action to take given a particular state — this is essentially what the agent is trying to learn and improve over time.
Example: a trained policy that, given a particular board position, reliably chooses moves that historically led to winning.
A
Action — the choice made
The actual decision or move the agent takes in a given state, which then affects what state it ends up in next.
Example: choosing to move a game piece to a specific square.
R
Reward — the feedback signal driving learning
A numeric feedback signal received after taking an action, which the agent uses to learn which actions and policies lead to better long-term outcomes; the goal is maximizing expected cumulative reward, not just the next single reward.
Example: gaining +1 for a winning move and -1 for a losing move, with the agent learning over thousands of games which policy maximizes total reward.
Applied Walkthrough
1
ChatGPT and Claude need to be aligned to produce outputs humans actually prefer, not just outputs that are technically fluent.
2
Human raters rank multiple candidate outputs from best to worst for the same prompt.
3
These rankings are used to train a separate reward model that learns to predict which outputs humans would prefer.
4
Reinforcement learning then optimizes the language model to maximize the reward model's score — this entire pipeline is called RLHF (Reinforcement Learning from Human Feedback), the standard technique for aligning modern LLMs.
Exam Application
Exams test whether you can name and define the core RL components (state, policy, action, reward) and whether you understand RLHF specifically as the technique connecting reinforcement learning to real-world LLM alignment — a frequently tested, highly current application. Also expect exploration-vs-exploitation questions: should the agent try new actions (explore) or stick with known good ones (exploit)?
⚠ Common Trap
The most common trap is confusing the reward model (trained from human rankings, used to score outputs) with the final language model itself (trained via RL to maximize that reward model's score). These are two separate models working together in the RLHF pipeline, not one single model.
✓ Quick Self-Check
1. What does the reward signal represent in reinforcement learning?
Feedback indicating how good or bad a particular action was, used to guide the agent toward better long-term behavior.
Tap to reveal / hide
2. What is a "policy" in reinforcement learning?
The learned strategy mapping states to actions — essentially the agent's decision-making rule.
Tap to reveal / hide
3. What does RLHF stand for?
Reinforcement Learning from Human Feedback.
Tap to reveal / hide
4. In RLHF, what do human raters actually do?
They rank multiple candidate model outputs from best to worst, and those rankings are used to train a reward model.
Tap to reveal / hide
5. Is the reward model the same thing as the final aligned language model?
No — the reward model is trained separately from human rankings, and is then used to guide reinforcement learning of the final language model.
Tap to reveal / hide