Alignment
RLHF · post-training · safety tuning
In short
Alignment is the process of shaping a trained model's behaviour so it follows instructions, tells the truth as best it can, and declines harmful requests. It happens after pre-training, mainly through supervised fine-tuning on demonstrations and reinforcement learning from human or AI feedback.
A model straight out of pre-training predicts text. It does not follow instructions, does not prefer truth to plausibility, and has no notion of what it should decline. Alignment is what turns that into an assistant.
The standard pipeline has two stages. Supervised fine-tuning trains on curated examples of good responses. Preference optimisation — RLHF, or methods such as DPO that skip the separate reward model — trains the model to prefer responses people rated higher.
Constitutional approaches substitute a written set of principles for much of the human labelling: the model critiques and revises its own outputs against stated rules, producing preference data at scale.
Two limits are worth being clear about. Alignment is behavioural, not epistemic — it makes a model act helpfully and refuse appropriately; it does not give it a mechanism for knowing whether a claim is true, which is why hallucination persists. And it is contested: whose values, which refusals, and how much caution before a model becomes useless are questions with no technical answer.
The research programme extends well beyond product behaviour — interpretability, evaluation of dangerous capabilities, scalable oversight — but in commercial use, "aligned" usually means the narrower thing described above.
Frequently asked questions
- What is RLHF?
- Reinforcement learning from human feedback: people rank alternative model responses, a reward model is trained on those rankings, and the model is optimised against it. Variants replace some human rankings with AI-generated ones.
- Does alignment make a model less capable?
- It can, at the margins — over-cautious refusal of benign requests is the usual complaint. Balancing helpfulness against caution is an explicit tuning trade-off, not an accident.