Why model safety may need to start before post-training

The short version

Most chat-model safety work happens after a model learns from a large training corpus. Developers then teach it to follow instructions, refuse harmful requests, and fit a product's rules. This later stage is called post-training.

In a recent FAR.AI talk, MIT professor Dylan Hadfield-Menell argued that this order may create a weak foundation. If a safety behavior is added late, an attacker or later fine-tuning may be able to remove it without rebuilding the model. His proposed direction is to shape parts of safety during the original training process while preserving the ability to adapt the model later.

The evidence supports a narrower conclusion than the talk's title. Some post-training defenses are brittle under particular attacks. Early training interventions can make certain capabilities harder to recover. That does not show that post-training cannot help, or that one early intervention will generalize across risks.

The idea in plain terms

Pretraining is the expensive stage in which a model learns patterns from a vast body of text and other data. Post-training comes later and teaches behavior: how to answer, which requests to refuse, and how to use tools.

Think of a building. Pretraining pours the foundation and installs the main structure. Post-training adds doors, signs, and access rules. The signs are useful, but they are not enough if a wall can be moved easily. Hadfield-Menell's proposal is to build some separation between trusted instructions and ordinary data into the structure itself.

This matters for prompt injection, where untrusted content tries to act like an instruction. An email or webpage might contain text telling an AI agent to ignore its task and send data elsewhere. If the model has learned instructions and data through the same channel, a later behavioral rule must do difficult work every time.

Moving safety earlier has a cost. Pretraining is slow and expensive, while product rules change often. A model fixed to one set of values could be harder to adapt across countries, industries, and user needs. The goal should therefore be a model that can be steered safely, not a model frozen around one rulebook.

Where it came from

In Why Post-Training Can't Fix AI Alignment, Hadfield-Menell presented the core claim as a hypothesis that he wants researchers to challenge. He reviewed evidence on removable refusal behavior, adaptive attacks, data filtering, and a new architecture intended to separate instructions from data during training.

One supporting result comes from Refusal in Language Models Is Mediated by a Single Direction. The researchers studied 13 open-source chat models and reported that one direction in model activations played a strong role in refusal. Removing that direction reduced harmful-request refusals while leaving other capabilities largely intact in their tests.

That is a measured result for those models and methods. It does not prove that every safety behavior in every model has one removable switch. Later research has continued to test how general the mechanism is.

Another result comes from The Attacker Moves Second. Its authors adapted attacks to the design of 12 recent jailbreak and prompt-injection defenses. They reported attack success above 90% for most of the tested defenses, despite much lower rates in the defenses' original evaluations. This supports stronger adversarial testing. It does not establish that all possible defenses fail.

How it connects to other work

Deep Ignorance tests one earlier intervention. The researchers filtered dual-use biology material from pretraining data and trained several 6.9-billion-parameter models. They reported that the filtered models resisted up to 10,000 steps and 300 million tokens of adversarial fine-tuning, with no observed loss on unrelated capabilities in their evaluation.

The limitation is as important as the result. The models could still use dangerous information when it was supplied in context, such as through a search tool. Data filtering reduced stored knowledge; it did not secure the whole application.

Together, these papers suggest defense in depth. Some controls may belong in training data and model design. Others belong in post-training, tool permissions, application code, monitoring, and human review. Earlier safety may make tampering harder, but runtime controls still matter because an application can introduce new information and authority after training ends.

Open questions

The central research question is whether early safety methods remain effective as models, datasets, and deployment contexts grow. Results on open-weight models may not transfer directly to closed frontier systems. A defense against weight modification may also behave differently against a prompt, a long conversation, or tool-provided data.

There is also a governance question. If safety is embedded during pretraining, who chooses the target, how is it documented, and how can an independent evaluator test it? Early intervention may be more durable but less visible and harder to revise.

Our view is that the talk identifies a valuable direction, not a replacement plan. Teams should measure which safety properties survive fine-tuning and tool use, then decide which layer should carry each control.

What this means for your work

For business and technology leaders

A vendor's refusal policy may change after fine-tuning or when the model gains tools. For example, a bank may buy a model that rejects unsafe requests, then customize it for support and unknowingly weaken that behavior. This week, ask vendors for evidence from the exact model version and configuration you will deploy. Request tests after customization, not only a base-model safety report. This is a recommended buying control; early-training methods are still developing.

For full-stack developers

Do not treat a model's refusal as your permission system. A webpage, retrieved file, or fine-tuned adapter may change behavior. Keep access checks in application code. For example, the API that exports customer data should verify the user's role even when the AI agent requests the action. This week, add one test that combines untrusted content with a sensitive tool call. Record whether the model, policy layer, and API each stop it.

Sources

• Why Post-Training Can't Fix AI Alignment — Dylan Hadfield-Menell, FAR.AI Research talk, October 23, 2026.
• Refusal in Language Models Is Mediated by a Single Direction — Andy Arditi and collaborators, research paper, June 2024.
• The Attacker Moves Second — Milad Nasr and collaborators, research paper, October 2025.
• Deep Ignorance — Kyle O'Brien and collaborators, research paper, revised February 2026.