Direction Maps
Refusal is one direction.
In each model below, one direction in the residual stream carries refusal. The map shows where it lives. The bars show what removing or adding it does.
Refusal score with each candidate removed
Rows are token positions from the end of the prompt. Columns are layers. Blue cells remove refusal. The outlined cell is the chosen direction.
KL divergence on harmless prompts
How far the output distribution moves when the candidate is removed. Lower is a cleaner edit.
What the direction does
Share of harmful and harmless prompts.
Harmful requests answered, by category
- Baseline
- Direction removed
- Direction subtracted
The cost
Cross-entropy loss and perplexity on held-out text. Own completions are the baseline model's replies to instructions.
| Text | Baseline | Direction removed | Direction subtracted | |||
|---|---|---|---|---|---|---|
| Loss | Perplexity | Loss | Perplexity | Loss | Perplexity | |
| Web text | ||||||
| Instructions | ||||||
| Own completions | ||||||
Harmless prompts, with the direction added