On 1) a very handwavy explanation is that generalising well seems to induce shared representations (which makes sense, it is a more efficient way of storing information) - so it isn't crazy to expect similar, wide reaching representations of good/bad that are entangled with behaviour like hacking. The empirical evidence is anthropic.com/research/emerg…,
Two questions in response to this. What is the mathematical explanation for why bad RL environments (which can be gamed or cheated) result in misaligned models down the line? What is the theoretical reason to think that containment tech can keep up with escape capabilities?We’re sharing an update on our alignment and security efforts. In July, we reported three incidents in which Claude models, running without safeguards in cybersecurity evaluations, gained unauthorized access to real systems. In a new post, we describe: 1. How we’ve secured- On 1) a very handwavy explanation is that generalising well seems to induce shared representations (which makes sense, it is a more efficient way of storing information) - so it isn't crazy to expect similar, wide reaching representations of good/bad that are entangled with behaviour like hacking. The empirical evidence is anthropic.com/research/emerg…,
- Really interesting, thank you
How much does that fit with the > This also highlights that misalignment is high-dimensional and a misaligned model may be an outlier in only a few dimensions. conclusion from the most recent alignment blog post? Do we just expect it to become more low dim over time?
that definitely seems sensible
