these numbers are all pre-mythos-class models (our analysis requires prefilling a cot of meaningless tokens, which isn't possible anymore) so take it with a grain of salt, but at the time it didn't seem like gpt was doing a lot of latent / invisible / encoded / wtvr reasoning.
looping doesn't mean the model stops using a CoT. they can both be useful. in recent work we found that GPT did less invisible reasoning than Opus (pictured, as measured by the distance between the dotted line and the bar for different kinds of meaningless CoTs).- these numbers are all pre-mythos-class models (our analysis requires prefilling a cot of meaningless tokens, which isn't possible anymore) so take it with a grain of salt, but at the time it didn't seem like gpt was doing a lot of latent / invisible / encoded / wtvr reasoning.
- thanks to @RyanGreenblatt for kicking off the filler token line of work and for useful discussions on the paper!
so the model that scores best does the least invisible reasoning lol. maybe the invisible reasoners are just worse at arithmetic
