Manish Shetty

@slimshetty_

AI capability evals @METR_Evals | Previously PhD @ UC Berkeley, research @googledeepmind, @msftresearch

Berkeley

manishs.org

Replies Reposts Media Articles

Manish Shetty
@slimshetty_
Sep 29
A GSO bench update after a long time. We're close to saturation, particularly because I expect at least ~5% of tasks in any benchmark (including my own) to have issues.
5
Manish Shetty
@slimshetty_
Aug 6
good day to revisit this banger from @idavidrein
david rein
@idavidrein
Apr 18
Article
AI Companies as 'hot springs' for growing AIs
When people think about rogue misaligned AI, often the story is about it 'escaping containment', getting out onto the internet, and acquiring resources in the wild. But AI labs have (a) tons of...
Manish Shetty
@slimshetty_
Jul 22
I'm curious about AIs optimizing model training due to its direct implications for RSI. It seems productive to study the dynamics of algorithmic efficiency wins humans vs agents get given enough expenditure. I think Expenditure Horizon helps with this, and I'm excited it's out.
METR
@METR_Evals
Jul 21
Introducing “expenditure horizon”: a proposed method for measuring AI capabilities on continuously-scored problems. The method compares performance as a function of spend for humans vs agents. The point where humans become more cost-effective is the agent’s expenditure horizon.
1
Manish Shetty
@slimshetty_
Jul 13
i personally find RSI (models) extremely daunting and complex to think about. i really love this attempt at simplifying from @ElasticityInst. thinking of AI production as a graph of elasticities between inputs and outputs with feedback loops (w/o specific modeling decisions) is
tom cunningham
@testingham
Jul 13
Very happy to share the first paper from @ElasticityInst: The Economics of Recursive Self-Improvement. Two parts: (1) a graphical representation of feedback loops, to formalize a variety of RSI-related arguments, where each arrow represents responsiveness (elasticity); (2) a
Manish Shetty
@slimshetty_
May 19
Incredible work from my colleagues at METR. This report makes me feel more hopeful about our ability to keep generating evidence on AI capabilities and risks. So cool to be working at a place that doesn't shy away from setting the standard *every* single time!!
METR
@METR_Evals
May 19
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
1