Human-Guided Agentic Research: A Research Agenda

We need to study how humans can best interpret and guide research performed by autonomous agents when those agents lack taste, tacit knowledge or competence, or may try to reward hack, sandbag or sabotage such research. This is one attempt to define the problem and the shape of potential solutions.A Story About the Future of Research Imagine yourself a year or two in the future. Recursive self-improvement (RSI) is accelerating. Agents work in swarms independently for days or weeks at a time doing research.You work in a frontier lab doing AI safety research. You sit in front of your computer and click into the input box, ready to kick off a new project. What do you type?“Solve AI alignment”? Beware giving a magic genie vague wishes.Think about that again: what exactly do you type? How do you know what you type is the best way to prompt this agent swarm into doing your bidding?When the lead agent comes back a week later, what exactly does that output look like? How do you use that output to launch the next phase of the project?How will you validate that output to ensure the agent hasn’t reward hacked, sabotaged or incompetently explored the research space?How will you know what key decisions the agent made? Which research paths they explored? Which research paths they intentionally or unintentionally left unexplored?How will you know how faithful the research the agent did was to the goal you gave it?Today these questions are productivity issues—soon they will be safety issues. If we don’t start researching how humans can effectively interpret and direct agentic research, we lose the ability to make safety claims about that research.Recursive Self-Improvement Is HereRecursive self-improvement (RSI) has already started.[1] Models working within agent harnesses are being used to develop the next model, accelerating AI development.Anthropic