AI Outperforms Law Professors in Stanford Law Study
Key takeaways
- The study, titled “Law Professors Prefer AI Over Peer Answers,” was conducted with 16 law professors across U.S.
- This study challenges important assumptions about AI s role in legal education, said Nyarko, who leads Stanford Law School’s Legal Innovation through Frontier Technology Lab, or liftlab.
- The study is particularly notable because previous AI evaluations have focused primarily on subjects with clear right-or-wrong answers.
A groundbreaking study led by Stanford Law School Professor Julian Nyarko reveals that law professors overwhelmingly prefer AI-generated answers to student questions over responses written by their fellow instructors—a finding that could reshape how legal education is delivered.
The study, titled “Law Professors Prefer AI Over Peer Answers,” was conducted with 16 law professors across U.S. law schools and tested whether large language models could serve as effective tutors for contract law courses.In a blind evaluation of nearly 3,000 anonymized comparisons, professors rated AI responses significantly higher than answers written by other professors, with AI winning 75% of head-to-head matchups.
This study challenges important assumptions about AI s role in legal education, said Nyarko, who leads Stanford Law School’s Legal Innovation through Frontier Technology Lab, or liftlab. He co-authored the paper with colleagues from Yale, NYU, University of Chicago, and other leading institutions. We focused on law precisely because it requires judgment, nuanced reasoning, and the ability to navigate ambiguity—not just factual recall.