AI is Not Normal Technology

Last year, Arvind Narayanan and Sayash Kapoor published a now well-circulated essay, AI as Normal Technology. The essay is popular, I think, primarily because people would like it to be true, myself included. It is a terrifying proposition to acknowledge how different AI may be from “normal” technology. However, acknowledge it we must –– the urgency of developing proper governance and technical progress on alignment cannot be overstated.I think the essay has significant flaws. This response walks through the most important ones in an effort to demonstrate the importance of acknowledging the tremendous nature of the changes superintelligent AI may bring.This is a linkpost, with minor modifications, for a blog post published yesterday. Post-publishing, I saw Scott Alexander's response and realized that I overlap with him substantially on some of the arguments. I'm posting regardless because where we overlap my response tends use alternative or updated examples. I also push on points that Alexander does not, including that biosecurity as the most concrete catastrophic risk, offering chess history as a parallel for how Narayanan and Kapoor's argument may age, and examining now-falsified or soon-to-be falsified empirical predictions in the essay about forecasting and persuasion. I invite pushback, particularly on the discussion of labor and how significant current uplift from models is.The essay consists of Narayanan and Kapoor laying out a scenario they deem to be the median outcome, in four parts, roughly as follows:Part I: Speed. They argue that transformative economic and societal impacts will be slow and there is a distinction between AI methods, AI applications, and AI adoption, including different timescales.Part II: Division of labor. They discuss potential division of labor in a manner they say does not apply to “superintelligent” AI, which they “view as incoherent as usually conceptualized.” They argue control is primarily in the hands of people and organizatio