Citations Needed: Magic Encyclopedias to Save the World

Last week FLF launched a competition “to find the best workflows and methodologies for using AI to produce reliable, trustworthy knowledge bases”. I had (and have ongoing) a substantial role in that effort. Why do I think it’s so important? It’s a lot of reasons actually! I’ll gesture at a few here.Conjuring a magic encyclopedia For now, assume with me that it can be done. Wish away with me the various technical and financial challenges. Great! Now we can rapidly conjure up a deeply, fully researched knowledge base on any topic. All claims point back to who’s said them, in what context, and (importantly) with what justifications and evidence (if any). Any quibbles or nuances which have been expressed on a point are similarly readily available. It’s not opinionated: all competing viewpoints with their associated justifications are associated and comparable.That’s way, way too much information! Imagine trying to read everything ever about diet or shipping or taxes or microbes. It’s not happening. So as well as this, we now magically have tools which gather similar points together, summarise, and can make a decent stab at which points we’ll consider most or least relevant. We can dig deeper (or send AI agents to scout deeper) as desired. And when new interesting and informative content arises, or in contexts where nuance and clarification are helpful, it can be bubbled up to our attention.All this is doable today: enough web searches, enough cross-referencing of tweets, articles, journals, following of citation chains, gathering and comparing of hypotheses and points of view, etc. will make progress. But it’s exhausting. When someone does go to those lengths, their partial — but heroic — efforts to map out what’s been said often languish either unpublished or unrecognised.Don’t we already have this? A shining example is Wikipedia, where the collective curatorial effort of a wide range of editors gradually maps out an expanding core of topics and commentary. But Wikipedi