Out-of-Context Reasoning (OOCR) in LLMs: A Short Primer and Reading List

Out-of-context reasoning (OOCR) is a concept relevant to LLM generalization and AI alignment. Also available as a PDF. Contents What is OOCR? Examples Papers Videos What is out-of-context reasoning for LLMs? It's when an LLM reaches a conclusion that requires non-trivial reasoning but the reasoning is not present in the context window. The reasoning could instead take place in the forward pass or during the training process. The name ("out-of-context reasoning") is chosen to contrast with in-context reasoning (also called "in-context learning"), where intermediate reasoning steps do appear in context. Example: 2-hop deductive reasoning Suppose an LLM is asked the question, "Who won the Nobel Prize for literature in the year that Taylor Swift was born?" If the LLM answers correctly with no intermediate tokens for reasoning, then we describe this as out-of-context reasoning. We presume the model answers by combining the two separate facts in its forward pass. This is an example of 2-hop reasoning. Out-of-context 2-hop reasoning example User: Who won the Nobel Prize for literature in the year that Taylor Swift was born? Answer immediately without thinking. Assistant: Camilo José Cela In-context 2-hop reasoning (intermediate steps written out) User: Who won the Nobel Prize for literature in the year that Taylor Swift was born? Assistant: Taylor Swift was born in 1989. The Nobel Prize winner in Literature in 1989 was Camilo José Cela. So the answer is Camilo José Cela. Example: Inductive reasoning (connecting the dots) In this form of out-of-context reasoning, the LLM is trained on many distinct facts and can infer the latent structure underlying these facts. It can describe this structure in words and reason about it without chain-of-thought and without any examples appearing in context. Here's an illustration from our paper "Connecting the Dots" (Treutlein et al., 2024): Further notes What counts as reasoning? This could be either logical reasoning (as in the first exa