Reinforcement learning towards broadly and persistently beneficial models

This is an unofficial automated linkpost. We find that reinforcement learning on realistic scenarios targeting beneficial traits can produce broad improvements across dozens of benchmarks measuring aligned and beneficial behavior. These alignment gains generalize beyond the domains used for training and persist under adversarial pressure. As AI systems become more capable and autonomous in high-stakes settings like health, science, education, and coding, they will need to remain helpful, honest, transparent, and safe in situations they have not seen before. This requires generalizing to new contexts, new pressures, longer and more complex interactions, and across domains that differ from those seen during training. A growing body of research has shown that misalignment can sometimes generalize in this way. Models trained to exhibit narrow forms of problematic behavior, such as writing insecure code or cheating in realistic scenarios, can begin to behave badly in broader settings unrelated to the original training task. This phenomenon, emergent misalignment, suggests that training on a narrow behavior in one setting can sometimes produce much broader changes in model behavior that extend beyond the training distribution. Continue reading at alignment.openai.com →Discuss