Mathematical Scientist for AI Safety Research

Montreal

Frontier AI companies are throwing billions of dollars into scaling existing architectures and methods such as next-token-prediction, direct preference optimization (DPO), reinforcement learning with human feedback (RLHF), and reinforcement learning with verified rewards (RLVR). These methods are very powerful, yet fundamentally flawed, resulting in misalignment, sycophancy, systematic biases, and other forms of harmful behavior that are already having severely negative consequences in our society.

LawZero is a non-profit founded by Yoshua Bengio developing a fundamentally new approach, the Scientist AI. We are inspired by the fact that scientific theories are both generally useful and , unlike an untrusted agent, equivariant to the consequences of their use (for a broad overview, see our blogpost ). We aim not only to build a novel, safe-by-design system, but to construct a theoretical blueprint ...