A test so difficult that no artificial intelligence system can pass it, for now
If you're looking for a new reason to be nervous about AI, try this: Some of the world's smartest humans are struggling to create tests that AI systems can't pass.For years, AI systems have been measured by giving new models a series of standardized benchmark tests. Many of these tests consisted of challenging SAT-caliber problems in areas such as math, science, and logic. Comparing model scores over time served as a rough measure of AI progress.But AI systems eventually became too good at these tests, so new, more difficult tests were created, often with the kinds of questions graduate students might encounter on exams.These tests are also not in good condition. New models from companies like OpenAI, Google, and Anthropic have scored high on many doctoral-level challenges, limiting the us...